Loss Functions Part-2

-

 

This is a continuation of this

Loss Functions used for Classification 

As we all know that for regression problems we use Least square error as the loss function. Through this, we get a convex loss function and we can optimize by finding its global minimal. But when it comes to logistic regression the concept is completely changed, the Least Square error will give us a non-convex loss function with, more than one local minima. Here we get a wavy curve due to the non-linear sigmoid function used in the logistic regression hypothesis so it has multiple local minima which are bad for gradient Descent which is used to find minima. 

Cross-Entropy Loss 

This is the most common setting for classification problems. Cross-entropy loss increases as the predicted probability diverge from the actual label. An important aspect of this is that cross-entropy loss penalizes heavily the predictions that are confident but wrong. We can’t give equal weight to all false results. 




 
For Example, if the confidence is low and predicted a result at the same time if confidence is too high and predicted the result but both times the predicted is
false. The error function should use high weights for the prediction with higher confidence. The below proof of binary cross-entropy loss function.


Below is find the gradient of a complex cost function for Logistic Regression.


HInge Loss 
We use Hinge Loss for Support vector Machines. Hinge Loss trains the classifier. The difference in hinge loss is convex but not differentiable. Hinge loss penalises the wrong side of the hyperplane but as it is not differentiable so Gradient or Stochastic Gradient Descent so we use Cross entropy most of the time.

Comments

Post a Comment

Popular posts from this blog

Learning Optimization(SGD) Through Examples

-
Image
Introduction The entire aim of optimization is to minimize the cost function . We will learn more about optimization in the later sections of the paper. Batch Gradient Descent Here we sum up all examples on every iteration while performing the updates for the weight or parameters. So for every update in weights, we need to sum over all examples. The weights and bias get updated based on gradient and learning rate(n). Mainly advantages when there is a straight trajectory towards minimum and it has an unbiased estimate of gradients and fixed learning rate during training. Disadvantageous when we use vector implementation because we have to go over all the training set again and again. Learning happens when we go through all data even when some examples are reductant and with no contribution to the updating. Stochastic Gradient Descent  Here unlike Branch Gradient Descent, we update the parameters on each example so learning happens on every example. So it converges more quickly than...
Read more

Revise Key Terms and Concepts of Deep Learning in 5 minutes

-
Image
Tensor : It is a mathematical object and  can be a number, vector, matrix, or an n-dimensional array. Padding : Increase the image size shape by adding the given  amount of pixels when it is being processed by the kernel of a CNN Stride : The value determines the kernel's jumping over how many pixels while moving the input. Max - Pooling : Decrease the height and width of the output tensor from each convolution layer by replacing the max of the block. Use of DataLoader : Split the dataset into batches of data of given size and also provides the utilities like shuffling, random sampling while forming a batch. Use of Validation set : Helps in evaluating the model during training i.e adjusting hyperparameters and pick the best version of the model. By this, we can also identify the occurrence of overfitting. Can accuracy be a loss function for a classification problem? No. The accuracy is not a differential function so we cant compute the gradients as there is no mathematical for...
Read more