Hence, we can see, it's a quite higher loss cause our probability was a small number. so, if we consider our particular example, it will be

if probability is less than 1, then log gives a negative number which becomes positive after multiplying the minus in the formula.Īs we can see in the graph, log gives the negative number when probability value is less than 1 and it gives 0 when the probability value is 1. If the probability is 1, the loss is log (1) = 0. As we know, the probabilities range from 0 to 1, so, higher the probability, minimum the loss. Log loss, also known known as logistic loss or cross entropy loss, used to calculate the loss on the basis of probabilities in cross entropy formula. Σ represents the sum of expressions for all classes, since there is only 1 class here, so we defined our expression for only once here.In entropy function, we'll use the target tensor value as an index for predicted tensor y_p and get the probability, which is representing the target value, which we are predicting (which is 4 here), so f(s) = y_p = out = 0.0835. f(s) represents the probability of the output of our model, the output here is the tensor y, and we just calculated the probability y by softmax function and stored it in y_p.t represents the probability of the target class, since target tensor t has only 1 class (which is 4) so the probability will be 1/1 = 1.Now let's break down what each component represents in this example Now if you recall, our cross-entropy loss formula looked like this

Output tensor(, grad_fn=Īs we can see that the probability of each value of y is calculated, and the sum of the probabilities is exactly equal to 1. The result of each value of y will be replaced by the 0 in the tensor s. Then in for loop we have iterated all the values of y and calculated their probabilities using the softmax formula which we defined. Here we have defined our softmax function, first of all we defined a zero values tensor of the same size as our tensor y. Now suppose we have which has some equal to 1. So, when we put this in our softmax formula, the probabilities generated for our negative number are very lower as compared to the probabilities generated by positive numbers. If we look at the graph of e^x, we can see that it has output close to 0 for negative numbers, while higher positive outputs for positive number. As we know, the cross-entropy loss function accepts the output in form of probabilities, so we need to convert our output values into probabilities. The softmax function is used to calculate the probabilities of set of numbers whose sum is always equal to 1. If index 4 (as we know index 4 represents the number 4) in the y has the highest probability, that means the model's output is correct and loss will be lower, if the index 4 has lower probability, that means the model's output is incorrect and loss will be higher. T is the target which we are trying to predict in this data, in other words t is the correct output. Hence, the index 0 represents the output value for number 0, the index 1 represents the output value for number 1 and so on. Suppose y is the output of some neural network which predicts the numbers from 0 to 4. Let's first import the torch library and define our output tensor and target tensor. I'll try to explain the complete working of cross entropy loss formula by taking 2 example and solving them in PyTorch.

C is the total number of target classes.

f(s) is the probability of the predicted output.

The term t is the probability of target class.

Torch cross entropy loss