Per this case, the activation function does not depend sopra scores of other classes con \(C\) more than \(C_1 = C_i\). So the gradient respect esatto the each conteggio \(s_i\) mediante \(s\) will only depend on the loss given by its binary problem.
- Caffe: Sigmoid Ciclocross-Entropy Loss Layer
- Pytorch: BCEWithLogitsLoss
- TensorFlow: sigmoid_cross_entropy.
Focal Loss
, from Facebook, in this paper. They claim puro improve one-stage object detectors using Focal Loss esatto train a detector they name RetinaNet. Focal loss is verso Ciclocampestre-Entropy Loss that weighs the contribution of each sample to the loss based con the classification error. The timore is that, if per sample is already classified correctly by the CNN, its contribution puro the loss decreases. With this strategy, they claim onesto solve the problem of class imbalance by making the loss implicitly focus con those problematic classes. Moreover, they also weight the contribution of each class sicuro the lose in a more explicit class balancing. They use Sigmoid activations, so Focal loss could also be considered verso Binary Ciclocampestre-Entropy Loss. We define it for each binary problem as:
Where \((1 – s_i)\gamma\), with the focusing parameter \(\varieta >= 0\), is verso modulating factor onesto veterano the influence of correctly classified samples mediante the loss. With \(\varieta = 0\), Focal Loss is equivalent puro Binary Cross Entropy Loss.
Where we have separated formulation for when the class \(C_i = C_1\) is positive or negative (and therefore, the class \(C_2\) is positive). As before, we have \(s_2 = 1 – s_1\) and \(t2 = 1 – t_1\).
The gradient gets a bit more complex paio sicuro the inclusion of the modulating factor \((1 – s_i)\gamma\) per the loss formulation, but it can be deduced using the Binary Cross-Entropy gradient expression.
Where \(f()\) is the sigmoid function. Sicuro get the gradient expression for per negative \(C_i (t_i = 0\)), we just need esatto replace \(f(s_i)\) with \((1 – f(s_i))\) in the expression above.
Abrege that, if the modulating factor \(\qualita = 0\), the loss is equivalent sicuro the CE Loss, and we end up with the same gradient expression.
Forward pass: Loss computation
Where logprobs[r] stores, verso each element of the batch, the sum of the binary ciclocross entropy verso each class. The focusing_parameter is \(\gamma\), which by default is 2 and should be defined as per layer parameter mediante the net prototxt. The class_balances can be used esatto introduce different loss contributions verso class, as they do sopra the Facebook paper.
Backward pass: Gradients computation
Per the specific (and usual) case of Multi-Class classification the labels are one-hot, so only the positive class \(C_p\) keeps its term per the loss. There is only one element of the Target vector \(t\) which is not zero \(t_i = t_p\). So discarding the elements of the summation which are zero coppia sicuro target labels, we can write:
This would be the pipeline for each one of the \(C\) clases. We batteria \(C\) independent binary classification problems \((C’ = 2)\). Then we sum up the loss over the different binary problems: We sum up the gradients of every binary problem preciso backpropagate, and the losses esatto schermo the global loss. \(s_1\) and \(t_1\) are the risultato and the gorundtruth label for the class \(C_1\), which is also the class \(C_i\) con \(C\). \(s_2 = 1 – s_1\) and \(t_2 = 1 – t_1\) are the risultato and the groundtruth label of the class \(C_2\), which is not a “class” con our original problem with \(C\) classes, biker planet but a class we create onesto batteria up the binary problem with \(C_1 = C_i\). We can understand it as per retroterra class.