Loss Functions

Cross-Entropy

Cross-entropy loss, or log loss, measures the performance of a classification model whose output is a probability value between 0 and 1. Cross-entropy loss increases as the predicted probability diverges from the actual label. So predicting a probability of .012 when the actual observation label is 1 would be bad and result in a high loss value. A perfect model would have a log loss of 0.

_images/cross_entropy.png

The graph above shows the range of possible loss values given a true observation (isDog = 1). As the predicted probability approaches 1, log loss slowly decreases. As the predicted probability decreases, however, the log loss increases rapidly. Log loss penalizes both types of errors, but especially those predictions that are confident and wrong!

Cross-entropy and log loss are slightly different depending on context, but in machine learning when calculating error rates between 0 and 1 they resolve to the same thing.

Code

def CrossEntropy(yHat, y):
    if y == 1:
      return -log(yHat)
    else:
      return -log(1 - yHat)

Math

In binary classification, where the number of classes \(M\) equals 2, cross-entropy can be calculated as:

\[-{(y\log(p) + (1 - y)\log(1 - p))}\]

If \(M > 2\) (i.e. multiclass classification), we calculate a separate loss for each class label per observation and sum the result.

\[-\sum_{c=1}^My_{o,c}\log(p_{o,c})\]

Note

  • M - number of classes (dog, cat, fish)
  • log - the natural log
  • y - binary indicator (0 or 1) if class label \(c\) is the correct classification for observation \(o\)
  • p - predicted probability observation \(o\) is of class \(c\)

Hinge

Used for classification.

Code

def Hinge(yHat, y):
    return np.max(0, 1 - yHat * y)

Huber

Typically used for regression. It’s less sensitive to outliers than the MSE.

Code

def Huber(yHat, y):
    pass

Kullback-Leibler

Code

def KLDivergence(yHat, y):
    pass

MAE (L1)

Mean Absolute Error, or L1 loss. Excellent overview below [6] and [10].

Code

def L1(yHat, y):
    return np.sum(np.absolute(yHat - y))