All the algorithms in machine learning rely on minimizing or maximizing a function, which we call \u201cobjective function\u201d. The group of functions that are minimized are called \u201closs functions\u201d. A loss function is a measure of how good a prediction model does in terms of being able to predict the expected outcome. A most commonly used method of finding the minimum point of function is \u201cgradient descent\u201d. Think of loss function like undulating mountain and gradient descent is like sliding down the mountain to reach the bottommost point.<\/p>\n
There is not a single loss function that works for all kind of data. It depends on a number of factors including the presence of outliers, choice of machine learning algorithm, time efficiency of gradient descent, ease of finding the derivatives and confidence of predictions. The purpose of this blog series is to learn about different losses and how each of them can help data scientists.<\/p>\n
Loss functions can be broadly categorized into 2 types: Classification and Regression Loss<\/strong>. In this post, I\u2019m focussing on regression loss. In future posts I cover loss functions in other categories. Please let me know in comments if I miss something. Also, all the codes and plots shown in this blog can be found in this notebook.<\/a><\/p>\n Regression functions predict a quantity, and classification functions predict a label.<\/em><\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n Mean Square Error (MSE)<\/a> is the most commonly used regression loss function. MSE is the sum of squared distances between our target variable and predicted values.<\/p>\n Below is a plot of an MSE function where the true target value is 100, and the predicted values range between -10,000 to 10,000. The MSE loss (Y-axis) reaches its minimum value at prediction (X-axis) = 100. The range is 0 to \u221e.<\/p>\n Mean Absolute Error<\/a> (MAE) is another loss function used for regression models. MAE is the sum of absolute differences between our target and predicted variables. So it measures the average magnitude of errors in a set of predictions, without considering their directions. (If we consider directions also, that would be called Mean Bias Error (MBE), which is a sum of residuals\/errors). The range is also 0 to \u221e.<\/p>\n In short,<\/strong> using the squared error is easier to solve, but using the absolute error is more robust to outliers. But let\u2019s understand why!<\/strong><\/p>\n Whenever we train a machine learning model, our goal is to find the point that minimizes loss function. Of course, both functions reach the minimum when the prediction is exactly equal to the true value.<\/p>\n Here\u2019s a quick review of python code for both. We can either write our own functions or use sklearn\u2019s built-in metrics functions:<\/p>\n <\/p>\n Let\u2019s see the values of MAE and Root Mean Square Error (RMSE, which is just the square root of MSE to make it on the same scale as MAE) for 2 cases. In the first case, the predictions are close to true values and the error has small variance among observations. In the second, there is one outlier observation, and the error is high.<\/p>\n What do we observe from this, and how can it help us to choose which loss function to use?<\/strong><\/p>\n Since MSE squares the error (y \u2014 y_predicted = e), the value of error (e) increases a lot if e > 1. If we have an outlier in our data, the value of e will be high and e\u00b2 will be >> |e|. This will make the model with MSE loss give more weight to outliers than a model with MAE loss. In the 2nd case above, the model with RMSE as loss will be adjusted to minimize that single outlier case at the expense of other common examples, which will reduce its overall performance.<\/p>\n MAE loss is useful<\/strong> if the training data is corrupted with outliers (i.e. we erroneously receive unrealistically huge negative\/positive values in our training environment, but not our testing environment).<\/p>\n Intuitively, we can think about it like this: If we only had to give one prediction for all the observations that try to minimize MSE, then that prediction should be the mean <\/strong>of all target values. But if we try to minimize MAE, that prediction would be the median<\/strong> of all observations. We know that median is more robust to outliers<\/a> than mean, which consequently makes MAE more robust to outliers than MSE.<\/p>\n One big problem in using MAE loss<\/strong> (for neural nets especially) is that its gradient is the same throughout, which means the gradient will be large even for small loss values. This isn\u2019t good for learning. To fix this, we can use dynamic learning rate which decreases as we move closer to the minima. MSE behaves nicely in this case and will converge even with a fixed learning rate. The gradient of MSE loss is high for larger loss values and decreases as loss approaches 0, making it more precise at the end of training (see figure below.)<\/p>\n Deciding which loss function to use I recommend reading this post with a nice study comparing the performance of a regression model using L1 loss and L2 loss<\/a> in both the presence and absence of outliers. Remember, L1 and L2 loss are just another names for MAE and MSE respectively.<\/p>\n L1 loss is more robust to outliers, but its derivatives are not continuous, making it inefficient to find the solution. L2 loss is sensitive to outliers, but gives a more stable and closed form solution (by setting its derivative to 0.)<\/em><\/mark><\/p><\/blockquote>\n Problems with both: <\/strong>There can be cases where neither loss function gives desirable predictions. For example, if 90% of observations in our data have true target value of 150 and the remaining 10% have target value between 0\u201330. Then a model with MAE as loss might predict 150 for all observations, ignoring 10% of outlier cases, as it will try to go towards median value. In the same case, a model using MSE would give many predictions in the range of 0 to 30 as it will get skewed towards outliers. Both results are undesirable in many business cases.<\/p>\n What to do in such a case? <\/strong>An easy fix would be to transform the target variables. Another way is to try a different loss function. This is the motivation behind our 3rd loss function, Huber loss.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n Huber loss<\/a> is less sensitive to outliers in data than the squared error loss. It\u2019s also differentiable at 0. It\u2019s basically absolute error, which becomes quadratic when error is small. How small that error has to be to make it quadratic depends on a hyperparameter, \ud835\udeff (delta), which can be tuned. Huber loss approaches MSE when \ud835\udeff ~ 0 and MAE when \ud835\udeff ~ \u221e (large numbers.)<\/strong><\/p>\n The choice of delta is critical because it determines what you\u2019re willing to consider as an outlier. Residuals larger than delta are minimized with L1 (which is less sensitive to large outliers), while residuals smaller than delta are minimized \u201cappropriately\u201d with L2.<\/p>\n Why use Huber Loss? Huber loss can be really helpful in such cases, as it curves around the minima which decreases the gradient. And it\u2019s more robust to outliers than MSE. Therefore, it combines good properties from both MSE and MAE. However, the problem with Huber loss<\/strong> is that we might need to train hyperparameter delta which is an iterative process.<\/p>\n Log-cosh is another function used in regression tasks that\u2019s smoother than L2. Log-cosh is the logarithm of the hyperbolic cosine of the prediction error.<\/p>\n Advantage:<\/strong> Why do we need a 2nd derivative? <\/strong>Many ML model implementations like XGBoost <\/a>use Newton\u2019s method to find the optimum, which is why the second derivative (Hessian) is needed. For ML frameworks like XGBoost, twice differentiable functions are more favorable.<\/p>\n But Log-cosh loss isn\u2019t perfect. It still suffers from the problem of gradient and hessian for very large off-target predictions being constant, therefore resulting in the absence of splits for XGBoost.<\/p>\n Python code for Huber and Log-cosh loss functions:<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n In most of the real-world prediction problems, we are often interested to know about the uncertainty in our predictions. Knowing about the range of predictions as opposed to only point estimates can significantly improve decision making processes for many business problems.<\/p>\n Quantile loss functions<\/a> turn out to be useful when we are interested in predicting an interval instead of only point predictions. Prediction interval from least square regression is based on an assumption that residuals (y \u2014 y_hat) have constant variance across values of independent variables. We can not trust linear regression models that violate this assumption. We can not also just throw away the idea of fitting a linear regression model as the baseline by saying that such situations would always be better modeled using non-linear functions or tree-based models. This is where quantile loss and quantile regression come to the rescue as regression-based on quantile loss provides sensible prediction intervals even for residuals with non-constant variance or non-normal distribution.<\/p>\n Let\u2019s see a working example to better understand why regression based on quantile loss performs well with heteroscedastic data.<\/p>\n Quantile regression vs. Ordinary Least Square regression<\/strong><\/p>\n Notebook link<\/a> with codes for quantile regression shown in the above plots.<\/p>\n Understanding the quantile loss function<\/strong><\/p>\n Quantile-based regression aims to estimate the conditional \u201cquantile\u201d of a response variable given certain values of predictor variables. Quantile loss is actually just an extension of MAE (when the quantile is 50th percentile, it is MAE).<\/p>\n The idea is to choose the quantile value based on whether we want to give more value to positive errors or negative errors. Loss function tries to give different penalties to overestimation and underestimation based on the value of the chosen quantile (\u03b3). For example, a quantile loss function of \u03b3 = 0.25 gives more penalty to overestimation and tries to keep prediction values a little below median<\/p>\n \u03b3 is the required quantile and has value between 0 and 1.<\/p>\n We can also use this loss function to calculate prediction intervals in neural nets or tree based models. Below is an example of Sklearn implementation for gradient boosted tree regressors.<\/p>\n The above figure shows a 90% prediction interval calculated using the quantile loss function available in GradientBoostingRegression of sklearn library. The upper bound is constructed \u03b3 = 0.95 and lower bound using \u03b3 = 0.05.<\/p>\n ___________________________________________________________________<\/p>\n A nice comparison simulation is provided in \u201cGradient boosting machines, a tutorial<\/a>\u201d. To demonstrate the properties of all the above loss functions, they\u2019ve simulated a dataset sampled from a sinc(x<\/em>)<\/a> function with two sources of artificially simulated noise: the Gaussian noise component \u03b5 ~ N<\/em>(0, \u03c32) and the impulsive noise component \u03be ~ Bern(p<\/em>). The impulsive noise term is added to illustrate the robustness effects. Below are the results of fitting a GBM regressor using different loss functions.<\/p>\n Some observations from the simulations:<\/strong><\/p>\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:486\/1*3MsFzl7zRZE3TihIC9JmaQ.png\")
<\/figure>Regression loss<\/h1>\n
1. Mean Square Error, Quadratic loss, L2 Loss<\/strong><\/h2>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:255\/1*mlXnpXGdhMefPybSQtRmDA.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:576\/1*EqTaoCB1NmJnsRYEezSACA.png\")
<\/figure>2. Mean Absolute Error, L1 Loss<\/strong><\/h2>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:257\/1*xjarhfIDtRcaNhp7ZEyEdg.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:576\/1*8BQhdKu1nk-tAAbOR17qGg.png\")
<\/figure>MSE vs. MAE (L2 loss vs L1 loss)<\/strong><\/h2>\n
![\"\"](\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/06\/mas-\u2013-Medium.jpg\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*KibGRET1M6Bu0-8XmjviMA.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*JTC4ReFwSeAt3kvTLq1YoA.png\")
<\/figure>
\n<\/strong>If the outliers represent anomalies that are important for business and should be detected, then we should use MSE. On the other hand, if we believe that the outliers just represent corrupted data, then we should choose MAE as loss.<\/p>\n3. Huber Loss, Smooth Mean Absolute Error<\/strong><\/h2>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:525\/1*0eoiZGyddDqltzzjoyfRzA.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:504\/1*jxidxadWSMLvwLDZz2mycg.png\")
<\/figure>
\n<\/strong>One big problem with using MAE for training of neural nets is its constantly large gradient, which can lead to missing minima at the end of training using gradient descent. For MSE, gradient decreases as the loss gets close to its minima, making it more precise.<\/p>\n4. Log-Cosh Loss<\/h2>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:436\/1*hj5n5273jYX7rclO7bnfJg.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:504\/1*BAbgW_JdwyAWLZR2dE1Ujg.png\")
<\/figure>log(cosh(x))<\/code> is approximately equal to (x ** 2) \/ 2<\/code> for small x<\/code>and to abs(x) - log(2)<\/code> for large x<\/code>. This means that ‘logcosh’ works mostly like the mean squared error, but will not be so strongly affected by the occasional wildly incorrect prediction. It has all the advantages of Huber loss, and it\u2019s twice differentiable everywhere, unlike Huber loss.<\/p>\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*FNxOsZLqXVZNFOxGoG9A1Q.png\")
<\/figure>![\"\"](\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/06\/loss_regression-\u2013-Medium-1.jpg\")
<\/figure>\n\n\n\n5. Quantile Loss<\/h2>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*A61Xn0hlPcoMKDns5KFD-A.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*h_iOn3gSUa2bk6o0foudDA.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:568\/1*hdqrLhTXity54wmfXAtBGw.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*ePh5hyrWS5f591nUORz8_A.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:504\/1*_Msrko0NVv1d43MaVfsZkA.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/0*DQ0t4YXq-xLFsWi1.png\")
<\/figure>Comparison Study:<\/strong><\/h2>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*46WnlaWhfPZaVWzSZPviIg.png\")
<\/figure>\n
All the loss functions in a single plot.<\/strong><\/h2>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*BploIBOUrhbgdoB1BK_sOg.png\")
<\/figure>