![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:1500\/1*Ocxi71mUXvhgo3rNg1XR4g.png\")
<\/figure>According to Wikipedia<\/a>, an outlier<\/strong> is an observation point that is distant from other observations. This definition is vague because it doesn\u2019t quantify the word \u201cdistant\u201d. In this blog, we\u2019ll try to understand the different interpretations of this \u201cdistant\u201d notion. We will also look into the outlier detection and treatment techniques while seeing their impact on different types of machine learning models.<\/p>\n Outliers arise due to changes in system behavior, fraudulent behavior, human error, instrument error, or simply through natural deviations in populations. A sample may have been contaminated with elements from outside the population being examined.<\/p>\n Many machine learning models, like linear & logistic regression<\/a>, are easily impacted by the outliers in the training data. Models like AdaBoost<\/a> increase the weights of misclassified points on every iteration and therefore might put high weights on these outliers as they tend to be often misclassified. This can become an issue if that outlier is an error of some type, or if we want our model to generalize well and not care for extreme values.<\/p>\n To overcome this issue, we can either change the model or metric, or we can make some changes in the data and use the same models. For the analysis, we will look into House Prices Kaggle Data<\/a>. All the codes for plots and implementation can be found on this GitHub Repository<\/a>.<\/p>\n Extreme values can be present in both dependent & independent variables, in the case of supervised learning methods.<\/p>\n These extreme values need not necessarily impact the model performance or accuracy, but when they do they are called \u201cInfluential\u201d<\/strong> points.<\/p>\n These are called points of \u201chigh leverage<\/strong>\u201d. With a single predictor, an extreme value is simply one that is particularly high or low. With multiple predictors, extreme values may be particularly high or low for one or more predictors (univariate analysis \u2014 analysis of one variable at a time)<\/em><\/strong> or may be \u201cunusual\u201d combinations of predictor values (multivariate analysis)<\/em><\/strong><\/p>\n In the following figure, all the points on the right-hand side of the orange line are leverage points.<\/p>\n Regression \u2014 these extreme values are termed as \u201coutliers\u201d<\/strong>. They may or may not be influential points, which we will see later. In the following figure, all the points above the orange line can be classified as outliers.<\/p>\n Classification: Here, we have two types of extreme values:<\/p>\n 1. Outliers:<\/strong> For example, in an image classification problem in which we\u2019re trying to identify dogs\/cats, one of the images in the training set has a gorilla (or any other category not part of the goal of the problem) by mistake. Here, the gorilla image is clearly noise. Detecting outliers here does not make sense because we already know which categories we want to focus on and which to discard<\/p>\n 2. Novelties:<\/strong> Many times we\u2019re dealing with novelties, and the problem is often called supervised anomaly detection<\/strong>. In this case, the goal is not to remove outliers or reduce their impact, but we are interested in detecting anomalies in new observations. Therefore we won\u2019t be discussing it in this post. It is especially used for fraud detection in credit-card transactions, fake calls, etc.<\/p>\n All the points we have discussed above, including influential points, will become very clear once we visualize the following figure.<\/p>\n Inference Our major focus will be outliers (extreme values in target variable<\/strong> for further investigation and treatment). We\u2019ll see the impact of these extreme values on the model\u2019s performance.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n When detecting outliers, we are either doing univariate analysis or multivariate analysis. When your linear model has a single predictor, then you can use univariate analysis. However, it can give misleading results if you use it for multiple predictors. One common way of performing outlier detection is to assume that the regular data come from a known distribution<\/strong> (e.g. data are Gaussian distributed). This assumption is discussed in the Z-Score method section below.<\/p>\n The quickest and easiest way to identify outliers is by visualizing them<\/a> using plots. If your dataset is not huge (approx. up to 10k observations & 100 features), I would highly recommend you build scatter plots & box-plots of variables. If there aren\u2019t outliers, you\u2019ll definitely gain some other insights like correlations, variability, or external factors like the impact of world war\/recession on economic factors. However, this method is not recommended for high dimensional data where the power of visualization fails.<\/p>\n The box plot uses inter-quartile range to detect outliers. Here, we first determine the quartiles Q<\/em>1 and Q<\/em>3.<\/p>\n Interquartile range is given by, IQR = Q3 \u2014 Q1<\/p>\n Upper limit = Q3+1.5*IQR<\/p>\n Lower limit = Q1\u20131.5*IQR<\/p>\n Anything below the lower limit and above the upper limit is considered an outlier<\/p>\n This is a multivariate approach for finding influential points. These points may or may not be outliers as explained above, but they have the power to influence the regression model. We will see their impact in the later part of the blog.<\/p>\n This method is used only for linear regression and therefore has a limited application. Cook\u2019s distance<\/a> measures the effect of deleting a given observation. It\u2019s represents the sum of all the changes in the regression model when observation \u201ci\u201d<\/strong> is removed from it.<\/p>\n Here, p is the number of predictors and s\u00b2 is the mean squared error of the regression model. There are different views regarding the cut-off values to use for spotting highly influential points. A rule of thumb is that D(i) > 4\/n, can be good cut off for influential points.<\/p>\n R has the car<\/a> (Companion to Applied Regression) package where you can directly find outliers using Cook\u2019s distance. Implementation is provided in this R-Tutorial<\/a>. Another similar approach is DFFITS<\/strong>, which you can see details of here<\/a>.<\/p>\n This method assumes that the variable has a Gaussian distribution. It represents the number of standard deviations an observation is away from the mean:<\/p>\n Here, we normally define outliers as points whose modulus of z-score is greater than a threshold value. This threshold value is usually greater than 2 (3 is a common value).<\/p>\n All the above methods are good for initial analysis of data, but they don\u2019t have much value in multivariate settings or with high dimensional data. For such datasets, we have to use advanced methods like PCA, LOF (Local Outlier Factor) & HiCS: High Contrast Subspaces for Density-Based Outlier Ranking<\/strong>.<\/p>\n We won\u2019t be discussing these methods in this blog, as they are beyond its scope. Our focus here is to see how various outlier treatment techniques affect the performance of models. You can read this blog<\/a> for details on these methods.<\/p>\n The impact of outliers can be seen not only in predictive modeling but also in statistical tests where it reduces the power of tests. Most parametric statistics, like means, standard deviations, and correlations, and every statistic based on these, are highly sensitive to outliers. But in this post, we are focusing only on the impact of outliers in predictive modeling.<\/p>\n I believe dropping data is always a harsh step and should be taken only in extreme conditions when we\u2019re very sure that the outlier is a measurement error<\/strong>, which we generally do not know. The data collection process is rarely provided. When we drop data, we lose information in terms of the variability in data. When we have too many observations and outliers are few<\/strong>, then we can think of dropping these observations.<\/p>\n In the following example we can see that the slope of the regression line changes a lot in the presence of the extreme values at the top. Hence, it is reasonable to drop them and get a better fit & more general solution.<\/p>\n For this comparison, I chose only four important predictors (Overall Quality, MSubClass, Total Basement Area, Ground living area) out of total 80 predictors and tried to predict Sales Price using these predictors. The idea is to see how outliers affect linear & tree-based methods.<\/p>\n From the above results, we can conclude that transformation techniques generally works better than dropping for improving the predictive accuracy of both linear & tree-based models. It is very important to treat outliers by either dropping or transforming them if you are using linear regression model.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n If I have missed any important techniques for outliers treatment, I would love to hear about them in comments. Thank you for reading.<\/p>\n About Me:<\/strong> Graduated with Masters in Data Science at USF. Interested in working with cross-functional groups to derive insights from data, and apply Machine Learning knowledge to solve complicated data science problems. https:\/\/alviraswalin.wixsite.com\/alvira<\/a><\/p>\n Check out my other blogs here<\/a>!<\/p>\n LinkedIn: <\/strong>www.linkedin.com\/in\/alvira-swalin<\/strong><\/a><\/p>\nWhat do we mean by outliers?<\/h1>\n
Extreme Values in Independent Variables<\/h2>\n
<\/figure>
Extreme Values in Target Variables<\/h2>\n
<\/figure>
<\/figure>
\n<\/em><\/strong>– Points in Q1: Outliers
\n– Points in Q3: Leverage Points
\n– Points in Q2: Both outliers & leverage but non-influential points
\n– Circled points: Example of Influential Points. There can be more but these are the prominent ones<\/em><\/p><\/blockquote>\nCommon Methods for Detecting Outliers<\/h1>\n
Box-Plot<\/h2>\n
<\/figure>
Cook\u2019s Distance<\/h2>\n
<\/figure>
Z-Score<\/h2>\n
<\/figure>
<\/figure>
Impact & Treatment of Outliers<\/h1>\n
To Drop or Not to Drop<\/h2>\n
<\/figure>
Other Data-Based Methods<\/h2>\n
\n
<\/figure>
\n
Model-Based Methods<\/h2>\n
\n
<\/figure>
<\/figure>
Case Study Comparison<\/h2>\n
<\/figure>
End Notes<\/h2>\n
\n
References:<\/h1>\n
\n