![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/format:webp\/1*o_KfyMzF7LITK2DlYm_wHw.png)
<\/picture>Often in machine learning, and specifically with classification problems, we encounter imbalanced datasets. This typically refers to an issue where the classes are not represented equally, which can cause huge problems for some algorithms.<\/p>\n
In this article, we\u2019ll explore a technique called resampling<\/strong>, which is used to reduce this effect on our machine learning algorithms.<\/p>\n This article presumes that you know some machine learning concepts and are familiar with Python and its data science libraries.<\/p><\/blockquote>\n The best way to learn something is through an example:<\/p>\n Say that you have a fraud detection binary classification model (two classes \u2014 \u201cFraud\u201d or \u201cNot-Fraud\u201d) problem with 100 instances (rows). A total of 80 instances are labeled as Fraud <\/strong>and the remaining 20 instances are labeled as a Not-Fraud. <\/strong>This is an imbalanced dataset, and the ratio of Fraud to Not-Fraud instances is 80:20, or 4:1.<\/p>\n Most classification datasets don\u2019t have exactly equal numbers of records in each class, but a small difference doesn\u2019t matter as much.<\/p>\n This class imbalance problem can occur in binary classification problems as well as multi-class classification problems, but most techniques can be used on either.<\/p>\n In addition to using the core Python libraries like NumPy, Pandas, and scikit-learn, we\u2019re going to use another great library called imbalanced-learn<\/a>, which is a part of scikit-learn-contrib projects.<\/p>\n imbalanced-learn<\/strong> provides more advanced methods to handle imbalanced datasets like SMOTE and Tomek Links.<\/p>\n Here are the commands to install it via pip or conda:<\/p>\n Most beginners struggle when dealing with imbalanced datasets for the first time. They tend to use accuracy as a metric to evaluate their machine learning models. This intuitively makes sense, as classification accuracy is often the first measure we use when evaluating such models.<\/p>\n Nevertheless, this can be misleading, because most of the algorithms we use are designed to achieve the best accuracy, so the classifier always \u201cpredicts\u201d the most common class without performing any features analysis. It will still have a high accuracy rate, but it will give false predictions, nevertheless.<\/p>\n Say we have 1000 emails as a dataset: 990 are spam emails and 10 aren\u2019t. If you build a simple model you\u2019ll get ~99% accuracy, which at first glance seems great.<\/p>\n But on the other hand, the algorithm doesn\u2019t perform any learning \u2014 the accuracy here is only reflecting the underlying class distribution because models look at the data and cleverly decide that the best thing to do is to always predict spam and achieve high accuracy. As such, the model\u2019s success and is just an illusion.<\/p>\n This is why the choice of metrics used <\/strong>when working with imbalanced datasets is extremely important.<\/p>\n You should have an imbalanced dataset to apply the methods described here\u2014 you can get started with this dataset<\/strong><\/a> from Kaggle.<\/p>\n You can use Seaborn<\/a> to plot the count of each class to see if your dataset presents imbalanced dataset problem like the following:<\/p>\n Or you can use the sklearn to compute class weight and get the class ratios as follows:<\/p>\n Join more than 14,000 of your fellow machine learners and data scientists. Subscribe to the premier newsletter for all things deep learning<\/a>.<\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n There are multiple ways to handle the issue of imbalanced datasets. The techniques we\u2019re going to use in this tutorials is called resampling.<\/p>\n Resampling is a widely-adopted technique for dealing with imbalanced datasets, and it is often very easy to implement, fast to run, and an excellent starting point.<\/p>\n Resampling changes the dataset into a more balanced one by adding instances to the minority class or deleting ones from the majority class, that way we build better machine learning models.<\/p>\n The way to introduce these changes in a given dataset is achieved via two main methods: Oversampling<\/strong> and Undersampling<\/strong>.<\/p>\n Notwithstanding the advantage of balancing classes, these techniques also have some drawbacks:<\/p>\n We\u2019ll now show the underlying techniques in each method, along with some code snippets.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n Random undersampling randomly deletes records from the majority class. You should consider trying this technique when you have a lot of data.<\/p>\n A simple undersampling technique is to undersample the majority class randomly and uniformly. This can potentially lead to information loss, though. But if the examples of the majority class are near to others in terms of distance, this method might yield good results.<\/p>\n Here is a code snippet:<\/p>\n Here\u2019s the set of parameters you can specify to the The NearMiss algorithm has been proposed to solve the issue of potential information loss. It\u2019s based on the nearest neighbor algorithm and has a lot of variations that we\u2019ll see in this section.<\/p>\n The basics of the NearMiss algorithms include the following:<\/p>\n Here are the different versions of this algorithm:<\/p>\n Here is a code snippet:<\/p>\n You can tune also the following parameters:<\/p>\n Tomek links are pairs of very close instances that belong to different classes. They\u2019re samples near the borderline between classes. By removing the examples of the majority<\/strong> class of each pair, we increase the space between the two classes and move toward balancing the dataset by deleting those points.<\/p>\n Here is a code snippet to resample the majority class:<\/p>\n The idea here is basically to remove the unimportant instance from the majority class. To decide whether an instance is important or not, we use the concept of clustering<\/strong> on the geometry of the feature space.<\/p>\n Clustering is an unsupervised learning approach, in which clusters are creating encircling data points belonging.<\/p>\n We will use it only to find cluster centroid <\/strong>that are obtained by averaging feature vector for all the features over the data points.<\/p>\n After finding the cluster centroid of the majority class, we decide the following:<\/p>\n Here is a code snippet for using cluster centroids:<\/p>\n Besides the previous parameter, here\u2019s another one you can tune to get better results:<\/p>\n Edited Nearest Neighbor Rule (or ENN) was proposed in 1972 to remove instances from the majority class (undersampling).<\/p>\n This technique removes any instance from the majority class whose class label is different from the class label of at least two of its three nearest neighbors. In general terms, it\u2019s near or around the borderline of different classes.<\/p>\n The point here is to increase the classification accuracy of minority instances rather than majority instances.<\/p>\n Here is a sample code snippet:<\/p>\n Neighborhood Cleaning Rule ( or NCR) deals with the majority and minority samples separately when sampling the datasets.<\/p>\n NCR starts by calculating the nearest three neighbors for all instances in the training set. We then do the following:<\/p>\n Here is a sample code snippet:<\/p>\n An important parameter here is Random oversampling randomly duplicate records from the minority class. Try this technique when you don\u2019t have a lot of data.<\/p>\n Random oversampling simply replicates random minority class examples. It\u2019s known to increase the likelihood of overfitting, which is a major drawback.<\/p>\n Here is a sample code snippet:<\/p>\n SMOTE stands for S<\/strong>ynthetic M<\/strong>inority O<\/strong>versampling Te<\/strong>chnique \u2014 it consists of creating or synthesizing elements or samples from the minority class rather than creating copies based on those that exist already. This is used to avoid model overfitting.<\/p>\n To create a synthetic instance, SMOTE finds the K-nearest neighbors of each minority instance, randomly selects one of them and then calculates linear interpolations to produce a new minority instance in the neighborhood. It can be also explained by changing this instance features one at a time by a random amount \u2014 so as a result, the new points are added between the neighbors.<\/p>\n Here is a code snippet showing how to resample the minority class:<\/p>\n ADASYN stands for Ada<\/strong>ptive Syn<\/strong>thetic sampling, and as SMOTE does, ADASYN generates samples of the minority class. But here, because of their density distributions<\/strong>, this technique receives wide attention.<\/p>\n Its purpose is to generate data for minority class samples that are harder to learn, as compared to those minority samples that are easier to learn.<\/p>\n It measures the K-nearest neighbors for all minority instances, then calculates the class ratio of the minority and majority instances to create new samples.<\/p>\n Repeating this process, we will adaptively <\/strong>shift the decision boundary to focus on those samples that are hard to learn.<\/p>\n Here is a code snippet:<\/p>\n We can combine oversampling and undersampling techniques to get better sampling results. Here are two ways that SMOTE & Tomek Links<\/strong> \u2014 Here\u2019s a code snippet:<\/p>\n SMOTE & Edited Nearest Neighbor<\/strong> \u2014 Here\u2019s a code snippet:<\/p>\n Besides the resampling methodologies we\u2019ve covered in this article, there are other intuitive and advanced techniques you can employ to deal with this problem. Here are some of them:<\/p>\n There are more resources out there to handle your imbalanced dataset. Here are a few to help you get started:<\/p>\n In this article, we confronted the problem of imbalanced datasets by exploring several different resampling techniques, which allow you to change your dataset\u2019s balance so your model can learn more effectively.<\/p>\n You\u2019ll need to experiment with these techniques on your specific machine learning problem to see what best fits your case \u2014 we don\u2019t have a technique that always allows us to build a better model while achieving the best performance. Otherwise, we wouldn\u2019t be able to say that we\u2019re machine learning engineers.<\/p>\n You can combine these methods to obtain more reliable models, but I suggest starting small and building upon what you learn.<\/p>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":" Photo by Martin Sanchez on Unsplash Often in machine learning, and specifically with classification problems, we encounter imbalanced datasets. This typically refers to an issue where the classes are not represented equally, which can cause huge problems for some algorithms. In this article, we\u2019ll explore a technique called resampling, which is used to reduce this […]<\/p>\n","protected":false},"author":66,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","footnotes":""},"categories":[6],"tags":[],"coauthors":[195],"class_list":["post-7617","post","type-post","status-publish","format-standard","hentry","category-machine-learning"],"yoast_head":"\nWhat is an Imbalanced Dataset?<\/h1>\n
Project setup<\/h1>\n
# using pip\npip install -U imbalanced-learn<\/span># using conda\nconda install -c conda-forge imbalanced-learn<\/span><\/pre>\nThe Metric Problem<\/h1>\n
Example<\/h2>\n
Investigate your dataset<\/h1>\n
# import the data sciecne libraries.\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# read the dataset\ndata = pd.read_csv('training.csv')\n\n# print the count of each class from the target vatiables\nprint(data.FraudResult.value_counts())\n\n# plot the count of each class from the target vatiables\nsns.countplot(data.FraudResult)<\/pre>\n# import the function to compute the class weights\nfrom sklearn.utils import compute_class_weight\n\n# calculate the class weight by providing the 'balanced' parameter.\nclass_weight = compute_class_weight('balanced', data.FraudResult.unique() , data.FraudResult)\n\n# print the result\nprint(class_weight)<\/pre>\n<\/div>\n<\/div>\n<\/div>\n\n\n\nResampling<\/h1>\n
\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*o_KfyMzF7LITK2DlYm_wHw.png\")
<\/figure>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*gHW_PLz7kWrhdl5t1sJRRA.png\")
<\/figure>Disadvantages<\/h2>\n
\n
Undersampling<\/h1>\n
Random Undersampling<\/h2>\n
# import the Random Under Sampler object.\nfrom imblearn.under_sampling import RandomUnderSampler\n\n# create the object.\nunder_sampler = RandomUnderSampler()\n\n# fit the object to the training data.\nx_train_under, y_train_under = under_sampler.fit_sample(x_train, y_train)<\/pre>\n
RandomUnderSampler<\/code> object (the same thing apply for the other objects from the imblearn library):<\/p>\n\n
sampling_strategy<\/code> : This parameter can be used to tell the object how to perform undersampling on our dataset. It can be majority<\/code> to resample only the majority class, not_minority<\/code> to resample all classes but the minority class, and auto<\/code> is the default one here, which stands for not_minority<\/code>. You can check out the documentation (included below in \u201cResources\u201d) to learn more.<\/li>\nreturn_indices<\/code> : Boolean on whether to return the indices of the removed instances or not.<\/li>\nrandom_state<\/code> : An integer that controls the randomness of the procedure, allowing you to reproduce the results.<\/li>\n<\/ul>\nNearMiss Undersampling<\/h2>\n
\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*8WM0gsh_naPEa9HTpE2c1A.png\")
<\/figure>\n
# import the NearMiss object.\nfrom imblearn.under_sampling import NearMiss\n\n# create the object with auto\nnear = NearMiss(sampling_strategy=\"not minority\")\n\n# fit the object to the training data.\nx_train_near, y_train_near = near.fit_sample(x_train, y_train)<\/pre>\n
\n
version<\/code> : the version of the near-miss algorithm, which can be 3,1, or 2.<\/li>\nn_neighbors<\/code> : the number of neighbors to consider to compute the average distance\u2014three is the default.<\/li>\n<\/ul>\nUndersampling with Tomek links<\/h2>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*pR35KsLpz7-_zvbvdm0frg.png\")
<\/figure># import the TomekLinks object.\nfrom imblearn.under_sampling import TomekLinks\n\n# instantiate the object with the right ratio strategy.\ntomek_links = TomekLinks(sampling_strategy='majority')\n\n# fit the object to the training data.\nx_train_tl, y_train_tl = tomek_links.fit_sample(x_train, y_train)<\/pre>\n
Undersampling with Cluster Centroids<\/h2>\n
\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*1XlHmnc9hKn1oPz48lrn7Q.png\")
<\/figure># import the ClusterCentroids object.\nfrom imblearn.under_sampling import ClusterCentroids\n\n# instantiate the object with the right ratio.\ncluster_centroids = ClusterCentroids(sampling_strategy=\"auto\")\n\n# fit the object to the training data.\nx_train_cc, y_train_cc = cluster_centroids.fit_sample(x_train, y_train)<\/pre>\n
\n
estimator<\/code>: An object that performs the clustering process for this method\u2014K-Means is the default here.<\/li>\n<\/ul>\nUndersampling with Edited Nearest Neighbor Rule<\/h2>\n
# import the EditedNearestNeighbours object.\nfrom imblearn.under_sampling import EditedNearestNeighbours\n\n# create the object to resample the majority class.\nenn = EditedNearestNeighbours(sampling_strategy=\"majority\")\n\n# fit the object to the training data.\nx_train_enn, y_train_enn = enn.fit_sample(x_train, y_train)<\/pre>\n
Undersampling with Neighborhood Cleaning Rule<\/h2>\n
\n
# import the NeighbourhoodCleaningRule object.\nfrom imblearn.under_sampling import NeighbourhoodCleaningRule\n\n# create the object to resample the majority class.\nncr = NeighbourhoodCleaningRule(sampling_strategy=\"majority\")\n\n# fit the object to the training data.\nx_train_ncr, y_train_ncr = ncr.fit_sample(x_train, y_train)<\/pre>\n
threshold_clearning<\/code>, which is a float number used after applying ENN, it tells the algorithm to consider a class or not during the cleaning.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\nOversampling<\/h1>\n
Random Oversampling<\/h2>\n
# import the Random Over Sampler object.\nfrom imblearn.over_sampling import RandomOverSampler\n\n# create the object.\nover_sampler = RandomOverSampler()\n\n# fit the object to the training data.\nx_train_over, y_train_over = over_sampler.fit_sample(x_train, y_train)<\/pre>\n
SMOTE Oversampling<\/h2>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*bSOwLuDleEEGiuw7PtooOQ.png\")
<\/figure># import the SMOTETomek\nfrom imblearn.over_sampling import SMOTE\n\n# create the object with the desired sampling strategy.\nsmote = SMOTE(sampling_strategy='minority')\n\n# fit the object to our training data\nx_train_smote, y_train_smote = smote.fit_sample(x_train, y_train)<\/pre>\n
ADASYN Oversampling<\/h2>\n
# import the ADASYN object.\nfrom imblearn.over_sampling import ADASYN\n\n# create the object to resample the majority class.\nadasyn = ADASYN(sampling_strategy=\"minority\")\n\n# fit the object to the training data.\nx_train_adasyn, y_train_adasyn = adasyn.fit_sample(x_train, y_train)<\/pre>\n
Combining Oversampling and Undersampling<\/h1>\n
imblearn<\/code> provides:<\/p>\n# import the SMOTETomek.\nfrom imblearn.combine import SMOTETomek\n\n# create the object with the desired sampling strategy.\nsmotemek = SMOTETomek(sampling_strategy='auto')\n\n# fit the object to our training data.\nx_train_smt, y_train_smt = smotemek.fit_sample(x_train, y_train)<\/pre>\n
# import the SMOTEENN.\nfrom imblearn.combine import SMOTEENN\n\n# create the object with the desired samplig strategy.\nsmoenn = SMOTEENN(sampling_strategy='minority')\n\n# fit the object to our training data.\nx_train_smtenn, y_train_smtenn = smoenn.fit_sample(x_train, y_train)<\/pre>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
Other Techniques<\/h1>\n
\n
class_weight<\/code> attribute.<\/li>\n<\/ol>\nResources<\/h1>\n
\n
imbalanced-learn<\/a><\/code> documentation.<\/li>\nConclusion<\/h1>\n