{"id":7878,"date":"2023-10-06T15:38:45","date_gmt":"2023-10-06T23:38:45","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=7878"},"modified":"2025-04-24T17:05:38","modified_gmt":"2025-04-24T17:05:38","slug":"major-problems-of-machine-learning-datasets-part-3","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/major-problems-of-machine-learning-datasets-part-3\/","title":{"rendered":"Major Problems of Machine Learning Datasets: Part 3"},"content":{"rendered":"\n\n\n\n\n
\n
\n
\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>
Photo by Tim Gouw<\/a> on Unsplash<\/a><\/figcaption><\/figure>\n

Note: This is the part 3 of the series Major Problems of Machine Learning Datasets<\/strong>. You can read part 1 here<\/strong><\/a> and part 2 here<\/strong><\/a>.<\/p>\n

Imbalanced data<\/h1>\n

Imbalanced data occurs when there is an uneven distribution of classes or labels. For example, in a credit card detection task, the number of non-fraudulent transactions will likely be much greater than the number of fraudulent credit card transactions.<\/p>\n

\n
\"\"<\/figure>
<\/picture><\/div>\n<\/figure>\n

Particularly with classification tasks, class balance is extremely important. When classes are imbalanced, the majority class will influence the output of the model more, making our classifier biased towards the majority class.<\/p>\n

Models trained with imbalanced data usually have high precision and recall scores for the majority class, whereas these scores will likely drop significantly for the minority class.<\/p>\n

Let\u2019s take a look at this example of a credit card fraud dataset<\/a>, and build a model on this imbalanced data. Our goal is to correctly identify fraudulent transactions:<\/p>\n

import pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt<\/span>df = pd.read_csv(\"creditcard.csv<\/a>\")<\/span># checking class distribution of column `Class`\nsns.countplot(data= df, x = \"Class\")<\/span><\/pre>\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/figure>\n
We have a clear class imbalance here. Let\u2019s see how much this will affect model performance:<\/p>\n
#  Splitting Data Into Training and Test Sets\nfrom sklearn.model_selection import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X,\n                                                    y,\n                                                    test_size=0.2,\n                                                    random_state=42)\nprint(X_train.shape, vX_test.shape)\n------------\n(227845, 30) (56962, 30)<\/span><\/pre>\n
Now that the data is divided into training and test sets, let\u2019s build a baseline logistic regression model.<\/p>\n
from sklearn.linear_model import LogisticRegression\nlr = LogisticRegression(solver='liblinear')\nlr.fit(X_train, y_train)<\/span><\/pre>\n
Now we evaluate the performance of the model with a classification report and confusion matrix:<\/p>\n
from sklearn.metrics import (plot_confusion_matrix,\n                             classification_report)\ndef gen_report(model):\n  preds = model.predict(X_test)\n  print(classification_report(preds, y_test))\n  plot_confusion_matrix(model, X_test, y_test)<\/span>print(\"---LOGISTIC REGRESSION MODEL---\")\ngen_report(model=lr)<\/span><\/pre>\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/figure>\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/figure>\n
A quick look at these metrics, and it would seem our model performs amazingly \u2014 100% accuracy, 100% precision, and 100% recall! But if we dig a little deeper, we\u2019ll see that our precision drops to 53% on fraudulent data (the minority class) and recall drops to 83%.<\/p>\n
How to deal with imbalanced data<\/h2>\n
The first thing we can do is change our sampling method. Instead of random sampling, we can use stratified sampling<\/strong>, which makes sure train and test data will have an almost equal ratio of fraudulent transactions. The downside to this method, however, is that with such a small proportion of total minority class cases in the total data, we will still have very few instances of fraudulent cases present for the model to learn from.<\/p>\n
Instead, we might try a resampling method that attempts to tackle the underlying problem of not enough representation of the minority class. Resampling<\/strong> includes both oversampling and under-sampling methods.<\/p>\n
Oversampling<\/strong> is a resampling technique using which generates more instances of the underrepresented class by randomly sampling from the existing instances.<\/p>\n
Under-sampling<\/strong> is a resampling technique in which the majority class is reduced to the size of the minority class by randomly sampling from the majority class.<\/p>\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/figure>\n
Scikit-learn\u2019s imblearn<\/code> is an open-source, MIT-licensed Python library that provides tools to deal with imbalanced classes. It has a class named SMOTETomek<\/code> that combines the concepts of under-sampling and oversampling techniques, theoretically providing a good compromise between the pros and cons of each (though in practice this is not always the case; for your dataset, you may need to experiment with each sampling method to see which performs best for your dataset).<\/p>\n
from imblearn.combine import SMOTETomek\nsmt = SMOTETomek()<\/span>X_res, y_res= smt.fit_resample(X, y)<\/span><\/pre>\n
Let\u2019s again create a model on our resampled data and see if performance improves:<\/p>\n
from sklearn.model_selection import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X_res,\n                                                    y_res,\n                                                    test_size=0.2,\n                                                    stratify=y,\n                                                    random_state=42\n                                                    )\nfrom sklearn.linear_model import LogisticRegression\nre_lrm = LogisticRegression(solver='liblinear')\nre_lrm.fit(X_train, y_train)\n<\/span>print(\"---LOGISTIC REGRESSION MODEL (RESAMPLED)---\")\ngen_report(model= re_lrm)<\/span><\/pre>\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/figure>\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/figure>\n
As you can see, we\u2019ve had a vast improvement in model performance!<\/p>\n
High-dimensional data<\/h1>\n
In machine learning, the number (or, degree) of features in a dataset is referred to as its dimensionality. Machine learning problems with high dimensionality have a variety of issues.<\/p>\n
As the number of features in a dataset increases, the time needed to train the model will also increase. But that\u2019s not all. High-dimensional datasets are extremely difficult (if not impossible) to visualize (imagine a 6-dimensional plot… neither can I!). The harder it is to visualize a dataset, the harder it can be to explain, which contributes to the \u201cblack box\u201d problem in machine learning. Furthermore, the amount of data needed to train a model typically grows exponentially with the number of features (or, dimensions) in a dataset. This is often referred to as the Curse of Dimensionality and can lead to a slew of statistical phenomena that do not occur in low-dimensional settings. To save our model from this problem, we can perform dimensionality reduction.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
\n
\n
\n
Want to see the evolution of AI-generated art projects? Visit our public project<\/a> to see time-lapses, experiment evolutions, and more!<\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
\n
\n
\n
How to deal with high dimensionality<\/h2>\n
Dimensionality reduction attempts to reduce the number of features in a dataset, while still preserving as much variation as possible from the original data. Dimensionality reduction can reduce the chances of overfitting, takes care of multicollinearity, removes noise from data, and can also be useful to transform non-linear data into linear data.<\/p>\n
There are two main techniques to perform dimensionality reduction: we can either remove the least important feature, or we can try to combine the original features into newer, fewer features. One of the most popular methods of linear dimensionality reduction is PCA (Principle Component Analysis).<\/p>\n
PCA transforms a set of correlated features into a smaller number of uncorrelated features, called principal components. PCA makes use of the correlation between features for reducing dimensions. Note that it is important to perform feature scaling before applying PCA because it is very sensitive to relative ranges of features.<\/p>\n
Sci-kit learn provides a built-in tool for this process with sklearn.decomposition.PCA<\/code>.<\/p>\n
Let\u2019s first scale the data using Standard Scaler:<\/p>\n
from sklearn.preprocessing import StandardScaler<\/span>scaler = StandardScaler()\nscaler.fit(X)\nX_scaled = scaler.transform(X)<\/span><\/pre>\n
And now let\u2019s apply PCA:<\/p>\n
from sklearn.decomposition import PCA<\/span>pca_10 = PCA(n_components = 10, random_state = 42)\nX_pca_10 = pca_10.fit_transform(X_scaled)<\/span># comparing before and after PCA shape\nprint(X.shape, X_pca_10.shape)\n------------\n((284807, 30), (284807, 10))<\/span><\/pre>\n
\n
\"\"<\/figure>
<\/picture><\/div>\n
Principal Components<\/figcaption>\n<\/figure>\n
Non-normal distribution of data<\/h1>\n
The normal distribution is a type of distribution of data in which data points are distributed in a symmetrical <\/strong>manner around the mean of data. It looks like a bell shape curve.<\/p>\n
It is important for most machine learning algorithms that features should follow a normal distribution. non-normal distribution of data affects model performance and generates wrong predictions normality is an important assumption for many machine learning models.<\/p>\n
In this blog, we will discuss the two most common types of non-normal distribution Left skewed and right-skewed distribution.<\/p>\n
The left-skewed distribution has a long tail with negatively-skewed data, whereas the right skewed has a long tail on the right side of the distribution.<\/p>\n
How to Deal With Non-Normal Distribution of Data<\/h2>\n
There are many transformation methods that are used to convert non-normal distribution into a normal distribution.<\/p>\n
    \n
  1. Log Normal Transformation: <\/strong>In this technique, we take the log of values of a particular feature.<\/li>\n<\/ol>\n
    import numpy as np\nimport matplotlib.pyplot as plt\nfrom scipy import stats<\/span># sample data generation\nnp.random.seed(42)\ndata = sorted(stats.lognorm.rvs(s=0.5,\n                                loc=1,\n                                scale=1000,\n                                size=1000))<\/span># fit lognormal distribution\nshape, loc, scale = stats.lognorm.fit(data, loc=0)\npdf_lognorm = stats.lognorm.pdf(data, shape, loc, scale)<\/span><\/pre>\n
    \n
    \"\"<\/figure>
    <\/picture><\/div>\n<\/figure>\n
    2. Box-Cox Transformation<\/strong>: Box-Cox transformation is a part of the power transformers family. It makes use of the exponent lambda (\u03bb), which ranges from -5 to 5, to convert non-normal distribution of dependent variables into a normal distribution.<\/p>\n
    import numpy as np\nimport matplotlib.pyplot as plt\nfrom scipy import stats<\/span># sample data generation\nnp.random.seed(42)\ndata = sorted(stats.lognorm.rvs(s=0.7, loc=3, scale=1000, size=1000))<\/span>pdf_boxcox = stats.boxcox(data)<\/span><\/pre>\n
    \n
    \"\"<\/figure>
    <\/picture><\/div>\n<\/figure>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
    Photo by Tim Gouw on Unsplash Note: This is the part 3 of the series Major Problems of Machine Learning Datasets. You can read part 1 here and part 2 here. Imbalanced data Imbalanced data occurs when there is an uneven distribution of classes or labels. For example, in a credit card detection task, the […]<\/p>\n","protected":false},"author":8,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","footnotes":""},"categories":[6],"tags":[],"coauthors":[140],"class_list":["post-7878","post","type-post","status-publish","format-standard","hentry","category-machine-learning"],"yoast_head":"\nMajor Problems of Machine Learning Datasets: Part 3 - Comet<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/major-problems-of-machine-learning-datasets-part-3\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Major Problems of Machine Learning Datasets: Part 3\" \/>\n<meta property=\"og:description\" content=\"Photo by Tim Gouw on Unsplash Note: This is the part 3 of the series Major Problems of Machine Learning Datasets. You can read part 1 here and part 2 here. Imbalanced data Imbalanced data occurs when there is an uneven distribution of classes or labels. For example, in a credit card detection task, the […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/blog\/major-problems-of-machine-learning-datasets-part-3\/\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2023-10-06T23:38:45+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-04-24T17:05:38+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*2zagnBfllq2sIoAhcKeK3A.jpeg\" \/>\n<meta name=\"author\" content=\"Abhay Parashar\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Abhay Parashar\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"8 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Major Problems of Machine Learning Datasets: Part 3 - Comet","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/blog\/major-problems-of-machine-learning-datasets-part-3\/","og_locale":"en_US","og_type":"article","og_title":"Major Problems of Machine Learning Datasets: Part 3","og_description":"Photo by Tim Gouw on Unsplash Note: This is the part 3 of the series Major Problems of Machine Learning Datasets. You can read part 1 here and part 2 here. Imbalanced data Imbalanced data occurs when there is an uneven distribution of classes or labels. For example, in a credit card detection task, the […]","og_url":"https:\/\/www.comet.com\/site\/blog\/major-problems-of-machine-learning-datasets-part-3\/","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2023-10-06T23:38:45+00:00","article_modified_time":"2025-04-24T17:05:38+00:00","og_image":[{"url":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*2zagnBfllq2sIoAhcKeK3A.jpeg","type":"","width":"","height":""}],"author":"Abhay Parashar","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Abhay Parashar","Est. reading time":"8 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.comet.com\/site\/blog\/major-problems-of-machine-learning-datasets-part-3\/#article","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/blog\/major-problems-of-machine-learning-datasets-part-3\/"},"author":{"name":"Team Comet Digital","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/6266601170c60a7a82b3e0043fbe8ddf"},"headline":"Major Problems of Machine Learning Datasets: Part 3","datePublished":"2023-10-06T23:38:45+00:00","dateModified":"2025-04-24T17:05:38+00:00","mainEntityOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/major-problems-of-machine-learning-datasets-part-3\/"},"wordCount":1064,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/major-problems-of-machine-learning-datasets-part-3\/#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*2zagnBfllq2sIoAhcKeK3A.jpeg","articleSection":["Machine Learning"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/blog\/major-problems-of-machine-learning-datasets-part-3\/","url":"https:\/\/www.comet.com\/site\/blog\/major-problems-of-machine-learning-datasets-part-3\/","name":"Major Problems of Machine Learning Datasets: Part 3 - Comet","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/major-problems-of-machine-learning-datasets-part-3\/#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/major-problems-of-machine-learning-datasets-part-3\/#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*2zagnBfllq2sIoAhcKeK3A.jpeg","datePublished":"2023-10-06T23:38:45+00:00","dateModified":"2025-04-24T17:05:38+00:00","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/blog\/major-problems-of-machine-learning-datasets-part-3\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/blog\/major-problems-of-machine-learning-datasets-part-3\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/blog\/major-problems-of-machine-learning-datasets-part-3\/#primaryimage","url":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*2zagnBfllq2sIoAhcKeK3A.jpeg","contentUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*2zagnBfllq2sIoAhcKeK3A.jpeg"},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/blog\/major-problems-of-machine-learning-datasets-part-3\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"Major Problems of Machine Learning Datasets: Part 3"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/6266601170c60a7a82b3e0043fbe8ddf","name":"Team Comet Digital","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/4f0c0a8cc7c0e87c636ff6a420a6647c","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/Screen-Shot-2023-08-12-at-8.58.50-AM-96x96.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/Screen-Shot-2023-08-12-at-8.58.50-AM-96x96.png","caption":"Team Comet Digital"},"sameAs":["https:\/\/www.comet.ml\/"],"url":"https:\/\/www.comet.com\/site\/blog\/author\/teamcometdigital\/"}]}},"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7878","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=7878"}],"version-history":[{"count":1,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7878\/revisions"}],"predecessor-version":[{"id":15502,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7878\/revisions\/15502"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=7878"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=7878"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=7878"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=7878"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}