{"id":7874,"date":"2023-10-06T15:34:59","date_gmt":"2023-10-06T23:34:59","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=7874"},"modified":"2025-04-24T17:05:41","modified_gmt":"2025-04-24T17:05:41","slug":"major-problems-of-machine-learning-datasets-part-1","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/major-problems-of-machine-learning-datasets-part-1\/","title":{"rendered":"Major Problems of Machine Learning Datasets: Part 1"},"content":{"rendered":"\n\n\n\n\n
\n
\n
\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>
Photo by JESHOOTS.COM<\/a> on Unsplash<\/a><\/figcaption><\/figure>\n

Data play a key role in machine learning, and the better and more relevant data you have, the more accurate the model you will build. Getting the perfect data, however, is still a dream for many data scientists. A lot of data comes from web scraping, APIs and other external sources, and most real-world datasets will just look like an ugly stack of information, at least at first. However, data will speak for itself, if you keep it organized.<\/p>\n

In this blog, I would love to share some major problems that occur with many supervised machine learning datasets, as well as how to deal with them.<\/p>\n

Missing Values<\/h1>\n

How to Deal with Missing Values In Datasets?<\/h2>\n

There are various ways of dealing with missing values, and you\u2019ll likely need to determine which method is right for your task at hand on a case-by-case basis. If only a very small percentage of your data is missing, then you might be able to simply drop all missing values. In some cases of extremely large amounts of missing data, it may even be better to consider finding a new dataset, additional datasets, increasing your domain knowledge, or reframing your problem. When about 10\u201350% of your data are missing, however, you may also consider imputation.<\/p>\n

There are two types of imputation: numerical and categorical.<\/p>\n

Imputing Missing Numerical Data<\/h2>\n
    \n
  1. Mean, median, or mode imputation<\/strong>: If the distribution of your data appears normal, you might consider calculating the mean of a particular feature to replace missing values. However, if the distribution of data appears left- or right-skewed, then you might be better off with median imputation. The following implementation assumes that your missing values are represented by NaN<\/code>s:<\/li>\n<\/ol>\n
    # import numpy as np<\/span># Normally-distributed data (fill with mean)\ndf['col'] = df['col'].fillna(df['col'].mean()) <\/span># Skewed data (fill with median)\ndf['col'] = df['col'].fillna(df['col'].median())<\/span><\/pre>\n
    2. Treat <\/strong>NaN<\/strong><\/code> as new category<\/strong>: This technique is useful when there is some relationship between missing values and non-missing values. In this method, we add a new column to our DataFrame with all missing values as 1<\/code> and non-missing values as 0<\/code>. Note that this method creates a category feature out of whether or not a missing value exists in a particular numerical feater, but does not replace, or otherwise handle, the original missing values. Later we should still replace missing values from the original column.<\/p>\n
    import numpy as np<\/span>df['col_nan']= np.where(df['col'].isnull(),1,0) <\/span>df['col'] = df['col'].fillna(df['col'].mean()) <\/span><\/pre>\n
    3. KNN Imputer<\/strong>: KNN Imputer is a distance-based imputation method that utilizes the k<\/code>-Nearest Neighbors algorithm to replace missing values in the dataset with the mean value of k<\/code> nearest neighbors found in training data. The value of k<\/code> is determined using the parameter n_neighbors<\/code>. By default, KNN uses the Euclidean distance metric to find the nearest neighbors. One thing to remember is that we need to normalize the data before passing it to KNN Imputer, otherwise replacements will be biased towards the larger range of features.<\/p>\n
    # Loading DataFrame\nimport pandas as pd\ndf = pd.read_csv('titanic.csv')<\/span># Extracting all Columns With Numerical Data\nnum = [col for col in df.columns if df[col].dtypes != 'O']<\/span># Scaling Data Using MinMaxScaler\nfrom sklearn.preprocessing import MinMaxScaler\nscaler = MinMaxScaler()\nnorm_df = pd.DataFrame(scaler.fit_transform(df[num]), columns = num)<\/span># Initialize Imputer\nfrom sklearn.impute import KNNImputer\nknn = KNNImputer(n_neighbors=5)<\/span># Fit Imputer and Transform Data\nknn.fit(norm_df)<\/span># Transform Data and Save It In a New Instance of DataFrame\nnon_nan_df=pd.DataFrame(knn.transform(norm_df), columns=num)<\/span><\/pre>\n
    \n
    \n
    \"\"<\/figure>
    <\/picture><\/div>\n<\/div>\n
    Image by author<\/figcaption>\n<\/figure>\n
    This method can also be used for categorical data, but for that, we first need to convert the data to numerical form using label- or one-hot-encoding.<\/p>\n
    4. Filling categories proportionally using their probability weights<\/strong>: In this technique, we replace missing categories with other categories based on their contribution to creating the whole feature. This method doesn\u2019t change the category proportion, which means In a feature if category A contributes 90%, B 7%, and C 3% then after filling missing values proportion of A, B and C will stay almost the same.<\/p>\n
    def fill_proportionally(col, dataset):\n    import random\n    random.seed(0)\n\n    # getting all unique values (without nan)\n    values = dataset[col].dropna().unique()\n\n    # getting weights for probability weighting\n    weights = dataset[col].value_counts().values \/ dataset[col].value_counts().values.sum()\n    print('Before Imputation Probablity Weights\\n',weights)\n    # filling\n    dataset[col] = dataset[col].apply(lambda x: random.choices(values, weights=weights)[0] if pd.isnull(x) else x)\n\nimport pandas as pd\ndf = pd.read_csv('https:\/\/raw.githubusercontent.com\/Abhayparashar31\/datasets\/master\/titanic.csv')\n\n### Imputing Missing Categories\nfill_proportionally('Embarked', df)<\/pre>\n
    \n
    \"\"<\/figure>
    <\/picture><\/div>\n<\/figure>\n
    There are other methods as well that make use of machine learning models to predict missing values. You can check them by visiting my Kaggle notebook<\/em><\/strong><\/a>.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
    \n
    \n
    \n
    Innovation and academia go hand-in-hand. Listen to our own CEO Gideon Mendels chat with the Stanford MLSys Seminar Series<\/a> team about the future of MLOps and give the Comet platform a try for free!<\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
    \n
    \n
    \n
    Categorical Data<\/h1>\n
    Many machine learning algorithms work only with numerical data including regression models. For this reason, it is important to convert categorical data to a numerical form before feeding them to a machine learning model. Categorical data refer to data with labels as values (e.g., sex, city, position, etc.).<\/p>\n
    How to deal with categorical data in datasets?<\/h2>\n
    One option when utilizing categorical data is to choose a tree-based model, when appropriate. If the situation calls for another type of model, however,
    \nthen another option is to convert the categorical values into numerical form.<\/p>\n
    Imputing Categorical Data (with no inherent order)<\/strong><\/p>\n
    Inherent order means there is a relationship between the order of our categories. In categories with no inherent order, we can use nominal encoding methods. One of the most common methods of nominal encoding is One-Hot Encoding<\/p>\n