{"id":6911,"date":"2023-07-24T15:47:34","date_gmt":"2023-07-24T23:47:34","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=6911"},"modified":"2025-04-24T17:15:08","modified_gmt":"2025-04-24T17:15:08","slug":"introduction-to-machine-learning-model-evaluation","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/introduction-to-machine-learning-model-evaluation\/","title":{"rendered":"Introduction to Machine Learning Model Evaluation"},"content":{"rendered":"\n\n\n\n\n
\n
\"\"<\/figure>
\n
<\/picture><\/figure>\n<\/div>\n
\n
\n

Machine learning continues to be an increasingly integral component of our lives, whether we\u2019re applying the techniques to research or business problems. Machine learning models ought to be able to give accurate predictions in order to create real value for a given organization.<\/p>\n

While training a model<\/a> is a key step, how the model generalizes on unseen data is an equally important aspect that should be considered in every machine learning pipeline. We need to know whether it actually works and, consequently, if we can trust its predictions. Could the model be merely memorizing the data it is fed with, and therefore unable to make good predictions on future samples, or samples that it hasn\u2019t seen before?<\/p>\n

In this article, we explain the techniques used in evaluating how well a machine learning model generalizes to new, previously unseen data. We\u2019ll also illustrate how common model evaluation metrics are implemented for classification and regression problems<\/a> using Python.<\/p>\n

Model Evaluation Techniques<\/h1>\n

The above issues can be handled by evaluating the performance of a machine learning model, which is an integral component of any data science project. Model evaluation aims to estimate the generalization accuracy of a model on future (unseen\/out-of-sample) data.<\/p>\n

Methods for evaluating a model\u2019s performance are divided into 2 categories: namely, holdout<\/a> and Cross-validation<\/a>. Both methods use a test set (i.e data not seen by the model) to evaluate model performance. It\u2019s not recommended to use the data we used to build the model to evaluate it. This is because our model will simply remember the whole training set, and will therefore always predict the correct label for any point in the training set. This is known as overfitting<\/a>.<\/p>\n

Holdout<\/h2>\n

The purpose of holdout evaluation is to test a model on different data than it was trained on. This provides an unbiased estimate of learning performance.<\/p>\n

In this method, the dataset is randomly<\/em> divided into three subsets:<\/p>\n

    \n
  1. Training set<\/strong> is a subset of the dataset used to build predictive models.<\/li>\n
  2. Validation set<\/strong> is a subset of the dataset used to assess the performance of the model built in the training phase. It provides a test platform for fine-tuning a model\u2019s parameters and selecting the best performing model. Not all modeling algorithms need a validation set.<\/li>\n
  3. Test set<\/strong>, or unseen data, is a subset of the dataset used to assess the likely future performance of a model. If a model fits to the training set much better than it fits the test set, overfitting is probably the cause.<\/li>\n<\/ol>\n

    The holdout approach is useful because of its speed, simplicity, and flexibility. However, this technique is often associated with high variability since differences in the training and test dataset can result in meaningful differences in the estimate of accuracy.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

    \n
    \n
    \n

    Cross-Validation<\/h2>\n

    Cross-validation<\/a> is a technique that involves partitioning the original observation dataset into a training set, used to train the model, and an independent set used to evaluate the analysis.<\/p>\n

    The most common cross-validation technique is k-fold cross-validation<\/a>, where the original dataset is partitioned into k equal size subsamples, called folds. The k is a user-specified number, usually with 5 or 10 as its preferred value. This is repeated k times, such that each time, one of the k subsets is used as the test set\/validation set and the other k-1 subsets are put together to form a training set. The error estimation is averaged over all k trials to get the total effectiveness of our model.<\/p>\n

    For instance, when performing five-fold cross-validation, the data is first partitioned into 5 parts of (approximately) equal size. A sequence of models is trained. The first model is trained using the first fold as the test set, and the remaining folds are used as the training set. This is repeated for each of these 5 splits of the data and the estimation of accuracy is averaged over all 5 trials to get the total effectiveness of our model.<\/p>\n

    As can be seen, every data point gets to be in a test set exactly once and gets to be in a training set k-1 times. This significantly reduces bias, as we\u2019re using most of the data for fitting, and it also significantly reduces variance, as most of the data is also being used in the test set. Interchanging the training and test sets also adds to the effectiveness of this method.<\/p>\n

    Model Evaluation Metrics<\/h1>\n

    Model evaluation metrics are required to quantify model performance. The choice of evaluation metrics depends on a given machine learning task (such as classification, regression, ranking, clustering, topic modeling, among others). Some metrics, such as precision-recall<\/a>, are useful for multiple tasks. Supervised learning tasks such as classification and regression constitutes a majority of machine learning applications. In this article, we focus on metrics for these two supervised learning models.<\/p>\n

    Classification Metrics<\/h1>\n

    In this section we will review some of the metrics used in classification problems, namely:<\/p>\n