Cross-validation is a technique used to measure and evaluate machine learning models performance. During training we create a number of partitions of the training set and train\/test on different subsets of those partitions.<\/p>\n\n\n\n
Cross-validation is frequently used to train, measure and finally select a machine learning model for a given dataset because it helps assess how the results of a model will generalize to an independent data set in practice<\/em>. Most importantly, cross-validation has been shown to produce models with lower bias than other methods.<\/p>\n\n\n\n This tutorial will focus on one variant of cross-validation named k-fold cross-validation<\/strong>.<\/p>\n\n\n\n In this tutorial we\u2019ll cover the following:<\/p>\n\n\n\n Cross-validation is a resampling technique used to evaluate machine learning models on a limited data set.<\/p>\n\n\n\n The most common use of cross-validation is the k-fold cross-validation method. Our training set is split into K<\/em> partitions, the model is trained on K-1 <\/em>partitions and the test error is predicted and computed on the Kth<\/em> partition. This is repeated for each unique group and the test errors are averaged across.<\/p>\n\n\n\n The same procedure is described by the following steps:<\/strong><\/p>\n\n\n\n For each partition:<\/p>\n\n\n\n 2. Set the partition is the test set<\/p>\n\n\n\n 3. Train a model on the rest of the partitions<\/p>\n\n\n\n 4. Measure performance on the test set<\/p>\n\n\n\n 5. Retain the performance metric<\/p>\n\n\n\n 6. Explore model performance over different folds<\/p>\n\n\n\n Cross-validation is commonly used since it\u2019s easy to interpret and since it generally results in a less biased or less optimistic estimates of the model performance than other methods, such as a simple train\/test split. One of the biggest downsides in using cross-validation is the increased training time as we are essentially training K times instead of 1.<\/p>\n\n\n\n Scikit-learn<\/strong><\/a> is a popular machine learning library that also provides many tools for data sampling, model evaluation and training. We\u2019ll use the This example trains a text classifier on the news groups dataset (you can find it here<\/a>). Given a piece of text (string), the model classifies it to one of the following classes: \u201catheism\u201d,\u201dchristian\u201d,\u201dcomputer graphics\u201d, \u201cmedicine\u201d.<\/p>\n\n\n\n\n
\n\n\nK-Fold Cross-Validation<\/h3>\n\n\n\n
\n
Cross-validation example using scikit-learn<\/h3>\n\n\n\n
Kfold<\/code> class to generate our folds. Here\u2019s a basic overview:<\/p>\n\n\n\nfrom<\/strong> sklearn.model_selection<\/strong> import<\/strong> KFold
X = [...] # My training dataset inputs\/features
y = [...] # My training dataset targets
kf = KFold(n_splits=2)
kf.get_n_splits(X)
for<\/strong> train_index, test_index in<\/strong> kf.split(X):
X_train, X_test = X[train_index], X[test_index]
y_train, y_test = y[train_index], y[test_index]
model = train_model(X_train,y_train)
score = eval_model(X_test,y_test)<\/pre>\n\n\n\nNow let\u2019s train an end-to-end example using scikit-learn and comet.ml.<\/h3>\n\n\n\n