{"id":7545,"date":"2023-09-19T11:33:55","date_gmt":"2023-09-19T19:33:55","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=7545"},"modified":"2025-04-29T13:37:53","modified_gmt":"2025-04-29T13:37:53","slug":"customer-churn-with-continuous-experiment-tracking","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/customer-churn-with-continuous-experiment-tracking\/","title":{"rendered":"Customer Churn With Continuous Experiment Tracking"},"content":{"rendered":"\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*Oz_ftD_Sn7TtrvR_wSXaBg.png\" alt=\"customer churn prediction with continuous tracking and Comet ML\"\/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Introduction<\/h3>\n\n\n\n<p>In today\u2019s competitive business environment, retaining customers is essential to a company\u2019s success. Customer churn, or the rate at which customers leave your service, is an important metric that directly affects your business bottom line. To address this challenge, data scientists harness the power of machine learning to predict customer churn and develop strategies for customer retention.<\/p>\n\n\n\n<p><a href=\"https:\/\/www.comet.com\/yennhi95zz\/customer-churn\/ce4189deb57943d281df0405dab75687?experiment-tab=panels&amp;showOutliers=true&amp;smoothing=0&amp;xAxis=wall\" target=\"_blank\" rel=\"noreferrer noopener\">Check out the final project here!\u00a0<\/a><\/p>\n\n\n\n<p>In this article, we take a deep dive into a machine learning project aimed at predicting customer churn and explore how Comet ML, a powerful machine learning experiment tracking platform, plays a key role in increasing project success.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p><em>\ud83d\udca1I write about Machine Learning on&nbsp;<\/em><a href=\"https:\/\/medium.com\/@yennhi95zz\/subscribe\" target=\"_blank\" rel=\"noreferrer noopener\"><em>Medium<\/em><\/a><em>&nbsp;||&nbsp;<\/em><a href=\"https:\/\/github.com\/yennhi95zz\" target=\"_blank\" rel=\"noreferrer noopener\"><em>Github<\/em><\/a><em>&nbsp;||&nbsp;<\/em><a href=\"https:\/\/www.kaggle.com\/nhiyen\/code\" target=\"_blank\" rel=\"noreferrer noopener\"><em>Kaggle<\/em><\/a><em>&nbsp;||&nbsp;<\/em><a href=\"https:\/\/www.linkedin.com\/in\/yennhi95zz\/\" target=\"_blank\" rel=\"noreferrer noopener\"><em>Linkedin<\/em><\/a><em>. \ud83d\udd14 Follow \u201cNhi Yen\u201d for future updates!<\/em><\/p>\n<\/blockquote>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/0*wfEvc8wRvp0YgG8C\" alt=\"customer churn prediction with continuous tracking and Comet ML\"\/><figcaption class=\"wp-element-caption\">Photo by&nbsp;<a href=\"https:\/\/unsplash.com\/@blakewisz?utm_source=medium&amp;utm_medium=referral\" target=\"_blank\" rel=\"noreferrer noopener\">Blake Wisz<\/a>&nbsp;on&nbsp;<a href=\"https:\/\/unsplash.com\/?utm_source=medium&amp;utm_medium=referral\" target=\"_blank\" rel=\"noreferrer noopener\">Unsplash<\/a><\/figcaption><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">I. Customer Churn: Why Does It&nbsp;Matter?<\/h3>\n\n\n\n<p>Customer churn refers to the phenomenon where&nbsp;<strong><em>customers stop using your service or product<\/em><\/strong>. This is an important metric for companies for the following reasons:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Impact on revenue: Loss of customers reduces revenue and growth opportunities.<\/li>\n\n\n\n<li>Acquisition costs: Acquiring new customers is usually more expensive than retaining existing customers.<\/li>\n\n\n\n<li>Customer Feedback: Understanding why customers leave provides valuable information to improve your service.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/0*8JhzIsWdzTp5K5gM\" alt=\"Customer churn prediction\"\/><figcaption class=\"wp-element-caption\">What is Customer Churn?\u200a\u2014\u200aCredit:&nbsp;<a href=\"https:\/\/www.google.com\/url?sa=i&amp;url=https%3A%2F%2Fkpi-max.com%2Fchurn-rate%2F&amp;psig=AOvVaw1ZDEZzrnTPDB7aN2RH0DQK&amp;ust=1694051597769000&amp;source=images&amp;cd=vfe&amp;opi=89978449&amp;ved=0CBAQjRxqFwoTCLDyiZDwlIEDFQAAAAAdAAAAABAE\" target=\"_blank\" rel=\"noreferrer noopener\">KPI&nbsp;Max<\/a><\/figcaption><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">II. Customer Churn Project and Dataset<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\">1. Project Objective<\/h4>\n\n\n\n<p>The goal of our project is to predict customer churn for telecommunications companies using a model stacking approach. Model stacking involves training multiple machine learning models and using another model to combine their predictions to improve accuracy.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">2. Dataset<\/h4>\n\n\n\n<p>This project uses the \u201c<a href=\"https:\/\/www.kaggle.com\/datasets\/blastchar\/telco-customer-churn\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>Telco Customer Churn<\/strong><\/a>\u201d dataset available on Kaggle. This dataset contains information about telecom customers, such as contract type, monthly fee, and whether the customer has canceled.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p>Tired of manually tracking your prompts and prompt variables?&nbsp;<a href=\"https:\/\/github.com\/comet-ml\/comet-llm\/?utm_source=Heartbeat&amp;utm_medium=referral&amp;utm_content=Heartbeat\" target=\"_blank\" rel=\"noreferrer noopener\">Try CometLLM, a free, open-source tool to log, visualize, and search your LLM prompts and metadata.<\/a><\/p>\n<\/blockquote>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">III. Continuous Experiment Tracking for Customer Churn with Comet ML<\/h3>\n\n\n\n<p>Comet ML is a versatile tool that helps data scientists optimize machine learning experiments. In our project, we use Comet ML to:<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">1. Experiment Tracking<\/h4>\n\n\n\n<p>Comet ML has a section where you can create and manage experiments. This is where you record information about your experiment, such as metrics, hyperparameters, and other relevant details.<\/p>\n\n\n\n<p>Comet ML allows you to record several metrics such as precision, log loss, and ROC AUC score at each step of your experiment. This detailed log is invaluable for tracking model performance and understanding how changes impact results.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">2. Visualization<\/h4>\n\n\n\n<p>Within Comet ML, you need tools to visualize the results of your experiments, such as tables and graphs showing metrics over time or across different runs.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">3. Hyperparameter Tuning<\/h4>\n\n\n\n<p>Hyperparameter optimization is critical to model performance. Comet ML seamlessly integrates with Optuna, an automated hyperparameter optimization framework. This allows you to efficiently tune the hyperparameters of your machine learning model.<\/p>\n\n\n\n<p><em>\ud83d\udc49 Read more about CometML\u200a\u2014\u200a<\/em><a href=\"https:\/\/www.comet.com\/docs\/v2\/guides\/tracking-ml-training\/starting-an-experiment\/\" target=\"_blank\" rel=\"noreferrer noopener\"><em>HERE<\/em><\/a><\/p>\n\n\n\n<p><em>You might be interested in:<\/em><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/medium.com\/mlearning-ai\/logging-the-effective-management-of-machine-learning-systems-e1eb04e74eb5\" target=\"_blank\" rel=\"noreferrer noopener\"><em>Logging\u200a\u2014\u200aThe Effective Management of Machine Learning Systems<\/em><\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/medium.com\/@yennhi95zz\/hyperparameter-tuning-in-machine-learning-a-key-to-optimize-model-performance-1d520934bb99\" target=\"_blank\" rel=\"noreferrer noopener\"><em>Hyperparameter Tuning in Machine Learning: A Key to Optimize Model Performance<\/em><\/a><\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">IV. Step-by-step guide: How the customer churn project works.<\/h3>\n\n\n\n<p>\ud83d\udc49 The entire code can be found on both&nbsp;<a href=\"https:\/\/github.com\/yennhi95zz\/customer-churn-prediction-with-model-stacking\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>GitHub<\/strong><\/a>&nbsp;and&nbsp;<a href=\"https:\/\/www.kaggle.com\/nhiyen\/customer-churn-prediction-with-model-stacking\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>Kaggle<\/strong><\/a>.<\/p>\n\n\n\n<p>Here\u2019s an overview of the steps we follow in our project:<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">1. Import Libraries<\/h3>\n\n\n\n<p>First, import the required Python libraries, such as Comet ML, Optuna, and scikit-learn. These libraries provide tools for data pre-processing, model training, and hyperparameter tuning.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">!pip install -q optuna comet_ml\nimport optuna\nimport comet_ml\nfrom comet_ml import Experiment<\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\">import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler, OneHotEncoder\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier\nfrom sklearn.svm import SVC\nfrom sklearn.metrics import accuracy_score, log_loss, roc_auc_score\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.ensemble import StackingClassifier\nfrom sklearn.metrics import accuracy_score, log_loss\n\nfrom kaggle_secrets import UserSecretsClient\n\n# Set display options to show all columns\npd.set_option('display.max_columns', None)<\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\">user_secrets = UserSecretsClient()\ncomet_api_key = user_secrets.get_secret(\"Comet API Key\")\n\nexperiment = Experiment(\n    api_key= YOUR_COMET_API,\n    project_name= YOUR_PROJECT_NAME,\n    workspace= YOUR_WORKSPACE\n)<\/pre>\n\n\n\n<p>In this project, I use a Kaggle notebook to schedule daily runs, and Comet ML records each run as an experiment. In a typical MLOps project, similar scheduling is essential to handle new data and track model performance continuously.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*z28pJ185dHBZ5imZ1swY1w.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">Experiment Tracking in CometML (Image by the&nbsp;Author)<\/figcaption><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">2. Load and Explore&nbsp;Data<\/h3>\n\n\n\n<p>We load the Telco Customer Churn dataset and perform exploratory data analysis (EDA). EDA is essential for gaining insights into the dataset\u2019s characteristics and identifying any data preprocessing requirements.<\/p>\n\n\n\n<p>During this step, for each plot, I use&nbsp;<code>experiment.log_figure(figure=plt)<\/code>&nbsp;to log the plot to Comet. You can access these plots by going to&nbsp;<em>[Experiment] &gt; Graphics<\/em>.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*tTNYJp5vmJUqvpZUBc0qug.png\" alt=\"\"\/><figcaption class=\"wp-element-caption\">Graphics View in Comet ML (Image by the&nbsp;author)<\/figcaption><\/figure>\n\n\n\n<p>For the final experiment I have run, this is the results:<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>#1. Customer Churn Distribution<\/strong><\/h4>\n\n\n\n<p>This plot shows the distribution of churn vs. non-churn customers. In it, you can see the number of customers who have churned (left the telecom service) and those who have not.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/0*Z_nFtgscGqj8s2gi.png\" alt=\"\"\/><\/figure>\n\n\n\n<p>The dataset shows an imbalance with 5,174 non-churned and 1,869 churned customers. Imbalanced data may require special model training techniques, like oversampling or undersampling, to handle class imbalance effectively.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>#2. Numeric Feature Distribution<\/strong><\/h4>\n\n\n\n<p>These histograms show the distribution of numeric features (tenure, MonthlyCharges, and TotalCharges) for the entire dataset.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/0*MXyiHffBg_s08Pbj.png\" alt=\"\"\/><\/figure>\n\n\n\n<p>You can observe how these numeric features are distributed. For instance, understanding the distribution of MonthlyCharges and TotalCharges can help in pricing strategy decisions. Are there clusters of customers with different spending patterns?<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>#3. Categorical Feature Distribution<\/strong><\/h4>\n\n\n\n<p>These plots show the distribution of categorical features (gender, SeniorCitizen, Partner, Dependents, Contract, PaymentMethod) split by churn status.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*v3ubQGfLgXgkEAt1vsciyQ.png\" alt=\"\"\/><\/figure>\n\n\n\n<p>These plots provide insights into how different categories of customers (e.g., seniors vs. non-seniors, customers with partners vs. without) are distributed in terms of churn. You can identify potential customer segments that are more likely to churn.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>#4. Correlation Heatmap:<\/strong><\/h4>\n\n\n\n<p>The heatmap displays the correlation between numeric features in the dataset.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/0*Vft-_xudNmnCTKSM.png\" alt=\"\"\/><\/figure>\n\n\n\n<p>Understanding feature correlations can help in feature selection. For instance, if monthly charges and total charges are highly correlated, you might choose to keep only one of them to avoid multicollinearity in your models. It also helps identify which features might be more important in predicting churn.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>#5. Monthly Charges vs. Total Charges:<\/strong><\/h4>\n\n\n\n<p>This scatterplot shows the relationship between monthly charges and total charges, with points colored by churn status.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/0*Syn6Oe-05OZx6Yrz.png\" alt=\"\"\/><\/figure>\n\n\n\n<p>In the graph above, it appears that customers who have higher Total Charges are less likely to churn. This suggests that long-term customers who spend more are more loyal. You can use this insight to focus on retaining high-value, long-term customers by offering loyalty programs or incentives.<\/p>\n\n\n\n<p>These business insights derived from EDA can guide feature engineering and model selection for your churn prediction project. They help you understand the data\u2019s characteristics and make informed decisions to optimize customer retention strategies.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">3. Preprocessing<\/h3>\n\n\n\n<p>Data preprocessing is a critical step. In it, we encode categorical features, scale numerical features, and split the data into training and validation sets.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"># Encode categorical features, scale numerical features\n\nencoder = OneHotEncoder(handle_unknown=\"ignore\", sparse=False)\nscaler = StandardScaler()\n\nX_train, X_val, y_train, y_val = train_test_split(data.drop(\"Churn\", axis=1), data[\"Churn\"], test_size=0.2, random_state=42)\n\nX_train_encoded = encoder.fit_transform(X_train[categorical_features])\nX_val_encoded = encoder.transform(X_val[categorical_features])\n\nX_train_scaled = scaler.fit_transform(X_train[numerical_features])\nX_val_scaled = scaler.transform(X_val[numerical_features])\n\nX_train_processed = np.concatenate((X_train_encoded, X_train_scaled), axis=1)\nX_val_processed = np.concatenate((X_val_encoded, X_val_scaled), axis=1)<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">4. Model&nbsp;Training<\/h3>\n\n\n\n<p>We train multiple machine learning models, including Logistic Regression, Random Forest, Gradient Boosting, and Support Vector Machine. These models serve as the basis for our ensemble approach.<\/p>\n\n\n\n<p><strong>Logistic Regression (logreg):<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Simple and interpretable model.<\/li>\n\n\n\n<li>Well-suited for binary classification tasks like churn prediction.<\/li>\n\n\n\n<li>Helps understand how features impact the chance of churn.<\/li>\n<\/ul>\n\n\n\n<p><strong>Random Forest Classifier (rf):<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Ensemble method combining multiple decision trees.<\/li>\n\n\n\n<li>Handles mixed feature types (categorical and numerical).<\/li>\n\n\n\n<li>Resistant to overfitting, good for complex datasets.<\/li>\n<\/ul>\n\n\n\n<p><strong>Gradient Boosting Classifier (gb):<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Sequential ensemble building strong predictive power.<\/li>\n\n\n\n<li>Captures complex relationships in data.<\/li>\n\n\n\n<li>Works well for various types of datasets.<\/li>\n<\/ul>\n\n\n\n<p><strong>Support Vector Machine (svm):<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Versatile model for linear and non-linear data.<\/li>\n\n\n\n<li>Can find complex decision boundaries.<\/li>\n\n\n\n<li>Useful when patterns between churn and non-churn are intricate.<\/li>\n<\/ul>\n\n\n\n<p><strong>Modeling Stacking<\/strong><\/p>\n\n\n\n<p>In the project, I am stacking models such as random forests, gradient boosting, and support vector machines, which each have different characteristics and can capture different aspects of the customer churn problem. This approach can help you achieve a more accurate and robust churn prediction model, ultimately leading to better customer retention strategies and business outcomes.<\/p>\n\n\n\n<p>Comet ML comes into play by allowing you to log the models\u2019 performance, hyperparameters, and other metadata.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">5. Hyperparameter Tuning &amp; Ensemble&nbsp;Modeling<\/h3>\n\n\n\n<p>Using Optuna, we optimize hyperparameters for the individual models. This step ensures that our models are fine-tuned for maximum accuracy.<\/p>\n\n\n\n<p>We create a stacking ensemble of models to combine their predictions. This will enhance our predictive performance.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">def objective(trial):\n    # Define hyperparameter search space for individual models\n    rf_params = {\n        'n_estimators': trial.suggest_int('rf_n_estimators', 100, 300),\n        'max_depth': trial.suggest_categorical('rf_max_depth', [None, 10, 20]),\n        'min_samples_split': trial.suggest_int('rf_min_samples_split', 2, 10),\n        'min_samples_leaf': trial.suggest_int('rf_min_samples_leaf', 1, 4),\n    }\n\n    gb_params = {\n        'n_estimators': trial.suggest_int('gb_n_estimators', 100, 300),\n        'learning_rate': trial.suggest_float('gb_learning_rate', 0.01, 0.2),\n        'max_depth': trial.suggest_categorical('gb_max_depth', [3, 4, 5]),\n    }\n\n    svm_params = {\n        'C': trial.suggest_categorical('svm_C', [0.1, 1, 10]),\n        'kernel': trial.suggest_categorical('svm_kernel', ['linear', 'rbf']),\n    }\n\n    # Create models with suggested hyperparameters\n    rf = RandomForestClassifier(**rf_params)\n    gb = GradientBoostingClassifier(**gb_params)\n    svm = SVC(probability=True, **svm_params)\n\n    # Train individual models\n    rf.fit(X_train_processed, y_train)\n    gb.fit(X_train_processed, y_train)\n    svm.fit(X_train_processed, y_train)\n\n    # Evaluate individual models on validation data\n    rf_predictions = rf.predict(X_val_processed)\n    gb_predictions = gb.predict(X_val_processed)\n    svm_predictions = svm.predict(X_val_processed)\n\n    # Calculate accuracy and ROC AUC for individual models\n    rf_accuracy = accuracy_score(y_val, rf_predictions)\n    gb_accuracy = accuracy_score(y_val, gb_predictions)\n    svm_accuracy = accuracy_score(y_val, svm_predictions)\n\n    rf_roc_auc = roc_auc_score(y_val, rf.predict_proba(X_val_processed)[:, 1])\n    gb_roc_auc = roc_auc_score(y_val, gb.predict_proba(X_val_processed)[:, 1])\n    svm_roc_auc = roc_auc_score(y_val, svm.predict_proba(X_val_processed)[:, 1])\n\n    # Create a stacking ensemble with trained models\n    estimators = [\n        ('random_forest', rf),\n        ('gradient_boosting', gb),\n        ('svm', svm)\n    ]\n\n    stacking_classifier = StackingClassifier(estimators=estimators, final_estimator=LogisticRegression())\n\n    # Train the stacking ensemble\n    stacking_classifier.fit(X_train_processed, y_train)\n\n    # Evaluate the stacking ensemble on validation data\n    stacking_predictions = stacking_classifier.predict(X_val_processed)\n    stacking_accuracy = accuracy_score(y_val, stacking_predictions)\n    stacking_roc_auc = roc_auc_score(y_val, stacking_classifier.predict_proba(X_val_processed)[:, 1])\n\n    # Log parameters and metrics to Comet ML\n    experiment.log_parameters({\n        'rf_n_estimators': rf_params['n_estimators'],\n        'rf_max_depth': rf_params['max_depth'],\n        'rf_min_samples_split': rf_params['min_samples_split'],\n        'rf_min_samples_leaf': rf_params['min_samples_leaf'],\n        'gb_n_estimators': gb_params['n_estimators'],\n        'gb_learning_rate': gb_params['learning_rate'],\n        'gb_max_depth': gb_params['max_depth'],\n        'svm_C': svm_params['C'],\n        'svm_kernel': svm_params['kernel']\n    })\n\n    experiment.log_metrics({\n        'rf_accuracy': rf_accuracy,\n        'gb_accuracy': gb_accuracy,\n        'svm_accuracy': svm_accuracy,\n        'rf_roc_auc': rf_roc_auc,\n        'gb_roc_auc': gb_roc_auc,\n        'svm_roc_auc': svm_roc_auc,\n        'stacking_accuracy': stacking_accuracy,\n        'stacking_roc_auc': stacking_roc_auc\n    })\n\n    # Return the negative accuracy as Optuna aims to minimize the objective\n    return -stacking_accuracy<\/pre>\n\n\n\n<p>As you can see, Comet ML can help you log and track the hyperparameter tuning process, allowing you to compare different runs and select the best hyperparameters.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">6. Optimization Results<\/h3>\n\n\n\n<p>Next, we display the best hyperparameters and accuracy scores achieved through hyperparameter tuning, providing transparency in our model selection process.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">from tabulate import tabulate\n\n# Create and optimize the study\nstudy = optuna.create_study(direction='minimize')  # Adjust direction based on your optimization goal\nstudy.optimize(objective, n_trials=100)  # You can adjust the number of trials\n\n# Get the best hyperparameters and results\nbest_rf_params = study.best_params\nbest_accuracy = -study.best_value  # Convert back to positive accuracy\n\n# Convert the dictionary to a list of key-value pairs for tabulation\nparam_table = [(key, value) for key, value in best_rf_params.items()]\n\n# Display the best_rf_params table\nbest_rf_params = tabulate(param_table, headers=[\"Parameter\", \"Value\"], tablefmt=\"grid\")\n\nprint(f\"Best RF Hyperparameters:\\n{best_rf_params}\")\nprint(f\"Best Accuracy: {best_accuracy}\")<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">7. End Experiment<\/h3>\n\n\n\n<p>Finally, we conclude the Comet ML experiment, ensuring all relevant information is logged for future reference.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">experiment.end()<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">8. Business&nbsp;Insights<\/h3>\n\n\n\n<p>After running an experiment, you can check the results by going to the&nbsp;<em>Respective Experiment &gt; Experiment &gt; Dashboards<\/em>&nbsp;or the<em>Respective Experiment &gt; Experiment &gt; Metrics.<\/em><\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*7q2xYRWk-_qXYCZz0PZ6Wg.png\" alt=\"Customer churn prediction with Comet ML\"\/><figcaption class=\"wp-element-caption\">Experiment Overview in CometML (Image by the&nbsp;Author)<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/2400\/1*Ryuxt9gqcRlfww577We8NQ.png\" alt=\"Predicting customer churn with Comet ML\"\/><figcaption class=\"wp-element-caption\">Logged Metrics in CometML (Image by the&nbsp;Author)<\/figcaption><\/figure>\n\n\n\n<p>Now, lets explore the business insights based on these optimization results:<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>#1. Model Selection:<\/strong><\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The project aimed to predict customer churn, a critical concern for telecom companies. Model stacking was used to enhance prediction accuracy.<\/li>\n\n\n\n<li>From the optimization results, we can see that Gradient Boosting (gb) achieved the highest accuracy (0.786) and ROC AUC (0.823) scores among individual models. It\u2019s a promising choice for predicting customer churn due to its good performance.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>#2. Ensemble Modeling:<\/strong><\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Stacking models (stacking_accuracy) achieved an accuracy of 0.783, slightly higher than the individual Random Forest (0.777) and Support Vector Machine (0.773) models.<\/li>\n\n\n\n<li>The stacking ensemble also outperformed in terms of ROC AUC (0.822), indicating its ability to balance false positives and true positives, which is crucial for customer churn prediction.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>#3. Hyperparameter Tuning:<\/strong><\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Hyperparameter tuning was conducted to optimize model performance.<\/li>\n\n\n\n<li>The optimized models, particularly Gradient Boosting and Random Forest, show improved accuracy compared to their default configurations.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\"><strong># 4. Decision-Making Insights:<\/strong><\/h4>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/0*N0ULKo9h1tTFHmW4.png\" alt=\"actionable customer churn prediction models live in the intersection between business knowledge and data science.\"\/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Telecom companies can use these models to identify customers at risk of churn.<\/li>\n\n\n\n<li>Strategies can be developed to retain high-risk customers, such as offering tailored promotions or better customer service.<\/li>\n\n\n\n<li>Insights from Gradient Boosting, which performed the best, can help identify key factors influencing churn, allowing the company to take proactive measures.<\/li>\n\n\n\n<li>The stacking ensemble provides robust predictions, combining the strengths of multiple models.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>#5. Monitoring and Continuous Improvement:<\/strong><\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Regular monitoring of churn prediction using these models can help the telecom company adapt to changing customer behaviors.<\/li>\n\n\n\n<li>Continuous hyperparameter tuning and model retraining can further enhance predictive accuracy over time.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Conclusion<\/h3>\n\n\n\n<p>In this article, we explored a churn prediction project using machine learning and Comet ML. A combination of model stacking, hyperparameter tuning, and insightful EDA will enable you to build robust churn prediction models.<\/p>\n\n\n\n<p>Predicting customer churn is just one application of machine learning in business, but the impact is significant. By leveraging tools like Comet ML, data scientists can optimize models and gain insights that ultimately contribute to improved customer retention strategies and business results.<\/p>\n\n\n\n<p>If you want to learn more about the world of machine learning and data science, keep an eye out for future articles. Remember that the power of data is in your hands and with the right tools and techniques you can make data-driven decisions that drive business success.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introduction In today\u2019s competitive business environment, retaining customers is essential to a company\u2019s success. Customer churn, or the rate at which customers leave your service, is an important metric that directly affects your business bottom line. To address this challenge, data scientists harness the power of machine learning to predict customer churn and develop strategies [&hellip;]<\/p>\n","protected":false},"author":95,"featured_media":7546,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[6,7],"tags":[40,14,68,69,16,53,56],"coauthors":[192],"class_list":["post-7545","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-machine-learning","category-tutorials","tag-comet","tag-comet-ml","tag-customer-churn","tag-fintech","tag-ml-experiment-management","tag-mlops","tag-optimization"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v25.9 (Yoast SEO v25.9) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Customer Churn With Continuous Experiment Tracking - Comet<\/title>\n<meta name=\"description\" content=\"In this hands-on project tutorial, learn why customer churn matters and how to predict it with machine learning, explained step-by-step.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/customer-churn-with-continuous-experiment-tracking\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Customer Churn With Continuous Experiment Tracking\" \/>\n<meta property=\"og:description\" content=\"In this hands-on project tutorial, learn why customer churn matters and how to predict it with machine learning, explained step-by-step.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/blog\/customer-churn-with-continuous-experiment-tracking\/\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2023-09-19T19:33:55+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-04-29T13:37:53+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/09\/Screen-Shot-2023-09-19-at-3.13.15-PM.png\" \/>\n\t<meta property=\"og:image:width\" content=\"600\" \/>\n\t<meta property=\"og:image:height\" content=\"602\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Nhi Yen\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Nhi Yen\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"13 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Customer Churn With Continuous Experiment Tracking - Comet","description":"In this hands-on project tutorial, learn why customer churn matters and how to predict it with machine learning, explained step-by-step.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/blog\/customer-churn-with-continuous-experiment-tracking\/","og_locale":"en_US","og_type":"article","og_title":"Customer Churn With Continuous Experiment Tracking","og_description":"In this hands-on project tutorial, learn why customer churn matters and how to predict it with machine learning, explained step-by-step.","og_url":"https:\/\/www.comet.com\/site\/blog\/customer-churn-with-continuous-experiment-tracking\/","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2023-09-19T19:33:55+00:00","article_modified_time":"2025-04-29T13:37:53+00:00","og_image":[{"width":600,"height":602,"url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/09\/Screen-Shot-2023-09-19-at-3.13.15-PM.png","type":"image\/png"}],"author":"Nhi Yen","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Nhi Yen","Est. reading time":"13 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.comet.com\/site\/blog\/customer-churn-with-continuous-experiment-tracking\/#article","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/blog\/customer-churn-with-continuous-experiment-tracking\/"},"author":{"name":"Nhi Yen","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/1a873c6cf609e07d582cd696f147609b"},"headline":"Customer Churn With Continuous Experiment Tracking","datePublished":"2023-09-19T19:33:55+00:00","dateModified":"2025-04-29T13:37:53+00:00","mainEntityOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/customer-churn-with-continuous-experiment-tracking\/"},"wordCount":1800,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/customer-churn-with-continuous-experiment-tracking\/#primaryimage"},"thumbnailUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/09\/Screen-Shot-2023-09-19-at-3.13.15-PM.png","keywords":["Comet","Comet ML","Customer Churn","FinTech","ML Experiment Management","MLOps","Optimization"],"articleSection":["Machine Learning","Tutorials"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/blog\/customer-churn-with-continuous-experiment-tracking\/","url":"https:\/\/www.comet.com\/site\/blog\/customer-churn-with-continuous-experiment-tracking\/","name":"Customer Churn With Continuous Experiment Tracking - Comet","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/customer-churn-with-continuous-experiment-tracking\/#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/customer-churn-with-continuous-experiment-tracking\/#primaryimage"},"thumbnailUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/09\/Screen-Shot-2023-09-19-at-3.13.15-PM.png","datePublished":"2023-09-19T19:33:55+00:00","dateModified":"2025-04-29T13:37:53+00:00","description":"In this hands-on project tutorial, learn why customer churn matters and how to predict it with machine learning, explained step-by-step.","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/blog\/customer-churn-with-continuous-experiment-tracking\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/blog\/customer-churn-with-continuous-experiment-tracking\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/blog\/customer-churn-with-continuous-experiment-tracking\/#primaryimage","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/09\/Screen-Shot-2023-09-19-at-3.13.15-PM.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/09\/Screen-Shot-2023-09-19-at-3.13.15-PM.png","width":600,"height":602},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/blog\/customer-churn-with-continuous-experiment-tracking\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"Customer Churn With Continuous Experiment Tracking"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/1a873c6cf609e07d582cd696f147609b","name":"Nhi Yen","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/cbe7005c33fc937d23d6bbbff99e5223","url":"https:\/\/secure.gravatar.com\/avatar\/ec9f8f996211d944f352679e89c48b4cdaf7a1609d7409846408ac93045893d9?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/ec9f8f996211d944f352679e89c48b4cdaf7a1609d7409846408ac93045893d9?s=96&d=mm&r=g","caption":"Nhi Yen"},"url":"https:\/\/www.comet.com\/site\/blog\/author\/nhi-yen\/"}]}},"jetpack_featured_media_url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/09\/Screen-Shot-2023-09-19-at-3.13.15-PM.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7545","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/95"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=7545"}],"version-history":[{"count":3,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7545\/revisions"}],"predecessor-version":[{"id":15804,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7545\/revisions\/15804"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media\/7546"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=7545"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=7545"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=7545"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=7545"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}