{"id":5851,"date":"2023-05-25T12:38:23","date_gmt":"2023-05-25T20:38:23","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=5851"},"modified":"2025-04-24T17:15:32","modified_gmt":"2025-04-24T17:15:32","slug":"model-ci-cd-with-comet","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/model-ci-cd-with-comet\/","title":{"rendered":"Model CI\/CD with Comet"},"content":{"rendered":"\n

Machine Learning Teams are moving their models from solutions like Github and storing them in MLOps platforms like Comet. Comet’s Model Registry allows ML Organizations to access the full training lineage of a Model.  <\/span>However, the Model Lifecycle doesn\u2019t end at a Model Store. Eventually,  teams deploy a model to production and monitor it for drift. As models drift, teams re-train the model on a new version of the data. Updating a model in production is an arduous task that raises a lot of questions. When does a model needs to be updated?  Can we verify if the re-trained model is better than the original? How can we automate the deployment of the updated models? Who has the final say to move a model production? <\/span>ML Teams need a Model Storing Solution like Comet that supports all these Model CI\/CD Workflows.
\n<\/span><\/p>\n\n\n\n

Detect the Need for Model Re-Training<\/span><\/h2>\n\n\n\n

Machine Learning Models Models are rarely “set and forget”. As models start to underperform or “drift” in production, ML teams need to be aware as-soon-as possible, so they can commence the re-training process. Comet’s Model Production Monitoring (MPM)  solution stores and analyzes a model’s predictions to detect where and how a model is failing.  Here are some of the ways MPM helps ML teams debug their models in productions<\/p>\n\n\n\n

    \n
  1. Detect Model Output Drift<\/li>\n\n\n\n
  2. Detect Model Input Feature Drift<\/li>\n\n\n\n
  3. Detect Model Accuracy Drift<\/li>\n\n\n\n
  4. Detect Data Drift from Training Distribution<\/li>\n\n\n\n
  5. Detect Model Bias via Segmentation<\/li>\n<\/ol>\n\n\n\n
    \"mpm<\/figure>\n\n\n\n

    <\/p>\n\n\n\n

    Any of the occurrences listed above warrants a model re-training; ideally on a new dataset with some added data from inference.<\/p>\n\n\n\n

    Compare and Contrast Model Versions<\/span><\/h2>\n\n\n\n

    Comet’s Model Registry has full lineage to the Experiment Management platform. Users can access the experiment run associated with a particular Model. Since Comet stores all the necessary metadata needed to reproduce a model training run, re-training a model on a new dataset is often trivial. New and Updated Dataset Versions are tracked via Comet Artifacts. Comet’s Built-In Panels can be used to compare model performance at high level metrics such as loss, accuracy, mAP, recall and also compare model predictions on specific data-samples.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    <\/p>\n\n\n\n

    Downstream Model Updates via Webhooks<\/span><\/h2>\n\n\n\n

     A re-trained model is added to the Model Registry with an incremented Model Version Number. Each model has an associated status field with the options to assign the following fields: “Development”, “QA”, “Staging”, “Production”. For comprehensive model management, Comet exposes changes in the Model Registry via Webhooks. Webhooks enable teams to automate<\/strong> their model promotion process. Here’s how Comet Customers are leveraging this feature in their workflows today.<\/span><\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    <\/p>\n\n\n\n

      \n
    1. Add\/Delete a Model from the Registry ->\u00a0 Send an alert message on Slack\/Teams<\/span><\/li>\n\n\n\n
    2. Change a Model’s Status to “QA” -> Create an automatic Jira ticket for someone to review it<\/span><\/li>\n\n\n\n
    3. Change a Model Status to \u201cStaging\u201d to \u201cProduction\u201d,\u00a0 trigger your deployment CI\/CD with Sagemaker or Github Actions. <\/span><\/li>\n<\/ol>\n\n\n\n

      Approve Model Promotions<\/span><\/h2>\n\n\n\n

      Machine Learning Models are making high-impact decisions for organizations across the globe. Driving Cars, Analyzing Medical Images, Recommending Loan Amounts, and Detecting Credit Card Fraud. A new model update can have a profound impact on a ML system. In the software world, pull requests are approved by a committee of engineers before it is merged into the main branch. <\/span><\/p>\n\n\n\n

      Since changing the Model Status can affect downstream systems with our Webhooks functionality, in Comet, ML Engineers have to \u201crequest\u201d to change the status of the model. Only Workspace Admin(s) then have the authority  to either \u201capprove\u201d or \u201creject\u201d the request. For organizations where Model Auditing is a requirement, data scientists use Comet Reports<\/a> to supplement a model status request. <\/span><\/p>\n\n\n\n

      \"model<\/figure>\n\n\n\n

      <\/p>\n\n\n\n

      Store Your Production Models in Comet<\/span><\/h2>\n\n\n\n

      As teams deploy more models to production, a cohesive Model CI\/CD process becomes a “must have” rather than “a nice to have”. Comet is incredibly easy to integrate with your current ML workflows. Sign-up<\/a> for a free account today and try it out!<\/p>\n","protected":false},"excerpt":{"rendered":"

      Machine Learning Teams are moving their models from solutions like Github and storing them in MLOps platforms like Comet. Comet’s Model Registry allows ML Organizations to access the full training lineage of a Model.  However, the Model Lifecycle doesn\u2019t end at a Model Store. Eventually,  teams deploy a model to production and monitor it for […]<\/p>\n","protected":false},"author":21,"featured_media":5887,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[8,9],"tags":[],"coauthors":[134],"class_list":["post-5851","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-comet-community-hub","category-product"],"yoast_head":"\nModel CI\/CD with Comet - Comet<\/title>\n<meta name=\"description\" content=\"Comet's MLOps platform helps teams build out a cohesive Model CI\/CD workflow\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/model-ci-cd-with-comet\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Model CI\/CD with Comet\" \/>\n<meta property=\"og:description\" content=\"Comet's MLOps platform helps teams build out a cohesive Model CI\/CD workflow\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/blog\/model-ci-cd-with-comet\/\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2023-05-25T20:38:23+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-04-24T17:15:32+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/05\/mr_snapshot.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1750\" \/>\n\t<meta property=\"og:image:height\" content=\"843\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Siddharth Mehta\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Siddharth Mehta\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Model CI\/CD with Comet - Comet","description":"Comet's MLOps platform helps teams build out a cohesive Model CI\/CD workflow","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/blog\/model-ci-cd-with-comet\/","og_locale":"en_US","og_type":"article","og_title":"Model CI\/CD with Comet","og_description":"Comet's MLOps platform helps teams build out a cohesive Model CI\/CD workflow","og_url":"https:\/\/www.comet.com\/site\/blog\/model-ci-cd-with-comet\/","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2023-05-25T20:38:23+00:00","article_modified_time":"2025-04-24T17:15:32+00:00","og_image":[{"width":1750,"height":843,"url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/05\/mr_snapshot.png","type":"image\/png"}],"author":"Siddharth Mehta","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Siddharth Mehta","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.comet.com\/site\/blog\/model-ci-cd-with-comet\/#article","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/blog\/model-ci-cd-with-comet\/"},"author":{"name":"Siddharth Mehta","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/652eb7d782d18f295922f50ea3b9e54c"},"headline":"Model CI\/CD with Comet","datePublished":"2023-05-25T20:38:23+00:00","dateModified":"2025-04-24T17:15:32+00:00","mainEntityOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/model-ci-cd-with-comet\/"},"wordCount":669,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/model-ci-cd-with-comet\/#primaryimage"},"thumbnailUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/05\/mr_snapshot.png","articleSection":["Comet Community Hub","Product"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/blog\/model-ci-cd-with-comet\/","url":"https:\/\/www.comet.com\/site\/blog\/model-ci-cd-with-comet\/","name":"Model CI\/CD with Comet - Comet","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/model-ci-cd-with-comet\/#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/model-ci-cd-with-comet\/#primaryimage"},"thumbnailUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/05\/mr_snapshot.png","datePublished":"2023-05-25T20:38:23+00:00","dateModified":"2025-04-24T17:15:32+00:00","description":"Comet's MLOps platform helps teams build out a cohesive Model CI\/CD workflow","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/blog\/model-ci-cd-with-comet\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/blog\/model-ci-cd-with-comet\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/blog\/model-ci-cd-with-comet\/#primaryimage","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/05\/mr_snapshot.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/05\/mr_snapshot.png","width":1750,"height":843,"caption":"model versions screen in Comet"},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/blog\/model-ci-cd-with-comet\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"Model CI\/CD with Comet"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/652eb7d782d18f295922f50ea3b9e54c","name":"Siddharth Mehta","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/940c7280faea9e1b8b086c2ed7ec01db","url":"https:\/\/secure.gravatar.com\/avatar\/27a672e997fa7a66796e4be0503e0efeec6bd34daae185bb6de163227a5a0739?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/27a672e997fa7a66796e4be0503e0efeec6bd34daae185bb6de163227a5a0739?s=96&d=mm&r=g","caption":"Siddharth Mehta"},"description":"ML Growth Engineer @ Comet. Interested in Computer Vision, Robotics, and Reinforcement Learning","sameAs":["https:\/\/www.comet.com\/"],"url":"https:\/\/www.comet.com\/site\/blog\/author\/siddharthmcomet-com\/"}]}},"jetpack_featured_media_url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/05\/mr_snapshot.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/5851","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=5851"}],"version-history":[{"count":1,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/5851\/revisions"}],"predecessor-version":[{"id":15621,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/5851\/revisions\/15621"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media\/5887"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=5851"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=5851"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=5851"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=5851"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}