{"id":8090,"date":"2023-11-02T10:16:15","date_gmt":"2023-11-02T18:16:15","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=8090"},"modified":"2025-04-24T17:04:44","modified_gmt":"2025-04-24T17:04:44","slug":"streamline-ml-model-development-with-gitlabs-devops-platform-and-comet","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/streamline-ml-model-development-with-gitlabs-devops-platform-and-comet\/","title":{"rendered":"Streamline ML Model Development with GitLab\u2019s DevOps Platform and Comet"},"content":{"rendered":"\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/streamline-ml-model-development-with-gitlabs-devops-platform-and-comet\">\n\n\n\n<div class=\"fk fl fm fn fo\">\n<div class=\"ab ca\">\n<div class=\"ch bg ew ex ey ez\">\n<figure class=\"mk ml mm mn mo mp mh mi paragraph-image\">\n<div class=\"mq mr ee ms bg mt\" tabindex=\"0\" role=\"button\">\n<figure><img loading=\"lazy\" decoding=\"async\" class=\"bg mu mv c\" role=\"presentation\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*F3DxwRnz8uUGcza2St0zCA.png\" alt=\"\" width=\"700\" height=\"394\"><\/figure><div class=\"mh mi mj\"><picture><\/picture><\/div>\n<\/div>\n<\/figure>\n<blockquote class=\"mw\"><p id=\"9867\" class=\"mx my fr be mz na nb nc nd ne nf ng dw\" data-selectable-paragraph=\"\">Editor\u2019s Note: This blog post was co-written alongside <a class=\"af nh\" href=\"https:\/\/www.linkedin.com\/in\/william-galindez-arias-87566542\/\" target=\"_blank\" rel=\"noopener ugc nofollow\">William Arias<\/a>, Technical Marketing Manager at GitLab, and has been syndicated with permission. <a class=\"af nh\" href=\"https:\/\/about.gitlab.com\/blog\/2021\/11\/08\/machine-learning-on-the-gitlab-devops-platform\/\" target=\"_blank\" rel=\"noopener ugc nofollow\">Check out the original version of this post on GitLab\u2019s blog.<\/a><\/p><\/blockquote>\n<h1 id=\"fe3a\" class=\"ni nj fr be nk nl nm gr nn no np gu nq nr ns nt nu nv nw nx ny nz oa ob oc od bj\" data-selectable-paragraph=\"\"><strong class=\"al\">Introduction<\/strong><\/h1>\n<p id=\"46d8\" class=\"pw-post-body-paragraph oe of fr be b gp og oh oi gs oj ok ol om on oo op oq or os ot ou ov ow ox ng fk bj\" data-selectable-paragraph=\"\">Building ML-powered applications comes with numerous challenges. When we talk about these challenges, there is a tendency to overly focus on problems related to the quality of a model\u2019s predictions \u2014 things like data drift, changes in model architectures, or inference latency.<\/p>\n<p id=\"15ec\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">While these are all problems worthy of deep consideration, an often overlooked challenge in ML development is the process of integrating a model into an existing software application.<\/p>\n<p id=\"fd82\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">If you\u2019re tasked with adding an ML feature to a product, you will almost certainly run into an existing codebase that must play nicely with your model. This is, to put it mildly, not an easy task.<\/p>\n<p id=\"cb65\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">ML is a highly iterative discipline. Teams often make many changes to their codebase and pipelines in the process of developing a model. Coupling an ML codebase to an application\u2019s dependencies, unit tests, and CI\/CD pipelines will significantly reduce the velocity with which ML teams can deliver on a solution, since each change would require running these downstream dependencies before a merge can be approved.<\/p>\n<p id=\"4572\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">In this post, we\u2019re going to demonstrate how you can use <a class=\"af nh\" href=\"https:\/\/www.comet.com\/site\/\" target=\"_blank\" rel=\"noopener ugc nofollow\">Comet<\/a> with GitLab\u2019s DevOps platform to streamline the workflow for your ML and Software Engineering teams, allowing them to collaborate without getting in each other\u2019s way.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n<div class=\"fk fl fm fn fo\">\n<div class=\"ab ca\">\n<div class=\"ch bg ew ex ey ez\">\n<blockquote class=\"mw\"><p id=\"003a\" class=\"mx my fr be mz na pl pm pn po pp ng dw\" data-selectable-paragraph=\"\">TL;DR \u2014 Check out this technical walkthrough for an inside look at how you can add reliable CI\/CD pipelines to your ML workflows with GitLab + Comet.<\/p><\/blockquote>\n<figure class=\"pq pr ps pt pu mp\">\n<div class=\"pv iu l ee\">\n<div class=\"pw px l\"><iframe loading=\"lazy\" class=\"eo n ff dy bg\" title=\"GitLab DevOps Platform + Comet ML. Automate Machine Learning Pipelines MNIST Demo\" src=\"https:\/\/cdn.embedly.com\/widgets\/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FW_DsNl5aAVk%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DW_DsNl5aAVk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FW_DsNl5aAVk%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube\" width=\"854\" height=\"480\" frameborder=\"0\" scrolling=\"no\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/div>\n<\/div>\n<\/figure>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n<div class=\"fk fl fm fn fo\">\n<div class=\"ab ca\">\n<div class=\"ch bg ew ex ey ez\">\n<h1 id=\"126c\" class=\"ni nj fr be nk nl py gr nn no pz gu nq nr qa nt nu nv qb nx ny nz qc ob oc od bj\" data-selectable-paragraph=\"\"><strong class=\"al\">The Challenge for ML Teams working with Application Teams<\/strong><\/h1>\n<p id=\"fc7e\" class=\"pw-post-body-paragraph oe of fr be b gp og oh oi gs oj ok ol om on oo op oq or os ot ou ov ow ox ng fk bj\" data-selectable-paragraph=\"\">Let\u2019s say your team is working on improving a feature engineering pipeline. You will likely have to test many combinations of features with some baseline model for the task, to see which combinations make an impact on model performance.<\/p>\n<p id=\"8a31\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">It is hard to know beforehand which features might be significant, so having to run multiple experiments is inevitable. If your ML code is a part of your application codebase, this would mean having to run your application\u2019s CI\/CD pipeline for every feature combination you might be trying.<\/p>\n<p id=\"94e0\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">This will certainly frustrate your Engineering and DevOps teams, since you would be unnecessarily tying up system resources, given that Software Engineering teams do not need to run their pipelines with the same frequency as ML teams do.<\/p>\n<p id=\"3df0\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">The other issue is that despite having to run numerous experiments, only a single set of outputs from these experiments will make it to your production application. Therefore, the rest of the assets produced through these experiments are not relevant to your application code.<\/p>\n<p id=\"8890\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">Keeping these two codebases separated will make life a lot easier for everyone \u2014 but it also introduces the problem of syncing the latest model between two codebases.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n<div class=\"fk fl fm fn fo\">\n<div class=\"ab ca\">\n<div class=\"ch bg ew ex ey ez\">\n<h1 id=\"f6e0\" class=\"ni nj fr be nk nl py gr nn no pz gu nq nr qa nt nu nv qb nx ny nz qc ob oc od bj\" data-selectable-paragraph=\"\"><strong class=\"al\">Using GitLab DevOps Platform and Comet for your Model Development Process<\/strong><\/h1>\n<p id=\"e6bc\" class=\"pw-post-body-paragraph oe of fr be b gp og oh oi gs oj ok ol om on oo op oq or os ot ou ov ow ox ng fk bj\" data-selectable-paragraph=\"\">With the GitLab DevOps platform and Comet, we can keep the workflows between ML and Engineering teams separated, while enabling cross-team collaboration by preserving the visibility and audit-ability of the entire model development process across teams.<\/p>\n<p id=\"d094\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">We will use two separate projects to demonstrate this process. One project will contain our application code for a handwritten digit recognizer, while the other will contain all the code relevant to training and evaluating our model.<\/p>\n<p id=\"ec3a\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">We will adopt a process where discussions, code reviews, and model performance metrics get automatically published and tracked within the GitLab DevOps platform, increasing the velocity and opportunity for collaboration between data scientists and software engineers for machine learning workflows.<\/p>\n<h2 id=\"16c3\" class=\"qd nj fr be nk qe qf qg nn qh qi qj nq om qk ql qm oq qn qo qp ou qq qr qs qt bj\" data-selectable-paragraph=\"\">Project Setup<\/h2>\n<p id=\"3192\" class=\"pw-post-body-paragraph oe of fr be b gp og oh oi gs oj ok ol om on oo op oq or os ot ou ov ow ox ng fk bj\" data-selectable-paragraph=\"\">Our application consists of two projects. The <a class=\"af nh\" href=\"https:\/\/gitlab.com\/tech-marketing\/devops-platform\/comet-model-trainer\" target=\"_blank\" rel=\"noopener ugc nofollow\"><strong class=\"be qu\">comet-model-trainer<\/strong><\/a> and <a class=\"af nh\" href=\"https:\/\/gitlab.com\/tech-marketing\/devops-platform\/canara-review-apps-testing\" target=\"_blank\" rel=\"noopener ugc nofollow\"><strong class=\"be qu\">ml-ui<\/strong><\/a>.<\/p>\n<\/div>\n<\/div>\n<div class=\"mp\">\n<div class=\"ab ca\">\n<div class=\"qv qw qx qy qz ra ce rb cf rc ch bg\">\n<figure class=\"mk ml mm mn mo mp re rf paragraph-image\">\n<div class=\"mq mr ee ms bg mt\" tabindex=\"0\" role=\"button\">\n<figure><img loading=\"lazy\" decoding=\"async\" class=\"bg mu mv c\" role=\"presentation\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/0*3y59sKAplhAv9kK2\" alt=\"\" width=\"1000\" height=\"424\"><\/figure><div class=\"mh mi rd\"><picture><source srcset=\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/0*3y59sKAplhAv9kK2 640w, https:\/\/miro.medium.com\/v2\/resize:fit:720\/0*3y59sKAplhAv9kK2 720w, https:\/\/miro.medium.com\/v2\/resize:fit:750\/0*3y59sKAplhAv9kK2 750w, https:\/\/miro.medium.com\/v2\/resize:fit:786\/0*3y59sKAplhAv9kK2 786w, https:\/\/miro.medium.com\/v2\/resize:fit:828\/0*3y59sKAplhAv9kK2 828w, https:\/\/miro.medium.com\/v2\/resize:fit:1100\/0*3y59sKAplhAv9kK2 1100w, https:\/\/miro.medium.com\/v2\/resize:fit:2000\/0*3y59sKAplhAv9kK2 2000w\" type=\"image\/webp\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 1000px\"><source srcset=\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/0*3y59sKAplhAv9kK2 640w, https:\/\/miro.medium.com\/v2\/resize:fit:720\/0*3y59sKAplhAv9kK2 720w, https:\/\/miro.medium.com\/v2\/resize:fit:750\/0*3y59sKAplhAv9kK2 750w, https:\/\/miro.medium.com\/v2\/resize:fit:786\/0*3y59sKAplhAv9kK2 786w, https:\/\/miro.medium.com\/v2\/resize:fit:828\/0*3y59sKAplhAv9kK2 828w, https:\/\/miro.medium.com\/v2\/resize:fit:1100\/0*3y59sKAplhAv9kK2 1100w, https:\/\/miro.medium.com\/v2\/resize:fit:2000\/0*3y59sKAplhAv9kK2 2000w\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 1000px\" data-testid=\"og\"><\/picture><\/div>\n<\/div>\n<\/figure>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"ab ca\">\n<div class=\"ch bg ew ex ey ez\">\n<p id=\"4a92\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">The <strong class=\"be qu\">comet-model-trainer <\/strong>repository contains scripts to train and evaluate a model on the MNIST dataset. We have set up the GitLab DevOps Platform in a way that runs the training and evaluation Pipeline whenever a new Merge Request is opened with the necessary changes.<\/p>\n<p id=\"5dc6\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">The <strong class=\"be qu\">ml-ui <\/strong>repository contains the necessary code to build the frontend of our ML application.<\/p>\n<p id=\"fc06\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">Since the code is integrated with Comet, your ML team can easily track the source code, hyperparameters, metrics, and other details related to the development of the model.<\/p>\n<p id=\"b317\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">Once the training and evaluation steps are completed, we can use Comet to fetch summary metrics from the Project as well as metrics from the Candidate model and display them within the Merge Request \u2014 this will allow the ML team to easily review the changes to the model.<\/p>\n<\/div>\n<\/div>\n<div class=\"mp\">\n<div class=\"ab ca\">\n<div class=\"qv qw qx qy qz ra ce rb cf rc ch bg\">\n<figure class=\"mk ml mm mn mo mp re rf paragraph-image\">\n<div class=\"mq mr ee ms bg mt\" tabindex=\"0\" role=\"button\">\n<figure><img loading=\"lazy\" decoding=\"async\" class=\"bg mu mv c\" role=\"presentation\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*RwxS9QGPlwp39Moz2yESuA.gif\" alt=\"\" width=\"1000\" height=\"563\"><\/figure><div class=\"mh mi rg\"><picture><source srcset=\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/1*RwxS9QGPlwp39Moz2yESuA.gif 640w, https:\/\/miro.medium.com\/v2\/resize:fit:720\/1*RwxS9QGPlwp39Moz2yESuA.gif 720w, https:\/\/miro.medium.com\/v2\/resize:fit:750\/1*RwxS9QGPlwp39Moz2yESuA.gif 750w, https:\/\/miro.medium.com\/v2\/resize:fit:786\/1*RwxS9QGPlwp39Moz2yESuA.gif 786w, https:\/\/miro.medium.com\/v2\/resize:fit:828\/1*RwxS9QGPlwp39Moz2yESuA.gif 828w, https:\/\/miro.medium.com\/v2\/resize:fit:1100\/1*RwxS9QGPlwp39Moz2yESuA.gif 1100w, https:\/\/miro.medium.com\/v2\/resize:fit:2000\/1*RwxS9QGPlwp39Moz2yESuA.gif 2000w\" type=\"image\/webp\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 1000px\"><source srcset=\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/1*RwxS9QGPlwp39Moz2yESuA.gif 640w, https:\/\/miro.medium.com\/v2\/resize:fit:720\/1*RwxS9QGPlwp39Moz2yESuA.gif 720w, https:\/\/miro.medium.com\/v2\/resize:fit:750\/1*RwxS9QGPlwp39Moz2yESuA.gif 750w, https:\/\/miro.medium.com\/v2\/resize:fit:786\/1*RwxS9QGPlwp39Moz2yESuA.gif 786w, https:\/\/miro.medium.com\/v2\/resize:fit:828\/1*RwxS9QGPlwp39Moz2yESuA.gif 828w, https:\/\/miro.medium.com\/v2\/resize:fit:1100\/1*RwxS9QGPlwp39Moz2yESuA.gif 1100w, https:\/\/miro.medium.com\/v2\/resize:fit:2000\/1*RwxS9QGPlwp39Moz2yESuA.gif 2000w\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 1000px\" data-testid=\"og\"><\/picture><\/div>\n<\/div><figcaption class=\"rh ri rj mh mi rk rl be b bf z dw\" data-selectable-paragraph=\"\"><a class=\"af nh\" href=\"https:\/\/gitlab.com\/tech-marketing\/devops-platform\/comet-model-trainer\/-\/commit\/fb6600dfd1e24a9a0aba3b9eb7e9a75960516c50\" target=\"_blank\" rel=\"noopener ugc nofollow\">Experiment Metrics Displayed in the Merge Request<\/a><\/figcaption><\/figure>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"ab ca\">\n<div class=\"ch bg ew ex ey ez\">\n<p id=\"d808\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">In our case, the average accuracy of the models in the Project is 97%. Our Candidate model achieved an accuracy of 99%, so it looks like it is a good fit to promote to production. The metrics displayed here are completely configurable and can be changed as necessary.<\/p>\n<p id=\"6db0\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">When the Merge Request is approved, the deployment pipeline is triggered and the model is pushed to Comet\u2019s Model Registry. The Model Registry versions each model and links it back to the Comet Experiment that produced it.<\/p>\n<p id=\"be77\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">Once the model is pushed to the Model Registry, it is available to the application code. When the Application Team wishes to deploy this new version of the model to their app, they simply have to trigger their specific deployment pipeline.<\/p>\n<h2 id=\"8ebf\" class=\"qd nj fr be nk qe qf qg nn qh qi qj nq om qk ql qm oq qn qo qp ou qq qr qs qt bj\" data-selectable-paragraph=\"\"><strong class=\"al\">Running the Pipeline<\/strong><\/h2>\n<figure class=\"mk ml mm mn mo mp mh mi paragraph-image\">\n<div class=\"mq mr ee ms bg mt\" tabindex=\"0\" role=\"button\">\n<figure><img loading=\"lazy\" decoding=\"async\" class=\"bg mu mv c\" role=\"presentation\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*VtPOX689OJ3SjRNK\" alt=\"\" width=\"700\" height=\"740\"><\/figure><div class=\"mh mi rm\"><picture><source srcset=\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/0*VtPOX689OJ3SjRNK 640w, https:\/\/miro.medium.com\/v2\/resize:fit:720\/0*VtPOX689OJ3SjRNK 720w, https:\/\/miro.medium.com\/v2\/resize:fit:750\/0*VtPOX689OJ3SjRNK 750w, https:\/\/miro.medium.com\/v2\/resize:fit:786\/0*VtPOX689OJ3SjRNK 786w, https:\/\/miro.medium.com\/v2\/resize:fit:828\/0*VtPOX689OJ3SjRNK 828w, https:\/\/miro.medium.com\/v2\/resize:fit:1100\/0*VtPOX689OJ3SjRNK 1100w, https:\/\/miro.medium.com\/v2\/resize:fit:1400\/0*VtPOX689OJ3SjRNK 1400w\" type=\"image\/webp\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px\"><source srcset=\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/0*VtPOX689OJ3SjRNK 640w, https:\/\/miro.medium.com\/v2\/resize:fit:720\/0*VtPOX689OJ3SjRNK 720w, https:\/\/miro.medium.com\/v2\/resize:fit:750\/0*VtPOX689OJ3SjRNK 750w, https:\/\/miro.medium.com\/v2\/resize:fit:786\/0*VtPOX689OJ3SjRNK 786w, https:\/\/miro.medium.com\/v2\/resize:fit:828\/0*VtPOX689OJ3SjRNK 828w, https:\/\/miro.medium.com\/v2\/resize:fit:1100\/0*VtPOX689OJ3SjRNK 1100w, https:\/\/miro.medium.com\/v2\/resize:fit:1400\/0*VtPOX689OJ3SjRNK 1400w\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px\" data-testid=\"og\"><\/picture><\/div>\n<\/div>\n<figcaption class=\"rh ri rj mh mi rk rl be b bf z dw\" data-selectable-paragraph=\"\">Pipeline Outline<\/figcaption>\n<\/figure>\n<p id=\"8157\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">We will run the process outlined below every time a team member creates a merge request to change code in the <code class=\"cw rn ro rp rq b\">build-neural-network<\/code> script.<\/p>\n<p id=\"2cc4\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">Now, let\u2019s take a look at the <code class=\"cw rn ro rp rq b\">yaml<\/code> config used to define our CI\/CD pipelines depicted in the previous diagram:<\/p>\n<pre class=\"mk ml mm mn mo rr rq rs rt ax ru bj\"><span id=\"668c\" class=\"qd nj fr rq b ic rv rw l is rx\" data-selectable-paragraph=\"\">image: python:3.8\n\ninclude:\n  - template: 'Workflows\/Branch-Pipelines.gitlab-ci.yml'\n\nbuild-neural-network:\n    script:\n        - mkdir artifacts\n        - pip install -r requirements.txt\n        - python mnist-dnn-rich.py\n\n    artifacts:\n        paths:\n            - \".\/artifacts\"\n        expire_in: 30 days\n\nwrite-report-mr:\n    needs: [build-neural-network]\n    image: dvcorg\/cml-py3:latest\n    script:\n        - cd artifacts\n        - echo &gt; plots.md\n        - cml-publish model_accuracy.png --md --title 'Model Accuracy' &gt;&gt; plots.md\n        - cml-send-comment summary.md\n        - cml-send-comment plots.md\n        - echo \"Published results to MR\"\n\nregister-model:\n    needs: [write-report-mr]\n    script:\n        - pip install -r requirements.txt\n        -  python register_model.py --workspace $COMET_WORKSPACE --project_name $COMET_PROJECT_NAME --model_name mnist-neural-net\n        - echo \"model registered\"\n    only:\n        - main\n\ndeploy-ml-ui:\n    trigger: tech-marketing\/devops-platform\/canara-review-apps-testing\n    rules:\n     - if: '$CI_COMMIT_BRANCH == \"main\"'\n       when: manual<\/span><\/pre>\n<p id=\"9b6e\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">Let\u2019s break down the CI\/CD pipeline by describing the gitlab-ci.yml file so you can use it and customize it to your needs<\/p>\n<p id=\"e65f\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">We start by instructing our GitLab runners to utilize python:3.8 to run the jobs specified in the pipeline:<\/p>\n<ul class=\"\">\n<li id=\"f5d2\" class=\"oe of fr be b gp oy oh oi gs oz ok ol om ry oo op oq rz os ot ou sa ow ox ng sb sc sd bj\" data-selectable-paragraph=\"\"><code class=\"cw rn ro rp rq b\"><strong class=\"be qu\">Image:<\/strong> python:3.8<\/code><\/li>\n<\/ul>\n<p id=\"ae71\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">Then, we define the job where we want to build and train the neural network:<\/p>\n<ul class=\"\">\n<li id=\"8bd5\" class=\"oe of fr be b gp oy oh oi gs oz ok ol om ry oo op oq rz os ot ou sa ow ox ng sb sc sd bj\" data-selectable-paragraph=\"\"><code class=\"cw rn ro rp rq b\"><strong class=\"be qu\">build-neural-network<\/strong><\/code><\/li>\n<\/ul>\n<p id=\"5f95\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">In this step, we start by creating a folder where we will store the artifacts generated by this job, install dependencies using the requirements.txt file, and finally execute the corresponding Python script that will be in charge of training the neural network. The training runs in the GitLab runner using the Python image defined above, along with its dependencies.<\/p>\n<p id=\"38e3\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">Once the <code class=\"cw rn ro rp rq b\">build-neural-network<\/code> job has finalized successfully, we move to the next job: <code class=\"cw rn ro rp rq b\">write-report-mr<\/code><\/p>\n<p id=\"4360\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">Here, we use another image created by DVC that will allow us to publish a report right in the merge request opened by the contributor who changed code in the neural network script \u2014 in this way, we\u2019ve brought software development workflows to the development of ML applications. With the report provided by this job, code and model review can be executed within the Merge Request view, enabling teams to collaborate not only around the code but also the model performance.<\/p>\n<p id=\"bf78\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">From the Merge Request Page, we get access to loss curves and other relevant performance metrics from the model we are training, along with a link to the Comet Experiment UI, where richer details are provided to evaluate the model performance. These details include interactive charts for model metrics, the model hyperparameters, and Confusion Matrices of the test set performance, to name a few.<\/p>\n<figure class=\"mk ml mm mn mo mp mh mi paragraph-image\">\n<div class=\"mq mr ee ms bg mt\" tabindex=\"0\" role=\"button\">\n<figure><img loading=\"lazy\" decoding=\"async\" class=\"bg mu mv c\" role=\"presentation\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*pPfB6D0ZSPAUVnn1qujSDg.gif\" alt=\"\" width=\"700\" height=\"375\"><\/figure><div class=\"mh mi se\"><picture><source srcset=\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/1*pPfB6D0ZSPAUVnn1qujSDg.gif 640w, https:\/\/miro.medium.com\/v2\/resize:fit:720\/1*pPfB6D0ZSPAUVnn1qujSDg.gif 720w, https:\/\/miro.medium.com\/v2\/resize:fit:750\/1*pPfB6D0ZSPAUVnn1qujSDg.gif 750w, https:\/\/miro.medium.com\/v2\/resize:fit:786\/1*pPfB6D0ZSPAUVnn1qujSDg.gif 786w, https:\/\/miro.medium.com\/v2\/resize:fit:828\/1*pPfB6D0ZSPAUVnn1qujSDg.gif 828w, https:\/\/miro.medium.com\/v2\/resize:fit:1100\/1*pPfB6D0ZSPAUVnn1qujSDg.gif 1100w, https:\/\/miro.medium.com\/v2\/resize:fit:1400\/1*pPfB6D0ZSPAUVnn1qujSDg.gif 1400w\" type=\"image\/webp\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px\"><source srcset=\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/1*pPfB6D0ZSPAUVnn1qujSDg.gif 640w, https:\/\/miro.medium.com\/v2\/resize:fit:720\/1*pPfB6D0ZSPAUVnn1qujSDg.gif 720w, https:\/\/miro.medium.com\/v2\/resize:fit:750\/1*pPfB6D0ZSPAUVnn1qujSDg.gif 750w, https:\/\/miro.medium.com\/v2\/resize:fit:786\/1*pPfB6D0ZSPAUVnn1qujSDg.gif 786w, https:\/\/miro.medium.com\/v2\/resize:fit:828\/1*pPfB6D0ZSPAUVnn1qujSDg.gif 828w, https:\/\/miro.medium.com\/v2\/resize:fit:1100\/1*pPfB6D0ZSPAUVnn1qujSDg.gif 1100w, https:\/\/miro.medium.com\/v2\/resize:fit:1400\/1*pPfB6D0ZSPAUVnn1qujSDg.gif 1400w\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px\" data-testid=\"og\"><\/picture><\/div>\n<\/div>\n<figcaption class=\"rh ri rj mh mi rk rl be b bf z dw\" data-selectable-paragraph=\"\">Open Experiment View From a Merge Request<\/figcaption>\n<\/figure>\n<p id=\"5e51\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">When the team is done with the code and model review, the Merge Request gets approved, and the script that generated the model is merged into the main codebase, along with its respective commit and CI pipeline associated to it, taking us to the next job:<\/p>\n<ul class=\"\">\n<li id=\"396c\" class=\"oe of fr be b gp oy oh oi gs oz ok ol om ry oo op oq rz os ot ou sa ow ox ng sb sc sd bj\" data-selectable-paragraph=\"\"><code class=\"cw rn ro rp rq b\"><strong class=\"be qu\">Register-Model<\/strong><\/code><\/li>\n<\/ul>\n<p id=\"aaa1\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">This job uses an integration between GitLab and Comet to upload the reviewed and accepted version of the model to the Comet Model Registry. If you recall, the Model Registry is where models intended for production can be logged and versioned. In order to run the commands that will register the model, we need to set up these variables:<\/p>\n<pre class=\"mk ml mm mn mo rr rq rs rt ax ru bj\"><span id=\"e2ed\" class=\"qd nj fr rq b ic rv rw l is rx\" data-selectable-paragraph=\"\">COMET_WORKSPACE\nCOMET_PROJECT_NAME<\/span><\/pre>\n<p id=\"c4f2\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">In order to do that, follow the steps described <a class=\"af nh\" href=\"https:\/\/docs.gitlab.com\/ee\/ci\/variables\/#add-a-cicd-variable-to-an-instance\" target=\"_blank\" rel=\"noopener ugc nofollow\">here<\/a>.<\/p>\n<p id=\"9f3a\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">It is worth noting that the <code class=\"cw rn ro rp rq b\">register-model<\/code> job only runs when the Merge Request gets reviewed and approved, and this behavior is obtained by setting <code class=\"cw rn ro rp rq b\">only:main<\/code>at the end of the job.<\/p>\n<p id=\"9182\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">Finally, we decide to let a team member have final control of the deployment; therefore we define a manual job:<\/p>\n<figure class=\"mk ml mm mn mo mp mh mi paragraph-image\">\n<figure><img loading=\"lazy\" decoding=\"async\" class=\"bg mu mv c\" role=\"presentation\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:563\/0*uSpMrdOypPghyXTg\" alt=\"\" width=\"971\" height=\"241\"><\/figure><div class=\"mh mi sf\"><picture><source srcset=\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/0*uSpMrdOypPghyXTg 640w, https:\/\/miro.medium.com\/v2\/resize:fit:720\/0*uSpMrdOypPghyXTg 720w, https:\/\/miro.medium.com\/v2\/resize:fit:750\/0*uSpMrdOypPghyXTg 750w, https:\/\/miro.medium.com\/v2\/resize:fit:786\/0*uSpMrdOypPghyXTg 786w, https:\/\/miro.medium.com\/v2\/resize:fit:828\/0*uSpMrdOypPghyXTg 828w, https:\/\/miro.medium.com\/v2\/resize:fit:1100\/0*uSpMrdOypPghyXTg 1100w, https:\/\/miro.medium.com\/v2\/resize:fit:1126\/0*uSpMrdOypPghyXTg 1126w\" type=\"image\/webp\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 563px\"><source srcset=\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/0*uSpMrdOypPghyXTg 640w, https:\/\/miro.medium.com\/v2\/resize:fit:720\/0*uSpMrdOypPghyXTg 720w, https:\/\/miro.medium.com\/v2\/resize:fit:750\/0*uSpMrdOypPghyXTg 750w, https:\/\/miro.medium.com\/v2\/resize:fit:786\/0*uSpMrdOypPghyXTg 786w, https:\/\/miro.medium.com\/v2\/resize:fit:828\/0*uSpMrdOypPghyXTg 828w, https:\/\/miro.medium.com\/v2\/resize:fit:1100\/0*uSpMrdOypPghyXTg 1100w, https:\/\/miro.medium.com\/v2\/resize:fit:1126\/0*uSpMrdOypPghyXTg 1126w\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 563px\" data-testid=\"og\"><\/picture><\/div>\n<\/figure>\n<p id=\"fb0a\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">When triggered, this job will import the model from Comet\u2019s Model Registry and automatically create the necessary containers to build the user interface and deploy to a Kubernetes cluster.<\/p>\n<\/div>\n<\/div>\n<div class=\"mp\">\n<div class=\"ab ca\">\n<div class=\"qv qw qx qy qz ra ce rb cf rc ch bg\">\n<figure class=\"mk ml mm mn mo mp re rf paragraph-image\">\n<div class=\"mq mr ee ms bg mt\" tabindex=\"0\" role=\"button\">\n<figure><img loading=\"lazy\" decoding=\"async\" class=\"bg mu mv c\" role=\"presentation\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/0*Qty-KwZzVZ2E-NOl\" alt=\"\" width=\"1000\" height=\"162\"><\/figure><div class=\"mh mi rd\"><picture><source srcset=\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/0*Qty-KwZzVZ2E-NOl 640w, https:\/\/miro.medium.com\/v2\/resize:fit:720\/0*Qty-KwZzVZ2E-NOl 720w, https:\/\/miro.medium.com\/v2\/resize:fit:750\/0*Qty-KwZzVZ2E-NOl 750w, https:\/\/miro.medium.com\/v2\/resize:fit:786\/0*Qty-KwZzVZ2E-NOl 786w, https:\/\/miro.medium.com\/v2\/resize:fit:828\/0*Qty-KwZzVZ2E-NOl 828w, https:\/\/miro.medium.com\/v2\/resize:fit:1100\/0*Qty-KwZzVZ2E-NOl 1100w, https:\/\/miro.medium.com\/v2\/resize:fit:2000\/0*Qty-KwZzVZ2E-NOl 2000w\" type=\"image\/webp\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 1000px\"><source srcset=\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/0*Qty-KwZzVZ2E-NOl 640w, https:\/\/miro.medium.com\/v2\/resize:fit:720\/0*Qty-KwZzVZ2E-NOl 720w, https:\/\/miro.medium.com\/v2\/resize:fit:750\/0*Qty-KwZzVZ2E-NOl 750w, https:\/\/miro.medium.com\/v2\/resize:fit:786\/0*Qty-KwZzVZ2E-NOl 786w, https:\/\/miro.medium.com\/v2\/resize:fit:828\/0*Qty-KwZzVZ2E-NOl 828w, https:\/\/miro.medium.com\/v2\/resize:fit:1100\/0*Qty-KwZzVZ2E-NOl 1100w, https:\/\/miro.medium.com\/v2\/resize:fit:2000\/0*Qty-KwZzVZ2E-NOl 2000w\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 1000px\" data-testid=\"og\"><\/picture><\/div>\n<\/div>\n<\/figure>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"ab ca\">\n<div class=\"ch bg ew ex ey ez\">\n<p id=\"0e19\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">This job triggers a downstream pipeline, which means that the UI for this MNIST application resides in a different project. This keeps the codebase for the UI and model training separated but integrated and connected at the moment of deploying the model to a production environment<\/p>\n<figure class=\"mk ml mm mn mo mp mh mi paragraph-image\">\n<figure><img loading=\"lazy\" decoding=\"async\" class=\"bg mu mv c\" role=\"presentation\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:521\/0*EWwJqV3KslyFbf7p\" alt=\"\" width=\"521\" height=\"176\"><\/figure><div class=\"mh mi sg\"><picture><source srcset=\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/0*EWwJqV3KslyFbf7p 640w, https:\/\/miro.medium.com\/v2\/resize:fit:720\/0*EWwJqV3KslyFbf7p 720w, https:\/\/miro.medium.com\/v2\/resize:fit:750\/0*EWwJqV3KslyFbf7p 750w, https:\/\/miro.medium.com\/v2\/resize:fit:786\/0*EWwJqV3KslyFbf7p 786w, https:\/\/miro.medium.com\/v2\/resize:fit:828\/0*EWwJqV3KslyFbf7p 828w, https:\/\/miro.medium.com\/v2\/resize:fit:1100\/0*EWwJqV3KslyFbf7p 1100w, https:\/\/miro.medium.com\/v2\/resize:fit:1042\/0*EWwJqV3KslyFbf7p 1042w\" type=\"image\/webp\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 521px\"><source srcset=\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/0*EWwJqV3KslyFbf7p 640w, https:\/\/miro.medium.com\/v2\/resize:fit:720\/0*EWwJqV3KslyFbf7p 720w, https:\/\/miro.medium.com\/v2\/resize:fit:750\/0*EWwJqV3KslyFbf7p 750w, https:\/\/miro.medium.com\/v2\/resize:fit:786\/0*EWwJqV3KslyFbf7p 786w, https:\/\/miro.medium.com\/v2\/resize:fit:828\/0*EWwJqV3KslyFbf7p 828w, https:\/\/miro.medium.com\/v2\/resize:fit:1100\/0*EWwJqV3KslyFbf7p 1100w, https:\/\/miro.medium.com\/v2\/resize:fit:1042\/0*EWwJqV3KslyFbf7p 1042w\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 521px\" data-testid=\"og\"><\/picture><\/div>\n<\/figure>\n<h2 id=\"91fa\" class=\"qd nj fr be nk qe qf qg nn qh qi qj nq om qk ql qm oq qn qo qp ou qq qr qs qt bj\" data-selectable-paragraph=\"\">Final Application<\/h2>\n<p id=\"0f79\" class=\"pw-post-body-paragraph oe of fr be b gp og oh oi gs oj ok ol om on oo op oq or os ot ou ov ow ox ng fk bj\" data-selectable-paragraph=\"\">Let\u2019s take a quick look at the model that been deployed to our application.<br>\nOur application is a simple input box, where you can draw in a digit and have the model predict what it thinks the digit is.<\/p>\n<\/div>\n<\/div>\n<div class=\"mp\">\n<div class=\"ab ca\">\n<div class=\"qv qw qx qy qz ra ce rb cf rc ch bg\">\n<figure class=\"mk ml mm mn mo mp re rf paragraph-image\">\n<div class=\"mq mr ee ms bg mt\" tabindex=\"0\" role=\"button\">\n<figure><img loading=\"lazy\" decoding=\"async\" class=\"bg mu mv c\" role=\"presentation\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*kG4lxqaLRHEzJzvv3YmNOw.gif\" alt=\"\" width=\"1000\" height=\"563\"><\/figure><div class=\"mh mi sh\"><picture><source srcset=\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/1*kG4lxqaLRHEzJzvv3YmNOw.gif 640w, https:\/\/miro.medium.com\/v2\/resize:fit:720\/1*kG4lxqaLRHEzJzvv3YmNOw.gif 720w, https:\/\/miro.medium.com\/v2\/resize:fit:750\/1*kG4lxqaLRHEzJzvv3YmNOw.gif 750w, https:\/\/miro.medium.com\/v2\/resize:fit:786\/1*kG4lxqaLRHEzJzvv3YmNOw.gif 786w, https:\/\/miro.medium.com\/v2\/resize:fit:828\/1*kG4lxqaLRHEzJzvv3YmNOw.gif 828w, https:\/\/miro.medium.com\/v2\/resize:fit:1100\/1*kG4lxqaLRHEzJzvv3YmNOw.gif 1100w, https:\/\/miro.medium.com\/v2\/resize:fit:2000\/1*kG4lxqaLRHEzJzvv3YmNOw.gif 2000w\" type=\"image\/webp\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 1000px\"><source srcset=\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/1*kG4lxqaLRHEzJzvv3YmNOw.gif 640w, https:\/\/miro.medium.com\/v2\/resize:fit:720\/1*kG4lxqaLRHEzJzvv3YmNOw.gif 720w, https:\/\/miro.medium.com\/v2\/resize:fit:750\/1*kG4lxqaLRHEzJzvv3YmNOw.gif 750w, https:\/\/miro.medium.com\/v2\/resize:fit:786\/1*kG4lxqaLRHEzJzvv3YmNOw.gif 786w, https:\/\/miro.medium.com\/v2\/resize:fit:828\/1*kG4lxqaLRHEzJzvv3YmNOw.gif 828w, https:\/\/miro.medium.com\/v2\/resize:fit:1100\/1*kG4lxqaLRHEzJzvv3YmNOw.gif 1100w, https:\/\/miro.medium.com\/v2\/resize:fit:2000\/1*kG4lxqaLRHEzJzvv3YmNOw.gif 2000w\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 1000px\" data-testid=\"og\"><\/picture><\/div>\n<\/div>\n<figcaption class=\"rh ri rj mh mi rk rl be b bf z dw\" data-selectable-paragraph=\"\">Model Deployed in an Application<\/figcaption>\n<\/figure>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"ab ca\">\n<div class=\"ch bg ew ex ey ez\">\n<h1 id=\"5c8c\" class=\"ni nj fr be nk nl nm gr nn no np gu nq nr si nt nu nv sj nx ny nz sk ob oc od bj\" data-selectable-paragraph=\"\"><strong class=\"al\">Conclusion\/Takeaways<\/strong><\/h1>\n<p id=\"db38\" class=\"pw-post-body-paragraph oe of fr be b gp og oh oi gs oj ok ol om on oo op oq or os ot ou ov ow ox ng fk bj\" data-selectable-paragraph=\"\">In this post, we addressed some of the challenges faced by ML and Software Teams when it comes to collaborating on delivering ML powered applications. Some of these challenges include<\/p>\n<ul class=\"\">\n<li id=\"9041\" class=\"oe of fr be b gp oy oh oi gs oz ok ol om ry oo op oq rz os ot ou sa ow ox ng sb sc sd bj\" data-selectable-paragraph=\"\">The discrepancy in the frequency with which each of these teams need to iterate on their codebases and CI\/CD pipelines<\/li>\n<li id=\"9648\" class=\"oe of fr be b gp sl oh oi gs sm ok ol om sn oo op oq so os ot ou sp ow ox ng sb sc sd bj\" data-selectable-paragraph=\"\">The fact that only a single set of experiment assets from an ML experimentation pipeline is relevant to the application<\/li>\n<li id=\"92ff\" class=\"oe of fr be b gp sl oh oi gs sm ok ol om sn oo op oq so os ot ou sp ow ox ng sb sc sd bj\" data-selectable-paragraph=\"\">The challenge of syncing a model or other experiment assets across independent codebases.<\/li>\n<\/ul>\n<p id=\"f0d0\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">Using the GitLab DevOps platform and Comet, we can start bridging the gap between ML and Software Engineering teams over the course of a project.<\/p>\n<p id=\"468a\" class=\"pw-post-body-paragraph oe of fr be b gp oy oh oi gs oz ok ol om pa oo op oq pb os ot ou pc ow ox ng fk bj\" data-selectable-paragraph=\"\">By having model performance metrics adopted into software development workflows like the one we saw in the issue and Merge Request, we can keep track of the code changes, discussions, experiments, and models created in the process. All the operations executed by the team are recorded, can be audited, are end-to end-traceable, and (most importantly) reproducible.<\/p>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Editor\u2019s Note: This blog post was co-written alongside William Arias, Technical Marketing Manager at GitLab, and has been syndicated with permission. Check out the original version of this post on GitLab\u2019s blog. Introduction Building ML-powered applications comes with numerous challenges. When we talk about these challenges, there is a tendency to overly focus on problems [&hellip;]<\/p>\n","protected":false},"author":8,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[8,23,9],"tags":[],"coauthors":[128],"class_list":["post-8090","post","type-post","status-publish","format-standard","hentry","category-comet-community-hub","category-integrations","category-product"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v25.9 (Yoast SEO v25.9) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Streamline ML Model Development with GitLab\u2019s DevOps Platform and Comet - Comet<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/streamline-ml-model-development-with-gitlabs-devops-platform-and-comet\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Streamline ML Model Development with GitLab\u2019s DevOps Platform and Comet\" \/>\n<meta property=\"og:description\" content=\"Editor\u2019s Note: This blog post was co-written alongside William Arias, Technical Marketing Manager at GitLab, and has been syndicated with permission. Check out the original version of this post on GitLab\u2019s blog. Introduction Building ML-powered applications comes with numerous challenges. When we talk about these challenges, there is a tendency to overly focus on problems [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/blog\/streamline-ml-model-development-with-gitlabs-devops-platform-and-comet\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2023-11-02T18:16:15+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-04-24T17:04:44+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*F3DxwRnz8uUGcza2St0zCA.png\" \/>\n<meta name=\"author\" content=\"Dhruv Nair\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Dhruv Nair\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"10 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Streamline ML Model Development with GitLab\u2019s DevOps Platform and Comet - Comet","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/blog\/streamline-ml-model-development-with-gitlabs-devops-platform-and-comet","og_locale":"en_US","og_type":"article","og_title":"Streamline ML Model Development with GitLab\u2019s DevOps Platform and Comet","og_description":"Editor\u2019s Note: This blog post was co-written alongside William Arias, Technical Marketing Manager at GitLab, and has been syndicated with permission. Check out the original version of this post on GitLab\u2019s blog. Introduction Building ML-powered applications comes with numerous challenges. When we talk about these challenges, there is a tendency to overly focus on problems [&hellip;]","og_url":"https:\/\/www.comet.com\/site\/blog\/streamline-ml-model-development-with-gitlabs-devops-platform-and-comet","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2023-11-02T18:16:15+00:00","article_modified_time":"2025-04-24T17:04:44+00:00","og_image":[{"url":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*F3DxwRnz8uUGcza2St0zCA.png","type":"","width":"","height":""}],"author":"Dhruv Nair","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Dhruv Nair","Est. reading time":"10 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.comet.com\/site\/blog\/streamline-ml-model-development-with-gitlabs-devops-platform-and-comet#article","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/blog\/streamline-ml-model-development-with-gitlabs-devops-platform-and-comet\/"},"author":{"name":"Team Comet Digital","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/6266601170c60a7a82b3e0043fbe8ddf"},"headline":"Streamline ML Model Development with GitLab\u2019s DevOps Platform and Comet","datePublished":"2023-11-02T18:16:15+00:00","dateModified":"2025-04-24T17:04:44+00:00","mainEntityOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/streamline-ml-model-development-with-gitlabs-devops-platform-and-comet\/"},"wordCount":1684,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/streamline-ml-model-development-with-gitlabs-devops-platform-and-comet#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*F3DxwRnz8uUGcza2St0zCA.png","articleSection":["Comet Community Hub","Integrations","Product"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/blog\/streamline-ml-model-development-with-gitlabs-devops-platform-and-comet\/","url":"https:\/\/www.comet.com\/site\/blog\/streamline-ml-model-development-with-gitlabs-devops-platform-and-comet","name":"Streamline ML Model Development with GitLab\u2019s DevOps Platform and Comet - Comet","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/streamline-ml-model-development-with-gitlabs-devops-platform-and-comet#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/streamline-ml-model-development-with-gitlabs-devops-platform-and-comet#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*F3DxwRnz8uUGcza2St0zCA.png","datePublished":"2023-11-02T18:16:15+00:00","dateModified":"2025-04-24T17:04:44+00:00","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/blog\/streamline-ml-model-development-with-gitlabs-devops-platform-and-comet#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/blog\/streamline-ml-model-development-with-gitlabs-devops-platform-and-comet"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/blog\/streamline-ml-model-development-with-gitlabs-devops-platform-and-comet#primaryimage","url":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*F3DxwRnz8uUGcza2St0zCA.png","contentUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*F3DxwRnz8uUGcza2St0zCA.png"},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/blog\/streamline-ml-model-development-with-gitlabs-devops-platform-and-comet#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"Streamline ML Model Development with GitLab\u2019s DevOps Platform and Comet"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/6266601170c60a7a82b3e0043fbe8ddf","name":"Team Comet Digital","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/4f0c0a8cc7c0e87c636ff6a420a6647c","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/Screen-Shot-2023-08-12-at-8.58.50-AM-96x96.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/Screen-Shot-2023-08-12-at-8.58.50-AM-96x96.png","caption":"Team Comet Digital"},"sameAs":["https:\/\/www.comet.ml\/"],"url":"https:\/\/www.comet.com\/site\/blog\/author\/teamcometdigital\/"}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/8090","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=8090"}],"version-history":[{"count":1,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/8090\/revisions"}],"predecessor-version":[{"id":15464,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/8090\/revisions\/15464"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=8090"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=8090"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=8090"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=8090"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}