{"id":9313,"date":"2024-02-26T14:35:47","date_gmt":"2024-02-26T22:35:47","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=9313"},"modified":"2025-04-24T17:03:09","modified_gmt":"2025-04-24T17:03:09","slug":"how-comet-achieved-zero-downtime","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/how-comet-achieved-zero-downtime\/","title":{"rendered":"How Comet Achieved Zero Downtime"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

Introduction<\/strong><\/h2>\n\n\n\n

In an era where developers and engineers are constantly evaluating and adopting cloud tools, one of the most important goals for any SaaS engineering team is to minimize production downtime<\/strong>.<\/p>\n\n\n\n

Comet is a tool that helps Data Scientists track all the relevant information for their model training runs (including code, performance metrics, hyper-parameters, and cpu\/data drift) and then monitor those models performance in production. Comet Cloud has over 100,000 end users and powers some of the most advanced ML teams at companies like Etsy, Assembly AI, and Affirm. Any downtime on Comet Cloud can have significant negative impacts:<\/p>\n\n\n\n

1. Disrupted Workflows: <\/strong>Many Data Scientists rely on Comet to inform them on their model training run progress. If the current training run is performing worse than the previous ones, Data Scientists often stop training runs early to save costs.<\/p>\n\n\n\n

2. Loss of Productivity: <\/strong>Comet improves ML team efficiency by 30%. Teams use Comet to determine which models are the latest and greatest. They then use Webhooks to update their downstream production systems.<\/p>\n\n\n\n

3. Delayed Failure Detection<\/strong> Comet Cloud serves as an observability tool for Machine Learning Models. Today these models are making important decisions in self-driving cars, recommendations systems, and financial services. A drifted financial model can have significant monetary impact for a business and therefore must be caught immediately.<\/p>\n\n\n\n

In 2022, Comet Cloud encountered several instances of service interruptions. For 2023, the Comet\u2019s Engineering and set an aggressive goal to maintain 99.97% uptime.<\/p>\n\n\n\n

Root Cause Analysis<\/h2>\n\n\n\n

The first step the Engineering team took was to examine the reasons why production downtime incidents were occuring. Taking a look back at their 2022 reports, they grouped the incidents into 3 separate categories<\/p>\n\n\n\n

    \n
  1. Incidents caused by the Infrastructure<\/li>\n\n\n\n
  2. Incidents caused by the Application<\/li>\n\n\n\n
  3. Incidents caused by Human Error<\/li>\n<\/ol>\n\n\n\n

    To achieve their goals, the engineering team had to leave no stone unturned and make sure they addressed all 3 facets of their environment.<\/p>\n\n\n\n

    Infrastructure Migration from EC2 to Kubernetes<\/h2>\n\n\n\n

    Comet\u2019s Cloud Infrastructure was initially built upon EC2. However, the team needed something that was more scalable, efficient, and reliable. They decided to transition their infrastructure to Kubernetes. Kubernetes facilitates seamless updates, load balancing, and a declarative configuration approach; fostering a dynamic and resilient cloud infrastructure.<\/p>\n\n\n\n

    The migration didn\u2019t happen overnight. The team spent the entire Q1 quarter 2023 meticulously planning and executing the transition. They used a phase approach of sending half of their data traffic to their Kubernetes to thoroughly stress test the system and iron out any kinks before making the complete switch.<\/p>\n\n\n\n

    In the initial phase, traffic splitting involved the addition of a dedicated HTTP header named \u201cRouting-Traffic\u201d. This header was incorporated into all HTTP requests coming in from our clients (Comet-ML SDK<\/a> or Comet-UI<\/a>). When the header\u2019s value was set to \u201cEKS\u201d, the AWS ALB Ingress Controller recognized it and directed the traffic to the Kubernetes instances, otherwise, EC2 instances were utilized. As the migration progressed, the team later implemented a 50\/50 random traffic split in subsequent stages.<\/p>\n\n\n\n

    To achieve a 50\/50 random allocation between a Kubernetes pod and EC2 instances, the team utilized header-based routing and random weighted traffic switching between targets while simultaneously monitoring the system to identify errors.<\/p>\n\n\n\n