{"id":13159,"date":"2025-03-13T18:40:19","date_gmt":"2025-03-13T18:40:19","guid":{"rendered":"https:\/\/comet-marketing-site.lndo.site\/?page_id=13159"},"modified":"2025-11-03T20:25:49","modified_gmt":"2025-11-03T20:25:49","slug":"uber","status":"publish","type":"page","link":"https:\/\/www.comet.com\/site\/customers\/uber\/","title":{"rendered":"How Scientists at Uber Use Comet to Manage AI Experiments"},"content":{"rendered":"\n<p>Scale is an interesting, often over-simplified challenge in machine learning. Intuitively, most everyone understands that bigger models require large amounts of resources (large datasets, computational firepower, etc.), but cost is just one piece of ML\u2019s scale problem. For example, the&nbsp;<em>complexity&nbsp;<\/em>of managing offline ML experiments at scale is a particularly thorny problem.<\/p>\n\n\n\n<p>In contrast with online A\/B tests, offline ML experiments are attempts to improve the accuracy or performance of an ML model by experimenting with changes to the model offline.<\/p>\n\n\n\n<p>At Uber, where ML is fundamental to most products, a mechanism to manage offline experiments easily is needed to improve developer velocity.&nbsp; To solve for this, Uber AI was looking for a solution that will potentially complement and extend its in-house experiment management and collaboration capabilities.<\/p>\n\n\n\n<figure class=\"wp-block-video\"><video height=\"1080\" style=\"aspect-ratio: 1920 \/ 1080;\" width=\"1920\" controls src=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/06\/Uber-Recording-4-1080p-2X-1.mp4\"><\/video><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">What is hard about managing experiments at Uber-scale?<\/h2>\n\n\n\n<p>Take a ML-powered feature of Uber, say estimating the ETA for every ride and delivery. Imagine every market where Uber is active, and then every product offering within each market. The sheer quantity and diversity of product and market segments present logistical challenges right out of the gate. As Olcay Cirit, research scientist at Uber AI, says, \u201cWe can\u2019t simply look at a global accuracy metric, rather we must use&nbsp;<em>sliced analysis<\/em>&nbsp;to gauge the improvements or degradations in each market and product category.\u201d<\/p>\n\n\n\n<p>This results in a huge volume of experiments, conducted by different teams with different processes, all of which must be compared in order to make optimal improvements to the model. Performing deep analysis on model results where each model might behave differently across segments is extremely difficult.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Building experiment management infrastructure at Uber<\/h2>\n\n\n\n<p>Comet supports most of the ML experiment tracking workload out-of-the-box. As Cirit explains, \u201cAs my team piloted Comet, it organized my team\u2019s deep learning experiments by tracking all of our hyperparameters, metrics, and code changes\u2026 Developers working on improvements to ML infrastructure can more easily gauge, for example, whether improvements in training speed might adversely impact model convergence.\u201d<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p>\u201cWe found that Comet had these critical customizability features, including code panels, and were also quite willing to engage with us during our evaluation.\u201d<\/p>\n<\/blockquote>\n\n\n\n<p>For Uber, customizability and extensibility are critical concerns, as Uber AI has many different teams working on state-of-the-art models in different domains. As Cirit explains:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>\u201cWe are working on integrating Comet into our in-house framework so that all users of the platform can benefit from experiment tracking\u2026 With Comet, it will be very easy for product team members to add new, project-specific metrics to track and visualize without having to make changes to the platform\u2026\u201d<\/li>\n\n\n\n<li>\u00a0\u201cUber has its own internally developed Bayesian Optimization package that we use to tune hyperparameters. Comet lets us visualize all of the trials using panels and parallel coordinate charts, so that we can derive high-level insights about the hyperparameter search space in addition to getting a tuned set of hyperparameters.\u201d<\/li>\n<\/ul>\n\n\n\n<p>What you need, at that stage, are tools with which you can build ML infrastructure suited to your specific needs.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Scale is an interesting, often over-simplified challenge in machine learning. Intuitively, most everyone understands that bigger models require large amounts of resources (large datasets, computational firepower, etc.), but cost is just one piece of ML\u2019s scale problem. For example, the&nbsp;complexity&nbsp;of managing offline ML experiments at scale is a particularly thorny problem. In contrast with online [&hellip;]<\/p>\n","protected":false},"author":140,"featured_media":18117,"parent":488,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"case-study","meta":{"customer_name":"Uber","customer_description":"Mobility-as-a-service provider","customer_industry":"Transportation - Rideshare & Delivery","customer_technologies":"Custom-built MLOps platform","customer_logo":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/10\/uber-logo-small-pad-white.svg","footnotes":""},"coauthors":[127],"class_list":["post-13159","page","type-page","status-publish","has-post-thumbnail","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v25.9 (Yoast SEO v25.9) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>How Scientists at Uber Use Comet to Manage ML Experiments<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/customers\/uber\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How Scientists at Uber Use Comet to Manage AI Experiments\" \/>\n<meta property=\"og:description\" content=\"Scale is an interesting, often over-simplified challenge in machine learning. Intuitively, most everyone understands that bigger models require large amounts of resources (large datasets, computational firepower, etc.), but cost is just one piece of ML\u2019s scale problem. For example, the&nbsp;complexity&nbsp;of managing offline ML experiments at scale is a particularly thorny problem. In contrast with online [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/customers\/uber\/\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:modified_time\" content=\"2025-11-03T20:25:49+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/10\/Case-study-Uber.webp\" \/>\n\t<meta property=\"og:image:width\" content=\"1842\" \/>\n\t<meta property=\"og:image:height\" content=\"650\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/webp\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n\t<meta name=\"twitter:label2\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data2\" content=\"Caroline Borders\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"How Scientists at Uber Use Comet to Manage ML Experiments","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/customers\/uber\/","og_locale":"en_US","og_type":"article","og_title":"How Scientists at Uber Use Comet to Manage AI Experiments","og_description":"Scale is an interesting, often over-simplified challenge in machine learning. Intuitively, most everyone understands that bigger models require large amounts of resources (large datasets, computational firepower, etc.), but cost is just one piece of ML\u2019s scale problem. For example, the&nbsp;complexity&nbsp;of managing offline ML experiments at scale is a particularly thorny problem. In contrast with online [&hellip;]","og_url":"https:\/\/www.comet.com\/site\/customers\/uber\/","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_modified_time":"2025-11-03T20:25:49+00:00","og_image":[{"width":1842,"height":650,"url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/10\/Case-study-Uber.webp","type":"image\/webp"}],"twitter_card":"summary_large_image","twitter_site":"@Cometml","twitter_misc":{"Est. reading time":"3 minutes","Written by":"Caroline Borders"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/customers\/uber\/","url":"https:\/\/www.comet.com\/site\/customers\/uber\/","name":"How Scientists at Uber Use Comet to Manage ML Experiments","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/customers\/uber\/#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/customers\/uber\/#primaryimage"},"thumbnailUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/10\/Case-study-Uber.webp","datePublished":"2025-03-13T18:40:19+00:00","dateModified":"2025-11-03T20:25:49+00:00","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/customers\/uber\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/customers\/uber\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/customers\/uber\/#primaryimage","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/10\/Case-study-Uber.webp","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/10\/Case-study-Uber.webp","width":1842,"height":650,"caption":"Comet x Uber"},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/customers\/uber\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"Customers","item":"https:\/\/www.comet.com\/site\/customers\/"},{"@type":"ListItem","position":3,"name":"How Scientists at Uber Use Comet to Manage AI Experiments"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]}]}},"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/pages\/13159","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/140"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=13159"}],"version-history":[{"count":2,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/pages\/13159\/revisions"}],"predecessor-version":[{"id":18149,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/pages\/13159\/revisions\/18149"}],"up":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/pages\/488"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media\/18117"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=13159"}],"wp:term":[{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=13159"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}