{"id":17615,"date":"2025-09-26T18:36:44","date_gmt":"2025-09-26T18:36:44","guid":{"rendered":"https:\/\/www.comet.com\/site\/?page_id=17615"},"modified":"2026-04-08T16:39:43","modified_gmt":"2026-04-08T16:39:43","slug":"pattern-ai-content-analysis","status":"publish","type":"page","link":"https:\/\/www.comet.com\/site\/customers\/pattern-ai-content-analysis\/","title":{"rendered":"Pattern\u2019s Data-Driven Approach to LLM Evaluation with Opik"},"content":{"rendered":"\n

Pattern Group Inc.<\/a> (\u201cPattern\u201d) is a leader in accelerating brands on global ecommerce marketplaces, leveraging proprietary technology and AI. At the core of these capabilities is the AI Operations team, which is responsible for building the foundational AI technologies that support both technical and non-technical users. Their work enables employees across the company to integrate AI into a wide range of projects, from automating internal processes to enhancing customer-facing products and services.<\/p>\n\n\n\n

We spoke with Jeremy Mumford, Lead AI Engineer, and Cayden Blake, AI Engineer, from Pattern\u2019s AI Ops team to learn how they leveraged Opik, Comet\u2019s open-source LLM evaluation framework<\/a> to achieve measurable efficiency gains and identify opportunities for significant cost savings, enhancing an already valuable workflow that drives growth for their customers.<\/p>\n\n\n\n

Automating Content Analysis to Optimize Marketplace Listings<\/h2>\n\n\n\n

One of the AI Ops team\u2019s responsibilities is maintaining a Content Brief<\/a>, an AI-powered, data-fueled, content optimization tool for creating product content that drives growth across every marketplace. Content Brief offers insights into how to improve copy and images by allowing brands to make data-backed decisions that improve content since it resonates more strongly with target buyers.<\/p>\n\n\n\n

Automating the Content Brief system with AI is imperative; manual processes would be too slow and resource-intensive to keep up with the increasing demand that users have across multiple marketplaces. The team relied on large models to power its Content Brief system; however, the approach proved costly. The team\u2019s challenge was to reduce costs without compromising on performance or quality that their clients depended on, and that is where Opik came in.<\/p>\n\n\n\n

\n

\u201cThe workflow was quite costly with the models we were using. Our hope was to investigate this process and use Opik to uncover a more cost-effective model without sacrificing performance\u201d<\/p>\nCayden Blake, AI Engineer<\/cite><\/blockquote>\n\n\n\n

The Evaluation Process With Opik<\/h2>\n\n\n\n

The AI Ops team initiated the project by extracting production data from their Content Briefs and converting it to reusable datasets for LLM evaluation<\/a> within Opik. With those datasets, they defined a set of LLM evaluation metrics<\/a> tailored to the specific workflow, which included user-defined metrics and custom LLM-as-a-judge<\/a> metrics to assess the outputs of various models and determine their performance.<\/p>\n\n\n\n

\"Screenshot
Opik’s Experiment Dashboard<\/figcaption><\/figure>\n\n\n\n

Using Opik\u2019s experiment dashboard<\/a>, they ran side-by-side comparisons of multiple models, analyzing both performance scores and cost data<\/a> in one centralized view. The team also leveraged the Prompt Playground<\/a> to test prompt variations before applying them to the full data set. To validate the results, they used light prompt tuning and a layer of human review to confirm the experiment’s findings.<\/p>\n\n\n\n

The results gave the team clear evidence of tradeoffs: some models delivered strong results, but at a high cost, while others were inexpensive but underperformed. By running structured evaluations in Opik, they identified the \u201csweet spot,\u201d using a smaller, more efficient model that delivered the same quality as their baseline approach. The average cost savings per Content Brief is expected to reduce aggregate annual costs by approximately $60,000.<\/p>\n\n\n\n

\n

\u201cThe biggest advantage of using Opik was achieving cost savings while maintaining quality. Benchmark evaluations gave us a scientific way to confirm that a smaller, more efficient model produced consistent results across a wide range of data.\u201d <\/p>\nJeremy Mumford, Lead AI Engineer<\/cite><\/blockquote>\n\n\n\n

Looking Ahead: Opik as a Core Part of AI Ops<\/h2>\n\n\n\n
\n

\u201cIt was very satisfying to feel more organized with my AI workflows, just being able to see everything in one place. It made me feel like I actually knew what was going on, instead of just hoping it worked in the background.\u201d<\/p>\nCayden Blake, AI Engineer<\/cite><\/blockquote>\n\n\n\n

Opik has provided the AI Ops team with a structured approach to evaluating AI systems, replacing trial and error with a centralized platform for running imperative testing before any system reaches production. This approach has resulted in strengthened individual workflows and created a more disciplined method for scaling AI across the company.<\/p>\n\n\n\n

\n

\u201cWe are making use of nearly every feature Opik has available today. We use tracing<\/a> features and the prompt library<\/a> for version control, the evaluation suite<\/a> to build benchmarks with both custom and out-of-the-box metrics, and online evaluation<\/a> to validate models in real time. We’re using the full spectrum of features and have exciting ideas for what to do next.\u201d <\/p>\nJeremy Mumford, Lead AI Engineer<\/cite><\/blockquote>\n\n\n\n

Looking ahead, they plan to integrate Opik into their CI\/CD pipelines so that every new or updated workflow undergoes standardized evaluation before release, establishing Opik as a standard checkpoint for engineers across the company.<\/p>\n\n\n\n

\n

\u201cOpik being open-source was one of the reasons we chose it. Beyond the peace of mind of knowing we can self-host if we want, the ability to debug and submit product requests when we notice things has been really helpful in making sure the product meets our needs.\u201d<\/p>\nJeremy Mumford, Lead AI Engineer<\/cite><\/blockquote>\n\n\n\n

For Pattern, Opik has evolved from a model evaluation tool into a hub for LLM observability<\/a>, enabling the company to innovate faster while maintaining confidence and control. If you are interested in learning more about how Opik can accelerate your AI workflows within Ecommerce, click the button below to learn more.<\/p>\n\n\n\n

\n
Contact Us<\/a><\/div>\n<\/div>\n\n\n\n
\n\n\n\n

Pattern accelerates brands on global ecommerce marketplaces, leveraging proprietary technology and AI. Utilizing more than 46 trillion data points, sophisticated machine learning and AI models, Pattern optimizes and automates all levers of ecommerce growth for global brands, including advertising, content management, logistics and fulfillment, pricing, forecasting, and customer service. Hundreds of global brands depend on Pattern’s ecommerce acceleration platform every day to drive profitable revenue growth across 60+ global marketplaces\u2014including Amazon, Walmart.com, Target.com, eBay, Tmall, TikTok Shop, JD, and Mercado Libre. To learn more, visit pattern.com or email press@pattern.com.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

Pattern Group Inc. (\u201cPattern\u201d) is a leader in accelerating brands on global ecommerce marketplaces, leveraging proprietary technology and AI. At the core of these capabilities is the AI Operations team, which is responsible for building the foundational AI technologies that support both technical and non-technical users. Their work enables employees across the company to integrate […]<\/p>\n","protected":false},"author":140,"featured_media":18120,"parent":488,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"case-study","meta":{"customer_name":"Pattern","customer_description":"Pattern provides ecommerce solutions to accelerate brands on marketplaces.","customer_industry":"Technology - Ecommerce","customer_technologies":"Opik","customer_logo":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/10\/pattern-logo-white.svg","footnotes":""},"coauthors":[127],"class_list":["post-17615","page","type-page","status-publish","has-post-thumbnail","hentry"],"yoast_head":"\nPattern\u2019s Data-Driven Approach to LLM Evaluation with Opik<\/title>\n<meta name=\"description\" content=\"Learn how Pattern leveraged Opik, to optimize AI-powered content analysis, cut costs, and build a scalable, production-ready eval workflow.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/customers\/pattern-ai-content-analysis\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Pattern\u2019s Data-Driven Approach to LLM Evaluation with Opik\" \/>\n<meta property=\"og:description\" content=\"Learn how Pattern leveraged Opik, to optimize AI-powered content analysis, cut costs, and build a scalable, production-ready eval workflow.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/customers\/pattern-ai-content-analysis\/\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:modified_time\" content=\"2026-04-08T16:39:43+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/10\/Case-study-Pattern.webp\" \/>\n\t<meta property=\"og:image:width\" content=\"1842\" \/>\n\t<meta property=\"og:image:height\" content=\"650\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/webp\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"4 minutes\" \/>\n\t<meta name=\"twitter:label2\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data2\" content=\"Caroline Borders\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Pattern\u2019s Data-Driven Approach to LLM Evaluation with Opik","description":"Learn how Pattern leveraged Opik, to optimize AI-powered content analysis, cut costs, and build a scalable, production-ready eval workflow.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/customers\/pattern-ai-content-analysis\/","og_locale":"en_US","og_type":"article","og_title":"Pattern\u2019s Data-Driven Approach to LLM Evaluation with Opik","og_description":"Learn how Pattern leveraged Opik, to optimize AI-powered content analysis, cut costs, and build a scalable, production-ready eval workflow.","og_url":"https:\/\/www.comet.com\/site\/customers\/pattern-ai-content-analysis\/","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_modified_time":"2026-04-08T16:39:43+00:00","og_image":[{"width":1842,"height":650,"url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/10\/Case-study-Pattern.webp","type":"image\/webp"}],"twitter_card":"summary_large_image","twitter_site":"@Cometml","twitter_misc":{"Est. reading time":"4 minutes","Written by":"Caroline Borders"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/customers\/pattern-ai-content-analysis\/","url":"https:\/\/www.comet.com\/site\/customers\/pattern-ai-content-analysis\/","name":"Pattern\u2019s Data-Driven Approach to LLM Evaluation with Opik","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/customers\/pattern-ai-content-analysis\/#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/customers\/pattern-ai-content-analysis\/#primaryimage"},"thumbnailUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/10\/Case-study-Pattern.webp","datePublished":"2025-09-26T18:36:44+00:00","dateModified":"2026-04-08T16:39:43+00:00","description":"Learn how Pattern leveraged Opik, to optimize AI-powered content analysis, cut costs, and build a scalable, production-ready eval workflow.","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/customers\/pattern-ai-content-analysis\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/customers\/pattern-ai-content-analysis\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/customers\/pattern-ai-content-analysis\/#primaryimage","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/10\/Case-study-Pattern.webp","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/10\/Case-study-Pattern.webp","width":1842,"height":650,"caption":"Opik x Pattern"},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/customers\/pattern-ai-content-analysis\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"Customers","item":"https:\/\/www.comet.com\/site\/customers\/"},{"@type":"ListItem","position":3,"name":"Pattern\u2019s Data-Driven Approach to LLM Evaluation with Opik"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]}]}},"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/pages\/17615","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/140"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=17615"}],"version-history":[{"count":2,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/pages\/17615\/revisions"}],"predecessor-version":[{"id":19483,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/pages\/17615\/revisions\/19483"}],"up":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/pages\/488"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media\/18120"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=17615"}],"wp:term":[{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=17615"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}