{"id":12869,"date":"2025-02-19T07:29:14","date_gmt":"2025-02-19T15:29:14","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=12869"},"modified":"2025-10-27T20:04:50","modified_gmt":"2025-10-27T20:04:50","slug":"a-simple-recipe-for-llm-observability","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/a-simple-recipe-for-llm-observability\/","title":{"rendered":"A Simple Recipe for LLM Observability"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

So, you\u2019re buil<\/span>ding an AI application on top of an LLM, and you\u2019re planning on setting it live in production. And thanks to the proliferation of frameworks and APIs, you\u2019re able to prototype, test, and deploy faster than ever. Before you go to production, however, you realize you still have one problem to solve: <\/span>How are you supposed to monitor this thing?<\/span><\/i><\/p>\n\n\n\n

One thing I\u2019ve learned over the course of 10+ years in production ML and AI, is the importance of monitoring these software systems. Despite our best efforts, models have a way of surprising us when they\u2019re deployed. Once an LLM is in production and in the hands of our users, we want to be proactively alerted to any disruptive behavior or bugs so we can preemptively resolve them before there is significant negative impact. Yet monitoring LLMs is complex, and although best practices can overlap and borrow from traditional software development, there are some fundamental differences.  <\/span><\/p>\n\n\n\n

Why Monitoring LLMs is Different<\/span><\/h2>\n\n\n\n

Production issues with an LLM look quite different from the issues that arise in traditional software. The LLM itself is now a component of the software architecture that needs to be monitored. These models are nondeterministic by nature, meaning that their output is probabilistically generated and there is natural variability expected in the results, resulting in strange behavior issues we\u2019ve never seen in software. Traditional software, for example, doesn\u2019t hallucinate or generate biased content. So LLMs have different failure points, and because of this, a different approach and set of tools is required to provide complete monitoring coverage to safeguard these models against their potential failure points. <\/span><\/p>\n\n\n\n

As LLMOps matures, we are discovering the exact monitoring paradigms we should implement before we flip the switch to live. <\/span>As developers, we want to keep tabs on the health of our apps and proactively monitor for potential issues. We\u2019d also like to visualize the behavior in a nice dashboard. That is actually really challenging with LLMs because of the nondeterministic nature of their output. For example, traditional software will throw an error when it fails\u2014at least, hopefully. Your LLM, on the other hand, isn\u2019t going to tell you if it\u2019s hallucinating. As a result, we have to be thoughtful about what we measure to assess the health of our system, and it can be very difficult to develop and generate the right set of numeric metrics to monitor the LLMs behavior online at scale.<\/span><\/p>\n\n\n\n

In this article, I will propose metric categories that must be considered before your LLM goes live, and provide a framework using OSS tools to configure a full coverage monitoring and debugging strategy. I\u2019ll be using <\/span>OpenAI\u2019s SDK<\/span><\/a> and <\/span>Opik<\/span><\/a> and providing easy copy paste coding samples so you can customize this monitoring strategy in your own apps. I was surprised by how quickly I was able to stand up production quality monitoring for a real LLM app using open source tools. This tutorial may look simple, but don\u2019t be fooled – a few lines of clean code can actually build a powerful and meaningful product! <\/span><\/p>\n\n\n\n

If you\u2019re looking to personalize a monitoring strategy for your own LLM app, this tutorial will give you some ideas about what you should be monitoring for, as well as provide a code example that can easily be customized to fit a monitoring strategy that is specific to your needs. And if you do use this monitoring strategy or code for your own projects, I\u2019d love to see what you built! Please tag me in your projects if you post about them on LI, or drop your Github in the comments down below. <\/span><\/p>\n\n\n\n

Monitoring Strategy for LLMs<\/span><\/h2>\n\n\n\n

So what are these unique failure points specific to live LLM models? What should we be worried about once our LLM project is live? I propose three categories of online metrics that should be considered for monitoring:<\/span> Cost, Quality, and Output.<\/b><\/p>\n\n\n\n

Cost Metrics<\/h2>\n\n\n\n

Running LLM inference can be expensive. Every call a customer makes to your app is likely one or multiple calls to an LLM model. Whether that model is hosted internally on your servers, or an API from an LLM provider like OpenAI, that call is going to cost you some $. Costs can escalate quickly, especially with high API usage, frequent model queries, and large inference workloads. So monitoring your cost metrics is crucial for maintaining ROI on your project. <\/span><\/p>\n\n\n\n

I typically monitor for the following cost metrics:<\/span><\/p>\n\n\n\n