{"id":20370,"date":"2026-07-20T19:41:26","date_gmt":"2026-07-20T19:41:26","guid":{"rendered":"https:\/\/www.comet.com\/site\/?p=20370"},"modified":"2026-07-22T15:15:59","modified_gmt":"2026-07-22T15:15:59","slug":"debugging-ai-agents","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/debugging-ai-agents\/","title":{"rendered":"Beyond the Single Trace: How We Built Agent Diagnostics for Opik"},"content":{"rendered":"\n

If you run an AI agent in production, you already know the drill. Something goes wrong, a user complains, or you just get a nagging feeling the agent isn’t behaving quite right. So you open Opik, your AI observability and agent tracing<\/a> tool, find the trace, and read through it until you spot the problem.<\/p>\n\n\n\n

That works for debugging one trace. It doesn’t work for the thousands your agent produces in a day.<\/p>\n\n\n\n

We wanted to fix that. Here’s how we built Diagnostics, our automated agent debugging tool, for Opik \u2014 and the two false starts it took to get there.<\/p>\n\n\n\n

\"two<\/figure>\n\n\n\n

The Problem With Reading Traces One at a Time<\/h2>\n\n\n\n

Talk to any team running agents in production and you’ll hear a version of the same workflow:<\/p>\n\n\n\n

    \n
  1. A developer, QA engineer, or end user notices something is off.<\/li>\n\n\n\n
  2. They go into Opik, find the trace, and download it.<\/li>\n\n\n\n
  3. They paste the trace JSON into Claude Code (or whatever LLM they have handy) to figure out what went wrong.<\/li>\n<\/ol>\n\n\n\n

    This works, but it’s entirely manual, and it only surfaces the one issue someone happened to trip over. The more disciplined teams also run periodic “review sessions,” where someone reads through a random sample of traces looking for patterns nobody’s reported yet. It’s a good instinct, but reading traces one at a time to spot patterns is like trying to understand a city’s traffic by watching a single car \u2014 it doesn’t scale past a handful of traces a week, and it’ll never tell you that the same tool call has been failing for a missing parameter all week, or that a prompt tweak quietly doubled how often the agent retries itself.<\/strong><\/p>\n\n\n\n

    Worse, the failures that matter most rarely throw an exception in the first place.<\/strong> An agent that quietly retries a failing tool three times with no strategy change, or burns ten short reasoning turns without making progress, doesn’t crash \u2014 it just produces a worse, slower, or more expensive answer. That kind of silent failure is invisible from a stack trace, and fixing it might mean touching a prompt, a tool definition, the agent’s code, or something further upstream like a RAG<\/a> pipeline feeding it bad context. None of that shows up unless you go looking for it.<\/p>\n\n\n\n

    We wanted an agent that goes looking for it \u2014 reading the traces so you don’t have to, and coming back with something more useful than “here’s a trace that looks weird.”<\/p>\n\n\n\n

    Agent Debugging Tools We Already Had to Work With<\/h2>\n\n\n\n

    We didn’t start from zero. Ollie, our in-house agent, already knew how to explore Opik’s traces and spans and reason about debugging them. We also had a daily report feature that would summarize a project’s overall activity \u2014 patterns, anomalies, and headline metrics for the day.<\/p>\n\n\n\n

    The daily report told you what happened<\/em>. What we needed next was something that could tell you why<\/em>, and what to do about it.<\/p>\n\n\n\n

    First Attempt: Point Ollie at the Outliers<\/h2>\n\n\n\n

    Our first idea followed naturally from the daily report: a fire-and-forget agent that would dig into causes rather than just describe symptoms.<\/p>\n\n\n\n

    We already had an endpoint that computed per-project statistics \u2014 run time, error rate, tool call counts, trace size \u2014 so we used that as a compass. The agent would pull the previous day’s statistics, flag traces that looked suspicious relative to the norm (too slow, too fast, too expensive, too cheap), and dig into just those. We built a dedicated TraceInvestigationAgent<\/code> sub-agent whose only job was to go deep on one flagged trace, so the parent agent’s context stayed lean instead of filling up with full trace bodies.<\/p>\n\n\n\n

    We tested it on the most available dataset we had: our own usage of Ollie. Which meant Ollie was investigating Ollie’s own mistakes \u2014 a genuinely fun bit of dogfooding, and every issue it found doubled as a bug report for this very project.<\/p>\n\n\n\n

    But this approach hit a wall we couldn’t tool our way out of: we had no way to confirm findings at scale.<\/strong> If the agent flagged an issue in one trace, we had no idea whether it was a one-off fluke or something affecting every fifth request. And answering that question the obvious way \u2014 having the agent read through a large sample of yesterday’s traces one by one \u2014 was both too slow and too expensive to ever ship. We eventually ripped out the dedicated stats tool and the standalone trace-investigation agent entirely; neither survived the rework.<\/p>\n\n\n\n

    The Rework: From Reading Traces to Querying Them<\/h2>\n\n\n\n

    Our first instinct for the rework was to split the agent debugging work across two models: an expensive one to generate theories, and a cheaper one to confirm them across the full dataset. We tried this locally with GLM-4.7-Flash in the “confirm at scale” role. It still wasn’t fast or cheap enough. The users who’d benefit most from Diagnostics are exactly the ones with the highest<\/em> trace volume, and no matter how inexpensive the model, reading every trace one at a time doesn’t scale to that.<\/p>\n\n\n\n

    We also looked at pre-computing a summary for every trace at ingestion time, so confirmation would just be a lookup instead of a fresh read. That ran into the same wall from the other direction: instead of paying the cost of a cheap model once, per suspicious trace, we’d be paying it on every single trace that came in, whether anyone ever asked about it or not. At real trace volumes, that’s not a discount \u2014 it’s a much bigger bill for something most of it would never be used for.<\/p>\n\n\n\n

    \n

    The real breakthrough was simpler than any of that:<\/p>\n\n\n\n

    <\/p>\n\n\n\n

    Stop asking the agent to read the data \u2014 let it query the data instead.<\/p>\n<\/blockquote>\n\n\n\n

    Every trace and span we store lives in ClickHouse, a column-oriented database built for exactly this kind of aggregate analysis. If the agent could query it, it wouldn’t need to read a single trace to answer “how often does this actually happen?” \u2014 it could just ask.<\/p>\n\n\n\n

    A quick local prototype confirmed the idea immediately. Not only did the findings improve, we could see the difference in kind: some issues we’d previously flagged turned out to barely matter once we could measure their real prevalence, while others we’d have never surfaced through spot-checking turned out to be widespread.<\/p>\n\n\n\n

    A Query Specialist, Not a Raw Database Connection<\/h3>\n\n\n\n

    We didn’t hand the main agent a database connection and walk away. For production, we built a dedicated ClickHouse sub-agent \u2014 one that knows our schema, our SQL dialect quirks, and the query patterns that are efficient against our tables \u2014 and gave it a single tool: clickhouse_select<\/code>.<\/p>\n\n\n\n

    The main agent gives that sub-agent a plain-language question \u2014 “how often does a tool call fail due to a missing parameter in this window?” \u2014 and the sub-agent turns it into SQL. If a query errors or comes back empty, it reads the error, adjusts, and retries on its own, within a small turn budget. Only once it’s satisfied does it hand a tight, aggregated answer back to the main agent: a sentence describing what the numbers are, the data itself, and a caveat only when one materially changes how to read the result. No speculation, no “you might also want to check” \u2014 the sub-agent’s job ends when the data is on the page, and the main agent owns all the interpretation.<\/p>\n\n\n\n

    That division of labor keeps the main agent’s context focused on reasoning about findings<\/em> instead of query mechanics, and it means query quality doesn’t depend on the orchestrating agent’s own SQL skills.<\/p>\n\n\n\n

    Making Direct Database Access Safe<\/h3>\n\n\n\n

    Giving an LLM-driven agent any path to a production database is worth being paranoid about, and we treated it that way. The clickhouse_select tool doesn’t talk to ClickHouse directly at all \u2014 it calls a purpose-built, read-only endpoint on the Opik backend, which is where the real enforcement lives:<\/p>\n\n\n\n