{"id":20285,"date":"2026-07-02T22:21:56","date_gmt":"2026-07-02T22:21:56","guid":{"rendered":"https:\/\/www.comet.com\/site\/?p=20285"},"modified":"2026-07-02T22:21:57","modified_gmt":"2026-07-02T22:21:57","slug":"edd-opik-project-example","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/edd-opik-project-example\/","title":{"rendered":"How Evaluation-Driven Development (EDD) Works"},"content":{"rendered":"\n

Turn every AI agent change into a measured experiment you compare before and after to detect regressions and measure performance.<\/em><\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

The scariest AI failures are the silent ones.<\/p>\n\n\n\n

You ship a change to your agent on a branch \u2014 a new feature, a prompt fix, a quick refactor. No errors. No complaints. Everything looks<\/em> fine. But does it still work, or did you quietly break something that worked yesterday?<\/p>\n\n\n\n

As Alejandro Aboy puts it: \u201cthe fact that they\u2019re not complaining doesn\u2019t mean there\u2019s no issue going on.\u201d<\/em> A quiet user is not a happy user. Usually, it\u2019s the opposite.<\/p>\n\n\n\n

\n

The more Alejandro and I talked about AI evals and EDD, the more his struggles sounded like mine. A story from builders to builder.<\/p>\n<\/blockquote>\n\n\n\n

    \n
  1. You can break what already worked.<\/strong> Change a prompt, refactor a tool, and an old feature quietly regresses. Alejandro lived it: cleaning noisy instructions out of his agent\u2019s system prompt made it start fabricating IDs it used to get right. You only catch that by running the same tests before and after the change, and comparing.<\/li>\n\n\n\n
  2. The feature is brand new, so you have nothing to test it on<\/strong>. No dataset, no historical traces, no ground truth. Yet you still need to know whether it works, and how well. So how do you generate realistic test data fast, then feed it to evaluators that turn it into hard performance numbers?<\/li>\n<\/ol>\n\n\n\n

    This is the case study that gives you a plan of attack for both: Evaluation-Driven Development (EDD)<\/strong><\/p>\n\n\n\n

    How to prove a new feature works, and didn\u2019t regress, before you merge. It comes from a recent conversation with Alejandro Aboy<\/a>, a senior data and AI engineer at Workpath who owns the entire data stack, built the Workpath AI Companion, and writes The Pipe and The Line Substack<\/em>.<\/p>\n\n\n\n

    And we won\u2019t keep it abstract. Every example comes from one real product: Workpath<\/strong>, a strategy-execution SaaS that keeps large companies\u2019 OKRs and initiatives aligned. (OKRs \u2014 Objectives and Key Results \u2014 are the goal-setting framework teams use to name what they want to achieve and the measurable results that prove they\u2019re getting there.) Its AI-native feature is the Workpath AI Companion<\/strong>: an agent that scans a company\u2019s strategy and OKR data end-to-end to keep enterprise teams aligned. It\u2019s the exact system Alejandro runs EDD on every day.<\/p>\n\n\n\n

    So when we say developing a new feature, picture shipping a change to that Companion and proving, before you merge, that it works and didn\u2019t regress.<\/p>\n\n\n\n

    \"\"
    The moving parts<\/figcaption><\/figure>\n\n\n\n

    The Develop-a-Feature Workflow<\/h2>\n\n\n\n

    Imagine. You start working on a new feature, branch, and develop the change. But before you merge, you have to answer 2 questions:<\/p>\n\n\n\n

      \n
    1. What\u2019s the performance of my new feature?<\/li>\n\n\n\n
    2. Did my change introduce any regressions into the existing codebase?<\/li>\n<\/ol>\n\n\n\n

      Only when both look good you accept the pull request. EDD helps you answer those 2 questions.<\/p>\n\n\n\n

      \"\"
      EDD is the offline validation gate between developing a change and merging it<\/figcaption><\/figure>\n\n\n\n

      Every feature is hypothesis-first. As Alejandro frames it, \u201cI have a hypothesis\u2026 <\/em>and everything should lie around that.\u201d Every change starts as a stated hypothesis on a branch.<\/p>\n\n\n\n

      Based on that hypothesis, EDD runs a simulation and scores the results to answer the 2 questions.<\/p>\n\n\n\n

      Every feature ends in a PR, backed by an experiment, with clear traces and metrics. Framing the eval results as an experiment allows you to compare current results to previous ones, detecting regressions or tracking improvements.<\/p>\n\n\n\n

      This is how you can compare two experiments in Opik:<\/p>\n\n\n\n

      \"\"
      Comparing two experiments in Opik<\/figcaption><\/figure>\n\n\n\n

      What about the process that happens between starting a new feature and its experiment?<\/p>\n\n\n\n

      From an architectural perspective, we have:<\/h3>\n\n\n\n