Optimizer benchmarks
We regularly evaluate every optimizer against shared datasets so you can make informed trade-offs. This page summarizes the latest results and explains how to reproduce them with the public benchmark scripts.
Datasets & metrics
Each run uses Opik datasets backed by open-source corpuses commonly used in academia:
Results shown below use gpt-4o-mini for evaluation on non multi-hop based runs. Scores will change if you select different models, metrics, agent configurations or prompt seeds.
Latest results
These are directional numbers. Some optimizers use more LLM/tool calls per trial than others (e.g., the HRPO Hierarchical Reflective Prompt Optimizer batches multiple analyses), so cost and runtime are not apples-to-apples even when the trial budget matches. Re-run the suite with your own datasets, models, and cost constraints before committing to a single optimizer.
Run benchmarks locally
- Install dependencies (ideally in a virtualenv):
- Configure provider keys (e.g.,
OPENAI_API_KEY). - Execute the runner:
- Inspect the JSON or load it into a notebook to compare against the published table.
The script spins up datasets defined in sdks/opik_optimizer/benchmarks/config.py, runs each optimizer with consistent trial budgets, and logs runs to Opik so you can review traces. Note that production use should include separate validation datasets to prevent overfitting—see Define datasets for guidance.
Looking for production-style examples beyond synthetic benchmarks? Check out the agent optimizations demos repo. It contains end-to-end scenarios (LangGraph, RAG, support bots) and shows how different optimizers behave in real workloads.
Next steps
- Learn how each optimizer works in the Algorithms overview.
- Customize the benchmark configs (datasets, metrics, budgets) to mirror your production workload.
- Share results or contribute improvements via GitHub.
