We regularly evaluate every optimizer against shared datasets so you can make informed trade-offs. This page summarizes the latest results and explains how to reproduce them with the public benchmark scripts.
Each run uses Opik datasets backed by open-source corpuses commonly used in academia:
Results shown below use openai/gpt-5-nano for evaluation on non multi-hop based runs. Scores will change if you select different models, metrics, agent configurations or prompt seeds.
These are directional numbers. Some optimizers use more LLM/tool calls per trial than others (e.g., the HRPO Hierarchical Reflective Prompt Optimizer batches multiple analyses), so cost and runtime are not apples-to-apples even when the trial budget matches. Re-run the suite with your own datasets, models, and cost constraints before committing to a single optimizer.
OPENAI_API_KEY).The script spins up datasets defined in sdks/opik_optimizer/benchmarks/config.py, runs each optimizer with consistent trial budgets, and logs runs to Opik so you can review traces. Note that production use should include separate validation datasets to prevent overfitting—see Define datasets for guidance.
Looking for production-style examples beyond synthetic benchmarks? Check out the agent optimizations demos repo. It contains end-to-end scenarios (LangGraph, RAG, support bots) and shows how different optimizers behave in real workloads.