Changelog

The single-run page in Optimization Studio has been rebuilt end to end. The header now shows the run’s dataset, model, algorithm, and metric as pills (hover the metric pill for its full config), and the KPI cards show the current value, a trend badge, and a baseline → best line, with the duration now measured to actual run completion rather than the last trial’s timestamp. The trials progress chart highlights the best trial by default and labels discarded trials clearly instead of the previous “pruned” terminology.

Previously, an optimization run whose worker crashed or lost contact could sit in “Initializing” or “Running” forever with no indication anything was wrong. A background job now detects runs stuck for too long (5 minutes with no worker pickup, or 8 hours still running) and marks them as failed with a system-generated reason written to the run’s logs.

• Workspace switcher now matches the projects menu — The workspace dropdown now groups workspaces into Pinned and Recently visited sections with per-row pin/unpin controls, instead of a single flat list, matching the pattern already used for the projects menu.

• Mobile web onboarding — Opening Opik on a phone now shows a short guided walkthrough (what a trace is, how issues get flagged, how to connect your first integration) instead of the desktop quickstart, with an option to email yourself setup instructions.

• Workspace home now always opens on Projects — Navigating to a workspace’s home URL now consistently lands on the Projects tab, instead of sometimes reopening whichever project you last viewed.

• Metadata filter dropdown no longer overlaps rows — Long metadata keys that wrapped onto multiple lines in the filter panel’s autocomplete dropdown could visually overlap the option below them; rows now grow to fit the wrapped text.

• GPT-5.6 models now behave as reasoning models — gpt-5.6-luna, gpt-5.6-sol, and gpt-5.6-terra were missing from the reasoning-model list, so the Playground and online scoring still tried to send temperature, which these models reject. They now show the reasoning-effort selector (including a new Max option) instead of temperature/Top P, consistent with other reasoning models.

• Perplexity model costs now load correctly — Perplexity was missing from the provider list used to load pricing data, so calls through Perplexity (including the sonar model family) were costed at $0. Pricing now loads correctly.

• Clearer errors from Python-based LLM-as-judge scoring — A metric evaluation that hit an empty or malformed response from the scoring service previously failed with an opaque server error; it now surfaces the actual underlying error message.

• Tool-call spans from OpenTelemetry now typed correctly — Spans carrying the OTel gen_ai.tool.call.arguments / gen_ai.tool.call.result attributes are now classified as tool spans in the UI, matching spans tracked directly through the SDK.

• Fixed intermittent 500s retrieving threads and experiment project lists — Two endpoints (thread lookup, and listing the projects an experiment spans) could return an unmapped server error when the underlying query legitimately returned more than one row for the same logical entity. Both now handle this correctly.

• More accurate error codes from LLM providers — Errors from OpenAI-compatible providers (including calls routed through OpenRouter) were sometimes parsed with the wrong error model, which could turn a rate-limit error into a generic server error. The correct error model is now selected based on the error’s shape, so the real status code is surfaced.

• LangChain integration: cost capture through proxies, and provider override — OpikTracer accepts a new provider argument to override provider auto-detection, useful when routing ChatOpenAI through a proxy such as LiteLLM. When the proxy reports its own cost via the x-litellm-response-cost response header (with include_response_headers=True on ChatOpenAI), Opik now uses that cost instead of estimating $0.

• opik migrate no longer risks OOM on partially-stale projects — When a migrated experiment referenced only deleted or timestamp-less traces, the CLI fell back to an unbounded read of the entire project’s spans. It now skips the unrecoverable span data for that batch (with a warning) and continues the migration instead of risking an out-of-memory crash.

• SDK tolerates new OpenAI usage fields and avoids a broken LiteLLM release — The SDK no longer breaks when OpenAI adds new fields to its usage payload, and the LiteLLM dependency now excludes the 1.92.* release line, which crashed on import without optional proxy dependencies installed.

• Dataset and test suite creation logs point to the right project — The URL logged by the SDK after creating a dataset or test suite now links directly to that resource inside its actual project, instead of a generic link that could resolve to the wrong project.

• Self-hosted: ClickHouse upgraded to 26.3 LTS — The bundled ClickHouse instance (Helm chart, docker-compose, and testcontainers) is now pinned to 26.3 LTS. No manual data migration is required, though major-version upgrades may take longer to complete.

• Self-hosted: ingestion load spreads more evenly across backend pods — The frontend nginx proxy now recycles its keepalive connections to the backend on a schedule, instead of holding them open indefinitely. On Kubernetes deployments this prevents a small number of backend pods from absorbing a disproportionate share of ingestion traffic while others sit idle.

Online LLM-as-judge scoring rules for traces and threads can now be given a Max cost per evaluation (USD) budget. Once an evaluation’s cumulative spend crosses the limit, the agentic scoring loop stops starting new tool-calling turns and returns a best-effort verdict from whatever evidence it has already gathered, instead of continuing to spend without bound. The monitoring trace for a capped run is tagged budget_exceeded so you can tell a budget-limited verdict apart from a fully-investigated one. The field is hidden for span-scope rules, since a single LLM call has no loop to cap.

Starting a new optimization run no longer navigates away from the runs list: the “New optimization run” form now opens as a side panel over the list (via the list’s ?new — or ?template / ?rerun — parameter) so you can start, clone, or re-run an optimization without losing your place. The runs list itself has a new Item source column (the same reusable cell used on the Experiments page), Run ID / Algorithm / Metric columns available to add, and the “Pruned” trial status is now labeled Discarded for clarity.

• Diagnostics: clearer failure copy and working billing link — Follow-ups to last week’s Diagnostics failure reasons: a permission-denied run failure now shows explanatory copy instead of a generic message, and the “View billing” link in the failure dialog now opens the right settings page.

• Claude Agent SDK / Claude Code traces now show real input/output — Spans emitted by the Claude Agent SDK and Claude Code carried their prompt, tool, and response content on attributes that no mapping rule recognized, so everything fell into a generic attribute bag instead of the trace’s input/output fields. These spans are now mapped correctly, and model/provider are set so cost is calculated instead of showing as $0.

• New: Mistral AI Python integration (track_mistral) — A native integration for the mistralai Python SDK: wrap a client with track_mistral(client) to trace chat.complete/chat.complete_async and chat.stream/stream_async calls (including structured-output parse calls and streamed responses aggregated into a single span), with input, output, token usage, and cost captured automatically.

• opik migrate dataset can resume after an interruption — A crash, network drop, or OOM during opik migrate dataset used to mean starting over. Progress now checkpoints after each completed experiment, so a re-run picks up where it left off, and the source dataset keeps its original name until the destination copy is fully verified — a failed run leaves only a discardable temporary copy behind rather than a renamed source.

• opik migrate dataset no longer runs out of memory on large datasets — The migration’s read path could request full-fidelity pages large enough to exhaust the process’s memory and get killed mid-run. Reads are now paged more conservatively, with an automatic page-size shrink on read timeouts/connection errors.

• opik export/opik import preserve tags for prompts, experiments, and datasets — Tags on prompts, experiments, and datasets were silently dropped when exporting or importing via the CLI (traces and spans already round-tripped correctly). All four entity types now preserve tags end-to-end.

• opik export ... all no longer runs out of memory on large projects — Exporting a project used to buffer every trace and span in memory before writing anything, so peak memory scaled with project size. Export now processes traces in bounded chunks, flushing and discarding each chunk as it’s written, and an interrupted export resumes mid-project instead of restarting.

• Fixed duplicate rows in the test suite detail view — Editing a dataset item could cause its experiment result to render multiple times in the test suite detail view, due to a join that matched more than one dataset-item version per item.

• Prompt names are now unique per project instead of per workspace — Two prompts with the same name in different projects no longer conflict.

• Custom metrics fail loudly on an unresolvable reference key — An equals/levenshtein_ratio/numerical_similarity metric configured with a reference key that matches no dataset field used to silently score every item 0, making a broken metric indistinguishable from a genuinely low-scoring run. This now raises a clear error at build time listing the available fields, and a per-item missing value is scored 0 with an explicit “Missing reference value” reason instead of a spurious perfect match.

• Fixed cost calculation for the highest LiteLLM pricing tiers — Models with above_128k_tokens (e.g. Gemini 1.5 Flash) or above_272k_tokens (e.g. the GPT-5.4/5.5 family and their Azure-hosted equivalents) pricing tiers were being undercharged, since only the above_200k_tokens tier was applied. All published tiers are now taken into account.

• Fixed an outbound gzip decompression error — Self-hosted deployments with jerseyClient.gzipEnabled: true could hit ZipException: Not in GZIP format on outbound calls that read a gzip-encoded response body (for example, some Ollama responses), because the response was decompressed twice. The stale Content-Encoding header is now stripped after the first decode.

• Faster Logs page loads on large projects — The Traces/Spans/Threads “log your first trace” onboarding check used to run a full project-wide aggregation query just to see if any row existed. It now uses a minimal existence check, removing a multi-second scan from every Logs page load.

• Reduced full-table scans in dataset and experiment queries — Several ClickHouse queries backing dataset-item filters and experiment views scanned far more data than needed — in one case, an experiment/dataset-item filter query read 75x fewer granules after being scoped to the relevant experiment’s trace IDs instead of the whole workspace. These queries now carry tighter project/trace-id/workspace bounds, reducing ClickHouse load on large workspaces.

Error, duration, and cost cells across the Traces, Spans, and Threads tables now have an “Explain” button (the Ollie owl icon) that streams a plain-language explanation of that specific value without leaving the table — no need to open the trace panel and dig through raw payloads to understand why a call errored, took as long as it did, or cost what it did.

LLM-as-a-judge scoring runs (on traces, spans, and threads) are now recorded as monitoring traces automatically, with a prepare_evaluation span and one span per scoring call, including token usage and cost. They’re hidden from the main Logs view by default, and each rule on the Online Evaluation page now has a Go to traces action that opens a scoped, filtered view of that rule’s evaluation activity — useful for confirming a rule is running, checking what it cost, or debugging why it errored. This is on by default for all workspaces.

• Diagnostics now explains why a run failed — Previously, a Diagnostics run that crashed or never started (for example, from exhausted LLM credits) just spun until it timed out, with no indication of what went wrong. Failed runs now show a specific reason (out of credits, rate limited, provider error, or never started) with a “Try again” action, and the failure timeout was cut from 12 minutes to 5.

• Playground and online scoring: Sonnet 5 and Fable 5 no longer error out — These models were missing from the capability list, so the Playground kept showing temperature/Top P sliders and sending those parameters, which the models reject. The sliders are now hidden and the parameters are no longer sent, matching the behavior already in place for Opus 4.7/4.8.

• Azure-hosted OpenAI model prices now load correctly — Model prices for the azure/* model family (gpt-4o, gpt-5, codex-mini, and others) were silently failing to load because of a provider-mapping gap, causing cost to show as unavailable for these models.

• Experiment and Playground traces now show up in scoped “Go to logs” views — Traces created by experiment runs and Playground runs were sometimes missing from their respective “Go to logs” tables even though the traces existed, due to a visibility-filter mismatch. These views now correctly scope by the experiment or run itself.

• Playground: dataset and metric selection redesigned — The “Run experiment” dialog is replaced by inline dataset and metrics dropdowns in the Playground header, so you can change either independently without reopening a modal. Metrics can also be created and edited in place.

• Permission gating extended to more pages — Users with the Annotator workspace role no longer see Agent Playground, Online Evaluation, or Alerts in the sidebar. Prompt Library view/edit access is now governed by the same permission system, at both the UI and API level.

• opik import --to-workspace for cross-workspace imports — The opik import CLI command accepts a --to-workspace option to import exported data into a different workspace than the one it was exported from, without needing to restructure the export directory as a workaround.

• opik migrate dataset --exclude-experiments — This flag skips migrating a dataset’s experiments and optimizations, migrating only the dataset and its version history. Useful for large datasets where the experiment/optimization cascade isn’t needed.

• Fixed intermittent errors loading prompt and dataset versions under load — A query pattern that caused MySQL to materialize large temporary tables could fail outright under load. Prompt and dataset version lookups are now rewritten to avoid the issue.

• Faster trace ingestion — Removed a redundant ClickHouse lookup that ran on every trace ingestion just to check the trace’s last-updated timestamp, reducing ingestion latency and ClickHouse load.

• SDK: shared connection resources across Opik() clients — Code that creates multiple Opik() clients (for example, one per task or request) now shares the underlying connection pool and background threads across clients with matching configuration instead of creating a full new stack each time, reducing thread and connection overhead.

• Self-hosted: ClickHouse FINAL reads no longer over-read on skip-indexed tables — Disabled the new ClickHouse 25.x default for use_skip_indexes_if_final_exact_mode, which was causing skip-indexed queries to read far more data than necessary; Opik’s queries already prune by primary key and project ID, so the exact mode isn’t needed.

Online LLM-as-judge scoring rules can now reason over files attached to traces — images, PDFs, documents, and other binary content — in addition to the trace’s text fields. Previously, attachment-aware evaluation was only available for thread (conversation) scoring; single-trace scoring rules now share the same capability.

When a trace carries attachments, the scorer automatically routes to an agentic tool-calling loop. The judge model receives a ReadTool that lists the attached files and a GetAttachmentTool that fetches each file as multimodal content. Up to eight attachments are injected per evaluation. When the configured provider does not support tool calls, a warning is surfaced rather than silently failing.

The opik export and opik import CLI commands now require a project argument, matching how Opik v2 organises data (every dataset, prompt, and experiment lives inside a project).

New command shape:

$
opik export WORKSPACE PROJECT ITEM [NAME] [OPTIONS]

Where ITEM is one of all, dataset, traces, experiment, or prompt. On-disk output is written to PATH/WORKSPACE/projects/<id>/ with a project.json name index, so the same --path value round-trips between export and import.

• MCP server: opik mcp status command + auto-detection of hosted server — opik configure and opik mcp configure now probe for a Comet-hosted MCP server (HTTP + OAuth) and fall back to the local uvx opik-mcp server only when the hosted one is unavailable. A new opik mcp status command (and the equivalent opik configure status) reports the active Opik configuration and which AI assistants have the MCP server registered, flagging any drift from ~/.opik.config. Use --local-server to force the local path.

• Vercel AI SDK v7 and Vercel eve agent support — OpikExporter now captures traces from AI SDK v4 through v7 and from the Vercel eve agent framework. Multi-turn eve conversations are automatically grouped into a single thread via the session ID, and cached-token usage is captured.

• track_openai: custom provider argument — An optional provider parameter (str or opik.LLMProvider) can now be passed to track_openai to override the auto-detected provider on every span. Useful for OpenAI-compatible APIs where the hostname-derived default is not descriptive.

  1
  client = track_openai(openai.OpenAI(base_url="..."), provider="my-provider")

• Configurable local runner poll interval — The opik connect / opik endpoint runner polls for jobs every 0.5 s by default (~120 requests/min). Setting OPIK_RUNNER_POLL_INTERVAL (seconds) reduces this for environments where firewalls or proxies throttle sustained polling. Sustained failures now escalate to an actionable warning with a firewall/proxy note rather than going silent.

• Assertion scoring errors now shown in the UI — When an LLM-as-judge assertion scorer fails (for example, because the configured model is unavailable), an orange Scoring error badge with an actionable tooltip is shown instead of a spinning indicator or a generic grey Skipped tag.

• Test suite items no longer lost after editing an assertion — A ClickHouse parameter-escaping bug caused the \n sequences inside LLM-judge prompts to be unescaped to literal newlines, producing invalid stored JSON and silently dropping the row on read. Items now survive edit round-trips correctly.

• ChrF metric: char_order and ignore_whitespace now forwarded to NLTK — These parameters were stored in the ChrF object but never passed to nltk.translate.chrf_score.sentence_chrf. char_order is now forwarded as max_len and ignore_whitespace is forwarded as-is.

• OpenAI audio models: audio input tokens billed at the correct rate — For models such as gpt-4o-audio-preview and gpt-4o-realtime, audio input tokens are now charged at the input_cost_per_audio_token rate (up to 16× the standard text rate). Previously they were silently billed at the standard text input rate.

• opik export span fetch performance — When exporting a filtered subset of traces, spans are now fetched with a trace_id range bound derived from the matched trace IDs rather than scanning the entire project. Large filtered exports that previously stalled now complete promptly.

• Optimization studio renamed to Optimization runs — The v2 UI section previously called “Optimization studio” is now labelled Optimization runs everywhere.

• Attachment base64 detection is now O(n) — The regex that detects whether an attachment payload is already base64-encoded was rewritten with a possessive quantifier, making it linear in the payload size and avoiding catastrophic backtracking on large binary attachments.

• Self-hosted: ClickHouse upgraded to 25.8 LTS — The bundled ClickHouse instance (Helm chart, docker-compose, and testcontainers) is now pinned to 25.8 LTS and the operator to 0.27.1. This release requires no manual migration; existing data is read correctly by the new version.

Traces from PydanticAI agents ingested over OpenTelemetry (via Logfire) now show complete, correctly-typed data and accurate cost. Several gaps in the OTel → Opik span mapping were fixed:

• Tool call I/O is now captured — tool span inputs and outputs now appear in the correct Input/Output fields. Previously, logfire’s tool_response attribute fell into the Input bucket and the Output field was empty.
• Errors surface at the trace level — when a span carries an OTel exception event or STATUS_CODE_ERROR status, the error is translated into error_info and propagated up to the parent trace so failed agent runs are flagged at the top level.
• Gemini/Google cost now calculated — PydanticAI reports the generic google provider name, which previously matched no pricing row and showed $0 cost. The backend now resolves it to google_vertexai or google_ai based on the server hostname.
• Agent-run spans correctly typed — spans from invoke_agent operations are now typed general instead of llm, so the trace tree accurately reflects the agent structure.

Setting up Claude Desktop (or any MCP client) to connect to Opik no longer requires manually editing JSON config files. Running opik configure now offers an opt-in MCP setup step at the end of the flow. The standalone opik mcp configure command runs the same wizard on its own. Both automatically write the Opik MCP server block to your Claude Desktop configuration.

• OpenClaw added to onboarding integrations — The Get Started page now includes OpenClaw with its four-step CLI setup guide (install plugin → configure → status → run), rendered with bash syntax highlighting. The integration details dialog also supports multi-step and non-pip integrations going forward.

• Logs view: search bar repositioned inline with filters — The search bar now sits alongside active filter chips in a single wrapping row. Toolbar controls (Columns, Row height, Refresh) moved to the tab row, reducing visual clutter in the main search area. Chart bars changed to violet and table tags to green.

• Annotation queue: scores scoped per queue — Feedback scores and comments now carry a source_queue_id so each queue’s metrics (average scores, reviewer counts) are computed independently. A score submitted through one queue no longer counts toward another queue’s statistics when the same trace appears in both.

• Annotation queue: false “another annotator is reviewing” message resolved — After submitting a feedback score, the lock heartbeat now correctly recognizes existing lock holders, so annotators no longer see a false “another annotator is reviewing this item” message immediately after scoring.

• Thread-scoped annotation queues no longer crash — Opening a thread annotation queue without additional queue filters no longer throws an Unknown expression or function identifier error and crashes the queue view.

• Experiments page: charts stay fixed during horizontal table scroll — The feedback-score charts now remain pinned to the top while the experiments table scrolls horizontally underneath.

• Playground: immediate failure on exhausted OpenAI quota — When an OpenAI API key has insufficient quota (HTTP 429 insufficient_quota), the Playground now returns an error immediately instead of retrying the request multiple times and delaying the response.

Long-running evaluation jobs that get cut short — by Ctrl-C, an OOM error, a failed scoring metric, or a network blip — can now be continued from where they stopped instead of restarting from scratch. opik.evaluate_resume(experiment_id, task, scoring_metrics=[...]) replays only the trials that did not complete, merges them with the ones that did, and returns a single EvaluationResult covering the whole experiment.