Changelog

evaluate_threads only accepted trace_input_transform and trace_output_transform, so a thread was flattened to plain role/content messages with no way to pass along the documents each answer was grounded on — judging relevance or faithfulness for multi-turn RAG agents meant falling back to scoring individual traces instead.

A new trace_context_transform argument receives the whole trace object and attaches the extracted context to that trace’s message:

• Daily briefings report “Not enough credits” instead of failing silently — A daily report that ran out of LLM credits mid-run used to just say “Briefing failed,” with the pod already allocated and the spend already burned. The trigger now checks credits up front, and a briefing that can’t run for that reason shows “Not enough credits” with a direct link to add more, alongside Run again.

• Provider errors keep their real HTTP status instead of turning into a 500 — Traces and online-scoring rules calling out to an LLM provider reported a generic server error whenever the provider’s response couldn’t be parsed into its usual error shape — a plain-text upstream outage message, or a rate-limit response wrapped by a retry layer. The provider’s actual status is now recovered from these cases too, so a caller-side or provider-side failure is no longer reported as an Opik fault.

• LLM-as-judge scoring retries on empty provider responses and no longer hangs indefinitely — An empty structured-output response from the judge model previously fell outside the retry policy and cost the item its whole score. These are now retried, and judge calls carry an explicit timeout and retry budget so a stalled provider can no longer block scoring indefinitely.

• Silent span-conversion failures in the ADK integration are now reported — When converting an ADK LlmResponse or extracting its usage failed, the span was still logged but silently missing output and usage, with nothing surfacing the failure. This is now logged as an error instead of being swallowed.

• Fixed two production NullPointerExceptions — Deleting or batch-updating dataset items, and querying span metrics, could 500 when a filter used an operator not supported by its field, instead of returning the same 400 other endpoints already gave. Separately, a judge response with no text (for example from a content filter) could crash scoring for the whole trace instead of reporting that nothing was scored.

• Prompts no longer leak across projects with the same name — A backward-compatibility fallback for looking up a prompt by name, when no project was specified, incorrectly matched a same-named prompt from any project in the workspace instead of only project-less legacy prompts, so a client without a project_name set could read, and unintentionally version, another project’s prompt.

• Grouped chart series and tag colors are easier to tell apart — Two adjacent colors in the series palette were close enough to be confused in a thin chart line; one has been replaced with a more distinct color, so any label that resolved to that palette slot now renders more legibly wherever it appears — chart series, tag chips, and feedback scores.

• Dataset item sorting by JSON fields now handles special characters correctly — Sorting by input.*, output.*, or metadata.* keys built the underlying query by interpolating the key text directly, unlike data.* sorting, which could corrupt results for keys containing quotes or other special characters. JSON sort keys are now passed as bound parameters like everywhere else.

• Faster traces table on projects with many feedback score columns — The traces table builds one column per feedback-score name, and every cell — editable or not — was paying the cost of wiring up inline editing. Non-editable score columns now render a lighter read-only cell, cutting a multi-second main-thread block down substantially on projects with hundreds of score columns and rows.

Experiment traces were previously reachable only through a small “Go to logs” link tucked into the metadata row, easy to miss — users who couldn’t find it would conclude nothing had been logged. They’re now a dedicated Logs tab on the experiment page, always scoped to that experiment or, when comparing, to every experiment in the comparison. Comparing experiments that live in different projects now shows an explicit message instead of silently dropping traces, since logs are read per project.

• LLM-as-judge online-eval rules score more consistently — Built-in judge prompt templates and the scoring engine disagreed on whether a judge’s answer should be a flat or nested JSON object, so a rule could non-deterministically return an empty score depending on which shape the model happened to pick. The engine now accepts both shapes, correctly parses quoted numbers ("0.8" no longer becomes 0.0) and common boolean spellings (yes/no, pass/fail, on/off), and reports — rather than silently drops — a score name the rule doesn’t recognize. Judge prompts are also no longer HTML-escaped before being sent, which had been corrupting values like base64 data URLs assembled from template variables.

• Unsupported LLM features now return a clear 400 instead of a mysterious 500 — Requesting a capability the selected provider doesn’t support (for example, requiring tool choice against a model that can’t do it) previously failed with a generic server error and, for online-scoring rules, burned through the full retry budget before the message was dropped. It’s now reported immediately as a 400 naming the unsupported feature, for both regular and streaming requests.

• Fixed a Vertex AI thread leak that could exhaust a backend pod — Every LLM call routed through Vertex AI built a brand-new client whose background threads were never released, which could accumulate to thousands of threads per pod over time and eventually require a restart. The client is now created and closed per request instead.

• Optimization cost totals now include GEPA reflection spend — LLM calls the optimizer makes internally during GEPA reflection belong to no single trial, so they were billed but excluded from a run’s total cost, undercounting real spend. That spend is now traced, tagged with the run, and included in total_optimization_cost (opik-optimizer 3.2.0). Upgrade both packages together — pip install -U "opik-optimizer>=3.2.0" "gepa>=0.1.0" — raising gepa on its own breaks at runtime against the older reflection-template format that 3.2.0 no longer speaks.

• Self-hosted: per-component image registry override, and a demo-data job fix — The Helm chart now lets you set a separate image registry for the backend, frontend, or Python backend individually, falling back to the chart-wide registry when unset. The demo-data job also now honors a component’s configured image repository instead of always defaulting to opik-python-backend.

• Faster trace counts on projects using guardrails filters — The traces-count query that fires on every Logs page load always joined against guardrails results and sorted every row to deduplicate, even when no guardrails filter was applied. Both are now skipped unless actually needed, cutting rows read roughly in half and cutting measured P99 latency by about 90% on production workspaces.

• Faster trace lookups by thread ID — Filtering traces or threads by thread ID wasn’t engaging an existing prefilter, so a lookup could scan a project’s entire span history just to find one thread’s traces. The prefilter now engages for thread-scoped lookups, cutting measured production query times from multiple seconds to under 200ms.

Three cases that previously produced no score and no actionable error are now reported: a metric that doesn’t declare **kwargs no longer fails on every item with a confusing “unexpected keyword argument” error (inputs are narrowed to what the metric’s signature actually accepts), an item-level evaluator with an unsupported type now produces a failed score explaining why instead of being silently skipped, and a scoring_key_mapping value that matches nothing is now logged as a warning instead of at debug level.

• Experiment.batch_upload_items for bulk experiment creation — A new method uploads experiment items together with their traces, spans, and feedback scores in a single call, instead of creating traces first and linking them afterward. Items are validated up front, batched to respect backend limits, and retried automatically on rate limiting.

• Stuck optimization runs are detected within minutes instead of hours — Following up on the stalled-run detection shipped previously, the reaper now derives liveness from actual trial and item progress rather than only the run’s last status change, catching a run orphaned by a worker restart in around an hour instead of up to eight. A run that’s still legitimately evaluating items is never mistaken for stalled, a wrongly-flagged run recovers automatically once its worker reports back, and it can now be cancelled instead of being stuck.

• Nebius and SambaNova model costs now load correctly — Both providers were missing from the internal provider list used to load pricing data, so calls through Nebius or SambaNova models (including their DeepSeek, Qwen, and Llama catalogs) were costed at $0. Pricing now loads correctly for both.

• Project lookups no longer leak across workspaces — Looking up a project by name cached the result under a key that didn’t account for the workspace, so switching organizations in the UI without a full page reload could keep resolving a project name to the previous organization’s project.

• Python UDF metric evaluation retries on transient connection drops — When the Python scoring backend closed its connection before responding, the error wasn’t recognized as retriable, so a transient connection drop failed the evaluation outright. These are now retried automatically.

• More accurate cost and usage on experiments with re-sent spans — A span re-sent under a different parent span could be counted twice when computing usage and cost rollups for experiments and optimizations. Spans are now deduplicated by their own id, so each span is counted once regardless of how it was re-sent.

• Faster trace loading for traces with many spans — Looking up traces by id no longer forces a full deduplication pass (FINAL) over the spans table; the query now filters to the relevant trace first and deduplicates only what’s left, cutting CPU cost by over 90% on traces with hundreds of spans.

• Faster project-level trace stats — The query behind trace statistics on the projects list no longer performs a full blocking sort over every span in a project’s history to deduplicate them, so project pages with long-running, high-volume projects load faster.

The single-run page in Optimization Studio has been rebuilt end to end. The header now shows the run’s dataset, model, algorithm, and metric as pills (hover the metric pill for its full config), and the KPI cards show the current value, a trend badge, and a baseline → best line, with the duration now measured to actual run completion rather than the last trial’s timestamp. The trials progress chart highlights the best trial by default and labels discarded trials clearly instead of the previous “pruned” terminology.

Previously, an optimization run whose worker crashed or lost contact could sit in “Initializing” or “Running” forever with no indication anything was wrong. A background job now detects runs stuck for too long (5 minutes with no worker pickup, or 8 hours still running) and marks them as failed with a system-generated reason written to the run’s logs.

• Workspace switcher now matches the projects menu — The workspace dropdown now groups workspaces into Pinned and Recently visited sections with per-row pin/unpin controls, instead of a single flat list, matching the pattern already used for the projects menu.

• Mobile web onboarding — Opening Opik on a phone now shows a short guided walkthrough (what a trace is, how issues get flagged, how to connect your first integration) instead of the desktop quickstart, with an option to email yourself setup instructions.

• Workspace home now always opens on Projects — Navigating to a workspace’s home URL now consistently lands on the Projects tab, instead of sometimes reopening whichever project you last viewed.

• Metadata filter dropdown no longer overlaps rows — Long metadata keys that wrapped onto multiple lines in the filter panel’s autocomplete dropdown could visually overlap the option below them; rows now grow to fit the wrapped text.

• GPT-5.6 models now behave as reasoning models — gpt-5.6-luna, gpt-5.6-sol, and gpt-5.6-terra were missing from the reasoning-model list, so the Playground and online scoring still tried to send temperature, which these models reject. They now show the reasoning-effort selector (including a new Max option) instead of temperature/Top P, consistent with other reasoning models.

• Perplexity model costs now load correctly — Perplexity was missing from the provider list used to load pricing data, so calls through Perplexity (including the sonar model family) were costed at $0. Pricing now loads correctly.

• Clearer errors from Python-based LLM-as-judge scoring — A metric evaluation that hit an empty or malformed response from the scoring service previously failed with an opaque server error; it now surfaces the actual underlying error message.

• Tool-call spans from OpenTelemetry now typed correctly — Spans carrying the OTel gen_ai.tool.call.arguments / gen_ai.tool.call.result attributes are now classified as tool spans in the UI, matching spans tracked directly through the SDK.

• Fixed intermittent 500s retrieving threads and experiment project lists — Two endpoints (thread lookup, and listing the projects an experiment spans) could return an unmapped server error when the underlying query legitimately returned more than one row for the same logical entity. Both now handle this correctly.

• More accurate error codes from LLM providers — Errors from OpenAI-compatible providers (including calls routed through OpenRouter) were sometimes parsed with the wrong error model, which could turn a rate-limit error into a generic server error. The correct error model is now selected based on the error’s shape, so the real status code is surfaced.

• LangChain integration: cost capture through proxies, and provider override — OpikTracer accepts a new provider argument to override provider auto-detection, useful when routing ChatOpenAI through a proxy such as LiteLLM. When the proxy reports its own cost via the x-litellm-response-cost response header (with include_response_headers=True on ChatOpenAI), Opik now uses that cost instead of estimating $0.

• opik migrate no longer risks OOM on partially-stale projects — When a migrated experiment referenced only deleted or timestamp-less traces, the CLI fell back to an unbounded read of the entire project’s spans. It now skips the unrecoverable span data for that batch (with a warning) and continues the migration instead of risking an out-of-memory crash.

• SDK tolerates new OpenAI usage fields and avoids a broken LiteLLM release — The SDK no longer breaks when OpenAI adds new fields to its usage payload, and the LiteLLM dependency now excludes the 1.92.* release line, which crashed on import without optional proxy dependencies installed.

• Dataset and test suite creation logs point to the right project — The URL logged by the SDK after creating a dataset or test suite now links directly to that resource inside its actual project, instead of a generic link that could resolve to the wrong project.

• Self-hosted: ClickHouse upgraded to 26.3 LTS — The bundled ClickHouse instance (Helm chart, docker-compose, and testcontainers) is now pinned to 26.3 LTS. No manual data migration is required, though major-version upgrades may take longer to complete.

• Self-hosted: ingestion load spreads more evenly across backend pods — The frontend nginx proxy now recycles its keepalive connections to the backend on a schedule, instead of holding them open indefinitely. On Kubernetes deployments this prevents a small number of backend pods from absorbing a disproportionate share of ingestion traffic while others sit idle.

Online LLM-as-judge scoring rules for traces and threads can now be given a Max cost per evaluation (USD) budget. Once an evaluation’s cumulative spend crosses the limit, the agentic scoring loop stops starting new tool-calling turns and returns a best-effort verdict from whatever evidence it has already gathered, instead of continuing to spend without bound. The monitoring trace for a capped run is tagged budget_exceeded so you can tell a budget-limited verdict apart from a fully-investigated one. The field is hidden for span-scope rules, since a single LLM call has no loop to cap.

Starting a new optimization run no longer navigates away from the runs list: the “New optimization run” form now opens as a side panel over the list (via the list’s ?new — or ?template / ?rerun — parameter) so you can start, clone, or re-run an optimization without losing your place. The runs list itself has a new Item source column (the same reusable cell used on the Experiments page), Run ID / Algorithm / Metric columns available to add, and the “Pruned” trial status is now labeled Discarded for clarity.

• Diagnostics: clearer failure copy and working billing link — Follow-ups to last week’s Diagnostics failure reasons: a permission-denied run failure now shows explanatory copy instead of a generic message, and the “View billing” link in the failure dialog now opens the right settings page.

• Claude Agent SDK / Claude Code traces now show real input/output — Spans emitted by the Claude Agent SDK and Claude Code carried their prompt, tool, and response content on attributes that no mapping rule recognized, so everything fell into a generic attribute bag instead of the trace’s input/output fields. These spans are now mapped correctly, and model/provider are set so cost is calculated instead of showing as $0.

• New: Mistral AI Python integration (track_mistral) — A native integration for the mistralai Python SDK: wrap a client with track_mistral(client) to trace chat.complete/chat.complete_async and chat.stream/stream_async calls (including structured-output parse calls and streamed responses aggregated into a single span), with input, output, token usage, and cost captured automatically.

• opik migrate dataset can resume after an interruption — A crash, network drop, or OOM during opik migrate dataset used to mean starting over. Progress now checkpoints after each completed experiment, so a re-run picks up where it left off, and the source dataset keeps its original name until the destination copy is fully verified — a failed run leaves only a discardable temporary copy behind rather than a renamed source.

• opik migrate dataset no longer runs out of memory on large datasets — The migration’s read path could request full-fidelity pages large enough to exhaust the process’s memory and get killed mid-run. Reads are now paged more conservatively, with an automatic page-size shrink on read timeouts/connection errors.

• opik export/opik import preserve tags for prompts, experiments, and datasets — Tags on prompts, experiments, and datasets were silently dropped when exporting or importing via the CLI (traces and spans already round-tripped correctly). All four entity types now preserve tags end-to-end.

• opik export ... all no longer runs out of memory on large projects — Exporting a project used to buffer every trace and span in memory before writing anything, so peak memory scaled with project size. Export now processes traces in bounded chunks, flushing and discarding each chunk as it’s written, and an interrupted export resumes mid-project instead of restarting.

• Fixed duplicate rows in the test suite detail view — Editing a dataset item could cause its experiment result to render multiple times in the test suite detail view, due to a join that matched more than one dataset-item version per item.

• Prompt names are now unique per project instead of per workspace — Two prompts with the same name in different projects no longer conflict.

• Custom metrics fail loudly on an unresolvable reference key — An equals/levenshtein_ratio/numerical_similarity metric configured with a reference key that matches no dataset field used to silently score every item 0, making a broken metric indistinguishable from a genuinely low-scoring run. This now raises a clear error at build time listing the available fields, and a per-item missing value is scored 0 with an explicit “Missing reference value” reason instead of a spurious perfect match.

• Fixed cost calculation for the highest LiteLLM pricing tiers — Models with above_128k_tokens (e.g. Gemini 1.5 Flash) or above_272k_tokens (e.g. the GPT-5.4/5.5 family and their Azure-hosted equivalents) pricing tiers were being undercharged, since only the above_200k_tokens tier was applied. All published tiers are now taken into account.

• Fixed an outbound gzip decompression error — Self-hosted deployments with jerseyClient.gzipEnabled: true could hit ZipException: Not in GZIP format on outbound calls that read a gzip-encoded response body (for example, some Ollama responses), because the response was decompressed twice. The stale Content-Encoding header is now stripped after the first decode.

• Faster Logs page loads on large projects — The Traces/Spans/Threads “log your first trace” onboarding check used to run a full project-wide aggregation query just to see if any row existed. It now uses a minimal existence check, removing a multi-second scan from every Logs page load.

• Reduced full-table scans in dataset and experiment queries — Several ClickHouse queries backing dataset-item filters and experiment views scanned far more data than needed — in one case, an experiment/dataset-item filter query read 75x fewer granules after being scoped to the relevant experiment’s trace IDs instead of the whole workspace. These queries now carry tighter project/trace-id/workspace bounds, reducing ClickHouse load on large workspaces.

Error, duration, and cost cells across the Traces, Spans, and Threads tables now have an “Explain” button (the Ollie owl icon) that streams a plain-language explanation of that specific value without leaving the table — no need to open the trace panel and dig through raw payloads to understand why a call errored, took as long as it did, or cost what it did.

LLM-as-a-judge scoring runs (on traces, spans, and threads) are now recorded as monitoring traces automatically, with a prepare_evaluation span and one span per scoring call, including token usage and cost. They’re hidden from the main Logs view by default, and each rule on the Online Evaluation page now has a Go to traces action that opens a scoped, filtered view of that rule’s evaluation activity — useful for confirming a rule is running, checking what it cost, or debugging why it errored. This is on by default for all workspaces.

• Diagnostics now explains why a run failed — Previously, a Diagnostics run that crashed or never started (for example, from exhausted LLM credits) just spun until it timed out, with no indication of what went wrong. Failed runs now show a specific reason (out of credits, rate limited, provider error, or never started) with a “Try again” action, and the failure timeout was cut from 12 minutes to 5.

• Playground and online scoring: Sonnet 5 and Fable 5 no longer error out — These models were missing from the capability list, so the Playground kept showing temperature/Top P sliders and sending those parameters, which the models reject. The sliders are now hidden and the parameters are no longer sent, matching the behavior already in place for Opus 4.7/4.8.

• Azure-hosted OpenAI model prices now load correctly — Model prices for the azure/* model family (gpt-4o, gpt-5, codex-mini, and others) were silently failing to load because of a provider-mapping gap, causing cost to show as unavailable for these models.

• Experiment and Playground traces now show up in scoped “Go to logs” views — Traces created by experiment runs and Playground runs were sometimes missing from their respective “Go to logs” tables even though the traces existed, due to a visibility-filter mismatch. These views now correctly scope by the experiment or run itself.

• Playground: dataset and metric selection redesigned — The “Run experiment” dialog is replaced by inline dataset and metrics dropdowns in the Playground header, so you can change either independently without reopening a modal. Metrics can also be created and edited in place.

• Permission gating extended to more pages — Users with the Annotator workspace role no longer see Agent Playground, Online Evaluation, or Alerts in the sidebar. Prompt Library view/edit access is now governed by the same permission system, at both the UI and API level.

• opik import --to-workspace for cross-workspace imports — The opik import CLI command accepts a --to-workspace option to import exported data into a different workspace than the one it was exported from, without needing to restructure the export directory as a workaround.

• opik migrate dataset --exclude-experiments — This flag skips migrating a dataset’s experiments and optimizations, migrating only the dataset and its version history. Useful for large datasets where the experiment/optimization cascade isn’t needed.

• Fixed intermittent errors loading prompt and dataset versions under load — A query pattern that caused MySQL to materialize large temporary tables could fail outright under load. Prompt and dataset version lookups are now rewritten to avoid the issue.