Troubleshooting

This guide covers common troubleshooting scenarios for self-hosted Opik deployments.

Common Issues

ClickHouse Migration Failures: Missing Cluster Macro

Problem Description

Opik requires ClickHouse to be configured with cluster macros, even for single-node deployments. Opik migrations use the ON CLUSTER '{cluster}' clause to ensure DDL operations execute consistently across all nodes in a cluster.

If the {cluster} macro is not configured in your ClickHouse instance, migrations will fail with the following error:

Code: 139. DB::Exception: No macro 'cluster' in config.

Symptoms:

  • Backend fails to start or enters CrashLoopBackOff state
  • Migration errors appear in backend logs
  • Error message: Code: 139. DB::Exception: No macro 'cluster' in config.

Automatic Configuration

Opik Helm Chart and Docker Compose deployments automatically configure the required cluster macros. If you’re using Opik’s provided deployment configurations, you should not encounter this issue.

Manual Configuration Required

If you’re running your own ClickHouse instance (not using Opik’s Helm chart or Docker Compose), you need to configure the cluster macros yourself.

1. Add Cluster Macro Configuration

Add the {cluster} macro to your ClickHouse configuration file. The location depends on your ClickHouse installation:

For standard ClickHouse installations:

Add the macros to /etc/clickhouse-server/config.d/macros.xml (or your equivalent config directory):

1<clickhouse>
2    <macros>
3        <cluster>single_node_cluster</cluster>
4        <shard>1</shard>
5        <replica>clickhouse</replica>
6    </macros>
7</clickhouse>

Note: For single-node setups, you can use any value for <cluster>. The value single_node_cluster is just an example. For multi-node clusters, use your actual cluster name that matches your <remote_servers> configuration.

2. Restart ClickHouse

After adding the configuration, restart ClickHouse for the changes to take effect:

3. Verify Configuration

You can verify the macro is configured by connecting to ClickHouse and running:

1SELECT * FROM system.macros WHERE macro = 'cluster';

You should see a row with the macro name and value.

4. Retry Migrations

After restarting ClickHouse, retry the backend deployment or migration. The backend should automatically retry after ClickHouse is ready.

Fresh Multi-Replica Install Migration Failures

Problem Description

A brand-new Opik install on a ClickHouse cluster with 2 or more replicas cannot complete its analytics migrations. The backend never starts (CrashLoopBackOff), and the migration step fails with errors such as:

Code: 60. DB::Exception: Could not find table: <table_name>. (UNKNOWN_TABLE)
Code: 81. DB::Exception: Database opik does not exist. (UNKNOWN_DATABASE)

opik is the default analytics database name (ANALYTICS_DB_DATABASE_NAME); substitute your own if you overrode it.

Symptoms:

  • Fresh install only (no existing data); backend in CrashLoopBackOff
  • Migration errors reference a table or the opik database that is missing on one replica
  • One replica holds the opik database and tables while another has few or none

Cause

Opik’s earliest analytics migrations predate cluster-aware DDL — they do not use ON CLUSTER '{cluster}' — so they create the opik database and base tables on a single replica only. Later, cluster-aware migrations fan out to every replica and fail on the one that never received that base schema. The ClickHouse operator only copies schema to a replica when it joins the cluster (a scale-up), not during a migration.

Resolution

Install High-Availability deployments in two phases so the schema exists before the extra replicas join:

1

Install (or reset to) clickhouse.replicasCount: 1.

2

Wait until opik-backend is Ready — all migrations are applied on the single replica.

3

Raise clickhouse.replicasCount (e.g. to 2) and upgrade. The operator provisions the new replica and copies the fully-migrated schema to it.

Single-replica installs and upgrades of already-migrated multi-replica clusters are unaffected. Cluster-aware migrations use ON CLUSTER '{cluster}', so once the base schema is present on all replicas, later migrations stay consistent automatically.

If a fresh multi-replica attempt was torn down, the dropped tables can leave orphaned /clickhouse/tables/... ZooKeeper paths that cause REPLICA_ALREADY_EXISTS on a retry. Since a fresh install has no data to preserve, recovery only needs those orphaned paths cleared before re-running — the full replica-restore rebuild in ClickHouse Zookeeper Metadata Loss is for recovering existing data and isn’t needed here.

ClickHouse Zookeeper Metadata Loss

Problem Description

If Zookeeper loses the metadata paths for ClickHouse tables, you will see coordination exceptions in the ClickHouse logs and potentially in the opik-backend service logs. These errors indicate that Zookeeper cannot find table metadata paths.

Symptoms:

Error messages appearing in ClickHouse logs and propagating to opik-backend service:

Code: 999. Coordination::Exception: Coordination error: No node, path /clickhouse/tables/0/default/DATABASECHANGELOG/log. (KEEPER_EXCEPTION)

This indicates that Zookeeper has lost the metadata paths for one or more ClickHouse tables.

Resolution Steps

Follow these steps to restore ClickHouse table metadata in Zookeeper:

1. Clean Zookeeper Paths (If Needed)

If only some table paths are missing in Zookeeper, you’ll need to delete the existing paths manually. Connect to the Zookeeper pod and use the Zookeeper CLI:

$# Connect to Zookeeper pod
$kubectl exec -it cometml-production-opik-zookeeper-0 -- zkCli.sh -server localhost:2181
$
$# Delete all ClickHouse table paths
$deleteall /clickhouse/tables

Warning: This operation removes all table metadata from Zookeeper. Proceed with caution.

2. Restart ClickHouse

Restart the ClickHouse pods so they become aware that Zookeeper no longer has the metadata:

$kubectl rollout restart statefulset/chi-opik-clickhouse-cluster-0-0
3. Restore Replica Definitions

Connect to the first ClickHouse replica and restore the replica definitions for each table:

$# Connect to the first ClickHouse replica
$kubectl exec -it chi-opik-clickhouse-cluster-0-0-0 -- clickhouse-client

Important: The Opik schema name is typically opik but may vary depending on your installation. Before proceeding, verify your schema name by running SHOW DATABASES; in ClickHouse and identifying the Opik database. Use that database name in all subsequent commands.

Run the SYSTEM RESTORE REPLICA command for each table:

1-- Restore system tables
2SYSTEM RESTORE REPLICA default.DATABASECHANGELOG;
3SYSTEM RESTORE REPLICA default.DATABASECHANGELOGLOCK;
4
5-- Verify your Opik database name
6SHOW DATABASES;
7
8-- List all Opik tables (replace 'opik' with your actual schema name if different)
9USE opik;
10SHOW TABLES;
11
12-- Restore each Opik table
13SYSTEM RESTORE REPLICA opik.attachments;
14SYSTEM RESTORE REPLICA opik.automation_rule_evaluator_logs;
15SYSTEM RESTORE REPLICA opik.comments;
16SYSTEM RESTORE REPLICA opik.dataset_items;
17SYSTEM RESTORE REPLICA opik.experiment_items;
18SYSTEM RESTORE REPLICA opik.experiments;
19SYSTEM RESTORE REPLICA opik.feedback_scores;
20SYSTEM RESTORE REPLICA opik.guardrails;
21SYSTEM RESTORE REPLICA opik.optimizations;
22SYSTEM RESTORE REPLICA opik.project_configurations;
23SYSTEM RESTORE REPLICA opik.spans;
24SYSTEM RESTORE REPLICA opik.traces;
25SYSTEM RESTORE REPLICA opik.trace_threads;
26SYSTEM RESTORE REPLICA opik.workspace_configurations;

Note: The exact list of tables may vary depending on your Opik version. Use the SHOW DATABASES; or \d command to list all tables in your database and restore each one.

4. Restart ClickHouse Again

Restart ClickHouse again to ensure it:

  • Re-establishes connections to Zookeeper
  • Verifies and synchronizes the newly restored metadata
  • Automatically resumes normal replication operations
$kubectl rollout restart statefulset/chi-opik-clickhouse-cluster-0-0
5. Validate the Recovery

After the restart completes, verify that the replica status is healthy:

1-- Check table creation
2SHOW CREATE TABLE opik.attachments;
3
4-- Verify replica status
5SELECT table, is_readonly, replica_is_active, zookeeper_exception
6FROM system.replicas;

Expected Results:

  • is_readonly = 0 (table is writable)
  • replica_is_active = 1 (replica is active)
  • zookeeper_exception = '' (no exceptions)

You can also verify from the Zookeeper side:

$# Connect to Zookeeper CLI
$kubectl exec -it cometml-production-opik-zookeeper-0 -- zkCli.sh -server localhost:2181
$
$# List tables (example path - adjust for your database name)
$ls /clickhouse/tables/0/<database_name>/<table_name>

ClickHouse TOO_MANY_PARTS Errors and Stuck Merges

Problem Description

Under sustained high-volume ingestion, span/trace batch inserts may start failing with HTTP 500s while ClickHouse rejects new parts:

Code: 252. DB::Exception: Too many parts (N) in table 'opik.spans'.
Merges are processing significantly slower than inserts:
While executing WaitForAsyncInsert. (TOO_MANY_PARTS)

Clients calling POST /api/v1/private/spans/batch (or /traces/batch) receive 500s. The active part count for the table has exceeded ClickHouse’s parts_to_throw_insert threshold (default 3000) and is not draining.

Symptoms:

  • 500s on the span/trace batch endpoints; TOO_MANY_PARTS (Code: 252) in the opik-backend logs
  • A high and still-growing active-part count for spans and/or traces
  • If it persists, the opik-backend ClickHouse connection pool can exhaust (ConnectionRequestTimeoutException), amplifying the impact

This async-insert behavior applies to all ClickHouse-backed writes — individual create/update calls as well as the batch endpoints, and other entities (feedback scores, dataset items, experiments, …) — because the setting lives on the connection. Spans and traces, especially the batch endpoints, generate by far the most volume, so they hit the threshold first; the tuning below applies globally.

This error has two very different root causes that need different fixes. Always run the diagnosis below before taking action — the recovery for one will not help the other.

A production Opik runs ClickHouse as a replicated cluster, so the commands below use the cluster-wide form and two placeholders: {cluster} (the cluster macro) and <database_name> (the analytics schema — default opik, set via ANALYTICS_DB_DATABASE_NAME). Substitute your own before running — find them with SELECT * FROM system.macros and SHOW DATABASES. On a single-node install, drop the clusterAllReplicas(...) wrapper (query the system.* table directly) and the ON CLUSTER '{cluster}' clause.

Diagnose first

1-- Active parts per node and table: is the count high and growing?
2SELECT hostName() AS host, table, count() AS active_parts
3FROM clusterAllReplicas('{cluster}', system.parts)
4WHERE active AND database = '<database_name>'
5GROUP BY host, table ORDER BY active_parts DESC;
6
7-- Is a merge stuck? Look for a queue entry with a rising num_tries and a repeating exception.
8SELECT hostName() AS host, database, table, type, num_tries, new_part_name, last_exception
9FROM clusterAllReplicas('{cluster}', system.replication_queue)
10WHERE database = '<database_name>'
11ORDER BY num_tries DESC;
12
13-- How many merges are actually running per node vs. the pool size?
14SELECT hostName() AS host, count() AS running_merges
15FROM clusterAllReplicas('{cluster}', system.merges)
16GROUP BY host;
17
18-- Replica health (rules out the ZooKeeper-metadata-loss scenario below).
19SELECT hostName() AS host, table, is_readonly, replica_is_active, zookeeper_exception
20FROM clusterAllReplicas('{cluster}', system.replicas);
  • Cause A — fragmentation (merges can’t keep up). Merges complete normally and quickly, running_merges is healthy, but parts are created faster than they merge. system.replication_queue shows no entry stuck with a high num_tries. This is driven by async-insert flush frequency under high concurrency.
  • Cause B — a stuck merge blocking the queue. running_merges is near zero (despite a large background_pool_size) and one replication_queue entry has a high, climbing num_tries with a repeating last_exception — commonly Code: 76 ... CANNOT_OPEN_FILE on corrupt/missing source-part files. That one poison entry jams the scheduler so nothing else merges and parts cannot drain, regardless of CPU or disk.

If system.replicas shows Code: 999 / KEEPER_EXCEPTION / No node, you have a different problem — see ClickHouse Zookeeper Metadata Loss. SYSTEM RESTORE REPLICA is for that scenario and will not clear a stuck merge.

Cause A resolution — tune async insert

Opik applies its async-insert settings on the ClickHouse connection (via custom_http_params in ANALYTICS_DB_QUERY_PARAMETERS), so they apply to every insert. The shipped defaults favor freshness over batching:

async_insert=1
wait_for_async_insert=1
async_insert_busy_timeout_min_ms=100
async_insert_busy_timeout_max_ms=250
async_insert_use_adaptive_busy_timeout=1

With async_insert=1, each server-side flush becomes a new part. A short busy-timeout window (100–250 ms) means frequent flushes, and under high concurrency this creates many small parts that merges must keep up with.

Because Opik pins the busy-timeout settings on the connection, tuning async_insert_busy_timeout_* in a ClickHouse server profile or user settings has no effect — the connection value wins. Change them through Opik configuration instead (below).

To reduce fragmentation for high-volume deployments, widen the flush window (fewer, larger parts) via these opik-backend environment variables:

$ANALYTICS_DB_ASYNC_INSERT_BUSY_TIMEOUT_MAX_MS=2000   # ceiling of the adaptive flush window (ms); e.g. 1000–3000
$ANALYTICS_DB_ASYNC_INSERT_BUSY_TIMEOUT_MIN_MS=1000   # floor of the adaptive flush window (ms); keep below max
$ANALYTICS_DB_ASYNC_INSERT_MAX_DATA_SIZE=52428800     # buffered bytes that force a flush; larger = fewer parts

Each of these env vars applies when set — it overrides the value in your ANALYTICS_DB_QUERY_PARAMETERS chain, or is added if the chain omits it — and leaves your value untouched when unset. When unset, the busy-timeout knobs fall back to the shipped chain defaults (100 / 250 ms) and async_insert_max_data_size to your ClickHouse server / user-profile value (Opik doesn’t pin it). To change any other connection setting, override the whole ANALYTICS_DB_QUERY_PARAMETERS string, keeping every existing key and adjusting only what you need.

Widening the flush window trades a little ingestion latency and buffer memory — rows become queryable up to max_ms later — for far fewer parts, a good trade for high-volume observability data. Under sustained high load, flushes are size-triggered anyway, so most of the added latency falls on quieter periods. After changing these, restart the opik-backend and watch the active part count and system.asynchronous_inserts, dialing max_ms between 1000–3000 ms to taste. New inserts will fragment less; an existing backlog still needs to merge down (it will, once the insert rate no longer outpaces merges).

Because ingestion is synchronous (wait_for_async_insert=1), a higher max_ms also raises the worst-case response time of the batch and create/update endpoints. Make sure your client request timeouts comfortably exceed max_ms: the official Opik SDKs already use a generous read timeout, so typical values are safe, but custom or direct HTTP clients should be checked and increased if needed so callers don’t time out before a successful insert returns.

Cause B resolution — unblock the stuck merge

Work from least to most invasive: the threshold bump (step 1) and diagnosis (steps 2–3) are non-destructive; DETACH (step 4) is recoverable but data-affecting; the ZooKeeper edit (step 6) is the last resort. Run the commands against the table your diagnosis flagged — replace <table> with spans or traces.

1

Restore ingestion immediately (reversible, no data loss). Temporarily raise the throw threshold so inserts succeed while you fix the root cause:

1ALTER TABLE <database_name>.<table> ON CLUSTER '{cluster}' MODIFY SETTING parts_to_throw_insert = 20000, parts_to_delay_insert = 20000;

On a Distributed deployment, target the underlying per-shard ReplacingMergeTree table, not the Distributed proxy. This is a safe, reversible stopgap — no data is deleted — so revert to the defaults once the backlog has drained.

2

Confirm replicas are healthy (is_readonly = 0, replica_is_active = 1, zookeeper_exception = ''). If you see Keeper No node errors, switch to ClickHouse Zookeeper Metadata Loss.

3

Identify the failing merge and its corrupt source parts from the system.replication_queue.last_exception values (the CANNOT_OPEN_FILE messages name the offending part directories).

4

Try to recover the corrupt source parts. If a part is still intact on another replica, DETACH the corrupt local copy (never DROP) — ClickHouse then re-fetches the good copy from a healthy replica, which lets the blocked merge complete. Detached parts are preserved under the table’s detached/ directory.

1ALTER TABLE <database_name>.<table> DETACH PART 'all_1_100_5';
2-- repeat for each corrupt part named in the exception

Data-affecting, expert operation — do it with ClickHouse expertise on hand. On a ReplicatedMergeTree the queued merge entry references its source parts by name, so DETACH does not by itself remove that entry: it only helps when the part is intact elsewhere and can be re-fetched. If the part is corrupt on all replicas there is nothing to re-fetch — the entry keeps retrying and you must proceed to step 6 to clear it (and removing the part also drops its rows, a small bounded window for observability data). Always DETACH (recoverable), never DROP.

5

Re-check the queue. If the stuck entry is gone and merges resume, the backlog will drain on its own. A SYSTEM RESTART REPLICA <database_name>.<table> can help a node re-read its queue.

6

Last resort — remove the stuck queue entry from ZooKeeper. When re-fetch/DETACH can’t clear it (steps 4–5), removing the poison entry directly is the reliable fix — but it is dangerous, so do it only after 4–5 and ideally with ClickHouse expertise on hand. Delete the specific stuck queue-XXXXXXXXXX node under every replica path, then restart the replicas:

/clickhouse/tables/<shard>/<database_name>/<table>/replicas/<replica>/queue/queue-XXXXXXXXXX

Editing ZooKeeper directly is dangerous and can corrupt replication if the wrong node is removed. Delete only the exact stuck entry, on all replicas, and prefer the DETACH PART path above whenever possible.

7

Verify recovery and revert. Watch the active-part count fall, then restore parts_to_throw_insert / parts_to_delay_insert to their defaults.

What does not clear a stuck merge: SYSTEM RESTART REPLICA on its own (it only resets num_tries and re-attempts the same entry), SYSTEM STOP/START MERGES (the entry survives the pause), and KILL MUTATION (a merge is not a mutation). Use the DETACH PART path instead.

Prevention

  • High-volume ingestion: raise async_insert_busy_timeout_max_ms (Cause A above) before scaling ingestion up, and monitor part counts.
  • ClickHouse major-version upgrades: a replica can transiently go read-only during a rolling upgrade of replicated tables, which may leave part files inconsistent and schedule a merge that later fails. Quiesce or throttle ingestion during the upgrade, and confirm all replicas are healthy (system.replicas) and system.replication_queue is clean before ramping ingestion back up.
  • Monitoring: alert on the cluster-wide active part count per table and on system.replication_queue entries with a rising num_tries, so a stuck merge is caught before it becomes TOO_MANY_PARTS.

Diagnostic Commands

Connecting to ClickHouse

Connect directly to ClickHouse pods for diagnostics:

$# Connect to first replica
$kubectl exec -it chi-opik-clickhouse-cluster-0-0-0 -- clickhouse-client
$
$# Connect to second replica (if running multiple replicas)
$kubectl exec -it chi-opik-clickhouse-cluster-0-1-0 -- clickhouse-client

Connecting to Zookeeper

Connect directly to Zookeeper pods:

$# Connect to Zookeeper pod
$kubectl exec -it cometml-production-opik-zookeeper-0 -- bash
$
$# Run Zookeeper client commands
$zkCli.sh -server localhost:2181

Common Zookeeper commands:

$# List tables in Zookeeper
$kubectl exec -it cometml-production-opik-zookeeper-0 -- \
>  zkCli.sh -server localhost:2181 ls /clickhouse/tables/0/opik
$
$# Remove a specific table from Zookeeper
$kubectl exec -it cometml-production-opik-zookeeper-0 -- \
>  zkCli.sh -server localhost:2181 \
>  deleteall /clickhouse/tables/0/opik/optimizations

Prevention and Best Practices

To avoid Zookeeper metadata loss issues:

  1. Regular Backups: Implement regular backups of ClickHouse data. See the Advanced ClickHouse Backup guide for details.

  2. Monitoring: Set up monitoring for Zookeeper health and ClickHouse replica status. Alert on zookeeper_exception in system.replicas.

  3. Resource Allocation: Ensure Zookeeper has adequate resources (CPU, memory, disk) to maintain metadata reliably.

  4. Persistent Storage: Use persistent volumes for Zookeeper to prevent data loss during pod restarts.

  5. Replica Validation: Regularly check replica status with the diagnostic queries above.

Getting Help

If you continue to experience issues after following this guide:

  1. Check the Opik GitHub Issues for similar problems
  2. Review ClickHouse and Zookeeper logs for additional error details
  3. Open a new issue on GitHub with:
    • Opik versions:
      • Backend version (opik-backend)
      • Frontend version (opik-frontend)
      • Helm chart version (if deployed via Helm)
    • ClickHouse version
    • Zookeeper version
    • Error logs from all services (ClickHouse, Zookeeper, opik-backend)
    • Steps taken to reproduce the issue