> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://www.comet.com/docs/opik/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://www.comet.com/docs/opik/_mcp/server. # ClickHouse Cold-Tier Migration > **Important:** If you're using or considering the enterprise version of Opik or Comet, reach out to [Sales@comet.com](mailto:Sales@comet.com) to access the enterprise deployment documentation. Opik's ClickHouse can use a tiered storage policy: hot data on a local disk and cold data on S3, fronted by a local read-through cache. When you enable the cold tier, the Helm chart runs a migration job that makes the transition safe and automatic. This matters most on a cluster running **more than one ClickHouse replica**, where enabling the cold tier can otherwise wedge a replica for several minutes. This page covers enabling the cold tier on an existing multi-replica cluster. For running multiple ClickHouse replicas in general — including the two-phase install required for a fresh HA deployment — see [Scaling Opik](/self-host/scaling#replication-strategy). ## Why This Is Needed Enabling the cold tier renders a new ClickHouse `storage.xml` that references the cache path (`/var/cache/clickhouse_s3` by default). On a multi-replica cluster, the ClickHouse operator can push this new configuration to a pod **before** it rebuilds that pod's StatefulSet to add the cache volume and mount. When that happens the pod fails to start: ``` mkdir /var/cache/clickhouse_s3: Permission denied ``` The pod enters `CrashLoopBackOff` and self-heals on its own once the operator rebuilds the StatefulSet — but that passive recovery takes roughly 7 minutes per replica. The migration job automates the recovery so the cold-tier enablement completes quickly and without manual intervention. ## Upgrade Instructions To enable the cold tier, set `clickhouse.tieredStorage.enabled: true` and provide the cold bucket configuration under `clickhouse.tieredStorage`: ```yaml clickhouse: tieredStorage: enabled: true cold: s3: endpoint: "https://.s3..amazonaws.com" region: "" ``` The migration job runs automatically whenever the cold tier is enabled. It is controlled by: ```yaml clickhouse: tieredStorage: coldTierMigration: enabled: true ``` `coldTierMigration.enabled` defaults to `true`, so upgrading a release that **already** has `clickhouse.tieredStorage.enabled: true` activates this job on the next upgrade without any action on your part. It only does work when the cold tier is enabled, and is a no-op when every replica already has the cache volume — so it is safe to leave on across upgrades, though it does add a post-upgrade hook to the release. Because it only bounces the operator (never deletes ClickHouse StatefulSets, pods, or data), it is safe on an existing release. To opt out **without** disabling the cold tier: ```yaml clickhouse: tieredStorage: coldTierMigration: enabled: false ``` The migration job is namespace-scoped to the release namespace, so it is **incompatible with `altinity-clickhouse-operator.namespaceOverride`** — with the cold tier on (and `coldTierMigration.enabled` defaulting to `true`), the chart will hard-fail at render if both are set. Either remove `namespaceOverride`, or set `coldTierMigration.enabled: false` and recover manually. The same applies to **`altinity-clickhouse-operator.nameOverride` / `fullnameOverride`**: the job locates the operator by its default `app.kubernetes.io/name`, so a renamed subchart would leave the bounce silently skipped. The chart hard-fails at render instead — remove the override, or set `coldTierMigration.enabled: false`. ### How the Migration Works The migration runs as a post-upgrade hook (or ArgoCD `PostSync` hook) and is **data-preserving**. It recovers the cluster by bouncing the ClickHouse operator — it never deletes ClickHouse StatefulSets, pods, or data: #### Detect Check whether any ClickHouse StatefulSet is still missing the cold-tier cache volume. If none are, the job exits as a no-op — replicas that are merely mid-roll do not count as missing, so a routine upgrade of an already-migrated release never bounces the operator. #### Bounce the operator Restart the altinity ClickHouse operator Deployment (`kubectl rollout restart`). A fresh operator re-reconciles from scratch. No pod is ever deleted. #### Rebuild with the cache volume On its first reconcile pass, the fresh operator rebuilds every ClickHouse StatefulSet **with** the cache volume — the affected pod remounts and recovers. #### Wait for readiness Wait until every ClickHouse pod is `Ready` and no StatefulSet is missing the cache volume. The job itself only restarts the operator — it never deletes a ClickHouse StatefulSet, pod, or PersistentVolumeClaim. Recovery is far faster than the \~7 min/replica passive self-heal, though the operator's own rebuild of the affected StatefulSet may still briefly restart the wedged pod. ### Configuration Reference All keys are under `clickhouse.tieredStorage.coldTierMigration`: | Value | Default | Description | | ------------------------------------------------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `clickhouse.tieredStorage.coldTierMigration.enabled` | `true` | Run the migration when the cold tier is enabled. Set `false` to opt out without disabling the cold tier. | | `clickhouse.tieredStorage.coldTierMigration.image` | `alpine/kubectl:1.35.0` | Image used by the migration job. | | `clickhouse.tieredStorage.coldTierMigration.pollTimeoutSeconds` | `1200` | Total budget for the run (bounce the operator, then wait for all replicas Ready). | | `clickhouse.tieredStorage.coldTierMigration.activeDeadlineSeconds` | `1320` | Kubernetes hard cap on the Job. Must be greater than `pollTimeoutSeconds`. | | `clickhouse.tieredStorage.coldTierMigration.serviceAccountName` | `""` | Use an externally-managed ServiceAccount instead of the chart-created one. When set, the account must already have `get/list/watch` on `statefulsets` and `pods`, and `get/patch` on the operator Deployment. | The migration ServiceAccount is granted `get`/`patch` on **only** the operator Deployment (by `resourceNames`) plus read-only access to StatefulSets and pods. It cannot delete any pod. Note that the Role and RoleBinding use `before-hook-creation` and so are **not** removed when you later set `coldTierMigration.enabled: false` — they persist on the cluster until deleted manually. Leave `serviceAccountName` empty to use the chart-created, Job-scoped account; if you supply your own, use a dedicated ServiceAccount rather than a shared one, since anything else using that account inherits this Role. ## Single-Replica Clusters On a single-replica cluster the cold-tier enablement is briefly disruptive and would self-heal on its own in roughly 6–7 minutes. The migration job still runs and recovers the replica the same way (bounce the operator so it rebuilds the StatefulSet with the cache volume), typically faster than waiting for the passive self-heal. No action is required either way. If the replica already has the cache volume, the job is a no-op. ## Need Help? If you encounter issues enabling the cold tier, please check the [troubleshooting guide](/self-host/troubleshooting) or reach out for support.