Enabling the ClickHouse Cold Tier

Important: If you’re using or considering the enterprise version of Opik or Comet, reach out to Sales@comet.com to access the enterprise deployment documentation.

Opik’s ClickHouse can use a tiered storage policy: hot data on a local disk and cold data on S3, fronted by a local read-through cache. When you enable the cold tier, the Helm chart runs a migration job that makes the transition safe and automatic. This matters most on a cluster running more than one ClickHouse replica, where enabling the cold tier can otherwise wedge a replica for several minutes.

This page covers enabling the cold tier on an existing multi-replica cluster. For running multiple ClickHouse replicas in general — including the two-phase install required for a fresh HA deployment — see Scaling Opik.

Why This Is Needed

Enabling the cold tier renders a new ClickHouse storage.xml that references the cache path (/var/cache/clickhouse_s3 by default). On a multi-replica cluster, the ClickHouse operator can push this new configuration to a pod before it rebuilds that pod’s StatefulSet to add the cache volume and mount. When that happens the pod fails to start:

mkdir /var/cache/clickhouse_s3: Permission denied

The pod enters CrashLoopBackOff and self-heals on its own once the operator rebuilds the StatefulSet — but that passive recovery takes roughly 7 minutes per replica. The migration job automates the recovery so the cold-tier enablement completes quickly and without manual intervention.

Upgrade Instructions

To enable the cold tier, set clickhouse.tieredStorage.enabled: true and provide the cold bucket configuration under clickhouse.tieredStorage:

clickhouse:
  tieredStorage:
    enabled: true
    cold:
      s3:
        endpoint: "https://<bucket>.s3.<region>.amazonaws.com"
        region: "<region>"

The migration job runs automatically whenever the cold tier is enabled. It is controlled by:

clickhouse:
  tieredStorage:
    coldTierMigration:
      enabled: true

coldTierMigration.enabled defaults to true, so upgrading a release that already has clickhouse.tieredStorage.enabled: true activates this job on the next upgrade without any action on your part. It only does work when the cold tier is enabled, and is a no-op when every replica already has the cache volume — so it is safe to leave on across upgrades, though it does add a post-upgrade hook to the release. Because it only bounces the operator (never deletes ClickHouse StatefulSets, pods, or data), it is safe on an existing release. To opt out without disabling the cold tier:

clickhouse:
  tieredStorage:
    coldTierMigration:
      enabled: false

The migration job is namespace-scoped to the release namespace, so it is incompatible with altinity-clickhouse-operator.namespaceOverride — with the cold tier on (and coldTierMigration.enabled defaulting to true), the chart will hard-fail at render if both are set. Either remove namespaceOverride, or set coldTierMigration.enabled: false and recover manually.

The same applies to altinity-clickhouse-operator.nameOverride / fullnameOverride: the job locates the operator by its default app.kubernetes.io/name, so a renamed subchart would leave the bounce silently skipped. The chart hard-fails at render instead — remove the override, or set coldTierMigration.enabled: false.

How the Migration Works

The migration runs as a post-upgrade hook (or ArgoCD PostSync hook) and is data-preserving. It recovers the cluster by bouncing the ClickHouse operator — it never deletes ClickHouse StatefulSets, pods, or data:

Detect

Check whether any ClickHouse StatefulSet is still missing the cold-tier cache volume. If none are, the job exits as a no-op — replicas that are merely mid-roll do not count as missing, so a routine upgrade of an already-migrated release never bounces the operator.

Bounce the operator

Restart the altinity ClickHouse operator Deployment (kubectl rollout restart). A fresh operator re-reconciles from scratch. No pod is ever deleted.

Rebuild with the cache volume

On its first reconcile pass, the fresh operator rebuilds every ClickHouse StatefulSet with the cache volume — the affected pod remounts and recovers.

Wait for readiness

Wait until every ClickHouse pod is Ready and no StatefulSet is missing the cache volume.

The job itself only restarts the operator — it never deletes a ClickHouse StatefulSet, pod, or PersistentVolumeClaim. Recovery is far faster than the ~7 min/replica passive self-heal, though the operator’s own rebuild of the affected StatefulSet may still briefly restart the wedged pod.

Configuration Reference

All keys are under clickhouse.tieredStorage.coldTierMigration:

Value	Default	Description
`clickhouse.tieredStorage.coldTierMigration.enabled`	`true`	Run the migration when the cold tier is enabled. Set `false` to opt out without disabling the cold tier.
`clickhouse.tieredStorage.coldTierMigration.image`	`alpine/kubectl:1.35.0`	Image used by the migration job.
`clickhouse.tieredStorage.coldTierMigration.pollTimeoutSeconds`	`1200`	Total budget for the run (bounce the operator, then wait for all replicas Ready).
`clickhouse.tieredStorage.coldTierMigration.activeDeadlineSeconds`	`1320`	Kubernetes hard cap on the Job. Must be greater than `pollTimeoutSeconds`.
`clickhouse.tieredStorage.coldTierMigration.serviceAccountName`	`""`	Use an externally-managed ServiceAccount instead of the chart-created one. When set, the account must already have `get/list/watch` on `statefulsets` and `pods`, and `get/patch` on the operator Deployment.

The migration ServiceAccount is granted get/patch on only the operator Deployment (by resourceNames) plus read-only access to StatefulSets and pods. It cannot delete any pod.

Note that the Role and RoleBinding use before-hook-creation and so are not removed when you later set coldTierMigration.enabled: false — they persist on the cluster until deleted manually. Leave serviceAccountName empty to use the chart-created, Job-scoped account; if you supply your own, use a dedicated ServiceAccount rather than a shared one, since anything else using that account inherits this Role.

Single-Replica Clusters

On a single-replica cluster the cold-tier enablement is briefly disruptive and would self-heal on its own in roughly 6–7 minutes. The migration job still runs and recovers the replica the same way (bounce the operator so it rebuilds the StatefulSet with the cache volume), typically faster than waiting for the passive self-heal. No action is required either way. If the replica already has the cache volume, the job is a no-op.

Need Help?

If you encounter issues enabling the cold tier, please check the troubleshooting guide or reach out for support.