> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://www.comet.com/docs/opik/llms.txt. # Redis Sentinel By default, the Opik backends connect directly to a single Redis endpoint through `REDIS_URL`. Enable Redis Sentinel when you want them to discover the active Redis master and reconnect after Sentinel promotes a replica. Opik configures the clients only; you must provide and operate the Redis master, replicas, and Sentinel processes. > **Warning** > > Configure Sentinel on **both** the Java backend and the Python backend, which runs the RQ workers. A service left on a fixed Redis address keeps writing to the old master after a failover: those writes are lost when Sentinel demotes it, and later ones fail with `READONLY`. ## Prerequisites Before enabling Sentinel mode, make sure that: * A Redis master and at least one replica are running and synchronized. * All Sentinel processes monitor the same master set name. * Sentinel-announced Redis and Sentinel addresses are reachable from both backends. * Authentication and TLS settings are consistent across the topology. * Production deployments use at least three Sentinel processes in independent failure domains. A typical three-Sentinel deployment uses a quorum of `2`. See the [Redis Sentinel documentation](https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/) for Redis-side deployment and quorum guidance. ## Configure the backends The example below uses: * `mymaster` as the Sentinel master set name * `sentinel-1:26379` as the initial Sentinel seed * `sentinel-2:26379` and `sentinel-3:26379` as additional seeds * `redis-password` for Redis data-node authentication * `sentinel-password` for Sentinel authentication * Redis database `0` #### Docker Compose Add the same overrides to both backends in `deployment/docker-compose/docker-compose.override.yaml`: ```yaml x-redis-sentinel: &redis-sentinel REDIS_URL: "redis://:redis-password@sentinel-1:26379/0" OPIK_REDIS_SENTINEL_ENABLED: "true" OPIK_REDIS_SENTINEL_MASTER_NAME: "mymaster" OPIK_REDIS_SENTINEL_NODES: "redis://sentinel-2:26379,redis://sentinel-3:26379" OPIK_REDIS_SENTINEL_PASSWORD: "sentinel-password" services: backend: environment: *redis-sentinel python-backend: environment: *redis-sentinel ``` Apply the change from the `deployment/docker-compose` directory: ```bash docker compose up -d backend python-backend ``` #### Kubernetes / Helm Add the same environment variables to both backends in your Helm values file: ```yaml component: backend: env: REDIS_URL: "redis://:redis-password@sentinel-1:26379/0" OPIK_REDIS_SENTINEL_ENABLED: "true" OPIK_REDIS_SENTINEL_MASTER_NAME: "mymaster" OPIK_REDIS_SENTINEL_NODES: "redis://sentinel-2:26379,redis://sentinel-3:26379" OPIK_REDIS_SENTINEL_PASSWORD: "sentinel-password" python-backend: env: REDIS_URL: "redis://:redis-password@sentinel-1:26379/0" OPIK_REDIS_SENTINEL_ENABLED: "true" OPIK_REDIS_SENTINEL_MASTER_NAME: "mymaster" OPIK_REDIS_SENTINEL_NODES: "redis://sentinel-2:26379,redis://sentinel-3:26379" OPIK_REDIS_SENTINEL_PASSWORD: "sentinel-password" ``` Apply the change: ```bash helm upgrade opik opik/opik -n opik -f values.yaml ``` ## How `REDIS_URL` is interpreted When Sentinel mode is disabled, `REDIS_URL` remains the direct single-node Redis URL. When Sentinel mode is enabled, the same URL is interpreted as follows: | URL part | Meaning in Sentinel mode | | ---------------------------------- | ----------------------------------------------------------------------------------------------------- | | Scheme (`redis://` or `rediss://`) | Connection scheme for the Sentinel topology | | Host and port | Initial Sentinel seed. The port is required, since Sentinel can listen on any port (commonly `26379`) | | Username and password | Credentials for Redis data nodes, not for Sentinel | | Path | Redis database selected on the discovered master | Sentinel authentication is configured independently with `OPIK_REDIS_SENTINEL_USERNAME` and `OPIK_REDIS_SENTINEL_PASSWORD`. For ACL-based authentication, use separate credentials for the data nodes and Sentinel: ```bash REDIS_URL="redis://redis-user:redis-password@sentinel-1:26379/0" OPIK_REDIS_SENTINEL_USERNAME="sentinel-user" OPIK_REDIS_SENTINEL_PASSWORD="sentinel-password" ``` If Sentinel does not require authentication, omit the Sentinel username and password variables. ## Configuration reference Configure these environment variables, with the same values, on both the Java backend and the Python backend. Both read them the same way, and both validate them at startup whenever Sentinel is enabled, refusing to start on an invalid value. Set to an empty value, `OPIK_REDIS_SENTINEL_ENABLED` means disabled and the nodes, username and password are treated as unset; any other empty variable is invalid. Set boolean variables to `true` or `false`. | Environment variable | Default | Description | | ------------------------------------------ | -------------------------------: | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `REDIS_URL` | `redis://:opik@localhost:6379/0` | Direct Redis URL in single-node mode. In Sentinel mode, its host and port identify the initial Sentinel seed, while its credentials and database apply to Redis data nodes. | | `OPIK_REDIS_SENTINEL_ENABLED` | `false` | Set to `true` to enable Sentinel-based master discovery, or `false` to keep single-node mode. | | `OPIK_REDIS_SENTINEL_MASTER_NAME` | Empty | Sentinel master set name. Required when Sentinel mode is enabled. | | `OPIK_REDIS_SENTINEL_NODES` | Empty | Comma-separated additional Sentinel seed addresses, each with a host, a port and the same `redis://` or `rediss://` scheme as `REDIS_URL`. Do not embed credentials; use the Sentinel username and password variables. | | `OPIK_REDIS_SENTINEL_USERNAME` | Empty | Username used to authenticate to Sentinel nodes when Sentinel ACLs are enabled. | | `OPIK_REDIS_SENTINEL_PASSWORD` | Empty | Password used to authenticate to Sentinel nodes. | | `OPIK_REDIS_SENTINEL_RETRY_ATTEMPTS` | `3` | Number of retry attempts for commands that fail during a topology change, between `0` and `20`. | | `OPIK_REDIS_SENTINEL_RETRY_BACKOFF_BASE` | `500ms` | Base of the full-jitter backoff: each retry waits a random time up to `min(base × 2^attempt, cap)`. | | `OPIK_REDIS_SENTINEL_RETRY_BACKOFF_CAP` | `2s` | Longest wait between retries. Must be at least the base. Keep the total retry time above your Sentinel promotion time, `down-after-milliseconds` plus the election. | | `OPIK_REDIS_SENTINEL_CHECK_SENTINELS_LIST` | `true` | Requires Sentinel discovery to report at least two Sentinel nodes. Set to `false` only for single-Sentinel development environments. | | `OPIK_REDIS_SENTINEL_CONNECT_TIMEOUT` | `10s` | Timeout for establishing Redis or Sentinel connections. | | `OPIK_REDIS_SENTINEL_TIMEOUT` | `5s` | Timeout for Redis command responses. | | `OPIK_REDIS_SENTINEL_SCAN_INTERVAL` | `2s` | Java backend only. Interval at which Redisson polls Sentinel for topology changes. The Python client needs no polling: it asks Sentinel for the master on every new connection and reconnects when the old master stops accepting writes. | Duration values accept forms such as `500ms`, `5s`, and `1m`. Timeouts and the scan interval must be between `100ms` and `5m`; the backoff base and cap between `1ms` and `5m`. > **Info** > > Sentinel mode cannot be combined with AWS IAM authentication (`redis.awsIamAuth`). ElastiCache and MemoryDB, where IAM authentication applies, do not offer Sentinel; point `REDIS_URL` at their primary endpoint in single-node mode instead. ## TLS Use the `rediss://` scheme in `REDIS_URL` and every address in `OPIK_REDIS_SENTINEL_NODES` when the Sentinel topology requires TLS. Opik does not expose separate connection schemes for Sentinel and Redis data nodes, so a topology that mixes plaintext Sentinel connections with TLS data-node connections, or the reverse, is not supported by this configuration. For TLS deployments, make sure Sentinel announces hostnames or addresses that: * Are reachable from both backends * Match the subject alternative names in the Redis server certificates ## Production guidance * Keep `OPIK_REDIS_SENTINEL_CHECK_SENTINELS_LIST=true` in production. * Provide multiple seed addresses through `OPIK_REDIS_SENTINEL_NODES` so backend startup does not depend on one Sentinel being reachable. * Place Sentinel processes in independent failure domains. * Ensure replicas are synchronized and eligible for promotion before testing failover. * Store Redis and Sentinel credentials in your deployment's secret manager rather than directly in a committed values or Compose file. * Remember that Redis replication is asynchronous. Sentinel improves availability but does not guarantee that every acknowledged write survives every failure. ## Verify the configuration ### Confirm Sentinel mode at startup Check the Java backend logs for: ```text Built redis config with address 'redis://sentinel-1:26379', database '0', sentinel 'true' ``` #### Docker Compose ```bash docker compose logs backend | grep "Built redis config" ``` #### Kubernetes / Helm ```bash kubectl logs -n opik deployment/opik-backend \ | grep "Built redis config" ``` If the log reports `sentinel 'false'`, Sentinel mode is not enabled in the Java backend process. The Python backend logs `Redis: using Sentinel master 'mymaster' through 3 sentinels, database 0` at startup. ### Check Redis health Query the backend health endpoint from inside the deployment or through a port-forward: #### Docker Compose ```bash docker compose exec backend \ curl -fsS "http://localhost:8080/health-check?name=redis" ``` #### Kubernetes / Helm In one terminal: ```bash kubectl port-forward -n opik deployment/opik-backend 8080:8080 ``` In another terminal: ```bash curl -i "http://localhost:8080/health-check?name=redis" ``` A healthy Redis topology returns HTTP `200`. > **Info** > > The Redis health check pings the master currently discovered through Sentinel, the only node that serves Opik traffic. A lost replica or Sentinel keeps the backend ready; the check can briefly report unhealthy while Sentinel promotes a new master. ### Validate failover > **Warning** > > Perform this procedure only in a development or staging environment. Triggering a failover changes the writable Redis node and can interrupt active requests. Keep both Opik backends running while performing this check. The commands below use `OPIK_REDIS_SENTINEL_USERNAME` and `OPIK_REDIS_SENTINEL_PASSWORD`. Set these variables in the shell running `redis-cli`. If the username is unset, the commands authenticate as the Redis `default` ACL user. #### Confirm the current master ```bash REDISCLI_AUTH="$OPIK_REDIS_SENTINEL_PASSWORD" redis-cli \ --user "${OPIK_REDIS_SENTINEL_USERNAME:-default}" \ -h sentinel-1 -p 26379 \ SENTINEL GET-MASTER-ADDR-BY-NAME mymaster ``` #### Trigger a controlled failover Or stop the current master according to your Redis operating procedure: ```bash REDISCLI_AUTH="$OPIK_REDIS_SENTINEL_PASSWORD" redis-cli \ --user "${OPIK_REDIS_SENTINEL_USERNAME:-default}" \ -h sentinel-1 -p 26379 \ SENTINEL FAILOVER mymaster ``` #### Wait for the promotion Repeat the first step until `SENTINEL GET-MASTER-ADDR-BY-NAME` reports the promoted replica. #### Check Opik still works Without restarting either backend, repeat a normal Opik operation that uses Redis and confirm it succeeds, and that both backend processes stayed up throughout the failover. Requests may briefly fail or time out while Sentinel promotes a replica and Redisson reconnects. Tune the retry and timeout settings to match your Sentinel failure-detection and failover timing. ## Troubleshooting | Symptom | Likely cause | Resolution | | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `Master node is undefined` or `GET-MASTER-ADDR-BY-NAME` returns no address | The configured master name does not match the Sentinel master set | Run `SENTINEL MASTERS` and set `OPIK_REDIS_SENTINEL_MASTER_NAME` to the reported name. | | Startup reports fewer than two Sentinel nodes | Sentinel discovery cannot reach enough nodes | Check network access and announced addresses. Use `OPIK_REDIS_SENTINEL_CHECK_SENTINELS_LIST=false` only for local single-Sentinel testing. | | Sentinel authentication succeeds but Redis commands fail with authentication errors | Sentinel credentials were also used as data-node credentials, or the reverse | Put Redis data-node credentials in `REDIS_URL` and Sentinel credentials in the dedicated Sentinel variables. | | TLS handshake or hostname verification fails | The URL scheme is incorrect, or Sentinel announces an address that does not match the certificate | Use the correct `redis://` or `rediss://` scheme and configure routable certificate-valid announced hostnames. | | Sentinel reports a new master, but Opik cannot reconnect | The promoted node's announced address is not reachable from the Java backend | Correct Redis and Sentinel announce settings, DNS, firewall rules, or network routes. | | The Python backend or RQ workers fail with `READONLY` after a failover | The Python backend still uses a fixed Redis address | Set the same `REDIS_URL` and `OPIK_REDIS_SENTINEL_*` variables on the Python backend as on the Java backend. | Useful Sentinel diagnostics: ```bash REDISCLI_AUTH="$OPIK_REDIS_SENTINEL_PASSWORD" redis-cli \ --user "${OPIK_REDIS_SENTINEL_USERNAME:-default}" \ -h sentinel-1 -p 26379 SENTINEL MASTERS REDISCLI_AUTH="$OPIK_REDIS_SENTINEL_PASSWORD" redis-cli \ --user "${OPIK_REDIS_SENTINEL_USERNAME:-default}" \ -h sentinel-1 -p 26379 SENTINEL SENTINELS mymaster REDISCLI_AUTH="$OPIK_REDIS_SENTINEL_PASSWORD" redis-cli \ --user "${OPIK_REDIS_SENTINEL_USERNAME:-default}" \ -h sentinel-1 -p 26379 SENTINEL REPLICAS mymaster REDISCLI_AUTH="$OPIK_REDIS_SENTINEL_PASSWORD" redis-cli \ --user "${OPIK_REDIS_SENTINEL_USERNAME:-default}" \ -h sentinel-1 -p 26379 SENTINEL CKQUORUM mymaster ```