Skip to navigation
Configuration

Redis Sentinel

Use Sentinel-based Redis master discovery in self-hosted deployments

View as Markdown

By default, the Opik backends connect directly to a single Redis endpoint through REDIS_URL. Enable Redis Sentinel when you want them to discover the active Redis master and reconnect after Sentinel promotes a replica. Opik configures the clients only; you must provide and operate the Redis master, replicas, and Sentinel processes.

Configure Sentinel on both the Java backend and the Python backend, which runs the RQ workers. A service left on a fixed Redis address keeps writing to the old master after a failover: those writes are lost when Sentinel demotes it, and later ones fail with READONLY.

Prerequisites

Before enabling Sentinel mode, make sure that:

  • A Redis master and at least one replica are running and synchronized.
  • All Sentinel processes monitor the same master set name.
  • Sentinel-announced Redis and Sentinel addresses are reachable from both backends.
  • Authentication and TLS settings are consistent across the topology.
  • Production deployments use at least three Sentinel processes in independent failure domains. A typical three-Sentinel deployment uses a quorum of 2.

See the Redis Sentinel documentation for Redis-side deployment and quorum guidance.

Configure the backends

The example below uses:

  • mymaster as the Sentinel master set name
  • sentinel-1:26379 as the initial Sentinel seed
  • sentinel-2:26379 and sentinel-3:26379 as additional seeds
  • redis-password for Redis data-node authentication
  • sentinel-password for Sentinel authentication
  • Redis database 0

Add the same overrides to both backends in deployment/docker-compose/docker-compose.override.yaml:

x-redis-sentinel: &redis-sentinel
  REDIS_URL: "redis://:redis-password@sentinel-1:26379/0"
  OPIK_REDIS_SENTINEL_ENABLED: "true"
  OPIK_REDIS_SENTINEL_MASTER_NAME: "mymaster"
  OPIK_REDIS_SENTINEL_NODES: "redis://sentinel-2:26379,redis://sentinel-3:26379"
  OPIK_REDIS_SENTINEL_PASSWORD: "sentinel-password"
services:
  backend:
    environment: *redis-sentinel
  python-backend:
    environment: *redis-sentinel

Apply the change from the deployment/docker-compose directory:

docker compose up -d backend python-backend

How REDIS_URL is interpreted

When Sentinel mode is disabled, REDIS_URL remains the direct single-node Redis URL. When Sentinel mode is enabled, the same URL is interpreted as follows:

URL partMeaning in Sentinel mode
Scheme (redis:// or rediss://)Connection scheme for the Sentinel topology
Host and portInitial Sentinel seed. The port is required, since Sentinel can listen on any port (commonly 26379)
Username and passwordCredentials for Redis data nodes, not for Sentinel
PathRedis database selected on the discovered master

Sentinel authentication is configured independently with OPIK_REDIS_SENTINEL_USERNAME and OPIK_REDIS_SENTINEL_PASSWORD.

For ACL-based authentication, use separate credentials for the data nodes and Sentinel:

REDIS_URL="redis://redis-user:redis-password@sentinel-1:26379/0"
OPIK_REDIS_SENTINEL_USERNAME="sentinel-user"
OPIK_REDIS_SENTINEL_PASSWORD="sentinel-password"

If Sentinel does not require authentication, omit the Sentinel username and password variables.

Configuration reference

Configure these environment variables, with the same values, on both the Java backend and the Python backend. Both read them the same way, and both validate them at startup whenever Sentinel is enabled, refusing to start on an invalid value. Set to an empty value, OPIK_REDIS_SENTINEL_ENABLED means disabled and the nodes, username and password are treated as unset; any other empty variable is invalid. Set boolean variables to true or false.

Environment variableDefaultDescription
REDIS_URLredis://:opik@localhost:6379/0Direct Redis URL in single-node mode. In Sentinel mode, its host and port identify the initial Sentinel seed, while its credentials and database apply to Redis data nodes.
OPIK_REDIS_SENTINEL_ENABLEDfalseSet to true to enable Sentinel-based master discovery, or false to keep single-node mode.
OPIK_REDIS_SENTINEL_MASTER_NAMEEmptySentinel master set name. Required when Sentinel mode is enabled.
OPIK_REDIS_SENTINEL_NODESEmptyComma-separated additional Sentinel seed addresses, each with a host, a port and the same redis:// or rediss:// scheme as REDIS_URL. Do not embed credentials; use the Sentinel username and password variables.
OPIK_REDIS_SENTINEL_USERNAMEEmptyUsername used to authenticate to Sentinel nodes when Sentinel ACLs are enabled.
OPIK_REDIS_SENTINEL_PASSWORDEmptyPassword used to authenticate to Sentinel nodes.
OPIK_REDIS_SENTINEL_RETRY_ATTEMPTS3Number of retry attempts for commands that fail during a topology change, between 0 and 20.
OPIK_REDIS_SENTINEL_RETRY_BACKOFF_BASE500msBase of the full-jitter backoff: each retry waits a random time up to min(base × 2^attempt, cap).
OPIK_REDIS_SENTINEL_RETRY_BACKOFF_CAP2sLongest wait between retries. Must be at least the base. Keep the total retry time above your Sentinel promotion time, down-after-milliseconds plus the election.
OPIK_REDIS_SENTINEL_CHECK_SENTINELS_LISTtrueRequires Sentinel discovery to report at least two Sentinel nodes. Set to false only for single-Sentinel development environments.
OPIK_REDIS_SENTINEL_CONNECT_TIMEOUT10sTimeout for establishing Redis or Sentinel connections.
OPIK_REDIS_SENTINEL_TIMEOUT5sTimeout for Redis command responses.
OPIK_REDIS_SENTINEL_SCAN_INTERVAL2sJava backend only. Interval at which Redisson polls Sentinel for topology changes. The Python client needs no polling: it asks Sentinel for the master on every new connection and reconnects when the old master stops accepting writes.

Duration values accept forms such as 500ms, 5s, and 1m. Timeouts and the scan interval must be between 100ms and 5m; the backoff base and cap between 1ms and 5m.

Sentinel mode cannot be combined with AWS IAM authentication (redis.awsIamAuth). ElastiCache and MemoryDB, where IAM authentication applies, do not offer Sentinel; point REDIS_URL at their primary endpoint in single-node mode instead.

TLS

Use the rediss:// scheme in REDIS_URL and every address in OPIK_REDIS_SENTINEL_NODES when the Sentinel topology requires TLS.

Opik does not expose separate connection schemes for Sentinel and Redis data nodes, so a topology that mixes plaintext Sentinel connections with TLS data-node connections, or the reverse, is not supported by this configuration.

For TLS deployments, make sure Sentinel announces hostnames or addresses that:

  • Are reachable from both backends
  • Match the subject alternative names in the Redis server certificates

Production guidance

  • Keep OPIK_REDIS_SENTINEL_CHECK_SENTINELS_LIST=true in production.
  • Provide multiple seed addresses through OPIK_REDIS_SENTINEL_NODES so backend startup does not depend on one Sentinel being reachable.
  • Place Sentinel processes in independent failure domains.
  • Ensure replicas are synchronized and eligible for promotion before testing failover.
  • Store Redis and Sentinel credentials in your deployment’s secret manager rather than directly in a committed values or Compose file.
  • Remember that Redis replication is asynchronous. Sentinel improves availability but does not guarantee that every acknowledged write survives every failure.

Verify the configuration

Confirm Sentinel mode at startup

Check the Java backend logs for:

Built redis config with address 'redis://sentinel-1:26379', database '0', sentinel 'true'
docker compose logs backend | grep "Built redis config"

If the log reports sentinel 'false', Sentinel mode is not enabled in the Java backend process.

The Python backend logs Redis: using Sentinel master 'mymaster' through 3 sentinels, database 0 at startup.

Check Redis health

Query the backend health endpoint from inside the deployment or through a port-forward:

docker compose exec backend \
  curl -fsS "http://localhost:8080/health-check?name=redis"

A healthy Redis topology returns HTTP 200.

The Redis health check pings the master currently discovered through Sentinel, the only node that serves Opik traffic. A lost replica or Sentinel keeps the backend ready; the check can briefly report unhealthy while Sentinel promotes a new master.

Validate failover

Perform this procedure only in a development or staging environment. Triggering a failover changes the writable Redis node and can interrupt active requests.

Keep both Opik backends running while performing this check.

The commands below use OPIK_REDIS_SENTINEL_USERNAME and OPIK_REDIS_SENTINEL_PASSWORD. Set these variables in the shell running redis-cli. If the username is unset, the commands authenticate as the Redis default ACL user.

1

Confirm the current master

REDISCLI_AUTH="$OPIK_REDIS_SENTINEL_PASSWORD" redis-cli \
  --user "${OPIK_REDIS_SENTINEL_USERNAME:-default}" \
  -h sentinel-1 -p 26379 \
  SENTINEL GET-MASTER-ADDR-BY-NAME mymaster
2

Trigger a controlled failover

Or stop the current master according to your Redis operating procedure:

REDISCLI_AUTH="$OPIK_REDIS_SENTINEL_PASSWORD" redis-cli \
  --user "${OPIK_REDIS_SENTINEL_USERNAME:-default}" \
  -h sentinel-1 -p 26379 \
  SENTINEL FAILOVER mymaster
3

Wait for the promotion

Repeat the first step until SENTINEL GET-MASTER-ADDR-BY-NAME reports the promoted replica.

4

Check Opik still works

Without restarting either backend, repeat a normal Opik operation that uses Redis and confirm it succeeds, and that both backend processes stayed up throughout the failover.

Requests may briefly fail or time out while Sentinel promotes a replica and Redisson reconnects. Tune the retry and timeout settings to match your Sentinel failure-detection and failover timing.

Troubleshooting

SymptomLikely causeResolution
Master node is undefined or GET-MASTER-ADDR-BY-NAME returns no addressThe configured master name does not match the Sentinel master setRun SENTINEL MASTERS and set OPIK_REDIS_SENTINEL_MASTER_NAME to the reported name.
Startup reports fewer than two Sentinel nodesSentinel discovery cannot reach enough nodesCheck network access and announced addresses. Use OPIK_REDIS_SENTINEL_CHECK_SENTINELS_LIST=false only for local single-Sentinel testing.
Sentinel authentication succeeds but Redis commands fail with authentication errorsSentinel credentials were also used as data-node credentials, or the reversePut Redis data-node credentials in REDIS_URL and Sentinel credentials in the dedicated Sentinel variables.
TLS handshake or hostname verification failsThe URL scheme is incorrect, or Sentinel announces an address that does not match the certificateUse the correct redis:// or rediss:// scheme and configure routable certificate-valid announced hostnames.
Sentinel reports a new master, but Opik cannot reconnectThe promoted node’s announced address is not reachable from the Java backendCorrect Redis and Sentinel announce settings, DNS, firewall rules, or network routes.
The Python backend or RQ workers fail with READONLY after a failoverThe Python backend still uses a fixed Redis addressSet the same REDIS_URL and OPIK_REDIS_SENTINEL_* variables on the Python backend as on the Java backend.

Useful Sentinel diagnostics:

REDISCLI_AUTH="$OPIK_REDIS_SENTINEL_PASSWORD" redis-cli \
  --user "${OPIK_REDIS_SENTINEL_USERNAME:-default}" \
  -h sentinel-1 -p 26379 SENTINEL MASTERS
REDISCLI_AUTH="$OPIK_REDIS_SENTINEL_PASSWORD" redis-cli \
  --user "${OPIK_REDIS_SENTINEL_USERNAME:-default}" \
  -h sentinel-1 -p 26379 SENTINEL SENTINELS mymaster
REDISCLI_AUTH="$OPIK_REDIS_SENTINEL_PASSWORD" redis-cli \
  --user "${OPIK_REDIS_SENTINEL_USERNAME:-default}" \
  -h sentinel-1 -p 26379 SENTINEL REPLICAS mymaster
REDISCLI_AUTH="$OPIK_REDIS_SENTINEL_PASSWORD" redis-cli \
  --user "${OPIK_REDIS_SENTINEL_USERNAME:-default}" \
  -h sentinel-1 -p 26379 SENTINEL CKQUORUM mymaster