Skip to main content
ALTCHA®

Clustering & high availability

Run several Sentinel replicas behind a load balancer when one instance isn't enough capacity, or when you need Sentinel to keep answering while a replica restarts. Replicas don't talk to each other directly: they share one database and one Redis or Valkey, and exchange notices over its pub/sub.

Prerequisite: a license whose instances limit covers your replica count. See Configure the license.

How a cluster fits together

Every replica runs the same image with the same sentinel.toml, and any replica can answer any request. What each kind of state lives in:

WhereWhatShared
Database ([database])Accounts, users, API keys, security groups, training data, threat sources and entries, alerts, audit logs, request logs (unless ClickHouse holds them), the license state, instance slots, and the cluster-wide locksYes, the source of truth
KV store ([redis])Rate-limit counters, penalties, replay protection of challenges and signed payloads, sign-in and OTP state, and the pub/sub channelsYes
Each replica's memoryThe API-key cache, compiled security-group rules, threat-list indexes, classifier lists, and the last hour of request metricsNo; rebuilt from the database and downloads
Each replica's data_dirDownloaded remote files and the indexes built from them; with embedded SQLite, the database itselfNo

Everything a replica keeps for itself it loads again at startup, so replicas are interchangeable and a new one needs no state copied in.

Requirements

  • An external database. Use PostgreSQL, MySQL, MariaDB, SQL Server, or Oracle, reachable from every replica; see Databases & KV store. Each replica opens up to [database] pool_max connections.
  • Redis or Valkey. The built-in KV store has no pub/sub and serves one instance only, so set [redis] nodes or [redis.sentinel]. Sizing is in Redis or Valkey.
  • The same [redis] key_prefix on every replica. The prefix (default "sentinel:") starts every key and pub/sub channel, so replicas only see each other under the same prefix. Two deployments can share one Redis with different prefixes; they then keep separate replica lists, change notices, rate limits, and replay protection.
  • The same secrets on every replica. Set [secrets] explicitly; generated secrets work only for a single instance on embedded SQLite. A mismatch looks like random 401 responses behind the load balancer. See Share them across replicas.
  • A unique, stable HOSTNAME per replica. It's the replica's node ID in the instance slots. Kubernetes sets it to the pod name; Docker sets it to the container ID unless you pass --hostname. Don't give two replicas the same value.

A minimal cluster configuration, identical on every replica:

[database]
url = "postgres://sentinel:${DB_PASSWORD}@db.internal:5432/sentinel"

[redis]
nodes      = ["redis://:${REDIS_PASSWORD}@redis.internal:6379"]
key_prefix = "prod:sentinel:"   # same on every replica; different per deployment sharing this Redis

[secrets]
jwt_secret         = "${file:/run/secrets/jwt_secret}"
altcha_hmac_secret = "${file:/run/secrets/altcha_hmac_secret}"
encryption_key     = "${file:/run/secrets/encryption_key}"

Replica discovery

Each replica publishes a status message over the KV store's pub/sub every 10 seconds, and keeps the last message of every other replica:

  • A new replica appears within 10 seconds of starting.
  • A replica that misses 3 messages is dropped, so a stopped replica disappears within 30 seconds.
  • A restarted replica comes back with a new id. Until its old entry expires, the same host name can appear twice.

Besides status messages, replicas notify each other over pub/sub when data changes, so a change made through one replica applies on the others within milliseconds:

ChangeEffect on the other replicas
API keys, security groups, account suspension or deletion, snapshot restoreReload the API-key cache
Threat sources and threat entriesResync from the database
Training dataResync the similarity index
A forced download of a remote fileDownload the file too

Pub/sub stores nothing. A notice a replica misses, for example while it reconnects to Redis, is caught up by the periodic reload: the API-key cache reloads every [api_key_cache] refresh_interval (default 10s), and threats and training data on their own sync intervals.

Background jobs

There is no leader replica. Every replica runs the scheduled jobs, and a cluster-wide lock in the database makes sure only one replica runs each one at a time:

  • schema migrations at startup (see Schema migrations);
  • retention of request logs, alerts, and audit logs, and the ClickHouse rollup;
  • scheduled snapshots, and every snapshot create or restore;
  • downloads of remote files and threat lists: one replica downloads, and the others adopt its result;
  • online license refresh.

Losing a replica in the middle of a job only delays it; another replica takes the lock once it expires.

Instance slots

The license limits how many replicas run at once (its instances limit; 3 without a usable license). Each replica holds one slot in the shared database, as a lease:

RuleValue
Lease length30 seconds, renewed every 10 seconds
Regular slotsThe license's instances limit, N
Burst slotsmax(1, ceil(N / 10)): 1 for up to 10 instances, 2 for up to 20, and so on
Longest burst hold30 minutes per replica at a time
Burst budget4 hours of burst use per 24 hours, shared by every replica of the license
Startup wait60 seconds, retried every 2 seconds
Runtime grace5 minutes without a slot before the replica goes over the limit

The slots in use are listed under instances in GET /v2/license and on the License page. With three replicas on a 3-instance license:

"instances": {
  "limit": 3,
  "used": 3,
  "state": "regular",
  "slots": [
    { "name": "lic_…:r:0", "kind": "regular", "node_id": "sentinel-2", "store": "database", "expires_at": 1795080448049 },
    { "name": "lic_…:r:1", "kind": "regular", "node_id": "sentinel-0", "store": "database", "expires_at": 1795080442142 },
    { "name": "lic_…:r:2", "kind": "regular", "node_id": "sentinel-1", "store": "database", "expires_at": 1795080442197 }
  ]
}

state is this replica's: regular, burst, acquiring, standby (a standby instance still waiting), over_limit, or disabled (an unlimited license).

Startup

A starting replica takes a free regular slot. When none is free, it takes a free burst slot, if the burst budget isn't used up. When neither is free, it retries for 60 seconds and then exits with code 1, naming the holders:

Error: licence allows 3 instances; database: slots held by sentinel-3, sentinel-2, sentinel-0, sentinel-1 (expire within 30s if those nodes are gone)

Your orchestrator restarts it, and it starts once a slot frees up.

Rolling deploys

Burst slots let a rolling update start a new replica before it stops an old one, even when you already run as many replicas as the license allows:

  1. The new replica starts on a burst slot (state: "burst") and serves requests normally.

  2. The orchestrator stops an old replica, which releases its regular slot on shutdown.

  3. Within 10 seconds, the new replica moves to the free regular slot and gives up the burst slot.

  4. The next new replica takes the burst slot, and so on.

Start at most as many extra replicas at once as there are burst slots: with Kubernetes, keep maxSurge at or below the burst slot count. A replica can hold a burst slot for 30 minutes at a time, and all bursts of a license together for 4 hours within 24 hours. A replica that runs past either limit without finding a regular slot gives up the burst slot and goes over the limit. If you run fewer replicas than the license allows, the free regular slots cover the surge and no burst slot is used.

Restarts and node IDs

A replica's node ID is the HOSTNAME environment variable, or the machine's host name when HOSTNAME is unset. With HOSTNAME set, a replica that restarts, even after a crash or kill -9, takes its previous slot back at once instead of waiting for the old lease to expire. Without it, a restarted replica waits up to 30 seconds for the old lease to expire, or uses a burst slot meanwhile.

A graceful shutdown (SIGTERM) releases the slot immediately.

If two live processes use the same HOSTNAME, they keep taking the slot from each other. After 3 such collisions within 10 minutes, Sentinel logs duplicate node_id …: another live process uses this node id; restart reclaim disabled and stops reclaiming slots by node ID. Give every replica its own HOSTNAME.

Over the limit

A running replica that loses its slot, for example after the license changes to fewer instances, tries to get one again. After 5 minutes without a slot, or when its burst runs out, it goes over the limit:

  • every /v2 request except /v2/auth/*, /v2/me*, /v2/app-config, /v2/license*, and /v2/doctor* answers 503 LICENSE_INSTANCE_LIMIT (see Error codes);
  • the license_instance_limit alert is raised (see Alerts);
  • the replica keeps trying, and serves again as soon as a slot frees up.

GET / keeps answering, so a load balancer probe on GET / doesn't take an over-the-limit replica out of rotation. Watch the altcha_sentinel_license_instances_used and altcha_sentinel_license_instances_limit metrics (Monitoring).

A replica that can't reach the database keeps its slot and doesn't go over the limit during the outage.

Standby instances

A standby instance is a warm failover replica that doesn't need a spare instance in the license, for example a second set of replicas in another availability zone that shares the production database. The license plan sets how many standbys a deployment may run; see Plans & Licensing. Mark a replica as a standby in its config:

[license]
key     = "${SENTINEL_LICENSE_KEY}"
standby = true   # wait for a free regular slot instead of taking one at startup

A standby behaves like this:

  • At startup, it connects to the database and applies or verifies the schema migrations, as any replica does, but takes no slot. It never exits for lack of a slot and never takes a burst slot.
  • While it waits, the main listener answers GET / with 200 and every other request, /.health included, with 503 LICENSE_STANDBY. The public API, admin UI, and Prometheus listeners aren't bound, and no background work runs: no bootstrap, snapshots, KV store, request logs, retention or cluster jobs, or alerts.
  • It takes over a regular slot once the slot has been free on two checks at least 15 seconds apart, so a rolling deploy's burst replica gets the slot first. It then logs licence: standby node acquired an instance slot; starting and starts like any other replica. Under an unlimited license, it starts at once.
  • Failback is manual. A promoted standby keeps its slot until it restarts. After a restart it waits as a standby again, and it doesn't reclaim its old slot by node ID.

GET / answers on a waiting standby, so a load balancer that probes only GET / sends it traffic that fails with 503. To route to standbys only after they take over, probe GET /v2/app-config: it needs no credentials, reads no storage, and answers 503 LICENSE_STANDBY while the standby waits. Give each standby its own HOSTNAME, as for every replica.

When the database is down

Each replica keeps the API keys in memory, with each key's security group and account state, and reloads them every refresh_interval:

[api_key_cache]
enabled          = true    # false: read the key, group, and account from the database per request
refresh_interval = "10s"   # background reload, also its timeout (1s–1h)
max_keys         = 10000   # keys beyond this are read from the database per request

A failed reload keeps the previous copy, so while the database is down:

Keeps workingStops working
GET /v2/challenge, POST /v2/verify, and POST /v2/verify/signature, with the keys, groups, and accounts as of the last successful reloadThe admin UI and every management endpoint; keys can't be created, edited, or revoked
Replay protection and rate limits in RedisEndpoints that read the database, such as training data and threat entries
Replicas that are already runningStarting a replica: migrations, the license, and the first cache load need the database

Keys beyond max_keys aren't cached, so requests with them fail with 500 during the outage. With an embedded database and no [redis] nodes, the KV store lives in the same database and fails with it; one more reason a cluster needs Redis.

Each replica raises the db_unavailable alert once the outage lasts [alerts.db_unavailable] duration (default 1m). Alert on the altcha_sentinel_api_key_cache_age_seconds metric well above refresh_interval, and on altcha_sentinel_api_key_cache_complete == 0 (more live keys than max_keys). Keys reference: [api_key_cache].

When the KV store is down

A replica starts and keeps serving while Redis is unreachable. The widget endpoints keep working with per-replica fallbacks: replay protection moves to the replica's memory (up to 100,000 claims, written back to Redis once it answers), and rate limits fall back to per-replica counters. During the outage, a solution sent to two different replicas can be accepted on both. Sign-in, OTP, passkeys, and step-up verification fail, because their replay checks need Redis. The details are in When the KV store is unavailable.

Replicas also can't publish status messages without Redis, so the others drop them from the replica list after 30 seconds, and change notices are caught up by the periodic reloads. Each replica raises the kv_unavailable alert once the outage lasts [alerts.kv_unavailable] duration (default 1m).

List the replicas

Root users see every live replica on the Cluster page of the admin UI, or through the API with a session token:

curl -H 'Authorization: Bearer <jwt>' https://sentinel.example.com/v2/cluster/replicas
{
  "data": [
    {
      "id": "rep_1k7pi23jm0011pln8cq",
      "hostname": "sentinel-0",
      "version": "2.0.0-beta.1",
      "startedAt": "2026-11-19T09:26:32.310830+00:00",
      "lastSeenAt": "2026-11-19T09:26:42.313089+00:00",
      "current": false,
      "memoryLimit": 2147483648,
      "metrics": [
        { "at": "2026-11-19T09:26:00+00:00", "requests": 0, "errors4xx": 0, "errors5xx": 0, "latencyMsAvg": null, "latencyMsMax": null, "cpu": 0.0, "cpuMax": 0.01, "memory": 94923434, "memoryMax": 97435648 }
      ],
      "similarity": { "provider": "none", "modelId": null, "loaded": false, … },
      "storage": {
        "kv": { "available": true, "checkedAt": "2026-11-19T09:26:42.150759+00:00", "latencyMs": 0.39, "error": null, "unavailableSince": null, "outages24h": 0, "downtime24hSecs": 0 },
        "db": { "available": true, "checkedAt": "2026-11-19T09:26:42.150759+00:00", "latencyMs": 0.08, "error": null, "unavailableSince": null, "outages24h": 0, "downtime24hSecs": 0 }
      },
      "clockOffsetMs": -1
    },
    {
      "id": "rep_1k7pi23l8001230obtc",
      "hostname": "sentinel-1",
      "version": "2.0.0-beta.1",
      "current": true,
      "metrics": [ … ],
      "clockOffsetMs": null,
      …
    }
  ]
}
  • Any replica can answer, and they all return the same list. current: true marks the replica that answered.
  • metrics holds the last hour in 1-minute buckets, oldest first; the last entry is the minute in progress. Metrics live only in memory and restart from zero with the replica.
  • Other replicas' entries are as of their last status message, up to 10 seconds old (lastSeenAt).
  • storage is the replica's own view of the KV store and the database, probed every 10 seconds.
  • version shows which replicas already run a new version during a rolling update.

The endpoint needs a root user's session (403 ROOT_PRIVILEGES_REQUIRED otherwise; API keys get 401). Every field is described in the API reference.

Multi-region deployments

A cluster is one database and one KV store. Every replica must reach both, and every KV command on the request path, such as a replay check during POST /v2/verify, waits for the round trip to Redis. Choose one of these layouts:

LayoutHowTrade-offs
One regionAll replicas, the database, and Redis in one region. Clients in other regions call it over the internet.Simplest. One set of accounts and keys. Latency for distant clients.
One deployment per regionA separate database, Redis, and license per region.Low latency everywhere. Accounts, keys, and rules are separate per region; replay protection, rate limits, and penalties don't cross regions.

With one deployment per region:

  • Each deployment has its own deployment ID, so each needs its own license.
  • Keep the widget and the backend of one site on the same region's deployment. API keys exist only in the deployment that created them, and replay claims aren't shared between deployments.
  • Keep accounts, keys, and security groups of the deployments in sync yourself.

Avoid stretching one cluster across regions with a remote Redis: every challenge and verification then pays the cross-region latency, and a network partition between the regions acts as a KV outage for the replicas on the far side.

Start typing to search...

↑ ↓ Navigate ↵ Select