# Clustering & high availability

> **Sentinel v2 (beta):** This page documents Sentinel v2, a beta rewrite that may change before the stable release. It does not apply to Sentinel v1, the current stable version: see the [Sentinel v1 documentation](/docs/sentinel.md).

Run several Sentinel replicas behind a load balancer when one instance isn't enough capacity, or when you need Sentinel to keep answering while a replica restarts. Replicas don't talk to each other directly: they share one database and one Redis or Valkey, and exchange notices over its pub/sub.

Prerequisite: a license whose `instances` limit covers your replica count. See [Configure the license](/docs/sentinel/v2/configure/licensing/).

## How a cluster fits together

Every replica runs the same image with the same `sentinel.toml`, and any replica can answer any request. What each kind of state lives in:

| Where | What | Shared |
| --- | --- | --- |
| Database (`[database]`) | Accounts, users, API keys, security groups, training data, threat sources and entries, alerts, audit logs, request logs (unless [ClickHouse](/docs/sentinel/v2/operations/clickhouse/) holds them), the license state, instance slots, and the cluster-wide locks | Yes, the source of truth |
| KV store (`[redis]`) | Rate-limit counters, penalties, replay protection of challenges and signed payloads, sign-in and OTP state, and the pub/sub channels | Yes |
| Each replica's memory | The [API-key cache](#when-the-database-is-down), compiled security-group rules, threat-list indexes, classifier lists, and the last hour of request metrics | No; rebuilt from the database and downloads |
| Each replica's `data_dir` | Downloaded remote files and the indexes built from them; with embedded SQLite, the database itself | No |

Everything a replica keeps for itself it loads again at startup, so replicas are interchangeable and a new one needs no state copied in.

## Requirements

- **An external database.** Use PostgreSQL, MySQL, MariaDB, SQL Server, or Oracle, reachable from every replica; see [Databases & KV store](/docs/sentinel/v2/configure/databases/). Each replica opens up to `[database] pool_max` connections.
- **Redis or Valkey.** The built-in KV store has no pub/sub and serves one instance only, so set `[redis] nodes` or `[redis.sentinel]`. Sizing is in [Redis or Valkey](/docs/sentinel/v2/install/#redis-or-valkey).
- **The same `[redis] key_prefix` on every replica.** The prefix (default `"sentinel:"`) starts every key and pub/sub channel, so replicas only see each other under the same prefix. Two deployments can share one Redis with different prefixes; they then keep separate replica lists, change notices, rate limits, and replay protection.
- **The same secrets on every replica.** Set `[secrets]` explicitly; generated secrets work only for a single instance on embedded SQLite. A mismatch looks like random `401` responses behind the load balancer. See [Share them across replicas](/docs/sentinel/v2/configure/secrets/#share-them-across-replicas).
- **A unique, stable `HOSTNAME` per replica.** It's the replica's node ID in the [instance slots](#instance-slots). Kubernetes sets it to the pod name; Docker sets it to the container ID unless you pass `--hostname`. Don't give two replicas the same value.

A minimal cluster configuration, identical on every replica:

```toml
[database]
url = "postgres://sentinel:${DB_PASSWORD}@db.internal:5432/sentinel"

[redis]
nodes      = ["redis://:${REDIS_PASSWORD}@redis.internal:6379"]
key_prefix = "prod:sentinel:"   # same on every replica; different per deployment sharing this Redis

[secrets]
jwt_secret         = "${file:/run/secrets/jwt_secret}"
altcha_hmac_secret = "${file:/run/secrets/altcha_hmac_secret}"
encryption_key     = "${file:/run/secrets/encryption_key}"
```

## Replica discovery

Each replica publishes a status message over the KV store's pub/sub every 10 seconds, and keeps the last message of every other replica:

- A new replica appears within 10 seconds of starting.
- A replica that misses 3 messages is dropped, so a stopped replica disappears within 30 seconds.
- A restarted replica comes back with a new `id`. Until its old entry expires, the same host name can appear twice.

Besides status messages, replicas notify each other over pub/sub when data changes, so a change made through one replica applies on the others within milliseconds:

| Change | Effect on the other replicas |
| --- | --- |
| API keys, security groups, account suspension or deletion, snapshot restore | Reload the [API-key cache](#when-the-database-is-down) |
| Threat sources and threat entries | Resync from the database |
| Training data | Resync the similarity index |
| A forced download of a remote file | Download the file too |

Pub/sub stores nothing. A notice a replica misses, for example while it reconnects to Redis, is caught up by the periodic reload: the API-key cache reloads every `[api_key_cache] refresh_interval` (default `10s`), and threats and training data on their own sync intervals.

## Background jobs

There is no leader replica. Every replica runs the scheduled jobs, and a cluster-wide lock in the database makes sure only one replica runs each one at a time:

- schema migrations at startup (see [Schema migrations](/docs/sentinel/v2/operations/schema-migrations/#replicas-and-rolling-updates));
- retention of request logs, alerts, and audit logs, and the ClickHouse rollup;
- scheduled [snapshots](/docs/sentinel/v2/operations/snapshots/), and every snapshot create or restore;
- downloads of remote files and threat lists: one replica downloads, and the others adopt its result;
- online license refresh.

Losing a replica in the middle of a job only delays it; another replica takes the lock once it expires.

## Instance slots

The license limits how many replicas run at once (its `instances` limit; 3 without a usable license). Each replica holds one slot in the shared database, as a lease:

| Rule | Value |
| --- | --- |
| Lease length | 30 seconds, renewed every 10 seconds |
| Regular slots | The license's `instances` limit, N |
| Burst slots | `max(1, ceil(N / 10))`: 1 for up to 10 instances, 2 for up to 20, and so on |
| Longest burst hold | 30 minutes per replica at a time |
| Burst budget | 4 hours of burst use per 24 hours, shared by every replica of the license |
| Startup wait | 60 seconds, retried every 2 seconds |
| Runtime grace | 5 minutes without a slot before the replica goes over the limit |

The slots in use are listed under `instances` in `GET /v2/license` and on the **License** page. With three replicas on a 3-instance license:

```json
"instances": {
"limit": 3,
"used": 3,
"state": "regular",
"slots": [
{ "name": "lic_…:r:0", "kind": "regular", "node_id": "sentinel-2", "store": "database", "expires_at": 1795080448049 },
{ "name": "lic_…:r:1", "kind": "regular", "node_id": "sentinel-0", "store": "database", "expires_at": 1795080442142 },
{ "name": "lic_…:r:2", "kind": "regular", "node_id": "sentinel-1", "store": "database", "expires_at": 1795080442197 }
]
}
```

`state` is this replica's: `regular`, `burst`, `acquiring`, `standby` (a [standby instance](#standby-instances) still waiting), `over_limit`, or `disabled` (an unlimited license).

### Startup

A starting replica takes a free regular slot. When none is free, it takes a free burst slot, if the burst budget isn't used up. When neither is free, it retries for 60 seconds and then exits with code `1`, naming the holders:

```text
Error: licence allows 3 instances; database: slots held by sentinel-3, sentinel-2, sentinel-0, sentinel-1 (expire within 30s if those nodes are gone)
```

Your orchestrator restarts it, and it starts once a slot frees up.

### Rolling deploys

Burst slots let a rolling update start a new replica before it stops an old one, even when you already run as many replicas as the license allows:

1. The new replica starts on a burst slot (`state: "burst"`) and serves requests normally.

2. The orchestrator stops an old replica, which releases its regular slot on shutdown.

3. Within 10 seconds, the new replica moves to the free regular slot and gives up the burst slot.

4. The next new replica takes the burst slot, and so on.

Start at most as many extra replicas at once as there are burst slots: with Kubernetes, keep `maxSurge` at or below the burst slot count. A replica can hold a burst slot for 30 minutes at a time, and all bursts of a license together for 4 hours within 24 hours. A replica that runs past either limit without finding a regular slot gives up the burst slot and goes over the limit. If you run fewer replicas than the license allows, the free regular slots cover the surge and no burst slot is used.

### Restarts and node IDs

A replica's node ID is the `HOSTNAME` environment variable, or the machine's host name when `HOSTNAME` is unset. With `HOSTNAME` set, a replica that restarts, even after a crash or `kill -9`, takes its previous slot back at once instead of waiting for the old lease to expire. Without it, a restarted replica waits up to 30 seconds for the old lease to expire, or uses a burst slot meanwhile.

A graceful shutdown (`SIGTERM`) releases the slot immediately.

If two live processes use the same `HOSTNAME`, they keep taking the slot from each other. After 3 such collisions within 10 minutes, Sentinel logs `duplicate node_id …: another live process uses this node id; restart reclaim disabled` and stops reclaiming slots by node ID. Give every replica its own `HOSTNAME`.

### Over the limit

A running replica that loses its slot, for example after the license changes to fewer instances, tries to get one again. After 5 minutes without a slot, or when its burst runs out, it goes over the limit:

- every `/v2` request except `/v2/auth/*`, `/v2/me*`, `/v2/app-config`, `/v2/license*`, and `/v2/doctor*` answers `503 LICENSE_INSTANCE_LIMIT` (see [Error codes](/docs/sentinel/v2/reference/errors/#licensing));
- the `license_instance_limit` alert is raised (see [Alerts](/docs/sentinel/v2/operations/alerts/));
- the replica keeps trying, and serves again as soon as a slot frees up.

`GET /` keeps answering, so a load balancer probe on `GET /` doesn't take an over-the-limit replica out of rotation. Watch the `altcha_sentinel_license_instances_used` and `altcha_sentinel_license_instances_limit` metrics ([Monitoring](/docs/sentinel/v2/operations/monitoring/)).

A replica that can't reach the database keeps its slot and doesn't go over the limit during the outage.

### Standby instances

A standby instance is a warm failover replica that doesn't need a spare instance in the license, for example a second set of replicas in another availability zone that shares the production database. The license plan sets how many standbys a deployment may run; see [Plans & Licensing](/docs/sentinel/v2/plans/#standby-instances). Mark a replica as a standby in its config:

```toml
[license]
key     = "${SENTINEL_LICENSE_KEY}"
standby = true   # wait for a free regular slot instead of taking one at startup
```

A standby behaves like this:

- **At startup**, it connects to the database and applies or verifies the schema migrations, as any replica does, but takes no slot. It never exits for lack of a slot and never takes a burst slot.
- **While it waits**, the main listener answers `GET /` with `200` and every other request, `/.health` included, with `503 LICENSE_STANDBY`. The public API, admin UI, and Prometheus listeners aren't bound, and no background work runs: no bootstrap, snapshots, KV store, request logs, retention or cluster jobs, or alerts.
- **It takes over** a regular slot once the slot has been free on two checks at least 15 seconds apart, so a rolling deploy's burst replica gets the slot first. It then logs `licence: standby node acquired an instance slot; starting` and starts like any other replica. Under an unlimited license, it starts at once.
- **Failback is manual.** A promoted standby keeps its slot until it restarts. After a restart it waits as a standby again, and it doesn't reclaim its old slot by node ID.

`GET /` answers on a waiting standby, so a load balancer that probes only `GET /` sends it traffic that fails with `503`. To route to standbys only after they take over, probe `GET /v2/app-config`: it needs no credentials, reads no storage, and answers `503 LICENSE_STANDBY` while the standby waits. Give each standby its own `HOSTNAME`, as for every replica.

## When the database is down

Each replica keeps the API keys in memory, with each key's security group and account state, and reloads them every `refresh_interval`:

```toml
[api_key_cache]
enabled          = true    # false: read the key, group, and account from the database per request
refresh_interval = "10s"   # background reload, also its timeout (1s–1h)
max_keys         = 10000   # keys beyond this are read from the database per request
```

A failed reload keeps the previous copy, so while the database is down:

| Keeps working | Stops working |
| --- | --- |
| `GET /v2/challenge`, `POST /v2/verify`, and `POST /v2/verify/signature`, with the keys, groups, and accounts as of the last successful reload | The admin UI and every management endpoint; keys can't be created, edited, or revoked |
| Replay protection and rate limits in Redis | Endpoints that read the database, such as training data and threat entries |
| Replicas that are already running | Starting a replica: migrations, the license, and the first cache load need the database |

Keys beyond `max_keys` aren't cached, so requests with them fail with `500` during the outage. With an embedded database and no `[redis] nodes`, the KV store lives in the same database and fails with it; one more reason a cluster needs Redis.

Each replica raises the `db_unavailable` alert once the outage lasts `[alerts.db_unavailable] duration` (default `1m`). Alert on the `altcha_sentinel_api_key_cache_age_seconds` metric well above `refresh_interval`, and on `altcha_sentinel_api_key_cache_complete == 0` (more live keys than `max_keys`). Keys reference: [`[api_key_cache]`](/docs/sentinel/v2/reference/configuration/#api_key_cache).

## When the KV store is down

A replica starts and keeps serving while Redis is unreachable. The widget endpoints keep working with per-replica fallbacks: replay protection moves to the replica's memory (up to 100,000 claims, written back to Redis once it answers), and rate limits fall back to per-replica counters. During the outage, a solution sent to two different replicas can be accepted on both. Sign-in, OTP, passkeys, and step-up verification fail, because their replay checks need Redis. The details are in [When the KV store is unavailable](/docs/sentinel/v2/configure/databases/#when-the-kv-store-is-unavailable).

Replicas also can't publish status messages without Redis, so the others drop them from the replica list after 30 seconds, and change notices are caught up by the periodic reloads. Each replica raises the `kv_unavailable` alert once the outage lasts `[alerts.kv_unavailable] duration` (default `1m`).

## List the replicas

Root users see every live replica on the **Cluster** page of the admin UI, or through the API with a session token:

```bash
curl -H 'Authorization: Bearer <jwt>' https://sentinel.example.com/v2/cluster/replicas
```

```json
{
"data": [
{
  "id": "rep_1k7pi23jm0011pln8cq",
  "hostname": "sentinel-0",
  "version": "2.0.0-beta.1",
  "startedAt": "2026-11-19T09:26:32.310830+00:00",
  "lastSeenAt": "2026-11-19T09:26:42.313089+00:00",
  "current": false,
  "memoryLimit": 2147483648,
  "metrics": [
    { "at": "2026-11-19T09:26:00+00:00", "requests": 0, "errors4xx": 0, "errors5xx": 0, "latencyMsAvg": null, "latencyMsMax": null, "cpu": 0.0, "cpuMax": 0.01, "memory": 94923434, "memoryMax": 97435648 }
  ],
  "similarity": { "provider": "none", "modelId": null, "loaded": false, … },
  "storage": {
    "kv": { "available": true, "checkedAt": "2026-11-19T09:26:42.150759+00:00", "latencyMs": 0.39, "error": null, "unavailableSince": null, "outages24h": 0, "downtime24hSecs": 0 },
    "db": { "available": true, "checkedAt": "2026-11-19T09:26:42.150759+00:00", "latencyMs": 0.08, "error": null, "unavailableSince": null, "outages24h": 0, "downtime24hSecs": 0 }
  },
  "clockOffsetMs": -1
},
{
  "id": "rep_1k7pi23l8001230obtc",
  "hostname": "sentinel-1",
  "version": "2.0.0-beta.1",
  "current": true,
  "metrics": [ … ],
  "clockOffsetMs": null,
  …
}
]
}
```

- Any replica can answer, and they all return the same list. `current: true` marks the replica that answered.
- `metrics` holds the last hour in 1-minute buckets, oldest first; the last entry is the minute in progress. Metrics live only in memory and restart from zero with the replica.
- Other replicas' entries are as of their last status message, up to 10 seconds old (`lastSeenAt`).
- `storage` is the replica's own view of the KV store and the database, probed every 10 seconds.
- `version` shows which replicas already run a new version during a rolling update.

The endpoint needs a root user's session (`403 ROOT_PRIVILEGES_REQUIRED` otherwise; API keys get `401`). Every field is described in the [API reference](/sentinel-api-docs-v2.html#tag/cluster/GET/v2/cluster/replicas).

## Multi-region deployments

A cluster is one database and one KV store. Every replica must reach both, and every KV command on the request path, such as a replay check during `POST /v2/verify`, waits for the round trip to Redis. Choose one of these layouts:

| Layout | How | Trade-offs |
| --- | --- | --- |
| One region | All replicas, the database, and Redis in one region. Clients in other regions call it over the internet. | Simplest. One set of accounts and keys. Latency for distant clients. |
| One deployment per region | A separate database, Redis, and license per region. | Low latency everywhere. Accounts, keys, and rules are separate per region; replay protection, rate limits, and penalties don't cross regions. |

With one deployment per region:

- Each deployment has its own deployment ID, so each needs its own license.
- Keep the widget and the backend of one site on the same region's deployment. API keys exist only in the deployment that created them, and replay claims aren't shared between deployments.
- Keep accounts, keys, and security groups of the deployments in sync yourself.

Avoid stretching one cluster across regions with a remote Redis: every challenge and verification then pays the cross-region latency, and a network partition between the regions acts as a [KV outage](#when-the-kv-store-is-down) for the replicas on the far side.

## Related

- [Install Sentinel: requirements and sizing](/docs/sentinel/v2/install/)
- [Databases & KV store](/docs/sentinel/v2/configure/databases/)
- [Configure the license](/docs/sentinel/v2/configure/licensing/)
- [Server secrets](/docs/sentinel/v2/configure/secrets/)
- [Updating Sentinel](/docs/sentinel/v2/operations/updating/)
- [Monitoring & metrics](/docs/sentinel/v2/operations/monitoring/)
- [Doctor & troubleshooting](/docs/sentinel/v2/operations/troubleshooting/)
