# Databases & KV store

> **Sentinel v2 (beta):** This page documents Sentinel v2, a beta rewrite that may change before the stable release. It does not apply to Sentinel v1, the current stable version: see the [Sentinel v1 documentation](/docs/sentinel.md).

Sentinel stores its data in a database, which is the source of truth, and keeps short-lived shared state in a key-value (KV) store. This page helps you choose both and connect Sentinel to them. Every key and default is listed in [`[database]`](/docs/sentinel/v2/reference/configuration/#database) and [`[redis]`](/docs/sentinel/v2/reference/configuration/#redis) of the Configuration reference.

## Choose a backend

The database holds accounts, users, API keys, security groups, training data, threat entries, request logs (unless you send them to ClickHouse), and the license state. The KV store holds rate-limit counters, penalties, replay protection for solved challenges, and the pub/sub messages that replicas use to find each other. What each one stores is described in [How Sentinel works](/docs/sentinel/v2/concepts/).

| Your deployment | Database | KV store |
| --- | --- | --- |
| Evaluation, or one replica with a persistent volume | Embedded SQLite (the default; nothing to configure) | Built-in (the default) |
| One replica, database managed outside the container | PostgreSQL, MySQL, MariaDB, SQL Server, or Oracle | Built-in, or Redis to keep replay protection up while the database is down |
| Two or more replicas | A shared external database | Redis or Valkey (required) |
| An existing ExotDB server | ExotDB server (`http(s)://`) | Redis, or the ExotDB server's KV port (required) |

Sentinel connects to the database you name in `[database] url`. The scheme selects the backend:

| Backend | `url` scheme | Supported versions | License feature |
| --- | --- | --- | --- |
| SQLite (embedded) | `sqlite:` | Bundled SQLite 3.54.1; the host's version doesn't matter | `exotdb.backend.sqlite` (in the baseline) |
| PostgreSQL | `postgres://` or `postgresql://` | 16 or later | `exotdb.backend.postgres` |
| MySQL | `mysql://` | 8.0.13 or later | `exotdb.backend.mysql` |
| MariaDB | `mariadb://` | 11.4 LTS or later | `exotdb.backend.mariadb` |
| Microsoft SQL Server | `mssql://` or `sqlserver://` | 2022 or later | `exotdb.backend.mssql` |
| Oracle | `oracle://` | 21c or later (needs `JSON_TRANSFORM`); only in the `-oracle` image | `exotdb.backend.oracle` |
| ExotDB server | `http://` or `https://` | — | Enforced by the ExotDB server |

MariaDB is the only backend whose version Sentinel checks: it refuses to start on MariaDB older than 11.4, on a MariaDB server behind a `mysql://` URL, and on a MySQL server behind a `mariadb://` URL. Use the scheme that matches your server.

Oracle connections need the Oracle Instant Client, which only the `-oracle` image tag ships (for example `2.0.0-beta.1-oracle`); see [Container image](/docs/sentinel/v2/install/#container-image). Use that image only with an Oracle database.

<Callout type="note" title="License required">

Every backend except SQLite needs its <code>exotdb.backend.&lt;kind&gt;</code> license feature, for example <code>exotdb.backend.postgres</code>. The built-in KV store is checked the same way, because it runs on the database's backend. Without the feature, Sentinel starts and you can sign in, but most `/v2` routes answer `403 BACKEND_NOT_LICENSED`. See [Plans & Licensing](/docs/sentinel/v2/plans/).

## Configure the database

Set `[database] url`, with the password read from a secret. Without a URL, Sentinel uses `sqlite:<data_dir>/sentinel.db`. Each tab shows a complete `[database]` section:

<Tabs padding={false}>

### SQLite

```toml
# The default is sqlite:<data_dir>/sentinel.db; set url only to put the file elsewhere.
[database]
url = "sqlite:/data/sentinel/sentinel.db"
```

### PostgreSQL

```toml
[database]
url = "postgres://sentinel:${DB_PASSWORD}@db.internal:5432/sentinel?sslmode=verify-full"
```

### MySQL

```toml
[database]
url = "mysql://sentinel:${DB_PASSWORD}@db.internal:3306/sentinel?ssl-mode=VERIFY_IDENTITY"
```

### MariaDB

```toml
[database]
url = "mariadb://sentinel:${DB_PASSWORD}@db.internal:3306/sentinel?ssl-mode=VERIFY_IDENTITY"
```

### SQL Server

```toml
# Encrypted and verified by default.
[database]
url = "mssql://sentinel:${DB_PASSWORD}@db.internal:1433/sentinel"
```

### Oracle

```toml
# -oracle image only. Everything after the last @ is an Oracle connect string.
[database]
url = "oracle://sentinel:${ORACLE_PASSWORD}@tcps://db.example.com:2484/FREEPDB1?ssl_server_dn_match=true"
```

### ExotDB server

```toml
[database]
url   = "https://exotdb.internal:3000"
name  = "sentinel"                                  # database on the server; default "sentinel"
token = "${file:/run/secrets/exotdb_token}"
tls_ca = "${file:/run/secrets/exotdb-ca.pem}"     # only for a private CA

[redis]
nodes = ["redis://:${file:/run/secrets/kv_password}@exotdb.internal:6379"]   # a Redis, or the server's [kvstore] port
```

Sentinel creates and migrates its tables at startup, so the database account needs the right to create tables. When several replicas start at once, one migrates while the others wait for its lock. If your database administrator applies schema changes instead, set `[database] auto_migrate = false` and generate the SQL with `altcha-sentinel schema sql`; see [Schema migrations](/docs/sentinel/v2/operations/schema-migrations/). With any backend other than embedded SQLite, set all three keys in `[secrets]` yourself; Sentinel doesn't generate them. See [Server secrets](/docs/sentinel/v2/configure/secrets/).

### Share a database with `table_prefix`

`[database] table_prefix` (default `""`) is prepended to every table, index, and relation name, and to Sentinel's database lock names. Use it to put Sentinel next to other applications in one database, or to run two deployments in one database:

```toml
[database]
url          = "postgres://sentinel:${DB_PASSWORD}@db.internal:5432/shared"
table_prefix = "sentinel_"
```

The prefix is up to 24 characters of `a-z`, `0-9`, and `_`, starting with a letter. Anything else stops startup with `database.table_prefix must be at most 24 characters of a-z 0-9 _, starting with a-z.` Changing the prefix later starts on new, empty tables; the old ones stay in place.

### SQLite on network storage

Embedded SQLite uses write-ahead logging (WAL) by default, so reads don't wait for writes. WAL keeps its index in shared memory (the `-shm` file next to the database), which works only when every process that opens the file runs on the same host. On a network file system such as AWS EFS, NFS, or an SMB share like Azure Files, it breaks. Switch the file to a rollback journal there:

```toml
[database]
sqlite_journal_mode = "delete"   # default "wal"
```

`delete` needs a file system whose file locks work, and reads and writes then block each other. Keep `wal` on local disks and block volumes such as Amazon EBS.

Sentinel sets the mode at startup, for the database and the built-in KV store, which share the file. Switching an existing file needs every other process to close it first: stop every container that uses it, and close any `sqlite3` session on it. While another process keeps it open, Sentinel retries for 5 seconds, then stops with an error. The key is ignored by every other backend.

Network storage doesn't make SQLite a shared database: embedded SQLite still serves one replica. For more, use an external database.

### Connection pool

Every SQL backend, SQLite included, uses a connection pool per replica:

```toml
[database]
pool_min        = 2       # connections kept open
pool_max        = 20      # most connections this replica opens
acquire_timeout = "5s"    # wait for a free connection, then the call fails
max_lifetime    = "30m"   # not SQLite
idle_timeout    = "10m"   # not SQLite
```

These are the defaults. Each replica has its own pool, so the database sees up to `pool_max` × replicas connections. Keep that below the database's connection limit, with room for rolling deploys, which briefly run extra replicas, and for other clients. Durations accept `"0s"` for no limit (`acquire_timeout = "0s"` waits forever). An ExotDB server sizes its own pool and ignores these keys; for it, `request_timeout` (default `60s`) bounds each HTTP request and must stay above `30s` while `auto_migrate` is on. See [`[database]`](/docs/sentinel/v2/reference/configuration/#database) for the limits of each key.

## Database TLS

PostgreSQL, MySQL, MariaDB, and SQL Server take TLS options as query parameters of `url`, with file paths for certificates:

| Backend | Verify the server | Private CA | Client certificate |
| --- | --- | --- | --- |
| PostgreSQL | `sslmode=verify-full` | `sslrootcert=/path/ca.pem` | `sslcert=…&sslkey=…` |
| MySQL, MariaDB | `ssl-mode=VERIFY_IDENTITY` | `ssl-ca=/path/ca.pem` | `ssl-cert=…&ssl-key=…` |
| SQL Server | Default | `trust_cert_ca=/path/ca.pem` | — |

Sentinel trusts the system CA store, the AWS RDS CA bundle, and the certificates in the top-level [`tls_extra_ca_certs`](/docs/sentinel/v2/reference/configuration/#top-level), unless the URL names its own CA. Amazon RDS and Aurora therefore verify with `verify-full` or `VERIFY_IDENTITY` and no CA file:

```toml
[database]
url = "postgres://sentinel:${DB_PASSWORD}@sentinel.abc123.eu-central-1.rds.amazonaws.com:5432/sentinel?sslmode=verify-full"
```

> **Warning:**
>
> PostgreSQL defaults to `sslmode=prefer` and MySQL and MariaDB to `ssl-mode=PREFERRED`: TLS when the server offers it, **without verifying the certificate**, and plain text when it doesn't. Anyone on the network path can read or impersonate the database. Set `verify-full` or `VERIFY_IDENTITY` for any database outside the host. SQL Server is the exception: it encrypts and verifies by default, and `trust_cert=true` or `encrypt=false` turn that off.

All modes and parameters are listed in [Database TLS](/docs/sentinel/v2/reference/configuration/#database-tls).

### Oracle

Sentinel passes the connect string after the last `@` of an `oracle://` URL unchanged to the Oracle Instant Client 23.26 in the image, so TLS is configured the Oracle way and `tls_extra_ca_certs` doesn't apply. User and password aren't percent-decoded.

- **Public CA:** use `tcps://host:port/service?ssl_server_dn_match=true`, as in the Oracle tab above. Instant Client verifies the server against the image's CA bundle.
- **Private CA or client certificates:** build an auto-login wallet (`cwallet.sso`) with `orapki` outside the image, which ships Instant Client Basic Light without `orapki`. Mount it read-only, readable by uid `65532`, and name its directory with `wallet_location=` in the connect string.
- **Net service name:** put the wallet and a `tnsnames.ora` in one directory and mount it at `/opt/oracle/instantclient/network/admin` (the default `TNS_ADMIN`) or point `TNS_ADMIN` at it. The URL then ends in the entry name, for example `@SENTINEL_DB`.

The wallet commands and a `tnsnames.ora` example are in [Oracle TLS](/docs/sentinel/v2/reference/configuration/#oracle-tls).

### ExotDB server

For an `https://` ExotDB server, set `[database] tls_ca` (PEM, trusted next to the system CA store) for a private CA, and `tls_cert` with `tls_key` for a client certificate. The keys take PEM contents, so read them from mounted files with `file:` expansion, as in the ExotDB server tab above. They apply only to an `https://` URL; with any other `url` they stop startup.

## Configure the KV store

### Built-in KV store

With `[redis] nodes` empty and no `[redis.sentinel]`, Sentinel runs a built-in KV store on its own database. It needs no setup, but it has no pub/sub, so it supports **one replica only**: other replicas can't see each other, reload their API-key cache only every `refresh_interval`, and keep separate rate limits and replay protection.

On an external database, the built-in store goes down with the database. A replica keeps verifying challenges through a database outage, but replay protection then holds per replica only. Set `[redis] nodes` if that matters to you.

The built-in store isn't available with an ExotDB server; Sentinel refuses to start with `redis.nodes must be set when database.url points to an ExotDB server (use its [kvstore] port or a Redis).`

Two keys tune the built-in store:

```toml
[redis]
builtin     = "database"   # or "memory": state is lost when Sentinel restarts
synchronous = "full"       # SQLite file only; "normal" skips the disk sync per commit
```

`builtin = "memory"` keeps replay protection and rate-limit state in process memory, and `doctor` warns about it. `synchronous = "normal"` writes faster and survives a process restart, but a power loss or OS crash can drop the last writes, so a solved challenge could be accepted again.

### Redis or Valkey

Set `[redis] nodes` to use an external Redis-protocol server. Every replica must use the same server and the same `key_prefix`. Which versions to run and how much memory to give it are covered in [Redis or Valkey](/docs/sentinel/v2/install/#redis-or-valkey).

Sentinel supports three topologies:

<Tabs padding={false}>

### Standalone

```toml
# One entry: a single Redis server.
[redis]
nodes = ["redis://:${file:/run/secrets/redis_password}@redis.internal:6379"]
```

### Redis Cluster

```toml
# Several entries: seed nodes of one Redis Cluster.
[redis]
nodes = [
"redis://:${file:/run/secrets/redis_password}@redis-0.internal:6379",
"redis://:${file:/run/secrets/redis_password}@redis-1.internal:6379",
"redis://:${file:/run/secrets/redis_password}@redis-2.internal:6379",
]
```

### Redis Sentinel

```toml
# [redis] nodes stays empty.
[redis.sentinel]
nodes    = ["redis://:${REDIS_SENTINEL_PASSWORD}@s1:26379", "redis://:${REDIS_SENTINEL_PASSWORD}@s2:26379", "redis://:${REDIS_SENTINEL_PASSWORD}@s3:26379"]
master   = "mymaster"
password = "${file:/run/secrets/redis_password}"
```

With **Redis Sentinel**, `nodes` lists the Sentinels, and the userinfo in those URLs, the same on every entry, authenticates to the Sentinels. `master` is the name the Sentinels monitor, and `username` and `password` authenticate to the master. Every new connection asks the Sentinels for the current master. After a failover, the demoted master rejects writes with `READONLY`; Sentinel drops that connection and the next command reconnects to the new master. Redis Sentinel doesn't work with an ExotDB server's KV port. The details are in [`[redis.sentinel]`](/docs/sentinel/v2/reference/configuration/#redissentinel).

An ExotDB server's `[kvstore]` port also works as a `nodes` entry. If that server keeps its KV store in memory, its keys are lost when it restarts, and `doctor` warns about it.

### Key prefix and timeout

`[redis] key_prefix` (default `"sentinel:"`) is prepended to every key and pub/sub channel, in the built-in store too. Replicas see each other only under the same prefix, so deployments sharing one Redis stay apart with different prefixes, for example `"prod:sentinel:"`. The prefix is 1–64 printable ASCII characters without spaces or curly braces. Changing it starts with empty counters, penalties, and replay claims.

`[redis] timeout` (default `"2s"`, from `1ms` to `60s` in whole milliseconds) bounds the connect and response time of every KV command, the built-in store included, and of each Redis Sentinel lookup and pub/sub subscribe. A command that times out counts as a KV failure (see [below](#when-the-kv-store-is-unavailable)). Keep it short: until Sentinel gives up on an unresponsive store, each KV call on the request path can wait this long.

### Redis TLS

Use `rediss://` URLs. Sentinel verifies the server against the system CA store, `tls_extra_ca_certs`, and `[redis] tls_ca`:

```toml
[redis]
nodes    = ["rediss://:${file:/run/secrets/redis_password}@redis.internal:6380"]
tls_ca   = "${file:/run/secrets/redis-ca.pem}"       # private CA only
tls_cert = "${file:/run/secrets/redis-client.pem}"   # client certificate, with tls_key
tls_key  = "${file:/run/secrets/redis-client.key}"
```

The `tls_*` keys take PEM contents and need every node to be `rediss://` (with `[redis.sentinel]`, every Sentinel address); otherwise they stop startup. With `[redis.sentinel]`, `rediss://` Sentinel URLs also connect to the master over TLS. Appending `#insecure` (`rediss://redis.internal:6380/#insecure`) skips certificate verification; use it only for testing.

### When the KV store is unavailable

A replica starts while an external KV store is unreachable, and keeps serving. A KV store that rejects the connection, for example with a wrong password, still stops startup. While the store is down:

- **The widget endpoints keep working.** `GET /v2/challenge`, `POST /v2/verify`, and `POST /v2/verify/signature` keep replay protection in the replica's memory, so each solution is accepted once per replica. Every 5 seconds the replica checks the store, and writes the claims it kept once the store answers.
- **After 3 consecutive failures**, the replica stops calling the store on the request path until it answers again, so requests don't wait for timeouts. Meanwhile the penalty reads as `0`, and header entropy, the HIS replay check, and similarity campaigns are skipped.
- **Rate limits** fall back to per-replica counters in memory. `[rate_limit] memory_fallback = false` lets requests through without a limit instead; see [`[rate_limit]`](/docs/sentinel/v2/reference/configuration/#rate_limit).
- **Sign-in, OTP, passkeys, and step-up verification** fail, because their replay checks need the store.

Sentinel raises the [`kv_unavailable`](/docs/sentinel/v2/reference/configuration/#alertskv_unavailable) and [`db_unavailable`](/docs/sentinel/v2/reference/configuration/#alertsdb_unavailable) alerts during outages. How the API-key cache keeps widget endpoints working while the database is down is covered in [API keys & access levels](/docs/sentinel/v2/concepts/api-keys/#when-the-database-is-down).

## Verify it works

Run `doctor` inside the running container, so it uses the server's configuration, environment, and network. It connects to the database and the KV store and reports how fast they answer:

```bash
docker exec sentinel /usr/local/bin/altcha-sentinel doctor
```

```text
altcha-sentinel 2.0.0-beta.1 doctor, config: sentinel.toml

storage
ok    storage.db                                   The database answered in 0.04 ms.
ok    storage.schema                               The database schema is current.
ok    storage.kv                                   The KV store answered in 0.11 ms.
skip  storage.clickhouse                           Request logs are stored in the database, not ClickHouse.
…
```

`storage.schema` counts the schema changes that this version of Sentinel would apply at startup. With `[database] auto_migrate = false`, pending changes fail the check, because the server would refuse to start; see [Check the schema](/docs/sentinel/v2/operations/schema-migrations/#check-the-schema).

When `doctor` runs before the first start with embedded SQLite, for example in a one-off `docker run --rm` container, it skips the storage checks with `The SQLite database ./data/sentinel.db does not exist yet; the server creates it at startup.` An unreachable database or KV store fails the check, and `doctor` exits with `1`:

```text
storage
FAIL  storage.db                                   The database is unreachable: server error 500 BACKEND_ERROR: Backend error: pool timed out while waiting for an open connection.
                                                 hint: Check the [database] url and that the database server is up.
skip  storage.schema                               The database is unreachable; see storage.db.
FAIL  storage.kv                                   The KV store is unreachable: Connection refused (os error 61).
                                                 hint: Check [redis] nodes or [redis.sentinel] and that the KV server is up.
```

On a running server, root users run the same checks under **Doctor → Run checks** in the admin UI. For monitoring, `GET /.health` on the main listener reports whether this replica reaches its storage:

```bash
curl -s http://localhost:3000/.health
```

```json
{"checks":{"db":{"ok":true},"kv":{"ok":true}},"status":"healthy"}
```

It answers `503` with `"status": "unhealthy"` when a check fails, and only to the client IPs allowed by `[server] health_allowed_ips`. Don't use it as a liveness or readiness probe; see [Health endpoint](/docs/sentinel/v2/reference/configuration/#health-endpoint).

## Related

- [`[database]`](/docs/sentinel/v2/reference/configuration/#database), [`[redis]`](/docs/sentinel/v2/reference/configuration/#redis), and [`[redis.sentinel]`](/docs/sentinel/v2/reference/configuration/#redissentinel) in the Configuration reference
- [Install: production essentials](/docs/sentinel/v2/install/#production-essentials)
- [Schema migrations](/docs/sentinel/v2/operations/schema-migrations/)
- [Server secrets](/docs/sentinel/v2/configure/secrets/)
- [Clustering & high availability](/docs/sentinel/v2/operations/clustering/)
- [Snapshots](/docs/sentinel/v2/operations/snapshots/)
- [`doctor` in the CLI reference](/docs/sentinel/v2/reference/cli/#doctor)
- [Plans & Licensing](/docs/sentinel/v2/plans/)
