Skip to main content
ALTCHA®

Databases & KV store

Sentinel stores its data in a database, which is the source of truth, and keeps short-lived shared state in a key-value (KV) store. This page helps you choose both and connect Sentinel to them. Every key and default is listed in [database] and [redis] of the Configuration reference.

Choose a backend

The database holds accounts, users, API keys, security groups, training data, threat entries, request logs (unless you send them to ClickHouse), and the license state. The KV store holds rate-limit counters, penalties, replay protection for solved challenges, and the pub/sub messages that replicas use to find each other. What each one stores is described in How Sentinel works.

Your deploymentDatabaseKV store
Evaluation, or one replica with a persistent volumeEmbedded SQLite (the default; nothing to configure)Built-in (the default)
One replica, database managed outside the containerPostgreSQL, MySQL, MariaDB, SQL Server, or OracleBuilt-in, or Redis to keep replay protection up while the database is down
Two or more replicasA shared external databaseRedis or Valkey (required)
An existing ExotDB serverExotDB server (http(s)://)Redis, or the ExotDB server's KV port (required)

Sentinel connects to the database you name in [database] url. The scheme selects the backend:

Backendurl schemeSupported versionsLicense feature
SQLite (embedded)sqlite:Bundled SQLite 3.54.1; the host's version doesn't matterexotdb.backend.sqlite (in the baseline)
PostgreSQLpostgres:// or postgresql://16 or laterexotdb.backend.postgres
MySQLmysql://8.0.13 or laterexotdb.backend.mysql
MariaDBmariadb://11.4 LTS or laterexotdb.backend.mariadb
Microsoft SQL Servermssql:// or sqlserver://2022 or laterexotdb.backend.mssql
Oracleoracle://21c or later (needs JSON_TRANSFORM); only in the -oracle imageexotdb.backend.oracle
ExotDB serverhttp:// or https://—Enforced by the ExotDB server

MariaDB is the only backend whose version Sentinel checks: it refuses to start on MariaDB older than 11.4, on a MariaDB server behind a mysql:// URL, and on a MySQL server behind a mariadb:// URL. Use the scheme that matches your server.

Oracle connections need the Oracle Instant Client, which only the -oracle image tag ships (for example 2.0.0-beta.1-oracle); see Container image. Use that image only with an Oracle database.

Configure the database

Set [database] url, with the password read from a secret. Without a URL, Sentinel uses sqlite:<data_dir>/sentinel.db. Each tab shows a complete [database] section:

# The default is sqlite:<data_dir>/sentinel.db; set url only to put the file elsewhere.
[database]
url = "sqlite:/data/sentinel/sentinel.db"
[database]
url = "postgres://sentinel:${DB_PASSWORD}@db.internal:5432/sentinel?sslmode=verify-full"
[database]
url = "mysql://sentinel:${DB_PASSWORD}@db.internal:3306/sentinel?ssl-mode=VERIFY_IDENTITY"
[database]
url = "mariadb://sentinel:${DB_PASSWORD}@db.internal:3306/sentinel?ssl-mode=VERIFY_IDENTITY"
# Encrypted and verified by default.
[database]
url = "mssql://sentinel:${DB_PASSWORD}@db.internal:1433/sentinel"
# -oracle image only. Everything after the last @ is an Oracle connect string.
[database]
url = "oracle://sentinel:${ORACLE_PASSWORD}@tcps://db.example.com:2484/FREEPDB1?ssl_server_dn_match=true"
[database]
url   = "https://exotdb.internal:3000"
name  = "sentinel"                                  # database on the server; default "sentinel"
token = "${file:/run/secrets/exotdb_token}"
tls_ca = "${file:/run/secrets/exotdb-ca.pem}"     # only for a private CA

[redis]
nodes = ["redis://:${file:/run/secrets/kv_password}@exotdb.internal:6379"]   # a Redis, or the server's [kvstore] port

Sentinel creates and migrates its tables at startup, so the database account needs the right to create tables. When several replicas start at once, one migrates while the others wait for its lock. If your database administrator applies schema changes instead, set [database] auto_migrate = false and generate the SQL with altcha-sentinel schema sql; see Schema migrations. With any backend other than embedded SQLite, set all three keys in [secrets] yourself; Sentinel doesn't generate them. See Server secrets.

Share a database with table_prefix

[database] table_prefix (default "") is prepended to every table, index, and relation name, and to Sentinel's database lock names. Use it to put Sentinel next to other applications in one database, or to run two deployments in one database:

[database]
url          = "postgres://sentinel:${DB_PASSWORD}@db.internal:5432/shared"
table_prefix = "sentinel_"

The prefix is up to 24 characters of a-z, 0-9, and _, starting with a letter. Anything else stops startup with database.table_prefix must be at most 24 characters of a-z 0-9 _, starting with a-z. Changing the prefix later starts on new, empty tables; the old ones stay in place.

SQLite on network storage

Embedded SQLite uses write-ahead logging (WAL) by default, so reads don't wait for writes. WAL keeps its index in shared memory (the -shm file next to the database), which works only when every process that opens the file runs on the same host. On a network file system such as AWS EFS, NFS, or an SMB share like Azure Files, it breaks. Switch the file to a rollback journal there:

[database]
sqlite_journal_mode = "delete"   # default "wal"

delete needs a file system whose file locks work, and reads and writes then block each other. Keep wal on local disks and block volumes such as Amazon EBS.

Sentinel sets the mode at startup, for the database and the built-in KV store, which share the file. Switching an existing file needs every other process to close it first: stop every container that uses it, and close any sqlite3 session on it. While another process keeps it open, Sentinel retries for 5 seconds, then stops with an error. The key is ignored by every other backend.

Network storage doesn't make SQLite a shared database: embedded SQLite still serves one replica. For more, use an external database.

Connection pool

Every SQL backend, SQLite included, uses a connection pool per replica:

[database]
pool_min        = 2       # connections kept open
pool_max        = 20      # most connections this replica opens
acquire_timeout = "5s"    # wait for a free connection, then the call fails
max_lifetime    = "30m"   # not SQLite
idle_timeout    = "10m"   # not SQLite

These are the defaults. Each replica has its own pool, so the database sees up to pool_max × replicas connections. Keep that below the database's connection limit, with room for rolling deploys, which briefly run extra replicas, and for other clients. Durations accept "0s" for no limit (acquire_timeout = "0s" waits forever). An ExotDB server sizes its own pool and ignores these keys; for it, request_timeout (default 60s) bounds each HTTP request and must stay above 30s while auto_migrate is on. See [database] for the limits of each key.

Database TLS

PostgreSQL, MySQL, MariaDB, and SQL Server take TLS options as query parameters of url, with file paths for certificates:

BackendVerify the serverPrivate CAClient certificate
PostgreSQLsslmode=verify-fullsslrootcert=/path/ca.pemsslcert=…&sslkey=…
MySQL, MariaDBssl-mode=VERIFY_IDENTITYssl-ca=/path/ca.pemssl-cert=…&ssl-key=…
SQL ServerDefaulttrust_cert_ca=/path/ca.pem—

Sentinel trusts the system CA store, the AWS RDS CA bundle, and the certificates in the top-level tls_extra_ca_certs, unless the URL names its own CA. Amazon RDS and Aurora therefore verify with verify-full or VERIFY_IDENTITY and no CA file:

[database]
url = "postgres://sentinel:${DB_PASSWORD}@sentinel.abc123.eu-central-1.rds.amazonaws.com:5432/sentinel?sslmode=verify-full"

All modes and parameters are listed in Database TLS.

Oracle

Sentinel passes the connect string after the last @ of an oracle:// URL unchanged to the Oracle Instant Client 23.26 in the image, so TLS is configured the Oracle way and tls_extra_ca_certs doesn't apply. User and password aren't percent-decoded.

  • Public CA: use tcps://host:port/service?ssl_server_dn_match=true, as in the Oracle tab above. Instant Client verifies the server against the image's CA bundle.
  • Private CA or client certificates: build an auto-login wallet (cwallet.sso) with orapki outside the image, which ships Instant Client Basic Light without orapki. Mount it read-only, readable by uid 65532, and name its directory with wallet_location= in the connect string.
  • Net service name: put the wallet and a tnsnames.ora in one directory and mount it at /opt/oracle/instantclient/network/admin (the default TNS_ADMIN) or point TNS_ADMIN at it. The URL then ends in the entry name, for example @SENTINEL_DB.

The wallet commands and a tnsnames.ora example are in Oracle TLS.

ExotDB server

For an https:// ExotDB server, set [database] tls_ca (PEM, trusted next to the system CA store) for a private CA, and tls_cert with tls_key for a client certificate. The keys take PEM contents, so read them from mounted files with file: expansion, as in the ExotDB server tab above. They apply only to an https:// URL; with any other url they stop startup.

Configure the KV store

Built-in KV store

With [redis] nodes empty and no [redis.sentinel], Sentinel runs a built-in KV store on its own database. It needs no setup, but it has no pub/sub, so it supports one replica only: other replicas can't see each other, reload their API-key cache only every refresh_interval, and keep separate rate limits and replay protection.

On an external database, the built-in store goes down with the database. A replica keeps verifying challenges through a database outage, but replay protection then holds per replica only. Set [redis] nodes if that matters to you.

The built-in store isn't available with an ExotDB server; Sentinel refuses to start with redis.nodes must be set when database.url points to an ExotDB server (use its [kvstore] port or a Redis).

Two keys tune the built-in store:

[redis]
builtin     = "database"   # or "memory": state is lost when Sentinel restarts
synchronous = "full"       # SQLite file only; "normal" skips the disk sync per commit

builtin = "memory" keeps replay protection and rate-limit state in process memory, and doctor warns about it. synchronous = "normal" writes faster and survives a process restart, but a power loss or OS crash can drop the last writes, so a solved challenge could be accepted again.

Redis or Valkey

Set [redis] nodes to use an external Redis-protocol server. Every replica must use the same server and the same key_prefix. Which versions to run and how much memory to give it are covered in Redis or Valkey.

Sentinel supports three topologies:

# One entry: a single Redis server.
[redis]
nodes = ["redis://:${file:/run/secrets/redis_password}@redis.internal:6379"]
# Several entries: seed nodes of one Redis Cluster.
[redis]
nodes = [
  "redis://:${file:/run/secrets/redis_password}@redis-0.internal:6379",
  "redis://:${file:/run/secrets/redis_password}@redis-1.internal:6379",
  "redis://:${file:/run/secrets/redis_password}@redis-2.internal:6379",
]
# [redis] nodes stays empty.
[redis.sentinel]
nodes    = ["redis://:${REDIS_SENTINEL_PASSWORD}@s1:26379", "redis://:${REDIS_SENTINEL_PASSWORD}@s2:26379", "redis://:${REDIS_SENTINEL_PASSWORD}@s3:26379"]
master   = "mymaster"
password = "${file:/run/secrets/redis_password}"

With Redis Sentinel, nodes lists the Sentinels, and the userinfo in those URLs, the same on every entry, authenticates to the Sentinels. master is the name the Sentinels monitor, and username and password authenticate to the master. Every new connection asks the Sentinels for the current master. After a failover, the demoted master rejects writes with READONLY; Sentinel drops that connection and the next command reconnects to the new master. Redis Sentinel doesn't work with an ExotDB server's KV port. The details are in [redis.sentinel].

An ExotDB server's [kvstore] port also works as a nodes entry. If that server keeps its KV store in memory, its keys are lost when it restarts, and doctor warns about it.

Key prefix and timeout

[redis] key_prefix (default "sentinel:") is prepended to every key and pub/sub channel, in the built-in store too. Replicas see each other only under the same prefix, so deployments sharing one Redis stay apart with different prefixes, for example "prod:sentinel:". The prefix is 1–64 printable ASCII characters without spaces or curly braces. Changing it starts with empty counters, penalties, and replay claims.

[redis] timeout (default "2s", from 1ms to 60s in whole milliseconds) bounds the connect and response time of every KV command, the built-in store included, and of each Redis Sentinel lookup and pub/sub subscribe. A command that times out counts as a KV failure (see below). Keep it short: until Sentinel gives up on an unresponsive store, each KV call on the request path can wait this long.

Redis TLS

Use rediss:// URLs. Sentinel verifies the server against the system CA store, tls_extra_ca_certs, and [redis] tls_ca:

[redis]
nodes    = ["rediss://:${file:/run/secrets/redis_password}@redis.internal:6380"]
tls_ca   = "${file:/run/secrets/redis-ca.pem}"       # private CA only
tls_cert = "${file:/run/secrets/redis-client.pem}"   # client certificate, with tls_key
tls_key  = "${file:/run/secrets/redis-client.key}"

The tls_* keys take PEM contents and need every node to be rediss:// (with [redis.sentinel], every Sentinel address); otherwise they stop startup. With [redis.sentinel], rediss:// Sentinel URLs also connect to the master over TLS. Appending #insecure (rediss://redis.internal:6380/#insecure) skips certificate verification; use it only for testing.

When the KV store is unavailable

A replica starts while an external KV store is unreachable, and keeps serving. A KV store that rejects the connection, for example with a wrong password, still stops startup. While the store is down:

  • The widget endpoints keep working. GET /v2/challenge, POST /v2/verify, and POST /v2/verify/signature keep replay protection in the replica's memory, so each solution is accepted once per replica. Every 5 seconds the replica checks the store, and writes the claims it kept once the store answers.
  • After 3 consecutive failures, the replica stops calling the store on the request path until it answers again, so requests don't wait for timeouts. Meanwhile the penalty reads as 0, and header entropy, the HIS replay check, and similarity campaigns are skipped.
  • Rate limits fall back to per-replica counters in memory. [rate_limit] memory_fallback = false lets requests through without a limit instead; see [rate_limit].
  • Sign-in, OTP, passkeys, and step-up verification fail, because their replay checks need the store.

Sentinel raises the kv_unavailable and db_unavailable alerts during outages. How the API-key cache keeps widget endpoints working while the database is down is covered in API keys & access levels.

Verify it works

Run doctor inside the running container, so it uses the server's configuration, environment, and network. It connects to the database and the KV store and reports how fast they answer:

docker exec sentinel /usr/local/bin/altcha-sentinel doctor
altcha-sentinel 2.0.0-beta.1 doctor, config: sentinel.toml

storage
  ok    storage.db                                   The database answered in 0.04 ms.
  ok    storage.schema                               The database schema is current.
  ok    storage.kv                                   The KV store answered in 0.11 ms.
  skip  storage.clickhouse                           Request logs are stored in the database, not ClickHouse.
…

storage.schema counts the schema changes that this version of Sentinel would apply at startup. With [database] auto_migrate = false, pending changes fail the check, because the server would refuse to start; see Check the schema.

When doctor runs before the first start with embedded SQLite, for example in a one-off docker run --rm container, it skips the storage checks with The SQLite database ./data/sentinel.db does not exist yet; the server creates it at startup. An unreachable database or KV store fails the check, and doctor exits with 1:

storage
  FAIL  storage.db                                   The database is unreachable: server error 500 BACKEND_ERROR: Backend error: pool timed out while waiting for an open connection.
                                                     hint: Check the [database] url and that the database server is up.
  skip  storage.schema                               The database is unreachable; see storage.db.
  FAIL  storage.kv                                   The KV store is unreachable: Connection refused (os error 61).
                                                     hint: Check [redis] nodes or [redis.sentinel] and that the KV server is up.

On a running server, root users run the same checks under Doctor → Run checks in the admin UI. For monitoring, GET /.health on the main listener reports whether this replica reaches its storage:

curl -s http://localhost:3000/.health
{"checks":{"db":{"ok":true},"kv":{"ok":true}},"status":"healthy"}

It answers 503 with "status": "unhealthy" when a check fails, and only to the client IPs allowed by [server] health_allowed_ips. Don't use it as a liveness or readiness probe; see Health endpoint.

Start typing to search...

↑ ↓ Navigate ↵ Select