# Server secrets

> **Sentinel v2 (beta):** This page documents Sentinel v2, a beta rewrite that may change before the stable release. It does not apply to Sentinel v1, the current stable version: see the [Sentinel v1 documentation](/docs/sentinel.md).

Sentinel signs, encrypts, and authenticates with a small set of server secrets in `[secrets]`. Set them explicitly in production, keep them out of `sentinel.toml`, and give every replica the same values. This page also covers `[bootstrap] root_password`, the password of the first root user.

## The four secrets

| Key | Protects | Format | Needed |
| --- | --- | --- | --- |
| `jwt_secret` | Signs (HS256) the session tokens of the admin UI and the management API, and the short-lived tokens of the second sign-in step and of step-up verification. | At least 24 characters | Always |
| `altcha_hmac_secret` | Signs the ALTCHA challenge on the admin sign-in page. Widget challenges and `verificationData` are signed with the secret of the API key instead. | At least 24 characters | Always |
| `encryption_key` | Encrypts (AES-256-GCM) secrets stored in the database: API key secrets and users' TOTP secrets. It also encrypts the OAuth sign-in state cookie and derives the daily [EDK](/docs/sentinel/v2/reference/configuration/#edk) device keys. | 64 hex characters (32 bytes) | Always |
| `vapid_private_key` | Signs web push notifications of [alerts](/docs/sentinel/v2/operations/alerts/). | Base64url P-256 private key | Only when `[alerts] enabled` (the default) and `[alerts.push] subject` is set |

Types and defaults are in [`[secrets]`](/docs/sentinel/v2/reference/configuration/#secrets) in the Configuration reference.

## Generate the values

Generate each secret once and store it as a file. `openssl rand -hex 32` prints 64 hex characters, which satisfies all three length rules:

```bash
mkdir -p secrets
openssl rand -hex 32 > secrets/jwt_secret
openssl rand -hex 32 > secrets/altcha_hmac_secret
openssl rand -hex 32 > secrets/encryption_key
```

For web push, generate a VAPID key pair and keep only the private key. The `web-push` package prints both keys; copy the line under **Private Key**:

```bash
npx web-push generate-vapid-keys
```

Without Node.js, extract the raw P-256 private key with OpenSSL and encode it as base64url:

```bash
openssl ecparam -name prime256v1 -genkey -noout \
| openssl ec -outform DER 2>/dev/null \
| tail -c +8 | head -c 32 \
| base64 | tr '+/' '-_' | tr -d '=\n' > secrets/vapid_private_key
```

Either way the result is 43 characters, such as `peQTcX48W3YFar4-6cxaYMHYFGPhO9Ffrq91Yv92UY0`. Sentinel derives the public key that browsers subscribe with from it.

## Pass them to Sentinel

Reference each file with `${file:…}`. Sentinel removes one trailing newline, so files written by `openssl` work as they are:

```toml
[secrets]
jwt_secret         = "${file:/run/secrets/jwt_secret}"
altcha_hmac_secret = "${file:/run/secrets/altcha_hmac_secret}"
encryption_key     = "${file:/run/secrets/encryption_key}"
vapid_private_key  = "${file:/run/secrets/vapid_private_key}"   # only with [alerts.push] subject
```

Environment variables work too (`"${JWT_SECRET}"`), but files keep the values out of process listings and container inspection output. How `${…}` references expand is explained in [The configuration file](/docs/sentinel/v2/configure/#environment-variables-and-secret-files).

With Docker Compose, declare the files as secrets; Compose mounts each one at `/run/secrets/<name>`:

```yaml
services:
sentinel:
secrets: [jwt_secret, altcha_hmac_secret, encryption_key]

secrets:
jwt_secret:
file: ./secrets/jwt_secret
altcha_hmac_secret:
file: ./secrets/altcha_hmac_secret
encryption_key:
file: ./secrets/encryption_key
```

With Kubernetes, create a Secret from the same files and mount it as a volume at `/run/secrets`:

```bash
kubectl create secret generic sentinel-secrets \
--from-file=jwt_secret=secrets/jwt_secret \
--from-file=altcha_hmac_secret=secrets/altcha_hmac_secret \
--from-file=encryption_key=secrets/encryption_key
```

```yaml
# StatefulSet or Deployment, pod template spec
automountServiceAccountToken: false   # the token mount at /var/run/secrets/… fails under a read-only /run/secrets
containers:
- name: sentinel
volumeMounts:
  - name: secrets
    mountPath: /run/secrets
    readOnly: true
volumes:
- name: secrets
secret:
  secretName: sentinel-secrets
```

The container runs as user `65532`, so the mounted files must be readable by it. In the image, `/var/run` links to `/run`, so with the Secret mounted at `/run/secrets` Kubernetes can't add its service account token there and the pod fails to start with `read-only file system`; Sentinel doesn't use the token, so turn it off. Complete examples are in [Docker Compose](/docs/sentinel/v2/install/docker-compose/) and [Kubernetes](/docs/sentinel/v2/install/kubernetes/).

## Validate them

`doctor --config-only` expands the references and checks every secret without starting the server. Run it with the same image, config file, and secret files as the server. The image's entrypoint is the `altcha-sentinel` binary, so the arguments after the image name go to it.

With the `secrets` folder from [Generate the values](#generate-the-values), put this `sentinel.toml` next to it:

```toml
[secrets]
jwt_secret         = "${file:/run/secrets/jwt_secret}"
altcha_hmac_secret = "${file:/run/secrets/altcha_hmac_secret}"
encryption_key     = "${file:/run/secrets/encryption_key}"
```

Then mount both into a one-off container and run the check:

```bash
docker run --rm \
-v "$PWD/sentinel.toml:/home/nonroot/sentinel.toml:ro" \
-v "$PWD/secrets:/run/secrets:ro" \
ghcr.io/altcha-org/sentinel:2.0.0-beta.1 doctor --config-only
```

```text
altcha-sentinel 2.0.0-beta.1 doctor, config: sentinel.toml

config
ok    config.file   Loaded sentinel.toml.
ok    config.valid  The configuration passes the startup checks.

2 ok, 0 warn, 0 fail, 0 skipped in 15 ms
```

With Docker Compose, `docker compose run --rm sentinel doctor --config-only` runs the same check with the service's mounts and secrets.

A value in the wrong format fails the check, the command exits with code `1`, and the server refuses to start with the same message. Here, `secrets/encryption_key` holds the output of `openssl rand -base64 32`:

```text
altcha-sentinel 2.0.0-beta.1 doctor, config: sentinel.toml

config
ok    config.file             Loaded sentinel.toml.
FAIL  config.valid (secrets)  secrets.encryption_key must be 64 hex characters (32 bytes).
                            hint: Fix this setting; the server does not start with it.

1 ok, 0 warn, 1 fail, 0 skipped in 2 ms
The other checks run once the configuration is valid.
```

The other format messages are `secrets.jwt_secret must be at least 24 characters` and `secrets.altcha_hmac_secret must be at least 24 characters`. A base64 value is long enough for those two but is rejected as `encryption_key`. A missing or unmounted file fails `config.file` with `reading secret file '/run/secrets/encryption_key': No such file or directory`. The VAPID key is checked when the server starts.

## Generated secrets on embedded SQLite

With the embedded SQLite database (`[database] url` unset), every secret you leave unset is generated on first start and saved to `<data_dir>/secrets.json`. The VAPID key is generated only when web push is on. The startup log notes it:

```text
INFO altcha_sentinel::secrets: generated secrets path=./data/secrets.json
```

The file is created with mode `600` and read on every start. Values set in `[secrets]` always take precedence over the file. `doctor` warns (`security.secrets_file`) when the file is readable or writable by group or others.

Generated secrets suit a trial or a single node. Back up `secrets.json` together with `sentinel.db`: a database restored without its `encryption_key` has unreadable API key secrets.

With any other database (PostgreSQL, MySQL, MariaDB, SQL Server, Oracle, or a remote ExotDB server), Sentinel generates nothing. Every replica must read the same values, so each secret must be set in the configuration. A missing one stops startup and fails `doctor --config-only`:

```text
Error: secrets.altcha_hmac_secret, secrets.encryption_key must be set when not using embedded SQLite
```

When web push is on, a missing VAPID key is listed as `secrets.vapid_private_key (alerts.push.subject is set)`. To move from embedded SQLite to another database, copy the values from `secrets.json` into your secret files first. Otherwise, the API key secrets in your data become unreadable after the move. See [Databases & KV store](/docs/sentinel/v2/configure/databases/).

## Share them across replicas

Every replica must use identical values. Sentinel doesn't compare them between replicas, and a mismatch looks like random failures behind a load balancer. A session issued by one replica returns `401 UNAUTHORIZED` on another, and an API key created on one replica can't be decrypted on another. See [Clustering & high availability](/docs/sentinel/v2/operations/clustering/).

## Rotating a secret

There is no rotation command and no grace period: Sentinel knows one value of each secret. Changing a value and restarting has these effects:

| Secret | What breaks | Recovery |
| --- | --- | --- |
| `jwt_secret` | Every session token, sign-in in progress, and step-up token becomes invalid. Requests with them get `401 UNAUTHORIZED`. | Users sign in again. Nothing is lost. |
| `altcha_hmac_secret` | Sign-in challenges issued before the restart fail. | Reload the sign-in page. Nothing is lost. |
| `encryption_key` | Every stored API key secret and TOTP secret becomes unreadable. Requests that use such a key, and reading or revealing it, fail with `500 INTERNAL_ERROR` (`failed to decrypt stored secret` in the log). Users with TOTP can't finish signing in. OAuth sign-ins in progress fail with `OAUTH_INVALID_STATE`. Every EDK changes, as at a day rollover. | Create new API keys and update the widgets and backends that use them. A root user turns off two-factor authentication for each affected user (`POST /v2/users/<id>/_disable_otp`), who then sets it up again. |
| `vapid_private_key` | Every web push subscription stops receiving notifications. | Each user turns **Push notifications** on again in the browser. |

To sign out a single user, you don't need to rotate `jwt_secret`: use **Log out everywhere** for that user, which ends every session they have. See [Revoke sessions](/docs/sentinel/v2/policies/accounts-users/#revoke-sessions).

> **Danger:**
>
> Treat `encryption_key` as permanent and back it up with the database. Snapshots store API key secrets and TOTP secrets encrypted with it, so restoring a snapshot into a server with another key leaves every API key unusable. `altcha-sentinel migrate` encrypts the converted secrets with the key of the configuration you run it with, so run it with the restoring server's configuration. See [Snapshots & backups](/docs/sentinel/v2/operations/snapshots/) and [Migrating data](/docs/sentinel/v2/upgrade/migrate-data/).

## The bootstrap root password

`[bootstrap] root_password` sets the password of the `root` user, which Sentinel creates when the database has no users:

```toml
[bootstrap]
root_password = "${file:/run/secrets/root_password}"   # at least 8 characters
```

```bash
openssl rand -base64 24 > secrets/root_password
```

The password must have at least 8 characters; a shorter one stops that first start with `bootstrap.root_password must be at least 8 characters`. When the key is unset, Sentinel generates a 24-character password and logs it once as a warning:

```text
WARN altcha_sentinel::bootstrap: created root user with generated password; change it username="root" password=98276a165e6faba41057d805
```

The key is read only while the root user is created. Once any user exists, changing or removing it has no effect; change the password in the admin UI under **user menu → Change password** instead. With several replicas starting at once, one replica creates the user under a lock, so the password is applied and logged once. Root users and their privileges are explained in [Accounts, users & roles](/docs/sentinel/v2/concepts/accounts-users/).

> **Warning:**
>
> A generated password stays in your log storage. Sign in, change it, and turn on two-factor authentication for root. `doctor` warns (`security.root_mfa`) about root users without it.

## Related

- [`[secrets]`](/docs/sentinel/v2/reference/configuration/#secrets) and [`[bootstrap]`](/docs/sentinel/v2/reference/configuration/#bootstrap) in the Configuration reference
- [The configuration file](/docs/sentinel/v2/configure/)
- [Install: production essentials](/docs/sentinel/v2/install/#production-essentials)
- [Databases & KV store](/docs/sentinel/v2/configure/databases/)
- [Snapshots & backups](/docs/sentinel/v2/operations/snapshots/)
- [API keys & access levels](/docs/sentinel/v2/concepts/api-keys/)
- [Production checklist](/docs/sentinel/v2/security/production-checklist/)
