# Doctor & troubleshooting

> **Sentinel v2 (beta):** This page documents Sentinel v2, a beta rewrite that may change before the stable release. It does not apply to Sentinel v1, the current stable version: see the [Sentinel v1 documentation](/docs/sentinel.md).

The doctor checks a Sentinel deployment and tells you what is wrong and how to fix it. Run it after a deploy, when something misbehaves, and before you ask for support. The second half of this page lists common symptoms with their causes and fixes.

The doctor comes in two forms:

| Form | Runs | Use it when |
| --- | --- | --- |
| `altcha-sentinel doctor` (CLI) | In a separate process, without the server. Validates the config, then checks storage, network, and the host. | Before a deploy, in CI, or when the server doesn't start. |
| `POST /v2/doctor/_run` and the **Doctor** page | Inside the running server, on the replica that answers. Adds the license, data, alert, and cluster checks. | The server is running and you want its live state. |

## Run the doctor from the command line

```bash
altcha-sentinel --config sentinel.toml doctor [--json] [--config-only]
```

`--config-only` only validates the config file, so it suits CI. `--json` prints the report as JSON. The command exits with `0` when no check failed (warnings included) and `1` otherwise. The [CLI reference](/docs/sentinel/v2/reference/cli/#doctor) lists the flags and exit codes.

The configuration comes first. When the file can't be loaded or fails a startup check, no other check runs:

```text
altcha-sentinel 2.0.0-beta.1 doctor, config: sentinel.toml

config
FAIL  config.file  loading config sentinel.toml: unknown field `url`, expected one of `nodes`, `sentinel`, `builtin`, `synchronous`, `key_prefix`, `timeout`, `tls_ca`, `tls_cert`, `tls_key`
                 in `redis`.
                 hint: Fix the file; the server does not start with it.

0 ok, 0 warn, 1 fail, 0 skipped in 17 ms
The other checks run once the configuration is valid.
```

With a valid file, `--config-only` prints two checks:

```text
altcha-sentinel 2.0.0-beta.1 doctor, config: sentinel.toml

config
ok    config.file   Loaded sentinel.toml.
ok    config.valid  The configuration passes the startup checks.

2 ok, 0 warn, 0 fail, 0 skipped in 21 ms
```

Without `--config-only`, it goes on to the checks that need no running server. Each line shows the status (`ok`, `warn`, `FAIL`, `skip`), the check ID, the message, and, for problems, a hint:

```text
altcha-sentinel 2.0.0-beta.1 doctor, config: sentinel.toml

storage
…
ok    storage.kv                                   The KV store answered in 0.47 ms.
skip  storage.clickhouse                           Request logs are stored in the database, not ClickHouse.
…
network
ok    network.host (https://directory.altcha.org)  https://directory.altcha.org answered HTTP 200 in 793 ms.
license
ok    license.server                               The licence server https://license.altcha.org is reachable.
…
security
warn  security.login_altcha                        Login does not require a solved ALTCHA challenge (auth.verify_altcha = false).
                                                 hint: Set [auth] verify_altcha = true; it guards password guessing with proof-of-work.
…
14 ok, 2 warn, 0 fail, 10 skipped in 819 ms
```

The CLI changes nothing except a probe file it writes and deletes in `data_dir`. It doesn't migrate the schema, create the database or `secrets.json`, or take a license instance slot, so it's safe to run next to a live server.

### In a container

Run the doctor where the server runs, with the server's config file, environment, and mounts, so it sees the same `${…}` values, files, and network. Inside the running container:

```bash
docker exec sentinel /usr/local/bin/altcha-sentinel --config /etc/sentinel/sentinel.toml doctor
```

When the server doesn't start, use a one-off container with the same mounts and environment instead. The image runs as uid `65532`, so the mounted files must be readable by that user:

```bash
docker run --rm \
--env-file sentinel.env \
-v "$PWD/sentinel.toml:/etc/sentinel/sentinel.toml:ro" \
-v "$PWD/secrets:/run/secrets:ro" \
-v sentinel-data:/home/nonroot/data \
ghcr.io/altcha-org/sentinel:2.0.0-beta.1 --config /etc/sentinel/sentinel.toml doctor
```

On embedded SQLite, stop the server first or use `docker exec`, so two containers don't open the same database files.

## Run the doctor in the server

`POST /v2/doctor/_run` runs every check on the replica that answers. It needs a **root** user's session; API keys get `401 UNAUTHORIZED`, and other users `403 ROOT_PRIVILEGES_REQUIRED`. It works without a license, with an unlicensed database backend, and over the instance limit, so you can always find out what is wrong.

```bash
curl -X POST -H 'Authorization: Bearer <jwt>' \
https://sentinel.example.com/v2/doctor/_run
```

```json
{
"data": {
"replica": { "id": "rep_1k7pi08ke00i1uocj7h", "hostname": "sentinel-1", "version": "2.0.0-beta.1" },
"generatedAt": "2026-11-19T09:25:37.066553+00:00",
"cached": false,
"durationMs": 458,
"summary": { "ok": 26, "warn": 6, "fail": 1, "skipped": 7 },
"checks": [
  {
    "id": "license.status",
    "category": "license",
    "subject": null,
    "status": "fail",
    "message": "Licence status is none; challenges, verification and feature endpoints are refused.",
    "hint": "Set [license] file or key in the configuration and restart.",
    "details": { "licensee": null, "phase": "ready", "plan": null, "source": "baseline", "status": "none" },
    "durationMs": 0
  },
  …
]
}
}
```

A report is reused for 30 seconds (`"cached": true`), per replica. Checks run concurrently, and one that takes longer than 20 seconds fails with `check timed out after 20s`. A failing check doesn't fail the request: the response is still `200`. The schema is in the [API reference](/sentinel-api-docs-v2.html#tag/doctor/POST/v2/doctor/_run).

In a cluster, each request reaches one replica. Run it on each replica (for example through its own address) when only some of them misbehave.

### The Doctor page

In the admin UI, root users open **Doctor** in the sidebar. The page runs the checks when it opens and shows only warnings and failures by default; switch **Checks shown** to **All** to see the rest. **Run checks** runs them again, and **Copy report** and **Download report** export the JSON. The app also runs the doctor when it loads and shows the number of failed checks next to **Doctor** in the sidebar.

## Read the report

Each check has a status:

| Status | Meaning |
| --- | --- |
| `ok` | Works as expected. |
| `warn` | Works, but needs attention: a risky setting, a resource running low, a feature that is configured but has no effect. |
| `fail` | Broken. Something Sentinel needs doesn't work. |
| `skipped` | Doesn't apply to this configuration, for example `storage.clickhouse` without `[clickhouse] url`. |

The JSON report of the CLI has `version`, `config` (the file read, `null` with the defaults), `generatedAt`, `durationMs`, `summary`, and `checks`. The API's report has `replica` instead of `version` and `config`, plus `cached`. `summary` counts the checks per status. Each entry in `checks` has these fields:

| Field | Type | Description |
| --- | --- | --- |
| `id` | string | `category.name`, for example `storage.db`. Stable, so match on it in scripts. |
| `category` | string | The part of `id` before the dot. |
| `subject` | string \| null | The item checked, for checks with one result per item: the origin for `network.host`, the config section for a failing `config.valid`. `null` otherwise. |
| `status` | string | `ok`, `warn`, `fail`, or `skipped`. |
| `message` | string | One English sentence. The wording can change between versions, so don't match on it. |
| `hint` | string \| null | A suggested fix; `null` when there's nothing to do. |
| `details` | object | Check-specific data, such as latencies, counts, or the failing items; `{}` when there is none. |
| `durationMs` | integer | How long the check took. |

To list only the problems from a JSON report:

```bash
altcha-sentinel --config sentinel.toml doctor --json \
| jq -r '.checks[] | select(.status == "warn" or .status == "fail") | "\(.status) \(.id) \(.message)"'
```

Reports are safe to share: URLs appear only as their origin (`scheme://host[:port]`), and no keys, tokens, passwords, or secret values are read or shown. Usernames appear only in `security.root_mfa`, API key names and IDs only in `security.api_key_ips`, and IP addresses only in `security.client_ip`.

## Checks

Thresholds are fixed, not configurable. "CLI" and "API" say where a check runs. In the CLI, `storage.db` and `storage.kv` have no outage history, `storage.db` is skipped while the embedded SQLite database doesn't exist yet, and `alerts.repeating`, `security.root_mfa`, and `security.api_key_ips` need a reachable database with a current schema.

| Check | Runs in | Warns or fails when |
| --- | --- | --- |
| `storage.db` | CLI, API | The database doesn't answer within 5 seconds (fail), answers in 500 ms or more, or had an outage in the last 24 hours (warn). |
| `storage.schema` | CLI | The schema can't be read, or changes are pending with `[database] auto_migrate = false` (fail). See [Schema migrations](/docs/sentinel/v2/operations/schema-migrations/). |
| `storage.kv` | CLI, API | The KV store doesn't answer (fail), answers in 100 ms or more, had an outage in the last 24 hours, or is an ExotDB KV server in memory mode (warn). |
| `storage.clickhouse` | CLI, API | `SELECT 1` fails (fail). |
| `storage.object` | CLI, API | Listing the snapshot or EML prefix of `[storage]` fails (fail), or gets `403` while only EML attachments use it (warn). |
| `network.host` | CLI, API | One result per outbound origin (remote files, threat sources, IP lookup APIs): DNS, TCP, or HTTP fails (fail). Any HTTP response counts as reachable. |
| `license.status` | API | The functional API is refused: no license, expired, revoked, not yet valid (fail), or the license runtime is still starting (warn). |
| `license.expiry` | API | The license ends within 3 days (fail) or 14 days (warn). |
| `license.server` | CLI, API | The license server is unreachable (fail), or reachable while the online refresh keeps failing (warn, API only). Skipped with `[license] file`. |
| `license.instances` | API | This replica is over the `instances` limit (fail), holds a burst slot, or every slot is in use (warn). |
| `license.quotas` | API | The `accounts`, `users`, `api_keys`, or `security_groups` count has reached its limit (warn). |
| `license.environment` | API | The license is for non-production use (warn). |
| `license.features` | API | The config uses a feature the license lacks: ClickHouse, audit logs, OAuth, or LDAP (warn). |
| `data.remote_files` | API | A remote file has never downloaded (fail), its last check failed, or its schedule is over an hour overdue (warn). |
| `data.threat_sources` | API | A threat source failed to load or isn't loaded yet (warn). |
| `data.mmdb` | CLI, API | A configured local `.mmdb` file is missing (fail) or older than 30 days (warn). |
| `data.similarity` | API | The embedding model failed or isn't loaded yet (warn). |
| `data.api_key_cache` | API | The API key cache is stale, or there are more live keys than `max_keys` (warn). |
| `alerts.delivery` | API | The last webhook or email delivery failed (fail), or alerts are disabled or only logged (warn). |
| `alerts.repeating` | CLI, API | A `high` system alert repeated in the last 24 hours (fail), or a lower-severity one did (warn). |
| `alerts.dropped` | API | This replica dropped alerts because its queue was full (warn). |
| `cluster.peers_storage` | API | Another replica can't reach the database or KV store (fail), or had an outage in the last 24 hours (warn). |
| `cluster.versions` | API | Replicas run different versions (warn). |
| `cluster.hostnames` | API | Two replicas share a hostname (fail). |
| `cluster.visibility` | API | The license counts more instances than this replica sees, so replicas can't see each other (warn). |
| `system.clock` | CLI, API | The clock is off by 60 seconds or more (fail) or 10 seconds or more (warn) from the probed servers and, in the API, the other replicas. |
| `system.data_dir` | CLI, API | `data_dir` is missing or not writable, or has less than 512 MiB or 2 % free (fail); less than 2 GiB or 10 % free (warn). The CLI warns instead of failing when `data_dir` doesn't exist yet. |
| `system.memory_limit` | CLI, API | No memory limit was found, so the `memory` alert never fires (warn). |
| `system.request_logs` | API | This replica lost request logs since it started (warn). |
| `security.client_ip` | CLI, API | `client_ip_header` is set without `trusted_proxies`, or (API) this request came through a proxy that isn't in `trusted_proxies` (warn). |
| `security.login_altcha` | CLI, API | `[auth] verify_altcha = false` (warn). |
| `security.prometheus` | CLI, API | The Prometheus listener is on a non-loopback address without `bearer_token` or `allowed_ips` (warn). |
| `security.admin_ui` | CLI, API | The admin UI listener is on a non-loopback address without `allowed_ips` (warn). |
| `security.health` | CLI, API | The main listener is on a non-loopback address and `[server] health_allowed_ips` is empty (warn). |
| `security.root_mfa` | CLI, API | An active root user has neither TOTP nor a passkey (warn). |
| `security.api_key_ips` | CLI, API | A `restricted` or `full` API key can be used from any IP, because no live rule gates on `ip` (warn). |
| `security.secrets_file` | CLI, API | `<data_dir>/secrets.json` is readable or writable by group or others (warn). |
| `security.build` | CLI, API | The binary isn't a production build (warn). |
| `config.file` | CLI | The file can't be read, isn't valid TOML, has an unknown key, or a `${…}` reference doesn't resolve (fail); no config file was found, so the defaults apply (warn). |
| `config.valid` | CLI | A startup check rejects a value, with one result per error and the config section as `subject` (fail). |
| `config.ip_resolver` | CLI, API | An `ip-api` provider has no `key`, or an `ipstack` provider has `https = false` (warn). |
| `config.passkeys` | CLI, API | The auth base URL's host isn't a domain name, so passkeys are disabled (warn). |
| `config.audio` | CLI, API | The code-challenge text-to-speech assets weren't found, so code challenges have no audio (warn). |

The doctor isn't a health check. For container and orchestrator probes, use `GET /` or `altcha-sentinel healthcheck`; see [Monitoring & metrics](/docs/sentinel/v2/operations/monitoring/).

## Symptoms and fixes

What each error code means is in [Error codes](/docs/sentinel/v2/reference/errors/). This table maps what you see to the usual cause.

| Symptom | Cause | Fix |
| --- | --- | --- |
| `403 LICENSE_NOT_ACTIVE` on `/v2/challenge`, `/v2/verify`, `/v2/classifier`, and other functional routes | No active license; `data.status` says why (`none`, `expired`, `refresh_overdue`, …). | Set `[license] key` or `file` and restart. Check `license.status` and `license.expiry`; see [Configure the license](/docs/sentinel/v2/configure/licensing/). |
| `403 BEARER_AUTH_PUBLIC_API_KEY` | A `restricted` or `full` key was sent by ID (Bearer or `apiKey`). | Send it with Basic auth, `-u 'key_…:sec_…'`. |
| `403 PUBLIC_API_KEY_REQUIRED`, `NON_PUBLIC_API_KEY_REQUIRED`, `BASIC_AUTH_NON_PUBLIC_API_KEY`, `401 API_KEY_REQUIRED` | The credential doesn't fit the endpoint: a server key in the widget, a `public` key on a backend endpoint, or a session token on a detection endpoint. | Widget endpoints take a `public` key by ID; backend endpoints a `restricted` or `full` key with its secret; management endpoints a session. See the [credential matrix](/docs/sentinel/v2/concepts/api-keys/#credential-matrix). |
| `403 BACKEND_NOT_LICENSED` on most `/v2` routes | `[database]` or `[redis]` uses a backend the license doesn't include. Without a usable license, only embedded SQLite is allowed. | Get a license with that backend, or switch to a licensed one. Sign-in, `/v2/license*`, and `/v2/doctor*` keep working. |
| `403 LICENSE_LIMIT_REACHED` when creating an account, user, API key, or security group | The license's limit for that kind is reached. `license.quotas` shows which one. | Delete unused records (deleted ones don't count) or get a higher limit. |
| `503 LICENSE_INSTANCE_LIMIT`, or a replica exits at startup | More replicas than the license's `instances`, or replicas sharing a `HOSTNAME`. | Give each replica a unique, stable `HOSTNAME`; stop surplus replicas. Check `license.instances` and `cluster.hostnames`; see [Clustering & HA](/docs/sentinel/v2/operations/clustering/). |
| CORS error in the browser console | The page calls an endpoint that isn't a widget route (detection and management APIs send no CORS headers), the proxy adds or strips CORS headers, or the real response is an error from a proxy. | Call detection APIs from your server, not the browser. Let Sentinel set CORS headers; see [CORS](/docs/sentinel/v2/configure/listeners/#cors). Check the response status in the network tab: a `421` or a proxy error page looks like a CORS failure. |
| `421 Misdirected Request` with the body `misdirected request` | The request's host isn't in the listener's `allowed_hosts`. Behind a proxy that rewrites `Host`, Sentinel sees the internal host. | Add the public host to `allowed_hosts`, or set `forwarded_host_header` and `trusted_proxies`; see [Host and allowed hosts](/docs/sentinel/v2/configure/listeners/#host-and-allowed-hosts). |
| Every request has the proxy's IP: rate limits hit all users at once, IP rules and request logs show one address | The proxy isn't in `trusted_proxies`, so `X-Forwarded-For` is ignored. `security.client_ip` warns when the doctor's own request shows this. | Add the proxy's addresses to `trusted_proxies` on that listener, or set `client_ip_header` for a CDN; see [Client IP](/docs/sentinel/v2/configure/listeners/#client-ip). |
| `403 IP_NOT_ALLOWED` on `GET /.health` | The prober isn't in `[server] health_allowed_ips` (private networks by default), or it sends forwarding headers through a proxy that isn't in `trusted_proxies`. | Add public probers to `health_allowed_ips`, or list the proxy in `trusted_proxies`; see [Health endpoint](/docs/sentinel/v2/reference/configuration/#health-endpoint). |
| Admin UI unreachable | `[server.admin_ui]` is off by default, and when enabled it listens on `127.0.0.1`, which isn't reachable through a container's published port. | Set `enabled = true` and `host = "0.0.0.0"`, restrict it with `allowed_ips`, and publish its port. Serve it over HTTPS: the session cookie is `Secure`, so sign-in works over plain HTTP only on `localhost`. See [Admin UI](/docs/sentinel/v2/configure/listeners/#admin-ui). |
| Startup stops with ``unknown field `…`, expected one of …`` | A misspelled key or a key in the wrong section; Sentinel rejects unknown keys. | Fix the key the message names; the next line names its section, for example ``in `server` ``. See [Unknown keys and invalid values stop startup](/docs/sentinel/v2/configure/#unknown-keys-and-invalid-values-stop-startup). |
| Startup stops with `invalid variable name` or `reading secret file '…': No such file or directory` | A `${…}` reference with an invalid name (such as `${JWT-SECRET}`), or a `${file:…}` path that isn't mounted. The error names the config key. | Fix the name or mount the file; see [Environment variables and secret files](/docs/sentinel/v2/configure/#environment-variables-and-secret-files). |
| Startup stops with `invalid type: string "3000", expected u16` | A number or boolean written as a `${…}` string; only strings are expanded. | Write numbers and booleans literally. |
| Lists or databases stay empty or stale | A remote file can't be downloaded: no outbound access, a proxy not set, DNS, or TLS. | Check `data.remote_files` for the file and `network.host` for its origin and failed `stage`. Set `HTTPS_PROXY` if outbound traffic needs a proxy; see [Remote files](/docs/sentinel/v2/configure/remote-files/) and [Air-gapped deployments](/docs/sentinel/v2/configure/air-gapped/). |
| OAuth sign-in returns to the login page with `OAUTH_FAILED`, or the provider reports a redirect URI mismatch | The redirect URI registered at the provider differs from `{base_url}/v2/auth/oauth/{provider}/callback`, where `base_url` is `[auth] base_url`, else an enabled `[server.admin_ui] base_url`, else `[server] base_url`. | Register the exact URI, including scheme, host, and any path prefix. See [Admin sign-in & SSO](/docs/sentinel/v2/security/sign-in/) and [OAuth redirect errors](/docs/sentinel/v2/reference/errors/#oauth-redirect-errors). |
| Startup stops with `timed out after 30000ms waiting for migration lock '__exotdb_migrate__'` | Another replica held the migration lock for over 30 seconds. | Restart the replica. See [Schema migrations](/docs/sentinel/v2/operations/schema-migrations/#errors). |
| Widget shows an error or never verifies | Usually one of the codes above on `/v2/challenge`: `LICENSE_NOT_ACTIVE`, a non-`public` key, `ACCESS_DENIED` from a security-group rule (for example an `origin` rule), or `RATE_LIMITED`. A clock off by seconds also breaks verification, because challenges have no clock leeway. | Open the network tab and read `code` in the failing response. Check `system.clock`. Test the key with `curl` as in [Widget integration](/docs/sentinel/v2/integrate/widget/#verify-it-works). |
| `500 INTERNAL_ERROR` | A server fault, such as the database or log store being unavailable. Details are only in the server log. | Check the log and run the doctor; `storage.db` and `storage.kv` usually show the cause. |

## Raise the log level

Sentinel logs to stdout. `[log] level` takes a filter directive: a default level, then `target=level` pairs for single modules. To log every 4xx and 5xx response with its error code, without raising the rest:

```toml
[log]
level = "info,sentinel::http=debug"   # "info,sentinel::http=trace" logs every response
```

Restart, and each failed request logs a line like this:

```text
2026-11-19T09:25:58.494974Z DEBUG http.server{… http.route="/v2/challenge" … http.response.status_code=403}: sentinel::http: request failed status=403 code="LICENSE_NOT_ACTIVE" latency_ms=0.204458
```

Set `[log] format = "json"` for log collectors. Lower the level again afterwards; `debug` and `trace` are verbose. See [`[log]`](/docs/sentinel/v2/reference/configuration/#log).

## Ask for support

Collect these before you open a support request:

1. **The doctor report.** The JSON from **Doctor → Download report**, or from `POST /v2/doctor/_run`, on the affected replica. If the server doesn't start, the output of `doctor --json` run with the server's config and mounts.

2. **The version and setup.** The image tag, the database and KV backend, the number of replicas, and what sits in front of Sentinel (proxy, CDN, load balancer).

3. **The failing request and response.** The method, path, and status, and the response body with its `code`. Remove API key secrets and session tokens.

4. **The server log** around the failure, with `[log] level = "info,sentinel::http=debug"` if the problem is with requests.

5. **The configuration**, with every secret removed. Secrets referenced as `${…}` are already safe to share; literal values are not.

> **Warning:**
>
> Doctor reports contain no secrets, but logs and config files can. Check them before you send them, and never send `secrets.json`, `[secrets]` values, license keys, or database passwords.

## Related

- [`doctor` in the CLI reference](/docs/sentinel/v2/reference/cli/#doctor)
- [Error codes](/docs/sentinel/v2/reference/errors/)
- [Monitoring & metrics](/docs/sentinel/v2/operations/monitoring/)
- [Production checklist](/docs/sentinel/v2/security/production-checklist/)
- [Run doctor in the API reference](/sentinel-api-docs-v2.html#tag/doctor/POST/v2/doctor/_run)
