Doctor & troubleshooting
The doctor checks a Sentinel deployment and tells you what is wrong and how to fix it. Run it after a deploy, when something misbehaves, and before you ask for support. The second half of this page lists common symptoms with their causes and fixes.
The doctor comes in two forms:
| Form | Runs | Use it when |
|---|---|---|
altcha-sentinel doctor (CLI) | In a separate process, without the server. Validates the config, then checks storage, network, and the host. | Before a deploy, in CI, or when the server doesn't start. |
POST /v2/doctor/_run and the Doctor page | Inside the running server, on the replica that answers. Adds the license, data, alert, and cluster checks. | The server is running and you want its live state. |
Run the doctor from the command line
altcha-sentinel --config sentinel.toml doctor [--json] [--config-only]--config-only only validates the config file, so it suits CI. --json prints the report as JSON. The command exits with 0 when no check failed (warnings included) and 1 otherwise. The CLI reference lists the flags and exit codes.
The configuration comes first. When the file can't be loaded or fails a startup check, no other check runs:
altcha-sentinel 2.0.0-beta.1 doctor, config: sentinel.toml
config
FAIL config.file loading config sentinel.toml: unknown field `url`, expected one of `nodes`, `sentinel`, `builtin`, `synchronous`, `key_prefix`, `timeout`, `tls_ca`, `tls_cert`, `tls_key`
in `redis`.
hint: Fix the file; the server does not start with it.
0 ok, 0 warn, 1 fail, 0 skipped in 17 ms
The other checks run once the configuration is valid.With a valid file, --config-only prints two checks:
altcha-sentinel 2.0.0-beta.1 doctor, config: sentinel.toml
config
ok config.file Loaded sentinel.toml.
ok config.valid The configuration passes the startup checks.
2 ok, 0 warn, 0 fail, 0 skipped in 21 msWithout --config-only, it goes on to the checks that need no running server. Each line shows the status (ok, warn, FAIL, skip), the check ID, the message, and, for problems, a hint:
altcha-sentinel 2.0.0-beta.1 doctor, config: sentinel.toml
storage
…
ok storage.kv The KV store answered in 0.47 ms.
skip storage.clickhouse Request logs are stored in the database, not ClickHouse.
…
network
ok network.host (https://directory.altcha.org) https://directory.altcha.org answered HTTP 200 in 793 ms.
license
ok license.server The licence server https://license.altcha.org is reachable.
…
security
warn security.login_altcha Login does not require a solved ALTCHA challenge (auth.verify_altcha = false).
hint: Set [auth] verify_altcha = true; it guards password guessing with proof-of-work.
…
14 ok, 2 warn, 0 fail, 10 skipped in 819 msThe CLI changes nothing except a probe file it writes and deletes in data_dir. It doesn't migrate the schema, create the database or secrets.json, or take a license instance slot, so it's safe to run next to a live server.
In a container
Run the doctor where the server runs, with the server's config file, environment, and mounts, so it sees the same ${…} values, files, and network. Inside the running container:
docker exec sentinel /usr/local/bin/altcha-sentinel --config /etc/sentinel/sentinel.toml doctorWhen the server doesn't start, use a one-off container with the same mounts and environment instead. The image runs as uid 65532, so the mounted files must be readable by that user:
docker run --rm \
--env-file sentinel.env \
-v "$PWD/sentinel.toml:/etc/sentinel/sentinel.toml:ro" \
-v "$PWD/secrets:/run/secrets:ro" \
-v sentinel-data:/home/nonroot/data \
ghcr.io/altcha-org/sentinel:2.0.0-beta.1 --config /etc/sentinel/sentinel.toml doctorOn embedded SQLite, stop the server first or use docker exec, so two containers don't open the same database files.
Run the doctor in the server
POST /v2/doctor/_run runs every check on the replica that answers. It needs a root user's session; API keys get 401 UNAUTHORIZED, and other users 403 ROOT_PRIVILEGES_REQUIRED. It works without a license, with an unlicensed database backend, and over the instance limit, so you can always find out what is wrong.
curl -X POST -H 'Authorization: Bearer <jwt>' \
https://sentinel.example.com/v2/doctor/_run{
"data": {
"replica": { "id": "rep_1k7pi08ke00i1uocj7h", "hostname": "sentinel-1", "version": "2.0.0-beta.1" },
"generatedAt": "2026-11-19T09:25:37.066553+00:00",
"cached": false,
"durationMs": 458,
"summary": { "ok": 26, "warn": 6, "fail": 1, "skipped": 7 },
"checks": [
{
"id": "license.status",
"category": "license",
"subject": null,
"status": "fail",
"message": "Licence status is none; challenges, verification and feature endpoints are refused.",
"hint": "Set [license] file or key in the configuration and restart.",
"details": { "licensee": null, "phase": "ready", "plan": null, "source": "baseline", "status": "none" },
"durationMs": 0
},
…
]
}
}A report is reused for 30 seconds ("cached": true), per replica. Checks run concurrently, and one that takes longer than 20 seconds fails with check timed out after 20s. A failing check doesn't fail the request: the response is still 200. The schema is in the API reference.
In a cluster, each request reaches one replica. Run it on each replica (for example through its own address) when only some of them misbehave.
The Doctor page
In the admin UI, root users open Doctor in the sidebar. The page runs the checks when it opens and shows only warnings and failures by default; switch Checks shown to All to see the rest. Run checks runs them again, and Copy report and Download report export the JSON. The app also runs the doctor when it loads and shows the number of failed checks next to Doctor in the sidebar.
Read the report
Each check has a status:
| Status | Meaning |
|---|---|
ok | Works as expected. |
warn | Works, but needs attention: a risky setting, a resource running low, a feature that is configured but has no effect. |
fail | Broken. Something Sentinel needs doesn't work. |
skipped | Doesn't apply to this configuration, for example storage.clickhouse without [clickhouse] url. |
The JSON report of the CLI has version, config (the file read, null with the defaults), generatedAt, durationMs, summary, and checks. The API's report has replica instead of version and config, plus cached. summary counts the checks per status. Each entry in checks has these fields:
| Field | Type | Description |
|---|---|---|
id | string | category.name, for example storage.db. Stable, so match on it in scripts. |
category | string | The part of id before the dot. |
subject | string | null | The item checked, for checks with one result per item: the origin for network.host, the config section for a failing config.valid. null otherwise. |
status | string | ok, warn, fail, or skipped. |
message | string | One English sentence. The wording can change between versions, so don't match on it. |
hint | string | null | A suggested fix; null when there's nothing to do. |
details | object | Check-specific data, such as latencies, counts, or the failing items; {} when there is none. |
durationMs | integer | How long the check took. |
To list only the problems from a JSON report:
altcha-sentinel --config sentinel.toml doctor --json \
| jq -r '.checks[] | select(.status == "warn" or .status == "fail") | "\(.status) \(.id) \(.message)"'Reports are safe to share: URLs appear only as their origin (scheme://host[:port]), and no keys, tokens, passwords, or secret values are read or shown. Usernames appear only in security.root_mfa, API key names and IDs only in security.api_key_ips, and IP addresses only in security.client_ip.
Checks
Thresholds are fixed, not configurable. "CLI" and "API" say where a check runs. In the CLI, storage.db and storage.kv have no outage history, storage.db is skipped while the embedded SQLite database doesn't exist yet, and alerts.repeating, security.root_mfa, and security.api_key_ips need a reachable database with a current schema.
| Check | Runs in | Warns or fails when |
|---|---|---|
storage.db | CLI, API | The database doesn't answer within 5 seconds (fail), answers in 500 ms or more, or had an outage in the last 24 hours (warn). |
storage.schema | CLI | The schema can't be read, or changes are pending with [database] auto_migrate = false (fail). See Schema migrations. |
storage.kv | CLI, API | The KV store doesn't answer (fail), answers in 100 ms or more, had an outage in the last 24 hours, or is an ExotDB KV server in memory mode (warn). |
storage.clickhouse | CLI, API | SELECT 1 fails (fail). |
storage.object | CLI, API | Listing the snapshot or EML prefix of [storage] fails (fail), or gets 403 while only EML attachments use it (warn). |
network.host | CLI, API | One result per outbound origin (remote files, threat sources, IP lookup APIs): DNS, TCP, or HTTP fails (fail). Any HTTP response counts as reachable. |
license.status | API | The functional API is refused: no license, expired, revoked, not yet valid (fail), or the license runtime is still starting (warn). |
license.expiry | API | The license ends within 3 days (fail) or 14 days (warn). |
license.server | CLI, API | The license server is unreachable (fail), or reachable while the online refresh keeps failing (warn, API only). Skipped with [license] file. |
license.instances | API | This replica is over the instances limit (fail), holds a burst slot, or every slot is in use (warn). |
license.quotas | API | The accounts, users, api_keys, or security_groups count has reached its limit (warn). |
license.environment | API | The license is for non-production use (warn). |
license.features | API | The config uses a feature the license lacks: ClickHouse, audit logs, OAuth, or LDAP (warn). |
data.remote_files | API | A remote file has never downloaded (fail), its last check failed, or its schedule is over an hour overdue (warn). |
data.threat_sources | API | A threat source failed to load or isn't loaded yet (warn). |
data.mmdb | CLI, API | A configured local .mmdb file is missing (fail) or older than 30 days (warn). |
data.similarity | API | The embedding model failed or isn't loaded yet (warn). |
data.api_key_cache | API | The API key cache is stale, or there are more live keys than max_keys (warn). |
alerts.delivery | API | The last webhook or email delivery failed (fail), or alerts are disabled or only logged (warn). |
alerts.repeating | CLI, API | A high system alert repeated in the last 24 hours (fail), or a lower-severity one did (warn). |
alerts.dropped | API | This replica dropped alerts because its queue was full (warn). |
cluster.peers_storage | API | Another replica can't reach the database or KV store (fail), or had an outage in the last 24 hours (warn). |
cluster.versions | API | Replicas run different versions (warn). |
cluster.hostnames | API | Two replicas share a hostname (fail). |
cluster.visibility | API | The license counts more instances than this replica sees, so replicas can't see each other (warn). |
system.clock | CLI, API | The clock is off by 60 seconds or more (fail) or 10 seconds or more (warn) from the probed servers and, in the API, the other replicas. |
system.data_dir | CLI, API | data_dir is missing or not writable, or has less than 512 MiB or 2 % free (fail); less than 2 GiB or 10 % free (warn). The CLI warns instead of failing when data_dir doesn't exist yet. |
system.memory_limit | CLI, API | No memory limit was found, so the memory alert never fires (warn). |
system.request_logs | API | This replica lost request logs since it started (warn). |
security.client_ip | CLI, API | client_ip_header is set without trusted_proxies, or (API) this request came through a proxy that isn't in trusted_proxies (warn). |
security.login_altcha | CLI, API | [auth] verify_altcha = false (warn). |
security.prometheus | CLI, API | The Prometheus listener is on a non-loopback address without bearer_token or allowed_ips (warn). |
security.admin_ui | CLI, API | The admin UI listener is on a non-loopback address without allowed_ips (warn). |
security.health | CLI, API | The main listener is on a non-loopback address and [server] health_allowed_ips is empty (warn). |
security.root_mfa | CLI, API | An active root user has neither TOTP nor a passkey (warn). |
security.api_key_ips | CLI, API | A restricted or full API key can be used from any IP, because no live rule gates on ip (warn). |
security.secrets_file | CLI, API | <data_dir>/secrets.json is readable or writable by group or others (warn). |
security.build | CLI, API | The binary isn't a production build (warn). |
config.file | CLI | The file can't be read, isn't valid TOML, has an unknown key, or a ${…} reference doesn't resolve (fail); no config file was found, so the defaults apply (warn). |
config.valid | CLI | A startup check rejects a value, with one result per error and the config section as subject (fail). |
config.ip_resolver | CLI, API | An ip-api provider has no key, or an ipstack provider has https = false (warn). |
config.passkeys | CLI, API | The auth base URL's host isn't a domain name, so passkeys are disabled (warn). |
config.audio | CLI, API | The code-challenge text-to-speech assets weren't found, so code challenges have no audio (warn). |
The doctor isn't a health check. For container and orchestrator probes, use GET / or altcha-sentinel healthcheck; see Monitoring & metrics.
Symptoms and fixes
What each error code means is in Error codes. This table maps what you see to the usual cause.
| Symptom | Cause | Fix |
|---|---|---|
403 LICENSE_NOT_ACTIVE on /v2/challenge, /v2/verify, /v2/classifier, and other functional routes | No active license; data.status says why (none, expired, refresh_overdue, …). | Set [license] key or file and restart. Check license.status and license.expiry; see Configure the license. |
403 BEARER_AUTH_PUBLIC_API_KEY | A restricted or full key was sent by ID (Bearer or apiKey). | Send it with Basic auth, -u 'key_…:sec_…'. |
403 PUBLIC_API_KEY_REQUIRED, NON_PUBLIC_API_KEY_REQUIRED, BASIC_AUTH_NON_PUBLIC_API_KEY, 401 API_KEY_REQUIRED | The credential doesn't fit the endpoint: a server key in the widget, a public key on a backend endpoint, or a session token on a detection endpoint. | Widget endpoints take a public key by ID; backend endpoints a restricted or full key with its secret; management endpoints a session. See the credential matrix. |
403 BACKEND_NOT_LICENSED on most /v2 routes | [database] or [redis] uses a backend the license doesn't include. Without a usable license, only embedded SQLite is allowed. | Get a license with that backend, or switch to a licensed one. Sign-in, /v2/license*, and /v2/doctor* keep working. |
403 LICENSE_LIMIT_REACHED when creating an account, user, API key, or security group | The license's limit for that kind is reached. license.quotas shows which one. | Delete unused records (deleted ones don't count) or get a higher limit. |
503 LICENSE_INSTANCE_LIMIT, or a replica exits at startup | More replicas than the license's instances, or replicas sharing a HOSTNAME. | Give each replica a unique, stable HOSTNAME; stop surplus replicas. Check license.instances and cluster.hostnames; see Clustering & HA. |
| CORS error in the browser console | The page calls an endpoint that isn't a widget route (detection and management APIs send no CORS headers), the proxy adds or strips CORS headers, or the real response is an error from a proxy. | Call detection APIs from your server, not the browser. Let Sentinel set CORS headers; see CORS. Check the response status in the network tab: a 421 or a proxy error page looks like a CORS failure. |
421 Misdirected Request with the body misdirected request | The request's host isn't in the listener's allowed_hosts. Behind a proxy that rewrites Host, Sentinel sees the internal host. | Add the public host to allowed_hosts, or set forwarded_host_header and trusted_proxies; see Host and allowed hosts. |
| Every request has the proxy's IP: rate limits hit all users at once, IP rules and request logs show one address | The proxy isn't in trusted_proxies, so X-Forwarded-For is ignored. security.client_ip warns when the doctor's own request shows this. | Add the proxy's addresses to trusted_proxies on that listener, or set client_ip_header for a CDN; see Client IP. |
403 IP_NOT_ALLOWED on GET /.health | The prober isn't in [server] health_allowed_ips (private networks by default), or it sends forwarding headers through a proxy that isn't in trusted_proxies. | Add public probers to health_allowed_ips, or list the proxy in trusted_proxies; see Health endpoint. |
| Admin UI unreachable | [server.admin_ui] is off by default, and when enabled it listens on 127.0.0.1, which isn't reachable through a container's published port. | Set enabled = true and host = "0.0.0.0", restrict it with allowed_ips, and publish its port. Serve it over HTTPS: the session cookie is Secure, so sign-in works over plain HTTP only on localhost. See Admin UI. |
Startup stops with unknown field `…`, expected one of … | A misspelled key or a key in the wrong section; Sentinel rejects unknown keys. | Fix the key the message names; the next line names its section, for example in `server`. See Unknown keys and invalid values stop startup. |
Startup stops with invalid variable name or reading secret file '…': No such file or directory | A ${…} reference with an invalid name (such as ${JWT-SECRET}), or a ${file:…} path that isn't mounted. The error names the config key. | Fix the name or mount the file; see Environment variables and secret files. |
Startup stops with invalid type: string "3000", expected u16 | A number or boolean written as a ${…} string; only strings are expanded. | Write numbers and booleans literally. |
| Lists or databases stay empty or stale | A remote file can't be downloaded: no outbound access, a proxy not set, DNS, or TLS. | Check data.remote_files for the file and network.host for its origin and failed stage. Set HTTPS_PROXY if outbound traffic needs a proxy; see Remote files and Air-gapped deployments. |
OAuth sign-in returns to the login page with OAUTH_FAILED, or the provider reports a redirect URI mismatch | The redirect URI registered at the provider differs from {base_url}/v2/auth/oauth/{provider}/callback, where base_url is [auth] base_url, else an enabled [server.admin_ui] base_url, else [server] base_url. | Register the exact URI, including scheme, host, and any path prefix. See Admin sign-in & SSO and OAuth redirect errors. |
Startup stops with timed out after 30000ms waiting for migration lock '__exotdb_migrate__' | Another replica held the migration lock for over 30 seconds. | Restart the replica. See Schema migrations. |
| Widget shows an error or never verifies | Usually one of the codes above on /v2/challenge: LICENSE_NOT_ACTIVE, a non-public key, ACCESS_DENIED from a security-group rule (for example an origin rule), or RATE_LIMITED. A clock off by seconds also breaks verification, because challenges have no clock leeway. | Open the network tab and read code in the failing response. Check system.clock. Test the key with curl as in Widget integration. |
500 INTERNAL_ERROR | A server fault, such as the database or log store being unavailable. Details are only in the server log. | Check the log and run the doctor; storage.db and storage.kv usually show the cause. |
Raise the log level
Sentinel logs to stdout. [log] level takes a filter directive: a default level, then target=level pairs for single modules. To log every 4xx and 5xx response with its error code, without raising the rest:
[log]
level = "info,sentinel::http=debug" # "info,sentinel::http=trace" logs every responseRestart, and each failed request logs a line like this:
2026-11-19T09:25:58.494974Z DEBUG http.server{… http.route="/v2/challenge" … http.response.status_code=403}: sentinel::http: request failed status=403 code="LICENSE_NOT_ACTIVE" latency_ms=0.204458Set [log] format = "json" for log collectors. Lower the level again afterwards; debug and trace are verbose. See [log].
Ask for support
Collect these before you open a support request:
The doctor report. The JSON from Doctor → Download report, or from
POST /v2/doctor/_run, on the affected replica. If the server doesn't start, the output ofdoctor --jsonrun with the server's config and mounts.The version and setup. The image tag, the database and KV backend, the number of replicas, and what sits in front of Sentinel (proxy, CDN, load balancer).
The failing request and response. The method, path, and status, and the response body with its
code. Remove API key secrets and session tokens.The server log around the failure, with
[log] level = "info,sentinel::http=debug"if the problem is with requests.The configuration, with every secret removed. Secrets referenced as
${…}are already safe to share; literal values are not.