Monitoring & metrics
Sentinel exposes a liveness endpoint for load balancers and orchestrators, a storage health endpoint for uptime monitors, Prometheus metrics, OpenTelemetry traces, and structured logs. This page shows what each one reports and how to wire it into your monitoring.
Every key mentioned here is listed with its default in the Configuration reference. Changes to these keys take effect after a restart.
What to use for what
| Endpoint or output | Where | Use it for | Access |
|---|---|---|---|
GET / | Main and public listeners | Liveness and readiness probes, healthcheck | Anyone |
GET /.health | Main listener only | Uptime monitoring and alerting on storage | Client IP in [server] health_allowed_ips |
/metrics | [prometheus] listener, off by default | Prometheus scraping, dashboards, alert rules | bearer_token, allowed_ips |
| OTLP traces | Outgoing, to your collector | Request latency breakdowns across Sentinel, its database, and outgoing calls | Collector credentials in [telemetry] headers |
| Logs | stdout | Errors, startup state, request failures | Your log pipeline |
None of these needs a license, and none of them is rate limited or written to request logs.
Liveness and readiness: GET /
GET / answers 200 with {"ok":true} as long as the process serves HTTP. It doesn't touch the database or Redis, so a storage outage doesn't take replicas out of service. It answers on any host, so probes that address the pod or container IP work even when allowed_hosts is set.
curl -i http://localhost:3000/HTTP/1.1 200 OK
content-type: application/json
cache-control: no-store
content-length: 11
{"ok":true}Use it for both liveness and readiness. A Kubernetes container spec, for example:
livenessProbe:
httpGet:
path: /
port: 3000
periodSeconds: 10
failureThreshold: 3
readinessProbe:
httpGet:
path: /
port: 3000
periodSeconds: 10With [server] tls set, add scheme: HTTPS to both probes. Behind a load balancer, point its health check at GET / on the listener it forwards to.
The healthcheck command
The container image has no curl or wget, so it ships a built-in check: altcha-sentinel healthcheck sends GET / to [server] port on this host and exits with 0 when the server answers {"ok": true}, or 1 otherwise. With [server] tls set, it uses HTTPS and skips certificate checks, because the certificate names the public host rather than the probed address.
The image's HEALTHCHECK runs it every 10 seconds against ./sentinel.toml. If you start Sentinel with --config, pass the same path to the check, so it probes the port the server listens on:
docker exec sentinel /usr/local/bin/altcha-sentinel --config /etc/sentinel/sentinel.toml healthcheck
echo $?0On failure, the reason goes to stderr. Without --config and without ./sentinel.toml, it probes the default port 3000 and says so:
http://127.0.0.1:3000/: unexpected answer …
no --config and no ./sentinel.toml: probed the default port 3000; pass the server's --configSee healthcheck for the exit codes, and Install with Docker for the --health-cmd override.
Storage health: GET /.health
GET /.health on the main listener reports whether this replica reaches its storage: kv (Redis or the built-in KV store), db (the database), and clickhouse when ClickHouse is configured. It answers from background probes that run every 10 seconds, so polling it never adds load to your database. A check whose last probe is older than 30 seconds fails.
curl -i http://10.0.0.20:3000/.healthHTTP/1.1 200 OK
content-type: application/json
cache-control: no-store
content-length: 65
{"checks":{"db":{"ok":true},"kv":{"ok":true}},"status":"healthy"}When any check fails, the status is 503 and "status" is "unhealthy". The response has no error messages or latencies; root users find those under Cluster in the admin UI, or in storage of GET /v2/cluster/replicas.
Who may call it
By default only loopback and private addresses (RFC 1918, 100.64.0.0/10, link-local, and their IPv6 equivalents) may call /.health. Everyone else gets 403 IP_NOT_ALLOWED:
curl -i https://sentinel.example.com/.healthHTTP/1.1 403 Forbidden
content-type: application/json
cache-control: no-store
content-length: 75
{"code":"IP_NOT_ALLOWED","error":"Client IP not allowed.","statusCode":403}Probes from Docker, Kubernetes, and AWS load balancers connect from private addresses and need no setting. Public probers, such as Google Cloud load balancers, Azure, and external uptime monitors, must be added. Setting the key replaces the default list, so keep the private ranges you still need:
[server]
health_allowed_ips = ["10.0.0.0/8", "35.191.0.0/16", "130.211.0.0/22"] # private network + Google Cloud health checkersA request that carries X-Forwarded-For or X-Real-IP from a peer outside trusted_proxies is also refused, so list your proxy in [server] trusted_proxies if it forwards /.health. The full access rules are in Health endpoint.
Prometheus metrics
The [prometheus] listener serves metrics in the Prometheus text format on its own port, so scrapes aren't counted in the request metrics, logged, or rate limited. Each replica serves its own metrics; scrape every replica.
Enable the listener
The listener binds to 127.0.0.1:3001 by default, which a scraper in another container or host can't reach. When you bind it beyond loopback, protect it with a bearer token, an IP allowlist, or both:
[prometheus]
enabled = true
host = "0.0.0.0"
port = 3001
bearer_token = "${file:/run/secrets/prometheus_token}"
allowed_ips = ["10.0.0.0/8"] # the TCP peer; X-Forwarded-For is not read
allowed_hosts = ["10.0.0.20"] # optional; scrapers usually address the IP
# tls = { cert = "/etc/sentinel/tls/fullchain.pem", key = "/etc/sentinel/tls/privkey.pem" }The listener answers in plain text, not the API's JSON envelope:
| Request | Response |
|---|---|
TCP peer outside allowed_ips | 403 with forbidden |
Missing or wrong Authorization: Bearer <token> | 401 with WWW-Authenticate: Bearer and unauthorized |
Host outside allowed_hosts | 421 with misdirected request |
curl -i http://10.0.0.20:3001/metricsHTTP/1.1 401 Unauthorized
content-type: text/plain; charset=utf-8
www-authenticate: Bearer
cache-control: no-store
content-length: 13
unauthorizedWith the token:
curl -H "Authorization: Bearer $PROMETHEUS_TOKEN" http://10.0.0.20:3001/metrics# HELP altcha_sentinel_build_info Version and identity of this replica; always 1.
# TYPE altcha_sentinel_build_info gauge
altcha_sentinel_build_info{version="2.0.0-beta.1",replica_id="rep_…",hostname="sentinel-0"} 1
# HELP altcha_sentinel_http_requests_total HTTP requests served by the API listener.
# TYPE altcha_sentinel_http_requests_total counter
altcha_sentinel_http_requests_total 3
…
# HELP altcha_sentinel_storage_up 1 when the last periodic probe of the service succeeded, else 0.
# TYPE altcha_sentinel_storage_up gauge
altcha_sentinel_storage_up{service="kv"} 1
altcha_sentinel_storage_up{service="db"} 1
…The startup log confirms the listener: prometheus metrics listening addr=0.0.0.0:3001 path=/metrics tls=false. See [prometheus] for path and the other keys.
Scrape configuration
Keep the token in a file that Prometheus reads, and list every replica:
scrape_configs:
- job_name: altcha-sentinel
scrape_interval: 30s
authorization:
type: Bearer
credentials_file: /etc/prometheus/secrets/sentinel_token
static_configs:
- targets: ["10.0.0.20:3001", "10.0.0.21:3001"]With tls set on the listener, add scheme: https and a tls_config that trusts your certificate. On Kubernetes, use your usual pod discovery (kubernetes_sd_configs or a PodMonitor) against the metrics port instead of static targets.
Metric families
Every name starts with altcha_sentinel_. Counters are totals since the replica started and reset on restart, so query them with rate() or increase(). A family whose source doesn't exist on this replica is left out rather than exported as zeros.
Requests and outcomes
| Metric | Type | Labels | Meaning |
|---|---|---|---|
altcha_sentinel_http_requests_total | counter | — | Requests served by the API: every request on the main and public listeners (including GET /, /.health, and 429s) and /v2 on the admin UI listener. Scrapes and admin app files aren't counted. |
altcha_sentinel_http_request_errors_total | counter | class (4xx, 5xx) | Responses with a 4xx or 5xx status. |
altcha_sentinel_http_request_duration_seconds | histogram | le | Latency to the response head. Buckets from 0.01 to 60 seconds. |
altcha_sentinel_http_requests_in_flight | gauge | — | Requests being served. |
altcha_sentinel_requests_blocked_total | counter | reason | Requests refused, by rule_deny, rate_limit_global, rate_limit_rule, rate_limit_audio, header_entropy, his, or ip_blocked. |
altcha_sentinel_challenges_issued_total | counter | algorithm, protection_level | Challenges created, e.g. algorithm="PBKDF2/SHA-256", protection_level="frictionless" or "code_challenge". Present after the first challenge. |
altcha_sentinel_verifications_total | counter | result, reason | result="verified", or result="failed" with the failure code as reason (e.g. CHALLENGE_EXPIRED, HIS_BLOCKED). |
altcha_sentinel_classifications_total | counter | endpoint, verdict | Classifier verdicts (GOOD, NEUTRAL, BAD) by endpoint: classifier, eml, or verify. Present after the first classification. |
Process
| Metric | Type | Labels | Meaning |
|---|---|---|---|
altcha_sentinel_build_info | gauge | version, replica_id, hostname | Always 1; identifies the replica. |
altcha_sentinel_process_cpu_percent | gauge | — | CPU used since the previous scrape, in percent of one core (200 = two cores busy). Absent on the first scrape. |
altcha_sentinel_process_resident_memory_bytes | gauge | — | Resident set size. |
altcha_sentinel_process_working_set_bytes | gauge | — | Cgroup working set: charged memory minus reclaimable page cache. Linux containers only. |
altcha_sentinel_process_memory_limit_bytes | gauge | — | The memory ceiling: the container's limit, or the host's RAM without one. |
altcha_sentinel_process_memory_limit_info | gauge | source (cgroup_v2, cgroup_v1, host) | Always 1; where the ceiling came from. |
altcha_sentinel_scrape_duration_seconds | gauge | — | Time spent rendering this scrape. |
process_cpu_percent covers the interval since the last scrape by anyone, so give each replica a single scraper, or read CPU from your container platform instead.
Storage and cluster
| Metric | Type | Labels | Meaning |
|---|---|---|---|
altcha_sentinel_storage_up | gauge | service (kv, db, clickhouse) | 1 when the last background probe succeeded, else 0. The same probes as /.health. |
altcha_sentinel_storage_probe_latency_seconds | gauge | service | Latency of the last successful probe. |
altcha_sentinel_replicas | gauge | — | Live replicas this one sees, itself included. |
altcha_sentinel_cluster_pubsub | gauge | — | 1 when replicas talk over Redis pub/sub, 0 with the built-in KV store. |
altcha_sentinel_kv_commands_total | counter | — | Commands run by the built-in KV store. Built-in KV store only. |
altcha_sentinel_kv_connections | gauge | — | Open connections to the built-in KV store. Built-in KV store only. |
altcha_sentinel_api_key_cache_entries | gauge | — | API keys in the cached snapshot. Absent when [api_key_cache] is off. |
altcha_sentinel_api_key_cache_complete | gauge | — | 1 when every live key fits [api_key_cache] max_keys; keys beyond it are read from the database. |
altcha_sentinel_api_key_cache_age_seconds | gauge | — | Age of the cached snapshot; grows while reloads fail. |
altcha_sentinel_api_key_cache_reload_failures_total | counter | — | Failed cache reloads. |
Remote files, threat sources, and similarity
| Metric | Type | Labels | Meaning |
|---|---|---|---|
altcha_sentinel_remote_file_updates_total | counter | file | Times the file's content changed: downloads, adoptions, and updates from other replicas. |
altcha_sentinel_remote_file_size_bytes | gauge | file | Size of the kept file. |
altcha_sentinel_remote_file_last_download_timestamp_seconds | gauge | file | Unix time of the last completed download. |
altcha_sentinel_remote_file_consecutive_failures | gauge | file | Consecutive failed checks and downloads. |
altcha_sentinel_remote_file_updating | gauge | file | 1 while a check or download runs. |
altcha_sentinel_threat_source_entries | gauge | source | Entries in the source's loaded index. |
altcha_sentinel_threat_source_last_load_timestamp_seconds | gauge | source | Unix time of the last successful load. |
altcha_sentinel_threat_source_error | gauge | source | 1 when the last registration or load failed. |
altcha_sentinel_similarity_info | gauge | provider, model_id | Always 1; the embedding provider and model. |
altcha_sentinel_similarity_loaded | gauge | — | 1 when an embedding model is ready. |
altcha_sentinel_similarity_samples | gauge | — | Training samples loaded. |
altcha_sentinel_similarity_embedded | gauge | — | Training samples with an embedding. |
file is the remote file's name, such as phishing or user-agents; threat lists appear as threat-<source id>. source is the threat source's ID (ths_…), not its name. See Remote files & data sources and Threat intelligence.
Request logs
| Metric | Type | Labels | Meaning |
|---|---|---|---|
altcha_sentinel_request_logs_written_total | counter | — | Request log records written. |
altcha_sentinel_request_logs_dropped_total | counter | reason (buffer_full, write_failed) | Records dropped because the queue was full or their batch failed to write. |
altcha_sentinel_request_logs_queue_depth | gauge | — | Messages waiting for the writer. |
The altcha_sentinel_license_* families are described in License alerts and metrics.
Example alert rules
groups:
- name: altcha-sentinel
rules:
- alert: SentinelStorageDown
expr: altcha_sentinel_storage_up == 0
for: 1m
- alert: SentinelHigh5xxRate
expr: rate(altcha_sentinel_http_request_errors_total{class="5xx"}[5m]) > 1
for: 5m
- alert: SentinelSlowRequests
expr: histogram_quantile(0.95, rate(altcha_sentinel_http_request_duration_seconds_bucket[5m])) > 0.5
for: 10m
- alert: SentinelRequestLogsDropped
expr: increase(altcha_sentinel_request_logs_dropped_total[15m]) > 0
- alert: SentinelRemoteFileFailing
expr: altcha_sentinel_remote_file_consecutive_failures >= 3Sentinel can also send many of these conditions itself, by webhook, email, or web push; see Alerts.
Tracing with OpenTelemetry
[telemetry] exports traces over OTLP/HTTP (protobuf) to an OpenTelemetry collector, or to any backend that accepts OTLP, such as Jaeger, Grafana Tempo, or Honeycomb.
[telemetry]
enabled = true
endpoint = "http://otel-collector:4318" # /v1/traces is appended
headers = { authorization = "Bearer ${OTEL_TOKEN}" }
sample_ratio = 0.1 # keep 10% of new traces (no incoming traceparent)Sentinel then logs telemetry: exporting traces over OTLP/HTTP with the endpoint at startup, and records:
- An
http.serverspan per API request, named after the method and route (GET /v2/challenge), withhttp.route,url.path, andhttp.response.status_code. A request with a W3Ctraceparentheader continues that trace. - Database, Redis, and rate-limit spans inside it. Redis and rate-limit spans are
debuglevel, so add them tofilterto export them:filter = "info,exotdb_redis=debug,exotdb_ratelimit=debug". - An
http.clientspan per outgoing call: webhooks, OAuth, remote files, IP lookups, AI and embedding APIs, and others. Sentinel passestraceparenton only to a remote ExotDB server, ClickHouse, and alert webhooks.
sample_ratio is parent-based: a request whose incoming traceparent is sampled is always exported. filter selects exported spans independently of [log] level. Without endpoint, Sentinel reads OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, then OTEL_EXPORTER_OTLP_ENDPOINT, then uses http://localhost:4318. Traces go through the outbound proxy when one is set; see Outbound proxy & air-gapped deployments. Metrics stay on Prometheus and logs on stdout. See [telemetry] for every key.
Logs
Sentinel writes logs to stdout. [log] sets what is logged and in which format:
[log]
level = "info"
format = "json"Log level
level is a filter directive: a default level, optionally followed by target=level pairs separated by commas. Levels are error, warn, info, debug, and trace.
level | What you get |
|---|---|
"info" | Startup, configuration, background jobs, warnings, and errors (default) |
"warn" | Only warnings and errors |
"info,sentinel::http=debug" | Also one line per 4xx or 5xx response, with its status, error code, and latency |
"info,sentinel::http=trace" | One line per response |
"info,tiberius=info" | Also the SQL Server driver's routine messages, which are otherwise limited to warn |
Responses aren't logged at info. To see why requests fail, raise sentinel::http to debug temporarily, or use request logs for API keys with logging on.
Log format
format = "text" (the default) writes one human-readable line per event:
2026-11-19T09:27:31.504337Z INFO altcha_sentinel: listening addr=0.0.0.0:3000 tls=false
2026-11-19T09:27:34.864593Z DEBUG http.server{otel.name=GET /v2/challenge … http.response.status_code=403}: sentinel::http: request failed status=403 code="LICENSE_NOT_ACTIVE" latency_ms=0.250959format = "json" writes one JSON object per line, for log collectors such as Loki, Elasticsearch, or Cloud Logging. Each object has timestamp, level, and target, the event's fields (including message) at the top level, and the enclosing span and spans when the event happens inside a request:
{"timestamp":"2026-11-19T09:27:08.780167Z","level":"INFO","message":"listening","addr":"0.0.0.0:3000","tls":false,"target":"altcha_sentinel"}
{"timestamp":"2026-11-19T09:27:11.629750Z","level":"DEBUG","message":"request failed","status":403,"code":"LICENSE_NOT_ACTIVE","latency_ms":0.126375,"target":"sentinel::http","span":{"http.request.method":"GET","http.response.status_code":403,"http.route":"/v2/challenge","otel.kind":"server","otel.name":"GET /v2/challenge","url.path":"/v2/challenge","name":"http.server"},"spans":[…]}With [privacy] anonymize_ip = true, public client IPs are anonymized in log lines too; see Privacy & data handling. See [log] and, for reading the common startup errors, Doctor & troubleshooting.
In the admin UI
The admin UI shows the same data without a monitoring stack:
- Dashboard: requests, verification results, errors, and visitor signals of API keys with request logging on. See Request logs & analytics.
- Cluster (root only): every replica's version, requests, CPU, memory, and storage status, refreshed every 10 seconds, with charts of the last hour. See Clustering & high availability.
- Doctor: live diagnostics of storage, connectivity, license, and configuration. See Doctor & troubleshooting.