Skip to main content
ALTCHA®

Monitoring & metrics

Sentinel exposes a liveness endpoint for load balancers and orchestrators, a storage health endpoint for uptime monitors, Prometheus metrics, OpenTelemetry traces, and structured logs. This page shows what each one reports and how to wire it into your monitoring.

Every key mentioned here is listed with its default in the Configuration reference. Changes to these keys take effect after a restart.

What to use for what

Endpoint or outputWhereUse it forAccess
GET /Main and public listenersLiveness and readiness probes, healthcheckAnyone
GET /.healthMain listener onlyUptime monitoring and alerting on storageClient IP in [server] health_allowed_ips
/metrics[prometheus] listener, off by defaultPrometheus scraping, dashboards, alert rulesbearer_token, allowed_ips
OTLP tracesOutgoing, to your collectorRequest latency breakdowns across Sentinel, its database, and outgoing callsCollector credentials in [telemetry] headers
LogsstdoutErrors, startup state, request failuresYour log pipeline

None of these needs a license, and none of them is rate limited or written to request logs.

Liveness and readiness: GET /

GET / answers 200 with {"ok":true} as long as the process serves HTTP. It doesn't touch the database or Redis, so a storage outage doesn't take replicas out of service. It answers on any host, so probes that address the pod or container IP work even when allowed_hosts is set.

curl -i http://localhost:3000/
HTTP/1.1 200 OK
content-type: application/json
cache-control: no-store
content-length: 11

{"ok":true}

Use it for both liveness and readiness. A Kubernetes container spec, for example:

livenessProbe:
  httpGet:
    path: /
    port: 3000
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet:
    path: /
    port: 3000
  periodSeconds: 10

With [server] tls set, add scheme: HTTPS to both probes. Behind a load balancer, point its health check at GET / on the listener it forwards to.

The healthcheck command

The container image has no curl or wget, so it ships a built-in check: altcha-sentinel healthcheck sends GET / to [server] port on this host and exits with 0 when the server answers {"ok": true}, or 1 otherwise. With [server] tls set, it uses HTTPS and skips certificate checks, because the certificate names the public host rather than the probed address.

The image's HEALTHCHECK runs it every 10 seconds against ./sentinel.toml. If you start Sentinel with --config, pass the same path to the check, so it probes the port the server listens on:

docker exec sentinel /usr/local/bin/altcha-sentinel --config /etc/sentinel/sentinel.toml healthcheck
echo $?
0

On failure, the reason goes to stderr. Without --config and without ./sentinel.toml, it probes the default port 3000 and says so:

http://127.0.0.1:3000/: unexpected answer …
no --config and no ./sentinel.toml: probed the default port 3000; pass the server's --config

See healthcheck for the exit codes, and Install with Docker for the --health-cmd override.

Storage health: GET /.health

GET /.health on the main listener reports whether this replica reaches its storage: kv (Redis or the built-in KV store), db (the database), and clickhouse when ClickHouse is configured. It answers from background probes that run every 10 seconds, so polling it never adds load to your database. A check whose last probe is older than 30 seconds fails.

curl -i http://10.0.0.20:3000/.health
HTTP/1.1 200 OK
content-type: application/json
cache-control: no-store
content-length: 65

{"checks":{"db":{"ok":true},"kv":{"ok":true}},"status":"healthy"}

When any check fails, the status is 503 and "status" is "unhealthy". The response has no error messages or latencies; root users find those under Cluster in the admin UI, or in storage of GET /v2/cluster/replicas.

Who may call it

By default only loopback and private addresses (RFC 1918, 100.64.0.0/10, link-local, and their IPv6 equivalents) may call /.health. Everyone else gets 403 IP_NOT_ALLOWED:

curl -i https://sentinel.example.com/.health
HTTP/1.1 403 Forbidden
content-type: application/json
cache-control: no-store
content-length: 75

{"code":"IP_NOT_ALLOWED","error":"Client IP not allowed.","statusCode":403}

Probes from Docker, Kubernetes, and AWS load balancers connect from private addresses and need no setting. Public probers, such as Google Cloud load balancers, Azure, and external uptime monitors, must be added. Setting the key replaces the default list, so keep the private ranges you still need:

[server]
health_allowed_ips = ["10.0.0.0/8", "35.191.0.0/16", "130.211.0.0/22"]   # private network + Google Cloud health checkers

A request that carries X-Forwarded-For or X-Real-IP from a peer outside trusted_proxies is also refused, so list your proxy in [server] trusted_proxies if it forwards /.health. The full access rules are in Health endpoint.

Prometheus metrics

The [prometheus] listener serves metrics in the Prometheus text format on its own port, so scrapes aren't counted in the request metrics, logged, or rate limited. Each replica serves its own metrics; scrape every replica.

Enable the listener

The listener binds to 127.0.0.1:3001 by default, which a scraper in another container or host can't reach. When you bind it beyond loopback, protect it with a bearer token, an IP allowlist, or both:

[prometheus]
enabled      = true
host         = "0.0.0.0"
port         = 3001
bearer_token = "${file:/run/secrets/prometheus_token}"
allowed_ips  = ["10.0.0.0/8"]            # the TCP peer; X-Forwarded-For is not read
allowed_hosts = ["10.0.0.20"]            # optional; scrapers usually address the IP
# tls = { cert = "/etc/sentinel/tls/fullchain.pem", key = "/etc/sentinel/tls/privkey.pem" }

The listener answers in plain text, not the API's JSON envelope:

RequestResponse
TCP peer outside allowed_ips403 with forbidden
Missing or wrong Authorization: Bearer <token>401 with WWW-Authenticate: Bearer and unauthorized
Host outside allowed_hosts421 with misdirected request
curl -i http://10.0.0.20:3001/metrics
HTTP/1.1 401 Unauthorized
content-type: text/plain; charset=utf-8
www-authenticate: Bearer
cache-control: no-store
content-length: 13

unauthorized

With the token:

curl -H "Authorization: Bearer $PROMETHEUS_TOKEN" http://10.0.0.20:3001/metrics
# HELP altcha_sentinel_build_info Version and identity of this replica; always 1.
# TYPE altcha_sentinel_build_info gauge
altcha_sentinel_build_info{version="2.0.0-beta.1",replica_id="rep_…",hostname="sentinel-0"} 1
# HELP altcha_sentinel_http_requests_total HTTP requests served by the API listener.
# TYPE altcha_sentinel_http_requests_total counter
altcha_sentinel_http_requests_total 3
…
# HELP altcha_sentinel_storage_up 1 when the last periodic probe of the service succeeded, else 0.
# TYPE altcha_sentinel_storage_up gauge
altcha_sentinel_storage_up{service="kv"} 1
altcha_sentinel_storage_up{service="db"} 1
…

The startup log confirms the listener: prometheus metrics listening addr=0.0.0.0:3001 path=/metrics tls=false. See [prometheus] for path and the other keys.

Scrape configuration

Keep the token in a file that Prometheus reads, and list every replica:

scrape_configs:
  - job_name: altcha-sentinel
    scrape_interval: 30s
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/secrets/sentinel_token
    static_configs:
      - targets: ["10.0.0.20:3001", "10.0.0.21:3001"]

With tls set on the listener, add scheme: https and a tls_config that trusts your certificate. On Kubernetes, use your usual pod discovery (kubernetes_sd_configs or a PodMonitor) against the metrics port instead of static targets.

Metric families

Every name starts with altcha_sentinel_. Counters are totals since the replica started and reset on restart, so query them with rate() or increase(). A family whose source doesn't exist on this replica is left out rather than exported as zeros.

Requests and outcomes

MetricTypeLabelsMeaning
altcha_sentinel_http_requests_totalcounter—Requests served by the API: every request on the main and public listeners (including GET /, /.health, and 429s) and /v2 on the admin UI listener. Scrapes and admin app files aren't counted.
altcha_sentinel_http_request_errors_totalcounterclass (4xx, 5xx)Responses with a 4xx or 5xx status.
altcha_sentinel_http_request_duration_secondshistogramleLatency to the response head. Buckets from 0.01 to 60 seconds.
altcha_sentinel_http_requests_in_flightgauge—Requests being served.
altcha_sentinel_requests_blocked_totalcounterreasonRequests refused, by rule_deny, rate_limit_global, rate_limit_rule, rate_limit_audio, header_entropy, his, or ip_blocked.
altcha_sentinel_challenges_issued_totalcounteralgorithm, protection_levelChallenges created, e.g. algorithm="PBKDF2/SHA-256", protection_level="frictionless" or "code_challenge". Present after the first challenge.
altcha_sentinel_verifications_totalcounterresult, reasonresult="verified", or result="failed" with the failure code as reason (e.g. CHALLENGE_EXPIRED, HIS_BLOCKED).
altcha_sentinel_classifications_totalcounterendpoint, verdictClassifier verdicts (GOOD, NEUTRAL, BAD) by endpoint: classifier, eml, or verify. Present after the first classification.

Process

MetricTypeLabelsMeaning
altcha_sentinel_build_infogaugeversion, replica_id, hostnameAlways 1; identifies the replica.
altcha_sentinel_process_cpu_percentgauge—CPU used since the previous scrape, in percent of one core (200 = two cores busy). Absent on the first scrape.
altcha_sentinel_process_resident_memory_bytesgauge—Resident set size.
altcha_sentinel_process_working_set_bytesgauge—Cgroup working set: charged memory minus reclaimable page cache. Linux containers only.
altcha_sentinel_process_memory_limit_bytesgauge—The memory ceiling: the container's limit, or the host's RAM without one.
altcha_sentinel_process_memory_limit_infogaugesource (cgroup_v2, cgroup_v1, host)Always 1; where the ceiling came from.
altcha_sentinel_scrape_duration_secondsgauge—Time spent rendering this scrape.

process_cpu_percent covers the interval since the last scrape by anyone, so give each replica a single scraper, or read CPU from your container platform instead.

Storage and cluster

MetricTypeLabelsMeaning
altcha_sentinel_storage_upgaugeservice (kv, db, clickhouse)1 when the last background probe succeeded, else 0. The same probes as /.health.
altcha_sentinel_storage_probe_latency_secondsgaugeserviceLatency of the last successful probe.
altcha_sentinel_replicasgauge—Live replicas this one sees, itself included.
altcha_sentinel_cluster_pubsubgauge—1 when replicas talk over Redis pub/sub, 0 with the built-in KV store.
altcha_sentinel_kv_commands_totalcounter—Commands run by the built-in KV store. Built-in KV store only.
altcha_sentinel_kv_connectionsgauge—Open connections to the built-in KV store. Built-in KV store only.
altcha_sentinel_api_key_cache_entriesgauge—API keys in the cached snapshot. Absent when [api_key_cache] is off.
altcha_sentinel_api_key_cache_completegauge—1 when every live key fits [api_key_cache] max_keys; keys beyond it are read from the database.
altcha_sentinel_api_key_cache_age_secondsgauge—Age of the cached snapshot; grows while reloads fail.
altcha_sentinel_api_key_cache_reload_failures_totalcounter—Failed cache reloads.

Remote files, threat sources, and similarity

MetricTypeLabelsMeaning
altcha_sentinel_remote_file_updates_totalcounterfileTimes the file's content changed: downloads, adoptions, and updates from other replicas.
altcha_sentinel_remote_file_size_bytesgaugefileSize of the kept file.
altcha_sentinel_remote_file_last_download_timestamp_secondsgaugefileUnix time of the last completed download.
altcha_sentinel_remote_file_consecutive_failuresgaugefileConsecutive failed checks and downloads.
altcha_sentinel_remote_file_updatinggaugefile1 while a check or download runs.
altcha_sentinel_threat_source_entriesgaugesourceEntries in the source's loaded index.
altcha_sentinel_threat_source_last_load_timestamp_secondsgaugesourceUnix time of the last successful load.
altcha_sentinel_threat_source_errorgaugesource1 when the last registration or load failed.
altcha_sentinel_similarity_infogaugeprovider, model_idAlways 1; the embedding provider and model.
altcha_sentinel_similarity_loadedgauge—1 when an embedding model is ready.
altcha_sentinel_similarity_samplesgauge—Training samples loaded.
altcha_sentinel_similarity_embeddedgauge—Training samples with an embedding.

file is the remote file's name, such as phishing or user-agents; threat lists appear as threat-<source id>. source is the threat source's ID (ths_…), not its name. See Remote files & data sources and Threat intelligence.

Request logs

MetricTypeLabelsMeaning
altcha_sentinel_request_logs_written_totalcounter—Request log records written.
altcha_sentinel_request_logs_dropped_totalcounterreason (buffer_full, write_failed)Records dropped because the queue was full or their batch failed to write.
altcha_sentinel_request_logs_queue_depthgauge—Messages waiting for the writer.

The altcha_sentinel_license_* families are described in License alerts and metrics.

Example alert rules

groups:
  - name: altcha-sentinel
    rules:
      - alert: SentinelStorageDown
        expr: altcha_sentinel_storage_up == 0
        for: 1m
      - alert: SentinelHigh5xxRate
        expr: rate(altcha_sentinel_http_request_errors_total{class="5xx"}[5m]) > 1
        for: 5m
      - alert: SentinelSlowRequests
        expr: histogram_quantile(0.95, rate(altcha_sentinel_http_request_duration_seconds_bucket[5m])) > 0.5
        for: 10m
      - alert: SentinelRequestLogsDropped
        expr: increase(altcha_sentinel_request_logs_dropped_total[15m]) > 0
      - alert: SentinelRemoteFileFailing
        expr: altcha_sentinel_remote_file_consecutive_failures >= 3

Sentinel can also send many of these conditions itself, by webhook, email, or web push; see Alerts.

Tracing with OpenTelemetry

[telemetry] exports traces over OTLP/HTTP (protobuf) to an OpenTelemetry collector, or to any backend that accepts OTLP, such as Jaeger, Grafana Tempo, or Honeycomb.

[telemetry]
enabled      = true
endpoint     = "http://otel-collector:4318"   # /v1/traces is appended
headers      = { authorization = "Bearer ${OTEL_TOKEN}" }
sample_ratio = 0.1                            # keep 10% of new traces (no incoming traceparent)

Sentinel then logs telemetry: exporting traces over OTLP/HTTP with the endpoint at startup, and records:

  • An http.server span per API request, named after the method and route (GET /v2/challenge), with http.route, url.path, and http.response.status_code. A request with a W3C traceparent header continues that trace.
  • Database, Redis, and rate-limit spans inside it. Redis and rate-limit spans are debug level, so add them to filter to export them: filter = "info,exotdb_redis=debug,exotdb_ratelimit=debug".
  • An http.client span per outgoing call: webhooks, OAuth, remote files, IP lookups, AI and embedding APIs, and others. Sentinel passes traceparent on only to a remote ExotDB server, ClickHouse, and alert webhooks.

sample_ratio is parent-based: a request whose incoming traceparent is sampled is always exported. filter selects exported spans independently of [log] level. Without endpoint, Sentinel reads OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, then OTEL_EXPORTER_OTLP_ENDPOINT, then uses http://localhost:4318. Traces go through the outbound proxy when one is set; see Outbound proxy & air-gapped deployments. Metrics stay on Prometheus and logs on stdout. See [telemetry] for every key.

Logs

Sentinel writes logs to stdout. [log] sets what is logged and in which format:

[log]
level  = "info"
format = "json"

Log level

level is a filter directive: a default level, optionally followed by target=level pairs separated by commas. Levels are error, warn, info, debug, and trace.

levelWhat you get
"info"Startup, configuration, background jobs, warnings, and errors (default)
"warn"Only warnings and errors
"info,sentinel::http=debug"Also one line per 4xx or 5xx response, with its status, error code, and latency
"info,sentinel::http=trace"One line per response
"info,tiberius=info"Also the SQL Server driver's routine messages, which are otherwise limited to warn

Responses aren't logged at info. To see why requests fail, raise sentinel::http to debug temporarily, or use request logs for API keys with logging on.

Log format

format = "text" (the default) writes one human-readable line per event:

2026-11-19T09:27:31.504337Z  INFO altcha_sentinel: listening addr=0.0.0.0:3000 tls=false
2026-11-19T09:27:34.864593Z DEBUG http.server{otel.name=GET /v2/challenge … http.response.status_code=403}: sentinel::http: request failed status=403 code="LICENSE_NOT_ACTIVE" latency_ms=0.250959

format = "json" writes one JSON object per line, for log collectors such as Loki, Elasticsearch, or Cloud Logging. Each object has timestamp, level, and target, the event's fields (including message) at the top level, and the enclosing span and spans when the event happens inside a request:

{"timestamp":"2026-11-19T09:27:08.780167Z","level":"INFO","message":"listening","addr":"0.0.0.0:3000","tls":false,"target":"altcha_sentinel"}
{"timestamp":"2026-11-19T09:27:11.629750Z","level":"DEBUG","message":"request failed","status":403,"code":"LICENSE_NOT_ACTIVE","latency_ms":0.126375,"target":"sentinel::http","span":{"http.request.method":"GET","http.response.status_code":403,"http.route":"/v2/challenge","otel.kind":"server","otel.name":"GET /v2/challenge","url.path":"/v2/challenge","name":"http.server"},"spans":[…]}

With [privacy] anonymize_ip = true, public client IPs are anonymized in log lines too; see Privacy & data handling. See [log] and, for reading the common startup errors, Doctor & troubleshooting.

In the admin UI

The admin UI shows the same data without a monitoring stack:

  • Dashboard: requests, verification results, errors, and visitor signals of API keys with request logging on. See Request logs & analytics.
  • Cluster (root only): every replica's version, requests, CPU, memory, and storage status, refreshed every 10 seconds, with charts of the last hour. See Clustering & high availability.
  • Doctor: live diagnostics of storage, connectivity, license, and configuration. See Doctor & troubleshooting.

Start typing to search...

↑ ↓ Navigate ↵ Select