# Monitoring & metrics

> **Sentinel v2 (beta):** This page documents Sentinel v2, a beta rewrite that may change before the stable release. It does not apply to Sentinel v1, the current stable version: see the [Sentinel v1 documentation](/docs/sentinel.md).

Sentinel exposes a liveness endpoint for load balancers and orchestrators, a storage health endpoint for uptime monitors, Prometheus metrics, OpenTelemetry traces, and structured logs. This page shows what each one reports and how to wire it into your monitoring.

Every key mentioned here is listed with its default in the [Configuration reference](/docs/sentinel/v2/reference/configuration/). Changes to these keys take effect after a restart.

## What to use for what

| Endpoint or output | Where | Use it for | Access |
| --- | --- | --- | --- |
| `GET /` | Main and public listeners | Liveness and readiness probes, `healthcheck` | Anyone |
| `GET /.health` | Main listener only | Uptime monitoring and alerting on storage | Client IP in `[server] health_allowed_ips` |
| `/metrics` | [`[prometheus]`](/docs/sentinel/v2/reference/configuration/#prometheus) listener, off by default | Prometheus scraping, dashboards, alert rules | `bearer_token`, `allowed_ips` |
| OTLP traces | Outgoing, to your collector | Request latency breakdowns across Sentinel, its database, and outgoing calls | Collector credentials in `[telemetry] headers` |
| Logs | stdout | Errors, startup state, request failures | Your log pipeline |

None of these needs a license, and none of them is rate limited or written to request logs.

<Callout type="note" title="Changed in v2">

Sentinel v2 has no `/.live`, `/.ready`, `/v1/ping`, `/.logs`, or `/.metrics` endpoints. Probe `GET /`, monitor `GET /.health`, and scrape metrics from the `[prometheus]` listener. See [Breaking changes](/docs/sentinel/v2/upgrade/breaking-changes/).

## Liveness and readiness: `GET /`

`GET /` answers `200` with `{"ok":true}` as long as the process serves HTTP. It doesn't touch the database or Redis, so a storage outage doesn't take replicas out of service. It answers on any host, so probes that address the pod or container IP work even when `allowed_hosts` is set.

```bash
curl -i http://localhost:3000/
```

```http
HTTP/1.1 200 OK
content-type: application/json
cache-control: no-store
content-length: 11

{"ok":true}
```

Use it for both liveness and readiness. A Kubernetes container spec, for example:

```yaml
livenessProbe:
httpGet:
path: /
port: 3000
periodSeconds: 10
failureThreshold: 3
readinessProbe:
httpGet:
path: /
port: 3000
periodSeconds: 10
```

With `[server] tls` set, add `scheme: HTTPS` to both probes. Behind a load balancer, point its health check at `GET /` on the listener it forwards to.

### The `healthcheck` command

The container image has no `curl` or `wget`, so it ships a built-in check: `altcha-sentinel healthcheck` sends `GET /` to `[server] port` on this host and exits with `0` when the server answers `{"ok": true}`, or `1` otherwise. With `[server] tls` set, it uses HTTPS and skips certificate checks, because the certificate names the public host rather than the probed address.

The image's `HEALTHCHECK` runs it every 10 seconds against `./sentinel.toml`. If you start Sentinel with `--config`, pass the same path to the check, so it probes the port the server listens on:

```bash
docker exec sentinel /usr/local/bin/altcha-sentinel --config /etc/sentinel/sentinel.toml healthcheck
echo $?
```

```text
0
```

On failure, the reason goes to stderr. Without `--config` and without `./sentinel.toml`, it probes the default port `3000` and says so:

```text
http://127.0.0.1:3000/: unexpected answer …
no --config and no ./sentinel.toml: probed the default port 3000; pass the server's --config
```

See [`healthcheck`](/docs/sentinel/v2/reference/cli/#healthcheck) for the exit codes, and [Install with Docker](/docs/sentinel/v2/install/docker/) for the `--health-cmd` override.

## Storage health: `GET /.health`

`GET /.health` on the main listener reports whether this replica reaches its storage: `kv` (Redis or the built-in KV store), `db` (the database), and `clickhouse` when [ClickHouse](/docs/sentinel/v2/operations/clickhouse/) is configured. It answers from background probes that run every 10 seconds, so polling it never adds load to your database. A check whose last probe is older than 30 seconds fails.

```bash
curl -i http://10.0.0.20:3000/.health
```

```http
HTTP/1.1 200 OK
content-type: application/json
cache-control: no-store
content-length: 65

{"checks":{"db":{"ok":true},"kv":{"ok":true}},"status":"healthy"}
```

When any check fails, the status is `503` and `"status"` is `"unhealthy"`. The response has no error messages or latencies; root users find those under **Cluster** in the admin UI, or in `storage` of [`GET /v2/cluster/replicas`](/sentinel-api-docs-v2.html#tag/cluster/GET/v2/cluster/replicas).

### Who may call it

By default only loopback and private addresses (RFC 1918, `100.64.0.0/10`, link-local, and their IPv6 equivalents) may call `/.health`. Everyone else gets `403 IP_NOT_ALLOWED`:

```bash
curl -i https://sentinel.example.com/.health
```

```http
HTTP/1.1 403 Forbidden
content-type: application/json
cache-control: no-store
content-length: 75

{"code":"IP_NOT_ALLOWED","error":"Client IP not allowed.","statusCode":403}
```

Probes from Docker, Kubernetes, and AWS load balancers connect from private addresses and need no setting. Public probers, such as Google Cloud load balancers, Azure, and external uptime monitors, must be added. Setting the key replaces the default list, so keep the private ranges you still need:

```toml
[server]
health_allowed_ips = ["10.0.0.0/8", "35.191.0.0/16", "130.211.0.0/22"]   # private network + Google Cloud health checkers
```

A request that carries `X-Forwarded-For` or `X-Real-IP` from a peer outside `trusted_proxies` is also refused, so list your proxy in `[server] trusted_proxies` if it forwards `/.health`. The full access rules are in [Health endpoint](/docs/sentinel/v2/reference/configuration/#health-endpoint).

<Callout type="warning" title="Don't probe /.health for liveness or readiness">

Every replica shares the same storage, so a database or Redis outage turns every replica's `/.health` to `503` at the same time. A readiness probe would take all replicas out of service, and a liveness probe would restart them without fixing anything. Probe `GET /` and alert on `/.health` or on `altcha_sentinel_storage_up`.

## Prometheus metrics

The `[prometheus]` listener serves metrics in the Prometheus text format on its own port, so scrapes aren't counted in the request metrics, logged, or rate limited. Each replica serves its own metrics; scrape every replica.

### Enable the listener

The listener binds to `127.0.0.1:3001` by default, which a scraper in another container or host can't reach. When you bind it beyond loopback, protect it with a bearer token, an IP allowlist, or both:

```toml
[prometheus]
enabled      = true
host         = "0.0.0.0"
port         = 3001
bearer_token = "${file:/run/secrets/prometheus_token}"
allowed_ips  = ["10.0.0.0/8"]            # the TCP peer; X-Forwarded-For is not read
allowed_hosts = ["10.0.0.20"]            # optional; scrapers usually address the IP
# tls = { cert = "/etc/sentinel/tls/fullchain.pem", key = "/etc/sentinel/tls/privkey.pem" }
```

The listener answers in plain text, not the API's JSON envelope:

| Request | Response |
| --- | --- |
| TCP peer outside `allowed_ips` | `403` with `forbidden` |
| Missing or wrong `Authorization: Bearer <token>` | `401` with `WWW-Authenticate: Bearer` and `unauthorized` |
| `Host` outside `allowed_hosts` | `421` with `misdirected request` |

```bash
curl -i http://10.0.0.20:3001/metrics
```

```http
HTTP/1.1 401 Unauthorized
content-type: text/plain; charset=utf-8
www-authenticate: Bearer
cache-control: no-store
content-length: 13

unauthorized
```

With the token:

```bash
curl -H "Authorization: Bearer $PROMETHEUS_TOKEN" http://10.0.0.20:3001/metrics
```

```text
# HELP altcha_sentinel_build_info Version and identity of this replica; always 1.
# TYPE altcha_sentinel_build_info gauge
altcha_sentinel_build_info{version="2.0.0-beta.1",replica_id="rep_…",hostname="sentinel-0"} 1
# HELP altcha_sentinel_http_requests_total HTTP requests served by the API listener.
# TYPE altcha_sentinel_http_requests_total counter
altcha_sentinel_http_requests_total 3
…
# HELP altcha_sentinel_storage_up 1 when the last periodic probe of the service succeeded, else 0.
# TYPE altcha_sentinel_storage_up gauge
altcha_sentinel_storage_up{service="kv"} 1
altcha_sentinel_storage_up{service="db"} 1
…
```

The startup log confirms the listener: `prometheus metrics listening addr=0.0.0.0:3001 path=/metrics tls=false`. See [`[prometheus]`](/docs/sentinel/v2/reference/configuration/#prometheus) for `path` and the other keys.

### Scrape configuration

Keep the token in a file that Prometheus reads, and list every replica:

```yaml
scrape_configs:
- job_name: altcha-sentinel
scrape_interval: 30s
authorization:
  type: Bearer
  credentials_file: /etc/prometheus/secrets/sentinel_token
static_configs:
  - targets: ["10.0.0.20:3001", "10.0.0.21:3001"]
```

With `tls` set on the listener, add `scheme: https` and a `tls_config` that trusts your certificate. On Kubernetes, use your usual pod discovery (`kubernetes_sd_configs` or a `PodMonitor`) against the metrics port instead of static targets.

### Metric families

Every name starts with `altcha_sentinel_`. Counters are totals since the replica started and reset on restart, so query them with `rate()` or `increase()`. A family whose source doesn't exist on this replica is left out rather than exported as zeros.

**Requests and outcomes**

| Metric | Type | Labels | Meaning |
| --- | --- | --- | --- |
| `altcha_sentinel_http_requests_total` | counter | — | Requests served by the API: every request on the main and public listeners (including `GET /`, `/.health`, and `429`s) and `/v2` on the admin UI listener. Scrapes and admin app files aren't counted. |
| `altcha_sentinel_http_request_errors_total` | counter | `class` (`4xx`, `5xx`) | Responses with a 4xx or 5xx status. |
| `altcha_sentinel_http_request_duration_seconds` | histogram | `le` | Latency to the response head. Buckets from `0.01` to `60` seconds. |
| `altcha_sentinel_http_requests_in_flight` | gauge | — | Requests being served. |
| `altcha_sentinel_requests_blocked_total` | counter | `reason` | Requests refused, by `rule_deny`, `rate_limit_global`, `rate_limit_rule`, `rate_limit_audio`, `header_entropy`, `his`, or `ip_blocked`. |
| `altcha_sentinel_challenges_issued_total` | counter | `algorithm`, `protection_level` | Challenges created, e.g. `algorithm="PBKDF2/SHA-256"`, `protection_level="frictionless"` or `"code_challenge"`. Present after the first challenge. |
| `altcha_sentinel_verifications_total` | counter | `result`, `reason` | `result="verified"`, or `result="failed"` with the failure code as `reason` (e.g. `CHALLENGE_EXPIRED`, `HIS_BLOCKED`). |
| `altcha_sentinel_classifications_total` | counter | `endpoint`, `verdict` | Classifier verdicts (`GOOD`, `NEUTRAL`, `BAD`) by `endpoint`: `classifier`, `eml`, or `verify`. Present after the first classification. |

**Process**

| Metric | Type | Labels | Meaning |
| --- | --- | --- | --- |
| `altcha_sentinel_build_info` | gauge | `version`, `replica_id`, `hostname` | Always `1`; identifies the replica. |
| `altcha_sentinel_process_cpu_percent` | gauge | — | CPU used since the previous scrape, in percent of one core (`200` = two cores busy). Absent on the first scrape. |
| `altcha_sentinel_process_resident_memory_bytes` | gauge | — | Resident set size. |
| `altcha_sentinel_process_working_set_bytes` | gauge | — | Cgroup working set: charged memory minus reclaimable page cache. Linux containers only. |
| `altcha_sentinel_process_memory_limit_bytes` | gauge | — | The memory ceiling: the container's limit, or the host's RAM without one. |
| `altcha_sentinel_process_memory_limit_info` | gauge | `source` (`cgroup_v2`, `cgroup_v1`, `host`) | Always `1`; where the ceiling came from. |
| `altcha_sentinel_scrape_duration_seconds` | gauge | — | Time spent rendering this scrape. |

`process_cpu_percent` covers the interval since the last scrape by anyone, so give each replica a single scraper, or read CPU from your container platform instead.

**Storage and cluster**

| Metric | Type | Labels | Meaning |
| --- | --- | --- | --- |
| `altcha_sentinel_storage_up` | gauge | `service` (`kv`, `db`, `clickhouse`) | `1` when the last background probe succeeded, else `0`. The same probes as `/.health`. |
| `altcha_sentinel_storage_probe_latency_seconds` | gauge | `service` | Latency of the last successful probe. |
| `altcha_sentinel_replicas` | gauge | — | Live replicas this one sees, itself included. |
| `altcha_sentinel_cluster_pubsub` | gauge | — | `1` when replicas talk over Redis pub/sub, `0` with the built-in KV store. |
| `altcha_sentinel_kv_commands_total` | counter | — | Commands run by the built-in KV store. Built-in KV store only. |
| `altcha_sentinel_kv_connections` | gauge | — | Open connections to the built-in KV store. Built-in KV store only. |
| `altcha_sentinel_api_key_cache_entries` | gauge | — | API keys in the cached snapshot. Absent when `[api_key_cache]` is off. |
| `altcha_sentinel_api_key_cache_complete` | gauge | — | `1` when every live key fits `[api_key_cache] max_keys`; keys beyond it are read from the database. |
| `altcha_sentinel_api_key_cache_age_seconds` | gauge | — | Age of the cached snapshot; grows while reloads fail. |
| `altcha_sentinel_api_key_cache_reload_failures_total` | counter | — | Failed cache reloads. |

**Remote files, threat sources, and similarity**

| Metric | Type | Labels | Meaning |
| --- | --- | --- | --- |
| `altcha_sentinel_remote_file_updates_total` | counter | `file` | Times the file's content changed: downloads, adoptions, and updates from other replicas. |
| `altcha_sentinel_remote_file_size_bytes` | gauge | `file` | Size of the kept file. |
| `altcha_sentinel_remote_file_last_download_timestamp_seconds` | gauge | `file` | Unix time of the last completed download. |
| `altcha_sentinel_remote_file_consecutive_failures` | gauge | `file` | Consecutive failed checks and downloads. |
| `altcha_sentinel_remote_file_updating` | gauge | `file` | `1` while a check or download runs. |
| `altcha_sentinel_threat_source_entries` | gauge | `source` | Entries in the source's loaded index. |
| `altcha_sentinel_threat_source_last_load_timestamp_seconds` | gauge | `source` | Unix time of the last successful load. |
| `altcha_sentinel_threat_source_error` | gauge | `source` | `1` when the last registration or load failed. |
| `altcha_sentinel_similarity_info` | gauge | `provider`, `model_id` | Always `1`; the embedding provider and model. |
| `altcha_sentinel_similarity_loaded` | gauge | — | `1` when an embedding model is ready. |
| `altcha_sentinel_similarity_samples` | gauge | — | Training samples loaded. |
| `altcha_sentinel_similarity_embedded` | gauge | — | Training samples with an embedding. |

`file` is the remote file's name, such as `phishing` or `user-agents`; threat lists appear as `threat-<source id>`. `source` is the threat source's ID (`ths_…`), not its name. See [Remote files & data sources](/docs/sentinel/v2/configure/remote-files/) and [Threat intelligence](/docs/sentinel/v2/features/threat-intelligence/).

**Request logs**

| Metric | Type | Labels | Meaning |
| --- | --- | --- | --- |
| `altcha_sentinel_request_logs_written_total` | counter | — | Request log records written. |
| `altcha_sentinel_request_logs_dropped_total` | counter | `reason` (`buffer_full`, `write_failed`) | Records dropped because the queue was full or their batch failed to write. |
| `altcha_sentinel_request_logs_queue_depth` | gauge | — | Messages waiting for the writer. |

The `altcha_sentinel_license_*` families are described in [License alerts and metrics](/docs/sentinel/v2/configure/licensing/#alerts-and-metrics).

### Example alert rules

```yaml
groups:
- name: altcha-sentinel
rules:
  - alert: SentinelStorageDown
    expr: altcha_sentinel_storage_up == 0
    for: 1m
  - alert: SentinelHigh5xxRate
    expr: rate(altcha_sentinel_http_request_errors_total{class="5xx"}[5m]) > 1
    for: 5m
  - alert: SentinelSlowRequests
    expr: histogram_quantile(0.95, rate(altcha_sentinel_http_request_duration_seconds_bucket[5m])) > 0.5
    for: 10m
  - alert: SentinelRequestLogsDropped
    expr: increase(altcha_sentinel_request_logs_dropped_total[15m]) > 0
  - alert: SentinelRemoteFileFailing
    expr: altcha_sentinel_remote_file_consecutive_failures >= 3
```

Sentinel can also send many of these conditions itself, by webhook, email, or web push; see [Alerts](/docs/sentinel/v2/operations/alerts/).

## Tracing with OpenTelemetry

`[telemetry]` exports traces over OTLP/HTTP (protobuf) to an OpenTelemetry collector, or to any backend that accepts OTLP, such as Jaeger, Grafana Tempo, or Honeycomb.

```toml
[telemetry]
enabled      = true
endpoint     = "http://otel-collector:4318"   # /v1/traces is appended
headers      = { authorization = "Bearer ${OTEL_TOKEN}" }
sample_ratio = 0.1                            # keep 10% of new traces (no incoming traceparent)
```

Sentinel then logs `telemetry: exporting traces over OTLP/HTTP` with the endpoint at startup, and records:

- **An `http.server` span per API request**, named after the method and route (`GET /v2/challenge`), with `http.route`, `url.path`, and `http.response.status_code`. A request with a W3C `traceparent` header continues that trace.
- **Database, Redis, and rate-limit spans** inside it. Redis and rate-limit spans are `debug` level, so add them to `filter` to export them: `filter = "info,exotdb_redis=debug,exotdb_ratelimit=debug"`.
- **An `http.client` span per outgoing call**: webhooks, OAuth, remote files, IP lookups, AI and embedding APIs, and others. Sentinel passes `traceparent` on only to a remote ExotDB server, ClickHouse, and alert webhooks.

`sample_ratio` is parent-based: a request whose incoming `traceparent` is sampled is always exported. `filter` selects exported spans independently of `[log] level`. Without `endpoint`, Sentinel reads `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, then `OTEL_EXPORTER_OTLP_ENDPOINT`, then uses `http://localhost:4318`. Traces go through the outbound proxy when one is set; see [Outbound proxy & air-gapped deployments](/docs/sentinel/v2/configure/air-gapped/). Metrics stay on Prometheus and logs on stdout. See [`[telemetry]`](/docs/sentinel/v2/reference/configuration/#telemetry) for every key.

## Logs

Sentinel writes logs to stdout. `[log]` sets what is logged and in which format:

```toml
[log]
level  = "info"
format = "json"
```

### Log level

`level` is a filter directive: a default level, optionally followed by `target=level` pairs separated by commas. Levels are `error`, `warn`, `info`, `debug`, and `trace`.

| `level` | What you get |
| --- | --- |
| `"info"` | Startup, configuration, background jobs, warnings, and errors (default) |
| `"warn"` | Only warnings and errors |
| `"info,sentinel::http=debug"` | Also one line per `4xx` or `5xx` response, with its status, error code, and latency |
| `"info,sentinel::http=trace"` | One line per response |
| `"info,tiberius=info"` | Also the SQL Server driver's routine messages, which are otherwise limited to `warn` |

Responses aren't logged at `info`. To see why requests fail, raise `sentinel::http` to `debug` temporarily, or use [request logs](/docs/sentinel/v2/operations/request-logs/) for API keys with logging on.

### Log format

`format = "text"` (the default) writes one human-readable line per event:

```text
2026-11-19T09:27:31.504337Z  INFO altcha_sentinel: listening addr=0.0.0.0:3000 tls=false
2026-11-19T09:27:34.864593Z DEBUG http.server{otel.name=GET /v2/challenge … http.response.status_code=403}: sentinel::http: request failed status=403 code="LICENSE_NOT_ACTIVE" latency_ms=0.250959
```

`format = "json"` writes one JSON object per line, for log collectors such as Loki, Elasticsearch, or Cloud Logging. Each object has `timestamp`, `level`, and `target`, the event's fields (including `message`) at the top level, and the enclosing `span` and `spans` when the event happens inside a request:

```json
{"timestamp":"2026-11-19T09:27:08.780167Z","level":"INFO","message":"listening","addr":"0.0.0.0:3000","tls":false,"target":"altcha_sentinel"}
{"timestamp":"2026-11-19T09:27:11.629750Z","level":"DEBUG","message":"request failed","status":403,"code":"LICENSE_NOT_ACTIVE","latency_ms":0.126375,"target":"sentinel::http","span":{"http.request.method":"GET","http.response.status_code":403,"http.route":"/v2/challenge","otel.kind":"server","otel.name":"GET /v2/challenge","url.path":"/v2/challenge","name":"http.server"},"spans":[…]}
```

With `[privacy] anonymize_ip = true`, public client IPs are anonymized in log lines too; see [Privacy & data handling](/docs/sentinel/v2/security/privacy/). See [`[log]`](/docs/sentinel/v2/reference/configuration/#log) and, for reading the common startup errors, [Doctor & troubleshooting](/docs/sentinel/v2/operations/troubleshooting/).

## In the admin UI

The admin UI shows the same data without a monitoring stack:

- **Dashboard**: requests, verification results, errors, and visitor signals of API keys with request logging on. See [Request logs & analytics](/docs/sentinel/v2/operations/request-logs/).
- **Cluster** (root only): every replica's version, requests, CPU, memory, and storage status, refreshed every 10 seconds, with charts of the last hour. See [Clustering & high availability](/docs/sentinel/v2/operations/clustering/).
- **Doctor**: live diagnostics of storage, connectivity, license, and configuration. See [Doctor & troubleshooting](/docs/sentinel/v2/operations/troubleshooting/).

## Related

- [Listeners & reverse proxy](/docs/sentinel/v2/configure/listeners/)
- [Configuration reference: `[prometheus]`](/docs/sentinel/v2/reference/configuration/#prometheus)
- [Configuration reference: `[telemetry]`](/docs/sentinel/v2/reference/configuration/#telemetry)
- [Health endpoint](/docs/sentinel/v2/reference/configuration/#health-endpoint)
- [CLI reference: `healthcheck`](/docs/sentinel/v2/reference/cli/#healthcheck)
- [Alerts](/docs/sentinel/v2/operations/alerts/)
- [Clustering & high availability](/docs/sentinel/v2/operations/clustering/)
- [Doctor & troubleshooting](/docs/sentinel/v2/operations/troubleshooting/)
- [Install with Docker](/docs/sentinel/v2/install/docker/)
- [Install on Kubernetes](/docs/sentinel/v2/install/kubernetes/)
