Monitoring and Logging

ALTCHA Sentinel provides several monitoring and logging endpoints for integration with external observability tools:

• GET /.health – Comprehensive health check
• GET /.ready – Readiness probe
• GET /.live – Liveness probe
• GET /.logs – Real-time log streaming
• GET /.metrics – Prometheus metrics

By default, these endpoints are accessible only from the internal (private) network. See Access Control for configuration details.

OpenTelemetry

Sentinel supports the OTLP HTTP exporter for OpenTelemetry collectors, enabling both traces and logs. To enable OpenTelemetry, set the OTEL_EXPORTER_OTLP_ENDPOINT environment variable.

Supported environment variables

• OTEL_EXPORTER_OTLP_ENDPOINT: The URL of the OpenTelemetry collector (e.g., https://localhost:4318).
• OTEL_EXPORTER_OTLP_HEADERS: Custom headers to include in OTLP requests (e.g., API keys or authentication tokens).
• OTEL_EXPORTER_OTLP_TIMEOUT: The timeout value for all outgoing data in milliseconds.
• OTEL_SERVICE_NAME: The logical name of the service that will appear in traces and logs (defaults to altcha-sentinel).

Example configuration

OTEL_SERVICE_NAME="altcha-sentinel"
OTEL_EXPORTER_OTLP_ENDPOINT="https://localhost:4318"
OTEL_EXPORTER_OTLP_HEADERS="x-api-key=your-api-key"

Caution

OpenTelemetry is available only with an Enterprise license.

Health Check Endpoints

Sentinel exposes three health-related endpoints for system monitoring, each serving a different purpose:

Comprehensive Health Check

GET /.health

Performs a full health check including critical subsystems like the database and Redis. Returns HTTP 503 if any component is unhealthy.

Example Response:

{
  "status": "healthy",
  "checks": {
    "database": {
      "ok": true
    },
    "redis": {
      "ok": true
    },
    "clickhouse": {
      "ok": true
    }
  },
  "timestamp": "2025-05-12T14:05:54.464Z",
  "version": "1.0.0"
}

Readiness Probe

GET /.ready

Checks whether the server is ready to receive traffic. Returns HTTP 200 if ready, or 503 if not.

Example Response:

{
  "status": "ready",
  "timestamp": "2025-05-12T14:06:08.460Z",
  "version": "1.0.0"
}

Liveness Probe

GET /.live

A basic endpoint that confirms the server is running. Always returns HTTP 200 if alive.

Example Response:

{
  "status": "alive",
  "timestamp": "2025-05-12T14:06:57.007Z",
  "version": "1.0.0"
}

Tip

For external health checks over the public Internet, prefer the GET /v1/ping endpoint instead of the internal health check URLs.

Metrics

Prometheus-Compatible Metrics

GET /.metrics

Exposes runtime metrics in a format compatible with Prometheus.

Available Metrics

The metrics endpoint provides system-level insights to help monitor your Sentinel instances. Key metrics include:

• sentinel_api_challenges_total: Total number of challenges generated (counter)
• sentinel_api_verifications_failed_total: Total number of failed verifications (counter)
• sentinel_api_verifications_success_total: Total number of successful verifications (counter)
• sentinel_http_requests_per_minute: Number of HTTP requests per minute (gauge)
• sentinel_http_requests_total: Total number of HTTP requests (counter)
• sentinel_http_status_4xx_per_minute: Number of HTTP 4xx responses per minute (gauge)
• sentinel_http_status_5xx_per_minute: Number of HTTP 5xx responses per minute (gauge)
• sentinel_http_status_4xx_total: Total number of HTTP status 4xx responses (counter)
• sentinel_http_status_5xx_total: Total number of HTTP status 5xx responses (counter)
• sentinel_http_request_duration_seconds: Duration of HTTP requests in seconds (histogram)
• sentinel_process_cpu_usage_percent: CPU usage of the process in percent (gauge)
• sentinel_process_memory_usage_percent: Memory usage of the process in percent (gauge)
• sentinel_process_resident_memory_bytes: Resident memory size in bytes (gauge)

To see all available metrics, view the response of GET /.metrics.

Logging

The Sentinel server implements the following logging capabilities:

1. Standard Output: All logs are written to stdout for capture by standard logging tools
2. Live Log Streaming: Provides real-time log access via HTTP

Note

This documentation covers server logging only. Request logging can be configured individually for each API key, and the retention period for request logs can be adjusted using the REQUEST_LOGS_TTL environment variable.

Live Log Streaming

GET /.logs

Streams server logs in real-time. Supports optional JSON formatting.

Query Parameters:

• format=json (optional): Outputs logs in JSON format

Examples:

• Default stream: GET /.logs
• JSON stream: GET /.logs?format=json

Access Control

To protect sensitive monitoring data, access is restricted using the following environment variables and, by default, is limited to internal (private) networks only:

MONITORING_IP_WHITELIST

Specifies which IP ranges are allowed to access monitoring endpoints. Default includes:

10.0.0.0/8           // Corporate networks
172.16.0.0/12        // Medium-sized private networks
192.168.0.0/16       // Office/home networks
127.0.0.1/32         // IPv4 localhost
::1/128              // IPv6 localhost
fd00::/8             // IPv6 unique local addresses
100.64.0.0/10        // Shared address space (CGNAT)

To allow access from any IP address, set MONITORING_IP_WHITELIST to an empty string ("").

MONITORING_HTTP_CREDENTIALS

Enables HTTP Basic Authentication using the format: <username>:<password>