Monitoring and Logging

ALTCHA Sentinel provides several monitoring and logging endpoints for integration with external observability tools:

• GET /.health – Comprehensive health check
• GET /.ready – Readiness probe
• GET /.live – Liveness probe
• GET /.logs – Real-time log streaming
• GET /.metrics – Prometheus metrics

By default, these endpoints are accessible only from the internal (private) network. See Access Control for configuration details.

OpenTelemetry

Sentinel supports the OTLP HTTP exporter for OpenTelemetry collectors, enabling both traces and logs. To enable OpenTelemetry, set the OTEL_EXPORTER_OTLP_ENDPOINT environment variable.

Supported environment variables

• OTEL_EXPORTER_OTLP_ENDPOINT: The URL of the OpenTelemetry collector (e.g., https://localhost:4318).
• OTEL_EXPORTER_OTLP_HEADERS: Custom headers to include in OTLP requests (e.g., API keys or authentication tokens).
• OTEL_EXPORTER_OTLP_TIMEOUT: The timeout value for all outgoing data in milliseconds.
• OTEL_SERVICE_NAME: The logical name of the service that will appear in traces and logs (defaults to altcha-sentinel).

Example configuration

OTEL_SERVICE_NAME="altcha-sentinel"
OTEL_EXPORTER_OTLP_ENDPOINT="https://localhost:4318"
OTEL_EXPORTER_OTLP_HEADERS="x-api-key=your-api-key"

Caution

OpenTelemetry is available only with an Enterprise license.

Health Check Endpoints

Sentinel exposes three health-related endpoints for system monitoring, each serving a different purpose:

Comprehensive Health Check

GET /.health

Performs a full health check including critical subsystems like the database and Redis. Returns HTTP 503 if any component is unhealthy.

Example Response:

{
  "status": "healthy",
  "checks": {
    "database": {
      "ok": true
    },
    "redis": {
      "ok": true
    }
  },
  "timestamp": "2025-05-12T14:05:54.464Z",
  "version": "1.0.0"
}

Readiness Probe

GET /.ready

Checks whether the server is ready to receive traffic. Returns HTTP 200 if ready, or 503 if not.

Example Response:

{
  "status": "ready",
  "timestamp": "2025-05-12T14:06:08.460Z",
  "version": "1.0.0"
}

Liveness Probe

GET /.live

A basic endpoint that confirms the server is running. Always returns HTTP 200 if alive.

Example Response:

{
  "status": "alive",
  "timestamp": "2025-05-12T14:06:57.007Z",
  "version": "1.0.0"
}

Tip

For external health checks over the public Internet, prefer the GET /v1/ping endpoint instead of the internal health check URLs.

Metrics

Prometheus-Compatible Metrics

GET /.metrics

Exposes runtime metrics in a format compatible with Prometheus.

Available Metrics

The metrics endpoint provides system-level insights to help monitor your Sentinel instances. Key metrics include:

• sentinel_process_cpu_usage_percent: CPU usage of the process, in percent (gauge)
• sentinel_process_memory_usage_percent: Memory usage of the process, in percent (gauge)
• sentinel_process_resident_memory_bytes: Resident memory size in bytes (gauge)
• sentinel_http_requests_per_minute: Number of HTTP requests per minute (gauge)
• sentinel_http_status_4xx_per_minute: Number of HTTP 4xx responses per minute (gauge)
• sentinel_http_status_5xx_per_minute: Number of HTTP 5xx responses per minute (gauge)
• sentinel_http_request_duration_seconds: Duration of HTTP requests in seconds (histogram)

Logging

The Sentinel server implements the following logging capabilities:

1. Standard Output: All logs are written to stdout for capture by standard logging tools
2. Live Log Streaming: Provides real-time log access via HTTP

Note

This documentation covers server logging only. Request logging can be configured individually for each API key, and the retention period for request logs can be adjusted using the REQUEST_LOGS_TTL environment variable.

Live Log Streaming

GET /.logs

Streams server logs in real-time. Supports optional JSON formatting.

Query Parameters:

• format=json (optional): Outputs logs in JSON format

Examples:

• Default stream: GET /.logs
• JSON stream: GET /.logs?format=json

Access Control

To protect sensitive monitoring data, access is restricted using the following environment variables and, by default, is limited to internal (private) networks only:

MONITORING_IP_WHITELIST

Specifies which IP ranges are allowed to access monitoring endpoints. Default includes:

10.0.0.0/8           // Corporate networks
172.16.0.0/12        // Medium-sized private networks
192.168.0.0/16       // Office/home networks
127.0.0.1/32         // IPv4 localhost
::1/128              // IPv6 localhost
fd00::/8             // IPv6 unique local addresses
100.64.0.0/10        // Shared address space (CGNAT)

To allow access from any IP address, set MONITORING_IP_WHITELIST to an empty string ("").

MONITORING_HTTP_CREDENTIALS

Enables HTTP Basic Authentication using the format: <username>:<password>