Skip to main content
ALTCHA®

Alerts

Sentinel watches logins, verifications, API key use, its own replicas, remote files, storage reachability, the license, and the release feed, and raises an alert when something needs attention. Use this page to tune the rules and to send alerts to your chat, inbox, incident tool, or browser.

How it works

Every alert is:

  • stored in the database and searchable with POST /v2/alerts/_search and under Alerts in the admin UI;
  • logged as a warning with the target sentinel::alerts;
  • delivered to every configured webhook, to the [alerts.email] recipients, and to the browsers of users who turned on push notifications.
WARN sentinel::alerts: alert: 3 failed logins from 127.0.0.1 within 10m rule="brute_force" severity="high" subject="127.0.0.1"

Scope

An alert either belongs to one account (accountId is the account's id) or is system-wide (accountId is null). System-wide alerts concern the whole deployment, such as login attacks, replicas, storage, the license, and releases, and only root users see them. See Who receives what.

Deduplication and cooldown

An alert is one row per rule, account, subject, and cooldown window. The subject names what the alert is about: a client IP, a username, an API key id, a replica's host name, and so on. Windows are [alerts] cooldown long (default 1h) and aligned to the Unix epoch.

  • The first occurrence in a window creates the alert and delivers it.
  • Later occurrences in the same window increase its occurrences and update lastTime, message, and details, without delivering it again.

A condition that persists, such as high memory use, therefore notifies about once per cooldown. Replicas share the alerts table, so an alert that several replicas raise at once notifies once.

Alerts last raised longer ago than [alerts] retention (default 90d) are deleted hourly, by one replica at a time. With [alerts] enabled = false, no new alert is raised; search and retention keep working.

Rules

RuleSeverityRaised whenScopeConfig
brute_forcehighOne client IP fails limit logins. A wrong current password on POST /v2/me/password counts as a failed login. subject: the IP.system[alerts.brute_force]
user_brute_forcehighOne username fails limit logins from any IPs: distributed guessing that stays under the per-IP limit. subject: the username, trimmed and lowercased.system[alerts.user_brute_force]
verification_failuresmediumAn account's failed POST /v2/verify calls per minute over window reach factor times its baseline, with at least min_failures failures. subject: the account id.account[alerts.verification_failures]
cpumediumA replica's CPU use stays at or above threshold percent of its available cores every minute of duration. subject: the replica's host name.system[alerts.cpu]
memoryhighA replica's memory (RSS) stays at or above threshold percent of its limit (the cgroup limit, else host RAM) every minute of duration.system[alerts.memory]
server_errorshighAt least threshold percent of a replica's responses over duration are 5xx, out of at least min_requests.system[alerts.server_errors]
root_login_countryhighOff by default. A root user signs in from a country they never signed in from before. subject: <username>:<country>.system[alerts.root_login_country]
download_failuresmediumA remote file fails its check or download failures times in a row on a replica. subject: the file name (threat-<id> for threat sources).system[alerts.download_failures]
kv_unavailablehighA replica can't reach the KV store for duration. subject: the replica's host name.system[alerts.kv_unavailable]
db_unavailablehighA replica can't reach the database for duration. subject: the replica's host name.system[alerts.db_unavailable]
api_key_networkshighA restricted or full API key is used from limit distinct networks within the window. subject: the key id.account[alerts.api_key_networks]
api_key_deniedhighA restricted or full API key with a valid secret is denied by its security group (403 ACCESS_DENIED). Whoever sent it holds the secret. subject: the key id.account[alerts.api_key_denied]
api_key_browserhighA restricted or full API key is sent with an Origin or Referer header, that is, from a browser page. subject: the key id.account[alerts.api_key_browser]
api_key_bearermediumA restricted or full API key id is sent without its secret (403 BEARER_AUTH_PUBLIC_API_KEY). subject: the key id.account[alerts.api_key_bearer]
api_key_wrong_secrethighA restricted or full API key gets limit wrong secrets. subject: the key id.account[alerts.api_key_wrong_secret]
license_expiredhighThe license expired, was revoked, or is overdue for its online refresh.system—
license_instance_limithighA replica went over the license's instance limit.system—
license_refresh_failingmediumThe online license refresh has been failing for 24 hours.system—
license_clock_rollbackmediumThe system clock is behind the license's stored time floor.system—
release_criticalhighA release newer than the running version is flagged [Critical], [Security], or UPDATE RECOMMENDED. subject: the highest such version.system[alerts.release_critical]
release_outdatedmediumReleases newer than the running version have been out longer than max_age. subject: the running version.system[alerts.release_outdated]

The license_* rules have no config section of their own; only [alerts] enabled = false stops them. See Alerts and metrics in the license guide.

How the rules count

  • Failed logins are every sign-in step the audit log records as failure: password, LDAP, OTP, passkey, and OAuth, whatever the error code. Failed step-up checks on POST /v2/me/verify count too. They count whether or not audit logs are enabled or licensed. Requests rejected by the rate limit (429) or with a malformed body don't count. The counters are fixed windows in the KV store, shared by all replicas, and each raises once per window, when it reaches limit.
  • Failed verifications are POST /v2/verify calls that return verified: false or 400 INVALID_PAYLOAD, for every API key, whatever its request logging. The baseline is the failures per minute over the baseline hours before the window, counting only hours with verifications, so quiet nights don't make normal daytime traffic look like a spike. Without a baseline yet (a new account or deployment), min_failures alone decides.
  • cpu, memory, and server_errors are checked every minute by each replica, on its own metrics. A minute without samples breaks the streak. CPU is relative to the cores the process may use (the cgroup quota on Linux), so threshold = 90 on a 2-core pod means 180% of one core.
  • root_login_country needs an IP resolver ([[ip_resolver.providers]]); with the rule on and no provider, startup fails. Countries are learned per root user and never expire. Each root user's first country, including the first sign-in after you turn the rule on, is learned silently. Private IPs and IPs no provider locates are skipped. It's off by default because IP geolocation is approximate: VPNs, mobile carriers, and corporate egress cause false alerts, and an attacker with a VPN in the victim's country avoids it.
  • download_failures counts failed checks and downloads in a row per file and replica. A completed download, an up-to-date check, or a restart resets it.
  • kv_unavailable and db_unavailable: each replica probes the KV store and the database every 10 seconds, with a 5-second timeout. While the database is down, the alert can't be stored, but it still reaches the log and webhooks (not web push). Recovery raises no alert; the replica logs storage reachable again.
  • api_key_* rules watch restricted and full keys only; public keys ship in widgets. api_key_denied, api_key_browser, and api_key_bearer raise at most once a minute per key and replica, and their details count the requests since the last one. Rules in monitor mode don't raise api_key_denied. Trying a server key in the API reference from a browser raises api_key_browser. See API keys & access levels.
  • release_* rules are checked after every fetch of the release feed ([releases], every 6h by default, and at startup). An unchanged subject is raised at most once a day. With [releases] enabled = false, there are no checks.

Configure the rules

Each rule has its own [alerts.<rule>] table with an enabled switch and the thresholds the rule table names. Defaults and ranges are in the Configuration reference. For example, to alert on fewer failed logins, turn on new-country alerts for root, and silence the CPU rule:

[alerts]
cooldown = "30m"            # notify about a persisting condition every 30 minutes

[alerts.brute_force]
limit = "5/15m"             # failures per client IP and window

[alerts.root_login_country]
enabled = true              # needs an [[ip_resolver.providers]] entry

[alerts.cpu]
enabled = false

Invalid values fail startup.

IP anonymization

Alerts that show client IPs (brute_force, user_brute_force, root_login_country, and the api_key_* rules) anonymize them like request logs (203.0.113.7 becomes 203.0.113.0) when [alerts] anonymize_ip is true, in the stored alert and in every delivery. Unset, it follows [privacy] anonymize_ip, which is off by default. See Privacy & data handling.

brute_force still counts failures per full IP. Anonymized, the IPs of one /24 (IPv4) or /112 (IPv6) share a subject, so their alerts in one cooldown window count onto one row.

Webhooks

Each [[alerts.webhooks]] table adds an endpoint that every new alert is POSTed to. Add as many as you need, each with its own filter:

[[alerts.webhooks]]
url          = "https://alerts.example.com/sentinel"
secret       = "${file:/run/secrets/alerts_webhook_secret}"   # signs each request; at least 16 characters
min_severity = "medium"                                       # low | medium | high

[[alerts.webhooks]]
url    = "${SLACK_WEBHOOK_URL}"   # the URL holds a token: keep it out of the file
format = "slack"
rules  = ["brute_force", "user_brute_force", "root_login_country"]

Payload

With format = "json" (the default), the body is the alert as the search returns it, without lastTime and occurrences, plus replica, the id of the replica that raised it:

{
  "accountId": null,
  "details": { "failures": 3, "ip": "203.0.113.7", "window": "10m" },
  "id": "alr_…",
  "message": "3 failed logins from 203.0.113.7 within 10m",
  "replica": "rep_…",
  "rule": "brute_force",
  "severity": "high",
  "subject": "203.0.113.7",
  "time": "2026-11-19T09:25:45.647000+00:00"
}

With format = "slack", the body is a single line of text, which Slack incoming webhooks and compatible services (Mattermost, Rocket.Chat, Discord's /slack webhook URL) accept:

{"text":"*[HIGH] Brute-force login*: 3 failed logins from 203.0.113.7 within 10m"}

Both are sent with Content-Type: application/json and User-Agent: ALTCHA-Sentinel/<version>, plus any headers you configure.

Verify the signature

With a secret, every request carries two headers:

HeaderValue
X-Sentinel-TimestampUnix time in seconds when the request was sent
X-Sentinel-Signaturesha256=<hex>: the HMAC-SHA256, keyed with the secret, of <timestamp>.<raw body>

Compute the HMAC over the raw request body, before any JSON parsing, and reject old timestamps to stop replays. A Node.js handler:

import { createHmac, timingSafeEqual } from 'node:crypto';
import express from 'express';

const secret = process.env.ALERTS_WEBHOOK_SECRET!;

function verifySentinelSignature(rawBody: string, timestamp?: string, signature?: string): boolean {
  if (!timestamp || !signature) return false;
  // Reject requests older (or newer) than 5 minutes.
  if (Math.abs(Date.now() / 1000 - Number(timestamp)) > 300) return false;
  const expected = 'sha256=' + createHmac('sha256', secret).update(`${timestamp}.${rawBody}`).digest('hex');
  const a = Buffer.from(expected);
  const b = Buffer.from(signature);
  return a.length === b.length && timingSafeEqual(a, b);
}

const app = express();

app.post('/sentinel', express.text({ type: 'application/json' }), (req, res) => {
  if (!verifySentinelSignature(req.body, req.get('X-Sentinel-Timestamp'), req.get('X-Sentinel-Signature'))) {
    return res.sendStatus(401);
  }
  const alert = JSON.parse(req.body);
  console.log(alert.severity, alert.rule, alert.message);
  res.sendStatus(204);
});

Retries

A request that fails to connect or gets a 429 or 5xx is retried twice, after 2 and 10 seconds, with a 10-second timeout per attempt. Any other response is final. Failures are logged with the webhook's origin only, never its URL or headers, and the alert stays stored. The last outcome of each webhook shows in the alerts.delivery check of Doctor.

Email

[alerts.email] sends every new alert as one plain-text message to all to recipients. It's on when host is set:

[alerts.email]
host     = "smtp.example.com"
tls      = "starttls"                       # starttls | implicit | none (local relays only)
username = "alerts@example.com"
password = "${file:/run/secrets/smtp_password}"
from     = "Sentinel <alerts@example.com>"
to       = ["ops@example.com", "security@example.com"]
min_severity = "high"
  • The subject is [Sentinel] [HIGH] Brute-force login: <message>. The body is the message, followed by the webhook JSON.
  • At most rate emails (default 20/1h) are sent per window, counted per SMTP server across replicas. Alerts beyond it are still stored and sent to webhooks and push.
  • A send that fails to connect, times out, or gets a 4xx reply is retried twice, after 2 and 10 seconds; a 5xx reply is final.
  • At startup, each replica checks that the SMTP server answers and logs the result. Startup doesn't fail on it.

SMTP connections never use the outbound HTTP proxy.

Web push

Users can receive alerts as browser notifications, even when the admin UI isn't open. Turn it on by setting [alerts.push] subject, a contact URL that push services can use to reach you:

[alerts.push]
subject = "mailto:security@example.com"

Notifications are signed with the VAPID key [secrets] vapid_private_key. With embedded SQLite, Sentinel generates it when unset; with any other database, startup fails without it. Changing the key invalidates every subscription, so users must turn push notifications on again. See Server secrets.

Each user then turns notifications on per browser in the admin UI, under the user menu → Push notifications (also linked from the Alerts page), and can send a test notification there. A user can have up to 20 subscribed devices. Each subscription receives at most [alerts.push] rate notifications per window (default 20/1h); alerts beyond it are still stored and sent to webhooks. high alerts are sent with Urgency: high, others with normal.

[alerts.push] endpoint_hosts limits the push services a browser may register, so users can't make Sentinel POST to arbitrary hosts. The default allows the services of Chrome, Edge, Firefox, and Safari. The endpoints are under /v2/me/push-subscriptions in the API reference.

Who receives what

AlertRecipients
System-wide (accountId: null)Active root users
An account'sActive root users, and users whose role in that account grants the alerts feature (the built-in member and readonly roles do)

The same rules decide who can search an alert. Recipients are checked when each alert is sent, so a role change or a deactivated user takes effect immediately. See Accounts, users & roles.

Webhooks and email have no recipient filter beyond rules and min_severity: they receive every alert, of every account.

Search alerts

POST /v2/alerts/_search accepts an admin session, or a full API key whose role grants alerts (see API keys & access levels):

curl -X POST https://sentinel.example.com/v2/alerts/_search \
  -H 'Authorization: Bearer <jwt>' \
  -H 'Content-Type: application/json' \
  -d '{ "filter": { "rule": "brute_force" }, "sortDir": "desc", "limit": 10 }'
{
  "data": [
    {
      "id": "alr_…",
      "time": "2026-11-19T09:25:45.647000+00:00",
      "lastTime": "2026-11-19T09:25:45.647000+00:00",
      "rule": "brute_force",
      "severity": "high",
      "accountId": null,
      "subject": "203.0.113.7",
      "message": "3 failed logins from 203.0.113.7 within 10m",
      "details": { "failures": 3, "ip": "203.0.113.7", "window": "10m" },
      "occurrences": 1
    }
  ],
  "pagination": { "limit": 10, "offset": 0, "total": 1 }
}

time is when the alert was first raised in its window, and lastTime the latest occurrence. details depends on the rule:

Ruledetails
brute_forceip, failures, window
user_brute_forceusername, failures, window, lastIp
verification_failuresfailures, verifications (in the window), window, failuresPerMinute, baselinePerMinute, baselineHours
cpureplica, host, cores, averagePercent, threshold, duration
memoryreplica, host, bytes, limitBytes, percent, threshold, duration
server_errorsreplica, host, requests, errors5xx, percent, threshold, duration
root_login_countryusername, userId, country, countryName, ip, asname, proxy, tor, hosting, providerErrors, knownCountries
download_failuresfile, failures, threshold, error, downloadedAt, threatSourceId, replica, host (never the file's URL)
kv_unavailable, db_unavailablereplica, host, since, downSeconds, error, duration
api_key_networksapiKeyId, apiKeyName, networks, window, network, ipv4Prefix, ipv6Prefix
api_key_deniedapiKeyId, apiKeyName, securityGroupId, rule (index), ruleName, ip, method, path, denials
api_key_browserapiKeyId, apiKeyName, origin, ip, path, requests
api_key_bearerapiKeyId, apiKeyName, via (bearer or query), origin, ip, attempts
api_key_wrong_secretapiKeyId, apiKeyName, failures, window, lastIp
release_criticalversion, title, link, publishedAt, runningVersion, versions
release_outdatedrunningVersion, latestVersion, latestLink, newerSince, days, maxAge
filterDescription
accountIdOne account's alerts. Absent: every alert; null: system-wide alerts only. Both are root only: a non-root caller without accountId gets 400 VALIDATION_ERROR, and with null gets 403 ROOT_PRIVILEGES_REQUIRED.
ruleOne of the rules; any other value is 400 VALIDATION_ERROR.
severitylow, medium, or high.
from, toRFC 3339; first raised at or after from, before to.

sortBy is time (default) or lastTime, and sortDir defaults to asc. See POST /v2/alerts/_search in the API reference.

Alerts have no acknowledged or resolved state. A condition that persists raises the same alert again in the next cooldown window, and old alerts expire after [alerts] retention.

Start typing to search...

↑ ↓ Navigate ↵ Select