Alerts
Sentinel watches logins, verifications, API key use, its own replicas, remote files, storage reachability, the license, and the release feed, and raises an alert when something needs attention. Use this page to tune the rules and to send alerts to your chat, inbox, incident tool, or browser.
How it works
Every alert is:
- stored in the database and searchable with
POST /v2/alerts/_searchand under Alerts in the admin UI; - logged as a warning with the target
sentinel::alerts; - delivered to every configured webhook, to the
[alerts.email]recipients, and to the browsers of users who turned on push notifications.
WARN sentinel::alerts: alert: 3 failed logins from 127.0.0.1 within 10m rule="brute_force" severity="high" subject="127.0.0.1"Scope
An alert either belongs to one account (accountId is the account's id) or is system-wide (accountId is null). System-wide alerts concern the whole deployment, such as login attacks, replicas, storage, the license, and releases, and only root users see them. See Who receives what.
Deduplication and cooldown
An alert is one row per rule, account, subject, and cooldown window. The subject names what the alert is about: a client IP, a username, an API key id, a replica's host name, and so on. Windows are [alerts] cooldown long (default 1h) and aligned to the Unix epoch.
- The first occurrence in a window creates the alert and delivers it.
- Later occurrences in the same window increase its
occurrencesand updatelastTime,message, anddetails, without delivering it again.
A condition that persists, such as high memory use, therefore notifies about once per cooldown. Replicas share the alerts table, so an alert that several replicas raise at once notifies once.
Alerts last raised longer ago than [alerts] retention (default 90d) are deleted hourly, by one replica at a time. With [alerts] enabled = false, no new alert is raised; search and retention keep working.
Rules
| Rule | Severity | Raised when | Scope | Config |
|---|---|---|---|---|
brute_force | high | One client IP fails limit logins. A wrong current password on POST /v2/me/password counts as a failed login. subject: the IP. | system | [alerts.brute_force] |
user_brute_force | high | One username fails limit logins from any IPs: distributed guessing that stays under the per-IP limit. subject: the username, trimmed and lowercased. | system | [alerts.user_brute_force] |
verification_failures | medium | An account's failed POST /v2/verify calls per minute over window reach factor times its baseline, with at least min_failures failures. subject: the account id. | account | [alerts.verification_failures] |
cpu | medium | A replica's CPU use stays at or above threshold percent of its available cores every minute of duration. subject: the replica's host name. | system | [alerts.cpu] |
memory | high | A replica's memory (RSS) stays at or above threshold percent of its limit (the cgroup limit, else host RAM) every minute of duration. | system | [alerts.memory] |
server_errors | high | At least threshold percent of a replica's responses over duration are 5xx, out of at least min_requests. | system | [alerts.server_errors] |
root_login_country | high | Off by default. A root user signs in from a country they never signed in from before. subject: <username>:<country>. | system | [alerts.root_login_country] |
download_failures | medium | A remote file fails its check or download failures times in a row on a replica. subject: the file name (threat-<id> for threat sources). | system | [alerts.download_failures] |
kv_unavailable | high | A replica can't reach the KV store for duration. subject: the replica's host name. | system | [alerts.kv_unavailable] |
db_unavailable | high | A replica can't reach the database for duration. subject: the replica's host name. | system | [alerts.db_unavailable] |
api_key_networks | high | A restricted or full API key is used from limit distinct networks within the window. subject: the key id. | account | [alerts.api_key_networks] |
api_key_denied | high | A restricted or full API key with a valid secret is denied by its security group (403 ACCESS_DENIED). Whoever sent it holds the secret. subject: the key id. | account | [alerts.api_key_denied] |
api_key_browser | high | A restricted or full API key is sent with an Origin or Referer header, that is, from a browser page. subject: the key id. | account | [alerts.api_key_browser] |
api_key_bearer | medium | A restricted or full API key id is sent without its secret (403 BEARER_AUTH_PUBLIC_API_KEY). subject: the key id. | account | [alerts.api_key_bearer] |
api_key_wrong_secret | high | A restricted or full API key gets limit wrong secrets. subject: the key id. | account | [alerts.api_key_wrong_secret] |
license_expired | high | The license expired, was revoked, or is overdue for its online refresh. | system | — |
license_instance_limit | high | A replica went over the license's instance limit. | system | — |
license_refresh_failing | medium | The online license refresh has been failing for 24 hours. | system | — |
license_clock_rollback | medium | The system clock is behind the license's stored time floor. | system | — |
release_critical | high | A release newer than the running version is flagged [Critical], [Security], or UPDATE RECOMMENDED. subject: the highest such version. | system | [alerts.release_critical] |
release_outdated | medium | Releases newer than the running version have been out longer than max_age. subject: the running version. | system | [alerts.release_outdated] |
The license_* rules have no config section of their own; only [alerts] enabled = false stops them. See Alerts and metrics in the license guide.
How the rules count
- Failed logins are every sign-in step the audit log records as
failure: password, LDAP, OTP, passkey, and OAuth, whatever the error code. Failed step-up checks onPOST /v2/me/verifycount too. They count whether or not audit logs are enabled or licensed. Requests rejected by the rate limit (429) or with a malformed body don't count. The counters are fixed windows in the KV store, shared by all replicas, and each raises once per window, when it reacheslimit. - Failed verifications are
POST /v2/verifycalls that returnverified: falseor400 INVALID_PAYLOAD, for every API key, whatever its request logging. The baseline is the failures per minute over thebaselinehours before the window, counting only hours with verifications, so quiet nights don't make normal daytime traffic look like a spike. Without a baseline yet (a new account or deployment),min_failuresalone decides. cpu,memory, andserver_errorsare checked every minute by each replica, on its own metrics. A minute without samples breaks the streak. CPU is relative to the cores the process may use (the cgroup quota on Linux), sothreshold = 90on a 2-core pod means 180% of one core.root_login_countryneeds an IP resolver ([[ip_resolver.providers]]); with the rule on and no provider, startup fails. Countries are learned per root user and never expire. Each root user's first country, including the first sign-in after you turn the rule on, is learned silently. Private IPs and IPs no provider locates are skipped. It's off by default because IP geolocation is approximate: VPNs, mobile carriers, and corporate egress cause false alerts, and an attacker with a VPN in the victim's country avoids it.download_failurescounts failed checks and downloads in a row per file and replica. A completed download, an up-to-date check, or a restart resets it.kv_unavailableanddb_unavailable: each replica probes the KV store and the database every 10 seconds, with a 5-second timeout. While the database is down, the alert can't be stored, but it still reaches the log and webhooks (not web push). Recovery raises no alert; the replica logsstorage reachable again.api_key_*rules watchrestrictedandfullkeys only;publickeys ship in widgets.api_key_denied,api_key_browser, andapi_key_bearerraise at most once a minute per key and replica, and theirdetailscount the requests since the last one. Rules inmonitormode don't raiseapi_key_denied. Trying a server key in the API reference from a browser raisesapi_key_browser. See API keys & access levels.release_*rules are checked after every fetch of the release feed ([releases], every6hby default, and at startup). An unchanged subject is raised at most once a day. With[releases] enabled = false, there are no checks.
Configure the rules
Each rule has its own [alerts.<rule>] table with an enabled switch and the thresholds the rule table names. Defaults and ranges are in the Configuration reference. For example, to alert on fewer failed logins, turn on new-country alerts for root, and silence the CPU rule:
[alerts]
cooldown = "30m" # notify about a persisting condition every 30 minutes
[alerts.brute_force]
limit = "5/15m" # failures per client IP and window
[alerts.root_login_country]
enabled = true # needs an [[ip_resolver.providers]] entry
[alerts.cpu]
enabled = falseInvalid values fail startup.
IP anonymization
Alerts that show client IPs (brute_force, user_brute_force, root_login_country, and the api_key_* rules) anonymize them like request logs (203.0.113.7 becomes 203.0.113.0) when [alerts] anonymize_ip is true, in the stored alert and in every delivery. Unset, it follows [privacy] anonymize_ip, which is off by default. See Privacy & data handling.
brute_force still counts failures per full IP. Anonymized, the IPs of one /24 (IPv4) or /112 (IPv6) share a subject, so their alerts in one cooldown window count onto one row.
Webhooks
Each [[alerts.webhooks]] table adds an endpoint that every new alert is POSTed to. Add as many as you need, each with its own filter:
[[alerts.webhooks]]
url = "https://alerts.example.com/sentinel"
secret = "${file:/run/secrets/alerts_webhook_secret}" # signs each request; at least 16 characters
min_severity = "medium" # low | medium | high
[[alerts.webhooks]]
url = "${SLACK_WEBHOOK_URL}" # the URL holds a token: keep it out of the file
format = "slack"
rules = ["brute_force", "user_brute_force", "root_login_country"]Payload
With format = "json" (the default), the body is the alert as the search returns it, without lastTime and occurrences, plus replica, the id of the replica that raised it:
{
"accountId": null,
"details": { "failures": 3, "ip": "203.0.113.7", "window": "10m" },
"id": "alr_…",
"message": "3 failed logins from 203.0.113.7 within 10m",
"replica": "rep_…",
"rule": "brute_force",
"severity": "high",
"subject": "203.0.113.7",
"time": "2026-11-19T09:25:45.647000+00:00"
}With format = "slack", the body is a single line of text, which Slack incoming webhooks and compatible services (Mattermost, Rocket.Chat, Discord's /slack webhook URL) accept:
{"text":"*[HIGH] Brute-force login*: 3 failed logins from 203.0.113.7 within 10m"}Both are sent with Content-Type: application/json and User-Agent: ALTCHA-Sentinel/<version>, plus any headers you configure.
Verify the signature
With a secret, every request carries two headers:
| Header | Value |
|---|---|
X-Sentinel-Timestamp | Unix time in seconds when the request was sent |
X-Sentinel-Signature | sha256=<hex>: the HMAC-SHA256, keyed with the secret, of <timestamp>.<raw body> |
Compute the HMAC over the raw request body, before any JSON parsing, and reject old timestamps to stop replays. A Node.js handler:
import { createHmac, timingSafeEqual } from 'node:crypto';
import express from 'express';
const secret = process.env.ALERTS_WEBHOOK_SECRET!;
function verifySentinelSignature(rawBody: string, timestamp?: string, signature?: string): boolean {
if (!timestamp || !signature) return false;
// Reject requests older (or newer) than 5 minutes.
if (Math.abs(Date.now() / 1000 - Number(timestamp)) > 300) return false;
const expected = 'sha256=' + createHmac('sha256', secret).update(`${timestamp}.${rawBody}`).digest('hex');
const a = Buffer.from(expected);
const b = Buffer.from(signature);
return a.length === b.length && timingSafeEqual(a, b);
}
const app = express();
app.post('/sentinel', express.text({ type: 'application/json' }), (req, res) => {
if (!verifySentinelSignature(req.body, req.get('X-Sentinel-Timestamp'), req.get('X-Sentinel-Signature'))) {
return res.sendStatus(401);
}
const alert = JSON.parse(req.body);
console.log(alert.severity, alert.rule, alert.message);
res.sendStatus(204);
});Retries
A request that fails to connect or gets a 429 or 5xx is retried twice, after 2 and 10 seconds, with a 10-second timeout per attempt. Any other response is final. Failures are logged with the webhook's origin only, never its URL or headers, and the alert stays stored. The last outcome of each webhook shows in the alerts.delivery check of Doctor.
[alerts.email] sends every new alert as one plain-text message to all to recipients. It's on when host is set:
[alerts.email]
host = "smtp.example.com"
tls = "starttls" # starttls | implicit | none (local relays only)
username = "alerts@example.com"
password = "${file:/run/secrets/smtp_password}"
from = "Sentinel <alerts@example.com>"
to = ["ops@example.com", "security@example.com"]
min_severity = "high"- The subject is
[Sentinel] [HIGH] Brute-force login: <message>. The body is the message, followed by the webhook JSON. - At most
rateemails (default20/1h) are sent per window, counted per SMTP server across replicas. Alerts beyond it are still stored and sent to webhooks and push. - A send that fails to connect, times out, or gets a 4xx reply is retried twice, after 2 and 10 seconds; a 5xx reply is final.
- At startup, each replica checks that the SMTP server answers and logs the result. Startup doesn't fail on it.
SMTP connections never use the outbound HTTP proxy.
Web push
Users can receive alerts as browser notifications, even when the admin UI isn't open. Turn it on by setting [alerts.push] subject, a contact URL that push services can use to reach you:
[alerts.push]
subject = "mailto:security@example.com"Notifications are signed with the VAPID key [secrets] vapid_private_key. With embedded SQLite, Sentinel generates it when unset; with any other database, startup fails without it. Changing the key invalidates every subscription, so users must turn push notifications on again. See Server secrets.
Each user then turns notifications on per browser in the admin UI, under the user menu → Push notifications (also linked from the Alerts page), and can send a test notification there. A user can have up to 20 subscribed devices. Each subscription receives at most [alerts.push] rate notifications per window (default 20/1h); alerts beyond it are still stored and sent to webhooks. high alerts are sent with Urgency: high, others with normal.
[alerts.push] endpoint_hosts limits the push services a browser may register, so users can't make Sentinel POST to arbitrary hosts. The default allows the services of Chrome, Edge, Firefox, and Safari. The endpoints are under /v2/me/push-subscriptions in the API reference.
Who receives what
| Alert | Recipients |
|---|---|
System-wide (accountId: null) | Active root users |
| An account's | Active root users, and users whose role in that account grants the alerts feature (the built-in member and readonly roles do) |
The same rules decide who can search an alert. Recipients are checked when each alert is sent, so a role change or a deactivated user takes effect immediately. See Accounts, users & roles.
Webhooks and email have no recipient filter beyond rules and min_severity: they receive every alert, of every account.
Search alerts
POST /v2/alerts/_search accepts an admin session, or a full API key whose role grants alerts (see API keys & access levels):
curl -X POST https://sentinel.example.com/v2/alerts/_search \
-H 'Authorization: Bearer <jwt>' \
-H 'Content-Type: application/json' \
-d '{ "filter": { "rule": "brute_force" }, "sortDir": "desc", "limit": 10 }'{
"data": [
{
"id": "alr_…",
"time": "2026-11-19T09:25:45.647000+00:00",
"lastTime": "2026-11-19T09:25:45.647000+00:00",
"rule": "brute_force",
"severity": "high",
"accountId": null,
"subject": "203.0.113.7",
"message": "3 failed logins from 203.0.113.7 within 10m",
"details": { "failures": 3, "ip": "203.0.113.7", "window": "10m" },
"occurrences": 1
}
],
"pagination": { "limit": 10, "offset": 0, "total": 1 }
}time is when the alert was first raised in its window, and lastTime the latest occurrence. details depends on the rule:
| Rule | details |
|---|---|
brute_force | ip, failures, window |
user_brute_force | username, failures, window, lastIp |
verification_failures | failures, verifications (in the window), window, failuresPerMinute, baselinePerMinute, baselineHours |
cpu | replica, host, cores, averagePercent, threshold, duration |
memory | replica, host, bytes, limitBytes, percent, threshold, duration |
server_errors | replica, host, requests, errors5xx, percent, threshold, duration |
root_login_country | username, userId, country, countryName, ip, asname, proxy, tor, hosting, providerErrors, knownCountries |
download_failures | file, failures, threshold, error, downloadedAt, threatSourceId, replica, host (never the file's URL) |
kv_unavailable, db_unavailable | replica, host, since, downSeconds, error, duration |
api_key_networks | apiKeyId, apiKeyName, networks, window, network, ipv4Prefix, ipv6Prefix |
api_key_denied | apiKeyId, apiKeyName, securityGroupId, rule (index), ruleName, ip, method, path, denials |
api_key_browser | apiKeyId, apiKeyName, origin, ip, path, requests |
api_key_bearer | apiKeyId, apiKeyName, via (bearer or query), origin, ip, attempts |
api_key_wrong_secret | apiKeyId, apiKeyName, failures, window, lastIp |
release_critical | version, title, link, publishedAt, runningVersion, versions |
release_outdated | runningVersion, latestVersion, latestLink, newerSince, days, maxAge |
filter | Description |
|---|---|
accountId | One account's alerts. Absent: every alert; null: system-wide alerts only. Both are root only: a non-root caller without accountId gets 400 VALIDATION_ERROR, and with null gets 403 ROOT_PRIVILEGES_REQUIRED. |
rule | One of the rules; any other value is 400 VALIDATION_ERROR. |
severity | low, medium, or high. |
from, to | RFC 3339; first raised at or after from, before to. |
sortBy is time (default) or lastTime, and sortDir defaults to asc. See POST /v2/alerts/_search in the API reference.
Alerts have no acknowledged or resolved state. A condition that persists raises the same alert again in the next cooldown window, and old alerts expire after [alerts] retention.