# Alerts

> **Sentinel v2 (beta):** This page documents Sentinel v2, a beta rewrite that may change before the stable release. It does not apply to Sentinel v1, the current stable version: see the [Sentinel v1 documentation](/docs/sentinel.md).

Sentinel watches logins, verifications, API key use, its own replicas, remote files, storage reachability, the license, and the release feed, and raises an **alert** when something needs attention. Use this page to tune the rules and to send alerts to your chat, inbox, incident tool, or browser.

<Callout type="note" title="Changed in v2">

The alert rules, the search filters, and the response fields changed, and delivery by webhook, email, and web push is new. See [Breaking changes](/docs/sentinel/v2/upgrade/breaking-changes/#alerts).

## How it works

Every alert is:

- **stored** in the database and searchable with `POST /v2/alerts/_search` and under **Alerts** in the admin UI;
- **logged** as a warning with the target `sentinel::alerts`;
- **delivered** to every configured webhook, to the `[alerts.email]` recipients, and to the browsers of users who turned on push notifications.

```text
WARN sentinel::alerts: alert: 3 failed logins from 127.0.0.1 within 10m rule="brute_force" severity="high" subject="127.0.0.1"
```

### Scope

An alert either belongs to one account (`accountId` is the account's id) or is system-wide (`accountId` is `null`). System-wide alerts concern the whole deployment, such as login attacks, replicas, storage, the license, and releases, and only root users see them. See [Who receives what](#who-receives-what).

### Deduplication and cooldown

An alert is one row per **rule, account, subject, and cooldown window**. The `subject` names what the alert is about: a client IP, a username, an API key id, a replica's host name, and so on. Windows are [`[alerts] cooldown`](/docs/sentinel/v2/reference/configuration/#alerts) long (default `1h`) and aligned to the Unix epoch.

- The first occurrence in a window creates the alert and delivers it.
- Later occurrences in the same window increase its `occurrences` and update `lastTime`, `message`, and `details`, without delivering it again.

A condition that persists, such as high memory use, therefore notifies about once per cooldown. Replicas share the alerts table, so an alert that several replicas raise at once notifies once.

Alerts last raised longer ago than `[alerts] retention` (default `90d`) are deleted hourly, by one replica at a time. With `[alerts] enabled = false`, no new alert is raised; search and retention keep working.

## Rules

| Rule | Severity | Raised when | Scope | Config |
| --- | --- | --- | --- | --- |
| `brute_force` | `high` | One client IP fails `limit` logins. A wrong current password on `POST /v2/me/password` counts as a failed login. `subject`: the IP. | system | [`[alerts.brute_force]`](/docs/sentinel/v2/reference/configuration/#alertsbrute_force) |
| `user_brute_force` | `high` | One username fails `limit` logins from any IPs: distributed guessing that stays under the per-IP limit. `subject`: the username, trimmed and lowercased. | system | [`[alerts.user_brute_force]`](/docs/sentinel/v2/reference/configuration/#alertsuser_brute_force) |
| `verification_failures` | `medium` | An account's failed `POST /v2/verify` calls per minute over `window` reach `factor` times its baseline, with at least `min_failures` failures. `subject`: the account id. | account | [`[alerts.verification_failures]`](/docs/sentinel/v2/reference/configuration/#alertsverification_failures) |
| `cpu` | `medium` | A replica's CPU use stays at or above `threshold` percent of its available cores every minute of `duration`. `subject`: the replica's host name. | system | [`[alerts.cpu]`](/docs/sentinel/v2/reference/configuration/#alertscpu) |
| `memory` | `high` | A replica's memory (RSS) stays at or above `threshold` percent of its limit (the cgroup limit, else host RAM) every minute of `duration`. | system | [`[alerts.memory]`](/docs/sentinel/v2/reference/configuration/#alertsmemory) |
| `server_errors` | `high` | At least `threshold` percent of a replica's responses over `duration` are 5xx, out of at least `min_requests`. | system | [`[alerts.server_errors]`](/docs/sentinel/v2/reference/configuration/#alertsserver_errors) |
| `root_login_country` | `high` | **Off by default.** A root user signs in from a country they never signed in from before. `subject`: `<username>:<country>`. | system | [`[alerts.root_login_country]`](/docs/sentinel/v2/reference/configuration/#alertsroot_login_country) |
| `download_failures` | `medium` | A [remote file](/docs/sentinel/v2/configure/remote-files/) fails its check or download `failures` times in a row on a replica. `subject`: the file name (`threat-<id>` for threat sources). | system | [`[alerts.download_failures]`](/docs/sentinel/v2/reference/configuration/#alertsdownload_failures) |
| `kv_unavailable` | `high` | A replica can't reach the KV store for `duration`. `subject`: the replica's host name. | system | [`[alerts.kv_unavailable]`](/docs/sentinel/v2/reference/configuration/#alertskv_unavailable) |
| `db_unavailable` | `high` | A replica can't reach the database for `duration`. `subject`: the replica's host name. | system | [`[alerts.db_unavailable]`](/docs/sentinel/v2/reference/configuration/#alertsdb_unavailable) |
| `api_key_networks` | `high` | A `restricted` or `full` API key is used from `limit` distinct networks within the window. `subject`: the key id. | account | [`[alerts.api_key_networks]`](/docs/sentinel/v2/reference/configuration/#alertsapi_key_networks) |
| `api_key_denied` | `high` | A `restricted` or `full` API key with a valid secret is denied by its security group (`403 ACCESS_DENIED`). Whoever sent it holds the secret. `subject`: the key id. | account | [`[alerts.api_key_denied]`](/docs/sentinel/v2/reference/configuration/#alertsapi_key_denied) |
| `api_key_browser` | `high` | A `restricted` or `full` API key is sent with an `Origin` or `Referer` header, that is, from a browser page. `subject`: the key id. | account | [`[alerts.api_key_browser]`](/docs/sentinel/v2/reference/configuration/#alertsapi_key_browser) |
| `api_key_bearer` | `medium` | A `restricted` or `full` API key id is sent without its secret (`403 BEARER_AUTH_PUBLIC_API_KEY`). `subject`: the key id. | account | [`[alerts.api_key_bearer]`](/docs/sentinel/v2/reference/configuration/#alertsapi_key_bearer) |
| `api_key_wrong_secret` | `high` | A `restricted` or `full` API key gets `limit` wrong secrets. `subject`: the key id. | account | [`[alerts.api_key_wrong_secret]`](/docs/sentinel/v2/reference/configuration/#alertsapi_key_wrong_secret) |
| `license_expired` | `high` | The license expired, was revoked, or is overdue for its online refresh. | system | — |
| `license_instance_limit` | `high` | A replica went over the license's instance limit. | system | — |
| `license_refresh_failing` | `medium` | The online license refresh has been failing for 24 hours. | system | — |
| `license_clock_rollback` | `medium` | The system clock is behind the license's stored time floor. | system | — |
| `release_critical` | `high` | A release newer than the running version is flagged `[Critical]`, `[Security]`, or `UPDATE RECOMMENDED`. `subject`: the highest such version. | system | [`[alerts.release_critical]`](/docs/sentinel/v2/reference/configuration/#alertsrelease_critical) |
| `release_outdated` | `medium` | Releases newer than the running version have been out longer than `max_age`. `subject`: the running version. | system | [`[alerts.release_outdated]`](/docs/sentinel/v2/reference/configuration/#alertsrelease_outdated) |

The `license_*` rules have no config section of their own; only `[alerts] enabled = false` stops them. See [Alerts and metrics](/docs/sentinel/v2/configure/licensing/#alerts-and-metrics) in the license guide.

### How the rules count

- **Failed logins** are every sign-in step the [audit log](/docs/sentinel/v2/operations/audit-logs/) records as `failure`: password, LDAP, OTP, passkey, and OAuth, whatever the error code. Failed step-up checks on `POST /v2/me/verify` count too. They count whether or not audit logs are enabled or licensed. Requests rejected by the rate limit (`429`) or with a malformed body don't count. The counters are fixed windows in the KV store, shared by all replicas, and each raises once per window, when it reaches `limit`.
- **Failed verifications** are `POST /v2/verify` calls that return `verified: false` or `400 INVALID_PAYLOAD`, for every API key, whatever its request logging. The baseline is the failures per minute over the `baseline` hours before the window, counting only hours with verifications, so quiet nights don't make normal daytime traffic look like a spike. Without a baseline yet (a new account or deployment), `min_failures` alone decides.
- **`cpu`, `memory`, and `server_errors`** are checked every minute by each replica, on its own metrics. A minute without samples breaks the streak. CPU is relative to the cores the process may use (the cgroup quota on Linux), so `threshold = 90` on a 2-core pod means 180% of one core.
- **`root_login_country`** needs an [IP resolver](/docs/sentinel/v2/configure/ip-resolvers/) (`[[ip_resolver.providers]]`); with the rule on and no provider, startup fails. Countries are learned per root user and never expire. Each root user's first country, including the first sign-in after you turn the rule on, is learned silently. Private IPs and IPs no provider locates are skipped. It's off by default because IP geolocation is approximate: VPNs, mobile carriers, and corporate egress cause false alerts, and an attacker with a VPN in the victim's country avoids it.
- **`download_failures`** counts failed checks and downloads in a row per file and replica. A completed download, an up-to-date check, or a restart resets it.
- **`kv_unavailable` and `db_unavailable`**: each replica probes the KV store and the database every 10 seconds, with a 5-second timeout. While the database is down, the alert can't be stored, but it still reaches the log and webhooks (not web push). Recovery raises no alert; the replica logs `storage reachable again`.
- **`api_key_*`** rules watch `restricted` and `full` keys only; `public` keys ship in widgets. `api_key_denied`, `api_key_browser`, and `api_key_bearer` raise at most once a minute per key and replica, and their `details` count the requests since the last one. Rules in `monitor` mode don't raise `api_key_denied`. Trying a server key in the API reference from a browser raises `api_key_browser`. See [API keys & access levels](/docs/sentinel/v2/concepts/api-keys/).
- **`release_*`** rules are checked after every fetch of the release feed (`[releases]`, every `6h` by default, and at startup). An unchanged subject is raised at most once a day. With `[releases] enabled = false`, there are no checks.

## Configure the rules

Each rule has its own `[alerts.<rule>]` table with an `enabled` switch and the thresholds the rule table names. Defaults and ranges are in the [Configuration reference](/docs/sentinel/v2/reference/configuration/#alerts). For example, to alert on fewer failed logins, turn on new-country alerts for root, and silence the CPU rule:

```toml
[alerts]
cooldown = "30m"            # notify about a persisting condition every 30 minutes

[alerts.brute_force]
limit = "5/15m"             # failures per client IP and window

[alerts.root_login_country]
enabled = true              # needs an [[ip_resolver.providers]] entry

[alerts.cpu]
enabled = false
```

Invalid values fail startup.

### IP anonymization

Alerts that show client IPs (`brute_force`, `user_brute_force`, `root_login_country`, and the `api_key_*` rules) anonymize them like request logs (`203.0.113.7` becomes `203.0.113.0`) when [`[alerts] anonymize_ip`](/docs/sentinel/v2/reference/configuration/#alerts) is `true`, in the stored alert and in every delivery. Unset, it follows `[privacy] anonymize_ip`, which is off by default. See [Privacy & data handling](/docs/sentinel/v2/security/privacy/).

`brute_force` still counts failures per full IP. Anonymized, the IPs of one `/24` (IPv4) or `/112` (IPv6) share a subject, so their alerts in one cooldown window count onto one row.

## Webhooks

Each [`[[alerts.webhooks]]`](/docs/sentinel/v2/reference/configuration/#alertswebhooks) table adds an endpoint that every new alert is POSTed to. Add as many as you need, each with its own filter:

```toml
[[alerts.webhooks]]
url          = "https://alerts.example.com/sentinel"
secret       = "${file:/run/secrets/alerts_webhook_secret}"   # signs each request; at least 16 characters
min_severity = "medium"                                       # low | medium | high

[[alerts.webhooks]]
url    = "${SLACK_WEBHOOK_URL}"   # the URL holds a token: keep it out of the file
format = "slack"
rules  = ["brute_force", "user_brute_force", "root_login_country"]
```

### Payload

With `format = "json"` (the default), the body is the alert as [the search](#search-alerts) returns it, without `lastTime` and `occurrences`, plus `replica`, the id of the replica that raised it:

```json
{
"accountId": null,
"details": { "failures": 3, "ip": "203.0.113.7", "window": "10m" },
"id": "alr_…",
"message": "3 failed logins from 203.0.113.7 within 10m",
"replica": "rep_…",
"rule": "brute_force",
"severity": "high",
"subject": "203.0.113.7",
"time": "2026-11-19T09:25:45.647000+00:00"
}
```

With `format = "slack"`, the body is a single line of text, which Slack incoming webhooks and compatible services (Mattermost, Rocket.Chat, Discord's `/slack` webhook URL) accept:

```json
{"text":"*[HIGH] Brute-force login*: 3 failed logins from 203.0.113.7 within 10m"}
```

Both are sent with `Content-Type: application/json` and `User-Agent: ALTCHA-Sentinel/<version>`, plus any `headers` you configure.

### Verify the signature

With a `secret`, every request carries two headers:

| Header | Value |
| --- | --- |
| `X-Sentinel-Timestamp` | Unix time in seconds when the request was sent |
| `X-Sentinel-Signature` | `sha256=<hex>`: the HMAC-SHA256, keyed with the secret, of `<timestamp>.<raw body>` |

Compute the HMAC over the raw request body, before any JSON parsing, and reject old timestamps to stop replays. A Node.js handler:

```ts
import { createHmac, timingSafeEqual } from 'node:crypto';
import express from 'express';

const secret = process.env.ALERTS_WEBHOOK_SECRET!;

function verifySentinelSignature(rawBody: string, timestamp?: string, signature?: string): boolean {
if (!timestamp || !signature) return false;
// Reject requests older (or newer) than 5 minutes.
if (Math.abs(Date.now() / 1000 - Number(timestamp)) > 300) return false;
const expected = 'sha256=' + createHmac('sha256', secret).update(`${timestamp}.${rawBody}`).digest('hex');
const a = Buffer.from(expected);
const b = Buffer.from(signature);
return a.length === b.length && timingSafeEqual(a, b);
}

const app = express();

app.post('/sentinel', express.text({ type: 'application/json' }), (req, res) => {
if (!verifySentinelSignature(req.body, req.get('X-Sentinel-Timestamp'), req.get('X-Sentinel-Signature'))) {
return res.sendStatus(401);
}
const alert = JSON.parse(req.body);
console.log(alert.severity, alert.rule, alert.message);
res.sendStatus(204);
});
```

### Retries

A request that fails to connect or gets a `429` or 5xx is retried twice, after 2 and 10 seconds, with a 10-second timeout per attempt. Any other response is final. Failures are logged with the webhook's origin only, never its URL or headers, and the alert stays stored. The last outcome of each webhook shows in the `alerts.delivery` check of [Doctor](/docs/sentinel/v2/operations/troubleshooting/).

## Email

[`[alerts.email]`](/docs/sentinel/v2/reference/configuration/#alertsemail) sends every new alert as one plain-text message to all `to` recipients. It's on when `host` is set:

```toml
[alerts.email]
host     = "smtp.example.com"
tls      = "starttls"                       # starttls | implicit | none (local relays only)
username = "alerts@example.com"
password = "${file:/run/secrets/smtp_password}"
from     = "Sentinel <alerts@example.com>"
to       = ["ops@example.com", "security@example.com"]
min_severity = "high"
```

- The subject is `[Sentinel] [HIGH] Brute-force login: <message>`. The body is the message, followed by the [webhook JSON](#payload).
- At most `rate` emails (default `20/1h`) are sent per window, counted per SMTP server across replicas. Alerts beyond it are still stored and sent to webhooks and push.
- A send that fails to connect, times out, or gets a 4xx reply is retried twice, after 2 and 10 seconds; a 5xx reply is final.
- At startup, each replica checks that the SMTP server answers and logs the result. Startup doesn't fail on it.

SMTP connections never use the outbound HTTP proxy.

## Web push

Users can receive alerts as browser notifications, even when the admin UI isn't open. Turn it on by setting [`[alerts.push] subject`](/docs/sentinel/v2/reference/configuration/#alertspush), a contact URL that push services can use to reach you:

```toml
[alerts.push]
subject = "mailto:security@example.com"
```

Notifications are signed with the VAPID key `[secrets] vapid_private_key`. With embedded SQLite, Sentinel generates it when unset; with any other database, startup fails without it. Changing the key invalidates every subscription, so users must turn push notifications on again. See [Server secrets](/docs/sentinel/v2/configure/secrets/).

Each user then turns notifications on per browser in the admin UI, under the user menu → **Push notifications** (also linked from the **Alerts** page), and can send a test notification there. A user can have up to 20 subscribed devices. Each subscription receives at most `[alerts.push] rate` notifications per window (default `20/1h`); alerts beyond it are still stored and sent to webhooks. `high` alerts are sent with `Urgency: high`, others with `normal`.

`[alerts.push] endpoint_hosts` limits the push services a browser may register, so users can't make Sentinel POST to arbitrary hosts. The default allows the services of Chrome, Edge, Firefox, and Safari. The endpoints are under `/v2/me/push-subscriptions` in the [API reference](/sentinel-api-docs-v2.html#tag/me/POST/v2/me/push-subscriptions).

### Who receives what

| Alert | Recipients |
| --- | --- |
| System-wide (`accountId: null`) | Active root users |
| An account's | Active root users, and users whose role in that account grants the `alerts` feature (the built-in `member` and `readonly` roles do) |

The same rules decide who can search an alert. Recipients are checked when each alert is sent, so a role change or a deactivated user takes effect immediately. See [Accounts, users & roles](/docs/sentinel/v2/concepts/accounts-users/#features-and-built-in-roles).

Webhooks and email have no recipient filter beyond `rules` and `min_severity`: they receive every alert, of every account.

## Search alerts

`POST /v2/alerts/_search` accepts an admin session, or a `full` API key whose role grants `alerts` (see [API keys & access levels](/docs/sentinel/v2/concepts/api-keys/)):

```bash
curl -X POST https://sentinel.example.com/v2/alerts/_search \
-H 'Authorization: Bearer <jwt>' \
-H 'Content-Type: application/json' \
-d '{ "filter": { "rule": "brute_force" }, "sortDir": "desc", "limit": 10 }'
```

```json
{
"data": [
{
  "id": "alr_…",
  "time": "2026-11-19T09:25:45.647000+00:00",
  "lastTime": "2026-11-19T09:25:45.647000+00:00",
  "rule": "brute_force",
  "severity": "high",
  "accountId": null,
  "subject": "203.0.113.7",
  "message": "3 failed logins from 203.0.113.7 within 10m",
  "details": { "failures": 3, "ip": "203.0.113.7", "window": "10m" },
  "occurrences": 1
}
],
"pagination": { "limit": 10, "offset": 0, "total": 1 }
}
```

`time` is when the alert was first raised in its window, and `lastTime` the latest occurrence. `details` depends on the rule:

| Rule | `details` |
| --- | --- |
| `brute_force` | `ip`, `failures`, `window` |
| `user_brute_force` | `username`, `failures`, `window`, `lastIp` |
| `verification_failures` | `failures`, `verifications` (in the window), `window`, `failuresPerMinute`, `baselinePerMinute`, `baselineHours` |
| `cpu` | `replica`, `host`, `cores`, `averagePercent`, `threshold`, `duration` |
| `memory` | `replica`, `host`, `bytes`, `limitBytes`, `percent`, `threshold`, `duration` |
| `server_errors` | `replica`, `host`, `requests`, `errors5xx`, `percent`, `threshold`, `duration` |
| `root_login_country` | `username`, `userId`, `country`, `countryName`, `ip`, `asname`, `proxy`, `tor`, `hosting`, `providerErrors`, `knownCountries` |
| `download_failures` | `file`, `failures`, `threshold`, `error`, `downloadedAt`, `threatSourceId`, `replica`, `host` (never the file's URL) |
| `kv_unavailable`, `db_unavailable` | `replica`, `host`, `since`, `downSeconds`, `error`, `duration` |
| `api_key_networks` | `apiKeyId`, `apiKeyName`, `networks`, `window`, `network`, `ipv4Prefix`, `ipv6Prefix` |
| `api_key_denied` | `apiKeyId`, `apiKeyName`, `securityGroupId`, `rule` (index), `ruleName`, `ip`, `method`, `path`, `denials` |
| `api_key_browser` | `apiKeyId`, `apiKeyName`, `origin`, `ip`, `path`, `requests` |
| `api_key_bearer` | `apiKeyId`, `apiKeyName`, `via` (`bearer` or `query`), `origin`, `ip`, `attempts` |
| `api_key_wrong_secret` | `apiKeyId`, `apiKeyName`, `failures`, `window`, `lastIp` |
| `release_critical` | `version`, `title`, `link`, `publishedAt`, `runningVersion`, `versions` |
| `release_outdated` | `runningVersion`, `latestVersion`, `latestLink`, `newerSince`, `days`, `maxAge` |

| `filter` | Description |
| --- | --- |
| `accountId` | One account's alerts. Absent: every alert; `null`: system-wide alerts only. Both are root only: a non-root caller without `accountId` gets `400 VALIDATION_ERROR`, and with `null` gets `403 ROOT_PRIVILEGES_REQUIRED`. |
| `rule` | One of the [rules](#rules); any other value is `400 VALIDATION_ERROR`. |
| `severity` | `low`, `medium`, or `high`. |
| `from`, `to` | RFC 3339; first raised at or after `from`, before `to`. |

`sortBy` is `time` (default) or `lastTime`, and `sortDir` defaults to `asc`. See [`POST /v2/alerts/_search`](/sentinel-api-docs-v2.html#tag/alerts/POST/v2/alerts/_search) in the API reference.

Alerts have no acknowledged or resolved state. A condition that persists raises the same alert again in the next cooldown window, and old alerts expire after `[alerts] retention`.

## Related

- [`[alerts]` in the Configuration reference](/docs/sentinel/v2/reference/configuration/#alerts)
- [Monitoring](/docs/sentinel/v2/operations/monitoring/)
- [Audit logs](/docs/sentinel/v2/operations/audit-logs/)
- [Configure the license](/docs/sentinel/v2/configure/licensing/)
- [Server secrets](/docs/sentinel/v2/configure/secrets/)
- [Privacy & data handling](/docs/sentinel/v2/security/privacy/)
