# Bots & AI agents

> **Sentinel v2 (beta):** This page documents Sentinel v2, a beta rewrite that may change before the stable release. It does not apply to Sentinel v1, the current stable version: see the [Sentinel v1 documentation](/docs/sentinel.md).

Sentinel identifies bots, crawlers, and AI agents, including agents that sign their requests with Web Bot Auth, so you can allow or block them with rules. It recognizes them by two independent signals: a verified request signature, and the client IP's presence on crawler and bot lists.

## How it works

### Web Bot Auth

AI agents and crawlers can sign their HTTP requests with [Web Bot Auth](https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/): HTTP Message Signatures ([RFC 9421](https://www.rfc-editor.org/rfc/rfc9421)) in the `Signature-Input` and `Signature` headers, optionally with `Signature-Agent`. Sentinel checks every API-key request that has a `Signature-Input` header against a curated list of trusted agent keys, the ALTCHA directory list `web-bot-auth`, refreshed hourly. Each listed key belongs to a named agent, such as `chatgpt-agent`, `google-agent` or `exa-crawler`, with a category that Sentinel maps to a kind:

| Kind | Agents that |
| --- | --- |
| `ai_training` | Collect content for AI model training. |
| `ai_search` | Index content for AI search and assistants. |
| `user_fetch` | Fetch a page on a user's request, such as an AI assistant operating a browser. |
| `crawler` | Anything else: search engines, SEO tools, link previews, archives, browser automation services. |

A request has one of three results:

| Request | Result |
| --- | --- |
| Signed by a listed key, valid now | **Verified**: the agent's name and kind are known. |
| Signed, but malformed, expired, by an unlisted key, or with a bad signature | **Unverified** |
| No `Signature-Input` header | Not checked |

A signature verifies only if all of these hold:

- The key is an Ed25519 key on the trusted list. Sentinel never fetches the key directory that `Signature-Agent` names.
- The signature has the `web-bot-auth` tag and the `created`, `expires` and `keyid` parameters. `expires` is in the future, and `created` is at most 60 seconds ahead of Sentinel's clock.
- It covers `@authority` (or `@target-uri`), plus `Signature-Agent` when the request has one, and the authority is Sentinel's own host: the listener's `[server] forwarded_host_header` when set and sent by a trusted proxy, otherwise `Host`. `@scheme` and `@target-uri` are always taken as `https`.

If your reverse proxy rewrites `Host`, set `forwarded_host_header` (for example `X-Forwarded-Host`), or every signature fails as unverified. See [Host and allowed hosts](/docs/sentinel/v2/configure/listeners/#host-and-allowed-hosts).

### Bot and crawler lists

Agents that don't sign their requests are recognized by their IP address, through [threat intelligence](/docs/sentinel/v2/features/threat-intelligence/). The default threat sources label IPs with these kinds:

| Source | Kind |
| --- | --- |
| `crawlers_ai_training` | `ai_training` |
| `crawlers_ai_search` | `ai_search` |
| `crawlers_user_fetch` | `user_fetch` |
| `crawlers_other` | `crawler` |
| `botscout_30d`, `blocklist_de_bots` | `bot` |

These sources only label IPs; none of them blocks on its own. An IP can be on more than one list: Google's crawler addresses, for example, are shared by Googlebot and Google's AI search crawler, so they're on both `crawlers_ai_search` and `crawlers_other`.

### Device `BOT` and IP `BOT`

The classifier has two rules named `BOT`, one per signal:

| | Device `BOT` | IP `BOT` |
| --- | --- | --- |
| Triggers when | The request is signed by a verified Web Bot Auth agent | A threat match has kind `bot`, `crawler`, `ai_training`, `ai_search` or `user_fetch` |
| Category | `device` | `ip` |
| Scored in | `POST /v2/verify`, `GET /v2/inspect` | `POST /v2/verify`, `POST /v2/classifier`, `GET /v2/inspect` |
| Can be faked | No: needs a listed private key | Not the IP itself, but a shared address carries the labels of everyone behind it |

The device rule only sees the signature of the request it scores. `POST /v2/classifier` classifies headers you pass in the body, so its device `BOT` never triggers. Scores are in [Classifier rules & scores](/docs/sentinel/v2/reference/classifier-rules/#device).

## Enabling it

Web Bot Auth is on by default:

```toml
[web_bot_auth]
enabled = true      # false: the list isn't downloaded and no request is checked
schedule = "@hourly"
```

`url` replaces the default list with your own CSV with the columns `name`, `keyid`, `category` and `jwk`. See [`[web_bot_auth]`](/docs/sentinel/v2/reference/configuration/#web_bot_auth). The crawler lists come with the default threat sources; see [Default threat sources](/docs/sentinel/v2/configure/remote-files/#default-threat-sources).

Detection has no effect until a security-group rule uses it. Two [presets](/docs/sentinel/v2/policies/presets/) cover the common cases:

- **[Verified agents](/docs/sentinel/v2/policies/presets/#verified-agents)** accepts verified agents that aren't AI training crawlers or user-driven fetchers, so verified search crawlers skip escalation and rate limits.
- **[Block AI agents](/docs/sentinel/v2/policies/presets/#block-ai-agents)** denies verified `ai_training` and `user_fetch` agents, any request whose signature doesn't verify, and IPs on an `ai_training` list.

Together, as the first rules of a group:

```json
[
{
"name": "verified-agents",
"action": "accept",
"conditions": [
  { "field": "agent", "operator": "=", "value": ["verified"] },
  { "field": "agent", "operator": "!=", "value": ["ai_training", "user_fetch"] }
]
},
{
"name": "block-ai",
"action": "deny",
"message": "Agent denied.",
"conditions": [{ "field": "agent", "operator": "=", "value": ["unverified", "ai_training", "user_fetch"] }]
},
{
"name": "block-ai-lists",
"action": "deny",
"message": "Agent denied.",
"conditions": [{ "field": "threat", "operator": "=", "value": ["ai_training"] }]
}
]
```

Most agents on the Web Bot Auth list today are `user_fetch` agents, such as assistants that open a page in a browser for their user. Denying `user_fetch` stops them from using your forms on a user's behalf; leave it out of the deny rule if you want to allow that.

## Rules & signals

### Rule conditions

| Condition | Matches | Values |
| --- | --- | --- |
| `agent` | The request's Web Bot Auth result. A verified agent matches `verified`, its name, and its kind; an unverified signature matches `unverified`; an unsigned request matches nothing. | `verified`, `unverified`, an agent name, `crawler`, `ai_training`, `ai_search`, `user_fetch`. Wildcards allowed, so `["*"]` matches every signed request. |
| `threat` | The client IP's threat matches, by source name or kind. | `bot`, `crawler`, `ai_training`, `ai_search`, `user_fetch`, a source name such as `crawlers_other`, … |
| `signal` | Classifier rules that trigger for the request. `BOT` holds when either the device or the IP `BOT` rule triggers. | `["BOT"]` |

An unsigned request matches no `agent` value, so `!=` conditions hold for it: `agent` `!=` `["verified"]` matches unsigned requests too. See [Condition fields](/docs/sentinel/v2/reference/rules/#condition-fields).

### `BOT` in the Recommended preset

The **Recommended** preset escalates requests with the `BOT` signal to a code challenge and a harder proof-of-work. That includes verified agents and IPs on crawler lists. To let verified search crawlers through without a challenge, add **Verified agents** before it. Crawlers that don't sign keep getting the challenge.

## Using the API

Test your rules with `GET /v2/inspect` and the [`securityRules` trace](/docs/sentinel/v2/policies/security-rules/#inspect-with-a-rule-trace). A request with a signature that Sentinel can't verify:

```bash
curl 'https://sentinel.example.com/v2/inspect?apiKey=key_…' \
-H 'Signature-Input: sig1=("@authority");created=1763520000;expires=1763520060;keyid="x";tag="web-bot-auth"' \
-H 'Signature: sig1=:AAAA:'
```

```json
{
"code": "ACCESS_DENIED",
"error": "Agent denied.",
"statusCode": 403,
"data": {
"device": {
  "rules": { "BOT": { "score": 0.0 }, … },
  …
},
"securityRules": [
  {
    "index": 0,
    "name": "verified-agents",
    "matched": false,
    "outcome": "continue",
    "conditions": [
      { "field": "agent", "matched": false, "signals": null },
      { "field": "agent", "matched": true, "signals": null }
    ],
    …
  },
  {
    "index": 1,
    "name": "block-ai",
    "matched": true,
    "outcome": "deny",
    "conditions": [{ "field": "agent", "matched": true, "signals": null }],
    …
  },
  …
],
…
}
}
```

The signature is `unverified`, so `block-ai` denies the request and device `BOT` stays at `0`. A verified agent would show `"outcome": "allow"` on `verified-agents` and device `BOT` above `0`. See the [API reference](/sentinel-api-docs-v2.html#tag/inspect/GET/v2/inspect).

To see the IP signal, classify an IP from a crawler list with `POST /v2/classifier` and a `restricted` key:

```bash
curl -X POST https://sentinel.example.com/v2/classifier \
-u 'key_…:sec_…' \
-H 'Content-Type: application/json' \
-d '{ "ip": "20.171.206.5" }'
```

```json
{
"data": {
"ip": {
  "threats": [
    { "source": "crawlers_ai_training", "kind": "ai_training", "action": null, "threatScore": null, "match": "20.171.206.0/24" }
  ],
  "action": null,
  "triggeredRules": ["BOT"],
  …
},
…
}
}
```

### Request logs

For API keys with request logging, each logged request records the Web Bot Auth result in `context.agent`: the verified agent's name, or `unverified`. Unsigned requests have no `agent`. Sentinel records it whether or not the group has an `agent` rule:

```json
{
"path": "/v2/challenge",
"statusCode": 403,
"error": "ACCESS_DENIED",
"context": { "agent": "unverified", "rules": "block-ai", "threatAction": "block", "threats": "firehol_level1" },
…
}
```

Review these requests under **Logs** in the admin UI. See [Request logs & analytics](/docs/sentinel/v2/operations/request-logs/).

## Tuning

- **Start in monitor mode.** Add `"monitor": true` to a new deny rule and watch its activity before blocking agents your users rely on.
- **Pick kinds, not names.** Agent names change as operators add keys; kinds stay stable. Use a name only to make an exception for one agent.
- **Exempt carefully.** `accept` skips every later rule, rate limits included. A verified agent that misbehaves is then limited only by the global `[rate_limit]`.

> **Warning:**
>
> A `deny` rule on `unverified` also blocks well-meaning clients whose signatures fail, for example because a proxy rewrites `Host` without `forwarded_host_header`. Check `context.agent` in the request logs before you turn it on.

## Limitations

- Only agents that sign with a key on the trusted list are verified. Other signed requests are `unverified`, even if their signature is valid for a key in their own directory.
- Only Ed25519 keys are supported.
- Sentinel keeps no nonce store. A captured signature can be replayed until its `expires`, but only against Sentinel's own host.
- Agents that don't sign are recognized only by their IP, and only while the crawler and bot lists include it. An agent that hides behind residential proxies and a browser User-Agent isn't detected as a bot. Other signals, such as [Human Interaction Signature](/docs/sentinel/v2/features/human-interaction-signature/) and the device rules, still apply.

## Related

- [Threat intelligence](/docs/sentinel/v2/features/threat-intelligence/)
- [Presets: Verified agents and Block AI agents](/docs/sentinel/v2/policies/presets/#add-ons)
- [Writing security rules: Exempt verified agents](/docs/sentinel/v2/policies/security-rules/#exempt-verified-agents)
- [Rules reference: condition fields](/docs/sentinel/v2/reference/rules/#condition-fields)
- [Classifier rules & scores](/docs/sentinel/v2/reference/classifier-rules/#device)
- [`[web_bot_auth]`](/docs/sentinel/v2/reference/configuration/#web_bot_auth) in the Configuration reference
