Bots & AI agents
Sentinel identifies bots, crawlers, and AI agents, including agents that sign their requests with Web Bot Auth, so you can allow or block them with rules. It recognizes them by two independent signals: a verified request signature, and the client IP's presence on crawler and bot lists.
How it works
Web Bot Auth
AI agents and crawlers can sign their HTTP requests with Web Bot Auth: HTTP Message Signatures (RFC 9421) in the Signature-Input and Signature headers, optionally with Signature-Agent. Sentinel checks every API-key request that has a Signature-Input header against a curated list of trusted agent keys, the ALTCHA directory list web-bot-auth, refreshed hourly. Each listed key belongs to a named agent, such as chatgpt-agent, google-agent or exa-crawler, with a category that Sentinel maps to a kind:
| Kind | Agents that |
|---|---|
ai_training | Collect content for AI model training. |
ai_search | Index content for AI search and assistants. |
user_fetch | Fetch a page on a user's request, such as an AI assistant operating a browser. |
crawler | Anything else: search engines, SEO tools, link previews, archives, browser automation services. |
A request has one of three results:
| Request | Result |
|---|---|
| Signed by a listed key, valid now | Verified: the agent's name and kind are known. |
| Signed, but malformed, expired, by an unlisted key, or with a bad signature | Unverified |
No Signature-Input header | Not checked |
A signature verifies only if all of these hold:
- The key is an Ed25519 key on the trusted list. Sentinel never fetches the key directory that
Signature-Agentnames. - The signature has the
web-bot-authtag and thecreated,expiresandkeyidparameters.expiresis in the future, andcreatedis at most 60 seconds ahead of Sentinel's clock. - It covers
@authority(or@target-uri), plusSignature-Agentwhen the request has one, and the authority is Sentinel's own host: the listener's[server] forwarded_host_headerwhen set and sent by a trusted proxy, otherwiseHost.@schemeand@target-uriare always taken ashttps.
If your reverse proxy rewrites Host, set forwarded_host_header (for example X-Forwarded-Host), or every signature fails as unverified. See Host and allowed hosts.
Bot and crawler lists
Agents that don't sign their requests are recognized by their IP address, through threat intelligence. The default threat sources label IPs with these kinds:
| Source | Kind |
|---|---|
crawlers_ai_training | ai_training |
crawlers_ai_search | ai_search |
crawlers_user_fetch | user_fetch |
crawlers_other | crawler |
botscout_30d, blocklist_de_bots | bot |
These sources only label IPs; none of them blocks on its own. An IP can be on more than one list: Google's crawler addresses, for example, are shared by Googlebot and Google's AI search crawler, so they're on both crawlers_ai_search and crawlers_other.
Device BOT and IP BOT
The classifier has two rules named BOT, one per signal:
Device BOT | IP BOT | |
|---|---|---|
| Triggers when | The request is signed by a verified Web Bot Auth agent | A threat match has kind bot, crawler, ai_training, ai_search or user_fetch |
| Category | device | ip |
| Scored in | POST /v2/verify, GET /v2/inspect | POST /v2/verify, POST /v2/classifier, GET /v2/inspect |
| Can be faked | No: needs a listed private key | Not the IP itself, but a shared address carries the labels of everyone behind it |
The device rule only sees the signature of the request it scores. POST /v2/classifier classifies headers you pass in the body, so its device BOT never triggers. Scores are in Classifier rules & scores.
Enabling it
Web Bot Auth is on by default:
[web_bot_auth]
enabled = true # false: the list isn't downloaded and no request is checked
schedule = "@hourly"url replaces the default list with your own CSV with the columns name, keyid, category and jwk. See [web_bot_auth]. The crawler lists come with the default threat sources; see Default threat sources.
Detection has no effect until a security-group rule uses it. Two presets cover the common cases:
- Verified agents accepts verified agents that aren't AI training crawlers or user-driven fetchers, so verified search crawlers skip escalation and rate limits.
- Block AI agents denies verified
ai_traininganduser_fetchagents, any request whose signature doesn't verify, and IPs on anai_traininglist.
Together, as the first rules of a group:
[
{
"name": "verified-agents",
"action": "accept",
"conditions": [
{ "field": "agent", "operator": "=", "value": ["verified"] },
{ "field": "agent", "operator": "!=", "value": ["ai_training", "user_fetch"] }
]
},
{
"name": "block-ai",
"action": "deny",
"message": "Agent denied.",
"conditions": [{ "field": "agent", "operator": "=", "value": ["unverified", "ai_training", "user_fetch"] }]
},
{
"name": "block-ai-lists",
"action": "deny",
"message": "Agent denied.",
"conditions": [{ "field": "threat", "operator": "=", "value": ["ai_training"] }]
}
]Most agents on the Web Bot Auth list today are user_fetch agents, such as assistants that open a page in a browser for their user. Denying user_fetch stops them from using your forms on a user's behalf; leave it out of the deny rule if you want to allow that.
Rules & signals
Rule conditions
| Condition | Matches | Values |
|---|---|---|
agent | The request's Web Bot Auth result. A verified agent matches verified, its name, and its kind; an unverified signature matches unverified; an unsigned request matches nothing. | verified, unverified, an agent name, crawler, ai_training, ai_search, user_fetch. Wildcards allowed, so ["*"] matches every signed request. |
threat | The client IP's threat matches, by source name or kind. | bot, crawler, ai_training, ai_search, user_fetch, a source name such as crawlers_other, … |
signal | Classifier rules that trigger for the request. BOT holds when either the device or the IP BOT rule triggers. | ["BOT"] |
An unsigned request matches no agent value, so != conditions hold for it: agent != ["verified"] matches unsigned requests too. See Condition fields.
BOT in the Recommended preset
The Recommended preset escalates requests with the BOT signal to a code challenge and a harder proof-of-work. That includes verified agents and IPs on crawler lists. To let verified search crawlers through without a challenge, add Verified agents before it. Crawlers that don't sign keep getting the challenge.
Using the API
Test your rules with GET /v2/inspect and the securityRules trace. A request with a signature that Sentinel can't verify:
curl 'https://sentinel.example.com/v2/inspect?apiKey=key_…' \
-H 'Signature-Input: sig1=("@authority");created=1763520000;expires=1763520060;keyid="x";tag="web-bot-auth"' \
-H 'Signature: sig1=:AAAA:'{
"code": "ACCESS_DENIED",
"error": "Agent denied.",
"statusCode": 403,
"data": {
"device": {
"rules": { "BOT": { "score": 0.0 }, … },
…
},
"securityRules": [
{
"index": 0,
"name": "verified-agents",
"matched": false,
"outcome": "continue",
"conditions": [
{ "field": "agent", "matched": false, "signals": null },
{ "field": "agent", "matched": true, "signals": null }
],
…
},
{
"index": 1,
"name": "block-ai",
"matched": true,
"outcome": "deny",
"conditions": [{ "field": "agent", "matched": true, "signals": null }],
…
},
…
],
…
}
}The signature is unverified, so block-ai denies the request and device BOT stays at 0. A verified agent would show "outcome": "allow" on verified-agents and device BOT above 0. See the API reference.
To see the IP signal, classify an IP from a crawler list with POST /v2/classifier and a restricted key:
curl -X POST https://sentinel.example.com/v2/classifier \
-u 'key_…:sec_…' \
-H 'Content-Type: application/json' \
-d '{ "ip": "20.171.206.5" }'{
"data": {
"ip": {
"threats": [
{ "source": "crawlers_ai_training", "kind": "ai_training", "action": null, "threatScore": null, "match": "20.171.206.0/24" }
],
"action": null,
"triggeredRules": ["BOT"],
…
},
…
}
}Request logs
For API keys with request logging, each logged request records the Web Bot Auth result in context.agent: the verified agent's name, or unverified. Unsigned requests have no agent. Sentinel records it whether or not the group has an agent rule:
{
"path": "/v2/challenge",
"statusCode": 403,
"error": "ACCESS_DENIED",
"context": { "agent": "unverified", "rules": "block-ai", "threatAction": "block", "threats": "firehol_level1" },
…
}Review these requests under Logs in the admin UI. See Request logs & analytics.
Tuning
- Start in monitor mode. Add
"monitor": trueto a new deny rule and watch its activity before blocking agents your users rely on. - Pick kinds, not names. Agent names change as operators add keys; kinds stay stable. Use a name only to make an exception for one agent.
- Exempt carefully.
acceptskips every later rule, rate limits included. A verified agent that misbehaves is then limited only by the global[rate_limit].
Limitations
- Only agents that sign with a key on the trusted list are verified. Other signed requests are
unverified, even if their signature is valid for a key in their own directory. - Only Ed25519 keys are supported.
- Sentinel keeps no nonce store. A captured signature can be replayed until its
expires, but only against Sentinel's own host. - Agents that don't sign are recognized only by their IP, and only while the crawler and bot lists include it. An agent that hides behind residential proxies and a browser User-Agent isn't detected as a bot. Other signals, such as Human Interaction Signature and the device rules, still apply.