Skip to main content
ALTCHA®

Bots & AI agents

Sentinel identifies bots, crawlers, and AI agents, including agents that sign their requests with Web Bot Auth, so you can allow or block them with rules. It recognizes them by two independent signals: a verified request signature, and the client IP's presence on crawler and bot lists.

How it works

Web Bot Auth

AI agents and crawlers can sign their HTTP requests with Web Bot Auth: HTTP Message Signatures (RFC 9421) in the Signature-Input and Signature headers, optionally with Signature-Agent. Sentinel checks every API-key request that has a Signature-Input header against a curated list of trusted agent keys, the ALTCHA directory list web-bot-auth, refreshed hourly. Each listed key belongs to a named agent, such as chatgpt-agent, google-agent or exa-crawler, with a category that Sentinel maps to a kind:

KindAgents that
ai_trainingCollect content for AI model training.
ai_searchIndex content for AI search and assistants.
user_fetchFetch a page on a user's request, such as an AI assistant operating a browser.
crawlerAnything else: search engines, SEO tools, link previews, archives, browser automation services.

A request has one of three results:

RequestResult
Signed by a listed key, valid nowVerified: the agent's name and kind are known.
Signed, but malformed, expired, by an unlisted key, or with a bad signatureUnverified
No Signature-Input headerNot checked

A signature verifies only if all of these hold:

  • The key is an Ed25519 key on the trusted list. Sentinel never fetches the key directory that Signature-Agent names.
  • The signature has the web-bot-auth tag and the created, expires and keyid parameters. expires is in the future, and created is at most 60 seconds ahead of Sentinel's clock.
  • It covers @authority (or @target-uri), plus Signature-Agent when the request has one, and the authority is Sentinel's own host: the listener's [server] forwarded_host_header when set and sent by a trusted proxy, otherwise Host. @scheme and @target-uri are always taken as https.

If your reverse proxy rewrites Host, set forwarded_host_header (for example X-Forwarded-Host), or every signature fails as unverified. See Host and allowed hosts.

Bot and crawler lists

Agents that don't sign their requests are recognized by their IP address, through threat intelligence. The default threat sources label IPs with these kinds:

SourceKind
crawlers_ai_trainingai_training
crawlers_ai_searchai_search
crawlers_user_fetchuser_fetch
crawlers_othercrawler
botscout_30d, blocklist_de_botsbot

These sources only label IPs; none of them blocks on its own. An IP can be on more than one list: Google's crawler addresses, for example, are shared by Googlebot and Google's AI search crawler, so they're on both crawlers_ai_search and crawlers_other.

Device BOT and IP BOT

The classifier has two rules named BOT, one per signal:

Device BOTIP BOT
Triggers whenThe request is signed by a verified Web Bot Auth agentA threat match has kind bot, crawler, ai_training, ai_search or user_fetch
Categorydeviceip
Scored inPOST /v2/verify, GET /v2/inspectPOST /v2/verify, POST /v2/classifier, GET /v2/inspect
Can be fakedNo: needs a listed private keyNot the IP itself, but a shared address carries the labels of everyone behind it

The device rule only sees the signature of the request it scores. POST /v2/classifier classifies headers you pass in the body, so its device BOT never triggers. Scores are in Classifier rules & scores.

Enabling it

Web Bot Auth is on by default:

[web_bot_auth]
enabled = true      # false: the list isn't downloaded and no request is checked
schedule = "@hourly"

url replaces the default list with your own CSV with the columns name, keyid, category and jwk. See [web_bot_auth]. The crawler lists come with the default threat sources; see Default threat sources.

Detection has no effect until a security-group rule uses it. Two presets cover the common cases:

  • Verified agents accepts verified agents that aren't AI training crawlers or user-driven fetchers, so verified search crawlers skip escalation and rate limits.
  • Block AI agents denies verified ai_training and user_fetch agents, any request whose signature doesn't verify, and IPs on an ai_training list.

Together, as the first rules of a group:

[
  {
    "name": "verified-agents",
    "action": "accept",
    "conditions": [
      { "field": "agent", "operator": "=", "value": ["verified"] },
      { "field": "agent", "operator": "!=", "value": ["ai_training", "user_fetch"] }
    ]
  },
  {
    "name": "block-ai",
    "action": "deny",
    "message": "Agent denied.",
    "conditions": [{ "field": "agent", "operator": "=", "value": ["unverified", "ai_training", "user_fetch"] }]
  },
  {
    "name": "block-ai-lists",
    "action": "deny",
    "message": "Agent denied.",
    "conditions": [{ "field": "threat", "operator": "=", "value": ["ai_training"] }]
  }
]

Most agents on the Web Bot Auth list today are user_fetch agents, such as assistants that open a page in a browser for their user. Denying user_fetch stops them from using your forms on a user's behalf; leave it out of the deny rule if you want to allow that.

Rules & signals

Rule conditions

ConditionMatchesValues
agentThe request's Web Bot Auth result. A verified agent matches verified, its name, and its kind; an unverified signature matches unverified; an unsigned request matches nothing.verified, unverified, an agent name, crawler, ai_training, ai_search, user_fetch. Wildcards allowed, so ["*"] matches every signed request.
threatThe client IP's threat matches, by source name or kind.bot, crawler, ai_training, ai_search, user_fetch, a source name such as crawlers_other, …
signalClassifier rules that trigger for the request. BOT holds when either the device or the IP BOT rule triggers.["BOT"]

An unsigned request matches no agent value, so != conditions hold for it: agent != ["verified"] matches unsigned requests too. See Condition fields.

The Recommended preset escalates requests with the BOT signal to a code challenge and a harder proof-of-work. That includes verified agents and IPs on crawler lists. To let verified search crawlers through without a challenge, add Verified agents before it. Crawlers that don't sign keep getting the challenge.

Using the API

Test your rules with GET /v2/inspect and the securityRules trace. A request with a signature that Sentinel can't verify:

curl 'https://sentinel.example.com/v2/inspect?apiKey=key_…' \
  -H 'Signature-Input: sig1=("@authority");created=1763520000;expires=1763520060;keyid="x";tag="web-bot-auth"' \
  -H 'Signature: sig1=:AAAA:'
{
  "code": "ACCESS_DENIED",
  "error": "Agent denied.",
  "statusCode": 403,
  "data": {
    "device": {
      "rules": { "BOT": { "score": 0.0 }, … },
      …
    },
    "securityRules": [
      {
        "index": 0,
        "name": "verified-agents",
        "matched": false,
        "outcome": "continue",
        "conditions": [
          { "field": "agent", "matched": false, "signals": null },
          { "field": "agent", "matched": true, "signals": null }
        ],
        …
      },
      {
        "index": 1,
        "name": "block-ai",
        "matched": true,
        "outcome": "deny",
        "conditions": [{ "field": "agent", "matched": true, "signals": null }],
        …
      },
      …
    ],
    …
  }
}

The signature is unverified, so block-ai denies the request and device BOT stays at 0. A verified agent would show "outcome": "allow" on verified-agents and device BOT above 0. See the API reference.

To see the IP signal, classify an IP from a crawler list with POST /v2/classifier and a restricted key:

curl -X POST https://sentinel.example.com/v2/classifier \
  -u 'key_…:sec_…' \
  -H 'Content-Type: application/json' \
  -d '{ "ip": "20.171.206.5" }'
{
  "data": {
    "ip": {
      "threats": [
        { "source": "crawlers_ai_training", "kind": "ai_training", "action": null, "threatScore": null, "match": "20.171.206.0/24" }
      ],
      "action": null,
      "triggeredRules": ["BOT"],
      …
    },
    …
  }
}

Request logs

For API keys with request logging, each logged request records the Web Bot Auth result in context.agent: the verified agent's name, or unverified. Unsigned requests have no agent. Sentinel records it whether or not the group has an agent rule:

{
  "path": "/v2/challenge",
  "statusCode": 403,
  "error": "ACCESS_DENIED",
  "context": { "agent": "unverified", "rules": "block-ai", "threatAction": "block", "threats": "firehol_level1" },
  …
}

Review these requests under Logs in the admin UI. See Request logs & analytics.

Tuning

  • Start in monitor mode. Add "monitor": true to a new deny rule and watch its activity before blocking agents your users rely on.
  • Pick kinds, not names. Agent names change as operators add keys; kinds stay stable. Use a name only to make an exception for one agent.
  • Exempt carefully. accept skips every later rule, rate limits included. A verified agent that misbehaves is then limited only by the global [rate_limit].

Limitations

  • Only agents that sign with a key on the trusted list are verified. Other signed requests are unverified, even if their signature is valid for a key in their own directory.
  • Only Ed25519 keys are supported.
  • Sentinel keeps no nonce store. A captured signature can be replayed until its expires, but only against Sentinel's own host.
  • Agents that don't sign are recognized only by their IP, and only while the crawler and bot lists include it. An agent that hides behind residential proxies and a browser User-Agent isn't detected as a bot. Other signals, such as Human Interaction Signature and the device rules, still apply.

Start typing to search...

↑ ↓ Navigate ↵ Select