Skip to main content
ALTCHA®

Remote files & data sources

Several Sentinel features work from data that it downloads: email domain lists, phishing URLs, threat IP lists, Web Bot Auth keys, malicious file hashes, and GeoIP databases. Each download is a remote file. Sentinel keeps a copy in its data directory and checks every remote file for updates on its own schedule.

You configure the server-side lists in sentinel.toml. Root users manage threat sources at runtime and watch every download in the admin UI or through /v2/remote-files.

What Sentinel downloads

Most lists come from the ALTCHA directory at https://directory.altcha.org, as gzipped CSV files (<list>.csv.gz). The lists in the first eight rows are enabled by default.

Remote file nameUsed byDefault sourceDefault scheduleSize limitConfigured in
disposable-email-domainsDISPOSABLEdirectory.altcha.org@daily10 MiB[classifier.disposable]
free-email-domainsFREE_PROVIDERdirectory.altcha.org@daily10 MiB[classifier.free_providers]
trusted-free-email-providersWell-known providers: never DISPOSABLE, half the FREE_PROVIDER scoredirectory.altcha.org@daily10 MiB[classifier.trusted_providers]
email-privacy-relaysPRIVACY_RELAYdirectory.altcha.org@daily10 MiB[classifier.privacy_relays]
user-agentsUSER_AGENT_HEADER_UNCOMMONdirectory.altcha.org@daily10 MiB[classifier.user_agents]
phishingURL_PHISHING, POST /v2/phishingdirectory.altcha.org@daily10 MiB[classifier.phishing]
malicious-file-hashesMALICIOUS_ATTACHMENT in POST /v2/emldirectory.altcha.org@daily100 MiB[eml.malicious_hashes]
web-bot-authThe security-group agent conditiondirectory.altcha.org@hourly10 MiB, fixed[web_bot_auth]
threat-<source id>Threat sources: IP lists for the threat condition and threat verdictsdirectory.altcha.org for the default sources, any URL for your own0 */4 * * *The source's maxSizeMbThreat Intelligence, or /v2/threat-sources
The provider's name, e.g. geolite2-cityGeoIP and network lookups of a maxmind or ipinfo-mmdb provider with a urlThe provider's url (MaxMind, IPinfo)@dailyNone[[ip_resolver.providers]]
similarity-model, similarity-tokenizerThe local embedding model of [similarity] provider = "local"huggingface.co@monthlyNone[similarity.local]
similarity-groupsBuilt-in builtin:* training groups, when enabledYour own CSV URL@weekly10 MiB, fixed[similarity.builtin]

An IP resolver provider or embedding model with a local path instead of a url isn't a remote file; Sentinel reads the file directly.

Default threat sources

On first boot, when the database has no threat sources, Sentinel creates 15 threat sources from the ALTCHA directory: FireHOL level 1, proxies and abusers, Spamhaus DROP (IPv4 and IPv6), blocklist.de lists, Tor exits, BotScout, and four crawler categories. Each checks for updates on the 0 */4 * * * schedule (every 4 hours, UTC) and has its own maxEntries and maxSizeMb, from 100000 entries and 10 MiB up to 5000000 entries and 50 MiB for firehol_proxies.

Set [threats] seed_defaults = false before the first start to skip them. After that, root users disable, delete, or change sources under Threat Intelligence or with PATCH /v2/threat-sources/<id>. See Threat intelligence.

Size and entry limits

Two limits protect memory and disk, per list:

  • max_size_mb caps the download in MiB, before extraction (1–1024). A larger download is stopped, fails, and the stored copy stays in use. The default is 10 for the classifier lists and 100 for [eml.malicious_hashes]. Threat sources use their maxSizeMb; a source without one has no limit.
  • max_entries caps the entries Sentinel loads from a downloaded list (1–10000000); the rest are ignored. The defaults are 100000 for [classifier.phishing] and 10000 for [classifier.user_agents] (the most-weighted user agents; the rest count as uncommon). The other classifier lists and [eml.malicious_hashes] have no limit unless you set one. Threat sources use their maxEntries.
[classifier.phishing]
max_entries = 250000   # default 100000
max_size_mb = 20       # default 10 MiB, before extraction

How updates work

Every remote file follows the same update cycle:

  • Checks. On each scheduled run, Sentinel sends a HEAD request and compares the response's ETag with the stored copy's, or Last-Modified when either side has no ETag. Only a changed file is downloaded. A server that sends neither header gets a full download on every check. When a server rejects HEAD with 403, 405, or 501 (for example, presigned URLs), the check uses a GET and closes the connection after the headers.
  • Downloads. A download replaces the stored copy only once it is complete and valid. A failed, partial, or oversized download leaves the previous copy in use and records the reason in the file's error. GeoIP databases must also open as MaxMind DB files.
  • Archives. For a .csv.gz URL, Sentinel keeps the .csv file inside it. GeoIP downloads may be a tar, tar.gz, zip, or gzip archive with one .mmdb file inside. The format is detected from the content, not the URL.
  • Retries. A file that has never been downloaded is retried after 30 seconds, then with a doubling delay up to every 30 minutes, until it succeeds. A file with a stored copy waits for its next scheduled check.
  • Schedules are 5-field cron expressions (minute hour day month weekday) or @hourly, @daily, @weekly, @monthly, and @yearly, evaluated in UTC.
  • Timeouts. A check fails after 30 seconds. A download fails when connecting takes longer than 10 seconds or when no data arrives for 60 seconds.
  • Concurrency. Each replica runs at most 4 checks or downloads at a time across all files. A file never has two downloads running on the same replica.

Requests carry User-Agent: altcha-sentinel/<version>. Requests to https://directory.altcha.org are also signed with an Authorization header that identifies Sentinel; other hosts receive only the headers you configure. Outbound requests honor HTTPS_PROXY; see Outbound proxy & air-gapped deployments.

At startup, Sentinel uses the copies already in the data directory and checks or downloads each file in the background. Startup never waits for a download.

Storage and replicas

Remote files are stored under <data_dir>/remote-files, one file per name plus a <name>.meta.json with its ETag, Last-Modified, and download time:

data/remote-files/
├── disposable-email-domains
├── disposable-email-domains.meta.json
├── phishing
├── phishing.meta.json
└── …

The directory is a cache: deleting it only causes fresh downloads. Keep data_dir on a persistent volume anyway, so a restart doesn't download every list again.

Each replica keeps its own remote files and schedule:

  • Separate data directories. Every replica downloads every file itself. With MaxMind, each replica's download counts against your daily download limit.
  • A shared data directory (a volume mounted by every replica). Replicas coordinate with a cluster-wide lock in the database: one downloads, and the others wait and then use its copy.

Replicas report their remote-file state to each other every 10 seconds over the KV store, so the API shows every live replica. Threat sources reach other replicas within [threats] sync_interval, and IP resolver providers come from each replica's own sentinel.toml. A file can therefore be missing on some replicas for a while, or for good when their configurations differ. See Clustering & high availability.

Change or disable a list

Each list table takes enabled, url, and schedule. Disabling a list stops its downloads; the list stays empty and its rule scores 0. Pointing url at another server, such as an internal mirror, keeps the list:

[classifier.user_agents]
enabled = false

[classifier.disposable]
url      = "https://mirror.example.internal/altcha/disposable-email-domains.csv.gz"
schedule = "0 */6 * * *"

A URL whose path ends in .csv.gz is read in the ALTCHA directory format, with the header line dropped, on any host. Classifier lists also accept a JSON array or one entry per line. To mirror the directory, see Mirror the lists. The key reference for each list is in Classifier lists.

Configuration changes apply at the next restart. Threat sources change at runtime: a new source downloads right away, and a changed url, schedule, headers, or maxSizeMb registers the file again.

Watch and refresh downloads

Root users see every remote file under Remote Files in the admin UI. The table shows Used by, Status, Replicas, Size, Downloaded, Last checked, and Next check, and it warns when replicas are out of sync. View status opens a file's state on each replica, with Check for updates. Force download downloads the file again on every replica.

The same operations are available through the API with a root user's session token. API keys aren't accepted (401 UNAUTHORIZED), and other users get 403 ROOT_PRIVILEGES_REQUIRED.

EndpointWhat it does
GET /v2/remote-filesLists every remote file and its state on each live replica.
POST /v2/remote-files/<name>/_checkChecks the URL now on the answering replica and starts a download if the file changed. Waits for the check, not the download.
POST /v2/remote-files/<name>/_updateDownloads now on every live replica that has the file, without checking first. Answers 202 Accepted.

You can't add, edit, or delete remote files through this API. They follow their owner: the config file or the threat source.

List remote files

curl -H 'Authorization: Bearer <jwt>' https://sentinel.example.com/v2/remote-files
{
  "data": [
    {
      "name": "disposable-email-domains",
      "replicas": [
        {
          "replicaId": "rep_…",
          "current": true,
          "file": {
            "name": "disposable-email-domains",
            "owner": "classifier_list",
            "url": "https://directory.altcha.org/disposable-email-domains.csv.gz",
            "schedule": "@daily",
            "extract": "*.csv",
            "downloaded": true,
            "member": "disposable-email-domains.csv",
            "size": 149294,
            "sha256": "c46893fb…",
            "etag": "\"78d9256e…\"",
            "lastModified": null,
            "downloadedAt": "2026-11-18T08:48:48.356294+00:00",
            "checkedAt": null,
            "nextCheckAt": "2026-11-19T00:00:00+00:00",
            "updating": false,
            "error": null,
            "failures": 0
          }
        }
      ]
    },
    …
  ]
}

replicas has one element per live replica; current marks the one that answered. file is null on a replica where the file isn't registered. The fields that answer the usual questions:

FieldTypeDescription
downloadedbooleantrue once a download of the current url has completed.
errorstring | nullThe last failed check or download, as text. Cleared by the next success. With downloaded: true, the replica is still using its previous copy.
failuresintegerFailed checks and downloads in a row on this replica. Reset by a success, by a restart, and when the file is registered again.
downloadedAtstring | nullWhen the stored copy was downloaded (RFC 3339). lastModified says how old the data is at its source.
sha256string | nullHash of the stored file. The same value on every replica means the same content.
checkedAt, nextCheckAtstring | nullThe last and next update check. checkedAt is null until the replica's first check, for example right after a download at startup.

Every field is in the API reference.

Check for an update

curl -X POST -H 'Authorization: Bearer <jwt>' \
  https://sentinel.example.com/v2/remote-files/phishing/_check
{
  "data": {
    "replicaId": "rep_…",
    "outcome": "up_to_date",
    "file": { "name": "phishing", "downloaded": true, "checkedAt": "2026-11-18T08:49:04.738111+00:00", "updating": false, "error": null, … }
  }
}
outcomeMeaning
up_to_dateThe URL reports the same ETag or Last-Modified as the stored copy. Nothing is downloaded.
update_startedThe file changed, was never downloaded, or has no header to compare. A download is running.
already_updatingA download was already running on this replica. No check was made.

To learn when the download finishes, poll GET /v2/remote-files until updating is false on that replica: a new downloadedAt means success, an unchanged one with error set means failure.

Errors

HTTPCodeMeaning
404NOT_FOUND_check: the file isn't registered on the answering replica. _update: no live replica has it.
502CHECK_FAILED_check only: the check failed with a network error, a timeout, or a 4xx/5xx status. error has the reason, and the stored copy stays in use.
503BROADCAST_FAILED_update only: the request couldn't reach the other replicas through the KV store. No download started.

Failed downloads never fail these requests, because they finish later; they show in the file's error. All codes are in Error codes.

Alerts for failed downloads

Each replica checks its remote files every minute. When a file's failures reaches [alerts.download_failures] failures (default 3), Sentinel raises the system-wide download_failures alert, with the file name as its subject. A file that keeps failing notifies about once per [alerts] cooldown.

[alerts.download_failures]
enabled  = true
failures = 3   # failed checks or downloads in a row (1-100)

The doctor's network.host checks probe the origin of every remote file the configuration registers, so they show which host is unreachable. See Alerts and Doctor & troubleshooting.

Start typing to search...

↑ ↓ Navigate ↵ Select