Remote files & data sources
Several Sentinel features work from data that it downloads: email domain lists, phishing URLs, threat IP lists, Web Bot Auth keys, malicious file hashes, and GeoIP databases. Each download is a remote file. Sentinel keeps a copy in its data directory and checks every remote file for updates on its own schedule.
You configure the server-side lists in sentinel.toml. Root users manage threat sources at runtime and watch every download in the admin UI or through /v2/remote-files.
What Sentinel downloads
Most lists come from the ALTCHA directory at https://directory.altcha.org, as gzipped CSV files (<list>.csv.gz). The lists in the first eight rows are enabled by default.
Remote file name | Used by | Default source | Default schedule | Size limit | Configured in |
|---|---|---|---|---|---|
disposable-email-domains | DISPOSABLE | directory.altcha.org | @daily | 10 MiB | [classifier.disposable] |
free-email-domains | FREE_PROVIDER | directory.altcha.org | @daily | 10 MiB | [classifier.free_providers] |
trusted-free-email-providers | Well-known providers: never DISPOSABLE, half the FREE_PROVIDER score | directory.altcha.org | @daily | 10 MiB | [classifier.trusted_providers] |
email-privacy-relays | PRIVACY_RELAY | directory.altcha.org | @daily | 10 MiB | [classifier.privacy_relays] |
user-agents | USER_AGENT_HEADER_UNCOMMON | directory.altcha.org | @daily | 10 MiB | [classifier.user_agents] |
phishing | URL_PHISHING, POST /v2/phishing | directory.altcha.org | @daily | 10 MiB | [classifier.phishing] |
malicious-file-hashes | MALICIOUS_ATTACHMENT in POST /v2/eml | directory.altcha.org | @daily | 100 MiB | [eml.malicious_hashes] |
web-bot-auth | The security-group agent condition | directory.altcha.org | @hourly | 10 MiB, fixed | [web_bot_auth] |
threat-<source id> | Threat sources: IP lists for the threat condition and threat verdicts | directory.altcha.org for the default sources, any URL for your own | 0 */4 * * * | The source's maxSizeMb | Threat Intelligence, or /v2/threat-sources |
The provider's name, e.g. geolite2-city | GeoIP and network lookups of a maxmind or ipinfo-mmdb provider with a url | The provider's url (MaxMind, IPinfo) | @daily | None | [[ip_resolver.providers]] |
similarity-model, similarity-tokenizer | The local embedding model of [similarity] provider = "local" | huggingface.co | @monthly | None | [similarity.local] |
similarity-groups | Built-in builtin:* training groups, when enabled | Your own CSV URL | @weekly | 10 MiB, fixed | [similarity.builtin] |
An IP resolver provider or embedding model with a local path instead of a url isn't a remote file; Sentinel reads the file directly.
Default threat sources
On first boot, when the database has no threat sources, Sentinel creates 15 threat sources from the ALTCHA directory: FireHOL level 1, proxies and abusers, Spamhaus DROP (IPv4 and IPv6), blocklist.de lists, Tor exits, BotScout, and four crawler categories. Each checks for updates on the 0 */4 * * * schedule (every 4 hours, UTC) and has its own maxEntries and maxSizeMb, from 100000 entries and 10 MiB up to 5000000 entries and 50 MiB for firehol_proxies.
Set [threats] seed_defaults = false before the first start to skip them. After that, root users disable, delete, or change sources under Threat Intelligence or with PATCH /v2/threat-sources/<id>. See Threat intelligence.
Size and entry limits
Two limits protect memory and disk, per list:
max_size_mbcaps the download in MiB, before extraction (1–1024). A larger download is stopped, fails, and the stored copy stays in use. The default is10for the classifier lists and100for[eml.malicious_hashes]. Threat sources use theirmaxSizeMb; a source without one has no limit.max_entriescaps the entries Sentinel loads from a downloaded list (1–10000000); the rest are ignored. The defaults are100000for[classifier.phishing]and10000for[classifier.user_agents](the most-weighted user agents; the rest count as uncommon). The other classifier lists and[eml.malicious_hashes]have no limit unless you set one. Threat sources use theirmaxEntries.
[classifier.phishing]
max_entries = 250000 # default 100000
max_size_mb = 20 # default 10 MiB, before extractionHow updates work
Every remote file follows the same update cycle:
- Checks. On each scheduled run, Sentinel sends a
HEADrequest and compares the response'sETagwith the stored copy's, orLast-Modifiedwhen either side has noETag. Only a changed file is downloaded. A server that sends neither header gets a full download on every check. When a server rejectsHEADwith403,405, or501(for example, presigned URLs), the check uses aGETand closes the connection after the headers. - Downloads. A download replaces the stored copy only once it is complete and valid. A failed, partial, or oversized download leaves the previous copy in use and records the reason in the file's
error. GeoIP databases must also open as MaxMind DB files. - Archives. For a
.csv.gzURL, Sentinel keeps the.csvfile inside it. GeoIP downloads may be a tar, tar.gz, zip, or gzip archive with one.mmdbfile inside. The format is detected from the content, not the URL. - Retries. A file that has never been downloaded is retried after 30 seconds, then with a doubling delay up to every 30 minutes, until it succeeds. A file with a stored copy waits for its next scheduled check.
- Schedules are 5-field cron expressions (
minute hour day month weekday) or@hourly,@daily,@weekly,@monthly, and@yearly, evaluated in UTC. - Timeouts. A check fails after 30 seconds. A download fails when connecting takes longer than 10 seconds or when no data arrives for 60 seconds.
- Concurrency. Each replica runs at most 4 checks or downloads at a time across all files. A file never has two downloads running on the same replica.
Requests carry User-Agent: altcha-sentinel/<version>. Requests to https://directory.altcha.org are also signed with an Authorization header that identifies Sentinel; other hosts receive only the headers you configure. Outbound requests honor HTTPS_PROXY; see Outbound proxy & air-gapped deployments.
At startup, Sentinel uses the copies already in the data directory and checks or downloads each file in the background. Startup never waits for a download.
Storage and replicas
Remote files are stored under <data_dir>/remote-files, one file per name plus a <name>.meta.json with its ETag, Last-Modified, and download time:
data/remote-files/
├── disposable-email-domains
├── disposable-email-domains.meta.json
├── phishing
├── phishing.meta.json
└── …The directory is a cache: deleting it only causes fresh downloads. Keep data_dir on a persistent volume anyway, so a restart doesn't download every list again.
Each replica keeps its own remote files and schedule:
- Separate data directories. Every replica downloads every file itself. With MaxMind, each replica's download counts against your daily download limit.
- A shared data directory (a volume mounted by every replica). Replicas coordinate with a cluster-wide lock in the database: one downloads, and the others wait and then use its copy.
Replicas report their remote-file state to each other every 10 seconds over the KV store, so the API shows every live replica. Threat sources reach other replicas within [threats] sync_interval, and IP resolver providers come from each replica's own sentinel.toml. A file can therefore be missing on some replicas for a while, or for good when their configurations differ. See Clustering & high availability.
Change or disable a list
Each list table takes enabled, url, and schedule. Disabling a list stops its downloads; the list stays empty and its rule scores 0. Pointing url at another server, such as an internal mirror, keeps the list:
[classifier.user_agents]
enabled = false
[classifier.disposable]
url = "https://mirror.example.internal/altcha/disposable-email-domains.csv.gz"
schedule = "0 */6 * * *"A URL whose path ends in .csv.gz is read in the ALTCHA directory format, with the header line dropped, on any host. Classifier lists also accept a JSON array or one entry per line. To mirror the directory, see Mirror the lists. The key reference for each list is in Classifier lists.
Configuration changes apply at the next restart. Threat sources change at runtime: a new source downloads right away, and a changed url, schedule, headers, or maxSizeMb registers the file again.
Watch and refresh downloads
Root users see every remote file under Remote Files in the admin UI. The table shows Used by, Status, Replicas, Size, Downloaded, Last checked, and Next check, and it warns when replicas are out of sync. View status opens a file's state on each replica, with Check for updates. Force download downloads the file again on every replica.
The same operations are available through the API with a root user's session token. API keys aren't accepted (401 UNAUTHORIZED), and other users get 403 ROOT_PRIVILEGES_REQUIRED.
| Endpoint | What it does |
|---|---|
GET /v2/remote-files | Lists every remote file and its state on each live replica. |
POST /v2/remote-files/<name>/_check | Checks the URL now on the answering replica and starts a download if the file changed. Waits for the check, not the download. |
POST /v2/remote-files/<name>/_update | Downloads now on every live replica that has the file, without checking first. Answers 202 Accepted. |
You can't add, edit, or delete remote files through this API. They follow their owner: the config file or the threat source.
List remote files
curl -H 'Authorization: Bearer <jwt>' https://sentinel.example.com/v2/remote-files{
"data": [
{
"name": "disposable-email-domains",
"replicas": [
{
"replicaId": "rep_…",
"current": true,
"file": {
"name": "disposable-email-domains",
"owner": "classifier_list",
"url": "https://directory.altcha.org/disposable-email-domains.csv.gz",
"schedule": "@daily",
"extract": "*.csv",
"downloaded": true,
"member": "disposable-email-domains.csv",
"size": 149294,
"sha256": "c46893fb…",
"etag": "\"78d9256e…\"",
"lastModified": null,
"downloadedAt": "2026-11-18T08:48:48.356294+00:00",
"checkedAt": null,
"nextCheckAt": "2026-11-19T00:00:00+00:00",
"updating": false,
"error": null,
"failures": 0
}
}
]
},
…
]
}replicas has one element per live replica; current marks the one that answered. file is null on a replica where the file isn't registered. The fields that answer the usual questions:
| Field | Type | Description |
|---|---|---|
downloaded | boolean | true once a download of the current url has completed. |
error | string | null | The last failed check or download, as text. Cleared by the next success. With downloaded: true, the replica is still using its previous copy. |
failures | integer | Failed checks and downloads in a row on this replica. Reset by a success, by a restart, and when the file is registered again. |
downloadedAt | string | null | When the stored copy was downloaded (RFC 3339). lastModified says how old the data is at its source. |
sha256 | string | null | Hash of the stored file. The same value on every replica means the same content. |
checkedAt, nextCheckAt | string | null | The last and next update check. checkedAt is null until the replica's first check, for example right after a download at startup. |
Every field is in the API reference.
Check for an update
curl -X POST -H 'Authorization: Bearer <jwt>' \
https://sentinel.example.com/v2/remote-files/phishing/_check{
"data": {
"replicaId": "rep_…",
"outcome": "up_to_date",
"file": { "name": "phishing", "downloaded": true, "checkedAt": "2026-11-18T08:49:04.738111+00:00", "updating": false, "error": null, … }
}
}outcome | Meaning |
|---|---|
up_to_date | The URL reports the same ETag or Last-Modified as the stored copy. Nothing is downloaded. |
update_started | The file changed, was never downloaded, or has no header to compare. A download is running. |
already_updating | A download was already running on this replica. No check was made. |
To learn when the download finishes, poll GET /v2/remote-files until updating is false on that replica: a new downloadedAt means success, an unchanged one with error set means failure.
Errors
| HTTP | Code | Meaning |
|---|---|---|
| 404 | NOT_FOUND | _check: the file isn't registered on the answering replica. _update: no live replica has it. |
| 502 | CHECK_FAILED | _check only: the check failed with a network error, a timeout, or a 4xx/5xx status. error has the reason, and the stored copy stays in use. |
| 503 | BROADCAST_FAILED | _update only: the request couldn't reach the other replicas through the KV store. No download started. |
Failed downloads never fail these requests, because they finish later; they show in the file's error. All codes are in Error codes.
Alerts for failed downloads
Each replica checks its remote files every minute. When a file's failures reaches [alerts.download_failures] failures (default 3), Sentinel raises the system-wide download_failures alert, with the file name as its subject. A file that keeps failing notifies about once per [alerts] cooldown.
[alerts.download_failures]
enabled = true
failures = 3 # failed checks or downloads in a row (1-100)The doctor's network.host checks probe the origin of every remote file the configuration registers, so they show which host is unreachable. See Alerts and Doctor & troubleshooting.