# Remote files & data sources

> **Sentinel v2 (beta):** This page documents Sentinel v2, a beta rewrite that may change before the stable release. It does not apply to Sentinel v1, the current stable version: see the [Sentinel v1 documentation](/docs/sentinel.md).

Several Sentinel features work from data that it downloads: email domain lists, phishing URLs, threat IP lists, Web Bot Auth keys, malicious file hashes, and GeoIP databases. Each download is a **remote file**. Sentinel keeps a copy in its data directory and checks every remote file for updates on its own schedule.

You configure the server-side lists in `sentinel.toml`. Root users manage threat sources at runtime and watch every download in the admin UI or through `/v2/remote-files`.

## What Sentinel downloads

Most lists come from the ALTCHA directory at `https://directory.altcha.org`, as gzipped CSV files (`<list>.csv.gz`). The lists in the first eight rows are enabled by default.

| Remote file `name` | Used by | Default source | Default schedule | Size limit | Configured in |
| --- | --- | --- | --- | --- | --- |
| `disposable-email-domains` | `DISPOSABLE` | `directory.altcha.org` | `@daily` | `10` MiB | [`[classifier.disposable]`](/docs/sentinel/v2/reference/configuration/#classifier-lists) |
| `free-email-domains` | `FREE_PROVIDER` | `directory.altcha.org` | `@daily` | `10` MiB | [`[classifier.free_providers]`](/docs/sentinel/v2/reference/configuration/#classifier-lists) |
| `trusted-free-email-providers` | Well-known providers: never `DISPOSABLE`, half the `FREE_PROVIDER` score | `directory.altcha.org` | `@daily` | `10` MiB | [`[classifier.trusted_providers]`](/docs/sentinel/v2/reference/configuration/#classifier-lists) |
| `email-privacy-relays` | `PRIVACY_RELAY` | `directory.altcha.org` | `@daily` | `10` MiB | [`[classifier.privacy_relays]`](/docs/sentinel/v2/reference/configuration/#classifier-lists) |
| `user-agents` | `USER_AGENT_HEADER_UNCOMMON` | `directory.altcha.org` | `@daily` | `10` MiB | [`[classifier.user_agents]`](/docs/sentinel/v2/reference/configuration/#classifier-lists) |
| `phishing` | `URL_PHISHING`, `POST /v2/phishing` | `directory.altcha.org` | `@daily` | `10` MiB | [`[classifier.phishing]`](/docs/sentinel/v2/reference/configuration/#classifier-lists) |
| `malicious-file-hashes` | `MALICIOUS_ATTACHMENT` in `POST /v2/eml` | `directory.altcha.org` | `@daily` | `100` MiB | [`[eml.malicious_hashes]`](/docs/sentinel/v2/reference/configuration/#emlmalicious_hashes) |
| `web-bot-auth` | The security-group `agent` condition | `directory.altcha.org` | `@hourly` | `10` MiB, fixed | [`[web_bot_auth]`](/docs/sentinel/v2/reference/configuration/#web_bot_auth) |
| `threat-<source id>` | Threat sources: IP lists for the `threat` condition and threat verdicts | `directory.altcha.org` for the default sources, any URL for your own | `0 */4 * * *` | The source's `maxSizeMb` | **Threat Intelligence**, or `/v2/threat-sources` |
| The provider's `name`, e.g. `geolite2-city` | GeoIP and network lookups of a `maxmind` or `ipinfo-mmdb` provider with a `url` | The provider's `url` (MaxMind, IPinfo) | `@daily` | None | [`[[ip_resolver.providers]]`](/docs/sentinel/v2/reference/configuration/#ip_resolverproviders) |
| `similarity-model`, `similarity-tokenizer` | The local embedding model of `[similarity] provider = "local"` | `huggingface.co` | `@monthly` | None | [`[similarity.local]`](/docs/sentinel/v2/reference/configuration/#similaritylocal) |
| `similarity-groups` | Built-in `builtin:*` training groups, when enabled | Your own CSV URL | `@weekly` | `10` MiB, fixed | [`[similarity.builtin]`](/docs/sentinel/v2/reference/configuration/#similaritybuiltin) |

An IP resolver provider or embedding model with a local `path` instead of a `url` isn't a remote file; Sentinel reads the file directly.

### Default threat sources

On first boot, when the database has no threat sources, Sentinel creates 15 threat sources from the ALTCHA directory: FireHOL level 1, proxies and abusers, Spamhaus DROP (IPv4 and IPv6), blocklist.de lists, Tor exits, BotScout, and four crawler categories. Each checks for updates on the `0 */4 * * *` schedule (every 4 hours, UTC) and has its own `maxEntries` and `maxSizeMb`, from 100000 entries and 10 MiB up to 5000000 entries and 50 MiB for `firehol_proxies`.

Set `[threats] seed_defaults = false` before the first start to skip them. After that, root users disable, delete, or change sources under **Threat Intelligence** or with `PATCH /v2/threat-sources/<id>`. See [Threat intelligence](/docs/sentinel/v2/features/threat-intelligence/).

## Size and entry limits

Two limits protect memory and disk, per list:

- **`max_size_mb`** caps the download in MiB, before extraction (`1`–`1024`). A larger download is stopped, fails, and the stored copy stays in use. The default is `10` for the classifier lists and `100` for `[eml.malicious_hashes]`. Threat sources use their `maxSizeMb`; a source without one has no limit.
- **`max_entries`** caps the entries Sentinel loads from a downloaded list (`1`–`10000000`); the rest are ignored. The defaults are `100000` for `[classifier.phishing]` and `10000` for `[classifier.user_agents]` (the most-weighted user agents; the rest count as uncommon). The other classifier lists and `[eml.malicious_hashes]` have no limit unless you set one. Threat sources use their `maxEntries`.

```toml
[classifier.phishing]
max_entries = 250000   # default 100000
max_size_mb = 20       # default 10 MiB, before extraction
```

## How updates work

Every remote file follows the same update cycle:

- **Checks.** On each scheduled run, Sentinel sends a `HEAD` request and compares the response's `ETag` with the stored copy's, or `Last-Modified` when either side has no `ETag`. Only a changed file is downloaded. A server that sends neither header gets a full download on every check. When a server rejects `HEAD` with `403`, `405`, or `501` (for example, presigned URLs), the check uses a `GET` and closes the connection after the headers.
- **Downloads.** A download replaces the stored copy only once it is complete and valid. A failed, partial, or oversized download leaves the previous copy in use and records the reason in the file's `error`. GeoIP databases must also open as MaxMind DB files.
- **Archives.** For a `.csv.gz` URL, Sentinel keeps the `.csv` file inside it. GeoIP downloads may be a tar, tar.gz, zip, or gzip archive with one `.mmdb` file inside. The format is detected from the content, not the URL.
- **Retries.** A file that has never been downloaded is retried after 30 seconds, then with a doubling delay up to every 30 minutes, until it succeeds. A file with a stored copy waits for its next scheduled check.
- **Schedules** are 5-field cron expressions (`minute hour day month weekday`) or `@hourly`, `@daily`, `@weekly`, `@monthly`, and `@yearly`, evaluated in UTC.
- **Timeouts.** A check fails after 30 seconds. A download fails when connecting takes longer than 10 seconds or when no data arrives for 60 seconds.
- **Concurrency.** Each replica runs at most 4 checks or downloads at a time across all files. A file never has two downloads running on the same replica.

Requests carry `User-Agent: altcha-sentinel/<version>`. Requests to `https://directory.altcha.org` are also signed with an `Authorization` header that identifies Sentinel; other hosts receive only the headers you configure. Outbound requests honor `HTTPS_PROXY`; see [Outbound proxy & air-gapped deployments](/docs/sentinel/v2/configure/air-gapped/).

At startup, Sentinel uses the copies already in the data directory and checks or downloads each file in the background. Startup never waits for a download.

## Storage and replicas

Remote files are stored under `<data_dir>/remote-files`, one file per `name` plus a `<name>.meta.json` with its `ETag`, `Last-Modified`, and download time:

```text
data/remote-files/
├── disposable-email-domains
├── disposable-email-domains.meta.json
├── phishing
├── phishing.meta.json
└── …
```

The directory is a cache: deleting it only causes fresh downloads. Keep `data_dir` on a persistent volume anyway, so a restart doesn't download every list again.

Each replica keeps its own remote files and schedule:

- **Separate data directories.** Every replica downloads every file itself. With MaxMind, each replica's download counts against your daily download limit.
- **A shared data directory** (a volume mounted by every replica). Replicas coordinate with a cluster-wide lock in the database: one downloads, and the others wait and then use its copy.

Replicas report their remote-file state to each other every 10 seconds over the KV store, so the API shows every live replica. Threat sources reach other replicas within `[threats] sync_interval`, and IP resolver providers come from each replica's own `sentinel.toml`. A file can therefore be missing on some replicas for a while, or for good when their configurations differ. See [Clustering & high availability](/docs/sentinel/v2/operations/clustering/).

## Change or disable a list

Each list table takes `enabled`, `url`, and `schedule`. Disabling a list stops its downloads; the list stays empty and its rule scores 0. Pointing `url` at another server, such as an internal mirror, keeps the list:

```toml
[classifier.user_agents]
enabled = false

[classifier.disposable]
url      = "https://mirror.example.internal/altcha/disposable-email-domains.csv.gz"
schedule = "0 */6 * * *"
```

A URL whose path ends in `.csv.gz` is read in the ALTCHA directory format, with the header line dropped, on any host. Classifier lists also accept a JSON array or one entry per line. To mirror the directory, see [Mirror the lists](/docs/sentinel/v2/configure/air-gapped/#mirror-the-lists). The key reference for each list is in [Classifier lists](/docs/sentinel/v2/reference/configuration/#classifier-lists).

Configuration changes apply at the next restart. Threat sources change at runtime: a new source downloads right away, and a changed `url`, `schedule`, `headers`, or `maxSizeMb` registers the file again.

## Watch and refresh downloads

Root users see every remote file under **Remote Files** in the admin UI. The table shows **Used by**, **Status**, **Replicas**, **Size**, **Downloaded**, **Last checked**, and **Next check**, and it warns when replicas are out of sync. **View status** opens a file's state on each replica, with **Check for updates**. **Force download** downloads the file again on every replica.

The same operations are available through the API with a root user's session token. API keys aren't accepted (`401 UNAUTHORIZED`), and other users get `403 ROOT_PRIVILEGES_REQUIRED`.

| Endpoint | What it does |
| --- | --- |
| `GET /v2/remote-files` | Lists every remote file and its state on each live replica. |
| `POST /v2/remote-files/<name>/_check` | Checks the URL now on the answering replica and starts a download if the file changed. Waits for the check, not the download. |
| `POST /v2/remote-files/<name>/_update` | Downloads now on every live replica that has the file, without checking first. Answers `202 Accepted`. |

You can't add, edit, or delete remote files through this API. They follow their owner: the config file or the threat source.

### List remote files

```bash
curl -H 'Authorization: Bearer <jwt>' https://sentinel.example.com/v2/remote-files
```

```json
{
"data": [
{
  "name": "disposable-email-domains",
  "replicas": [
    {
      "replicaId": "rep_…",
      "current": true,
      "file": {
        "name": "disposable-email-domains",
        "owner": "classifier_list",
        "url": "https://directory.altcha.org/disposable-email-domains.csv.gz",
        "schedule": "@daily",
        "extract": "*.csv",
        "downloaded": true,
        "member": "disposable-email-domains.csv",
        "size": 149294,
        "sha256": "c46893fb…",
        "etag": "\"78d9256e…\"",
        "lastModified": null,
        "downloadedAt": "2026-11-18T08:48:48.356294+00:00",
        "checkedAt": null,
        "nextCheckAt": "2026-11-19T00:00:00+00:00",
        "updating": false,
        "error": null,
        "failures": 0
      }
    }
  ]
},
…
]
}
```

`replicas` has one element per live replica; `current` marks the one that answered. `file` is `null` on a replica where the file isn't registered. The fields that answer the usual questions:

| Field | Type | Description |
| --- | --- | --- |
| `downloaded` | boolean | `true` once a download of the current `url` has completed. |
| `error` | string \| null | The last failed check or download, as text. Cleared by the next success. With `downloaded: true`, the replica is still using its previous copy. |
| `failures` | integer | Failed checks and downloads in a row on this replica. Reset by a success, by a restart, and when the file is registered again. |
| `downloadedAt` | string \| null | When the stored copy was downloaded (RFC 3339). `lastModified` says how old the data is at its source. |
| `sha256` | string \| null | Hash of the stored file. The same value on every replica means the same content. |
| `checkedAt`, `nextCheckAt` | string \| null | The last and next update check. `checkedAt` is `null` until the replica's first check, for example right after a download at startup. |

Every field is in the [API reference](/sentinel-api-docs-v2.html#tag/remote-files/GET/v2/remote-files).

### Check for an update

```bash
curl -X POST -H 'Authorization: Bearer <jwt>' \
https://sentinel.example.com/v2/remote-files/phishing/_check
```

```json
{
"data": {
"replicaId": "rep_…",
"outcome": "up_to_date",
"file": { "name": "phishing", "downloaded": true, "checkedAt": "2026-11-18T08:49:04.738111+00:00", "updating": false, "error": null, … }
}
}
```

| `outcome` | Meaning |
| --- | --- |
| `up_to_date` | The URL reports the same `ETag` or `Last-Modified` as the stored copy. Nothing is downloaded. |
| `update_started` | The file changed, was never downloaded, or has no header to compare. A download is running. |
| `already_updating` | A download was already running on this replica. No check was made. |

To learn when the download finishes, poll `GET /v2/remote-files` until `updating` is `false` on that replica: a new `downloadedAt` means success, an unchanged one with `error` set means failure.

> **Tip:**
>
> Prefer **Check for updates** (`_check`) over **Force download** (`_update`). A check doesn't download an unchanged file, while a forced download runs on every replica, which counts against provider limits such as MaxMind's daily database downloads. Force a download when a server sends wrong `ETag`/`Last-Modified` headers, rejects both `HEAD` and `GET` checks, or you suspect a damaged copy.

### Errors

| HTTP | Code | Meaning |
| --- | --- | --- |
| 404 | `NOT_FOUND` | `_check`: the file isn't registered on the answering replica. `_update`: no live replica has it. |
| 502 | `CHECK_FAILED` | `_check` only: the check failed with a network error, a timeout, or a 4xx/5xx status. `error` has the reason, and the stored copy stays in use. |
| 503 | `BROADCAST_FAILED` | `_update` only: the request couldn't reach the other replicas through the KV store. No download started. |

Failed downloads never fail these requests, because they finish later; they show in the file's `error`. All codes are in [Error codes](/docs/sentinel/v2/reference/errors/).

## Alerts for failed downloads

Each replica checks its remote files every minute. When a file's `failures` reaches `[alerts.download_failures] failures` (default `3`), Sentinel raises the system-wide `download_failures` alert, with the file name as its subject. A file that keeps failing notifies about once per `[alerts] cooldown`.

```toml
[alerts.download_failures]
enabled  = true
failures = 3   # failed checks or downloads in a row (1-100)
```

The doctor's `network.host` checks probe the origin of every remote file the configuration registers, so they show which host is unreachable. See [Alerts](/docs/sentinel/v2/operations/alerts/) and [Doctor & troubleshooting](/docs/sentinel/v2/operations/troubleshooting/).

## Related

- [Classifier lists in the Configuration reference](/docs/sentinel/v2/reference/configuration/#classifier-lists)
- [Outbound proxy & air-gapped deployments](/docs/sentinel/v2/configure/air-gapped/)
- [IP resolvers](/docs/sentinel/v2/configure/ip-resolvers/)
- [Threat intelligence](/docs/sentinel/v2/features/threat-intelligence/)
- [Classifier](/docs/sentinel/v2/features/classifier/)
- [Remote files in the API reference](/sentinel-api-docs-v2.html#tag/remote-files)
