# Performance & sizing

> **Sentinel v2 (beta):** This page documents Sentinel v2, a beta rewrite that may change before the stable release. It does not apply to Sentinel v1, the current stable version: see the [Sentinel v1 documentation](/docs/sentinel.md).

This page shows how fast one Sentinel replica answers its widget and feature endpoints, explains which work costs CPU and memory, and lists the settings that change it. For the minimum and production sizes of a replica, see [Sizing](/docs/sentinel/v2/install/#sizing).

## Benchmark setup

One Sentinel `2.0.0-beta.1` container ran on an AWS `c7a.medium` (1 vCPU, 2 GiB, compute-optimized) with an external Redis (`[redis] nodes`). A separate client with 4 CPUs in the same network sent the requests over HTTP.

- **Concurrency 1 and 16.** For each scenario and concurrency, the client sent 50 warm-up requests, then timed 1,000 requests with 1 or 16 in flight at a time. Every request succeeded.
- **Throughput** is successful requests per second (req/s) over the measurement. **Latency** is the round trip measured by the client, in milliseconds, from sending a request until its response body was read.
- **Widget key.** A `public` key whose security group keeps the default PBKDF2 `pow.cost`, so `GET /v2/challenge` derives every challenge at production cost, but sets a low `pow.counter` so the client could solve challenges quickly. Challenges were fetched and solved before each verify measurement, unmeasured.
- **Feature key.** A `restricted` key in a security group without rules, for `POST /v2/verify/signature` and the feature endpoints.
- **Fixed inputs.** The request bodies rotate through a short list of IPs, emails, texts, time zones, and user agents, so lookups hit Sentinel's caches and the classifier sees repeated texts. Results for unique traffic can be lower.
- **Production path.** Every request went through request logging, rate limiting, and penalties, as in production; the rate limit was raised so the benchmark didn't hit it. No similarity model was configured.

## Results

### Widget endpoints

`GET /v2/challenge` derives a key for every challenge at the default PBKDF2 cost. `POST /v2/verify` checks a solution and records it for replay protection; the "with fields" rows also send form fields (`email`, `message`) and a time zone, which the classifier scores. `POST /v2/verify/signature` is the server-side check of a verified payload.

| Endpoint | Conc. | req/s | p50 (ms) | p90 (ms) | p99 (ms) |
| --- | --: | --: | --: | --: | --: |
| `GET /v2/challenge` | 1 | 1294 | 0.72 | 0.73 | 0.85 |
| `GET /v2/challenge` | 16 | 1511 | 10.34 | 10.90 | 11.24 |
| `POST /v2/verify` | 1 | 2532 | 0.33 | 0.34 | 0.60 |
| `POST /v2/verify` | 16 | 4425 | 3.64 | 4.15 | 5.05 |
| `POST /v2/verify` with fields | 1 | 862 | 1.12 | 1.30 | 1.72 |
| `POST /v2/verify` with fields | 16 | 1038 | 14.98 | 18.66 | 26.32 |
| `POST /v2/verify/signature` | 1 | 3195 | 0.25 | 0.26 | 0.29 |
| `POST /v2/verify/signature` | 16 | 6944 | 1.87 | 2.58 | 3.25 |

### Classifier and feature endpoints

| Endpoint | Conc. | req/s | p50 (ms) | p90 (ms) | p99 (ms) |
| --- | --: | --: | --: | --: | --: |
| `POST /v2/classifier` | 1 | 778 | 1.24 | 1.44 | 1.67 |
| `POST /v2/classifier` | 16 | 909 | 17.85 | 20.39 | 25.70 |
| `POST /v2/ip` | 1 | 2165 | 0.34 | 0.38 | 1.66 |
| `POST /v2/ip` | 16 | 3759 | 4.19 | 4.41 | 5.34 |
| `POST /v2/threat` | 1 | 2625 | 0.33 | 0.34 | 0.36 |
| `POST /v2/threat` | 16 | 3831 | 4.14 | 4.19 | 5.41 |
| `POST /v2/language` | 1 | 1300 | 0.76 | 0.90 | 1.06 |
| `POST /v2/language` | 16 | 1524 | 10.56 | 11.27 | 22.68 |
| `POST /v2/user-agent` | 1 | 3546 | 0.22 | 0.24 | 0.26 |
| `POST /v2/user-agent` | 16 | 6667 | 2.28 | 2.43 | 3.59 |
| `POST /v2/time-zone` | 1 | 4902 | 0.15 | 0.16 | 0.18 |
| `POST /v2/time-zone` | 16 | 12346 | 1.04 | 1.15 | 1.54 |
| `POST /v2/phishing` | 1 | 4854 | 0.16 | 0.16 | 0.18 |
| `POST /v2/phishing` | 16 | 11628 | 1.09 | 1.22 | 3.38 |
| `POST /v2/eml` (small message) | 1 | 902 | 1.08 | 1.24 | 1.45 |
| `POST /v2/eml` (small message) | 16 | 1059 | 14.81 | 18.10 | 26.29 |

With remote [`[ip_resolver]`](/docs/sentinel/v2/reference/configuration/#ip_resolver) providers, an uncached IP or threat lookup also waits for the provider.

## Reading the results

- **Challenge derivation is CPU-bound.** On 1 vCPU, `GET /v2/challenge` went from 1294 to 1511 req/s between 1 and 16 concurrent requests, while its p50 latency grew from 0.72 to 10.34 ms: the requests queued for the CPU. More vCPUs, with [`[pow] cpus`](/docs/sentinel/v2/reference/configuration/#pow) raised to match, derive more challenges at once.
- **Verification is cheap.** `POST /v2/verify` checks a solution and records it in Redis for replay protection, and `POST /v2/verify/signature` checks a signature; both served several times the challenge rate.
- **Classification is the expensive part of a verification.** Sending form fields to classify cut `POST /v2/verify` from 4425 to 1038 req/s at 16 concurrent, close to `POST /v2/classifier` alone (909 req/s).
- **Cached lookups are fastest.** `user-agent`, `time-zone`, and `phishing` answered in under 4 ms at p99 with 16 concurrent requests.

These numbers use the default PBKDF2 cost, a fixed set of inputs, and a single 1-vCPU replica. Use compute-optimized instances, as [Sizing](/docs/sentinel/v2/install/#sizing) recommends, measure your own traffic mix before you size a cluster, and watch each replica's CPU and memory in [Monitoring](/docs/sentinel/v2/operations/monitoring/).

## Where CPU and memory go

| Work | Cost | Controlled by |
| --- | --- | --- |
| Challenge derivation (`GET /v2/challenge`, `POST /v2/challenge`) | CPU for every challenge, in proportion to `pow.cost`. | `pow.cost` and `pow.algorithm` in the security group; [`[pow] cpus`](/docs/sentinel/v2/reference/configuration/#pow) caps parallel derivations. |
| Memory-hard algorithms (`ARGON2ID`, `SCRYPT`) | CPU plus the memory cost per derivation: `pow.memoryCost` for `ARGON2ID` (default `65536` KiB), `pow.cost` for `SCRYPT`. Peak memory is about `[pow] cpus` × memory cost. | [`[pow] max_memory_cost`](/docs/sentinel/v2/reference/configuration/#pow) (default `262144` KiB) caps the memory cost of every rule. |
| Classification (`POST /v2/classifier`, `POST /v2/verify` with fields) | CPU for language detection, text checks, and campaign tracking. | The security group's classifier rules. |
| Local similarity model (`[similarity] provider = "local"`) | About 135 MB to download (a 118 MB int8 ONNX model and a 17 MB tokenizer). Each session holds its own copy of the model in memory, and every embedding uses CPU. | [`[similarity.local] sessions`](/docs/sentinel/v2/reference/configuration/#similaritylocal) and `threads`; [`[similarity] max_chunks`](/docs/sentinel/v2/reference/configuration/#similarity). |
| AI decider (`[classifier.ai]`) | Outbound requests to the model provider, not local CPU. | [`[classifier.ai] max_concurrency`](/docs/sentinel/v2/reference/configuration/#classifierai), `queue_timeout`, `timeout`. |
| Request logs | A database write per batch of logged requests, for keys with request logging on. | [`[request_logs]`](/docs/sentinel/v2/reference/configuration/#request_logs), or [ClickHouse](/docs/sentinel/v2/operations/clickhouse/). |

<Callout type="tip" title="Memory-hard proof of work">

With `ARGON2ID` at its default memory cost of `65536` KiB (64 MiB) and `[pow] cpus = 4`, challenge derivation alone can use 256 MiB at peak. Size memory for `[pow] cpus` × the largest memory cost your security groups use, or lower `max_memory_cost`.

## When to add Redis

Add an external Redis or Valkey ([`[redis] nodes`](/docs/sentinel/v2/reference/configuration/#redis) or [`[redis.sentinel]`](/docs/sentinel/v2/reference/configuration/#redissentinel)) when:

- you run more than one replica, which requires it; see [Clustering](/docs/sentinel/v2/operations/clustering/);
- `POST /v2/verify` or `POST /v2/verify/signature` carries real traffic: each verification writes a replay-protection key and every request updates rate-limit counters, which the built-in KV store writes to the main database;
- your main database is a network database and you want to keep per-request KV writes off it.

The benchmark above ran with an external Redis. A single small replica with light traffic works with the built-in KV store.

## When to add ClickHouse

Request logs are buffered and written to the main database in batches, and raw logs are kept for [`[request_logs] retention`](/docs/sentinel/v2/reference/configuration/#request_logs) (default `72h`). Move them to ClickHouse when many API keys log requests at high volume, when you want to keep raw logs longer than a few days, or when log writes and queries compete with the main database. See [ClickHouse](/docs/sentinel/v2/operations/clickhouse/).

## Tuning settings

| Key | Effect |
| --- | --- |
| [`[pow] cpus`](/docs/sentinel/v2/reference/configuration/#pow) | Challenge derivations at once (default: half the CPUs). Raise to the vCPU count when challenges dominate. |
| [`[pow] max_memory_cost`](/docs/sentinel/v2/reference/configuration/#pow) | Caps the memory of one `ARGON2ID` or `SCRYPT` derivation. |
| [`[redis] nodes`](/docs/sentinel/v2/reference/configuration/#redis) | Moves replay protection, rate limits, and penalties from the database to Redis. |
| [`[redis] synchronous`](/docs/sentinel/v2/reference/configuration/#redis) | Built-in KV store on SQLite: `normal` skips the per-commit disk sync for faster writes, at the risk of losing the last writes on a power loss. |
| [`[database] sqlite_journal_mode`](/docs/sentinel/v2/reference/configuration/#database) | Embedded SQLite file: keep the default `wal`, where reads don't wait for writes. `delete`, where reads and writes block each other, is only for network storage; see [SQLite on network storage](/docs/sentinel/v2/configure/databases/#sqlite-on-network-storage). |
| [`[database] pool_min`, `pool_max`](/docs/sentinel/v2/reference/configuration/#database) | Database connections per replica (defaults `2` and `20`). The database sees up to `pool_max` × replicas. |
| [`[database] acquire_timeout`](/docs/sentinel/v2/reference/configuration/#database) | How long a request waits for a free connection before it fails. |
| [`[api_key_cache]`](/docs/sentinel/v2/reference/configuration/#api_key_cache) `enabled`, `max_keys`, `refresh_interval` | Keeps API keys, security groups, and accounts in memory so widget requests skip the database. Raise `max_keys` above your key count. |
| [`[request_logs] flush_interval`, `batch_size`](/docs/sentinel/v2/reference/configuration/#request_logs) | How often and in what batch size buffered logs are written. Larger batches mean fewer writes. |
| [`[rate_limit] memory_max_keys`](/docs/sentinel/v2/reference/configuration/#rate_limit) | Memory bound of the in-memory rate-limit fallback while the KV store is down. |
| [`[similarity.local] sessions`, `threads`](/docs/sentinel/v2/reference/configuration/#similaritylocal) | Parallel embeddings; each session adds a copy of the model in memory. |
| [`[similarity] queue_timeout`](/docs/sentinel/v2/reference/configuration/#similarity) | Wait for a free model session before `EMBEDDING_UNAVAILABLE`. |
| [`[classifier.ai] max_concurrency`, `queue_timeout`](/docs/sentinel/v2/reference/configuration/#classifierai) | Concurrent AI requests and the wait for a free slot. |

## Related

- [Sizing](/docs/sentinel/v2/install/#sizing)
- [Clustering](/docs/sentinel/v2/operations/clustering/)
- [ClickHouse](/docs/sentinel/v2/operations/clickhouse/)
- [Monitoring](/docs/sentinel/v2/operations/monitoring/)
- [Configuration reference](/docs/sentinel/v2/reference/configuration/)
