Performance & sizing
This page shows how fast one Sentinel replica answers its widget and feature endpoints, explains which work costs CPU and memory, and lists the settings that change it. For the minimum and production sizes of a replica, see Sizing.
Benchmark setup
One Sentinel 2.0.0-beta.1 container ran on an AWS c7a.medium (1 vCPU, 2 GiB, compute-optimized) with an external Redis ([redis] nodes). A separate client with 4 CPUs in the same network sent the requests over HTTP.
- Concurrency 1 and 16. For each scenario and concurrency, the client sent 50 warm-up requests, then timed 1,000 requests with 1 or 16 in flight at a time. Every request succeeded.
- Throughput is successful requests per second (req/s) over the measurement. Latency is the round trip measured by the client, in milliseconds, from sending a request until its response body was read.
- Widget key. A
publickey whose security group keeps the default PBKDF2pow.cost, soGET /v2/challengederives every challenge at production cost, but sets a lowpow.counterso the client could solve challenges quickly. Challenges were fetched and solved before each verify measurement, unmeasured. - Feature key. A
restrictedkey in a security group without rules, forPOST /v2/verify/signatureand the feature endpoints. - Fixed inputs. The request bodies rotate through a short list of IPs, emails, texts, time zones, and user agents, so lookups hit Sentinel's caches and the classifier sees repeated texts. Results for unique traffic can be lower.
- Production path. Every request went through request logging, rate limiting, and penalties, as in production; the rate limit was raised so the benchmark didn't hit it. No similarity model was configured.
Results
Widget endpoints
GET /v2/challenge derives a key for every challenge at the default PBKDF2 cost. POST /v2/verify checks a solution and records it for replay protection; the "with fields" rows also send form fields (email, message) and a time zone, which the classifier scores. POST /v2/verify/signature is the server-side check of a verified payload.
| Endpoint | Conc. | req/s | p50 (ms) | p90 (ms) | p99 (ms) |
|---|---|---|---|---|---|
GET /v2/challenge | 1 | 1294 | 0.72 | 0.73 | 0.85 |
GET /v2/challenge | 16 | 1511 | 10.34 | 10.90 | 11.24 |
POST /v2/verify | 1 | 2532 | 0.33 | 0.34 | 0.60 |
POST /v2/verify | 16 | 4425 | 3.64 | 4.15 | 5.05 |
POST /v2/verify with fields | 1 | 862 | 1.12 | 1.30 | 1.72 |
POST /v2/verify with fields | 16 | 1038 | 14.98 | 18.66 | 26.32 |
POST /v2/verify/signature | 1 | 3195 | 0.25 | 0.26 | 0.29 |
POST /v2/verify/signature | 16 | 6944 | 1.87 | 2.58 | 3.25 |
Classifier and feature endpoints
| Endpoint | Conc. | req/s | p50 (ms) | p90 (ms) | p99 (ms) |
|---|---|---|---|---|---|
POST /v2/classifier | 1 | 778 | 1.24 | 1.44 | 1.67 |
POST /v2/classifier | 16 | 909 | 17.85 | 20.39 | 25.70 |
POST /v2/ip | 1 | 2165 | 0.34 | 0.38 | 1.66 |
POST /v2/ip | 16 | 3759 | 4.19 | 4.41 | 5.34 |
POST /v2/threat | 1 | 2625 | 0.33 | 0.34 | 0.36 |
POST /v2/threat | 16 | 3831 | 4.14 | 4.19 | 5.41 |
POST /v2/language | 1 | 1300 | 0.76 | 0.90 | 1.06 |
POST /v2/language | 16 | 1524 | 10.56 | 11.27 | 22.68 |
POST /v2/user-agent | 1 | 3546 | 0.22 | 0.24 | 0.26 |
POST /v2/user-agent | 16 | 6667 | 2.28 | 2.43 | 3.59 |
POST /v2/time-zone | 1 | 4902 | 0.15 | 0.16 | 0.18 |
POST /v2/time-zone | 16 | 12346 | 1.04 | 1.15 | 1.54 |
POST /v2/phishing | 1 | 4854 | 0.16 | 0.16 | 0.18 |
POST /v2/phishing | 16 | 11628 | 1.09 | 1.22 | 3.38 |
POST /v2/eml (small message) | 1 | 902 | 1.08 | 1.24 | 1.45 |
POST /v2/eml (small message) | 16 | 1059 | 14.81 | 18.10 | 26.29 |
With remote [ip_resolver] providers, an uncached IP or threat lookup also waits for the provider.
Reading the results
- Challenge derivation is CPU-bound. On 1 vCPU,
GET /v2/challengewent from 1294 to 1511 req/s between 1 and 16 concurrent requests, while its p50 latency grew from 0.72 to 10.34 ms: the requests queued for the CPU. More vCPUs, with[pow] cpusraised to match, derive more challenges at once. - Verification is cheap.
POST /v2/verifychecks a solution and records it in Redis for replay protection, andPOST /v2/verify/signaturechecks a signature; both served several times the challenge rate. - Classification is the expensive part of a verification. Sending form fields to classify cut
POST /v2/verifyfrom 4425 to 1038 req/s at 16 concurrent, close toPOST /v2/classifieralone (909 req/s). - Cached lookups are fastest.
user-agent,time-zone, andphishinganswered in under 4 ms at p99 with 16 concurrent requests.
These numbers use the default PBKDF2 cost, a fixed set of inputs, and a single 1-vCPU replica. Use compute-optimized instances, as Sizing recommends, measure your own traffic mix before you size a cluster, and watch each replica's CPU and memory in Monitoring.
Where CPU and memory go
| Work | Cost | Controlled by |
|---|---|---|
Challenge derivation (GET /v2/challenge, POST /v2/challenge) | CPU for every challenge, in proportion to pow.cost. | pow.cost and pow.algorithm in the security group; [pow] cpus caps parallel derivations. |
Memory-hard algorithms (ARGON2ID, SCRYPT) | CPU plus the memory cost per derivation: pow.memoryCost for ARGON2ID (default 65536 KiB), pow.cost for SCRYPT. Peak memory is about [pow] cpus × memory cost. | [pow] max_memory_cost (default 262144 KiB) caps the memory cost of every rule. |
Classification (POST /v2/classifier, POST /v2/verify with fields) | CPU for language detection, text checks, and campaign tracking. | The security group's classifier rules. |
Local similarity model ([similarity] provider = "local") | About 135 MB to download (a 118 MB int8 ONNX model and a 17 MB tokenizer). Each session holds its own copy of the model in memory, and every embedding uses CPU. | [similarity.local] sessions and threads; [similarity] max_chunks. |
AI decider ([classifier.ai]) | Outbound requests to the model provider, not local CPU. | [classifier.ai] max_concurrency, queue_timeout, timeout. |
| Request logs | A database write per batch of logged requests, for keys with request logging on. | [request_logs], or ClickHouse. |
When to add Redis
Add an external Redis or Valkey ([redis] nodes or [redis.sentinel]) when:
- you run more than one replica, which requires it; see Clustering;
POST /v2/verifyorPOST /v2/verify/signaturecarries real traffic: each verification writes a replay-protection key and every request updates rate-limit counters, which the built-in KV store writes to the main database;- your main database is a network database and you want to keep per-request KV writes off it.
The benchmark above ran with an external Redis. A single small replica with light traffic works with the built-in KV store.
When to add ClickHouse
Request logs are buffered and written to the main database in batches, and raw logs are kept for [request_logs] retention (default 72h). Move them to ClickHouse when many API keys log requests at high volume, when you want to keep raw logs longer than a few days, or when log writes and queries compete with the main database. See ClickHouse.
Tuning settings
| Key | Effect |
|---|---|
[pow] cpus | Challenge derivations at once (default: half the CPUs). Raise to the vCPU count when challenges dominate. |
[pow] max_memory_cost | Caps the memory of one ARGON2ID or SCRYPT derivation. |
[redis] nodes | Moves replay protection, rate limits, and penalties from the database to Redis. |
[redis] synchronous | Built-in KV store on SQLite: normal skips the per-commit disk sync for faster writes, at the risk of losing the last writes on a power loss. |
[database] sqlite_journal_mode | Embedded SQLite file: keep the default wal, where reads don't wait for writes. delete, where reads and writes block each other, is only for network storage; see SQLite on network storage. |
[database] pool_min, pool_max | Database connections per replica (defaults 2 and 20). The database sees up to pool_max × replicas. |
[database] acquire_timeout | How long a request waits for a free connection before it fails. |
[api_key_cache] enabled, max_keys, refresh_interval | Keeps API keys, security groups, and accounts in memory so widget requests skip the database. Raise max_keys above your key count. |
[request_logs] flush_interval, batch_size | How often and in what batch size buffered logs are written. Larger batches mean fewer writes. |
[rate_limit] memory_max_keys | Memory bound of the in-memory rate-limit fallback while the KV store is down. |
[similarity.local] sessions, threads | Parallel embeddings; each session adds a copy of the model in memory. |
[similarity] queue_timeout | Wait for a free model session before EMBEDDING_UNAVAILABLE. |
[classifier.ai] max_concurrency, queue_timeout | Concurrent AI requests and the wait for a free slot. |