# Similarity & training data

> **Sentinel v2 (beta):** This page documents Sentinel v2, a beta rewrite that may change before the stable release. It does not apply to Sentinel v1, the current stable version: see the [Sentinel v1 documentation](/docs/sentinel.md).

Similarity compares submitted text with example messages you label for your account, so a spam message that you've seen once is caught again when it's reworded, translated, or lightly edited. It also flags the same message arriving from many client IPs, with no training data at all.

You manage training data with an admin session (the `training_data` [permission feature](/docs/sentinel/v2/concepts/accounts-users/)) or from your backend with a `restricted` or `full` API key. Semantic matching also needs an embedding model on the server.

## How it works

The classifier runs three similarity rules on the text of `POST /v2/classifier`, `POST /v2/verify` (the widget's form fields), and `POST /v2/eml`:

| Rule | Detects | Needs |
| --- | --- | --- |
| `SIMILARITY` | The text means the same as a training sample, even when worded differently or in another language. | Training data and an embedding model |
| `NEAR_DUPLICATE` | The text is a training sample with small edits, such as one changed word. | Training data |
| `CAMPAIGN` | The same message (near-duplicates included) arrives from many client IPs within a time window. | Nothing |

### Groups and samples

A **sample** is one text (1–4000 characters) with a **label**:

- `positive` samples are what the group catches.
- `negative` samples are look-alikes that must **not** match, for example a legitimate "How do I buy bitcoin on your exchange?" next to crypto-scam positives.

A **group** is a named set of samples in one account, such as `crypto_scam` or `job_offers`. Group names are 1–64 characters `a-z`, `0-9`, `_` and `-`, starting with a letter or digit; they are trimmed and lowercased. Groups have no record of their own: a group exists while it has samples. An account holds at most `[similarity] max_samples_per_account` samples (default `10000`).

Each sample also has:

- a **weight** from `0` to `10` (default `1`), which scales the rule score of a match. A match on a sample with weight `1` or more makes the classification `BAD`; lower weights only add to the score, and `0` records the match without affecting the verdict.
- an optional **threshold** from `0` to `1`: the cosine similarity a text needs to match the sample semantically. Without one, `[similarity] threshold` applies (default `0.80`).
- a **source**: `manual`, `import`, `feedback`, or `suggestion`, depending on how it was added.
- `embedded`: `true` once the sample has an embedding of the server's current model.

### How a group matches

Every sample in a group gets a score against the incoming text:

- `1` when the text is a **near-duplicate**: the 64-bit SimHash fingerprints of the two texts differ in at most `[similarity] near_duplicate_distance` bits (default `3`). Texts under 12 canonical characters must be identical instead.
- Otherwise, when both are embedded, the **cosine similarity** of their embeddings: about `0` for unrelated texts, `1` for the same meaning. Texts longer than about 1000 characters are split into chunks (up to `[similarity] max_chunks`, default `8`), and the best chunk counts.

The group matches when its best positive sample reaches its threshold **and** beats the best negative sample's score by `[similarity] negative_margin` (default `0.05`). A near-duplicate always passes its threshold, so a negative near-duplicate suppresses every positive in its group. Among positives with the same score, the higher weight wins.

Near-duplicate matching suits messages of a few hundred characters. In a single sentence, one changed word usually moves the fingerprint by more than 3 bits, so short texts match as near-duplicates only when they are identical after canonicalization. Use semantic matching for short texts.

### Canonical text

Before comparing, Sentinel normalizes each text. Texts with the same canonical form count as identical for near-duplicate matching and duplicate checks, and embedding models receive the canonical text:

- Unicode compatibility forms are folded (`ｆｒｅｅ` → `free`) and the text is lowercased.
- Invisible characters, such as zero-width spaces, are removed.
- In text that mixes Latin with Cyrillic or Greek letters, look-alike letters become Latin (`сrypto` with a Cyrillic `с` → `crypto`).
- In words with an ASCII letter and at least as many letters as digits or `@ $ € |`, those characters are read as letters (`fr33` → `free`). `!` and `+` are never folded, and mostly numeric words such as `4x4` stay as they are.
- Runs of three or more equal characters shrink to two (`!!!!` → `!!`), and whitespace is collapsed.

### Embedding providers

`[similarity] provider` chooses the model that computes embeddings:

| `provider` | Model |
| --- | --- |
| `none` (default) | No semantic matching. `NEAR_DUPLICATE`, `CAMPAIGN`, sample management, and suggestions still work. |
| `local` | An ONNX model that runs inside Sentinel; by default a multilingual MiniLM (about 135 MB download). |
| `openai` | An OpenAI-compatible `/embeddings` API: OpenAI, or a self-hosted Ollama, vLLM, or TEI. |

Sentinel embeds samples in the background, in batches of 64, usually within seconds of a create or text change. Until then a sample can only match as a near-duplicate. A different model (another provider, model file, model name, or `dimensions`) is a different vector space, so every sample is embedded again after a change. The local model downloads at startup; until it has loaded, `SIMILARITY` scores `null`.

### Campaigns

`CAMPAIGN` counts how many **distinct client IPs** sent the same message to one account within `[similarity.campaign] window` (default `1h`):

- Below `min_sources` (default `5`) distinct sources, it scores `0`. From there it scores `5 × sources / min_sources`, at most `10`: with the defaults, 5 sources score `5` and 10 sources score `10`.
- The source is the client IP for `/v2/verify`, the body's `ip` for `/v2/classifier`, and the sending server's IP for `/v2/eml`. A `/v2/classifier` call **without** `ip` counts as a new source every time, so send `ip` whenever you have it.
- All endpoints of one account count together.
- Texts under `min_chars` canonical characters (default `30`) aren't counted, and `CAMPAIGN` scores `null` for them. Short messages like "thanks!" are too common to signal a campaign.
- Counting is approximate: near-duplicates are found through parts of their fingerprint, so occasionally unrelated texts that share a part are counted together. The counters live in the KV store, shared by all replicas, with up to 8 short-lived keys per distinct message.

### Built-in groups

The operator can load read-only groups that every account can use, from a CSV file you host yourself (the ALTCHA directory doesn't publish one). They are named `builtin:` plus the CSV's `group` value, for example `builtin:crypto_scam`, and their sample ids are `builtin:<group>:<line>`. Built-in groups don't appear in the training-data API and aren't included in `*`: name them explicitly. Account group names can't contain `:`, so `crypto_scam` and `builtin:crypto_scam` never collide.

## Enabling it

### Server configuration

Matching against training data, near-duplicates, and campaigns work without configuration. For semantic matching, choose an embedding provider and restart:

```toml
[similarity]
provider  = "local"   # ONNX model, ~135 MB download at startup
threshold = 0.80      # default for samples without their own threshold

[similarity.campaign]
min_sources = 5       # distinct client IPs within the window

[similarity.suggestions]
enabled = true        # stores message text; see Privacy below
```

For an OpenAI-compatible API instead:

```toml
[similarity]
provider = "openai"

[similarity.openai]
api_key = "${OPENAI_API_KEY}"
model   = "text-embedding-3-small"
```

For built-in groups, set `[similarity.builtin] enabled = true` and `url` to your CSV (`group,label,text,weight` header, plain or `.csv.gz`). Sentinel downloads it as the [remote file](/docs/sentinel/v2/configure/remote-files/) `similarity-groups`, by default `@weekly`.

Every replica must use the same model. Each key is in the Configuration reference: [`[similarity]`](/docs/sentinel/v2/reference/configuration/#similarity), [`[similarity.local]`](/docs/sentinel/v2/reference/configuration/#similaritylocal), [`[similarity.openai]`](/docs/sentinel/v2/reference/configuration/#similarityopenai), [`[similarity.campaign]`](/docs/sentinel/v2/reference/configuration/#similaritycampaign), [`[similarity.builtin]`](/docs/sentinel/v2/reference/configuration/#similaritybuiltin), and [`[similarity.suggestions]`](/docs/sentinel/v2/reference/configuration/#similaritysuggestions). **Doctor → Run checks** reports the **Embedding model** state, and `GET /v2/cluster/replicas` shows each replica's provider, model, sample counts, and last embedding error.

### Choosing groups

By default the classifier compares text with every group of the API key's account (`*`). A security group's `classifier.similarityGroups` set field narrows or extends that list: `*`, group names, and `builtin:` groups, up to 50, separated by `,`, `;`, or whitespace. Unknown names are skipped; an empty value selects no group, so only `CAMPAIGN` runs.

```json
[
{
"name": "similarity",
"action": "set",
"set": [{ "field": "classifier.similarityGroups", "value": "crypto_scam, builtin:crypto_scam" }]
},
{
"name": "comments",
"action": "set",
"conditions": [{ "field": "origin", "operator": "=", "value": ["https://forum.example.com"] }],
"set": [{ "field": "classifier.similarityGroups", "value": "comment_spam crypto_scam" }]
}
]
```

A later matching `set` rule overrides an earlier one, so the second rule switches groups for the forum. An invalid name is rejected with `400 INVALID_RULES` (`invalid group name 'bad/name'`). See [Classifier set fields](/docs/sentinel/v2/reference/rules/#classifier).

Per request, the `POST /v2/classifier` body's `similarityGroups` array (for example `["*", "builtin:crypto_scam"]`) and the `X-Similarity-Groups` header of `POST /v2/eml` replace the group's list. On `/v2/verify`, the widget must send its form fields; see [Verifying submissions](/docs/sentinel/v2/integrate/verification/).

## Managing training data

In the admin UI, open **Training Data**. The **Samples** tab lists, searches, exports, creates (**Create sample**), and imports (**Import**) samples; **Groups** shows each group's positive, negative, and pending counts and has **Evaluate thresholds**; **Suggestions** is the review queue. Every action has an API equivalent under `/v2/training-data` with an admin session. See the [training-data API reference](/sentinel-api-docs-v2.html#tag/training-data/POST/v2/training-data).

| Call | Does |
| --- | --- |
| `POST /v2/training-data` | Creates a sample (`accountId`, `group`, `text`, optional `label`, `weight`, `threshold`). |
| `PATCH /v2/training-data/{id}` | Updates a sample. `threshold: null` clears the threshold; a changed `text` is embedded again. |
| `DELETE /v2/training-data/{id}` | Deletes a sample permanently. |
| `POST /v2/training-data/_search` | Lists samples (`filter.accountId` required; `group`, `label`, `search`). |
| `POST /v2/training-data/_groups` | Per group: `positives`, `negatives`, and `pending` (samples not yet embedded). |
| `POST /v2/training-data/_import` | Adds up to 1000 texts to one group. |
| `POST /v2/training-data/_evaluate` | Suggests a threshold for a group. See [Tuning](#tuning). |
| `POST /v2/training-data/_feedback` | Adds a sample from your backend, with an API key. |

Create and update don't check for duplicates. Import, feedback, and accepting a suggestion do.

### Bulk import

`_import` adds texts to one group in a single all-or-nothing call; `label`, `weight`, and `threshold` apply to all of them:

```bash
curl https://sentinel.example.com/v2/training-data/_import \
-H 'Authorization: Bearer <jwt>' \
-H 'Content-Type: application/json' \
-d '{
"accountId": "acc_…",
"group": "crypto_scam",
"label": "positive",
"texts": [
  "Double your bitcoin in 24 hours. Guaranteed returns, no risk!",
  "Our AI trading bot made me $4,000 last week. DM me to join.",
  "  ",
  "our ai trading bot made me $4,000 last week. dm me to join."
]
}'
```

```json
{"data":{"created":1,"ids":["trd_…"],"skipped":3}}
```

Blank texts are skipped, and so is any text whose canonical form equals an existing sample in the group or an earlier text in the request. Here the first text already existed and the last differs from the second only in case. A text over 4000 characters fails the whole request with `400 VALIDATION_ERROR`, and an import that would exceed the sample limit fails with `400 TRAINING_DATA_LIMIT`; nothing is stored in either case.

### Feedback from your application

`_feedback` turns a moderator's decision into a sample, for example a "This is spam" button or restoring a false positive. Call it from your backend with a `restricted` or `full` key; the sample goes into the key's account, and the key's security-group rules apply:

```bash
curl https://sentinel.example.com/v2/training-data/_feedback \
-u 'key_…:sec_…' \
-H 'Content-Type: application/json' \
-d '{"group": "crypto_scam", "text": "Bitcoin doubler: send any amount and get twice as much back within 24 hours. Act fast!", "label": "positive"}'
```

```json
{"data":{"duplicate":false,"id":"trd_…"}}
```

New samples get weight `1`, no threshold, and `source: "feedback"`. If the group already has a near-duplicate with the same label, nothing is added and the response returns its `id` with `duplicate: true`, so repeated clicks are harmless. A near-duplicate with the **opposite** label returns `409 LABEL_CONFLICT`, naming the conflicting sample:

```json
{"code":"LABEL_CONFLICT","error":"Sample 'trd_…' has the opposite label.","statusCode":409}
```

Decide which label is right, then fix or delete that sample. A `public` key gets `403 NON_PUBLIC_API_KEY_REQUIRED`, and a session token `401 API_KEY_REQUIRED`.

### Training suggestions

With `[similarity.suggestions] enabled = true` (off by default), Sentinel queues texts that the built-in [text rules](/docs/sentinel/v2/features/text-analysis/) find spam, so a campaign that the word lists catch today becomes training data that catches its rewordings tomorrow. A text is captured when:

- it was classified for an API key, and
- the text category alone is `BAD` (`text.score` ≥ 2); other categories neither cause nor prevent a capture, and
- it isn't a near-duplicate of any sample in the account, in any group, with either label.

A near-duplicate of a queued suggestion increases that suggestion's `hits` instead (within about 10 seconds). When the account has `max_per_account` suggestions (default `1000`, counted per replica), new texts are dropped. Suggestions are deleted after `retention` (default `7d`).

Review them under **Training Data → Suggestions**: **Accept** adds the text to a group as a sample, and deleting removes it from the queue. With the API:

```bash
curl https://sentinel.example.com/v2/training-suggestions/trs_…/_accept \
-H 'Authorization: Bearer <jwt>' \
-H 'Content-Type: application/json' \
-d '{"group": "pharma", "label": "positive"}'
```

```json
{
"data": {
"sample": {
  "id": "trd_…",
  "group": "pharma",
  "text": "Buy viagra now, cheap pills, discreet shipping worldwide, no prescription needed! Casino bonus free spins.",
  "label": "positive",
  "weight": 1.0,
  "threshold": null,
  "source": "suggestion",
  …
},
"duplicate": false
}
}
```

Accept with `label: "negative"` to record a false positive of the text rules for this account. Duplicate and conflict handling are the same as for `_feedback`. A suggestion that someone else already accepted or deleted returns `404 NOT_FOUND`. List the queue with `POST /v2/training-suggestions/_search` (`sortBy`: `createdAt` or `hits`); see the [training-suggestions API reference](/sentinel-api-docs-v2.html#tag/training-suggestions/POST/v2/training-suggestions/_search).

## Rules & signals

| Rule | Meaning |
| --- | --- |
| `SIMILARITY` | The text matches a selected group semantically. `null` when no selected group has embedded samples or the text couldn't be embedded. |
| `NEAR_DUPLICATE` | The text is a near-duplicate of a positive sample in a selected group. `null` when the selected groups have no samples. |
| `CAMPAIGN` | At least `min_sources` distinct sources sent this message within the window. `null` for short texts or with campaigns off. |

A match on a sample with weight `1` or more raises the classification to `BAD`. The classification's `similarity` object lists the matched groups (`group`, `sampleId`, `matchType`, `score`, `weight`), `campaign` (`sources`, `windowSec`, or `null` when the check didn't run), and `error` (`EMBEDDING_UNAVAILABLE` when semantic matching was skipped). Scores, the category formula, and floors are in [Classifier rules & scores](/docs/sentinel/v2/reference/classifier-rules/#similarity). Turn rules off with `classifier.disableRules` or the body's `disableRules`, for example `["CAMPAIGN"]` for test traffic.

An excerpt of a `POST /v2/classifier` response for a reworded scam message:

```json
"similarity": {
"matches": [
{ "group": "crypto_scam", "sampleId": "trd_…", "matchType": "semantic", "score": 0.9804, "weight": 1.0 }
],
"campaign": { "sources": 1, "windowSec": 3600 },
"error": null,
"rules": {
"SIMILARITY": { "score": 4.9 },
"NEAR_DUPLICATE": { "score": 0.0 },
"CAMPAIGN": { "score": 0.0 }
},
"score": 4.9,
"time": 5.387,
"triggeredRules": ["SIMILARITY"]
}
```

## Using the API

`POST /v2/similarity` explains how a text compares with training data: which samples it resembles, how closely, and why each group matched or didn't. It's read-only: it doesn't classify, count campaigns, capture suggestions, or write request logs. Call it with a `restricted` or `full` key:

```bash
curl https://sentinel.example.com/v2/similarity \
-u 'key_…:sec_…' \
-H 'Content-Type: application/json' \
-d '{"text": "Give us your bitcoin and our trading bot doubles it in a day, guaranteed.", "groups": ["*"], "limit": 3}'
```

```json
{
"data": {
"modelId": "local:d6ea442ff6a891da:mean",
"error": null,
"time": 4.062,
"groups": [
  {
    "group": "crypto_scam",
    "matched": true,
    "score": 0.7563,
    "threshold": 0.6600000262260437,
    "suppressedBy": null,
    "matches": [
      {
        "sampleId": "trd_…",
        "text": "Our crypto bot doubles your deposit in one day, zero risk, guaranteed profit.",
        "label": "positive",
        "score": 0.7563,
        "matchType": "semantic",
        "distance": 22,
        "aboveThreshold": true,
        "span": { "start": 0, "end": 73 }
      },
      …
    ]
  }
]
}
}
```

| Request field | Default | Notes |
| --- | --- | --- |
| `text` | — | Required, 1–16000 characters. |
| `groups` | `["*"]` | Groups to compare with, as in `classifier.similarityGroups`; `[]` compares with none. |
| `examples` | `[]` | Up to 100 unsaved positive texts (1–4000 characters each), compared as an extra group named `examples`, with weight 1 and the default threshold. |
| `limit` | `10` | Samples listed per group, 1–50. |
| `includeBelowThreshold` | `false` | Also list the closest samples below their threshold. |

Each entry in `groups` (highest `score` first, only groups that exist) has `matched`, the best positive's `score` and `threshold`, `suppressedBy` (the negative sample that blocked a match, else `null`), and `matches`. Each match has `matchType` (`near_duplicate` or `semantic`), `distance` (SimHash bits that differ: `0` for the same text, about 32 for unrelated texts), `aboveThreshold`, and `span`, the UTF-8 **byte** offsets of the part of your text that matched best (`null` for near-duplicates). Stored thresholds are 32-bit floats, so `0.66` comes back as `0.6600000262260437`.

Changing one word in a 440-character message gives a near-duplicate:

```json
{ "score": 1.0, "matchType": "near_duplicate", "distance": 3, "aboveThreshold": true, "span": null, … }
```

Semantic matching can be unavailable while near-duplicate matching still runs. The response is then `200` with `error` set to `EMBEDDINGS_DISABLED` (`provider = "none"`) or `EMBEDDING_UNAVAILABLE` (model not loaded yet, busy past `[similarity] queue_timeout`, or failed), and `modelId` is `null` without a model. See the [similarity API reference](/sentinel-api-docs-v2.html#tag/similarity/POST/v2/similarity).

<Callout type="warning" title="Training texts are readable with API keys">

`POST /v2/similarity` returns the texts of matching samples. Anyone holding a `restricted` or `full` key of the account can read its training data through it.

`/v2/similarity` and classification use the same matching, but classification compares only the groups the security group selects, reports only matched groups, follows `disableRules`, and needs weight ≥ 1 to make the verdict `BAD` on its own. A text that matches here but not in classification usually means one of those differs.

## Tuning

### Thresholds

Cosine similarities depend on the model, so pick thresholds from your data rather than relying on the default `0.80`. With the default local model, rewordings of a message often score around 0.6–0.75. `_evaluate` suggests a threshold for a group from its own samples (**Training Data → Groups → Evaluate thresholds**):

```bash
curl https://sentinel.example.com/v2/training-data/_evaluate \
-H 'Authorization: Bearer <jwt>' \
-H 'Content-Type: application/json' \
-d '{"accountId": "acc_…", "group": "crypto_scam"}'
```

```json
{
"data": {
"currentThreshold": 0.8,
"curve": [
  { "precision": 0.7778, "recall": 0.7, "threshold": 0.5 },
  { "precision": 0.8333, "recall": 0.5, "threshold": 0.55 },
  { "precision": 1.0, "recall": 0.5, "threshold": 0.6 },
  …
  { "precision": 1.0, "recall": 0.2, "threshold": 0.8 },
  …
],
"group": "crypto_scam",
"negatives": 2,
"pending": 0,
"positives": 10,
"precision": 1.0,
"recall": 0.5,
"suggestedThreshold": 0.66
}
}
```

Each embedded positive is scored against the group's **other** positives (leave-one-out), standing in for a new message of the same kind, and each embedded negative against the positives. Across thresholds 0.50–0.99, **recall** is the share of positives that would match and **precision** the share of matches that are positives. `suggestedThreshold` has the best F0.5 score, which values precision twice as much as recall; without negatives, it's the highest threshold that keeps recall at 0.9 or above. `curve` shows steps of 0.05.

`_evaluate` changes nothing. To apply a value, set the `threshold` of the group's positive samples: **Apply threshold** in the dialog does that, or `PATCH` each sample. The call needs at least 3 embedded positives (`400 INSUFFICIENT_SAMPLES`, also for a group that doesn't exist) and an embedding model (`400 EMBEDDINGS_DISABLED`).

### False positives and misses

| What you see in `/v2/similarity` | What to do |
| --- | --- |
| A group is missing from `groups` | Check the name in `_groups`; name `builtin:` groups explicitly. |
| `matches: []` | Send `includeBelowThreshold: true` to see the closest samples; check `error` and `pending`. |
| High `score`, `matched: false`, `suppressedBy: null` | The best positive is below its threshold: lower it, guided by `_evaluate`. |
| `matched: false` with `suppressedBy` set | A negative is about as close as the best positive. Add the text as a positive if it's spam, or fix the negative. |
| `matched: true` for a legitimate text | Add it as a negative sample (for example with `_feedback`) or raise the threshold. |

More positives and realistic negatives give better suggestions; add the legitimate messages that you see falsely matched as negatives. To watch a new group without affecting verdicts, give its samples weight `0`, or use a weight below `1` for groups that should raise suspicion without deciding alone.

## Limitations

- Near-duplicate matching is unreliable for one-sentence texts; use semantic matching for short messages.
- Without an embedding model, only near-duplicates and campaigns are detected. With a model, a new or changed sample matches only as a near-duplicate until it's embedded.
- Text beyond `max_chunks` chunks (about 8000 characters by default) can only match as a near-duplicate.
- Other replicas pick up sample changes through the KV store's pub/sub within moments, and at the latest after `[similarity] sync_interval` (default `5m`). The duplicate checks of `_import`, `_feedback`, and accepting a suggestion use the handling replica's view, so a sample written on another replica a moment earlier can be missed.
- Campaign counting is approximate, and `/v2/classifier` calls without `ip` inflate it.

## Privacy

Samples and suggestions store message text verbatim in the database, and texts can contain personal data. Suggestions are deleted after `[similarity.suggestions] retention`; samples are kept until you delete them, and `DELETE` removes them for good. With `provider = "openai"`, the text of every sample and every classified text is sent to the configured embeddings API; with `local`, nothing leaves the server. See [Privacy & data handling](/docs/sentinel/v2/security/privacy/).

## Related

- [Classifier](/docs/sentinel/v2/features/classifier/) and [Classifier rules & scores](/docs/sentinel/v2/reference/classifier-rules/#similarity)
- [Spam, profanity & injection](/docs/sentinel/v2/features/text-analysis/), which feeds training suggestions
- [AI decider](/docs/sentinel/v2/features/ai-decider/)
- [`classifier.similarityGroups`](/docs/sentinel/v2/reference/rules/#classifier) in the Rules reference
- [`[similarity]`](/docs/sentinel/v2/reference/configuration/#similarity) in the Configuration reference
- [Training data errors](/docs/sentinel/v2/reference/errors/#training-data)
- API reference: [training data](/sentinel-api-docs-v2.html#tag/training-data/POST/v2/training-data), [training suggestions](/sentinel-api-docs-v2.html#tag/training-suggestions/POST/v2/training-suggestions/_search), [similarity](/sentinel-api-docs-v2.html#tag/similarity/POST/v2/similarity)
