Similarity & training data
Similarity compares submitted text with example messages you label for your account, so a spam message that you've seen once is caught again when it's reworded, translated, or lightly edited. It also flags the same message arriving from many client IPs, with no training data at all.
You manage training data with an admin session (the training_data permission feature) or from your backend with a restricted or full API key. Semantic matching also needs an embedding model on the server.
How it works
The classifier runs three similarity rules on the text of POST /v2/classifier, POST /v2/verify (the widget's form fields), and POST /v2/eml:
| Rule | Detects | Needs |
|---|---|---|
SIMILARITY | The text means the same as a training sample, even when worded differently or in another language. | Training data and an embedding model |
NEAR_DUPLICATE | The text is a training sample with small edits, such as one changed word. | Training data |
CAMPAIGN | The same message (near-duplicates included) arrives from many client IPs within a time window. | Nothing |
Groups and samples
A sample is one text (1–4000 characters) with a label:
positivesamples are what the group catches.negativesamples are look-alikes that must not match, for example a legitimate "How do I buy bitcoin on your exchange?" next to crypto-scam positives.
A group is a named set of samples in one account, such as crypto_scam or job_offers. Group names are 1–64 characters a-z, 0-9, _ and -, starting with a letter or digit; they are trimmed and lowercased. Groups have no record of their own: a group exists while it has samples. An account holds at most [similarity] max_samples_per_account samples (default 10000).
Each sample also has:
- a weight from
0to10(default1), which scales the rule score of a match. A match on a sample with weight1or more makes the classificationBAD; lower weights only add to the score, and0records the match without affecting the verdict. - an optional threshold from
0to1: the cosine similarity a text needs to match the sample semantically. Without one,[similarity] thresholdapplies (default0.80). - a source:
manual,import,feedback, orsuggestion, depending on how it was added. embedded:trueonce the sample has an embedding of the server's current model.
How a group matches
Every sample in a group gets a score against the incoming text:
1when the text is a near-duplicate: the 64-bit SimHash fingerprints of the two texts differ in at most[similarity] near_duplicate_distancebits (default3). Texts under 12 canonical characters must be identical instead.- Otherwise, when both are embedded, the cosine similarity of their embeddings: about
0for unrelated texts,1for the same meaning. Texts longer than about 1000 characters are split into chunks (up to[similarity] max_chunks, default8), and the best chunk counts.
The group matches when its best positive sample reaches its threshold and beats the best negative sample's score by [similarity] negative_margin (default 0.05). A near-duplicate always passes its threshold, so a negative near-duplicate suppresses every positive in its group. Among positives with the same score, the higher weight wins.
Near-duplicate matching suits messages of a few hundred characters. In a single sentence, one changed word usually moves the fingerprint by more than 3 bits, so short texts match as near-duplicates only when they are identical after canonicalization. Use semantic matching for short texts.
Canonical text
Before comparing, Sentinel normalizes each text. Texts with the same canonical form count as identical for near-duplicate matching and duplicate checks, and embedding models receive the canonical text:
- Unicode compatibility forms are folded (
free→free) and the text is lowercased. - Invisible characters, such as zero-width spaces, are removed.
- In text that mixes Latin with Cyrillic or Greek letters, look-alike letters become Latin (
сryptowith a Cyrillicс→crypto). - In words with an ASCII letter and at least as many letters as digits or
@ $ € |, those characters are read as letters (fr33→free).!and+are never folded, and mostly numeric words such as4x4stay as they are. - Runs of three or more equal characters shrink to two (
!!!!→!!), and whitespace is collapsed.
Embedding providers
[similarity] provider chooses the model that computes embeddings:
provider | Model |
|---|---|
none (default) | No semantic matching. NEAR_DUPLICATE, CAMPAIGN, sample management, and suggestions still work. |
local | An ONNX model that runs inside Sentinel; by default a multilingual MiniLM (about 135 MB download). |
openai | An OpenAI-compatible /embeddings API: OpenAI, or a self-hosted Ollama, vLLM, or TEI. |
Sentinel embeds samples in the background, in batches of 64, usually within seconds of a create or text change. Until then a sample can only match as a near-duplicate. A different model (another provider, model file, model name, or dimensions) is a different vector space, so every sample is embedded again after a change. The local model downloads at startup; until it has loaded, SIMILARITY scores null.
Campaigns
CAMPAIGN counts how many distinct client IPs sent the same message to one account within [similarity.campaign] window (default 1h):
- Below
min_sources(default5) distinct sources, it scores0. From there it scores5 × sources / min_sources, at most10: with the defaults, 5 sources score5and 10 sources score10. - The source is the client IP for
/v2/verify, the body'sipfor/v2/classifier, and the sending server's IP for/v2/eml. A/v2/classifiercall withoutipcounts as a new source every time, so sendipwhenever you have it. - All endpoints of one account count together.
- Texts under
min_charscanonical characters (default30) aren't counted, andCAMPAIGNscoresnullfor them. Short messages like "thanks!" are too common to signal a campaign. - Counting is approximate: near-duplicates are found through parts of their fingerprint, so occasionally unrelated texts that share a part are counted together. The counters live in the KV store, shared by all replicas, with up to 8 short-lived keys per distinct message.
Built-in groups
The operator can load read-only groups that every account can use, from a CSV file you host yourself (the ALTCHA directory doesn't publish one). They are named builtin: plus the CSV's group value, for example builtin:crypto_scam, and their sample ids are builtin:<group>:<line>. Built-in groups don't appear in the training-data API and aren't included in *: name them explicitly. Account group names can't contain :, so crypto_scam and builtin:crypto_scam never collide.
Enabling it
Server configuration
Matching against training data, near-duplicates, and campaigns work without configuration. For semantic matching, choose an embedding provider and restart:
[similarity]
provider = "local" # ONNX model, ~135 MB download at startup
threshold = 0.80 # default for samples without their own threshold
[similarity.campaign]
min_sources = 5 # distinct client IPs within the window
[similarity.suggestions]
enabled = true # stores message text; see Privacy belowFor an OpenAI-compatible API instead:
[similarity]
provider = "openai"
[similarity.openai]
api_key = "${OPENAI_API_KEY}"
model = "text-embedding-3-small"For built-in groups, set [similarity.builtin] enabled = true and url to your CSV (group,label,text,weight header, plain or .csv.gz). Sentinel downloads it as the remote file similarity-groups, by default @weekly.
Every replica must use the same model. Each key is in the Configuration reference: [similarity], [similarity.local], [similarity.openai], [similarity.campaign], [similarity.builtin], and [similarity.suggestions]. Doctor → Run checks reports the Embedding model state, and GET /v2/cluster/replicas shows each replica's provider, model, sample counts, and last embedding error.
Choosing groups
By default the classifier compares text with every group of the API key's account (*). A security group's classifier.similarityGroups set field narrows or extends that list: *, group names, and builtin: groups, up to 50, separated by ,, ;, or whitespace. Unknown names are skipped; an empty value selects no group, so only CAMPAIGN runs.
[
{
"name": "similarity",
"action": "set",
"set": [{ "field": "classifier.similarityGroups", "value": "crypto_scam, builtin:crypto_scam" }]
},
{
"name": "comments",
"action": "set",
"conditions": [{ "field": "origin", "operator": "=", "value": ["https://forum.example.com"] }],
"set": [{ "field": "classifier.similarityGroups", "value": "comment_spam crypto_scam" }]
}
]A later matching set rule overrides an earlier one, so the second rule switches groups for the forum. An invalid name is rejected with 400 INVALID_RULES (invalid group name 'bad/name'). See Classifier set fields.
Per request, the POST /v2/classifier body's similarityGroups array (for example ["*", "builtin:crypto_scam"]) and the X-Similarity-Groups header of POST /v2/eml replace the group's list. On /v2/verify, the widget must send its form fields; see Verifying submissions.
Managing training data
In the admin UI, open Training Data. The Samples tab lists, searches, exports, creates (Create sample), and imports (Import) samples; Groups shows each group's positive, negative, and pending counts and has Evaluate thresholds; Suggestions is the review queue. Every action has an API equivalent under /v2/training-data with an admin session. See the training-data API reference.
| Call | Does |
|---|---|
POST /v2/training-data | Creates a sample (accountId, group, text, optional label, weight, threshold). |
PATCH /v2/training-data/{id} | Updates a sample. threshold: null clears the threshold; a changed text is embedded again. |
DELETE /v2/training-data/{id} | Deletes a sample permanently. |
POST /v2/training-data/_search | Lists samples (filter.accountId required; group, label, search). |
POST /v2/training-data/_groups | Per group: positives, negatives, and pending (samples not yet embedded). |
POST /v2/training-data/_import | Adds up to 1000 texts to one group. |
POST /v2/training-data/_evaluate | Suggests a threshold for a group. See Tuning. |
POST /v2/training-data/_feedback | Adds a sample from your backend, with an API key. |
Create and update don't check for duplicates. Import, feedback, and accepting a suggestion do.
Bulk import
_import adds texts to one group in a single all-or-nothing call; label, weight, and threshold apply to all of them:
curl https://sentinel.example.com/v2/training-data/_import \
-H 'Authorization: Bearer <jwt>' \
-H 'Content-Type: application/json' \
-d '{
"accountId": "acc_…",
"group": "crypto_scam",
"label": "positive",
"texts": [
"Double your bitcoin in 24 hours. Guaranteed returns, no risk!",
"Our AI trading bot made me $4,000 last week. DM me to join.",
" ",
"our ai trading bot made me $4,000 last week. dm me to join."
]
}'{"data":{"created":1,"ids":["trd_…"],"skipped":3}}Blank texts are skipped, and so is any text whose canonical form equals an existing sample in the group or an earlier text in the request. Here the first text already existed and the last differs from the second only in case. A text over 4000 characters fails the whole request with 400 VALIDATION_ERROR, and an import that would exceed the sample limit fails with 400 TRAINING_DATA_LIMIT; nothing is stored in either case.
Feedback from your application
_feedback turns a moderator's decision into a sample, for example a "This is spam" button or restoring a false positive. Call it from your backend with a restricted or full key; the sample goes into the key's account, and the key's security-group rules apply:
curl https://sentinel.example.com/v2/training-data/_feedback \
-u 'key_…:sec_…' \
-H 'Content-Type: application/json' \
-d '{"group": "crypto_scam", "text": "Bitcoin doubler: send any amount and get twice as much back within 24 hours. Act fast!", "label": "positive"}'{"data":{"duplicate":false,"id":"trd_…"}}New samples get weight 1, no threshold, and source: "feedback". If the group already has a near-duplicate with the same label, nothing is added and the response returns its id with duplicate: true, so repeated clicks are harmless. A near-duplicate with the opposite label returns 409 LABEL_CONFLICT, naming the conflicting sample:
{"code":"LABEL_CONFLICT","error":"Sample 'trd_…' has the opposite label.","statusCode":409}Decide which label is right, then fix or delete that sample. A public key gets 403 NON_PUBLIC_API_KEY_REQUIRED, and a session token 401 API_KEY_REQUIRED.
Training suggestions
With [similarity.suggestions] enabled = true (off by default), Sentinel queues texts that the built-in text rules find spam, so a campaign that the word lists catch today becomes training data that catches its rewordings tomorrow. A text is captured when:
- it was classified for an API key, and
- the text category alone is
BAD(text.score≥ 2); other categories neither cause nor prevent a capture, and - it isn't a near-duplicate of any sample in the account, in any group, with either label.
A near-duplicate of a queued suggestion increases that suggestion's hits instead (within about 10 seconds). When the account has max_per_account suggestions (default 1000, counted per replica), new texts are dropped. Suggestions are deleted after retention (default 7d).
Review them under Training Data → Suggestions: Accept adds the text to a group as a sample, and deleting removes it from the queue. With the API:
curl https://sentinel.example.com/v2/training-suggestions/trs_…/_accept \
-H 'Authorization: Bearer <jwt>' \
-H 'Content-Type: application/json' \
-d '{"group": "pharma", "label": "positive"}'{
"data": {
"sample": {
"id": "trd_…",
"group": "pharma",
"text": "Buy viagra now, cheap pills, discreet shipping worldwide, no prescription needed! Casino bonus free spins.",
"label": "positive",
"weight": 1.0,
"threshold": null,
"source": "suggestion",
…
},
"duplicate": false
}
}Accept with label: "negative" to record a false positive of the text rules for this account. Duplicate and conflict handling are the same as for _feedback. A suggestion that someone else already accepted or deleted returns 404 NOT_FOUND. List the queue with POST /v2/training-suggestions/_search (sortBy: createdAt or hits); see the training-suggestions API reference.
Rules & signals
| Rule | Meaning |
|---|---|
SIMILARITY | The text matches a selected group semantically. null when no selected group has embedded samples or the text couldn't be embedded. |
NEAR_DUPLICATE | The text is a near-duplicate of a positive sample in a selected group. null when the selected groups have no samples. |
CAMPAIGN | At least min_sources distinct sources sent this message within the window. null for short texts or with campaigns off. |
A match on a sample with weight 1 or more raises the classification to BAD. The classification's similarity object lists the matched groups (group, sampleId, matchType, score, weight), campaign (sources, windowSec, or null when the check didn't run), and error (EMBEDDING_UNAVAILABLE when semantic matching was skipped). Scores, the category formula, and floors are in Classifier rules & scores. Turn rules off with classifier.disableRules or the body's disableRules, for example ["CAMPAIGN"] for test traffic.
An excerpt of a POST /v2/classifier response for a reworded scam message:
"similarity": {
"matches": [
{ "group": "crypto_scam", "sampleId": "trd_…", "matchType": "semantic", "score": 0.9804, "weight": 1.0 }
],
"campaign": { "sources": 1, "windowSec": 3600 },
"error": null,
"rules": {
"SIMILARITY": { "score": 4.9 },
"NEAR_DUPLICATE": { "score": 0.0 },
"CAMPAIGN": { "score": 0.0 }
},
"score": 4.9,
"time": 5.387,
"triggeredRules": ["SIMILARITY"]
}Using the API
POST /v2/similarity explains how a text compares with training data: which samples it resembles, how closely, and why each group matched or didn't. It's read-only: it doesn't classify, count campaigns, capture suggestions, or write request logs. Call it with a restricted or full key:
curl https://sentinel.example.com/v2/similarity \
-u 'key_…:sec_…' \
-H 'Content-Type: application/json' \
-d '{"text": "Give us your bitcoin and our trading bot doubles it in a day, guaranteed.", "groups": ["*"], "limit": 3}'{
"data": {
"modelId": "local:d6ea442ff6a891da:mean",
"error": null,
"time": 4.062,
"groups": [
{
"group": "crypto_scam",
"matched": true,
"score": 0.7563,
"threshold": 0.6600000262260437,
"suppressedBy": null,
"matches": [
{
"sampleId": "trd_…",
"text": "Our crypto bot doubles your deposit in one day, zero risk, guaranteed profit.",
"label": "positive",
"score": 0.7563,
"matchType": "semantic",
"distance": 22,
"aboveThreshold": true,
"span": { "start": 0, "end": 73 }
},
…
]
}
]
}
}| Request field | Default | Notes |
|---|---|---|
text | — | Required, 1–16000 characters. |
groups | ["*"] | Groups to compare with, as in classifier.similarityGroups; [] compares with none. |
examples | [] | Up to 100 unsaved positive texts (1–4000 characters each), compared as an extra group named examples, with weight 1 and the default threshold. |
limit | 10 | Samples listed per group, 1–50. |
includeBelowThreshold | false | Also list the closest samples below their threshold. |
Each entry in groups (highest score first, only groups that exist) has matched, the best positive's score and threshold, suppressedBy (the negative sample that blocked a match, else null), and matches. Each match has matchType (near_duplicate or semantic), distance (SimHash bits that differ: 0 for the same text, about 32 for unrelated texts), aboveThreshold, and span, the UTF-8 byte offsets of the part of your text that matched best (null for near-duplicates). Stored thresholds are 32-bit floats, so 0.66 comes back as 0.6600000262260437.
Changing one word in a 440-character message gives a near-duplicate:
{ "score": 1.0, "matchType": "near_duplicate", "distance": 3, "aboveThreshold": true, "span": null, … }Semantic matching can be unavailable while near-duplicate matching still runs. The response is then 200 with error set to EMBEDDINGS_DISABLED (provider = "none") or EMBEDDING_UNAVAILABLE (model not loaded yet, busy past [similarity] queue_timeout, or failed), and modelId is null without a model. See the similarity API reference.
/v2/similarity and classification use the same matching, but classification compares only the groups the security group selects, reports only matched groups, follows disableRules, and needs weight ≥ 1 to make the verdict BAD on its own. A text that matches here but not in classification usually means one of those differs.
Tuning
Thresholds
Cosine similarities depend on the model, so pick thresholds from your data rather than relying on the default 0.80. With the default local model, rewordings of a message often score around 0.6–0.75. _evaluate suggests a threshold for a group from its own samples (Training Data → Groups → Evaluate thresholds):
curl https://sentinel.example.com/v2/training-data/_evaluate \
-H 'Authorization: Bearer <jwt>' \
-H 'Content-Type: application/json' \
-d '{"accountId": "acc_…", "group": "crypto_scam"}'{
"data": {
"currentThreshold": 0.8,
"curve": [
{ "precision": 0.7778, "recall": 0.7, "threshold": 0.5 },
{ "precision": 0.8333, "recall": 0.5, "threshold": 0.55 },
{ "precision": 1.0, "recall": 0.5, "threshold": 0.6 },
…
{ "precision": 1.0, "recall": 0.2, "threshold": 0.8 },
…
],
"group": "crypto_scam",
"negatives": 2,
"pending": 0,
"positives": 10,
"precision": 1.0,
"recall": 0.5,
"suggestedThreshold": 0.66
}
}Each embedded positive is scored against the group's other positives (leave-one-out), standing in for a new message of the same kind, and each embedded negative against the positives. Across thresholds 0.50–0.99, recall is the share of positives that would match and precision the share of matches that are positives. suggestedThreshold has the best F0.5 score, which values precision twice as much as recall; without negatives, it's the highest threshold that keeps recall at 0.9 or above. curve shows steps of 0.05.
_evaluate changes nothing. To apply a value, set the threshold of the group's positive samples: Apply threshold in the dialog does that, or PATCH each sample. The call needs at least 3 embedded positives (400 INSUFFICIENT_SAMPLES, also for a group that doesn't exist) and an embedding model (400 EMBEDDINGS_DISABLED).
False positives and misses
What you see in /v2/similarity | What to do |
|---|---|
A group is missing from groups | Check the name in _groups; name builtin: groups explicitly. |
matches: [] | Send includeBelowThreshold: true to see the closest samples; check error and pending. |
High score, matched: false, suppressedBy: null | The best positive is below its threshold: lower it, guided by _evaluate. |
matched: false with suppressedBy set | A negative is about as close as the best positive. Add the text as a positive if it's spam, or fix the negative. |
matched: true for a legitimate text | Add it as a negative sample (for example with _feedback) or raise the threshold. |
More positives and realistic negatives give better suggestions; add the legitimate messages that you see falsely matched as negatives. To watch a new group without affecting verdicts, give its samples weight 0, or use a weight below 1 for groups that should raise suspicion without deciding alone.
Limitations
- Near-duplicate matching is unreliable for one-sentence texts; use semantic matching for short messages.
- Without an embedding model, only near-duplicates and campaigns are detected. With a model, a new or changed sample matches only as a near-duplicate until it's embedded.
- Text beyond
max_chunkschunks (about 8000 characters by default) can only match as a near-duplicate. - Other replicas pick up sample changes through the KV store's pub/sub within moments, and at the latest after
[similarity] sync_interval(default5m). The duplicate checks of_import,_feedback, and accepting a suggestion use the handling replica's view, so a sample written on another replica a moment earlier can be missed. - Campaign counting is approximate, and
/v2/classifiercalls withoutipinflate it.
Privacy
Samples and suggestions store message text verbatim in the database, and texts can contain personal data. Suggestions are deleted after [similarity.suggestions] retention; samples are kept until you delete them, and DELETE removes them for good. With provider = "openai", the text of every sample and every classified text is sent to the configured embeddings API; with local, nothing leaves the server. See Privacy & data handling.
Related
- Classifier and Classifier rules & scores
- Spam, profanity & injection, which feeds training suggestions
- AI decider
classifier.similarityGroupsin the Rules reference[similarity]in the Configuration reference- Training data errors
- API reference: training data, training suggestions, similarity