Skip to main content
ALTCHA®

Similarity & training data

Similarity compares submitted text with example messages you label for your account, so a spam message that you've seen once is caught again when it's reworded, translated, or lightly edited. It also flags the same message arriving from many client IPs, with no training data at all.

You manage training data with an admin session (the training_data permission feature) or from your backend with a restricted or full API key. Semantic matching also needs an embedding model on the server.

How it works

The classifier runs three similarity rules on the text of POST /v2/classifier, POST /v2/verify (the widget's form fields), and POST /v2/eml:

RuleDetectsNeeds
SIMILARITYThe text means the same as a training sample, even when worded differently or in another language.Training data and an embedding model
NEAR_DUPLICATEThe text is a training sample with small edits, such as one changed word.Training data
CAMPAIGNThe same message (near-duplicates included) arrives from many client IPs within a time window.Nothing

Groups and samples

A sample is one text (1–4000 characters) with a label:

  • positive samples are what the group catches.
  • negative samples are look-alikes that must not match, for example a legitimate "How do I buy bitcoin on your exchange?" next to crypto-scam positives.

A group is a named set of samples in one account, such as crypto_scam or job_offers. Group names are 1–64 characters a-z, 0-9, _ and -, starting with a letter or digit; they are trimmed and lowercased. Groups have no record of their own: a group exists while it has samples. An account holds at most [similarity] max_samples_per_account samples (default 10000).

Each sample also has:

  • a weight from 0 to 10 (default 1), which scales the rule score of a match. A match on a sample with weight 1 or more makes the classification BAD; lower weights only add to the score, and 0 records the match without affecting the verdict.
  • an optional threshold from 0 to 1: the cosine similarity a text needs to match the sample semantically. Without one, [similarity] threshold applies (default 0.80).
  • a source: manual, import, feedback, or suggestion, depending on how it was added.
  • embedded: true once the sample has an embedding of the server's current model.

How a group matches

Every sample in a group gets a score against the incoming text:

  • 1 when the text is a near-duplicate: the 64-bit SimHash fingerprints of the two texts differ in at most [similarity] near_duplicate_distance bits (default 3). Texts under 12 canonical characters must be identical instead.
  • Otherwise, when both are embedded, the cosine similarity of their embeddings: about 0 for unrelated texts, 1 for the same meaning. Texts longer than about 1000 characters are split into chunks (up to [similarity] max_chunks, default 8), and the best chunk counts.

The group matches when its best positive sample reaches its threshold and beats the best negative sample's score by [similarity] negative_margin (default 0.05). A near-duplicate always passes its threshold, so a negative near-duplicate suppresses every positive in its group. Among positives with the same score, the higher weight wins.

Near-duplicate matching suits messages of a few hundred characters. In a single sentence, one changed word usually moves the fingerprint by more than 3 bits, so short texts match as near-duplicates only when they are identical after canonicalization. Use semantic matching for short texts.

Canonical text

Before comparing, Sentinel normalizes each text. Texts with the same canonical form count as identical for near-duplicate matching and duplicate checks, and embedding models receive the canonical text:

  • Unicode compatibility forms are folded (free → free) and the text is lowercased.
  • Invisible characters, such as zero-width spaces, are removed.
  • In text that mixes Latin with Cyrillic or Greek letters, look-alike letters become Latin (сrypto with a Cyrillic с → crypto).
  • In words with an ASCII letter and at least as many letters as digits or @ $ € |, those characters are read as letters (fr33 → free). ! and + are never folded, and mostly numeric words such as 4x4 stay as they are.
  • Runs of three or more equal characters shrink to two (!!!! → !!), and whitespace is collapsed.

Embedding providers

[similarity] provider chooses the model that computes embeddings:

providerModel
none (default)No semantic matching. NEAR_DUPLICATE, CAMPAIGN, sample management, and suggestions still work.
localAn ONNX model that runs inside Sentinel; by default a multilingual MiniLM (about 135 MB download).
openaiAn OpenAI-compatible /embeddings API: OpenAI, or a self-hosted Ollama, vLLM, or TEI.

Sentinel embeds samples in the background, in batches of 64, usually within seconds of a create or text change. Until then a sample can only match as a near-duplicate. A different model (another provider, model file, model name, or dimensions) is a different vector space, so every sample is embedded again after a change. The local model downloads at startup; until it has loaded, SIMILARITY scores null.

Campaigns

CAMPAIGN counts how many distinct client IPs sent the same message to one account within [similarity.campaign] window (default 1h):

  • Below min_sources (default 5) distinct sources, it scores 0. From there it scores 5 × sources / min_sources, at most 10: with the defaults, 5 sources score 5 and 10 sources score 10.
  • The source is the client IP for /v2/verify, the body's ip for /v2/classifier, and the sending server's IP for /v2/eml. A /v2/classifier call without ip counts as a new source every time, so send ip whenever you have it.
  • All endpoints of one account count together.
  • Texts under min_chars canonical characters (default 30) aren't counted, and CAMPAIGN scores null for them. Short messages like "thanks!" are too common to signal a campaign.
  • Counting is approximate: near-duplicates are found through parts of their fingerprint, so occasionally unrelated texts that share a part are counted together. The counters live in the KV store, shared by all replicas, with up to 8 short-lived keys per distinct message.

Built-in groups

The operator can load read-only groups that every account can use, from a CSV file you host yourself (the ALTCHA directory doesn't publish one). They are named builtin: plus the CSV's group value, for example builtin:crypto_scam, and their sample ids are builtin:<group>:<line>. Built-in groups don't appear in the training-data API and aren't included in *: name them explicitly. Account group names can't contain :, so crypto_scam and builtin:crypto_scam never collide.

Enabling it

Server configuration

Matching against training data, near-duplicates, and campaigns work without configuration. For semantic matching, choose an embedding provider and restart:

[similarity]
provider  = "local"   # ONNX model, ~135 MB download at startup
threshold = 0.80      # default for samples without their own threshold

[similarity.campaign]
min_sources = 5       # distinct client IPs within the window

[similarity.suggestions]
enabled = true        # stores message text; see Privacy below

For an OpenAI-compatible API instead:

[similarity]
provider = "openai"

[similarity.openai]
api_key = "${OPENAI_API_KEY}"
model   = "text-embedding-3-small"

For built-in groups, set [similarity.builtin] enabled = true and url to your CSV (group,label,text,weight header, plain or .csv.gz). Sentinel downloads it as the remote file similarity-groups, by default @weekly.

Every replica must use the same model. Each key is in the Configuration reference: [similarity], [similarity.local], [similarity.openai], [similarity.campaign], [similarity.builtin], and [similarity.suggestions]. Doctor → Run checks reports the Embedding model state, and GET /v2/cluster/replicas shows each replica's provider, model, sample counts, and last embedding error.

Choosing groups

By default the classifier compares text with every group of the API key's account (*). A security group's classifier.similarityGroups set field narrows or extends that list: *, group names, and builtin: groups, up to 50, separated by ,, ;, or whitespace. Unknown names are skipped; an empty value selects no group, so only CAMPAIGN runs.

[
  {
    "name": "similarity",
    "action": "set",
    "set": [{ "field": "classifier.similarityGroups", "value": "crypto_scam, builtin:crypto_scam" }]
  },
  {
    "name": "comments",
    "action": "set",
    "conditions": [{ "field": "origin", "operator": "=", "value": ["https://forum.example.com"] }],
    "set": [{ "field": "classifier.similarityGroups", "value": "comment_spam crypto_scam" }]
  }
]

A later matching set rule overrides an earlier one, so the second rule switches groups for the forum. An invalid name is rejected with 400 INVALID_RULES (invalid group name 'bad/name'). See Classifier set fields.

Per request, the POST /v2/classifier body's similarityGroups array (for example ["*", "builtin:crypto_scam"]) and the X-Similarity-Groups header of POST /v2/eml replace the group's list. On /v2/verify, the widget must send its form fields; see Verifying submissions.

Managing training data

In the admin UI, open Training Data. The Samples tab lists, searches, exports, creates (Create sample), and imports (Import) samples; Groups shows each group's positive, negative, and pending counts and has Evaluate thresholds; Suggestions is the review queue. Every action has an API equivalent under /v2/training-data with an admin session. See the training-data API reference.

CallDoes
POST /v2/training-dataCreates a sample (accountId, group, text, optional label, weight, threshold).
PATCH /v2/training-data/{id}Updates a sample. threshold: null clears the threshold; a changed text is embedded again.
DELETE /v2/training-data/{id}Deletes a sample permanently.
POST /v2/training-data/_searchLists samples (filter.accountId required; group, label, search).
POST /v2/training-data/_groupsPer group: positives, negatives, and pending (samples not yet embedded).
POST /v2/training-data/_importAdds up to 1000 texts to one group.
POST /v2/training-data/_evaluateSuggests a threshold for a group. See Tuning.
POST /v2/training-data/_feedbackAdds a sample from your backend, with an API key.

Create and update don't check for duplicates. Import, feedback, and accepting a suggestion do.

Bulk import

_import adds texts to one group in a single all-or-nothing call; label, weight, and threshold apply to all of them:

curl https://sentinel.example.com/v2/training-data/_import \
  -H 'Authorization: Bearer <jwt>' \
  -H 'Content-Type: application/json' \
  -d '{
    "accountId": "acc_…",
    "group": "crypto_scam",
    "label": "positive",
    "texts": [
      "Double your bitcoin in 24 hours. Guaranteed returns, no risk!",
      "Our AI trading bot made me $4,000 last week. DM me to join.",
      "  ",
      "our ai trading bot made me $4,000 last week. dm me to join."
    ]
  }'
{"data":{"created":1,"ids":["trd_…"],"skipped":3}}

Blank texts are skipped, and so is any text whose canonical form equals an existing sample in the group or an earlier text in the request. Here the first text already existed and the last differs from the second only in case. A text over 4000 characters fails the whole request with 400 VALIDATION_ERROR, and an import that would exceed the sample limit fails with 400 TRAINING_DATA_LIMIT; nothing is stored in either case.

Feedback from your application

_feedback turns a moderator's decision into a sample, for example a "This is spam" button or restoring a false positive. Call it from your backend with a restricted or full key; the sample goes into the key's account, and the key's security-group rules apply:

curl https://sentinel.example.com/v2/training-data/_feedback \
  -u 'key_…:sec_…' \
  -H 'Content-Type: application/json' \
  -d '{"group": "crypto_scam", "text": "Bitcoin doubler: send any amount and get twice as much back within 24 hours. Act fast!", "label": "positive"}'
{"data":{"duplicate":false,"id":"trd_…"}}

New samples get weight 1, no threshold, and source: "feedback". If the group already has a near-duplicate with the same label, nothing is added and the response returns its id with duplicate: true, so repeated clicks are harmless. A near-duplicate with the opposite label returns 409 LABEL_CONFLICT, naming the conflicting sample:

{"code":"LABEL_CONFLICT","error":"Sample 'trd_…' has the opposite label.","statusCode":409}

Decide which label is right, then fix or delete that sample. A public key gets 403 NON_PUBLIC_API_KEY_REQUIRED, and a session token 401 API_KEY_REQUIRED.

Training suggestions

With [similarity.suggestions] enabled = true (off by default), Sentinel queues texts that the built-in text rules find spam, so a campaign that the word lists catch today becomes training data that catches its rewordings tomorrow. A text is captured when:

  • it was classified for an API key, and
  • the text category alone is BAD (text.score ≥ 2); other categories neither cause nor prevent a capture, and
  • it isn't a near-duplicate of any sample in the account, in any group, with either label.

A near-duplicate of a queued suggestion increases that suggestion's hits instead (within about 10 seconds). When the account has max_per_account suggestions (default 1000, counted per replica), new texts are dropped. Suggestions are deleted after retention (default 7d).

Review them under Training Data → Suggestions: Accept adds the text to a group as a sample, and deleting removes it from the queue. With the API:

curl https://sentinel.example.com/v2/training-suggestions/trs_…/_accept \
  -H 'Authorization: Bearer <jwt>' \
  -H 'Content-Type: application/json' \
  -d '{"group": "pharma", "label": "positive"}'
{
  "data": {
    "sample": {
      "id": "trd_…",
      "group": "pharma",
      "text": "Buy viagra now, cheap pills, discreet shipping worldwide, no prescription needed! Casino bonus free spins.",
      "label": "positive",
      "weight": 1.0,
      "threshold": null,
      "source": "suggestion",
      …
    },
    "duplicate": false
  }
}

Accept with label: "negative" to record a false positive of the text rules for this account. Duplicate and conflict handling are the same as for _feedback. A suggestion that someone else already accepted or deleted returns 404 NOT_FOUND. List the queue with POST /v2/training-suggestions/_search (sortBy: createdAt or hits); see the training-suggestions API reference.

Rules & signals

RuleMeaning
SIMILARITYThe text matches a selected group semantically. null when no selected group has embedded samples or the text couldn't be embedded.
NEAR_DUPLICATEThe text is a near-duplicate of a positive sample in a selected group. null when the selected groups have no samples.
CAMPAIGNAt least min_sources distinct sources sent this message within the window. null for short texts or with campaigns off.

A match on a sample with weight 1 or more raises the classification to BAD. The classification's similarity object lists the matched groups (group, sampleId, matchType, score, weight), campaign (sources, windowSec, or null when the check didn't run), and error (EMBEDDING_UNAVAILABLE when semantic matching was skipped). Scores, the category formula, and floors are in Classifier rules & scores. Turn rules off with classifier.disableRules or the body's disableRules, for example ["CAMPAIGN"] for test traffic.

An excerpt of a POST /v2/classifier response for a reworded scam message:

"similarity": {
  "matches": [
    { "group": "crypto_scam", "sampleId": "trd_…", "matchType": "semantic", "score": 0.9804, "weight": 1.0 }
  ],
  "campaign": { "sources": 1, "windowSec": 3600 },
  "error": null,
  "rules": {
    "SIMILARITY": { "score": 4.9 },
    "NEAR_DUPLICATE": { "score": 0.0 },
    "CAMPAIGN": { "score": 0.0 }
  },
  "score": 4.9,
  "time": 5.387,
  "triggeredRules": ["SIMILARITY"]
}

Using the API

POST /v2/similarity explains how a text compares with training data: which samples it resembles, how closely, and why each group matched or didn't. It's read-only: it doesn't classify, count campaigns, capture suggestions, or write request logs. Call it with a restricted or full key:

curl https://sentinel.example.com/v2/similarity \
  -u 'key_…:sec_…' \
  -H 'Content-Type: application/json' \
  -d '{"text": "Give us your bitcoin and our trading bot doubles it in a day, guaranteed.", "groups": ["*"], "limit": 3}'
{
  "data": {
    "modelId": "local:d6ea442ff6a891da:mean",
    "error": null,
    "time": 4.062,
    "groups": [
      {
        "group": "crypto_scam",
        "matched": true,
        "score": 0.7563,
        "threshold": 0.6600000262260437,
        "suppressedBy": null,
        "matches": [
          {
            "sampleId": "trd_…",
            "text": "Our crypto bot doubles your deposit in one day, zero risk, guaranteed profit.",
            "label": "positive",
            "score": 0.7563,
            "matchType": "semantic",
            "distance": 22,
            "aboveThreshold": true,
            "span": { "start": 0, "end": 73 }
          },
          …
        ]
      }
    ]
  }
}
Request fieldDefaultNotes
text—Required, 1–16000 characters.
groups["*"]Groups to compare with, as in classifier.similarityGroups; [] compares with none.
examples[]Up to 100 unsaved positive texts (1–4000 characters each), compared as an extra group named examples, with weight 1 and the default threshold.
limit10Samples listed per group, 1–50.
includeBelowThresholdfalseAlso list the closest samples below their threshold.

Each entry in groups (highest score first, only groups that exist) has matched, the best positive's score and threshold, suppressedBy (the negative sample that blocked a match, else null), and matches. Each match has matchType (near_duplicate or semantic), distance (SimHash bits that differ: 0 for the same text, about 32 for unrelated texts), aboveThreshold, and span, the UTF-8 byte offsets of the part of your text that matched best (null for near-duplicates). Stored thresholds are 32-bit floats, so 0.66 comes back as 0.6600000262260437.

Changing one word in a 440-character message gives a near-duplicate:

{ "score": 1.0, "matchType": "near_duplicate", "distance": 3, "aboveThreshold": true, "span": null, … }

Semantic matching can be unavailable while near-duplicate matching still runs. The response is then 200 with error set to EMBEDDINGS_DISABLED (provider = "none") or EMBEDDING_UNAVAILABLE (model not loaded yet, busy past [similarity] queue_timeout, or failed), and modelId is null without a model. See the similarity API reference.

/v2/similarity and classification use the same matching, but classification compares only the groups the security group selects, reports only matched groups, follows disableRules, and needs weight ≥ 1 to make the verdict BAD on its own. A text that matches here but not in classification usually means one of those differs.

Tuning

Thresholds

Cosine similarities depend on the model, so pick thresholds from your data rather than relying on the default 0.80. With the default local model, rewordings of a message often score around 0.6–0.75. _evaluate suggests a threshold for a group from its own samples (Training Data → Groups → Evaluate thresholds):

curl https://sentinel.example.com/v2/training-data/_evaluate \
  -H 'Authorization: Bearer <jwt>' \
  -H 'Content-Type: application/json' \
  -d '{"accountId": "acc_…", "group": "crypto_scam"}'
{
  "data": {
    "currentThreshold": 0.8,
    "curve": [
      { "precision": 0.7778, "recall": 0.7, "threshold": 0.5 },
      { "precision": 0.8333, "recall": 0.5, "threshold": 0.55 },
      { "precision": 1.0, "recall": 0.5, "threshold": 0.6 },
      …
      { "precision": 1.0, "recall": 0.2, "threshold": 0.8 },
      …
    ],
    "group": "crypto_scam",
    "negatives": 2,
    "pending": 0,
    "positives": 10,
    "precision": 1.0,
    "recall": 0.5,
    "suggestedThreshold": 0.66
  }
}

Each embedded positive is scored against the group's other positives (leave-one-out), standing in for a new message of the same kind, and each embedded negative against the positives. Across thresholds 0.50–0.99, recall is the share of positives that would match and precision the share of matches that are positives. suggestedThreshold has the best F0.5 score, which values precision twice as much as recall; without negatives, it's the highest threshold that keeps recall at 0.9 or above. curve shows steps of 0.05.

_evaluate changes nothing. To apply a value, set the threshold of the group's positive samples: Apply threshold in the dialog does that, or PATCH each sample. The call needs at least 3 embedded positives (400 INSUFFICIENT_SAMPLES, also for a group that doesn't exist) and an embedding model (400 EMBEDDINGS_DISABLED).

False positives and misses

What you see in /v2/similarityWhat to do
A group is missing from groupsCheck the name in _groups; name builtin: groups explicitly.
matches: []Send includeBelowThreshold: true to see the closest samples; check error and pending.
High score, matched: false, suppressedBy: nullThe best positive is below its threshold: lower it, guided by _evaluate.
matched: false with suppressedBy setA negative is about as close as the best positive. Add the text as a positive if it's spam, or fix the negative.
matched: true for a legitimate textAdd it as a negative sample (for example with _feedback) or raise the threshold.

More positives and realistic negatives give better suggestions; add the legitimate messages that you see falsely matched as negatives. To watch a new group without affecting verdicts, give its samples weight 0, or use a weight below 1 for groups that should raise suspicion without deciding alone.

Limitations

  • Near-duplicate matching is unreliable for one-sentence texts; use semantic matching for short messages.
  • Without an embedding model, only near-duplicates and campaigns are detected. With a model, a new or changed sample matches only as a near-duplicate until it's embedded.
  • Text beyond max_chunks chunks (about 8000 characters by default) can only match as a near-duplicate.
  • Other replicas pick up sample changes through the KV store's pub/sub within moments, and at the latest after [similarity] sync_interval (default 5m). The duplicate checks of _import, _feedback, and accepting a suggestion use the handling replica's view, so a sample written on another replica a moment earlier can be missed.
  • Campaign counting is approximate, and /v2/classifier calls without ip inflate it.

Privacy

Samples and suggestions store message text verbatim in the database, and texts can contain personal data. Suggestions are deleted after [similarity.suggestions] retention; samples are kept until you delete them, and DELETE removes them for good. With provider = "openai", the text of every sample and every classified text is sent to the configured embeddings API; with local, nothing leaves the server. See Privacy & data handling.

Start typing to search...

↑ ↓ Navigate ↵ Select