# Chat & forum moderation

> **Sentinel v2 (beta):** This page documents Sentinel v2, a beta rewrite that may change before the stable release. It does not apply to Sentinel v1, the current stable version: see the [Sentinel v1 documentation](/docs/sentinel.md).

Every chat message, comment or forum post is a new chance for spam, scams, phishing links and abuse, from accounts that passed sign-up long ago. This page screens each message from your backend with `POST /v2/classifier` before it's published, and feeds your moderators' decisions back into training data so that reworded copies are caught next time.

Prerequisites: a running Sentinel with an active license, and an admin user who can edit security groups, API keys and training data. Protect the sign-up form itself as in [Protecting user registration](/docs/sentinel/v2/use-cases/registration/).

## What you're defending against

| Threat | What stops it |
| --- | --- |
| Spam and promotional posts | The deterministic spam filter (`SPAM_WORDS`, `URL`, `LINK_MARKUP`, …) in 26 languages, including disguised words (`OBFUSCATION`). |
| Phishing links | `URL_PHISHING`, which makes the classification `BAD`. |
| Profanity, harassment and hate | `PROFANITY`, graded by severity, and optionally the AI decider's `AI_ABUSE`. |
| Reworded or translated repeats of known spam | `SIMILARITY` and `NEAR_DUPLICATE` against the training groups you build from moderator decisions. |
| The same message posted from many IPs | `CAMPAIGN`, without any training data. |
| Flooding | A per-user rate limit in the same call. |
| Scams and off-topic posts that use ordinary words | The AI decider's `AI_SCAM` and `AI_OFF_TOPIC`. |

Screening runs from your backend, because a chat message has no form for the widget to protect. For forums with a classic post form, you can add the widget as in [Protecting contact & lead forms](/docs/sentinel/v2/use-cases/forms/) as well.

## Recommended configuration

### Server configuration

Turn on semantic similarity and training suggestions. The AI decider is optional; it needs a TypeSafe or OpenRouter API key:

```toml
[similarity]
provider = "local"    # ONNX model, ~135 MB download at startup

[similarity.suggestions]
enabled = true        # queues spam the text rules catch, for review; stores message text

[classifier.ai]       # optional
provider = "openrouter"
api_key  = "${OPENROUTER_API_KEY}"
```

See [Similarity & training data](/docs/sentinel/v2/features/similarity/#enabling-it) and [AI decider](/docs/sentinel/v2/features/ai-decider/#server-configuration).

### Backend group and key

Create a `restricted` security group for your moderation backend, apply the [Backend API (allowed IPs)](/docs/sentinel/v2/policies/presets/#backend-api-allowed-ips) preset with your servers' IPs, and add a rule after it that tunes the classifier for chat:

```json
{
"name": "moderation",
"action": "set",
"set": [
{ "field": "classifier.disableRules", "value": "EMOJI EXCLAMATION CAPITALIZATION" },
{ "field": "classifier.similarityGroups", "value": "chat_spam chat_abuse" },
{ "field": "classifier.ai", "value": "true" },
{ "field": "classifier.aiPurpose", "value": "Community forum about home espresso machines: questions, reviews, buying advice and repair tips" }
]
}
```

- **`classifier.disableRules`.** Emojis, `!!` runs and all-caps words are normal in chat. Keep them on for long-form forums.
- **`classifier.similarityGroups`.** The training groups to compare with. Groups exist once they have samples; [Step 2](#step-2--feed-moderator-decisions-back) creates them.
- **`classifier.ai*`.** Leave these out without an AI provider. `classifier.aiPurpose` turns on `AI_OFF_TOPIC`; describe everything the forum is meant for. See [Writing a purpose and instructions](/docs/sentinel/v2/features/ai-decider/#writing-a-purpose-and-instructions).

Create a `restricted` API key in this group with **Request logging** set to **All**, so every classification is logged with its triggered rules. Raise `[rate_limit] global` above your peak message rate, because every message is a request from your backend's IP. See [Backend & API protection](/docs/sentinel/v2/use-cases/api-protection/).

## Integration steps

### Step 1 — Classify each message before you publish it

Send the message text, the author's IP and a per-user rate limit to `POST /v2/classifier`:

```bash
curl -X POST https://sentinel.example.com/v2/classifier \
-u 'key_…:sec_…' \
-H 'Content-Type: application/json' \
-d '{
"text": "Earn $500 a day from home!!! Join my crypto signals group: t.me/quickprofits",
"ip": "203.0.113.45",
"languages": ["en"],
"rateLimit": { "limit": "10/1m,200/1d", "customKey": "user:8812" }
}'
```

```json
{
"data": {
"classification": "BAD",
"score": 2.48,
"triggeredRules": ["SPAM_WORDS", "URL", "CURRENCY"],
"text": {
  "language": null,
  "terms": [{ "term": "earn", "rule": "SPAM_WORDS", "kind": "EXACT", "language": "en" }],
  "score": 4.97,
  "triggeredRules": ["SPAM_WORDS", "URL", "CURRENCY"],
  …
},
"similarity": { "matches": [], "campaign": { "sources": 1, "windowSec": 3600 }, "score": 0.0, … },
"rateLimit": { "consumedPoints": 1, "remainingPoints": 9, "resetInSec": 60, "score": 0.0, "triggeredRules": [], … },
"ip": { "threats": [], "action": null, "score": 0.0, "triggeredRules": [], … },
…
}
}
```

The text category scores `4.97` and the IP category `0`, so the overall score is their mean, (4.97 + 0) / 2 = `2.48`, `BAD`. The rate limit wasn't exceeded and the similarity check found nothing, so neither counts. See [From rules to a verdict](/docs/sentinel/v2/features/classifier/#from-rules-to-a-verdict).

- **Send `ip`.** `CAMPAIGN` counts distinct source IPs; a call without `ip` counts as a new source every time. The IP rules (`MALICIOUS`, `TOR`, …) also join the score.
- **Send `languages` when you know them,** for example the forum's or the user's language. Detection is less reliable on short chat messages. `text.language` is `null` when you pass `languages`.
- **Headers are optional.** Pass the author's `user-agent`, `accept` and `accept-language` in `headers` if you have them; the device rules then join the score.
- **Limits.** `text` is up to 16000 characters. See the [`POST /v2/classifier` API reference](/sentinel-api-docs-v2.html#tag/classifier/POST/v2/classifier).

Then map the result to a moderation action. In TypeScript:

```ts
type Action = 'publish' | 'hold' | 'reject' | 'throttle';

const SENTINEL_AUTH =
'Basic ' + Buffer.from(`key_…:${process.env.SENTINEL_API_KEY_SECRET}`).toString('base64');

export async function screenMessage(text: string, authorId: string, authorIp: string): Promise<Action> {
const res = await fetch('https://sentinel.example.com/v2/classifier', {
method: 'POST',
headers: { Authorization: SENTINEL_AUTH, 'Content-Type': 'application/json' },
body: JSON.stringify({
  text,
  ip: authorIp,
  rateLimit: { limit: '10/1m,200/1d', customKey: `user:${authorId}` },
}),
signal: AbortSignal.timeout(5000),
});
if (!res.ok) return 'hold'; // Sentinel unavailable or rejected the call: queue for a moderator
const { data } = await res.json();
const rules: string[] = data.triggeredRules;

if (rules.includes('RATE_LIMIT')) return 'throttle';
if (data.classification === 'BAD') return 'reject';
if (data.classification === 'NEUTRAL' || rules.includes('PROFANITY') || rules.includes('CAMPAIGN')) {
return 'hold';
}
return 'publish';
}
```

- **`BAD`** covers phishing URLs, injection attempts, strong similarity matches and confident AI answers, which raise the score to at least `2` whatever the other categories say, and spam that outweighs the clean categories. See [From rules to a verdict](/docs/sentinel/v2/features/classifier/#from-rules-to-a-verdict).
- **Hold, don't drop.** A held message waits for a moderator, which is also where your training data comes from.
- **Latency.** Without the AI decider, a call takes milliseconds. The AI decider adds one request to the provider (timeout `3s` by default). For live chat, publish first and retract when the result is `reject`, or hold only messages from new accounts.

### Step 2 — Feed moderator decisions back

When a moderator removes a message as spam or abuse, or restores a held one, send it to `POST /v2/training-data/_feedback` with the backend key:

```bash
curl https://sentinel.example.com/v2/training-data/_feedback \
-u 'key_…:sec_…' \
-H 'Content-Type: application/json' \
-d '{"group": "chat_spam", "label": "positive", "text": "Join my crypto signals group and earn $500 a day, link in my profile"}'
```

```json
{ "data": { "duplicate": false, "id": "trd_…" } }
```

- `positive` samples are what the group catches; send restored messages as `negative` to the same group, so similar legitimate posts stop matching.
- A match on a positive sample with weight `1` (the feedback default) makes the classification `BAD`. Repeated clicks are harmless: a near-duplicate with the same label returns `duplicate: true`.
- A near-duplicate with the opposite label returns `409 LABEL_CONFLICT`; decide which moderator was right and fix that sample under **Training Data → Samples**.

See [Feedback from your application](/docs/sentinel/v2/features/similarity/#feedback-from-your-application).

### Step 3 — Review training suggestions

With `[similarity.suggestions]` on, Sentinel queues messages whose text category alone is `BAD`. Under **Training Data → Suggestions**, accept the real spam into `chat_spam` and the false positives as `negative` samples. Sort by **hits** to handle the most repeated texts first. See [Training suggestions](/docs/sentinel/v2/features/similarity/#training-suggestions).

## Monitoring & tuning

- **Check thresholds once a group has samples.** **Training Data → Groups → Evaluate thresholds** (`POST /v2/training-data/_evaluate`) suggests a similarity threshold from the group's positives and negatives. See [Similarity: Tuning](/docs/sentinel/v2/features/similarity/#tuning).
- **Explain a match.** `POST /v2/similarity` shows which samples a text resembles and why a group matched or didn't, without classifying or counting a campaign.
- **Find noisy rules.** The `triggeredRules` breakdown of `POST /v2/request-logs/_stats` for the backend key shows which rules fire most. Disable rules that fit your community's normal content in the `moderation` rule, for example `HTML_INJECTION` and `SQL_INJECTION` on a developer forum where people paste code. See [Request logs & analytics](/docs/sentinel/v2/operations/request-logs/#read-statistics).
- **Different rules per community.** Give each community or channel type its own key and group, or switch `classifier.similarityGroups` and `classifier.aiPurpose` per request: the body's `similarityGroups` and `disableRules` replace the group's lists.
- **Repeat offenders.** Report the IPs of banned spammers with `POST /v2/threat/_report`, as in [Protecting user registration](/docs/sentinel/v2/use-cases/registration/#step-3--report-abusive-accounts). Their IP rules (`MALICIOUS`, `BOT`) then raise the score of their next messages.

## Related

- [Classifier](/docs/sentinel/v2/features/classifier/)
- [Spam, profanity & injection detection](/docs/sentinel/v2/features/text-analysis/)
- [Similarity & training data](/docs/sentinel/v2/features/similarity/)
- [AI decider](/docs/sentinel/v2/features/ai-decider/)
- [Phishing detection](/docs/sentinel/v2/features/phishing/)
- [Protecting contact & lead forms](/docs/sentinel/v2/use-cases/forms/)
