Chat & forum moderation
Every chat message, comment or forum post is a new chance for spam, scams, phishing links and abuse, from accounts that passed sign-up long ago. This page screens each message from your backend with POST /v2/classifier before it's published, and feeds your moderators' decisions back into training data so that reworded copies are caught next time.
Prerequisites: a running Sentinel with an active license, and an admin user who can edit security groups, API keys and training data. Protect the sign-up form itself as in Protecting user registration.
What you're defending against
| Threat | What stops it |
|---|---|
| Spam and promotional posts | The deterministic spam filter (SPAM_WORDS, URL, LINK_MARKUP, …) in 26 languages, including disguised words (OBFUSCATION). |
| Phishing links | URL_PHISHING, which makes the classification BAD. |
| Profanity, harassment and hate | PROFANITY, graded by severity, and optionally the AI decider's AI_ABUSE. |
| Reworded or translated repeats of known spam | SIMILARITY and NEAR_DUPLICATE against the training groups you build from moderator decisions. |
| The same message posted from many IPs | CAMPAIGN, without any training data. |
| Flooding | A per-user rate limit in the same call. |
| Scams and off-topic posts that use ordinary words | The AI decider's AI_SCAM and AI_OFF_TOPIC. |
Screening runs from your backend, because a chat message has no form for the widget to protect. For forums with a classic post form, you can add the widget as in Protecting contact & lead forms as well.
Recommended configuration
Server configuration
Turn on semantic similarity and training suggestions. The AI decider is optional; it needs a TypeSafe or OpenRouter API key:
[similarity]
provider = "local" # ONNX model, ~135 MB download at startup
[similarity.suggestions]
enabled = true # queues spam the text rules catch, for review; stores message text
[classifier.ai] # optional
provider = "openrouter"
api_key = "${OPENROUTER_API_KEY}"See Similarity & training data and AI decider.
Backend group and key
Create a restricted security group for your moderation backend, apply the Backend API (allowed IPs) preset with your servers' IPs, and add a rule after it that tunes the classifier for chat:
{
"name": "moderation",
"action": "set",
"set": [
{ "field": "classifier.disableRules", "value": "EMOJI EXCLAMATION CAPITALIZATION" },
{ "field": "classifier.similarityGroups", "value": "chat_spam chat_abuse" },
{ "field": "classifier.ai", "value": "true" },
{ "field": "classifier.aiPurpose", "value": "Community forum about home espresso machines: questions, reviews, buying advice and repair tips" }
]
}classifier.disableRules. Emojis,!!runs and all-caps words are normal in chat. Keep them on for long-form forums.classifier.similarityGroups. The training groups to compare with. Groups exist once they have samples; Step 2 creates them.classifier.ai*. Leave these out without an AI provider.classifier.aiPurposeturns onAI_OFF_TOPIC; describe everything the forum is meant for. See Writing a purpose and instructions.
Create a restricted API key in this group with Request logging set to All, so every classification is logged with its triggered rules. Raise [rate_limit] global above your peak message rate, because every message is a request from your backend's IP. See Backend & API protection.
Integration steps
Step 1 — Classify each message before you publish it
Send the message text, the author's IP and a per-user rate limit to POST /v2/classifier:
curl -X POST https://sentinel.example.com/v2/classifier \
-u 'key_…:sec_…' \
-H 'Content-Type: application/json' \
-d '{
"text": "Earn $500 a day from home!!! Join my crypto signals group: t.me/quickprofits",
"ip": "203.0.113.45",
"languages": ["en"],
"rateLimit": { "limit": "10/1m,200/1d", "customKey": "user:8812" }
}'{
"data": {
"classification": "BAD",
"score": 2.48,
"triggeredRules": ["SPAM_WORDS", "URL", "CURRENCY"],
"text": {
"language": null,
"terms": [{ "term": "earn", "rule": "SPAM_WORDS", "kind": "EXACT", "language": "en" }],
"score": 4.97,
"triggeredRules": ["SPAM_WORDS", "URL", "CURRENCY"],
…
},
"similarity": { "matches": [], "campaign": { "sources": 1, "windowSec": 3600 }, "score": 0.0, … },
"rateLimit": { "consumedPoints": 1, "remainingPoints": 9, "resetInSec": 60, "score": 0.0, "triggeredRules": [], … },
"ip": { "threats": [], "action": null, "score": 0.0, "triggeredRules": [], … },
…
}
}The text category scores 4.97 and the IP category 0, so the overall score is their mean, (4.97 + 0) / 2 = 2.48, BAD. The rate limit wasn't exceeded and the similarity check found nothing, so neither counts. See From rules to a verdict.
- Send
ip.CAMPAIGNcounts distinct source IPs; a call withoutipcounts as a new source every time. The IP rules (MALICIOUS,TOR, …) also join the score. - Send
languageswhen you know them, for example the forum's or the user's language. Detection is less reliable on short chat messages.text.languageisnullwhen you passlanguages. - Headers are optional. Pass the author's
user-agent,acceptandaccept-languageinheadersif you have them; the device rules then join the score. - Limits.
textis up to 16000 characters. See thePOST /v2/classifierAPI reference.
Then map the result to a moderation action. In TypeScript:
type Action = 'publish' | 'hold' | 'reject' | 'throttle';
const SENTINEL_AUTH =
'Basic ' + Buffer.from(`key_…:${process.env.SENTINEL_API_KEY_SECRET}`).toString('base64');
export async function screenMessage(text: string, authorId: string, authorIp: string): Promise<Action> {
const res = await fetch('https://sentinel.example.com/v2/classifier', {
method: 'POST',
headers: { Authorization: SENTINEL_AUTH, 'Content-Type': 'application/json' },
body: JSON.stringify({
text,
ip: authorIp,
rateLimit: { limit: '10/1m,200/1d', customKey: `user:${authorId}` },
}),
signal: AbortSignal.timeout(5000),
});
if (!res.ok) return 'hold'; // Sentinel unavailable or rejected the call: queue for a moderator
const { data } = await res.json();
const rules: string[] = data.triggeredRules;
if (rules.includes('RATE_LIMIT')) return 'throttle';
if (data.classification === 'BAD') return 'reject';
if (data.classification === 'NEUTRAL' || rules.includes('PROFANITY') || rules.includes('CAMPAIGN')) {
return 'hold';
}
return 'publish';
}BADcovers phishing URLs, injection attempts, strong similarity matches and confident AI answers, which raise the score to at least2whatever the other categories say, and spam that outweighs the clean categories. See From rules to a verdict.- Hold, don't drop. A held message waits for a moderator, which is also where your training data comes from.
- Latency. Without the AI decider, a call takes milliseconds. The AI decider adds one request to the provider (timeout
3sby default). For live chat, publish first and retract when the result isreject, or hold only messages from new accounts.
Step 2 — Feed moderator decisions back
When a moderator removes a message as spam or abuse, or restores a held one, send it to POST /v2/training-data/_feedback with the backend key:
curl https://sentinel.example.com/v2/training-data/_feedback \
-u 'key_…:sec_…' \
-H 'Content-Type: application/json' \
-d '{"group": "chat_spam", "label": "positive", "text": "Join my crypto signals group and earn $500 a day, link in my profile"}'{ "data": { "duplicate": false, "id": "trd_…" } }positivesamples are what the group catches; send restored messages asnegativeto the same group, so similar legitimate posts stop matching.- A match on a positive sample with weight
1(the feedback default) makes the classificationBAD. Repeated clicks are harmless: a near-duplicate with the same label returnsduplicate: true. - A near-duplicate with the opposite label returns
409 LABEL_CONFLICT; decide which moderator was right and fix that sample under Training Data → Samples.
See Feedback from your application.
Step 3 — Review training suggestions
With [similarity.suggestions] on, Sentinel queues messages whose text category alone is BAD. Under Training Data → Suggestions, accept the real spam into chat_spam and the false positives as negative samples. Sort by hits to handle the most repeated texts first. See Training suggestions.
Monitoring & tuning
- Check thresholds once a group has samples. Training Data → Groups → Evaluate thresholds (
POST /v2/training-data/_evaluate) suggests a similarity threshold from the group's positives and negatives. See Similarity: Tuning. - Explain a match.
POST /v2/similarityshows which samples a text resembles and why a group matched or didn't, without classifying or counting a campaign. - Find noisy rules. The
triggeredRulesbreakdown ofPOST /v2/request-logs/_statsfor the backend key shows which rules fire most. Disable rules that fit your community's normal content in themoderationrule, for exampleHTML_INJECTIONandSQL_INJECTIONon a developer forum where people paste code. See Request logs & analytics. - Different rules per community. Give each community or channel type its own key and group, or switch
classifier.similarityGroupsandclassifier.aiPurposeper request: the body'ssimilarityGroupsanddisableRulesreplace the group's lists. - Repeat offenders. Report the IPs of banned spammers with
POST /v2/threat/_report, as in Protecting user registration. Their IP rules (MALICIOUS,BOT) then raise the score of their next messages.