Skip to main content
ALTCHA®

Chat & forum moderation

Every chat message, comment or forum post is a new chance for spam, scams, phishing links and abuse, from accounts that passed sign-up long ago. This page screens each message from your backend with POST /v2/classifier before it's published, and feeds your moderators' decisions back into training data so that reworded copies are caught next time.

Prerequisites: a running Sentinel with an active license, and an admin user who can edit security groups, API keys and training data. Protect the sign-up form itself as in Protecting user registration.

What you're defending against

ThreatWhat stops it
Spam and promotional postsThe deterministic spam filter (SPAM_WORDS, URL, LINK_MARKUP, …) in 26 languages, including disguised words (OBFUSCATION).
Phishing linksURL_PHISHING, which makes the classification BAD.
Profanity, harassment and hatePROFANITY, graded by severity, and optionally the AI decider's AI_ABUSE.
Reworded or translated repeats of known spamSIMILARITY and NEAR_DUPLICATE against the training groups you build from moderator decisions.
The same message posted from many IPsCAMPAIGN, without any training data.
FloodingA per-user rate limit in the same call.
Scams and off-topic posts that use ordinary wordsThe AI decider's AI_SCAM and AI_OFF_TOPIC.

Screening runs from your backend, because a chat message has no form for the widget to protect. For forums with a classic post form, you can add the widget as in Protecting contact & lead forms as well.

Server configuration

Turn on semantic similarity and training suggestions. The AI decider is optional; it needs a TypeSafe or OpenRouter API key:

[similarity]
provider = "local"    # ONNX model, ~135 MB download at startup

[similarity.suggestions]
enabled = true        # queues spam the text rules catch, for review; stores message text

[classifier.ai]       # optional
provider = "openrouter"
api_key  = "${OPENROUTER_API_KEY}"

See Similarity & training data and AI decider.

Backend group and key

Create a restricted security group for your moderation backend, apply the Backend API (allowed IPs) preset with your servers' IPs, and add a rule after it that tunes the classifier for chat:

{
  "name": "moderation",
  "action": "set",
  "set": [
    { "field": "classifier.disableRules", "value": "EMOJI EXCLAMATION CAPITALIZATION" },
    { "field": "classifier.similarityGroups", "value": "chat_spam chat_abuse" },
    { "field": "classifier.ai", "value": "true" },
    { "field": "classifier.aiPurpose", "value": "Community forum about home espresso machines: questions, reviews, buying advice and repair tips" }
  ]
}
  • classifier.disableRules. Emojis, !! runs and all-caps words are normal in chat. Keep them on for long-form forums.
  • classifier.similarityGroups. The training groups to compare with. Groups exist once they have samples; Step 2 creates them.
  • classifier.ai*. Leave these out without an AI provider. classifier.aiPurpose turns on AI_OFF_TOPIC; describe everything the forum is meant for. See Writing a purpose and instructions.

Create a restricted API key in this group with Request logging set to All, so every classification is logged with its triggered rules. Raise [rate_limit] global above your peak message rate, because every message is a request from your backend's IP. See Backend & API protection.

Integration steps

Step 1 — Classify each message before you publish it

Send the message text, the author's IP and a per-user rate limit to POST /v2/classifier:

curl -X POST https://sentinel.example.com/v2/classifier \
  -u 'key_…:sec_…' \
  -H 'Content-Type: application/json' \
  -d '{
    "text": "Earn $500 a day from home!!! Join my crypto signals group: t.me/quickprofits",
    "ip": "203.0.113.45",
    "languages": ["en"],
    "rateLimit": { "limit": "10/1m,200/1d", "customKey": "user:8812" }
  }'
{
  "data": {
    "classification": "BAD",
    "score": 2.48,
    "triggeredRules": ["SPAM_WORDS", "URL", "CURRENCY"],
    "text": {
      "language": null,
      "terms": [{ "term": "earn", "rule": "SPAM_WORDS", "kind": "EXACT", "language": "en" }],
      "score": 4.97,
      "triggeredRules": ["SPAM_WORDS", "URL", "CURRENCY"],
      …
    },
    "similarity": { "matches": [], "campaign": { "sources": 1, "windowSec": 3600 }, "score": 0.0, … },
    "rateLimit": { "consumedPoints": 1, "remainingPoints": 9, "resetInSec": 60, "score": 0.0, "triggeredRules": [], … },
    "ip": { "threats": [], "action": null, "score": 0.0, "triggeredRules": [], … },
    …
  }
}

The text category scores 4.97 and the IP category 0, so the overall score is their mean, (4.97 + 0) / 2 = 2.48, BAD. The rate limit wasn't exceeded and the similarity check found nothing, so neither counts. See From rules to a verdict.

  • Send ip. CAMPAIGN counts distinct source IPs; a call without ip counts as a new source every time. The IP rules (MALICIOUS, TOR, …) also join the score.
  • Send languages when you know them, for example the forum's or the user's language. Detection is less reliable on short chat messages. text.language is null when you pass languages.
  • Headers are optional. Pass the author's user-agent, accept and accept-language in headers if you have them; the device rules then join the score.
  • Limits. text is up to 16000 characters. See the POST /v2/classifier API reference.

Then map the result to a moderation action. In TypeScript:

type Action = 'publish' | 'hold' | 'reject' | 'throttle';

const SENTINEL_AUTH =
  'Basic ' + Buffer.from(`key_…:${process.env.SENTINEL_API_KEY_SECRET}`).toString('base64');

export async function screenMessage(text: string, authorId: string, authorIp: string): Promise<Action> {
  const res = await fetch('https://sentinel.example.com/v2/classifier', {
    method: 'POST',
    headers: { Authorization: SENTINEL_AUTH, 'Content-Type': 'application/json' },
    body: JSON.stringify({
      text,
      ip: authorIp,
      rateLimit: { limit: '10/1m,200/1d', customKey: `user:${authorId}` },
    }),
    signal: AbortSignal.timeout(5000),
  });
  if (!res.ok) return 'hold'; // Sentinel unavailable or rejected the call: queue for a moderator
  const { data } = await res.json();
  const rules: string[] = data.triggeredRules;

  if (rules.includes('RATE_LIMIT')) return 'throttle';
  if (data.classification === 'BAD') return 'reject';
  if (data.classification === 'NEUTRAL' || rules.includes('PROFANITY') || rules.includes('CAMPAIGN')) {
    return 'hold';
  }
  return 'publish';
}
  • BAD covers phishing URLs, injection attempts, strong similarity matches and confident AI answers, which raise the score to at least 2 whatever the other categories say, and spam that outweighs the clean categories. See From rules to a verdict.
  • Hold, don't drop. A held message waits for a moderator, which is also where your training data comes from.
  • Latency. Without the AI decider, a call takes milliseconds. The AI decider adds one request to the provider (timeout 3s by default). For live chat, publish first and retract when the result is reject, or hold only messages from new accounts.

Step 2 — Feed moderator decisions back

When a moderator removes a message as spam or abuse, or restores a held one, send it to POST /v2/training-data/_feedback with the backend key:

curl https://sentinel.example.com/v2/training-data/_feedback \
  -u 'key_…:sec_…' \
  -H 'Content-Type: application/json' \
  -d '{"group": "chat_spam", "label": "positive", "text": "Join my crypto signals group and earn $500 a day, link in my profile"}'
{ "data": { "duplicate": false, "id": "trd_…" } }
  • positive samples are what the group catches; send restored messages as negative to the same group, so similar legitimate posts stop matching.
  • A match on a positive sample with weight 1 (the feedback default) makes the classification BAD. Repeated clicks are harmless: a near-duplicate with the same label returns duplicate: true.
  • A near-duplicate with the opposite label returns 409 LABEL_CONFLICT; decide which moderator was right and fix that sample under Training Data → Samples.

See Feedback from your application.

Step 3 — Review training suggestions

With [similarity.suggestions] on, Sentinel queues messages whose text category alone is BAD. Under Training Data → Suggestions, accept the real spam into chat_spam and the false positives as negative samples. Sort by hits to handle the most repeated texts first. See Training suggestions.

Monitoring & tuning

  • Check thresholds once a group has samples. Training Data → Groups → Evaluate thresholds (POST /v2/training-data/_evaluate) suggests a similarity threshold from the group's positives and negatives. See Similarity: Tuning.
  • Explain a match. POST /v2/similarity shows which samples a text resembles and why a group matched or didn't, without classifying or counting a campaign.
  • Find noisy rules. The triggeredRules breakdown of POST /v2/request-logs/_stats for the backend key shows which rules fire most. Disable rules that fit your community's normal content in the moderation rule, for example HTML_INJECTION and SQL_INJECTION on a developer forum where people paste code. See Request logs & analytics.
  • Different rules per community. Give each community or channel type its own key and group, or switch classifier.similarityGroups and classifier.aiPurpose per request: the body's similarityGroups and disableRules replace the group's lists.
  • Repeat offenders. Report the IPs of banned spammers with POST /v2/threat/_report, as in Protecting user registration. Their IP rules (MALICIOUS, BOT) then raise the score of their next messages.

Start typing to search...

↑ ↓ Navigate ↵ Select