AI Toxicity Detector — Check Text for Toxic & Offensive Language

The page runs two independent analyses on the same text and shows both side by side. They use very different methods, and watching them agree — or disagree — is the quickest way to judge how much to trust a single reading.

1. Neural model (server-side)

The primary classifier is unitary/toxic-bert, a BERT-base · 110M parameters · multi-label sigmoid head fine-tuned on the Jigsaw Toxic Comment Classification corpus. We call the Hugging Face Inference API from a server-only route handler with the body parameter top_k: null so the response includes every category's probability. Because the heads are independent sigmoids (not a softmax), the six numbers do not sum to 1 — a comment can be both an insult and a threat. Your browser never downloads model weights; it sends text and receives a small JSON result. For multi-sentence input the whole text and each sentence (up to a cap) are scored in one request.

2. Lexical heuristic (cross-check, always on)

The second pass is pure, deterministic JavaScript that never leaves the server. It combines five shape signals, each contributing 0 to 1: the fraction of long words in all-caps (shouting), exclamation-mark density, repeated-letter elongation such as "stuuupid", the directed you are <word> insult pattern, and hits against a small frozen list of 21 mild insult words. The five signals are averaged into one heuristic score, then projected onto the six categories with fixed coefficients so the cross-check is in the same shape as the model output:

heuristicScore = (shout + bang + elongation + directInsult + mildInsult) / 5

category[c] = clamp01(heuristicScore × projection[c])   for each of the 6 heads

This lexicon is intentionally small and public — it is a transparent baseline, not a moderation denylist. It gives the tool a usable floor for obvious cases and, crucially, means a result is still returned when neural inference is not configured on this build.

3. Verdict and aggregation

For multi-sentence text, the document score for each category is the maximum across sentences — one toxic sentence pollutes the document, which matches the aggregation the Detoxify authors recommend. The verdict then follows the flag threshold you set (default 0.50): if any category is at or above it, the text is flagged; if the top category sits within 0.10 below it, the text is borderline and worth a human re-read; otherwise it is clean. The comparison is inclusive (≥), so a score of exactly the threshold flags. Both passes are deterministic given the same input and threshold.

• Hugging Face — unitary/toxic-bert (model card)
• Detoxify (Unitary AI) — canonical implementation and category definitions
• Jigsaw / Conversation AI — Toxic Comment Classification Challenge (six-label taxonomy)
• Borkan et al., 2019 — Nuanced Metrics for Measuring Unintended Bias (arXiv:1903.04561)
• Hugging Face Inference API — official documentation

The model card, category taxonomy, and inference endpoint were last cross-checked on 2026-05-12. The page is reviewed whenever the Hugging Face Inference API contract changes or the upstream model repo is re-uploaded.