Content Moderation Pipeline at 50K Posts/Minute

The Challenge

A social media platform with 40M monthly active users was moderating content using a single-stage classifier that flagged posts for human review. The classifier had a 71% recall rate (missing 29% of policy violations) and generated so many false positives that the 200-person moderation team could only review 60% of flagged content within the SLA. Unreviewed content remained visible for an average of 18 hours before action was taken.

Our Approach

We built a multi-stage content moderation pipeline that processes text, images, and video at 50,000+ posts per minute with human-quality accuracy.

Stage 1 — Pre-Filter: A lightweight classifier (distilled BERT, 67M parameters) runs on every post in real-time. Posts scoring below the safe threshold (< 0.05 violation probability) are passed through immediately. Posts above the unsafe threshold (> 0.95) are auto-removed. Posts in the middle range proceed to Stage 2. This stage handles 82% of all content with sub-10ms latency.

Stage 2 — Deep Analysis: A multi-modal model analyses text content, attached images (using a fine-tuned CLIP model for visual policy violations), and metadata context (account age, posting history, network signals). This stage takes 50–200ms per post and produces a detailed classification across 23 policy categories (hate speech, harassment, sexual content, violence, misinformation, etc.).

Stage 3 — Human Review: Posts that the automated system cannot confidently classify (10–15% of volume) are routed to human reviewers. The routing system considers reviewer expertise (some reviewers are trained on specific policy areas), language proficiency, and emotional load balancing (reviewers are not exposed to the most disturbing content categories for more than 2 hours continuously).

Perceptual Hashing: Known violating images are hashed using pHash and stored in a database of 12M+ hashes. New images are compared against this database in real-time, catching re-uploads of previously removed content with 99.8% accuracy even when images are cropped, resized, or watermarked.

Results