Content Moderation Platform

Part of the worlds-biggest-software-project initiative.

AI-powered text and image moderation with a built-in human review queue, designed as an open, self-hostable alternative to proprietary SaaS moderation platforms.

The Content Moderation Platform combines automated multi-modal content classification with a structured human-in-the-loop review workflow. It targets social platforms, marketplaces, gaming communities, and enterprise platforms that need scalable moderation with regulatory compliance (DSA, CSAM reporting) without surrendering data to a hyperscaler.

Why Content Moderation Platform?

Every enterprise-grade incumbent (AWS Rekognition, Azure AI Content Safety, Hive, Checkstep, ActiveFence, Two Hat) is proprietary SaaS — platforms with data sovereignty or GDPR concerns have no credible self-hosted option.
Hyperscaler APIs (AWS, Azure, OpenAI Moderation) ship without a built-in human review queue or appeals workflow; teams must assemble these from scratch.
DSA compliance tooling — Statement of Reasons, structured appeals, transparency reporting — is built only into enterprise-tier products priced for VLOPs, leaving smaller platforms under-served.
Most tools classify content in isolation, ignoring conversation context and cultural locale, which drives both false positives and missed harms.
Policy changes typically require classifier retraining or threshold tuning; an LLM-driven natural-language policy engine can apply updates immediately.

Key Features

Multi-Modal Detection

REST API accepting text and image payloads with JSON confidence score responses
Pre-built detection categories: hate speech, NSFW/adult, violence, spam/profanity, self-harm
Asynchronous video moderation via frame-extraction pipeline (v1.1)
AI-generated content and deepfake detection (backlog)
Audio and voice toxicity moderation (backlog)

Human Review and Appeals

Moderator interface with queue assignment, action, and escalation
Appeal submission and tracking workflow with structured resolution states (v1.1)
Per-decision audit log with timestamp, content hash, score, and action taken
DSA-compliant Statement of Reasons generation for moderation actions
NCMEC CyberTipline reporting integration for CSAM compliance (backlog)

Policy and Configuration

Configurable threshold profiles per content category for per-platform policy
Natural language policy configuration via LLM-powered policy-to-rule translation (v1.1)
Per-user context and trust-score tracking (v1.1)
No-code policy workflow builder with drag-and-drop rule logic (backlog)

Integration and Operations

Webhook notifications for asynchronous moderation results (v1.1)
Multi-language text support targeting 20+ languages (v1.1)
Real-time live stream moderation with sub-2-second latency (backlog)
Threat intelligence feed integration for coordinated inauthentic behaviour (backlog)

AI-Native Advantage

Unlike incumbents whose policies are encoded in trained classifiers requiring retraining cycles, this platform expresses moderation policy in natural language and applies it immediately to new content. LLMs generate human-readable explanations citing specific policy clauses — a direct fit for DSA appeal requirements. Context-aware severity adjustment lets a generative model distinguish, for example, medical discussion of self-harm from instructional self-harm content. Purpose-built detectors can target generative-model artefact signatures to address the AI-generated content category that no incumbent fully covers.

Tech Stack & Deployment

The platform is positioned as a self-hostable alternative to proprietary SaaS moderation. Integration is REST API first, with JSON confidence-score responses and webhook events for asynchronous processing. Compliance interfaces target the EU Digital Services Act, NCMEC CyberTipline (US CSAM reporting), and C2PA / PhotoDNA for media provenance and known-illegal-content matching. AVID (AI Vulnerability Database) is referenced for benchmarking moderation models against known failure modes.

Market Context

The automated content moderation sub-segment was valued at USD 1.23 billion in 2025, growing at 20.1% CAGR (SNS Insider / GlobeNewswire, 2025). The broader content moderation services market is projected to reach USD 30.75 billion by 2032. Hyperscaler image moderation prices around USD 0.001–0.003 per call; managed human review adds USD 0.02–0.10 per item; enterprise platform contracts run USD 50K–500K/yr. Primary buyers are social media and UGC platforms, online marketplaces, gaming communities, enterprise HR/collaboration platforms, and financial services compliance teams.

Project Status

This project is in the research and specification phase.
Contributions, feedback, and domain expertise are welcome.

Contributing

We welcome contributions from developers, domain experts, and potential users. See CONTRIBUTING.md for guidelines.

Important: All contributions must be your own original work or clearly attributed open-source material with a compatible licence. Copyright infringement and licence violations will not be tolerated and will result in immediate removal of the offending contribution. If you are unsure whether a piece of code, text, or other material is safe to contribute, open an issue and ask before submitting.

Licence

Licence to be determined. See discussion for context.