Discord AI bug mistakenly banned over 8,000 accounts

Chat platform Discord has admitted on X that a bug in its AI-assisted moderation system wrongly banned around 8,200 user accounts over the past two months. The system mistakenly flagged ordinary images, including grid patterns and screenshots, as child sexual abuse material (CSAM).

Approximately 8,200 accounts were affected between May and early July. An additional 200 users were erroneously banned over the weekend due to the same fault. Discord has since reinstated all affected accounts. The company stated its systems use similarity matching against known CSAM to flag potentially illegal content for review.

A software bug triggered permanent account bans instead of temporarily pausing uploads during that review process. Furthermore, the same bug prevented cleared accounts from being automatically reinstated, leaving users suspended until Discord manually restored their accounts.

Discord employs machine learning models to analyze image content for potential CSAM not yet in databases. Normally, human reviewers verify potential matches before any enforcement action is taken. The incident raises questions about the reliability of AI-driven moderation and the risks of automation in content safety.