You've Got (Too Much) Mail: Behind the Scenes of the 3/25/26 Voice Outage
Discord experienced a major voice outage on March 25-26 due to a sudden spike in connection requests that overwhelmed internal monitoring systems. The failure of the monitoring layer prevented automated scaling from triggering, leading to a prolonged service disruption. The engineering team resolved the issue by bypassing the broken monitoring and manually scaling the infrastructure.
Background
- Discord is a popular free voice, video, and text chat platform used by millions of gamers, communities, and remote workers.
- On March 25-26, 2025, Discord suffered a major voice outage that prevented users from joining or hearing others in voice calls for over a day.
- The root cause was a massive surge in internal "queue mail" — system messages that coordinate tasks between servers — triggered by a single server cluster repeatedly failing and restarting, which flooded the system with 1.3 trillion messages in 48 hours.
- Discord's engineers eventually had to manually modify the source code of the running production system (a risky move called a "hot patch") to discard the offending messages and restore service.
- This post is a rare, transparent postmortem from Discord's engineering team, aimed at explaining a complex distributed-systems failure to a technical audience.