Invalid GitHub events leading to job queuing
An incident on August 1, 2024, caused invalid GitHub events to be sent to Namespace's ingress, resulting in jobs being queued without valid triggers. The issue was resolved by 17:08 UTC after rolling back a change and processing the backlog.
Background
- **namespace-status.com** is the status page for **Namespace Labs**, the company behind **NVMe/NTCP cloud servers** and the **Portunus** edge-compute platform. The page reports real-time infrastructure incidents.
- This incident (01KVBEJCE25NBJGE12VBQHJ54F) describes a bug where **invalid GitHub webhook events** (e.g., malformed push or PR payloads) caused automated CI/CD jobs to pile up in a queue, delaying legitimate builds.
- **GitHub webhooks** are HTTP callbacks triggered by repo events (pushes, PRs, etc.); services like Namespace listen to these to start workflows. When bad data arrives, systems can fail to process or discard it gracefully, leading to backlogs.
- The issue matters because it disrupted job scheduling for users relying on Namespace's infrastructure for deployment pipelines — not a security breach, but a reliability bug in event handling.