What happened after 2k people tried to hack my AI assistant
After opening his AI assistant Claw to public hacking attempts, the author reports that over 2,000 people participated, successfully jailbreaking the assistant and demonstrating vulnerabilities in AI safety measures. The experiment highlighted how easily AI systems can be exploited when exposed to motivated attackers.
Background
- The author, Fernando I., built a custom AI assistant using Claw (an open-source agentic coding tool) and posted a challenge inviting people to try to "hack" or break it through a public interface.
- Over 2,000 people tested the assistant, probing for prompt injections (tricking the AI into ignoring instructions), data leaks, and other security flaws.
- The post documents what successful attacks looked like, how the author hardened the system over time, and lessons for anyone deploying LLM-based agents with tool access.
- Key context: LLM-powered agents that can run code, read files, or call APIs face unique risks because a clever prompt can make them act against their instructions — unlike traditional software, the "attack surface" includes the model's own reasoning.