Show HN: OSS Tests to Fix AI Gen Code. 110 Test for Major API – Supabase, Auth0
A developer open-sourced a collection of 110 tests for major APIs like Supabase and Auth0 to catch bad code generated by AI tools. The tests are based on official documentation and aim to fix common issues such as writable user metadata and exposed service role keys in client code.
Background
- **Claude Code** is an AI coding assistant by Anthropic that works inside large codebases. As the project grows ("context compacts"), the AI tends to generate plausible-looking but incorrect code ("slop"), especially for API integrations.
- Common mistakes include exposing sensitive keys (e.g., Supabase "service role" keys) in client-side code like Next.js, and misconfiguring security rules (e.g., letting end users write to user metadata).
- Prompts and guidelines files ("skills.md") are the usual way to steer the AI, but the author found these unreliable without concrete test failures.
- **api-doctor** is an open-source test suite that uses official API documentation (not AI-generated tests) to verify correctness. It covers major services like Supabase (backend-as-a-service with auth/storage) and Auth0 (authentication platform) — catching the kinds of mistakes that AI coding tools routinely produce.