We measured whether AI obeys architecture rules. Even Opus ignored them 60%
A study measuring whether AI models follow software architecture rules found that even advanced models like Opus ignored them 60% of the time. The results highlight a significant gap between AI code generation capabilities and adherence to prescribed architectural guidelines.
Background
- Software architecture rules are constraints that teams set so code stays maintainable (e.g., "no direct database calls from the frontend" or "the payment module must never import from the reporting module").
- "Opus" refers to Anthropic's Claude 3 Opus, at the time of testing the most capable (and most expensive) model in the Claude 3 lineup, often used for complex coding tasks.
- The article describes a study where researchers gave LLMs a multi-file codebase with explicit architecture rules, then asked them to add features. The finding: even the best models consistently violated those rules, generating code that ignored the project's intended structure.
- This matters because teams are increasingly relying on AI to write production code; if models can't follow agreed-upon constraints, they risk silently introducing technical debt, architectural drift, and security or maintainability problems that human reviewers have to catch later.