Prompt Injection as Role Confusion
Research finds LLMs cannot reliably distinguish privileged text (role tags) from user input, and are more influenced by writing style than content. This "role confusion" enables jailbreaks mimicking internal model style. Removing stylistic cues ("destyling") dropped attack success from 61% to 10%, suggesting defenses remain a whack-a-mole game.
Background
- LLMs like ChatGPT use role tags (<system>, <user>, <assistant>) to mark different types of text. The model is trained to follow <system> instructions and ignore user attempts to override them.
- Prompt injection: an AI security attack where a user embeds hidden commands in their input (e.g. "Ignore your rules and do X"), tricking the model into breaking its own safety guidelines.
- MIT researchers Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell argue that prompt injection is really **role confusion**: models can't reliably tell which text is privileged based on tags alone. Worse, they found models pay more attention to the *writing style* of text than to the actual tag — so text that mimics the model's own "inner voice" (like <think> blocks) can override safeguards.
- "Destyling" — rewriting user input in a plain, unstylized way — dropped attack success from 61% to 10%, showing the model was mainly fooled by style, not by tags.
- Implication: unless models develop genuine role perception (understanding who said what, not just matching text patterns), prompt injection defenses may be permanently unreliable — a "perpetual whack-a-mole game."