关于角色混淆的思考
本文探讨了"角色混淆"这一概念，即大型语言模型（LLM）在推理时会忽略特定的角色标签（如<system>、<user>、<think>），而更倾向于根据文本的语气推断角色身份。这种现象解释了许多越狱攻击（jailbreak）的工作原理——例如，用户可以通过模仿模型自身推理痕迹的语气来欺骗模型，使其违反安全策略。文章还讨论了一种编号猜谜游戏的变体仍能成功愚弄ChatGPT 5.5，并提出了通过直接修改嵌入向量来区分角色、从而修复此问题的几种可能方案。
背景速读
- **LLM jailbreaks** trick AI models into bypassing safety filters. This post explores a new explanation: models infer who is speaking from *tone*, not role tags (`<system>`, `<user>`). So a user can write input that *sounds* like the model's own reasoning — and the model trusts it.
- **Charles Ye, Jasmine Cui and Dylan Hadfield-Menell** (MIT) authored the paper "Prompt Injection as Role Confusion" (2026). They show special delimiters fail because LLMs judge roles by textual style.
- **Simon Willison** is a prominent developer covering prompt injection. **Pliny the Liberator** is a pseudonymous jailbreak researcher.
- Proposed fix: encode role info directly into token embeddings (like BERT's segment embeddings or a rotation scheme). **ASIDE** is a related 2025 paper on separating instructions from data.
- **Giles Thomas** writes the "LLM from Scratch" tutorial series.