ICML paper finds LLMs cannot be fully secured against chain-of-thought forgery attacks exploiting instruction-source confusion
Researchers at ICML, a top AI conference, argue LLMs can't be made fully secure due to a flaw in how they identify who is instructing them, per MIT Technology Review. A forged policy note in a model's reasoning trace got GPT-oss-20b and GPT-5 to comply with requests they're train
Researchers presenting a paper at ICML, a top AI conference, argue it is impossible to make large language models fully secure against attack because of a fundamental flaw in how the models identify who or what is giving them instructions, according to MIT Technology Review. The researchers describe the technique as a "chain-of-thought forgery" attack: by inserting a forged policy note into a model's own internal reasoning trace, they were able to get OpenAI's GPT-oss-20b and GPT-5 to comply with requests the models are trained to refuse. The paper's attacks focus mainly on OpenAI's models, but the researchers report seeing similar results when testing models from Anthropic, Alibaba and DeepSeek, suggesting the vulnerability is not specific to one company's approach to model design. The finding reinforces a separate mathematical proof NIST published in June, authored by senior scientist Apostol Vassilev, which draws on Godel's incompleteness theorems to argue that no finite set of guardrails can be built to block every possible adversarial prompt. Together, the two results point to the same practical conclusion for security teams deploying LLMs in production: guardrails need continuous updating and monitoring rather than being treated as a one-time fix.