A surprising number of production AI agents will genuinely follow an instruction embedded in user input telling them to ignore their system prompt — because nothing in the setup distinguishes "trusted instructions" from "untrusted user text."
The fix:
never treat user input as equally trustworthy as your system prompt. Where possible, use a model/framework that supports distinct instruction-vs-input roles, and validate any agent output that would trigger a real action (a refund, a data change) before it actually executes — don't let the model's word alone be the final authority.
Comments
No comments yet — be the first.