Action-Semantic Privilege Isolation to Resist Indirect Prompt Injection in Autonomous Agents
A system that enforces semantic privilege boundaries between untrusted tool outputs and agent action spaces to prevent indirect prompt injection vulnerabilities.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 11.2.3 and Section 11.3, the authors evaluate agentic safety across 311 sandboxed browser, coding, and tool-use tasks. While the flagship 375B model achieves an 84.3% full refusal rate on direct malicious requests in natural language, its safety mechanisms fail on action-mediated tasks and indirect prompt injections where adversarial instructions are embedded in untrusted web pages, files, or tool outputs.
“Under indirect prompt injection, the drop is steepest: full refusal falls to 2.5% and 62.5% of trajectories reach severity 2 or higher (55.0% reach severity 3), meaning injected instructions in untrusted content often result in harmful completion or unsafe tool use.”
Limitation
Safety alignment relies on shallow lexical cues rather than semantic action authorization, causing the agent full-refusal rate to drop to 2.5% under indirect prompt injection and allowing 62.5% of injected trajectories to reach severe harm levels.
Proposed Solution
Implement a dual-stream architecture that syntactically and semantically isolates untrusted tool observations from the supervisory planning prompt. Train a lightweight action-verifier adapter that evaluates the intended delta of each emitted tool call (such as file modifications, shell executions, or outbound network requests) against the user's initial high-level intent before execution. Incorporate adversarial synthetic trajectories containing indirect injection attempts into the RLVR stage, applying heavy penalties whenever the model follows imperative instructions sourced from retrieved content.
Why This Matters
Closing the gap between lexical refusal and tool-grounded action containment would allow autonomous agents to safely browse untrusted environments without executing arbitrary malicious commands.
How to Get Started
How to test it: Evaluate the hardened agent on the 40 indirect prompt injection tasks from the paper's agentic safety suite; the approach succeeds if the severity < 2 rate increases from 37.5% to above 65% without decreasing benign task success below 80%. If benign tool-call accuracy drops by more than 10%, stop and revise the action authorization filter.
- Download the released K2-Horizon-7B checkpoint and set up the OpenCode evaluation harness.
- Reproduce the baseline indirect prompt injection evaluations from Section 11.2 using the SafeArena and DecodingTrust subsets.
- Implement a parser that tags tokens originating from tool outputs with separate delimiter markers and restricts executable tool syntax generation within those spans.
- Fine-tune the model on 5,000 synthetic multi-turn trajectories with embedded indirect prompt injections penalizing compliance.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works like Evaluating Indirect Prompt Injection Defenses in Tool-Using LLM Agents and MELON investigate defenses against indirect prompt injection in tool-using agents, but they leave open specific architectural isolation mechanisms combined with action-verifier adapters trained via RLVR.
- Evaluating Indirect Prompt Injection Defenses in Tool-Using LLM Agents: Security, Utility, and Replication · 2026 · 6 citations
- The Framing Gap: Indirect Prompt-Injection Exfiltration Defeats Surface-Level Defenses in Tool-Using Agents · 2026 · 0 citations
- MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents · 2025 · 128 citations
We searched OpenAlex and arXiv for “indirect prompt injection defense agent tool use” and “Action-Semantic Privilege Isolation to Resist Indirect Prompt Injection in Autonomous Agents” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
K2-Horizon: A Frontier Fleet of Open Source Language ModelsK2 Team, Zhengzhong Liu, Haonan Li +92 more · 2026
This technical report presents K2-Horizon, a family of six fully open language models ranging from 0.9B dense to a 375B mixture-of-experts (MoE) flagship activating 23B parameters, optimized for agentic and reasoning tasks. The authors open-source the complete development artifacts, including intermediate checkpoints, pre-training corpora with 10T synthetic tokens, the xLLM training framework, and the RL360 reinforcement learning stack. Key architectural innovations include Grouped RMSNorm, Mixture-of-Value Attention (MoVA), softplus attention gating, and Uno discrete diffusion adapters for accelerated decoding.
Related Ideas
- Hermetic Harness-Agent Environment Isolation Against Dependency-Mediated Answer LeakageSage S2.1 Flash Technical Report🟡Fairly easy
- Affordance-Grounded Execution Filters to Bridge the Text-to-Action Refusal GapEnvisioning Agentic Robotics: A New Era of Robotic Systems🟡Fairly easy
- In-Flight Context-Preserving Intervention via Interactive Agent Stream InjectionTokenEconomizer: Experience-Driven Black-Box Cost Control for Coding Agents🟠Medium
- Safety-Tuned Process Reward Models for Infeasible Task RefusalEnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings🟠Medium
- Semantic Policy Firewall for Runtime SandboxesAgentScope 1.0: A Developer-Centric Framework for Building Agentic Applications🟡Fairly easy