Closed-Loop Adaptive Red-Teaming and Adversarially Trained Span-Veto Sanitization
Train an automated red-teaming loop and learned span detector to harden deterministic agent guards against adaptive prompt-injection attacks that bypass hand-crafted rules.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 3.2, Section 5, and Section 7, SPILLWAY relies on static regex/rule groups in its ingress sanitizer—including payee-change and address rules added specifically after inspecting four failing AutoDojo pairs—and defends its recovery path via the span veto S and the local gemma4:31b two-view judge. Furthermore, RL and optimizer attacks (PISmith and AutoDojo) were only replayed open-loop rather than optimized interactively against SPILLWAY's specific judge, delegation rules, and sanitizer patterns.
“The judge, the delegation rules, and the lexicon are the surface an adaptive attacker (Zhan et al., 2025; Hofer et al., 2026) would target, which the four AutoDojo pairs on the P3 path already exercise.”
Limitation
The rule-based ingress sanitizer and deterministic delegation checks are brittle against adaptive or paraphrased attacks tailored to evade fixed regexes (such as the second payee transfer missed on held-out AutoDojo Banking), leaving path P3 exposed when an attacker crafts non-matching text that tricks the local judge.
Proposed Solution
Build a closed-loop adaptive red-teaming harness (adapting PISmith or AutoDojo black-box optimization directly against SPILLWAY's full pipeline) to systematically discover injections that bypass S and persuade the two-view judge or delegation rules. Use the resulting adversarial trajectories and benign tool outputs to distill a lightweight, token-level sequence-labeling span detector (e.g., a small DeBERTa/ModernBERT encoder running in milliseconds on a single GPU) that augments the deterministic turn scrubber and populates S without requiring manual regex authoring for new attack templates.
Why This Matters
Evaluating against closed-loop adaptive attackers would establish true worst-case robustness, while a learned span-veto model could generalize span detection beyond benchmark-specific templates at negligible latency.
How to Get Started
How to test it: Test whether an adversarially trained span-veto model reduces closed-loop adaptive ASR on held-out AgentDojo Banking and Slack suites by at least 75% relative to SPILLWAY's static regex sanitizer while keeping per-episode latency overhead under 100 ms and benign utility within 2%. Baseline is spillway-guard 1.0.0 and CommandSans; stop if benign false-positive span masking drops benign utility by more than 5%.
- Set up a closed-loop AutoDojo or PISmith attacker targeting spillway-guard 1.0.0 on AgentDojo Banking and Workspace to measure the true adaptive ASR on path P3.
- Collect and label failing adaptive attack spans versus benign tool outputs across AgentDojo and AgentDyn to form a train/held-out split.
- Fine-tune a compact encoder (e.g., DeBERTa-v3-small or ModernBERT) as a token-level span detector to replace or augment SPILLWAY's regex instruction rules and feed S.
- Evaluate the learned span veto under fresh closed-loop adaptive attacks and measure latency, UAU, BU, and ASR on held-out suites.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defenses and AutoDojo: A Generative Benchmark for Evaluating Prompt Injection Defenses in LLM Agents explore adaptive red-teaming harnesses and benchmarking for prompt injections. However, they do not specifically build a closed-loop adaptive red-teaming harness to distill a token-level span detector using adversarial trajectories as proposed here.
- PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defenses · 2026 · 28 citations
- AutoDojo: A Generative Benchmark for Evaluating Prompt Injection Defenses in LLM Agents · 2026 · 15 citations
We searched OpenAlex and arXiv for “adaptive prompt injection red teaming agent defense span sanitization” and “Closed-Loop Adaptive Red-Teaming and Adversarially Trained Span-Veto Sanitization” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
SPILLWAY: Separating Target Admission from Sanitization to Defend Tool-Using Agents against Prompt InjectionWoosang Lim, Yeseul Chang, Jaewoo Lee · 2026
This paper introduces SPILLWAY, a runtime defense against indirect prompt injection for tool-using LLM agents that separates target admission from content sanitization. Rather than invoking a language model on every step or requiring a per-task planner, it enforces a deterministic egress floor using typed tool return schemas while using a rule-based ingress sanitizer to strip instructions and constrain a local recovery judge. Across three backbones and seven suites on AgentDojo and AgentDyn, SPILLWAY achieves zero in-scope attack success under standard and forged-template attacks and 0.005 under a reinforcement-learning attacker, with higher utility and far fewer model calls than existing defenses.
Related Ideas
- Automated Adversarial Red-Teaming via LLM AgentsGPT-4 Technical Report🔴Hard
- Affordance-Grounded Execution Filters to Bridge the Text-to-Action Refusal GapEnvisioning Agentic Robotics: A New Era of Robotic Systems🟡Fairly easy
- Action-Semantic Privilege Isolation to Resist Indirect Prompt Injection in Autonomous AgentsK2-Horizon: A Frontier Fleet of Open Source Language Models🟠Medium
- Adversarial Specification Probing and Multi-Seed Contrast Verification for Verifier-Free Harness EditsHarness^2: Recursive Agent Harnessing for an Open World🟠Medium
- Conformal Multimodal Outcome Verification to Mitigate Premature Completion BiasEnvisioning Agentic Robotics: A New Era of Robotic Systems🟡Fairly easy