Skip to content
AI-generated research idea🔴Hard🔬Significant ResearchRelated work found

Closed-Loop Adaptive Red-Teaming and Adversarially Trained Span-Veto Sanitization

Train an automated red-teaming loop and learned span detector to harden deterministic agent guards against adaptive prompt-injection attacks that bypass hand-crafted rules.

Time

a couple of months

Compute level

Multiple GPUs

Resources

Multi-GPU node or 2-4 high-VRAM GPUs for closed-loop RL/black-box attack optimization and encoder fine-tuning, plus OpenAI API access and AgentDojo/AutoDojo/PISmith codebases.

What the paper used

One NVIDIA RTX PRO 6000 Blackwell GPU for local Ollama models (qwen3:8b, qwen3-coder:30b, gemma4:31b Q4_K_M, gemma4:12b 4-bit) plus OpenAI API for gpt-4o-mini-2024-07-18; evaluated across 949 AgentDojo v1.2.2 pairs per backbone, 560 AgentDyn pairs, 665 ChatInject pairs, 389+288 AutoDojo pairs, and 949 PISmith pairs.

Prerequisites

Adversarial machine learning, reinforcement learning red-teaming, transformer sequence labeling.

Scope

new model architecture

The Gap

In Section 3.2, Section 5, and Section 7, SPILLWAY relies on static regex/rule groups in its ingress sanitizer—including payee-change and address rules added specifically after inspecting four failing AutoDojo pairs—and defends its recovery path via the span veto S and the local gemma4:31b two-view judge. Furthermore, RL and optimizer attacks (PISmith and AutoDojo) were only replayed open-loop rather than optimized interactively against SPILLWAY's specific judge, delegation rules, and sanitizer patterns.

“The judge, the delegation rules, and the lexicon are the surface an adaptive attacker (Zhan et al., 2025; Hofer et al., 2026) would target, which the four AutoDojo pairs on the P3 path already exercise.”
From the paper, Section 7, page 10. Matched against the PDF text.

Limitation

The rule-based ingress sanitizer and deterministic delegation checks are brittle against adaptive or paraphrased attacks tailored to evade fixed regexes (such as the second payee transfer missed on held-out AutoDojo Banking), leaving path P3 exposed when an attacker crafts non-matching text that tricks the local judge.

Proposed Solution

Build a closed-loop adaptive red-teaming harness (adapting PISmith or AutoDojo black-box optimization directly against SPILLWAY's full pipeline) to systematically discover injections that bypass S and persuade the two-view judge or delegation rules. Use the resulting adversarial trajectories and benign tool outputs to distill a lightweight, token-level sequence-labeling span detector (e.g., a small DeBERTa/ModernBERT encoder running in milliseconds on a single GPU) that augments the deterministic turn scrubber and populates S without requiring manual regex authoring for new attack templates.

Why This Matters

Evaluating against closed-loop adaptive attackers would establish true worst-case robustness, while a learned span-veto model could generalize span detection beyond benchmark-specific templates at negligible latency.

How to Get Started

How to test it: Test whether an adversarially trained span-veto model reduces closed-loop adaptive ASR on held-out AgentDojo Banking and Slack suites by at least 75% relative to SPILLWAY's static regex sanitizer while keeping per-episode latency overhead under 100 ms and benign utility within 2%. Baseline is spillway-guard 1.0.0 and CommandSans; stop if benign false-positive span masking drops benign utility by more than 5%.

  1. Set up a closed-loop AutoDojo or PISmith attacker targeting spillway-guard 1.0.0 on AgentDojo Banking and Workspace to measure the true adaptive ASR on path P3.
  2. Collect and label failing adaptive attack spans versus benign tool outputs across AgentDojo and AgentDyn to form a train/held-out split.
  3. Fine-tune a compact encoder (e.g., DeBERTa-v3-small or ModernBERT) as a token-level span detector to replace or augment SPILLWAY's regex instruction rules and feed S.
  4. Evaluate the learned span veto under fresh closed-loop adaptive attacks and measure latency, UAU, BU, and ASR on held-out suites.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defenses and AutoDojo: A Generative Benchmark for Evaluating Prompt Injection Defenses in LLM Agents explore adaptive red-teaming harnesses and benchmarking for prompt injections. However, they do not specifically build a closed-loop adaptive red-teaming harness to distill a token-level span detector using adversarial trajectories as proposed here.

We searched OpenAlex and arXiv for “adaptive prompt injection red teaming agent defense span sanitization” and “Closed-Loop Adaptive Red-Teaming and Adversarially Trained Span-Veto Sanitization” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

SPILLWAY: Separating Target Admission from Sanitization to Defend Tool-Using Agents against Prompt Injection

Woosang Lim, Yeseul Chang, Jaewoo Lee · 2026

This paper introduces SPILLWAY, a runtime defense against indirect prompt injection for tool-using LLM agents that separates target admission from content sanitization. Rather than invoking a language model on every step or requiring a per-task planner, it enforces a deterministic egress floor using typed tool return schemas while using a rule-based ingress sanitizer to strip instructions and constrain a local recovery judge. Across three backbones and seven suites on AgentDojo and AgentDyn, SPILLWAY achieves zero in-scope attack success under standard and forged-template attacks and 0.005 under a reinforcement-learning attacker, with higher utility and far fewer model calls than existing defenses.

Related Ideas

Is this a good idea?

Share onXLinkedIn