Retrieval-Augmented Policy Context for Scalability
Integrate dense retrieval before the router to dynamically inject only relevant policies into the prompt, removing hard limits on total policy count.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
Section 4.2 describes the input prompt x as containing 'the set of all possible route policies C'. This places a hard limit on the number of routing policies (scalable only up to the context window size) and increases latency linearly as the number of policies grows.
Limitation
The current architecture requires injecting every possible policy description into the prompt for every inference pass. This prevents the system from handling large-scale enterprise environments with hundreds or thousands of specific routing policies.
Proposed Solution
Integrate a dense retrieval step (RAG) before the router. Use a lightweight embedding model to retrieve the top-k most relevant policies from a vector store based on the user query, and inject only those k policies into the Arch-Router prompt. This decouples the total number of policies from the inference cost.
Why This Matters
This enables the router to scale to thousands of user-defined policies without hitting context window limits or degrading inference speed, essential for large-scale SaaS platforms.
How to Get Started
How to test it: We hypothesize that a dense retrieval step will maintain routing accuracy while supporting thousands of policies without exceeding context limits. We will compare against full-prompt inclusion using top-1 routing accuracy and prompt token count, stopping if accuracy drops by more than 3% at 1,000 policies.
- Set up a vector database using FAISS to index the full set of available routing policies.
- Select a lightweight embedding model to encode incoming user queries for similarity matching.
- Modify the router inference pipeline to retrieve the top-k most relevant policies and format them into the prompt.
- Benchmark inference latency and routing accuracy as the total number of system policies scales significantly.
Code linked to this paper on Hugging Face: github.com/katanemo/archgw
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The provided literature consists of broad surveys on Retrieval-Augmented Generation (RAG) and federated RAG systems, such as 'Graph Retrieval-Augmented Generation: A Survey' and 'Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG'. None of these listed results cover using dense retrieval to filter and inject specific policies into an architecture router prompt to decouple policy count from inference cost.
We searched OpenAlex for “Integrate dense retrieval before the router to dynamically inject only relevant policies into the prompt, removing hard limits on total policy count.” and “Retrieval-Augmented Policy Context for Scalability” and an AI compared the idea with the top 6 results. A small search can miss relevant work.
Source Paper
Arch-Router: Aligning LLM Routing with Human PreferencesCo Tran, Salman Paracha, Adil Hafeez +1 more · 2025
This paper introduces Arch-Router, a compact 1.5B parameter model designed to route LLM queries based on user-defined natural language policies rather than opaque performance benchmarks. By decoupling intent classification from model assignment and utilizing a synthetic data pipeline with Domain-Action taxonomies, the authors achieve state-of-the-art routing accuracy and reduced latency compared to proprietary models like GPT-4o.
Related Ideas
- Domain-Grounded Test-Time Compute Routing for Policy WorkflowsEnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings🟡Fairly easy
- Neuro-Symbolic Constraint Enforcement ModuleMobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios🟠Medium
- Off-Policy Correction for Asynchronous Agent RLGLM-5: from Vibe Coding to Agentic Engineering🟠Medium
- Token-Level Importance-Weighted Off-Policy Correction for Asynchronous Agent RLTeleChat to Xing4: Advancing Frontier Agentic Models on Domestic Chips and Frameworks🟠Medium
- Feedback-Loop Retrieval VerificationSimpleMem: Efficient Lifelong Memory for LLM Agents🟢Easy