Skip to content
AI-generated research idea🟡Fairly easy🔧Minor TweakNo close match found

Uncertainty-Aware Active Routing for Ambiguous Policies

Quantify output uncertainty during policy routing to trigger clarification questions when incoming queries match overlapping or ambiguous policies.

Time

1 week

Compute level

Single GPU or less

Resources

1x NVIDIA A100 GPU (80GB) for fine-tuning the 1.5B model on an ambiguous-policy dataset slice.

What the paper used

Supervised fine-tuning using Llama-Factory with full-parameter updates in bfloat16 precision, AdamW optimizer, maximum 4 epochs on a single NVIDIA A100 GPU over 43,000 samples.

Prerequisites

basic ML knowledge, uncertainty estimation

Scope

modify existing code

The Gap

Section 6.3 states that 'routing accuracy is constrained by the precision of the policy set; ambiguous or overlapping policy descriptions directly degrade routing performance.' The current Cross-Entropy training objective forces the model to make a hard decision between overlapping policies (e.g., 'review contract' vs 'legal analysis').

“First, routing accuracy is constrained by the precision of the policy set; ambiguous or overlapping policy descriptions directly degrade routing performance.”
From the paper, Section 6.3, page 9. Matched against the PDF text.

Limitation

The model lacks a mechanism to handle ambiguity. When user-defined policies overlap, the router acts deterministically (likely hallucinating certainty), rather than asking for clarification or merging policies.

Proposed Solution

Implement uncertainty quantification (e.g., measuring entropy of the output logits over route tokens). If the entropy exceeds a threshold, the router should output a special 'AMBIGUOUS' token or generate a clarification question prompting the user to distinguish between the competing policies. This requires fine-tuning on a new dataset slice where the 'correct' label is a clarification request when policies overlap.

Why This Matters

This prevents 'silent failures' where the router confidently sends a query to the wrong expert due to poor user descriptions, significantly improving robustness in real-world deployments.

How to Get Started

How to test it: We hypothesize that measuring logit entropy will reliably detect overlapping policies and prompt for clarification. We will compare against a deterministic routing baseline using classification accuracy and clarification appropriateness, stopping if ambiguity detection precision falls below 85%.

  1. Extract output logits from a baseline routing model and calculate entropy across candidate policy tokens.
  2. Synthesize a dataset slice containing intentionally ambiguous queries where the correct label is a clarification request.
  3. Fine-tune the routing policy to output a special ambiguous token when entropy exceeds a defined threshold.
  4. Measure the reduction in misclassification errors on overlapping policy benchmarks.

Code linked to this paper on Hugging Face: github.com/katanemo/archgw

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 10 results

None of the listed papers address uncertainty-aware active routing for ambiguous model policies using clarification tokens and fine-tuning on a specialized dataset slice. While papers like 'Learning When to Collaborate: Selective Multi-Agent Medical Reasoning via Uncertainty-Aware Routing' touch on uncertainty routing, they do not implement the specific ambiguity-handling mechanism proposed here.

We searched OpenAlex for “Quantify output uncertainty during policy routing to trigger clarification questions when incoming queries match overlapping or ambiguous policies.” and “Uncertainty-Aware Active Routing for Ambiguous Policies” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Arch-Router: Aligning LLM Routing with Human Preferences

Co Tran, Salman Paracha, Adil Hafeez +1 more · 2025

This paper introduces Arch-Router, a compact 1.5B parameter model designed to route LLM queries based on user-defined natural language policies rather than opaque performance benchmarks. By decoupling intent classification from model assignment and utilizing a synthetic data pipeline with Domain-Action taxonomies, the authors achieve state-of-the-art routing accuracy and reduced latency compared to proprietary models like GPT-4o.

Related Ideas

Is this a good idea?

Share onXLinkedIn