Safety-Tuned Process Reward Models for Infeasible Task Refusal
Train a lightweight process reward model to score policy compliance and trigger safe refusals when agents encounter infeasible tasks.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 4.3, the evaluation of 30 infeasible tasks reveals that even the best frontier models (e.g., GPT-5.2 Low, DeepSeek-V3.2) only successfully refuse unsatisfiable requests 53.8-53.9% of the time, often attempting unauthorized actions or hallucinating premature completion.
“Both failures share the same pattern: accepting the first plausible result without cross-validating against task context or policy constraints.”
Limitation
Agents lack robust negative-constraint reasoning; they accept the first plausible tool output without explicitly cross-validating against task context or system policies, leading to unsafe execution of infeasible tasks.
Proposed Solution
Develop a lightweight Process Reward Model (PRM) trained specifically to score trajectory steps on policy compliance rather than just goal progression. Use this PRM as an early-stopping mechanism during decoding (e.g., in a Best-of-N search) to trigger a structured 'Safe Escalation' action when the safety score of a proposed tool call drops below a verified threshold.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Real-world enterprise deployment requires agents to do no harm. Improving safe refusal capabilities prevents silent error propagation, unauthorized access, and database corruption in production systems.
How to Get Started
How to test it: Hypothesis: A policy-compliance Process Reward Model (PRM) used to trigger safe escalation during Best-of-N decoding will improve infeasible-task refusal without hurting feasible task completion. Compare against ReAct baselines (DeepSeek-V3.2 High and Claude Sonnet 4.5) on the 30 infeasible tasks and a stratified feasible subset using clean-refusal rate and pass@1 across 3 runs; stop if Best-of-N with PRM fails to surpass the 53.9% top refusal baseline or lowers feasible pass@1 by more than 3 percentage points.
- Collect a dataset of feasible and infeasible task trajectories with step-level binary safety labels.
- Train a small reward model using an architecture like DeBERTa or an open-source language model to predict step compliance.
- Integrate the trained PRM into a Best-of-N decoding loop as an early-stopping filter.
- Evaluate the refusal accuracy on the set of infeasible test cases specified in the paper.
Code linked to this paper on Hugging Face: github.com/ServiceNow/EnterpriseOps-Gym
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The listed results cover safety challenges, vulnerability analyses, and general reward modeling like LongRM, but none of them develop a Process Reward Model specifically trained on policy compliance to serve as an early-stopping mechanism for unsafe tool calls during decoding.
- Why Agents Compromise Safety Under Pressure · 2026 · 8 citations
- The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agents · 2026 · 14 citations
- LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling · 2025 · 8 citations
We searched OpenAlex and arXiv for “Train a lightweight process reward model to score policy compliance and trigger safe refusals when agents encounter infeasible tasks.” and “Safety-Tuned Process Reward Models for Infeasible Task Refusal” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise SettingsShiva Krishna Reddy Malay, Shravan Nayak, Jishnu Sethumadhavan Nair +6 more · 2026
This paper introduces ENTERPRISEOPS-GYM, a highly complex, sandboxed benchmark featuring 164 database tables and 512 tools designed to evaluate LLM agents in realistic, policy-constrained enterprise workflows. The study reveals that current frontier models struggle significantly with long-horizon strategic planning, cascading state dependencies, and safely refusing infeasible tasks, indicating they are not yet ready for autonomous enterprise deployment.
Related Ideas
- Dense Process Reward Optimization for Long-Horizon GUI AgentsMobile-Agent-v3: Foundamental Agents for GUI Automation🟠Medium
- Action-Semantic Privilege Isolation to Resist Indirect Prompt Injection in Autonomous AgentsK2-Horizon: A Frontier Fleet of Open Source Language Models🟠Medium
- Affordance-Grounded Execution Filters to Bridge the Text-to-Action Refusal GapEnvisioning Agentic Robotics: A New Era of Robotic Systems🟡Fairly easy
- Verifiable Tree-Search with Partial State FeedbackDeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints🟠Medium
- Granular Credit Assignment via Learned Value FunctionsAgent Lightning: Train ANY AI Agents with Reinforcement Learning🟠Medium