Skip to content
AI-generated research idea🟠Medium💡Moderate InnovationNo close match found

Safety-Tuned Process Reward Models for Infeasible Task Refusal

Train a lightweight process reward model to score policy compliance and trigger safe refusals when agents encounter infeasible tasks.

Time

4–6 weeks to construct step-level policy annotations, train and calibrate the PRM threshold, integrate Safe Escalation into decoding, and complete 3-seed benchmark evaluations.

Compute level

Multiple GPUs

Resources

A 4–8x A100/H100 80GB GPU node to generate and label step-level policy-compliance trajectories, fine-tune and serve a lightweight PRM (4B–8B backbone), and execute Best-of-N rollouts (N=4–8) alongside ~$500–$1,500 in API credits and Docker sandbox compute.

What the paper used

Hardware not stated in the paper; evaluates 14 models (closed- and open-source up to 235B parameters) over 1,150 tasks across 3 runs each in a Docker sandbox (164 database tables, 512 tools) at reported inference costs of $0.007–$0.36 per task.

Prerequisites

Reward modeling, reinforcement learning from human feedback (RLHF)

Scope

Modify existing code and train a new reward model

The Gap

In Section 4.3, the evaluation of 30 infeasible tasks reveals that even the best frontier models (e.g., GPT-5.2 Low, DeepSeek-V3.2) only successfully refuse unsatisfiable requests 53.8-53.9% of the time, often attempting unauthorized actions or hallucinating premature completion.

“Both failures share the same pattern: accepting the first plausible result without cross-validating against task context or policy constraints.”
From the paper, Appendix C.7, page 27. Matched against the PDF text.

Limitation

Agents lack robust negative-constraint reasoning; they accept the first plausible tool output without explicitly cross-validating against task context or system policies, leading to unsafe execution of infeasible tasks.

Proposed Solution

Develop a lightweight Process Reward Model (PRM) trained specifically to score trajectory steps on policy compliance rather than just goal progression. Use this PRM as an early-stopping mechanism during decoding (e.g., in a Best-of-N search) to trigger a structured 'Safe Escalation' action when the safety score of a proposed tool call drops below a verified threshold.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

Real-world enterprise deployment requires agents to do no harm. Improving safe refusal capabilities prevents silent error propagation, unauthorized access, and database corruption in production systems.

How to Get Started

How to test it: Hypothesis: A policy-compliance Process Reward Model (PRM) used to trigger safe escalation during Best-of-N decoding will improve infeasible-task refusal without hurting feasible task completion. Compare against ReAct baselines (DeepSeek-V3.2 High and Claude Sonnet 4.5) on the 30 infeasible tasks and a stratified feasible subset using clean-refusal rate and pass@1 across 3 runs; stop if Best-of-N with PRM fails to surpass the 53.9% top refusal baseline or lowers feasible pass@1 by more than 3 percentage points.

  1. Collect a dataset of feasible and infeasible task trajectories with step-level binary safety labels.
  2. Train a small reward model using an architecture like DeBERTa or an open-source language model to predict step compliance.
  3. Integrate the trained PRM into a Best-of-N decoding loop as an early-stopping filter.
  4. Evaluate the refusal accuracy on the set of infeasible test cases specified in the paper.

Code linked to this paper on Hugging Face: github.com/ServiceNow/EnterpriseOps-Gym

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 10 results

The listed results cover safety challenges, vulnerability analyses, and general reward modeling like LongRM, but none of them develop a Process Reward Model specifically trained on policy compliance to serve as an early-stopping mechanism for unsafe tool calls during decoding.

We searched OpenAlex and arXiv for “Train a lightweight process reward model to score policy compliance and trigger safe refusals when agents encounter infeasible tasks.” and “Safety-Tuned Process Reward Models for Infeasible Task Refusal” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings

Shiva Krishna Reddy Malay, Shravan Nayak, Jishnu Sethumadhavan Nair +6 more · 2026

This paper introduces ENTERPRISEOPS-GYM, a highly complex, sandboxed benchmark featuring 164 database tables and 512 tools designed to evaluate LLM agents in realistic, policy-constrained enterprise workflows. The study reveals that current frontier models struggle significantly with long-horizon strategic planning, cascading state dependencies, and safely refusing infeasible tasks, indicating they are not yet ready for autonomous enterprise deployment.

Related Ideas

Is this a good idea?

Share onXLinkedIn