Full-Trajectory Deployment Cost and Retry-Overhead Profiling for Medical AI Agents
Profile end-to-end agent deployment costs including failed retries, external judges, and sandbox overhead so healthcare teams can compare true operational affordability.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Sections 4.4, 5.4, and 6.3 (and Appendix A.2), inference costs used to construct the Pareto frontier are computed solely from recorded input, output, and cached token prices on valid runs, omitting API/safety-block retries (up to 10 retries on HealthBench Professional, unlimited retries until valid on PhysicianBench and CHI-Bench), task timeouts (up to 7,200 s on ALE-CLI), and tool/sandbox/judge charges.
“They exclude failed runs and retries for the four public benchmarks, as well as judge, search, tool, sandbox, and internal platform charges (Appendix A.2), and therefore do not represent total deployment expenditure.”
Limitation
Excluding failed attempts, retries, tool execution overhead, and automated judge fees underestimates real-world deployment costs and may misclassify brittle models with high retry or timeout rates as Pareto-optimal.
Proposed Solution
Instrument the FHIR and CLI evaluation harnesses with OpenTelemetry-style full-trajectory billing that logs token consumption across every retry and safety-block fallback, wall-clock container/sandbox compute-seconds, external search/FHIR API calls, and LLM-judge token costs. Compute risk-adjusted cost-per-successful-task metrics and re-derive the performance-cost Pareto frontier under realistic hospital concurrency and SLA latency constraints.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Full-trajectory accounting should give clinical IT teams accurate budget projections and expose hidden reliability taxes in agentic healthcare deployments.
How to Get Started
How to test it: Test the hypothesis that including retry, timeout, and sandbox overhead shifts at least two models off the current five-model Pareto frontier by measuring full cost per successful task across the top 10 models on PhysicianBench and CHI-Bench. Abandon full-suite re-runs if initial pilot traces show retry/sandbox overhead adds less than 5% variance to token-only cost.
- Fork the CHI-Bench, PhysicianBench, and ALE-CLI harness wrappers to wrap every API call, retry, timeout, and Docker container execution with token, latency, and compute-second loggers.
- Re-run the five Pareto-frontier models plus five mid-tier models on CHI-Bench (75 tasks) and PhysicianBench (100 tasks) to capture failed-run and retry telemetry.
- Compare token-only mean cost per task against total cost per successful task (including retries and container runtime) and test whether the Pareto frontier in Figure 5 changes.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Cost-Utility Alignment in LLM Agent Trajectories directly addresses profiling and evaluation of agent costs and trajectories in clinical contexts. The current proposal extends this by integrating FHIR-specific infrastructure and hospital-level SLA latency constraints, which remain an open research frontier in medical agent deployment.
- Cost-Utility Alignment in LLM Agent Trajectories:Profiling,Attribution,Diagnosis,Adaptation,and Evaluation · 2026 · 2 citations
- Do More Agents Help? Evaluating Accuracy and Cost in LLM Multi-Agent Workflows · 2026 · 2 citations
- A comprehensive survey of AI agents in healthcare · 2026 · 55 citations
We searched OpenAlex and arXiv for “LLM agent deployment cost retries sandbox latency Pareto” and “Full-Trajectory Deployment Cost and Retry-Overhead Profiling for Medical AI Agents” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
InclusionMed: Benchmarking AI for Inclusive HealthcareThe InclusionMed Collaboration · 2026
InclusionMed introduces an open medical AI evaluation framework featuring the nine-cell InclusionMed Atlas (three healthcare roles by three AI capabilities) and the InclusionMed Index, which benchmarks 22 frontier language models across five audited evaluations. The study reveals substantial rank reversals across roles and capabilities, weaker clinical decision-making and action execution relative to patient communication, and up to 24-fold inference cost disparities among similarly scoring models.
Related Ideas
- Execution-Time and Hardware-Cost-Aware Scoring Beyond Token Budgets in Agentic Kernel OptimizationAscendGYM: A Cost-Aware Kernel Optimization Benchmark for Real-World Workloads on Ascend NPUs🟡Fairly easy
- Requirement-Coverage and Execution-Trace-Aware Candidate Deliverable SelectionHarness^2: Recursive Agent Harnessing for an Open World🟢Easy
- Multi-Shape Generalization and End-to-End Workload Evaluation for NPU Kernel OptimizationAscendGYM: A Cost-Aware Kernel Optimization Benchmark for Real-World Workloads on Ascend NPUs🟠Medium
- Hierarchical Edge-Cloud Architecture for Sub-Second Reactive ReplanningForceHarness: Turning Force into Action and Physics for Frozen Frontier Models🟠Medium
- Artifact-to-Claim Cross-Verification and Process Consistency Rewards for Agentic DeliverablesSage S2.1 Flash Technical Report🟡Fairly easy