Skip to content
AI-generated research idea🟠Medium💡Moderate InnovationRelated work found

Cross-Family Harness Standardization and Multi-Judge Debiasing in Agentic Clinical Evaluation

Standardize agent execution harnesses across model families and evaluate with multi-judge ensembles so workflow scores measure model capability rather than tooling artifacts.

Time

a couple of months

Compute level

Single GPU or less

Resources

API credits for 6-8 flagship models and 3 cross-family judge models across CHI-Bench, ALE-CLI, and diagnostic case subsets, plus standard CPU/Docker environment

What the paper used

Evaluated 22 commercial and open-weight LLM releases via API and agent harnesses across 1,034 tasks from five benchmarks (up to 8 runs per task on 525 cases, 3 runs on 75 cases, and 1 run on 434 cases), plus GPT-5.4 and GPT-6.1 Sol as automated judges, with cited per-task API costs from $0.064 to $4.12; local GPU hardware is not stated in the paper.

Prerequisites

LLM agent harnesses, LLM-as-a-judge bias mitigation, factorial experimental design

Scope

modify existing code

The Gap

In Appendix A.2 and Table S2, agency benchmarks (CHI-Bench and ALE-CLI v1.0) evaluate GPT models with Codex, Gemini models with Gemini CLI, and all other 16 models with Claude Code (furthermore using Claude Code 2.1.280 for Opus 5.5 versus 2.1.170 for others). Separately, Appendix C.3 evaluates InclusionMed-DxFiles solely using GPT-6.1 Sol as the judge, even though GPT-6.1 Sol is a ranked top-5 competitor in the Index.

“For both CHI-Bench and ALE-CLI v1.0: Health and Medicine, we used Codex for the four GPT releases, Gemini CLI for the two Gemini releases, and Claude Code for the remaining 16 releases.”
From the paper, Appendix A.2, page 18. Matched against the PDF text.

Limitation

Confounding model families with vendor-specific CLI harnesses makes it unclear whether agency gaps (e.g., GPT-6 Luna ranking 8th in reasoning but 20th in agency) stem from the base model or the scaffold, while single-model judging with an evaluated model risks self-preference bias.

Proposed Solution

Run a controlled factorial evaluation using a single open-source, model-agnostic CLI and tool-use harness (e.g., an OpenHands or standardized ReAct/MCP scaffold) across all flagship families alongside vendor-native harnesses to decompose model versus harness variance. Replace single-model judging on diagnostic reasoning and rubric tasks with a cross-family judge panel (e.g., combining Claude, Gemini, and GPT judges with blinded outputs and human-adjudicated calibration checks).

Why This Matters

Isolating base-model agency from harness design and removing single-judge bias would make leaderboard comparisons fair across providers and prevent over-crediting proprietary scaffolding.

How to Get Started

How to test it: Measure the variance in CHI-Bench pass@1 explained by model versus harness (standardized open harness vs Codex/Gemini CLI/Claude Code) across six flagship models via two-way ANOVA, and measure judge self-preference bias on diagnostic cases using human physician ground truth. Conclude the confound is negligible if rank correlation across harnesses exceeds 0.95 and judge-family bias is under 2 percentage points.

  1. Implement a unified open-source CLI/MCP agent adapter that exposes identical prompts, bash/file tools, and retry logic to GPT-6 Astra, Claude Opus 5.5, Gemini 4 Argon, DeepSeek V4.1 Flash, and Qwen3.8 Max.
  2. Evaluate all selected models on CHI-Bench (75 tasks) and ALE-CLI v1.0: Health and Medicine (22 tasks) under the unified harness and compare against the Table S2 vendor-harness scores.
  3. Run a cross-family three-judge ensemble (GPT, Claude, Gemini) on a 100-case stratified sample of HealthBench Professional and open NEJM MGH diagnostic cases to quantify inter-judge agreement and family self-preference.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Research such as Better Deck or Different Judge? and Agreement by Construction explore the role of judges and harnesses in evaluation, specifically within niche or corporate contexts. While these acknowledge the impact of judging variance, they do not provide a standard cross-family framework specifically tailored to the clinical reasoning and multi-agent scaffold validation described here.

We searched OpenAlex and arXiv for “LLM agent harness confounding self-preference judge bias clinical benchmark” and “Cross-Family Harness Standardization and Multi-Judge Debiasing in Agentic Clinical Evaluation” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

InclusionMed: Benchmarking AI for Inclusive Healthcare

The InclusionMed Collaboration · 2026

InclusionMed introduces an open medical AI evaluation framework featuring the nine-cell InclusionMed Atlas (three healthcare roles by three AI capabilities) and the InclusionMed Index, which benchmarks 22 frontier language models across five audited evaluations. The study reveals substantial rank reversals across roles and capabilities, weaker clinical decision-making and action execution relative to patient communication, and up to 24-fold inference cost disparities among similarly scoring models.

Related Ideas

Is this a good idea?

Share onXLinkedIn