Skip to content

Source paper

InclusionMed: Benchmarking AI for Inclusive Healthcare

The InclusionMed Collaboration
arXiv 2611.21218 · 202695 upvotes on Hugging Face

Research ideas from this paper (5)

Generated by AI. The paper’s authors did not write these.

Patient-Facing, Multilingual, and Multimodal Benchmark Expansion for Uncovered Atlas Cells

Build a multilingual, multimodal benchmark suite for patient-facing health conversations so medical AI evaluations reflect non-Western care settings and everyday patient needs.

InclusionMed: Benchmarking AI for Inclusive Healthcare
🔴Hard🔬Significant ResearchSingle GPU or lessNo close match found#Data & evaluation#Multimodal

Sensitivity-Aware Composite Indexing and Item-Response Calibration Across Heterogeneous Benchmarks

Calibrate composite benchmark indices using item-response modeling and leave-one-out weighting sensitivity so model rankings reflect true clinical capability rather than arbitrary portfolio weights.

InclusionMed: Benchmarking AI for Inclusive Healthcare
🟡Fairly easy💡Moderate InnovationSingle GPU or lessRelated work found#Data & evaluation#LLMs

Full-Trajectory Deployment Cost and Retry-Overhead Profiling for Medical AI Agents

Profile end-to-end agent deployment costs including failed retries, external judges, and sandbox overhead so healthcare teams can compare true operational affordability.

InclusionMed: Benchmarking AI for Inclusive Healthcare
🟡Fairly easy🔧Minor TweakSingle GPU or lessRelated work found#Efficiency#Agents

Cross-Family Harness Standardization and Multi-Judge Debiasing in Agentic Clinical Evaluation

Standardize agent execution harnesses across model families and evaluate with multi-judge ensembles so workflow scores measure model capability rather than tooling artifacts.

InclusionMed: Benchmarking AI for Inclusive Healthcare
🟠Medium💡Moderate InnovationSingle GPU or lessRelated work found#Agents#Data & evaluation

Precondition-Verified EHR Action Execution and Proactive Clinical Clarification Scaffolding

Augment clinical agents with proactive information-gathering prompts and schema-level action verification so models ask clarifying questions and execute electronic health record updates reliably.

InclusionMed: Benchmarking AI for Inclusive Healthcare
🟠Medium💡Moderate InnovationMultiple GPUsRelated work found#Agents#Reasoning

AI summary

InclusionMed introduces an open medical AI evaluation framework featuring the nine-cell InclusionMed Atlas (three healthcare roles by three AI capabilities) and the InclusionMed Index, which benchmarks 22 frontier language models across five audited evaluations. The study reveals substantial rank reversals across roles and capabilities, weaker clinical decision-making and action execution relative to patient communication, and up to 24-fold inference cost disparities among similarly scoring models.

How it works (AI summary)

The authors curated 388 candidate benchmarks via LLM-assisted benchmark cards, screened them through a six-criterion audit and independent reproduction of eight shortlisted suites, and retained four public benchmarks (HealthBench Professional, PhysicianBench, ALE-CLI v1.0: Health and Medicine, and CHI-Bench) alongside a private diagnostic suite (InclusionMed-DxFiles, 312 NEJM MGH cases). Models were evaluated via direct API calls, a FHIR EHR agent, and CLI harnesses (Codex, Gemini CLI, and Claude Code), scored equally (20% each) into the InclusionMed Index, annotated across eight task attributes, and compared against token-based API cost estimates.