Patient-Facing, Multilingual, and Multimodal Benchmark Expansion for Uncovered Atlas Cells
Build a multilingual, multimodal benchmark suite for patient-facing health conversations so medical AI evaluations reflect non-Western care settings and everyday patient needs.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In the first release of the InclusionMed Atlas and task-attribute audit (Section 6.1 and Appendix B.2.1), the five included benchmarks cover only three of nine role-capability cells, leaving all three cells for patients and the public, all three knowledge cells, and administrative reasoning empty. Across the 1,034 retained tasks, 93.23% are in English, 94.08% of tasks with an identifiable country are US-focused, and 99.81% are text-only.
“The current Atlas also includes no benchmark primarily intended to evaluate assistance for patients and the public.”
Limitation
Current Index scores provide no evidence on how medical LLMs assist patients and the general public, handle non-English languages or low-resource clinical guidelines, or interpret medical images and multi-turn symptom descriptions.
Proposed Solution
Construct a multi-turn, multilingual, and multimodal evaluation benchmark targeting the empty patient-facing cells (knowledge, reasoning, and agency) by pairing clinically validated patient vignettes and personal health workflows across at least five non-US regions with simulated patient personas that withhold key history until queried. Incorporate non-English local clinical guidelines, laboratory images or skin/radiology attachments, and longitudinal state tracking using physician-authored rubrics plus programmatic tool-use checks.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Filling these Atlas cells should make composite medical AI rankings representative of global patient populations rather than English-speaking US clinician and administrative workflows.
How to Get Started
How to test it: Test whether evaluating top Index models on a multilingual, multi-turn patient benchmark changes top-10 rankings compared to the current US/English clinician portfolio using Spearman rank correlation and subgroup pass-rate gaps. Stop or revise rubric calibration if inter-physician rubric agreement falls below Cohen's kappa of 0.70 on pilot cases.
- Audit the 79 multi-turn candidate benchmarks and non-English candidates from the 388-benchmark InclusionMed inventory via the released repository to catalog reusable task templates.
- Collaborate with clinicians in three non-US health systems to adapt 150 multi-turn patient-triage and care-navigation scenarios covering local guidelines and visual inputs (such as lab reports or dermatology photos).
- Implement a simulated-patient harness modeled on HealthBench Professional's rubric scorer with multi-turn information-gathering penalties and verify baseline runs on Gemini 4 Argon, Claude Opus 5.5, and GPT-5.6 Luna.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
While existing works like AgentClinic and Lingshu focus on general medical reasoning and multimodal evaluation, they do not incorporate the specific combination of patient-facing multilingual vignettes, simulated persona-driven history withholding, or non-US regional clinical guidelines. These listed studies provide foundations for medical benchmarking but leave open the specific synthesis of these diverse, patient-centered requirements.
- AgentClinic: a multimodal benchmark for tool-using clinical AI agents · 2026 · 403 citations
- Lingshu: Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning · 2026 · 350 citations
- Multimodal and Agentic AI for Cognitive Accessibility and Linguistic Preservation · 2025 · 0 citations
We searched OpenAlex and arXiv for “multilingual multimodal patient-facing medical LLM benchmark multi-turn” and “Patient-Facing, Multilingual, and Multimodal Benchmark Expansion for Uncovered Atlas Cells” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
InclusionMed: Benchmarking AI for Inclusive HealthcareThe InclusionMed Collaboration · 2026
InclusionMed introduces an open medical AI evaluation framework featuring the nine-cell InclusionMed Atlas (three healthcare roles by three AI capabilities) and the InclusionMed Index, which benchmarks 22 frontier language models across five audited evaluations. The study reveals substantial rank reversals across roles and capabilities, weaker clinical decision-making and action execution relative to patient communication, and up to 24-fold inference cost disparities among similarly scoring models.
Related Ideas
- End-to-End Benchmarking of Native Scientific Perception Within Interactive Agentic WorkflowsSage S2.1 Flash Technical Report🟠Medium
- Domain-Expert Validation Framework for Automated Scientific IllustrationsAutoFigure-Edit: Generating Editable Scientific Illustration🔴Hard
- Multimodal and Full-Document Soundness Assessment Across Figures and AppendicesPaperCritic: Training AI Reviewers to Assess the Soundness of Papers🔴Hard
- Multilingual Layout and OCR GeneralizationMinerU: An Open-Source Solution for Precise Document Content Extraction🟡Fairly easy
- Controlled Multi-Agent Causal Benchmark for Autonomous Research HarnessesARIS: Autonomous Research via Adversarial Multi-Agent Collaboration🟠Medium