Skip to content
AI-generated research idea🔴Hard🔬Significant ResearchNo close match found

Patient-Facing, Multilingual, and Multimodal Benchmark Expansion for Uncovered Atlas Cells

Build a multilingual, multimodal benchmark suite for patient-facing health conversations so medical AI evaluations reflect non-Western care settings and everyday patient needs.

Time

half a year or more

Compute level

Single GPU or less

Resources

API access to frontier LLMs for baseline evaluation, multilingual clinical experts and patient advocates for task/rubric authoring, and modest cloud compute for sandboxed personal health tools

What the paper used

Evaluated 22 commercial and open-weight LLM releases via API and agent harnesses across 1,034 tasks from five benchmarks (up to 8 runs per task on 525 cases, 3 runs on 75 cases, and 1 run on 434 cases), plus GPT-5.4 and GPT-6.1 Sol as automated judges, with cited per-task API costs from $0.064 to $4.12; local GPU hardware is not stated in the paper.

Prerequisites

clinical NLP, multilingual benchmark design, rubric-based evaluation, medical informatics

Scope

large-scale data collection

The Gap

In the first release of the InclusionMed Atlas and task-attribute audit (Section 6.1 and Appendix B.2.1), the five included benchmarks cover only three of nine role-capability cells, leaving all three cells for patients and the public, all three knowledge cells, and administrative reasoning empty. Across the 1,034 retained tasks, 93.23% are in English, 94.08% of tasks with an identifiable country are US-focused, and 99.81% are text-only.

“The current Atlas also includes no benchmark primarily intended to evaluate assistance for patients and the public.”
From the paper, Section 6.1, page 14. Matched against the PDF text.

Limitation

Current Index scores provide no evidence on how medical LLMs assist patients and the general public, handle non-English languages or low-resource clinical guidelines, or interpret medical images and multi-turn symptom descriptions.

Proposed Solution

Construct a multi-turn, multilingual, and multimodal evaluation benchmark targeting the empty patient-facing cells (knowledge, reasoning, and agency) by pairing clinically validated patient vignettes and personal health workflows across at least five non-US regions with simulated patient personas that withhold key history until queried. Incorporate non-English local clinical guidelines, laboratory images or skin/radiology attachments, and longitudinal state tracking using physician-authored rubrics plus programmatic tool-use checks.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

Filling these Atlas cells should make composite medical AI rankings representative of global patient populations rather than English-speaking US clinician and administrative workflows.

How to Get Started

How to test it: Test whether evaluating top Index models on a multilingual, multi-turn patient benchmark changes top-10 rankings compared to the current US/English clinician portfolio using Spearman rank correlation and subgroup pass-rate gaps. Stop or revise rubric calibration if inter-physician rubric agreement falls below Cohen's kappa of 0.70 on pilot cases.

  1. Audit the 79 multi-turn candidate benchmarks and non-English candidates from the 388-benchmark InclusionMed inventory via the released repository to catalog reusable task templates.
  2. Collaborate with clinicians in three non-US health systems to adapt 150 multi-turn patient-triage and care-navigation scenarios covering local guidelines and visual inputs (such as lab reports or dermatology photos).
  3. Implement a simulated-patient harness modeled on HealthBench Professional's rubric scorer with multi-turn information-gathering penalties and verify baseline runs on Gemini 4 Argon, Claude Opus 5.5, and GPT-5.6 Luna.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 10 results

While existing works like AgentClinic and Lingshu focus on general medical reasoning and multimodal evaluation, they do not incorporate the specific combination of patient-facing multilingual vignettes, simulated persona-driven history withholding, or non-US regional clinical guidelines. These listed studies provide foundations for medical benchmarking but leave open the specific synthesis of these diverse, patient-centered requirements.

We searched OpenAlex and arXiv for “multilingual multimodal patient-facing medical LLM benchmark multi-turn” and “Patient-Facing, Multilingual, and Multimodal Benchmark Expansion for Uncovered Atlas Cells” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

InclusionMed: Benchmarking AI for Inclusive Healthcare

The InclusionMed Collaboration · 2026

InclusionMed introduces an open medical AI evaluation framework featuring the nine-cell InclusionMed Atlas (three healthcare roles by three AI capabilities) and the InclusionMed Index, which benchmarks 22 frontier language models across five audited evaluations. The study reveals substantial rank reversals across roles and capabilities, weaker clinical decision-making and action execution relative to patient communication, and up to 24-fold inference cost disparities among similarly scoring models.

Related Ideas

Is this a good idea?

Share onXLinkedIn