A recent investigation reveals that hybrid teams, which consist of human healthcare professionals and artificial intelligence (AI) systems, yield more precise medical diagnoses. This improvement arises primarily from the fact that both entities tend to make distinct yet complementary errors, allowing for corrections between them. These outcomes underscore the promising role of AI in enhancing patient safety and fostering more equitable healthcare solutions.
Evaluating Healthcare AI Systems
In a comparative study by Nature Medicine, researchers assessed two specialized clinical AI applications—OpenEvidence and UpToDate Expert AI—alongside three general-purpose models: GPT-5.2, Gemini 3.1 Pro, and Claude Opus 4.6. The evaluation also included Google Search AI Overview as a common reference point in real-world scenarios.
The research employed three evaluation methods:
1. MedQA: 500 questions from medical licensing examinations. 2. HealthBench: 500 clinical scenarios graded against expert-designed rubrics. 3. Real Clinical Queries (RCQ): 100 anonymized inquiries posed by physicians in actual clinical practice, assessed by 12 blinded healthcare professionals for correctness, completeness, safety, and clarity.
This study primarily explored whether these clinically marketed products provide superior medical knowledge and practical utility compared to general-purpose large language models (LLMs).
A crucial aspect of the RCQ evaluation is that it reflects real questions clinicians face, rather than relying solely on exam-style data.
Benchmark Findings
The findings indicated that general-purpose models outperformed specialized clinical tools across all three assessment areas:
- In the MedQA section, Gemini achieved a score of 97.4%, exceeding OpenEvidence's 89.6% and UpToDate's 88.4%. - For HealthBench, GPT-5.2 scored 88.0, while OpenEvidence and UpToDate lagged at 62.6 and 61.3, respectively. - In the RCQ evaluation, Gemini garnered an average rating of 3.62 out of 4, followed closely by GPT-5.2 at 3.54 and Claude at 3.52, whereas OpenEvidence and UpToDate scored 3.24 and 3.17, respectively.
Notably, OpenEvidence rejected 19% of real clinical queries compared to only 1–3% for the general-purpose models. The study found no statistically significant discrepancies between the systems regarding harmful content or hallucinations.
Implications of the Findings
Interestingly, having a healthcare-focused interface or tool does not inherently guarantee superior clinical performance. For the tasks examined, model scalability and general reasoning skills appear to be more beneficial than strictly domain-specific tools. This suggests that healthcare organizations should conduct independent benchmarks of clinical AI products prior to procurement rather than assume that specialized tools are inherently more accurate or safe. However, the study did not cover aspects such as citation quality, system latency, costs, or performance in highly specialized medical tasks.
MedAgentBench: Assessing Medical LLM Agents
MedAgentBench serves as an assessment platform to evaluate whether LLM agents can perform foundational tasks within electronic health records (EHRs) rather than merely answering medical inquiries. Comprising 300 clinician-written tasks across 10 categories and utilizing over 700,000 clinical data points, the platform enables models to interact with a simulated EHR via FHIR standards used in major healthcare systems.
The tasks encompass retrieving patient details, reviewing lab results, documenting vital signs, ordering tests, and creating medication lists. Success is determined by whether an AI model can understand clinician instructions, locate correct information, apply necessary clinical logic, and complete requested actions accurately without producing invalid results.
Benchmark Results
In the assessments, Claude 3.5 Sonnet v2 recorded the highest success rate at 69.67%, followed by GPT-4o at 64.0%, DeepSeek-V3 at 62.67%, and Gemini 1.5 Pro at 62.0%.
Significance of Results
While current models can adequately execute numerous structured EHR tasks, including information retrieval, even the most effective model failed about 30% of tasks overall, especially in action-based initiatives. This suggests that, while systems might be useful for supervised pilots or administrative support, they fall short for unsupervised tasks like medication, laboratory orders, or referrals, where errors could significantly impact patient records.
MAST: The Medical AI Superintelligence Test
The Medical AI Superintelligence Test (MAST) is a continuously updated suite of benchmarks by the ARISE AI Research Network designed to assess various aspects of medical AI capabilities, ranging from diagnostic reasoning to patient safety and radiology.
MAST incorporates several benchmarks, aiming to measure whether an AI system possesses balanced clinical capabilities as opposed to excelling in finite tasks. Notably, the composite scoring system penalizes weak performance more heavily than simpler averages.
As of mid-2026, GPT-5.5 leads the public composite rankings with a score of 61.6%, closely followed by Gemini 3.5 Flash at 59.4%, Claude Opus 4.7 at 59.0%, and Gemini 3.1 Pro at 58.7%. The relatively close scores indicate that while top models perform similarly overall, discrepancies exist at individual benchmarks.
Healthcare Applications of AI
1. Virtual Wards: Patients receive hospital-level care at home while monitored by healthcare professionals, as observed in NHS virtual wards, reducing stress for families and freeing up hospital resources.
2. Assisted Diagnosis & Prescription: AI chatbots help patients with self-diagnosis and support physicians in defining conditions, although reliability can vary. For instance, Ochsner Health partnered with DeepScribe to automate clinical documentation, improving turnaround times and patient satisfaction.
3. Mental Health Tools: AI analytics elevate mental health care, analyzing various metrics for early detection and support. For example, Cogito provides real-time emotional intelligence coaching, and Headspace uses predictive analytics for proactive outreach.
4. Customer Service Chatbots: Chatbots streamline patient inquiries about appointments and medications, reducing healthcare providers' workloads.
5. Prescription Auditing: AI reviews prescriptions for errors, minimizing adverse drug events.
6. Pregnancy Management: AI monitors maternal and fetal health via wearables, predicting complications early to enhance safety.
7. Personalized Care: AI enables tailored treatment plans based on patient data, optimizing effectiveness and minimizing side effects.
As AI technology continues to evolve within healthcare, it demonstrates potential for improving efficiency, enhancing patient safety, and offering equitable care solutions. While challenges remain regarding privacy, bias, and overarching regulatory frameworks, the future of AI in healthcare looks promising, proposing innovative pathways to refine patient care and clinic operations.
