Included Health, an innovative healthcare provider focused on delivering comprehensive and personalized services to millions across the United States, recently shared an insightful case study published in NEJM Catalyst Innovations in Care Delivery. The study presents a practical framework designed for the deployment of patient-facing generative AI in healthcare, emphasizing safety and clinical oversight.
The paper outlines Included Health’s development and testing of a clinically regulated AI assistant. This assistant aims to provide reliable health advice for common inquiries while ensuring that complex or high-risk situations are escalated to qualified healthcare professionals. As more individuals seek quick medical responses from public AI platforms, the study advocates for a healthcare model specifically designed to assess risks and clearly communicate limitations, all while ensuring ongoing human supervision.
Dr. Ami Parekh, Chief Health Officer at Included Health, stated, "We believe that healthcare AI must surpass the standards set for general-purpose tools. Individuals require prompt assistance but equally need a trustworthy system that delivers accurate and safe recommendations. This means that from the outset, we must clearly define what AI can and cannot do, be aware of potential risks, and involve clinical expertise in critical moments."
The framework is aligned with Included Health’s AI + EQ strategy and is grounded in four fundamental principles:
1. Cross-functional governance that includes executive and clinical oversight from the beginning. 2. Proactive risk assessment and extensive pre-launch testing to detect potential failures and validate safety measures. 3. A three-tier risk classification system to categorize inquiries as standard, high-risk, or emergency-related. 4. Continuous involvement of clinicians to review interactions and enhance the AI system over time.
Throughout a seven-week pilot program, the risk-assessed assistant was implemented for 50% of Included Health’s patients, with daily manual reviews of all clinical interactions. The clinical evaluations indicated that this approach successfully upheld a high safety standard and decreased unnecessary referrals for standard-risk inquiries by 65%, allowing more patients to receive timely and clinically appropriate guidance. The patient experience remained high and comparable to that of the control group.
Dr. Ankoor Shah, Vice President of Clinical Excellence at Included Health, remarked, "We believe the framework outlined in this publication serves as a practical guide for healthcare organizations aiming to implement patient-centric AI while integrating safety, transparency, and necessary human oversight from day one. This is the kind of standard the healthcare sector should aspire to achieve, fostering governance that elevates the trustworthy deployment of these tools throughout the industry."
