A recent commentary in the Journal of the Royal Society of Medicine highlights significant shortcomings in the current risk-based regulatory frameworks governing the use of Artificial Intelligence (AI) in healthcare. The article expresses concerns that these approaches may not adequately safeguard patients, potentially resulting in both over-treatment and under-treatment, as well as introducing bias against certain patient demographics.

The authors note that while AI and machine learning can improve the precision of clinical decision-making, serious doubts persist regarding their reliability, transparency, and susceptibility to bias. These issues, they argue, are insufficiently addressed by the European Union’s AI Act, which was enacted in 2025 and categorizes medical AI as "high risk," imposing stringent regulations on its developers and deployers.

According to the authors, the existing risk-centric regulatory model fails to consider three pivotal aspects: individual patient preferences, the broader systemic impacts of AI deployment, and the marginalization of patient voices within the regulatory framework. 

Patients have diverse perspectives on what constitutes acceptable accuracy, bias, and the integration of AI in their healthcare. Therefore, regulations should transcend merely ensuring system-wide safety and also empower individual rights and engagement in the decision-making process. Professor Thomas Ploug, the lead author and a specialist in Data and AI Ethics at Aalborg University in Denmark, emphasizes the importance of recognizing these variations in patient values.

The commentary advocates for specific rights for patients regarding AI-assisted diagnosis and treatment, including the ability to:

- Request explanations of AI-driven outcomes,
- Provide or revoke consent,
- Obtain second opinions, and
- Decline diagnoses or screenings based on non-consensual data.

The authors caution that without proactive involvement from various healthcare stakeholders—such as clinicians, regulators, and patient advocacy groups—these essential rights may be neglected amid the swift advancement of AI technologies in healthcare.

"AI is revolutionizing the healthcare landscape, but that transformation should not compromise patient independence and trust," Professor Ploug asserts. "We must now articulate the rights that will safeguard and empower individuals within an AI-infused healthcare environment."

This vital discussion on patient rights in the realm of AI-driven healthcare emphasizes that relying solely on risk-based regulatory measures is inadequate.

New commentary calls for AI regulations focused on patients in healthcare systems.

A sunlit hospital consultation room: a doctor in a white coat and an adult patient sit across a wooden table bathed in warm natural light. Between them lies a printed policy document titled “AI Healthcare Regulation,” a large red rubber stamp reading “PATIENT-CENTERED” pressed firmly across its center. Behind them, a sleek computer monitor displays a clean AI medical interface with patient charts. Through the window, leafy greenery glows softly, underscoring a hopeful, patient-first approach to AI regulation.

New commentary urges patient-centered AI regulation in healthcare systems - News-Medical

- Current AI regulatory approaches in healthcare fail to adequately protect patients' rights.  
- The EU's AI Act overlooks critical issues like patient preferences and systemic impacts.  
- Experts urge for patient rights to ensure autonomy, consent, and participation in AI use.  

New commentary published in the Journal of the Royal Society of Medicine warns that current risk-based regulatory approaches to Artificial Intelligence (AI) in healthcare fall short in protecting patients, potentially leading to over- and undertreatment as well as discrimination against patient groups.

The authors found that while AI and machine learning systems can enhance clinical accuracy, concerns remain over their inherent inaccuracy, opacity, and potential for bias which are not adequately addressed by the current regulatory efforts introduced by the European Union's AI Act.

Passed in 2025, the AI Act categorises medical AI as "high risk" and introduces strict controls on providers and deployers. But the authors argue this risk-based framework overlooks three critical issues: individual patient preferences, systemic and long-term effects of AI implementation, and the disempowerment of patients in regulatory processes.

Patients have different values when it comes to accuracy, bias, or the role AI plays in their care. Regulation must move beyond system-level safety and account for individual rights and participation." Thomas Ploug, lead author, Professor of Data and AI Ethics, Aalborg University, Denmark

The authors call for the introduction of patient rights relating to AI-generated diagnosis or treatment planning, including the right to:

request an explanation;

give or withdraw consent;

seek a second opinion; and

refuse diagnosis or screening based on publicly available data without consent.

They warn that without urgent engagement from healthcare stakeholders - including clinicians, regulators, and patient groups - these rights risk being left behind in the rapid evolution of AI in healthcare.

"AI is transforming healthcare, but it must not do so at the expense of patient autonomy and trust," said Professor Ploug. "It is time to define the rights that will protect and empower patients in an AI-driven health system."

The need for patient rights in AI-driven healthcare - risk-based regulation is not enough (DOI: 10.1177/01410768251344707) is by Dr Thomas Ploug, Rikke Frank Jørgensen, Hanne Marie Motzfeldt, Naomi Ploug and Søren Holm.

New commentary urges patient-centered AI regulation in healthcare systems

Every significant advancement in science starts with a fundamental inquiry: can this treatment enhance longevity, restore equilibrium, or significantly transform patient care? At Ipsen, we are adopting a similar methodology in the realm of artificial intelligence, questioning whether it can aid our teams in analyzing data in innovative ways, minimize repetitive tasks, and facilitate quicker, more informed decision-making focused on purposeful scientific outcomes.

To explore this potential, we have initiated a pilot program in collaboration with the Claude team specializing in healthcare and life sciences. This initiative aims to test theories within actual research and business processes, assess impacts, and gain insights swiftly while embedding a sense of responsibility from the ground up.

Launched in May 2026, this pilot involves more than 40 team members from Ipsen's Research & Development and Medical divisions, who are engaged in a specific set of use cases. These encompass external innovation evaluations, synthesizing scientific literature, generating knowledge, and enhancing clinical development workflows. The focus is on high-value tasks that require teams to sift through significant volumes of intricate data while consistently delivering high-quality results. Participants are utilizing resources such as Claude Science and Claude Code to explore the platform's capabilities.

When it comes to healthcare, the application of AI is just as critical as its functionalities. Accordingly, this pilot emphasizes strong governance, appropriate access controls, and responsible usage of these tools to ensure that teams engage with them ethically.

This initiative is not about adopting AI for its own sake. Ipsen's goal is to expedite the process of delivering medications to patients. The pilot serves as a deliberate examination of how AI can sustainably support this objective, grounded in scientific rigor. While we recognize that not every aspect of medication delivery can be hastened, optimizing decision-making processes and alleviating challenges in complex analytical tasks can lead to significant improvements.

Evaluating the theory: a targeted pilot for artificial intelligence in healthcare

A bright, sunlit hospital ward with a senior doctor and a nurse standing beside an adult patient’s bed, studying a tablet screen that shows colorful AI‐driven vitals and graphs. Behind them, a whiteboard reads “Pilot Study” surrounded by hand‐drawn charts and flow diagrams. The scene feels hopeful and focused, bathed in warm natural light.

Testing the hypothesis: a focused pilot for AI in healthcare - Ipsen

- Ipsen tests AI with Claude to enhance data analysis and decision-making in healthcare.  
- Over 40 Ipsen colleagues are involved in pilot focusing on high-value research activities.  
- The initiative ensures responsible AI use with governance and control measures in place.  

Every scientific breakthrough begins with a question: can this medicine extend life, restore balance, or meaningfully change what is possible for patients? At Ipsen, we are bringing that same discipline to artificial intelligence, asking whether it can help our teams analyse data in new ways, reduce rework and make better, faster decisions in pursuit of science with purpose.

This is the aim of our pilot with the Claude for healthcare and life science team: to test hypotheses in real research and business workflows, measure impact and learn quickly with responsibility embedded from the outset.

Launched in May 2026, the pilot brings together over 40 colleagues across Ipsen’s Research & Development and Medical organizations to work on a defined set of use cases spanning external innovation assessments, scientific literature synthesis, knowledge generation and clinical development workflows. The objective is to focus on high-value activities where teams routinely navigate large amounts of complex data and must produce consistent, high-quality outputs. Participants are exploring the platform using tools including Claude Science and Claude Code.



In healthcare, how AI is used matters as much as what it can do. This pilot is therefore being run with a strong focus on governance, appropriate access, ensuring the right controls are in place and that teams use these tools responsibly.



This is not AI for AI’s sake. Ipsen’s ambition is to bring medicines to patients sooner. This pilot is a focused test of where AI can support that ambition, responsibly and with scientific rigour. Not every step in the delivery of medicines to patients can be accelerated but improving how and when decisions are made and reducing friction in complex analytical work, can make a meaningful difference.

Testing the hypothesis: a focused pilot for AI in healthcare

For over a year, six schools operated by the UN Relief and Works Agency (UNRWA) have been cut off from access, while another six have shut down entirely, reported UN Spokesperson Stéphane Dujarric to reporters in New York.

Dujarric noted that ten schools in Area C of the West Bank have been abandoned due to ongoing settler violence and access limitations.

On Wednesday, UN Deputy Humanitarian Coordinator for the Occupied Palestinian Territory, Suzanna Tkalec, spearheaded a joint humanitarian effort visiting a former UN school in Gaza, which is currently sheltering 18 families displaced by conflict.

Dujarric emphasized that the living conditions for these families have been severely compromised due to limited humanitarian aid access. He stated, "Our humanitarian partners are dedicated to mobilizing resources to meet the community's most pressing needs."

While the Israeli military continues its actions in Gaza and the West Bank, humanitarian workers are steadfast in their commitment to provide essential assistance. In the last month, UN partner organizations have distributed over 378,000 essential items—including tarps, cleaning supplies, and water containers—to civilians in Gaza.

In June alone, the UN World Food Programme (WFP) extended food vouchers and cash assistance to around 286,000 individuals in the West Bank. Nonetheless, the UN raised concerns about potential shortages of necessary supplies as winter approaches without increased financial support. Dujarric warned, "Without additional funding, our dwindling stockpiles can't be replenished, putting vulnerable families at heightened risk, especially with winter approaching."

The World Health Organization (WHO) highlighted the necessity of establishing health-focused regulations for artificial intelligence (AI) during a press release on Wednesday. It urged that AI governance must work to alleviate health disparities rather than exacerbate them. This alert coincided with a high-level conference in Lisbon, co-hosted by the WHO and the Government of Portugal, where representatives from 37 countries gathered to discuss responsible AI implementation in healthcare.

According to an April report by WHO, while 74% of EU Member States utilize AI for diagnostic assistance, only 11% have developed a health-specific AI strategy and a mere 8% possess liability standards for AI mishaps. Dr. Hans Kluge, the WHO's regional director for Europe, commented on the global struggle for responsible AI governance, emphasizing the need for healthcare systems to prioritize patient needs over those who can afford the technology.

Dr. Kluge pointed out the urgency of AI regulation, noting that patients increasingly rely on chatbots for medical queries. He stated, "Regulating AI in health is challenging, but failing to do so carries risks for patients, trust, and equality."

In 2021, WHO released guidance on the ethical use of AI in healthcare, stressing the importance of employing the technology safely and ethically to benefit patients and communities. The conference aims to develop strategies that ensure equitable access to healthcare technology for underserved populations as AI becomes more prevalent.

At the recent 19th session of the UN Expert Mechanism on the Rights of Indigenous Peoples, UN human rights chief Volker Türk remarked that the transition to clean energy should not infringe upon the rights of Indigenous Peoples. He pointed out that Indigenous rights defenders frequently encounter threats, violence, and legal challenges, especially when opposing development initiatives affecting their lands.

Between 2023 and 2025, it is estimated that 15% of human rights defenders who are killed or forcibly disappeared will be Indigenous individuals, Türk reported. He called attention to cases such as Brooklyn Rivera, an Indigenous leader from Nicaragua who passed away in state detention, urging Member States to uphold Indigenous rights.

Türk also celebrated recent victories, such as an Indigenous group in Australia that achieved unprecedented compensation for mining activities carried out without their consent. He affirmed, "Rights are not distant goals; they are immediate obligations. Where these rights are protected, society becomes fairer and more resilient."

He highlighted that Indigenous Peoples are often at the forefront of efforts concerning biodiversity, climate resilience, and resource management, noting, "Sustainable peace relies on justice, participation, and the recognition of rights." He called for acknowledgment of Indigenous Peoples' rights to self-determination and land control, emphasizing the historical injustices they have faced.

Türk urged countries to support the UN Voluntary Fund for Indigenous Peoples and the Global Alliance for Human Rights, which promote Indigenous involvement in international discussions. "By working together, we can place human rights at the core of decision-making everywhere," he said.

Global News Summary: West Bank schools shut down, AI applications in healthcare, Indigenous rights issues

A triptych-style thumbnail:  
Left panel (dark theme): a deserted school entrance in the West Bank at dusk, its metal gate barred and weathered, a Palestinian flag draped over the fence, heavy clouds overhead.  
Center panel (natural light): a bright hospital ward where a doctor in scrubs reviews a glowing 3D brain scan on a sleek monitor, warm sunlight streaming through large windows.  
Right panel (natural light): an Indigenous elder in traditional dress standing atop a grassy hill at sunrise, holding a hand-woven rights banner as a small group of supporters listens.

World News in Brief: Schools closed in West Bank, AI in healthcare, Indigenous rights - UN News

- Six UNRWA schools in the West Bank remain closed due to settler attacks and access issues.  
- Humanitarian aid continues in Gaza despite military strikes, reaching thousands with essential supplies.  
- WHO urges global AI health regulations to ensure equitable access and safe patient care.  

For more than a year, six schools established by the UN Relief and Works Agency (UNRWA) have been inaccessible, and another six have been closed, UN Spokesperson Stéphane Dujarric told journalists in New York.

Ten schools in Area C of the West Bank were recently abandoned due to settler attacks and access restrictions, he added.

On Wednesday, Suzanna Tkalec, the UN’s deputy humanitarian coordinator for the Occupied Palestinian Territory, led a joint humanitarian mission to a former UN school in Gaza currently hosting 18 displaced families.

Mr. Dujarric said that according to the UN aid coordination office, OCHA, those families’ living conditions have been undermined by a lack of access to humanitarian aid.

“Our humanitarian partners are committed to mobilizing assistance to address the community’s most urgent needs,” he said.

Aid reaches thousands in the Occupied Palestinian Territory

Even as the Israeli military continues to strike Gaza and launch raids into the West Bank, humanitarian workers continue to provide lifesaving assistance.

The UN’s partner organizations have distributed more than 378,000 items, including tarpaulins, cleaning kits and jerrycans to civilians in Gaza in the past month.

In the West Bank, the UN World Food Programme (WFP) provided food vouchers and cash assistance to about 286,000 people in June.

The UN is concerned, however, that the required supplies may not be available by the winter season without increased support.

“Without additional funding, depleted stockpiles cannot be replenished, putting vulnerable families at even greater risk, particularly as humanitarians prepare for winter,” Mr. Dujarric said.

© WHO/Uka Borregaard

WHO urges countries to develop health-specific AI regulations

Artificial intelligence (AI) governance strategies must ensure the technology remedies health inequities in access and treatment, rather than amplifying them, the World Health Organization (WHO) said on Wednesday in a press release.

The warning comes as the UN continues its push to bring Member States together around global AI governance. The WHO and the Government of Portugal kicked off a high-level conference in Lisbon on Wednesday, hosting government representatives from 37 countries across all six WHO regions to share strategies for responsible AI use in healthcare.

In an April report, the WHO found that 74 per cent of European Union (EU) Member States already used AI to assist with diagnostics, but only 11 per cent had a health-specific AI strategy and just eight per cent had liability standards for AI in case something goes wrong.

“Every country in the world is wrestling with the same questions right now: how do we govern AI responsibly, how do we build the health workforce to use it safely, and how do we make sure it serves patients rather than just those who can afford the technology,” said Dr. Hans Kluge, WHO regional director for Europe.

Regulation push prioritizes patients

In the press release, Dr. Kluge said part of the reason AI regulation is so urgent is that patients are already using chatbots to ask about their symptoms.

“Regulating AI in health is hard,” he said. “But not regulating it is harder – measured in patients harmed, trust lost and inequalities widened.

The WHO released guidance on the ethical use of AI in healthcare in 2021, focusing on how it can be used safely and ethically to advance the interests of patients and communities

A key focus of the conference is to develop strategies to ensure that patients with less access to the healthcare system and to technology are not left behind as countries and providers adopt AI tools

“No country should have to navigate the AI transformation alone,” the WHO wrote.

© Unsplash/Gustavo Leighton

Indigenous rights activists face disproportionate dangers

The world’s transition to clean energy must not come at the cost of Indigenous Peoples’ rights, UN human rights chief Volker Türk said on Wednesday at the 19th session of the UN-appointed Expert Mechanism on the Rights of Indigenous Peoples.

Indigenous rights defenders often face intimidation, harassment, violence, criminalisation and prosecution, especially as they defend their land from development projects such as mining and large-scale agriculture.

Between 2023 and 2025, at least 15 per cent of the human rights defenders killed or forcibly disappeared around the world were Indigenous Persons, according to the human rights chief.

Mr. Türk highlighted the case of Brooklyn Rivera, an Indigenous leader in Nicaragua who died in state custody in May, and called for all Member States to respect Indigenous rights.

He also highlighted recent successes such as an Indigenous group in Australia that in May secured record compensation for mining conducted without their consent.

Tweet URL

“Indigenous Peoples’ rights are not theoretical aspirations for the future; they are obligations for the present, now,” Mr. Türk said. “And wherever those rights are respected, all of society becomes more just, more peaceful and more resilient.”

Highlighting Indigenous contributions

Indigenous Peoples around the world lead efforts on biodiversity conservation, climate resilience, resource management, peacebuilding and human rights.

“Durable peace requires inclusion, justice, meaningful participation at every stage and recognition of rights,” he said. “And this must involve recognition of Indigenous Peoples’ rights to self-determination and to control over their lands.”

Majority populations have historically dispossessed Indigenous communities of land and resources, denied them self-determination, degraded their environments, and threatened their governance systems and cultures, human rights chief noted.

Mr. Türk urged Member States to support the UN Voluntary Fund for Indigenous Peoples and the Global Alliance for Human Rights, both of which strengthen Indigenous participation in international fora.

“Together we can work to place all human rights at the centre of decision-making everywhere,” he said.

World News in Brief: Schools closed in West Bank, AI in healthcare, Indigenous rights

The healthcare sector stands as one of the most vital industries within our society and presents a significant opportunity for the integration of artificial intelligence (AI). This integration aims to enhance support for clinicians, improve patient outcomes, and streamline operations within health systems. Despite the rapid advancements of AI technology, implementing it effectively within real clinical environments continues to prove challenging. A recent study indicates that 77% of health systems identify underdeveloped AI tools as a primary obstacle to their integration. In response, Scale is collaborating with Mayo Clinic to create and implement AI solutions that are specifically tailored to enhance clinical care.

Transforming Healthcare Through AI

This partnership is setting the stage for a comprehensive shift toward AI-driven healthcare solutions, emphasizing the importance of reliability, precision, and practicality. Utilizing the Scale Generative AI Platform, our joint efforts are concentrating on three critical areas:

1. **Enhancing Clinical Processes**: By minimizing the time clinicians spend reviewing patient health records before initial consultations, healthcare professionals can allocate more focus to patient interaction rather than paperwork. Since the initiative began, healthcare providers have increased their one-on-one time with patients by an average of eleven minutes, all while sustaining a high standard of care.

2. **Increasing Patient Safety and Quality of Care**: The initiative aims to automate the identification of vital safety incidents, such as wrong-site surgeries or serious patient falls, which often get lost amid routine reporting. This automation reduces the need for extensive manual oversight by administrative teams.

3. **Maximizing Healthcare Workers' Licenses**: By alleviating administrative burdens, the collaboration enhances continuous patient care while laying the groundwork for future AI innovations.

Matthew R. Callstrom, M.D., Ph.D., who serves as the medical director of the AI program at Mayo Clinic, stated, “This initiative is fundamentally about leveraging innovation to provide patients with more timely, coordinated, and compassionate care. Our collaboration merges the strengths of both organizations to enhance care delivery and improve patient experiences.”

Jason Droege, CEO of Scale AI, praised Mayo Clinic’s commitment to patient-centered innovation, remarking, “Mayo Clinic exemplifies the future of patient-focused healthcare, and we are excited to partner with them in developing reliable AI systems that not only assist clinicians but also lead to improved patient outcomes.”

Paving the Way for AI in Healthcare

These targeted initiatives represent crucial early milestones in the application of AI to address real clinical challenges. We aim to deliver tangible benefits for healthcare providers and patients alike while establishing a robust foundation for future advancements in the field. Committed to maintaining privacy and compliance, all data will remain secure within Mayo Clinic's HIPAA-compliant framework. Furthermore, our systems are built with the highest standards for data security and regulatory compliance in mind. As our collaboration progresses, we are dedicated to advancing meticulously evaluated AI systems that enhance care delivery and transform the operations of the healthcare sector.

To discover more about how AI is reshaping healthcare, visit scale.com/enterprise/healthcare.

Scale AI and Mayo Clinic Revolutionizing Clinical Care

A sunlit hospital examination room with warm, natural light flooding through tall windows. A physician in scrubs gently taps a sleek glass touchscreen on a stainless steel desk, revealing a richly detailed 3D organ model with color-coded diagnostic overlays. The glass display bears a prominent Scale AI logo etched into its frame, while the Mayo Clinic shield hangs on the wall behind, softly illuminated by sunlight. The scene conveys a harmonious blend of cutting-edge AI and clinical expertise transforming patient care.

How Scale AI and Mayo Clinic Are Transforming Clinical Care - Scale AI

- AI is being deployed to improve clinical care and support healthcare systems effectively.  
- Collaboration with Mayo Clinic focuses on reliable AI solutions to enhance patient outcomes.  
- Initial efforts streamline clinical processes and enhance patient safety while ensuring data security.  

Healthcare is one of the most important industries in our society, and one of the most consequential places to apply AI. In practice, this means supporting clinicians, improving patient outcomes, and helping health systems operate more effectively. While AI has advanced quickly, deploying it in real clinical settings remains a challenge. In fact, 77% of health systems cite immature AI tools as a significant barrier to adoption based on a 2025 study. That’s why Scale is working with Mayo Clinic to develop and deploy AI applications designed to meaningfully improve clinical care.

Driving AI Transformation in Healthcare

The collaboration establishes a framework for AI transformation, focusing on solutions that are reliable, accurate, and helpful. Together, and powered by the Scale Generative AI Platform, we are prioritizing three key areas:

Streamlining Clinical Processes : Reducing clinician review time of patients' health records before the initial consultation, allowing doctors and nurses to focus on direct care rather than manual documentation. Since launch, doctors are spending an average of eleven more minutes with each patient while reliably maintaining an expert standard of care for patients.

: Reducing clinician review time of patients' health records before the initial consultation, allowing doctors and nurses to focus on direct care rather than manual documentation. Since launch, doctors are spending an average of eleven more minutes with each patient while reliably maintaining an expert standard of care for patients. Enhancing Patient Safety and Quality of Care : Automating the detection of critical safety events, such as wrong-site surgeries or serious falls, hidden within routine reporting noise, reducing administrative manual review.

: Automating the detection of critical safety events, such as wrong-site surgeries or serious falls, hidden within routine reporting noise, reducing administrative manual review. Helping Mayo Healthcare Workers Operate at Top of License: Reducing administrative tasks, improving 24/7 patient care, and establishing a scalable foundation for future AI applications.

“Ultimately, this work is about helping patients by using innovation to deliver more timely, coordinated and compassionate care,” said Matthew R. Callstrom, M.D., Ph.D., medical director, AI program at Mayo Clinic. “This collaboration combines each organization’s expertise to advance care delivery and patient experience.”

“Mayo Clinic has been a model for what patient-centered innovation should look like, and it’s why they’re at the forefront of modern healthcare. We’re proud to work alongside their teams to build reliable AI systems that support clinicians and deliver better outcomes for patients.” – Jason Droege, CEO of Scale AI.

Charting a New Path for AI in Healthcare

These initial areas of focus mark important early steps in applying AI to real clinical needs. This work is designed to deliver practical value for clinicians and patients today, while creating a foundation for broader innovation over time. All of this is grounded in privacy and compliance: data remains within Mayo Clinic's secure, HIPAA-compliant environment, and our systems are built from the ground up to meet the highest standards of data security and regulatory oversight. As our efforts evolve, we will continue advancing rigorously evaluated AI systems that support care delivery and improve how the healthcare industry operates.

Visit us at scale.com/enterprise/healthcare to learn more about how AI is transforming healthcare.

How Scale AI and Mayo Clinic Are Transforming Clinical Care

In the past year, venture capital investments in healthcare artificial intelligence (AI) have surged to nearly $18 billion. However, the realization of a substantial return on these investments remains elusive, with many projects only reaching the experimental phase. Approximately 83% of healthcare systems have initiated generative AI trials, but a mere 5 to 10% of these projects successfully expand to enterprise-wide implementation. Many face obstacles such as procurement challenges, integration hurdles, or limited departmental launches that fail to scale.

Contributors like UCSF Health, Kleiner Perkins, and Doerr Capital believe the issue lies not just with the technology itself but rather with where it's being developed. According to exclusive insights shared with Fortune, they are introducing UCSF Health Converge, an AI accelerator designed with a unique approach: partnering with only two to three companies annually, immersing them within UCSF’s clinical processes, IT frameworks, and operational teams from the outset.

Suresh Gunasekaran, the CEO of UCSF Health, candidly critiques the existing landscape, stating that most solutions coming out of traditional accelerators lack readiness for system-wide implementation. “That final mile is missing,” he explained, indicating a significant gap in the transitional process from pilot projects to widespread adoption.

UCSF’s initiative is not unmatched in the field; Mayo Clinic has successfully accelerated over 70 startups since 2022, providing access to patient data and expert insights in exchange for an equity stake. Similarly, the Cleveland Clinic made headlines last fall by partnering with Khosla Ventures, allowing portfolio companies to test their solutions on actual patients and healthcare providers. UCSF’s stance is clear: the process of developing a product within a healthcare setting is inherently distinct from merely testing it there.

The effectiveness of this new model, however, remains uncertain. Current details about the Converge initiative are sparse, including the absence of a defined equity arrangement or investment specifics from Kleiner and Doerr. Gunasekaran mentioned that the terms are intentionally flexible, tailored to each participating company.

Kleiner Perkins, which recently secured $3.5 billion across two funds with a strong emphasis on AI, will consider investments in Converge participants based on a consensus from its six-member partnership. “It wouldn’t just be up to me,” explained Mamoon Hamid, a partner at Kleiner Perkins, underscoring the collaborative decision-making process.

John Doerr of Doerr Capital, who is co-leading the program alongside Kleiner, acknowledged a pressing issue: 100 million Americans lack access to primary care, and the nation cannot solely rely on training more healthcare professionals to alleviate this crisis. “We have to leverage AI to elevate the roles of everyone within our healthcare teams,” he emphasized.

He draws a historical analogy to Genentech, the pioneering biotech company that originated in the 1970s with close collaboration between scientists and investors within Kleiner Perkins' offices. Doerr believes this form of engagement is vital for catalyzing the healthcare AI movement, stating, “We underestimated the challenge of changing human behavior—patients, caregivers, and specialists alike.” He argues that AI's true potential lies in enabling personalized care at scale, whether by motivating patients to adhere to their treatment plans or implementing seamless documentation systems that free healthcare professionals to spend more quality time with their families.

Kleiner Perkins, John Doerr, and UCSF Health believe healthcare AI is fundamentally flawed.

A dimly lit hospital corridor at night, a humanoid medical robot in a cracked white coat lies broken on glossy floor tiles, its chest panel flung open with tangled circuit boards and streaming red error code; on the far wall a large Kleiner Perkins logo glows in deep red, beside it a shadowed UCSF Health emblem above a clinic door; off to one side, a lone silhouette of John Doerr stands with arms crossed, surveying the wreckage, dark and dramatic mood.

Kleiner Perkins, John Doerr, and UCSF Health think healthcare AI has been built all wrong - Fortune

- Nearly $18 billion has been invested in healthcare AI, but many projects fail to scale.  
- UCSF Health Converge aims to embed AI companies directly within clinical workflows from inception.  
- The initiative seeks to address primary care access issues using AI rather than just training more providers.  

Venture capital has poured nearly $18 billion into healthcare AI in a single year. The return on that investment, by most accounts, is a smorgasbord of pilots. Roughly 83% of health systems are running generative AI experiments—and only 5 to 10% of those ever reach enterprise-wide deployment. The rest die in procurement, drown in integration reviews, or launch in two departments and quietly stop expanding.

UCSF Health, Kleiner Perkins, and Doerr Capital don’t think that’s purely a technology problem. They’re betting that the technology is being built in the wrong place.

The trio, Fortune learned exclusively, is launching UCSF Health Converge, a health AI accelerator built around a deliberately radical constraint: two to three companies a year, embedded inside UCSF’s actual clinical workflows, IT systems, and operational teams from day one. UCSF Health CEO Suresh Gunasekaran is blunt about what the program is pushing back against. “The vast majority of solutions, when they exit the accelerator and come to pitch us, are not ready to implement at the system level,” he told Fortune. “That final mile is missing.”

It’s a direct shot at an industry that has spent the better part of a decade building healthcare AI from the outside looking in. And UCSF isn’t the first to try a different approach. Mayo Clinic has run more than 70 startups through an accelerator program since 2022 that gives companies access to its patient data and expert feedback in exchange for a stake in the company. Cleveland Clinic struck a deal with Khosla Ventures last fall that lets the firm’s portfolio companies bring their products in and test them on real patients and providers. UCSF’s argument is that testing a product inside a hospital is fundamentally different from building it there.

Whether the model actually delivers is an open question, and the details Converge has put on the table so far don’t exactly answer it. There’s no disclosed equity structure—meaning it’s unclear whether UCSF gets an ownership stake in the companies it helps build—no stated program length, and no specific dollar commitment from Kleiner or Doerr. Gunasekaran described the terms as “flexible” and dependent on each company’s situation.

Kleiner, which raised $3.5 billion across two new funds in March with a heavy focus on AI, will decide whether to invest in Converge companies the same way it decides everything else—through its full six-person partnership. “It wouldn’t just be up to me,” Mamoon Hamid, partner at Kleiner Perkins, told Fortune.

John Doerr, whose personal vehicle Doerr Capital is co-anchoring the program alongside Kleiner, put the stakes in terms that are hard to dismiss: 100 million Americans currently have zero access to primary care, and the U.S. isn’t going to train its way out of that shortage. “We’re not going to be able to address that need by training and hiring more doctors,” Doerr told Fortune. “We’ve got to use AI to raise the license level at which everyone in our healthcare teams participates.”

He sees a historical parallel in Genentech—the biotech pioneer that was essentially born inside Kleiner Perkins’ offices in the 1970s, with scientists and investors building in the same room rather than across a procurement process. That proximity, he argued, is what made the biotech revolution real.

“We underestimated the challenge of changing human behavior—and by humans, I mean patients, caregivers, neurosurgeons,” he told Fortune. What AI changes, in his telling, is personalization at scale: the nudge that gets a patient to take their medication, the ambient note-taking system that sends a physician home to their family instead of their Epic charts.

Kleiner Perkins, John Doerr, and UCSF Health think healthcare AI has been built all wrong

Sheba Medical Center's utilization of AISAP's AI technology has revolutionized cardiac diagnosis, reducing the time it takes to perform bedside assessments to less than five minutes. 

On July 15, 2026, it was announced that AISAP, a point-of-care ultrasound (POCUS) platform cleared by the FDA, has been highlighted as Case Study #5 in a report by the World Health Organization titled "Bridging Theory and Practice." This report provides insights into the practical applications of artificial intelligence in healthcare systems around the globe. AISAP's recognition among numerous evaluated AI companies underscores its impactful collaboration with Sheba Medical Center, a leader in global health care.

The focus of this case study is on the implementation of AISAP's technology in both the internal medicine and emergency departments at Sheba. Between July 2022 and December 2023, AI-enhanced cardiac ultrasounds were conducted on 660 patients by physicians who lacked formal training in sonography. Significant findings from the research, published in the Mayo Clinic Proceedings: Digital Health, indicate that the AI-enhanced Focused Ultrasound (FoCUS) identified clinically important cardiac issues in 193 patients (about 29%). This included 147 new diagnoses (22%) that had not been previously documented, which led to medication adjustments for 49 patients (7%) and 9 patients (1.4%) undergoing interventional procedures. In a more focused group of 117 patients, the FoCUS reached an impressive 100% sensitivity for significant left ventricular dysfunction and maintained a specificity of 96.9%. For aortic stenosis, it also achieved 100% sensitivity with 87.6% specificity when validated against formal echocardiography.

Robert Klempfner, the Chief Medical Officer and Co-Founder of AISAP, stated, "This recognition by the World Health Organization confirms our experience at Sheba: AI-guided POCUS can deliver high-level cardiac diagnostics to clinicians at the patient’s bedside in mere minutes instead of hours. Validated AI is transforming how we provide rapid, expert care across different hospital departments."

The WHO's case study relayed findings from Sheba's clinical study, reporting that over 30% of cases resulted in changes to treatment plans, accelerated discharges, or care transitions, with scans frequently concluding in under five minutes. By the second quarter of 2025, the AISAP platform had facilitated more than 3,000 scans across six departments and the emergency room, with 150 clinicians trained to integrate it into their everyday practices.

Having received FDA clearance in September 2024 (510(k) K234141) for the AISAP CARDIO V1.0 system, AISAP was validated based on a substantial multicenter study in the US that indicated sensitivities and specificities exceeding 93% for identifying significant valvular and ventricular issues, as confirmed by an independent core lab at Mass General Brigham. Additionally, Sheba has documented further operational results from the extensive rollout of the platform, beyond the results of the peer-reviewed study.

The full WHO report, "Bridging Theory and Practice," can be accessed online for more in-depth insights.

AISAP is redefining diagnostic standards through FDA-cleared algorithms integrated into PocusOS™, its AI system designed for use with any point-of-care ultrasound device. The platform sees application in academic medical centers, community facilities, and cardiology practices worldwide, and counts research institutions such as Stanford, Mayo Clinic, Mass General Brigham, Innova, and Jefferson Health among its collaborators. For additional information, visit aisap.ai. 

Original content can be viewed for multimedia downloads through available press sources.

AISAP Highlighted as a Case Study in Recent WHO Report on AI in Healthcare

A sunlit modern research office: a polished wooden table holds an open WHO report titled “AI in Healthcare” with the WHO emblem on its cover. Beside it, a sleek glass plaque etched with the bold AISAP logo catches shafts of morning light through floor-to-ceiling windows. In the softly blurred background, a laptop shows colorful patient data charts and medical graphs, hinting at the innovative case study.

AISAP Named a Featured Case Study in New WHO Report on AI in Healthcare - Morningstar

- AISAP's AI platform enables bedside cardiac diagnosis in under five minutes at Sheba Medical Center.  
- The deployment resulted in over 30% of cases leading to treatment changes or early discharge.  
- AISAP received FDA clearance in 2024 for its highly sensitive cardiac diagnostic technology.

Sheba Medical Center deployment cited for cutting bedside cardiac diagnosis to under five minutes

NEW YORK, July 15, 2026 /PRNewswire/ -- AISAP, the FDA-cleared AI platform for point of care ultrasound (POCUS), has been featured as Case Study #5 in the World Health Organization's new report, "Bridging Theory and Practice" (page 21), which examines real-world applications of artificial intelligence in health systems worldwide. AISAP's inclusion, among thousands of AI companies evaluated globally, highlights its work with Sheba Medical Center, one of the world's leading hospitals.

The case study centers on AISAP's deployment within Sheba's internal medicine and emergency departments, where 660 patients were evaluated between July 2022 and December 2023 using AI-enhanced, AI-guided cardiac ultrasound performed by physicians without formal sonography training. Results, published in Mayo Clinic Proceedings: Digital Health (2026;4(2):100355), showed that AI-enhanced FoCUS identified clinically relevant cardiac findings in 193 patients (29%), including 147 previously undocumented diagnoses (22%), and prompted medication changes in 49 patients (7%) and interventional procedures in 9 patients (1.4%). In a diagnostic-accuracy subset of 117 patients validated against formal echocardiography, AI-enhanced FoCUS showed 100% sensitivity and 96.9% specificity for detecting significant left ventricular dysfunction, and 100% sensitivity and 87.6% specificity for aortic stenosis.

"Being recognized by the World Health Organization validates what we've seen firsthand at Sheba: AI-guided POCUS can bring specialist-level cardiac diagnostics to any clinician, at any bedside, in minutes rather than hours. This is what validated AI can deliver, rapid, expert-level care directly to patients, across hospital departments," said Robert Klempfner, Chief Medical Officer & Co-Founder at AISAP.

The WHO case study cited results from a prospective clinical study at Sheba: more than 30% of cases led to treatment changes, early discharge, or care transitions, with scans completed in under five minutes. As of the second quarter of 2025, the platform had been used for more than 3,000 scans across six hospital departments and the emergency room, with 150 trained clinicians actively using it in routine clinical workflows.

AISAP received FDA clearance in September 2024 (510(k) K234141) for its AISAP CARDIO V1.0 system, based on a distinct US multicenter study that found more than 93% sensitivity and specificity for detecting significant valvular and ventricular abnormalities, validated against an independent core lab at Mass General Brigham. Sheba has also reported additional operational results from its enterprise-wide rollout of the platform, separate from the peer-reviewed clinical study.

The WHO report, "Bridging Theory and Practice," is available in full at iris.who.int/items/28faa031-ee08-4c06-b55b-233dba8ca500.

About AISAP AISAP is redefining the standard of care with FDA-cleared diagnostic algorithms embedded in PocusOS™, an AI operating system for any point of care ultrasound device. AISAP is used across academic medical centers, community hospitals, and cardiology clinics worldwide, including research partners such as Stanford, Mayo Clinic, Mass General Brigham, Innova and Jefferson Health. Learn more at aisap.ai.

View original content to download multimedia:https://www.prnewswire.com/news-releases/aisap-named-a-featured-case-study-in-new-who-report-on-ai-in-healthcare-302826203.html

SOURCE AISAP

AISAP Named a Featured Case Study in New WHO Report on AI in Healthcare

In a recent episode of the medical drama "The Pitt," AI-driven speech-to-text technology took center stage, highlighting its potential in the healthcare sector. The character Dr. Baran Al-Hashimi proclaimed, "Studies show that you can spend 80% less time charting," while demonstrating a new AI tool to her fellow doctors. However, her colleague voiced skepticism when the AI misidentified a patient's medication, raising a pertinent question: Are the benefits of AI tools in medicine worth the inherent risks?

Nelly Elsayed, an associate professor at the University of Cincinnati, sought to address this concern and propose strategies for developing safe and effective AI systems in medical documentation. Her recently published paper, titled "Socio-technical risks of clinical speech-to-text systems: Transparency, privacy, and reliability challenges in AI-driven documentation" in the International Journal of Medical Informatics, delves into existing research, ethical considerations, and regulatory frameworks to assess the rapid advancement of AI against the need for oversight.

While the quality of AI tools continues to improve, Elsayed emphasized that additional factors must be addressed to enhance their efficiency and transparency. She identified five primary risks associated with clinical speech-to-text applications:

1. Inconsistent practices in disclosure and consent
2. Reduced effectiveness when processing accented and disordered speech
3. Background noise in clinical environments compromising AI accuracy
4. Insufficient human review of AI-generated text, leading to unchecked errors
5. Ambiguity regarding accountability for mistakes: Is the responsibility with the software or the clinician?

According to Elsayed, incorporating human oversight in the review process before finalizing data could significantly mitigate these issues. "We need to have a human in the loop to verify that the text accurately reflects what was spoken," she stated, underscoring the importance of thorough checks rather than limiting reviews to just the initial statements.

She also pointed out that various real-world elements can hinder an AI's ability to record accurately. Most AI systems are trained in idealized environments, devoid of the typical distractions found in busy clinical settings, such as machines beeping and conversations among healthcare providers. For AI to prove reliable in practice, it must be trained to understand diverse accents and speech variations.

Additionally, ensuring that clinicians receive adequate training on the software before implementation is crucial. "The organization developing the system must provide clear guidelines to doctors on effective use, potential pitfalls, and what to monitor," Elsayed advised.

In summary, while AI speech-to-text technologies hold great promise for streamlining medical documentation, careful attention to transparency, human oversight, and real-world application is essential for their successful integration into healthcare practices.

Study reveals that AI speech-to-text requires greater regulation in healthcare.

A dim hospital transcription office at night bathed in warm lamp light. On the desk, a clipboard holds AI-generated medical notes with glaring red mistakes. A doctor in a white coat frowns, pen poised over the page, while a regulatory inspector’s gloved hand holds a magnifying glass over the errors. Heavy shadows frame the scene, evoking urgent oversight.

AI speech-to-text needs stronger oversight in healthcare, study finds - News-Medical

- AI speech-to-text tools can save time in medical documentation but pose significant risks.  
- Key risks include inconsistent practices, performance issues, and lack of human oversight.  
- Human review and proper training can improve reliability and transparency of AI systems.

AI speech-to-text had a moment in the spotlight on the medical drama series, "The Pitt," just earlier this year.

"Studies show that you can spend 80% less time charting," says character Baran Al-Hashimi, a doctor showing her peers a new AI speech-to-text tool. But a skeptical colleague raised concerns after the tool misheard the name of a patient's medication.

While the show is fictional, it poses a real question: Do the risks of these AI tools outweigh the reward?

Nelly Elsayed, associate professor at the University of Cincinnati, wanted to not only answer this question but find specific solutions to advancing safe and reliable AI systems in medical documentation.

Her paper, "Socio-technical risks of clinical speech-to-text systems: Transparency, privacy, and reliability challenges in AI-driven documentation," was just published in the International Journal of Medical Informatics. The study examined a multitude of existing research, ethical guidelines and government regulations to spot how AI's adoption is outpacing its oversight.

These AI tools, the quality is improving, but there are other aspects we have to care about to make them more efficient and transparent." Nelly Elsayed, associate professor, University of Cincinnati

Five key risks of clinical speech-to-text:

Inconsistent disclosure and consent practices

Decreased performance for accented and disordered speech

Extraneous noises at clinical facilities lowering the accuracy of AI

Lack of human review over AI-generated text leading to unchecked mistakes

Unclear accountability for errors: Is the software or the clinician responsible?

Elsayed said that simply having a human review data before it is finalized can reduce a fair amount of these concerns.

"We need to have a human in the loop to check whether the text is exactly what has been spoken," Elsayed explained. "And that test needs to be done for the entire text, not just for the first couple statements."

Elsayed also found that there are a multitude of factors in real-life settings that can throw off an AI's ability to record properly. AI is often trained in an "ideal" setting, with none of the bustle of a real doctor's office like machines beeping and doctors chatting. Without training large-language models in specific scenarios, for different accents and for speech disorders, they cannot be reliable in real life.

Another way to curb error is to train the clinicians on the software before they adopt it in their practices.

"The organization developing the system needs to give guidelines for the doctor, what they can use, what they cannot use and what to look out for," she said.

AI speech-to-text needs stronger oversight in healthcare, study finds

Artificial intelligence (AI) has emerged as a transformative force within the healthcare sector, showcasing remarkable capabilities that enhance operational efficiency.

Over recent years, AI applications have significantly streamlined processes such as coding, utilization management, and reimbursement decisions, all of which were once handled manually. For healthcare professionals facing heavy workloads, this innovative technology has alleviated much of the administrative burden. Tools that transcribe speech into text and create draft notes have proven particularly valuable, allowing providers to focus more on patient care while enhancing the accuracy of medical documentation.

As AI becomes more integral to daily healthcare delivery and administrative functions—including tasks like determining coverage, reviewing medical necessity, and ensuring claims accuracy—organizations in the sector must prepare for an increase in regulatory scrutiny. This oversight may come from both state and federal authorities, necessitating that healthcare entities validate their AI-driven payment decisions align with guidelines established by public and private payers.

The rapid proliferation of AI in healthcare has also introduced complex legal challenges, as highlighted in discussions by the National Law Review. These challenges include privacy risks, the cost of data storage and review, concerns about record integrity, and billing complications. To address these issues, a "human-in-the-loop" approach is recommended, where human involvement in the AI process is essential for maintaining accuracy and ethical standards.

A significant impediment to AI's progress is the legal framework's inability to keep pace with its rapid development. Currently, the United States lacks an overarching federal AI legislation, leaving healthcare organizations that utilize AI technology in a precarious position amidst a fragmented regulatory landscape. Recent trends indicate a rise in regulatory activity, with the Department of Justice (DOJ) showing a willingness to apply aspects of the False Claims Act (FCA) regarding AI’s role in healthcare reimbursement, as noted by Morgan Lewis.

In addition, states such as Texas and California are rolling out healthcare regulations that limit AI's use in determining medical necessity and impose requirements for substantial human oversight in clinical decisions. Many states now demand explicit notice and consent for recording communications, directly impacting how AI tools can be deployed in healthcare contexts.

Concerns arise when decisions around claim determinations are automated and human judgment is diminished or absent altogether. This may lead to compliance issues as regulators probe whether AI systems create undue financial incentives, result in systematic errors like upcoding or wrongful denials, or obscure clinical accountability. For instance, inaccuracies in AI-generated transcriptions could lead to incorrect coding and billing, escalating the risk of overpayment if not properly scrutinized by a human.

As AI tools become ever more embedded in healthcare delivery, regulatory enforcement related to these technologies is expected to intensify throughout the decade. Agencies such as the Centers for Medicare & Medicaid Services (CMS) and the Health and Human Services (HHS) Office of Inspector General are likely to apply established fraud and abuse laws to their oversight, as suggested by Healthcare IT News.

Jeff Wurzburg, a healthcare partner with Norton Rose Fulbright, emphasized that utilizing AI does not absolve providers or health plans from liability for claims submitted to federal healthcare programs. The shift toward automation could lead to systemic errors and other issues that are ripe for scrutiny under the False Claims Act, as regulators increase their focus on potential risks posed by AI.

As healthcare organizations embrace AI, they must remain vigilant about compliance and privacy risks associated with data use and the management of Protected Health Information (PHI). To navigate this evolving landscape, organizations should consider several compliance strategies:

1. Establishing an AI governance committee that brings together legal, technology, and business stakeholders to oversee AI initiatives and conduct audits for bias and fairness.

2. Engaging in proactive legal oversight to ensure that AI operations are recognized as critical from a compliance standpoint, particularly as they may have implications in future litigation.

3. Keeping up-to-date with the rapidly changing regulatory landscape at both state and federal levels, monitoring potential shifts in legislation or enforcement actions.

4. Enhancing transparency regarding automated decision-making processes facilitated by AI technologies, ensuring that public disclosures align with the actual capabilities of the systems in place. 

In summary, while AI's role in healthcare holds promising potential, it also introduces significant regulatory and legal challenges that healthcare organizations must effectively manage.

Regulators are expected to establish laws regarding AI usage in healthcare. Here's how to get ready.

A sunlit hospital policy office bathed in warm morning light. At a polished wooden conference table, a doctor in a white coat carefully thumbs through a thick dossier stamped “AI Healthcare Regulations.” Opposite, a regulator’s hand brings down a small wooden gavel to stamp “APPROVED” on a law document next to a pair of brass scales. Nearby, a sleek medical AI unit with a softly glowing interface stands ready. Morning sunbeams filter through tall arched windows, casting hopeful highlights on this pivotal moment of preparation for new AI healthcare laws.

Regulators are likely to pass laws on AI use in healthcare. Here’s how to prepare - Compliance Week

- AI tools are significantly reducing administrative workloads for healthcare providers and improving documentation accuracy.  
- Heightened regulatory oversight is expected as AI becomes integral to healthcare decision-making.  
- Legal and compliance challenges arise from AI use, including privacy concerns and potential inaccuracies.

Artificial intelligence tools hold breathtaking potential in the healthcare ecosystem.

In just a few short years, AI has already greatly expedited functions such as coding, utilization management, and reimbursement determinations that were previously executed manually.

For overworked healthcare providers worldwide, the next-generation technology has been a godsend in reducing their administrative workload—transcription tools that convert speech to text and generate draft notes have proven to be particularly facilitative—and allowing more time for actual patient care. For good measure, such AI-powered administrative tools oftentimes bolster documentation accuracy within the realm of medical recordkeeping.

David Ostrowsky

But as AI technology becomes more ingrained in the fabric of everyday care delivery and healthcare administration by underpinning coverage decisions, medical necessity reviews, and claims accuracy determinations, healthcare organizations need to brace for heightened regulatory oversight and enforcement of their various AI uses—both at the state and perhaps even federal level.

In doing so, such institutions will need to ensure that AI-fueled payment decisions are justifiable under both public and private healthcare payer roles. Meanwhile, the rampant use of AI in the healthcare arena has ushered in a wave of thorny legal issues, according to the National Law Review, including privacy exposure, storage and review costs, record-integrity concerns, and billing risk.

What is required to address these concerns is a “human-in-the-loop” methodology, in which living and breathing people are actually somewhat involved in the AI workflow to safeguard accuracy and make ethical decisions.

The current AI legislation patchwork

One of the most glaring drawbacks to AI’s meteoric evolution is that legal frameworks have not been able to adapt quickly enough to properly govern the technology.

While there may not be an all-encompassing federal AI legislation framework in the United States, healthcare companies deploying AI tools for the aforementioned functions still face significant risk within an enforcement environment remaining largely convoluted and disjointed.

In fact, regulatory and litigation activity is escalating in many areas. For example, the Department of Justice (DOJ) has appeared inclined to apply tenets of the False Claims Act (FCA) towards adjudicating the deployment of AI tools in healthcare reimbursement activity, according to a Morgan Lewis blog post.

Some state legislatures are scrambling to pass AI enforcement laws grounded in consumer protection, antitrust, and false claims statutes.

Two large states, Texas and California, recently imposed healthcare-focused regulations that restrict the use of AI in medical necessity determinations and mandate significant human oversight in clinical decision-making processes, according to another blog post by Morgan Lewis.

Several states have established notice and consent requirements for recording communications, while others are tackling the deployment of AI in healthcare interactions more directly. Several states have mandated that licensed professionals supply notice of the recording and obtain both verbal (on the recording) as well as written consent prior to utilizing AI tools to record conversations, .

Furthermore, with no overarching nationwide AI legislation in place, some federal agencies are pivoting towards current statutes to police AI activity on the healthcare front, according to a Holland & Knight blog post.

AI tools impacting coverage determinations and clinical decisions

When claim determinations are automated and human judgment is muted—or in some cases, completely absent—organizations may open themselves up to a series of liability issues, as state regulators and perhaps even attorneys general will want to closely scrutinize whether AI tools embed improper financial incentives, produce systemic upcoding or denials, or obscure clinical responsibility.

One of the most glaring examples would be AI-generated transcriptions that fail to accurately and thoroughly record conversations, leading to incorrect coding, billing, and utilization review. In the event that AI-produced language is included in a patient’s medical record, without appropriate human review and ultimately undergirds an inaccurate claim, overpayment issues could very well ensue, according to the National Law Review.

Bracing for heightened enforcement around AI use in healthcare

Regulatory enforcement surrounding the AI healthcare landscape may be in its nascent stages, but with AI tools becoming increasingly more integrated in healthcare operations, it is naturally expected to amplify in the balance of the decade.

More specifically, the Centers for Medicare & Medicaid Services (CMS), Health and Human Services (HHS) Office of Inspector General, and the Department of Justice are projected to implement longstanding fraud and abuse laws in their oversight efforts, according to Healthcare IT News.

Jeff Wurzburg, healthcare partner at Norton Rose Fulbright, a prominent global law firm, recently told Healthcare IT News, “The use of AI does not shift liability away from providers or health plans submitting claims to federal healthcare programs. To the contrary, large‑scale automation raises the risk of systemic errors, such as embedded upcoding, inappropriate denials or algorithmic bias toward revenue optimization—all of which are fertile ground for False Claims Act scrutiny by DOJ and oversight by CMS and the HHS‑OIG.”

Moreover, as AI becomes incrementally more embedded in clinical and operational work, health systems may encounter greater HIPAA-related risk from opaque data use, unauthorized release of Protected Health Information (PHI), and a disconnect between AI vendor practices and well-established privacy and security requirements.

Important considerations for healthcare organizations using AI

For healthcare organizations trying to drive forward innovation by developing new methods for deploying AI, it may be prudent to consider the following compliance measures:

Bolster oversight measures : Consider initiating an AI governance committee that incorporates legal, technology, and business stakeholders. Also, for AI-powered claim review processes, it may be worthwhile to conduct bias and fairness audits while subsequently making records of the results.

: Consider initiating an AI governance committee that incorporates legal, technology, and business stakeholders. Also, for AI-powered claim review processes, it may be worthwhile to conduct bias and fairness audits while subsequently making records of the results. In-house counsel responsibilities : In conjunction with an AI governance committee, the in-house counsel should also view AI activity as an important legal and compliance matter. In other words, AI is not merely within the domain of technology and business, and an organization’s in-house counsel needs to bear in mind that AI tools can factor into litigation down the road. Also, it would be prudent for in-house counsel to make sure that Business Associate Agreements (BAAs) clearly delineate breach notification procedures among other compliance matters.

: In conjunction with an AI governance committee, the in-house counsel should also view AI activity as an important legal and compliance matter. In other words, AI is not merely within the domain of technology and business, and an organization’s in-house counsel needs to bear in mind that AI tools can factor into litigation down the road. Also, it would be prudent for in-house counsel to make sure that Business Associate Agreements (BAAs) clearly delineate breach notification procedures among other compliance matters. Stay current on regulatory developments : It’s critical to remember that at both the state and federal levels, the AI regulatory landscape remains very fluid. There is constant change, and for companies with a national reach, there’s the added pressure of monitoring evolving state-level regulations while also staying up to date on both federal agency enforcement activity and executive actions.

: It’s critical to remember that at both the state and federal levels, the AI regulatory landscape remains very fluid. There is constant change, and for companies with a national reach, there’s the added pressure of monitoring evolving state-level regulations while also staying up to date on both federal agency enforcement activity and executive actions. Clearly articulate disclosures: Given the current environment, many courts and regulators are now expecting greater transparency into automated decision-making via AI technology. Particularly regarding AI-generated claim review decisions, public disclosures about AI activity should be in concert with actual system capabilities.

David Ostrowsky serves as the Manager of Corporate Communications for The Phia Group, a healthcare cost containment company headquartered in Canton, Massachusetts. He has previously written for The Self-Insurer, Compliance Today, and DC Journal, as well as other healthcare publications.

Regulators are likely to pass laws on AI use in healthcare. Here’s how to prepare

HCA has provided an initial look at the impacts of recent federal policy modifications on coverage under the Affordable Care Act. This analysis reveals how new regulations and guidelines are reshaping the landscape for health insurance, highlighting the effects these changes have on accessibility and affordability for consumers. As policymakers continue to refine healthcare laws, understanding these dynamics becomes essential for those navigating this complex system.

Google aims to establish a foundation of evidence for healthcare AI.

A sunlit medical research lab bathed in warm morning light. A doctor in a white coat and a data scientist lean over a curved touchscreen table displaying layered anatomical scans and colorful statistical graphs. Behind them, the Google logo is etched in frosted glass on a partition wall. On a nearby shelf, neatly labeled binders read “Evidence Base” and “Clinical Trials,” suggesting rigorous collaboration to build healthcare AI foundations.

Google wants to build an evidence base for healthcare AI - Modern Healthcare

- HCA reveals initial impacts of federal policy changes on Affordable Care Act coverage.  
- New data highlights shifts in enrollment and insurance affordability due to policy adjustments.  
- Federal updates appear to influence both access and quality of healthcare for enrollees.

HCA offers a first glimpse of how federal policy changes are affecting Affordable Care Act coverage.

Google wants to build an evidence base for healthcare AI

Three years ago, I penned my first piece on the role of AI in healthcare, coinciding with the debut of OpenAI's ChatGPT. At that time, I explored the transformative potential of generative AI in the medical field and felt a surge of optimism about the advancements it could bring. With the excitement still fresh in my memory, I decided this holiday weekend was the perfect moment to reflect on that initial article and assess whether we've made real strides in the areas I predicted would be most impactful.

When I first categorized the potential effects of AI on healthcare, I identified three primary domains:

1. Patient care and delivery
2. Research, diagnostics, and treatment
3. Clinical and non-clinical workflow

**Patient Care and Delivery**

I anticipated that AI would enhance patient care and streamline delivery through improved triage systems, error detection, and greater patient engagement. Now, three years later, I’m pleasantly surprised not just by the innovations, but also by the revelation that many of the challenges in healthcare were never truly technical in nature.

The triaging and access segment has blossomed into a market exceeding $100 million, growing 20-fold year over year. Companies like Counsel Health, Doctronic, and Torch Health leverage conversational AI to evaluate symptoms and direct patients toward appropriate care. Meanwhile, scheduling automation tools such as Assort Health and Hello Patient have simplified the often tedious appointment booking process. This aligns closely with the initiative I pursued with Nayroo, aimed at minimizing no-shows by efficiently matching patients with available slots.

The real excitement lies in patient engagement. AI-driven navigation platforms like Hippocratic AI, Ferry Health, and Aidify have taken on roles that were historically neglected—the gaps between clinical visits. They follow up with test results, arrange subsequent appointments, and provide care coordination, even answering queries at odd hours. Personally, I've integrated DoximityGPT into my discharge workflow, allowing me to quickly transform physician summaries into accessible explanations for patients and their families. These updates are not only read but also referenced, ensuring that primary care providers receive crucial information during vulnerable post-discharge phases.

However, it’s important to note that most of these advancements represent improvements in efficiency rather than true transformation. We're enhancing existing processes instead of fundamentally fixing the flaws within them. As a result, we find ourselves automating inefficiencies without addressing their root causes.

**Research, Diagnostics, and Treatment**

The evolution of AI in diagnostics and clinical decision support diverged from my initial expectations. Instead of AI taking over clinical reasoning, we have seen the rise of AI-powered medical references that enhance a clinician's ability to make informed decisions.

UpToDate has introduced Expert AI, which mimics the decision-making process of experienced clinicians while being fully transparent about its assumptions and reasoning. Additionally, OpenEvidence secured $210 million for quick literature analysis, and Doximity acquired Pathway Medical for $63 million, integrating it into DoxGPT. Other platforms like ClinicalKey, DynaMed, and Glass Health have also embraced AI to bolster clinical decision support.

The key takeaway is that the information provided by these tools is not only transparent but also draws from reliable sources, making them valuable additions to the clinician's toolkit rather than replacements for human judgment. If utilized properly, these tools could empower the next generation of healthcare providers to be exceptionally knowledgeable.

**Clinical and Non-clinical Workflows**

I remain a strong advocate for leveraging AI to alleviate administrative burdens, simplify documentation processes, and eradicate redundant tasks. Yet, I’ve grown increasingly skeptical about the so-called AI arms race in this domain.

AI scribes have become ubiquitous in the healthcare landscape, akin to the essential pagers of yesteryear—considered standard by all practitioners. Companies like Abridge, Suki, and Doximity automate clinical documentation, providing much-needed relief to healthcare professionals. However, they do little to address the underlying reasons for excessive documentation demands.

The surge in applications for prior authorization has been significant, with spending skyrocketing from $10 million in 2024 to $100 million in 2025. Platforms like Latent Health and Tandem now automate the filling of insurance forms directly from electronic health records, enabling physicians to submit claims swiftly. Simultaneously, payers utilize their own AI systems, such as Optum Real, for immediate coverage validation. This ongoing race sees both sides pouring billions into making a flawed system more efficient without questioning its very existence.

Furthermore, the Centers for Medicare & Medicaid Services recently introduced AI-driven prior authorization to Traditional Medicare, compensating AI contractors a share of denied claims to review services deemed "unnecessarily used." This move brings controversial aspects of Medicare Advantage into a system that functioned effectively without such burdens.

**Final Thoughts**

Reflecting on what I’ve learned about AI in healthcare over these past three years, it becomes clear that we are addressing symptoms, not fundamental issues.

We’re automating prior authorization processes without analyzing their necessity, accelerating claims processing without reimagining payment structures, and improving documentation workflows without questioning why physicians allocate so much time to record-keeping.

AI has the capacity to enhance healthcare, but only if it shines a light on the misaligned incentives that we’ve previously blamed on the intricacies of the system. Currently, we attribute inefficiencies to complexity, but what occurs when AI eliminates that rationale? When AI can streamline processes in seconds or reduce administrative costs significantly, the issue of inefficiency shifts from capability to incentive.

Healthcare organizations that wish to thrive must confront these misaligned incentives directly and leverage AI to realign them toward enhancing efficiency and patient care. Those that fail to do so will merely manage to operate their flawed systems at an accelerated pace.

Three Years of AI in Healthcare: Efficiency Compared to Transformation

A bright, sunlit modern hospital corridor: on the left, a nurse in scrubs taps a sleek tablet displaying AI-driven dashboards streamlining patient intake and vital signs; on the right, a physician in a white coat studies a glossy genomics workstation with a glowing digital DNA strand and molecular charts on a transparent screen. The warm natural light highlights clean lines and subtle motion as data flows around them, symbolizing “Efficiency vs Transformation” in healthcare’s AI evolution.

Three Years of AI in Healthcare: Efficiency vs Transformation - HLTH

- AI has significantly improved patient care delivery through automation and enhanced engagement tools.  
- Diagnostics have evolved to AI-augmented support, emphasizing transparency and clinical judgment enhancement.  
- AI in workflows mainly increases efficiency without addressing foundational issues like unnecessary documentation.

Three years ago—hard to believe—I wrote my first article on AI in healthcare, right as OpenAI launched ChatGPT. I discussed where I thought generative AI would have the biggest impact on healthcare and medicine. I vividly remember writing it. The possibilities felt euphoric. I could envision how care might be transformed.

Now, three years later, it's a holiday weekend and no one wants to read about health policy or drug pricing. So I thought I'd revisit that first article. The question: are we actually implementing AI in the areas I thought would matter most?





Healthcare Implications of AI Revisited





I initially organized AI's healthcare implications into three main buckets:

Patient care and delivery

Research, diagnostics, and treatment

Clinical and non-clinical workflow





Patient Care and Delivery





I envisioned AI improving patient care and delivery through triaging, detecting errors, and engaging patients in their care plans.

Three years later, this is where I've been most surprised, not because AI hasn't made progress, but because the progress exposed how much of the problem was never technical in the first place.

Triaging and access turned into a $100+ million market that grew 20x year over year. AI platforms like Counsel Health, Doctronic, and Torch Health now assess symptoms conversationally and route patients to appropriate care. Scheduling automation tools like Assort Health and Hello Patient eliminate manual appointment booking. This was the space I tried to crack with Nayroo—matching patients to open slots and reducing no-shows.

turned into a $100+ million market that grew 20x year over year. AI platforms like Counsel Health, Doctronic, and Torch Health now assess symptoms conversationally and route patients to appropriate care. Scheduling automation tools like Assort Health and Hello Patient eliminate manual appointment booking. This was the space I tried to crack with Nayroo—matching patients to open slots and reducing no-shows. Patient engagement is where the fun and real transformation happened. AI care navigation platforms like Hippocratic AI, Ferry Health, and Aidify now handle what we used to abandon patients on: the dead zone between clinical encounters. They call with results. They schedule follow-ups. They coordinate care transitions. They answer questions at 2 AM. Even within my own discharge workflow, I’ve incorporated using DoximityGPT. I now routinely convert my physician-directed discharge summaries into patient-friendly hospital course explanations and warm handoff letters for PCPs—both in under a minute. Patients actually read them and family members reference them. PCPs get actionable information during the highest-risk post-discharge window instead of operating blind.

One caveat with all of these tools, though. Most of them are efficiency AI, not transformative AI. They make broken processes faster. They help us navigate dysfunction more smoothly. They don't fix the dysfunction itself. We're automating abandonment instead of eliminating it.





Research, Diagnostics, and Treatment





The diagnostic and clinical decision support space evolved differently than I thought. Instead of AI replacing clinical reasoning, we got AI-augmented medical references.

UpToDate launched Expert AI that emulates how expert clinicians reason through problems, with full transparency into assumptions, sources, and reasoning steps. OpenEvidence raised $210M for rapid literature synthesis. Doximity acquired Pathway Medical for $63M, integrating it into DoxGPT. ClinicalKey, DynaMed, AMBOSS, and Glass Health all launched AI-powered clinical decision support.

What matters is that the answers are transparent, grounded in trusted content, and designed to augment rather than replace clinical judgment. This transparency matters enormously for adoption.

If physicians and trainees use these tools correctly—that is, with intent to learn and improve their knowledge base—then I do think the next generation of physicians will be some of the brightest ever. I’m biased, though.





Clinical and Non-clinical Workflows





I’m big on using AI to reduce administrative burden, streamline documentation, and eliminate unnecessary work. This is where the AI arms race is happening, and also where I've become most skeptical.

AI scribes are now ubiquitous. They're the "pager" of healthcare now: everyone has one, and if you don't, something's wrong. Abridge, Suki, Nabla, Doximity and dozens of others automate clinical documentation. That's valuable. It buys physicians time. But it doesn't question why we're generating so much documentation in the first place.

are now ubiquitous. They're the "pager" of healthcare now: everyone has one, and if you don't, something's wrong. Abridge, Suki, Nabla, Doximity and dozens of others automate clinical documentation. That's valuable. It buys physicians time. But it doesn't question why we're generating so much documentation in the first place. Prior authorization AI exploded as spending grew 10x from $10 million in 2024 to $100 million in 2025. Latent Health, Tandem, Mandolin, Squad Health, Flow Auth, and others now auto-fill payer forms by pulling data from the EHR. Physicians submit claims instantly. Payers respond with their own AI like Optum Real to deliver instant coverage validation. We have an escalating AI arms race where both sides are investing billions to fight the same battle more efficiently. Neither side is asking whether the battle should exist.

exploded as spending grew 10x from $10 million in 2024 to $100 million in 2025. Latent Health, Tandem, Mandolin, Squad Health, Flow Auth, and others now auto-fill payer forms by pulling data from the EHR. Physicians submit claims instantly. Payers respond with their own AI like Optum Real to deliver instant coverage validation. We have an escalating AI arms race where both sides are investing billions to fight the same battle more efficiently. Neither side is asking whether the battle should exist. CMS just brought AI-driven prior authorization to Traditional Medicare through the WISeR Model. They're paying AI contractors a percentage of denied claims to review services "prone to unnecessary use." We're importing Medicare Advantage's most criticized feature into a system that was already working well.





Dashevsky’s Dissection





So this captures everything I've learned about AI in healthcare over three years: we're treating symptoms, not root causes.

We're automating prior authorization instead of questioning why it exists, speeding up claims processing instead of redesigning how we pay for care, optimizing physician documentation instead of asking why physicians spend half their time on notes.

AI won't fix healthcare by making broken processes faster. It will only fix healthcare if it exposes the misaligned incentives we've been able to blame on complexity.

Right now, we can excuse inefficiency by saying the system is too complicated. But what happens when AI eliminates that excuse? When AI can automate prior auth in seconds, streamline care transitions seamlessly, and slash administrative costs? Suddenly the inefficiency becomes about incentive, not capability.

The healthcare organizations that thrive will be the ones willing to confront these misaligned incentives head-on and use AI to realign them around efficiency and patient care. The ones that don't will just run their broken systems faster.

Three Years of AI in Healthcare: Efficiency vs Transformation

Artificial intelligence (AI) is rapidly transforming the landscape of contemporary healthcare, emerging as one of the most groundbreaking technologies in the field. Its applications span a wide array of domains, including radiology, pathology, patient triage, and drug development, with AI systems now tackling tasks that were once the exclusive purview of trained medical professionals. The next significant advancement on the horizon is the emergence of autonomous AI—capable systems that go beyond merely advising healthcare providers to actively execute clinical functions, make treatment recommendations, and, in some cases, initiate actions with minimal human intervention.

The discussion surrounding autonomous healthcare AI is increasingly becoming a pressing issue rather than a theoretical concept. In various regions, including the United States, Canada, Israel, and the European Union, regulators are working on new frameworks to oversee more autonomous healthcare systems. These initiatives underscore a growing acknowledgment that current medical device regulations do not adequately address systems that can learn, adapt, and even make decisions independently.

Transitioning from decision support to automated decision-making

At present, the majority of healthcare AI applications serve as clinical decision support tools. These systems are adept at identifying suspicious lesions in medical imaging, prioritizing patient cases, and flagging potential drug interactions, with the ultimate decision still resting on the clinician's shoulders.

Autonomous systems represent a paradigm shift in this context. Rather than just delivering information, they may suggest treatment plans, adjust medication dosages, interpret test results, or monitor patients from a distance and trigger necessary interventions. For instance, Israel's newly established healthcare AI regulatory sandbox is testing autonomous pregnancy ultrasound assessments and AI-assisted heart failure management, showcasing how swiftly the technology is evolving.

Global healthcare systems are grappling with challenges such as workforce shortages, mounting costs, and aging populations. Automation presents a potential solution by enhancing efficiency and expanding clinical capacities. However, increased autonomy also brings with it additional risks.

Addressing errors made by algorithms

While human clinicians can and do make errors, they usually comprehend their decision-making process and can elucidate their reasoning when mistakes are made. AI systems, however, operate in a fundamentally different manner.

Many advanced machine learning models function as "black boxes" that generate outputs difficult to interpret, even by their creators. If an autonomous system suggests an inappropriate treatment or fails to recognize a clinical decline, pinpointing the source of the error can prove to be quite complex.

This raises significant concerns surrounding patient safety. While a healthcare professional might catch an obvious mistake from a less experienced colleague, recognizing a subtle misjudgment generated by an algorithm that seems statistically sophisticated is considerably more challenging. Additionally, an overreliance on automation can lead to “automation bias,” where medical staff are less inclined to scrutinize machine-generated recommendations.

The effectiveness of AI is heavily contingent on the quality of the data used for its training.

Healthcare datasets often exhibit issues such as incompleteness, inconsistency, or bias towards certain demographic groups. An algorithm trained predominantly on one population may yield less accurate results when applied to others. Variations in ethnicity, socioeconomic status, geography, age, and access to healthcare all affect clinical outcomes and can influence model performance.

Such challenges are particularly pronounced when technologies developed in one country are implemented globally. An AI system validated on patient demographics specific to North America, Europe, or Israel may not meet the same standards elsewhere.

Subpar data quality can result in consistently erroneous decisions on a large scale, which poses a risk that eclipses individual human error, as algorithms can replicate the same mistake across numerous patients. Health Canada has issued guidance on machine-learning-based medical devices, emphasizing the need for proper data management, validation, transparency, and ongoing monitoring.

One of the trickiest issues regarding autonomous AI pertains to accountability. If a clinician makes a poor decision, established frameworks for responsibility exist. When it comes to autonomous AI systems that may cause patient harm, the lines of accountability become blurred.

Is the clinician liable for relying on the software? Is the hospital responsible for the deployment of the system? Is liability assigned to the software developer? What happens if the algorithm evolves post-approval?

These questions have gained urgency as regulators strive to craft frameworks that accommodate adaptive machine learning systems. The U.S. Food and Drug Administration and Health Canada have both been considering mechanisms such as predetermined change control plans to oversee future modifications while maintaining regulatory vigilance.

Moreover, healthcare is one of the most frequently targeted sectors for cyberattacks, and the introduction of autonomous AI further complicates this landscape. Such systems can be vulnerable to adversarial attacks, where malicious actors subtly manipulate inputs in ways that are undetectable to humans but can lead to erroneous outputs. An attacker might aim to disrupt diagnostic algorithms, compromise patient monitoring systems, or interfere with treatment recommendations.

As healthcare increasingly relies on interconnected technology—such as smartphones, wearable devices, cloud platforms, and remote monitoring solutions— the potential for compromise escalates. A breached autonomous clinical system could impact patient care in ways far exceeding traditional software failures.

The necessity for human involvement

The World Health Organization emphasizes that AI's primary role should be to enhance human well-being through principles of transparency, accountability, and respect for human rights. Human clinicians bring contextual understanding, ethical considerations, empathy, and a sense of responsibility—qualities that are challenging, if not impossible, to replicate in algorithms.

The task at hand is to ascertain where human input is essential, and where automation can safely improve efficiency. Insufficient oversight brings inherent risks, while excessive oversight may negate many advantages that autonomous systems have to offer.

The potential of autonomous AI in healthcare: Balancing innovation with patient safety concerns.

A sunlit modern hospital room: a sleek, white autonomous medical robot gently adjusts an IV drip beside an adult patient’s bed while its touchscreen displays stable vitals; nearby, a secondary monitor glows red with a blinking “ERROR” alert as a concerned nurse watches. Warm morning light pours through large windows, casting natural shadows that contrast the promise of innovation with the looming safety challenge.

Autonomous AI in healthcare: Innovation’s promise, patient safety’s challenge - Digital Journal

- Autonomous AI in healthcare is evolving from decision support to making clinical decisions independently.  
- Regulators are creating frameworks for managing risks associated with autonomous healthcare AI systems.  
- Data quality and accountability present significant challenges in the deployment of autonomous AI technologies.  

Artificial intelligence has become one of the most disruptive technologies in modern healthcare. From radiology and pathology to patient triage and drug discovery, AI systems are increasingly capable of performing tasks that were once considered the exclusive domain of trained healthcare professionals. The next frontier is autonomous AI: systems that do not merely advise clinicians but actively perform clinical functions, make recommendations, and in some circumstances initiate actions with limited human oversight.

The growing debate around autonomous healthcare AI is no longer theoretical. Regulators in several jurisdictions, including Israel, the U.S., Canada, and the European Union, are actively developing new frameworks to govern increasingly autonomous medical systems. These efforts reflect recognition that existing medical device regulations were not designed for systems capable of learning, adapting, and, in some cases, making decisions independently.

The evolution from decision support to decision making

Most healthcare AI deployed today functions as clinical decision support. Such systems can identify suspicious lesions on medical images, help prioritize patients, or flag potential drug interactions. The clinician remains firmly responsible for the final decision.

Autonomous systems represent a fundamental shift. Instead of merely providing information, they may recommend treatments, adjust medication regimens, interpret diagnostic tests, or monitor patients remotely and trigger interventions. Israel’s recently launched healthcare AI regulatory sandbox includes pilots for autonomous pregnancy ultrasound assessment and AI-supported heart failure management, illustrating how rapidly the technology is advancing.

Healthcare systems globally face workforce shortages, rising costs, and ageing populations. Automation appears to offer a solution by increasing efficiency and extending clinical capacity. Yet greater autonomy inevitably introduces additional risk.

When algorithms make mistakes

Human clinicians make mistakes, but they generally understand the reasoning behind their decisions and can explain that reasoning when errors occur. AI systems operate differently.

Many advanced machine learning models function as so-called “black boxes,” producing outputs that are difficult to interpret even by their developers. If an autonomous system recommends an inappropriate treatment or fails to identify clinical deterioration, determining exactly why the error occurred may prove challenging.

This creates significant concerns for patient safety. A healthcare professional may detect an obvious error from a junior colleague; it can be much harder to recognise a subtle mistake generated by an algorithm that appears statistically sophisticated. Furthermore, excessive reliance on automation can create what researchers call “automation bias,” where humans become less likely to challenge machine-generated outputs.

The effectiveness of AI depends heavily on the quality of the data used to train it.

Healthcare datasets are often incomplete, inconsistent, or biased toward particular populations. An algorithm trained primarily on one demographic group may perform less accurately when applied to others. Differences in ethnicity, socioeconomic status, geography, age, and healthcare access can all influence clinical outcomes and affect model performance.

This challenge is especially significant when technologies developed in one country are deployed globally. An AI system validated using patient populations in North America, Europe, or Israel may not automatically perform to the same standard in other regions.

Poor-quality data can therefore produce systematically poor decisions at scale, which is a risk significantly greater than isolated human error because algorithms can replicate the same mistake across thousands of patients. Health Canada’s guidance on machine learning-enabled medical devices specifically highlights the importance of data management, validation, transparency, and ongoing monitoring.

The most difficult question surrounding autonomous AI concerns responsibility. If a physician makes a poor clinical decision, accountability frameworks are well established. If an autonomous AI system contributes to patient harm, responsibility becomes less clear.

Is the clinician accountable because they relied on the software? Is the hospital responsible for deploying the system? Does liability belong to the software developer? What if the algorithm has changed and evolved since its original approval?

These questions are becoming increasingly important as regulators develop frameworks capable of supporting adaptive machine learning systems. The U.S. Food and Drug Administration and Health Canada have both explored mechanisms such as predetermined change control plans to manage future modifications while maintaining regulatory oversight.

As further complication arises by healthcare already representing one of the most frequently targeted sectors for cyberattacks. Autonomous AI introduces additional vulnerabilities. Adversarial attacks can potentially manipulate AI systems by subtly altering inputs in ways that are imperceptible to humans but capable of producing incorrect outputs. An attacker might seek to disrupt diagnostic algorithms, compromise patient monitoring systems, or interfere with therapeutic recommendations.

As healthcare becomes more connected through smartphones, wearable devices, cloud computing platforms, and remote monitoring technologies, the attack surface continues to expand. A compromised autonomous clinical system could potentially affect patient care on a scale far beyond traditional software failures.

Human in the loop

The World Health Organization has repeatedly stressed that AI should support human wellbeing while respecting principles of transparency, accountability, and human rights. Human clinicians contribute contextual understanding, ethical judgement, empathy, and professional responsibility. These are qualities that remain difficult to replicate through algorithms.

The challenge is identifying where human involvement is essential and where automation can safely improve efficiency. Too little oversight introduces risk. Too much oversight may eliminate many of the benefits that autonomous systems promise.

New commentary calls for AI regulations focused on patients in healthcare systems.

More from AI in healthcare

Evaluating the theory: a targeted pilot for artificial intelligence in healthcare

Global News Summary: West Bank schools shut down, AI applications in healthcare, Indigenous rights issues

Scale AI and Mayo Clinic Revolutionizing Clinical Care

Kleiner Perkins, John Doerr, and UCSF Health believe healthcare AI is fundamentally flawed.