Federal and state agencies are increasingly leveraging artificial intelligence (AI), and the healthcare sector is no exception. The U.S. Department of Health and Human Services (HHS) has reported significant rises in AI implementation among key health organizations from fiscal years 2024 to 2025. The U.S. Food and Drug Administration (FDA) saw a remarkable 148 percent increase, while the U.S. Centers for Disease Control and Prevention (CDC) experienced an 87 percent boost. The Centers for Medicare and Medicaid Services (CMS) and the National Institutes of Health (NIH) recorded increases of 78 percent and 51 percent, respectively. State governments are also embracing AI technologies.
While the drive towards AI is motivated by goals of enhancing efficiency, improving accuracy, and achieving cost reductions, this rapid integration into public health raises significant concerns regarding the appropriateness of AI in certain governmental decisions, particularly when human oversight may be required. In critical areas like healthcare, as AI systems become more sophisticated and less transparent, the risk of detrimental consequences increases. The complexity of these systems makes it more difficult to identify and correct errors, potentially leading to serious oversight issues where mistakes can go unnoticed, with even rare miscalculations capable of severely affecting patient care.
The types of AI employed in the healthcare sector have evolved noticeably. Previously, agencies primarily used rule-based algorithms, which were straightforward and transparent, allowing for easier audits. For instance, during the onset of the COVID-19 pandemic in 2020, states deployed these simple algorithms to prioritize access to limited monoclonal antibody supplies based on race and sex, thereby clarifying their decision-making processes and ensuring accountability under anti‑discrimination laws.
By 2023, during the Medicaid eligibility assessments known as “unwinding,” states also relied on rule-based algorithms. Unfortunately, due to programmer errors, data integrity issues, and insufficient oversight, many individuals were incorrectly disenrolled. A class action lawsuit against TennCare highlighted alleged violations of the Medicaid Act and the Americans with Disabilities Act, partly due to the shortcomings of its computerized Eligibility Determination System. Similar issues arose from Deloitte Consulting’s software in Texas, raising alarms about the dependability of automated eligibility evaluations used across various states.
Since 2023, AI capabilities have advanced rapidly, marked by improvements in model performance and real-world applications. Federal agencies are progressively adopting machine-learning models and generative large language models (LLMs). Recent data from the U.S. Government Accountability Office (GAO) indicates a doubling of overall AI use cases across 11 federal entities between 2023 and 2024, with generative AI cases skyrocketing by a factor of nine, led by health agencies. States are also exploring generative AI applications and developing guidance on its utilization.
Although the recorded negatives linked to predictive machine learning and generative AI in healthcare remain relatively scarce, emerging evidence points to alarming risks. Studies have documented instances of diagnostic errors, unsafe triage recommendations, privacy violations, and concerning uses of these technologies in public decision-making. Notably, a class action suit from Medicare Advantage beneficiaries claimed that an AI system used by UnitedHealth Group incorrectly denied necessary medical care for post-acute treatments. A federal court later mandated the disclosure of the AI's error rates, which the plaintiffs contended were as high as 90 percent.
These cases demonstrate the critical need for healthcare agencies to conduct thorough evaluations prior to implementing AI systems that could lead to significant harm. Current federal policies necessitate AI impact assessments for high-stakes applications that could impact individual rights or safety. However, these frameworks primarily address risk mitigation rather than questioning the initial appropriateness of AI in specific decisions. They require agencies to clarify the system's objectives, evaluate data quality, assess privacy implications, analyze costs, and ensure independent review, but they do not confront the essential question of whether AI should be employed at all.
Agencies currently lack a structured approach to determine when it's appropriate to utilize AI, how much human judgment should be involved, and how the specific governmental function should inform the design and distribution of automated systems. This absence of a consistent framework poses a significant risk of making ill-considered AI deployment decisions that do not adhere to principles of administrative law.
To address this gap, I advocate for a Five Factor Framework (FFF) that focuses on various interconnected aspects of the decision-making context and the foundational principles guiding administrative processes. This framework encourages agencies to:
1. Identify the interests involved, including affected parties and the potential harms or rights at stake. 2. Examine the nature and ethical dimensions of the decision, encompassing its functional needs and procedural demands as aligned with the Administrative Procedure Act's (APA) reasoned decision-making standards. 3. Assess the characteristics of the proposed technology—its abilities, limitations, and error risks—through the lens of doctrines relevant to arbitrary and capricious review. 4. Evaluate the integration of human oversight in decision-making—how human judgement complements or replaces automated interventions, ensuring compliance with nondelegation principles and due process requirements. 5. Identify the legal parameters governing the decision, including protections related to due process and civil rights along with any sector-specific statutory obligations.
Collectively, these inquiries create a robust, legally grounded foundation for assessing whether technological interventions serve the public good. If an agency concludes that the advantages of incorporating AI for a critical decision outweigh the potential downsides, it should engage with the public for feedback on its evaluation. This could take the form of public notices, community meetings, or consultations with advocacy groups representing affected populations. Achieving democratic legitimacy necessitates not only technical conformity but also citizen participation in decisions that influence healthcare access.
Ultimately, the core issue is not whether AI can enhance the efficiency of government healthcare systems, but whether such efficiency justifies the risks associated with AI errors that could impact individuals’ health, rights, or lives. In the absence of a principled framework dictating when AI should be applied or avoided, agencies run the risk of systemically automating harm. The public deserves a better approach, and the legal system demands it.
