Picture yourself in a doctor's office, recounting your discomfort while your physician's gaze shifts from you to a nearby screen. That screen indicates a low-risk status, yet your body feels different. In this scenario, a crucial question arises: whose perspective should be trusted—the technology or your own lived experience?
Across the globe, healthcare facilities are rapidly integrating algorithms into patient care, utilizing everything from symptom checkers that help prioritize cases to AI tools aimed at enhancing diagnostic accuracy. The primary selling points emphasize improved efficiency, streamlined coordination, and enhanced precision in medical assessments.
However, these digital tools often overlook an important reality. They implicitly determine which sources of information are deemed credible, which symptoms warrant attention, and how patients' narratives should be interpreted.
I characterize this phenomenon as algorithmic gaslighting.
Redefining Medical Gaslighting
The term “medical gaslighting” refers to scenarios where patients’ genuine symptoms are undermined or misinterpreted by medical professionals—sometimes influenced by factors like gender, race, or chronic illness. Critics argue that the term is sometimes overused, as gaslighting suggests intentional deception, whereas many dismissive healthcare providers operate based on unconscious biases and longstanding institutional norms.
Regardless of the underlying intentions, the result is troubling: patients are left doubting their own experiences, feeling disconnected from the language necessary to articulate their conditions.
Algorithms exacerbate this dynamic. A computer program lacks intent and cannot perform deception; however, its decisions can overshadow patient accounts with a veneer of impartiality. These systems can constrain personal experiences into rigid schemas, leading individuals to question the validity of their own symptoms.
Key Issues in Algorithmic Healthcare
Firstly, there's what can be termed “data violence.” Medical data often skews towards populations that are predominantly white, male, and economically advantaged. This bias renders patients who do not fit that profile statistically invisible. For instance, symptoms like chest pain manifest differently in women than men, yet much cardiovascular research predominantly reflects male experiences.
Secondly, there's a troubling lack of transparency. While patients can challenge a physician's judgment, they find it difficult to dispute the results of a process whose inner workings are often opaque, even to the healthcare providers themselves. Take the Epic Sepsis Model, for example—a proprietary alert system used across many U.S. hospitals. Research revealed it performed significantly worse than advertised, identifying only about one-third of sepsis cases while generating frequent false alarms, all without clinicians or patients understanding the rationale behind any specific alert.
The third issue pertains to automated authority. When algorithms generate conclusions that contradict a patient's description, it can create a climate where individuals doubt their own experiences, and medical professionals feel reluctant to disregard what seems like an unbiased recommendation.
This authority can sometimes extend beyond safety. Internal documents later disclosed serious flaws in IBM’s Watson for Oncology, a tool promoted as a reliable decision-making aid. The tool often provided unsafe and incorrect treatment suggestions, having been largely trained on hypothetical scenarios rather than actual patient cases.
Lastly, there's the feedback loop effect. Patients and clinicians quickly learn which symptoms are likely to be categorized as "valid cases," leading them to adjust their descriptions accordingly. These new formulations then enter the data stream, reinforcing the limitations of the original categorizations that failed to accurately capture a patient's experience.
Over time, patients may find themselves framing their symptoms based on how they believe algorithms will interpret them rather than expressing their genuine feelings.
Understanding the Underlying Reality
Many may think these issues are mere technical glitches that superior design could remedy. Yet, the underlying analysis suggests something more systemic at play, challenging the notion that better engineering alone can resolve the complexities of algorithmic healthcare.
