Advertisement

Self-fulfilling prophecies and machine learning in resuscitation science

  • Maria De-Arteaga
    Affiliations
    Information, Risk and Operations Management Department, McCombs School of Business, University of Texas at Austin, Austin, TX, USA
    Search for articles by this author
  • Jonathan Elmer
    Correspondence
    Corresponding author at: Iroquois Building, Suite 400A, 3600 Forbes Avenue, Pittsburgh, PA 15206, USA.
    Affiliations
    Departments of Emergency Medicine, Critical Care Medicine and Neurology, University of Pittsburgh School of Medicine, Pittsburgh, PA, USA
    Search for articles by this author

      Abstract

      Introduction

      Growth of machine learning (ML) in healthcare has increased potential for observational data to guide clinical practice systematically. This can create self-fulfilling prophecies (SFPs), which arise when prediction of an outcome increases the chance that the outcome occurs.

      Methods

      We performed a scoping review, searching PubMed and ArXiv using terms related to machine learning, algorithmic fairness and bias. We reviewed results and selected manuscripts for inclusion based on expert opinion of well-designed or key studies and review articles. We summarized these articles to explore how use of ML can create, perpetuate or compound SFPs, and offer recommendations to mitigate these risks.

      Results

      We identify-four key mechanisms through which SFPs may be reproduced or compounded by ML. First, imperfect human beliefs and behavior may be encoded as SFPs when treatment decisions are not accounted for. Since patient outcomes are influenced by a myriad of clinical actions, many of which are not collected in data, this is common. Second, human–machine interaction may compound SFPs through a cycle of mutual reinforcement. Third, ML may introduce new SFPs stemming from incorrect predictions. Finally, historically correct clinical choices may become SFPs in the face of medical progress.

      Conclusion

      There is a need for broad recognition of SFPs as ML is increasingly applied in resuscitation science and across medicine. Acknowledging this challenge is crucial to inform research and practice that can transform ML from a tool that risks obfuscating and compounding SFPs into one that sheds light on and mitigates SFPs.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Resuscitation
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Grene D.
        • Aeschylus S.
        • Euripides.
        Three Greek tragedies in translation.
        The University of Chicago press, Chicago, Ill1942
        • Merton R.K.
        The Self-Fulfilling Prophecy.
        Antioch Rev. 1948; 8: 193-210
        • Smith J.D.
        The Mahābhārata.
        Penguin, New Delhi2009
        • Wilkinson D.
        The self-fulfilling prophecy in intensive care.
        Theor Med Bioeth. 2009; 30: 401-410
        • Chen I.Y.
        • Pierson E.
        • Rose S.
        • Joshi S.
        • Ferryman K.
        • Ghassemi M.
        Ethical Machine Learning in Healthcare.
        Ann Rev Biomed Data Sci. 2021; 4
        • Barocas S.
        • Selbst A.D.
        Big Data's Disparate Impact.
        California Law Rev. 2016; 104: 671-732
        • Mitchell S.
        • Potash E.
        • Barocas S.
        • D'Amour A.
        • Lum K.
        Algorithmic Fairness: Choices, Assumptions, and Definitions.
        Annu Rev Stat Appl. 2021; 8: 141-163
        • Day S.J.
        • Altman D.G.
        Blinding in clinical trials and other studies.
        BMJ. 2000; 321: 504
        • Callahan A.
        • Shah N.H.
        Chapter 19 - Machine Learning in Healthcare.
        in: Sheikh A. Cresswell K.M. Wright A. Bates D.W. Key Advances in Clinical Informatics. Academic Press, 2017: 279-291
        • Wiens J.
        • Shenoy E.S.
        Machine Learning for Healthcare: On the Verge of a Major Shift in Healthcare Epidemiology.
        Clin Infect Dis. 2018; 66: 149-153
        • Research and Reporting Considerations for Observational Studies Using Electronic Health Record Data
        Ann Int Med. 2020; 172: S79-S84
        • Sherman R.E.
        • Anderson S.A.
        • Dal Pan G.J.
        • et al.
        Real-World Evidence - What Is It and What Can It Tell Us?.
        N Engl J Med. 2016; 375: 2293-2297
        • Berg K.M.
        • Cheng A.
        • Panchal A.R.
        • et al.
        Part 7: Systems of Care: 2020 American Heart Association Guidelines for Cardiopulmonary Resuscitation and Emergency Cardiovascular Care.
        Circulation. 2020; 142: S580-S604
        • Elmer J.
        • Callaway C.W.
        • Chang C.H.
        • et al.
        Long-Term Outcomes of Out-of-Hospital Cardiac Arrest Care at Regionalized Centers.
        Ann Emerg Med. 2019; 73: 29-39
        • Caruana R.
        • Lou Y.
        • Gehrke J.
        • Koch P.
        • Sturm M.
        • Elhadad N.
        Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission. Association for Computing Machinery, 2015: 1721-1730
        • London A.J.
        Artificial Intelligence and Black-Box Medical Decisions: Accuracy versus Explainability.
        Hastings Cent Rep. 2019; 49: 15-21
        • Mertens M.
        • King O.C.
        • van Putten M.
        • Boenink M.
        Can we learn from hidden mistakes? Self-fulfilling prophecy and responsible neuroprognostic innovation.
        J Med Ethics. 2021;
        • McCracken D.J.
        • Lovasik B.P.
        • McCracken C.E.
        • et al.
        The Intracerebral Hemorrhage Score: A Self-Fulfilling Prophecy?.
        Neurosurgery. 2019; 84: 741-748
        • Hemphill 3rd, J.C.
        • Newman J.
        • Zhao S.
        • Johnston S.C.
        Hospital usage of early do-not-resuscitate orders and outcome after intracerebral hemorrhage.
        Stroke. 2004; 35: 1130-1134
        • Hemphill 3rd J.C.
        • White D.B.
        Clinical nihilism in neuroemergencies.
        Emerg Med Clin North Am. 2009; 27: 27-37
      1. Park SY, Kuo P-Y, Barbarin A, et al. Identifying Challenges and Opportunities in Human-AI Collaboration in Healthcare. Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing; 2019; Austin, TX, USA.

        • Bansal G.
        • Nushi B.
        • Kamar E.
        • Weld D.S.
        • Lasecki W.S.
        • Horvitz E.
        Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff.
        Proc AAAI Conf Artif Intell. 2019; 33: 2429-2437
      2. De-Arteaga M, Fogliato R, Chouldechova A. A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems; 2020; Honolulu, HI, USA.

        • Dietvorst B.J.
        • Simmons J.P.
        • Massey C.
        Algorithm aversion: People erroneously avoid algorithms after seeing them err.
        J Exp Psychol Gen. 2015; 144: 114-126
        • Goddard K.
        • Roudsari A.
        • Wyatt J.C.
        Automation bias: a systematic review of frequency, effect mediators, and mitigators.
        J Am Med Inform Assoc. 2012; 19: 121-127
        • Lebovitz S.
        • Lifshitz-Assaf H.
        • Levina N.
        To Incorporate or Not to Incorporate AI for Critical Judgments: The Importance of Ambiguity in Professionals’ Judgment Process.
        NYU Stern School Bus. 2020;
        • Steinberg A.
        • Grayek E.
        • Arnold R.M.
        • et al.
        Physicians' cognitive approach to prognostication after cardiac arrest.
        Resuscitation. 2022;
        • Reynolds J.C.
        • Grunau B.E.
        • Rittenberger J.C.
        • Sawyer K.N.
        • Kurz M.C.
        • Callaway C.W.
        Association Between Duration of Resuscitation and Favorable Outcome After Out-of-Hospital Cardiac Arrest: Implications for Prolonging or Terminating Resuscitation.
        Circulation. 2016; 134: 2084-2094
        • Drennan I.R.
        • Case E.
        • Verbeek P.R.
        • et al.
        A comparison of the universal TOR Guideline to the absence of prehospital ROSC and duration of resuscitation in predicting futility from out-of-hospital cardiac arrest.
        Resuscitation. 2017; 111: 96-102
        • Obermeyer Z.
        • Powers B.
        • Vogeli C.
        • Mullainathan S.
        Dissecting racial bias in an algorithm used to manage the health of populations.
        Science. 2019; 366: 447-453
        • Campbell B.C.
        • Mitchell P.J.
        • Kleinig T.J.
        • et al.
        Endovascular therapy for ischemic stroke with perfusion-imaging selection.
        N Engl J Med. 2015; 372: 1009-1018
        • Hacke W.
        • Kaste M.
        • Bluhmki E.
        • et al.
        Thrombolysis with alteplase 3 to 4.5 hours after acute ischemic stroke.
        N Engl J Med. 2008; 359: 1317-1329
        • Nogueira R.G.
        • Jadhav A.P.
        • Haussen D.C.
        • et al.
        Thrombectomy 6 to 24 Hours after Stroke with a Mismatch between Deficit and Infarct.
        N Engl J Med. 2018; 378: 11-21
        • Nieuwkamp D.J.
        • Setz L.E.
        • Algra A.
        • Linn F.H.
        • de Rooij N.K.
        • Rinkel G.J.
        Changes in case fatality of aneurysmal subarachnoid haemorrhage over time, according to age, sex, and region: a meta-analysis.
        Lancet Neurol. 2009; 8: 635-642
        • Gerke S.
        • Babic B.
        • Evgeniou T.
        • Cohen I.G.
        The need for a system view to regulate artificial intelligence/machine learning-based software as medical device.
        npj Digital Med. 2020; 3: 53
        • Kompa B.
        • Snoek J.
        • Beam A.L.
        Second opinion needed: communicating uncertainty in medical machine learning.
        npj Digital Med. 2021; 4: 4
        • Raghu M.
        • Blumer K.
        • Sayres R.
        • et al.
        Direct Uncertainty Prediction for Medical Second Opinions.
        Proceedings of Machine Learning Research, 2019
        • Perdomo J.
        • Zrnic T.
        • Mendler-Dünner C.
        • Hardt M.
        Performative Prediction.
        Proceedings of Machine Learning Research, 2020
      3. Coston A, Kennedy EH, Chouldechova A. Counterfactual Predictions under Runtime Confounding. ArXiv. 2020;abs/2006.16916.

        • Schulam P.
        • Saria S.
        Reliable decision support using counterfactual models.
        Adv Neural Inform Process Syst. 2017; : 1698-1709
        • Gigerenzer G.
        • Gaissmaier W.
        • Kurz-Milcke E.
        • Schwartz L.M.
        • Woloshin S.
        Helping Doctors and Patients Make Sense of Health Statistics.
        Psychol Sci Public Interest. 2007; 8: 53-96
        • Geocadin R.G.
        • Callaway C.W.
        • Fink E.L.
        • et al.
        Standards for Studies of Neurological Prognostication in Comatose Survivors of Cardiac Arrest: A Scientific Statement From the American Heart Association.
        Circulation. 2019; 140: e517-e542
        • Scarpino M.
        • Lolli F.
        • Lanzo G.
        • et al.
        Neurophysiology and neuroimaging accurately predict poor neurological outcome within 24 hours after cardiac arrest: The ProNeCA prospective multicentre prognostication study.
        Resuscitation. 2019; 143: 115-123
        • Rudin C.
        Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.
        Nat Mach Intel. 2019; 1: 206-215
        • Tonekaboni S.
        • Joshi S.
        • McCradden M.D.
        • Goldenberg A.
        What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use.
        Proceedings of Machine Learning Research, 2019
        • Kaur H.
        • Nori H.
        • Jenkins S.
        • Caruana R.
        • Wallach H.
        • Vaughan J.W.
        Interpreting Interpretability: Understanding Data Scientists' Use of Interpretability Tools for Machine Learning. Association for Computing Machinery, 2020: 1-14
      4. Lakkaraju H, Bastani O. “How do I fool you?”: Manipulating User Trust via Misleading Black Box Explanations. 2019:arXiv:1911.06473. https://ui.adsabs.harvard.edu/abs/2019arXiv191106473L. Accessed November 01, 2019.

        • Poursabzi-Sangdeh F.
        • Goldstein D.G.
        • Hofman J.M.
        • Vaughan J.W.W.
        • Wallach H.
        Manipulating and Measuring Model Interpretability.
        In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery. 2021 (Article 237)
      5. De-Arteaga M, Dubrawski A, Chouldechova A. Learning under selective labels in the presence of expert consistency. ArXiv. 2018;abs/1807.00905.