Download PDF
Appl Clin Inform 2023; 14(04): 743-751
DOI: 10.1055/a-2121-8380
Research Article

Doctors Identify Hemorrhage Better during Chart Review when Assisted by Artificial Intelligence

Martin S. Laursen   *
1   SDU Robotics, The Maersk Mc-Kinney Moller Institute, University of Southern Denmark, Odense, Denmark
,
Jannik S. Pedersen   *
1   SDU Robotics, The Maersk Mc-Kinney Moller Institute, University of Southern Denmark, Odense, Denmark
,
Rasmus S. Hansen
2   Department of Clinical Biochemistry, Odense University Hospital, Odense, Denmark
,
Thiusius R. Savarimuthu
1   SDU Robotics, The Maersk Mc-Kinney Moller Institute, University of Southern Denmark, Odense, Denmark
,
Rasmus B. Lynggaard
2   Department of Clinical Biochemistry, Odense University Hospital, Odense, Denmark
,
Pernille J. Vinholt
2   Department of Clinical Biochemistry, Odense University Hospital, Odense, Denmark
› Author Affiliations Funding This work was supported by The Odense University Hospital Innovation Fund.
› Further Information
    Permissions and Reprints

Abstract

Objectives This study evaluated if medical doctors could identify more hemorrhage events during chart review in a clinical setting when assisted by an artificial intelligence (AI) model and medical doctors' perception of using the AI model.

Methods To develop the AI model, sentences from 900 electronic health records were labeled as positive or negative for hemorrhage and categorized into one of 12 anatomical locations. The AI model was evaluated on a test cohort consisting of 566 admissions. Using eye-tracking technology, we investigated medical doctors' reading workflow during manual chart review. Moreover, we performed a clinical use study where medical doctors read two admissions with and without AI assistance to evaluate performance when using and perception of using the AI model.

Results The AI model had a sensitivity of 93.7% and a specificity of 98.1% on the test cohort. In the use studies, we found that medical doctors missed more than 33% of relevant sentences when doing chart review without AI assistance. Hemorrhage events described in paragraphs were more often overlooked compared with bullet-pointed hemorrhage mentions. With AI-assisted chart review, medical doctors identified 48 and 49 percentage points more hemorrhage events than without assistance in two admissions, and they were generally positive toward using the AI model as a supporting tool.

Conclusion Medical doctors identified more hemorrhage events with AI-assisted chart review and they were generally positive toward using the AI model.

Keywords

electronic health records - hemorrhage - artificial intelligence - decision support systems - natural language processing

Data Sharing Statement

We cannot make data publicly available due to sensitive information, but we encourage researchers to contact us for sharing possibilities.


Protection of Human and Animal Subjects

This study only included EHR data that were stored and processed on a secure platform in compliance with the General Data Protection Regulations. The study was approved by The Danish Data Protection Agency at the OUH (Journal nr 21/25172) and the regional secretary and Law Department in the Region of Southern Denmark (Journal nr 21/22013). According to section 14(2) of the Danish Act on Research Ethics Review of Health Research Projects, studies using retrospective data that do not involve human biological material do not require ethical approval.


Ethics

The study was approved by The Danish Data Protection Agency at the OUH (Journal nr 21/25172) and the Regional Secretary and Law Department in the Region of Southern Denmark (Journal nr 21/22013).


* Equal contribution.


Supplementary Material



Publication History

Received: 04 April 2023

Accepted: 29 June 2023

Accepted Manuscript online:
03 July 2023

Article published online:
20 September 2023

© 2023. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Entzeridou E, Markopoulou E, Mollaki V. Public and physician's expectations and ethical concerns about electronic health record: benefits outweigh risks except for information security. Int J Med Inform 2018; 110: 98-107
    CrossrefPubMedGoogle Scholar
  • 2 Miotto R, Wang F, Wang S, Jiang X, Dudley JT. Deep learning for healthcare: review, opportunities and challenges. Brief Bioinform 2018; 19 (06) 1236-1246
    CrossrefPubMedGoogle Scholar
  • 3 Tayefi M, Ngo P, Chomutare T. et al. Challenges and opportunities beyond structured data in analysis of electronic health records. Wiley Interdiscip Rev Comput Stat 2021; 13 (06) e1549
    CrossrefPubMedGoogle Scholar
  • 4 Valkhoff VE, Coloma PM, Masclee GMC. et al; EU-ADR Consortium. Validation study in four health-care databases: upper gastrointestinal bleeding misclassification affects precision but not magnitude of drug-related upper gastrointestinal bleeding risk. J Clin Epidemiol 2014; 67 (08) 921-931
    CrossrefPubMedGoogle Scholar
  • 5 Øie LR, Madsbu MA, Giannadakis C. et al. Validation of intracranial hemorrhage in the Norwegian Patient Registry. Brain Behav 2018; 8 (02) e00900
    CrossrefPubMedGoogle Scholar
  • 6 Delekta J, Hansen SM, AlZuhairi KS, Bork CS, Joensen AM. The validity of the diagnosis of heart failure (I50.0-I50.9) in the Danish National Patient Register. Dan Med J 2018; 65 (04) A5470
    PubMedGoogle Scholar
  • 7 Higgins TL, Deshpande A, Zilberberg MD. et al. Assessment of the accuracy of using ICD-9 diagnosis codes to identify pneumonia etiology in patients hospitalized with pneumonia. JAMA Network Open 2020; 3 (07) e207750-e207750
    CrossrefPubMedGoogle Scholar
  • 8 Wabe N, Li L, Lindeman R. et al. Evaluation of the accuracy of diagnostic coding for influenza compared to laboratory results: the availability of test results before hospital discharge facilitates improved coding accuracy. BMC Med Inform Decis Mak 2021; 21 (01) 168
    CrossrefPubMedGoogle Scholar
  • 9 Lee HJ, Jiang M, Wu Y. et al. A comparative study of different methods for automatic identification of clopidogrel-induced bleedings in electronic health records. AMIA Jt Summits Transl Sci Proc 2017; 2017: 185-192
    PubMedGoogle Scholar
  • 10 Taggart M, Chapman WW, Steinberg BA. et al. Comparison of 2 natural language processing methods for identification of bleeding among critically ill patients. JAMA Netw Open 2018; 1 (06) e183451-e183451
    CrossrefPubMedGoogle Scholar
  • 11 Li R, Hu B, Liu F. et al. Detection of bleeding events in electronic health record notes using convolutional neural network models enhanced with recurrent neural network autoencoders: deep learning approach. JMIR Med Inform 2019; 7 (01) e10788
    CrossrefPubMedGoogle Scholar
  • 12 Elkin PL, Mullin S, Mardekian J. et al. Using artificial intelligence with natural language processing to combine electronic health record's structured and free text data to identify nonvalvular atrial fibrillation to decrease strokes and death: evaluation and case-control study. J Med Internet Res 2021; 23 (11) e28946
    CrossrefPubMedGoogle Scholar
  • 13 Mitra A, Rawat BPS, McManus D, Kapoor A, Yu H. Bleeding entity recognition in electronic health records: a comprehensive analysis of end-to-end systems. In: AMIA Annual Symposium Proceedings. Vol. 2020. American Medical Informatics Association; 2020: 860
    PubMedGoogle Scholar
  • 14 Mitra A, Rawat BPS, McManus DD, Yu H. Relation classification for bleeding events from electronic health records using deep learning systems: an empirical study. JMIR Med Inform 2021; 9 (07) e27527
    CrossrefPubMedGoogle Scholar
  • 15 Shung D, Tsay C, Laine L. et al. Early identification of patients with acute gastrointestinal bleeding using natural language processing and decision rules. J Gastroenterol Hepatol 2021; 36 (06) 1590-1597
    CrossrefPubMedGoogle Scholar
  • 16 Pedersen JS, Laursen MS, Rajeeth Savarimuthu T. et al. Deep learning detects and visualizes bleeding events in electronic health records. Res Pract Thromb Haemost 2021; 5 (04) e12505
    CrossrefPubMedGoogle Scholar
  • 17 Rajpurkar P, Chen E, Banerjee O, Topol EJ. AI in health and medicine. Nat Med 2022; 28 (01) 31-38
    CrossrefPubMedGoogle Scholar
  • 18 Decousus H, Tapson VF, Bergmann JF. et al; IMPROVE Investigators. Factors at admission associated with bleeding risk in medical patients: findings from the IMPROVE investigators. Chest 2011; 139 (01) 69-79
    CrossrefPubMedGoogle Scholar
  • 19 National Guideline Centre #x0028;UK#x0029;. Venous thromboembolism in over 16s: Reducing the risk of hospital-acquired deep vein thrombosis or pulmonary embolism. London: National Institute for Health and Care Excellence #x0028;NICE#x0029;; March 2018.
    PubMed
  • 20 Amin A, Stemkowski S, Lin J, Yang G. Thromboprophylaxis rates in US medical centers: success or failure?. J Thromb Haemost 2007; 5 (08) 1610-1616
    CrossrefPubMedGoogle Scholar
  • 21 Rwabihama JP, Audureau E, Laurent M. et al; améliorer la prophylaxie de la Maladie ThromboEmbolique veineuse en milieu gériatrique (MATEV) Study Group. Prophylaxis of venous thromboembolism in geriatric settings: a cluster-randomized multicomponent interventional trial. J Am Med Dir Assoc 2018; 19 (06) 497-503
    CrossrefPubMedGoogle Scholar
  • 22 Amin A, Stemkowski S, Lin J, Yang G. Appropriate thromboprophylaxis in hospitalized cancer patients. Clin Adv Hematol Oncol 2008; 6 (12) 910-920
    PubMedGoogle Scholar
  • 23 Matthiesen S, Meboldt M, Ruckpaul A, Mussgnug M. et al. Eye tracking, a method for engineering design research on engineers' behavior while analyzing technical systems. In: DS 75-7: Proceedings of the 19th International Conference on Engineering Design (ICED13), Design for Harmonies, Vol. 7: Human Behaviour in Design,. Seoul, Korea, August 19–22, 2013: 277-286
    PubMedGoogle Scholar
  • 24 Clark K, Luong MT, Le QV, Manning CD. ELECTRA: pre-training text encoders as discriminators rather than generators. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia,. April 26–30, 2020 . OpenReview.net. https://openreview.net/forum?id=r1xMH1BtvB
    PubMedGoogle Scholar
  • 25 Wang A, Singh A, Michael J, Hill F, Levy O, Bowman S. GLUE: a multi-task benchmark and analysis platform for natural language understanding. In: Proceedings of the 2018 EMNLP Workshop BLackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Association for Computational Linguistics; 2018: 353-355
    PubMedGoogle Scholar
  • 26 Wang A, Pruksachatkun Y, Nangia N. et al. SuperGLUE: a stickier benchmark for general-purpose language understanding systems. In: Wallach H, Larochelle H, Beygelzimer A, d'Alché-Buc F, Fox E, Garnett R. eds. Advances in Neural Information Processing Systems. Vol 32. Curran Associates, Inc.; 2019. https://proceedings.neurips.cc/paper_files/paper/2019/file/4496bf24afe7fab6f046bf4923da8de6-Paper.pdf
    Google Scholar
  • 27 Sun T, Gaut A, Tang S. et al. Mitigating gender bias in natural language processing: literature review. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019: 1630-1640
    PubMedGoogle Scholar
  • 28 Pedersen JS, Laursen MS, Vinholt PJ, Alnor AB, Savarimuthu TR. Investigating anatomical bias in clinical machine learning algorithms. In: Findings of the European Chapter of the Association for Computational Linguistics: EACL 2023. Association for Computational Linguistics; 2023
    CrossrefGoogle Scholar
  • 29 Rogers EM, Singhal A, Quinlan MM. Diffusion of innovations: an integrated approach to communication theory and research. Routledge; 2014: 432-448
    Google Scholar
  • 30 Kahneman D. Thinking, Fast and Slow. Macmillan; 2011: 19-31
    PubMedGoogle Scholar
  • 31 Pedersen JS, Laursen MS, Soguero-Ruiz C, Savarimuthu TR, Hansen RS, Vinholt PJ. Domain over size: clinical ELECTRA surpasses general BERT for bleeding site classification in the free text of electronic health records. In: 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI). IEEE; 2022: 1-4
    PubMedGoogle Scholar
  • 32 Bojanowski P, Grave E, Joulin A, Mikolov T. Enriching word vectors with subword information. Trans Assoc Comput Linguist 2017; 5: 135-146
    CrossrefPubMedGoogle Scholar
  • 33 Laursen MS, Pedersen JS, Vinholt PJ, Hansen RS, Savarimuthu TR. Benchmark for evaluation of Danish clinical word embeddings. North Eur J Lang Technol. 2023;9(01):
    PubMedGoogle Scholar
  • 34 Salton G, Wong A, Yang CS. A vector space model for automatic indexing. Commun ACM 1975; 18 (11) 613-620
    CrossrefPubMedGoogle Scholar