Download PDF
Appl Clin Inform 2022; 13(02): 456-467
DOI: 10.1055/s-0042-1745829
Research Article

Usability and Acceptability of Clinical Decision Support Based on the KIIDS-TBI Tool for Children with Mild Traumatic Brain Injuries and Intracranial Injuries

Jacob K. Greenberg
1   Department of Neurological Surgery, Washington University School of Medicine in St. Louis, St. Louis, Missouri, United States
,
Ayodamola Otun
1   Department of Neurological Surgery, Washington University School of Medicine in St. Louis, St. Louis, Missouri, United States
,
Pyi Theim Kyaw
2   McKelvey School of Engineering, Washington University School of Medicine in St. Louis, St. Louis, Missouri, United States
,
Christopher R. Carpenter
3   Department of Emergency Medicine, Washington University School of Medicine in St. Louis, St. Louis, Missouri, United States
,
Ross C. Brownson
4   Brown School of Social Work, Washington University School of Medicine in St. Louis, St. Louis, Missouri, United States
,
Nathan Kuppermann
5   Department of Emergency Medicine, University of California Davis, Davis, California, United States
,
David D Limbrick Jr.
1   Department of Neurological Surgery, Washington University School of Medicine in St. Louis, St. Louis, Missouri, United States
,
Randi E. Foraker   *
6   Institute for Informatics, Washington University School of Medicine in St. Louis, St. Louis, Missouri, United States
,
Po-Yin Yen   *
6   Institute for Informatics, Washington University School of Medicine in St. Louis, St. Louis, Missouri, United States
› Author Affiliations Funding This study was supported by the U.S. Department of Health and Human Services, Agency for Healthcare Research and Quality (1F32HS027075-01A1), and Thrasher Research Fund (#15024).
› Further Information
   Permissions and Reprints

Abstract

Background The Kids Intracranial Injury Decision Support tool for Traumatic Brain Injury (KIIDS-TBI) tool is a validated risk prediction model for managing children with mild traumatic brain injuries (mTBI) and intracranial injuries. Electronic clinical decision support (CDS) may facilitate the clinical implementation of this evidence-based guidance.

Objective Our objective was to evaluate the acceptability and usability of an electronic CDS tool for managing children with mTBI and intracranial injuries.

Methods Emergency medicine and neurosurgery physicians (10 each) from 10 hospitals in the United States were recruited to participate in usability testing of a novel CDS prototype in a simulated electronic health record environment. Testing included a think-aloud protocol, an acceptability and usability survey, and a semi-structured interview. The prototype was updated twice during testing to reflect user feedback. Usability problems recorded in the videos were categorized using content analysis. Interview transcripts were analyzed using thematic analysis.

Results Among the 20 participants, most worked at teaching hospitals (80%), freestanding children's hospitals (95%), and level-1 trauma centers (75%). During the two prototype updates, problems with clarity of terminology and navigating through the CDS interface were identified and corrected. Corresponding to these changes, the number of usability problems decreased from 35 in phase 1 to 8 in phase 3 and the number of mistakes made decreased from 18 (phase 1) to 2 (phase 3). Through the survey, participants found the tool easy to use (90%), useful for determining a patient's level of care (95%), and likely to improve resource use (90%) and patient safety (79%). Interview themes related to the CDS's ability to support evidence-based decision-making and improve clinical workflow proposed implementation strategies and potential pitfalls.

Conclusion After iterative evaluation and refinement, the KIIDS-TBI CDS tool was found to be highly usable and useful for aiding the management of children with mTBI and intracranial injuries.

Keywords

clinical decision support - clinical information - traumatic brain injury - children - mixed methods

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects and was reviewed and approved by the authors' institutional review board. The authors' institutional review board reviewed and approved the study procedures with a waiver of documentation of consent (IRB #201902091). Therefore, participants were provided with a consent document, a verbal study description, and an opportunity to ask questions before verbally agreeing to proceed with the study.


* P.Y. and R.E.F. shared equal responsibility for study supervision.


Supplementary Material



Publication History

Received: 14 August 2021

Accepted: 18 February 2022

Article published online:
27 April 2022

© 2022. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 National Center for Injury Prevention and Control (US).. Report to Congress on mild traumatic brain injury in the United States: steps to prevent a serious public health problem. Centers for Disease Control and Prevention;; 2003
    Google Scholar
  • 2 McKinlay A, Grace RC, Horwood LJ, Fergusson DM, Ridder EM, MacFarlane MR. Prevalence of traumatic brain injury among children, adolescents and young adults: prospective evidence from a birth cohort. Brain Inj 2008; 22 (02) 175-181
    CrossrefPubMedGoogle Scholar
  • 3 Schneier AJ, Shields BJ, Hostetler SG, Xiang H, Smith GA. Incidence of pediatric traumatic brain injury and associated hospital resource utilization in the United States. Pediatrics 2006; 118 (02) 483-492
    CrossrefPubMedGoogle Scholar
  • 4 Koepsell TD, Rivara FP, Vavilala MS. et al. Incidence and descriptive epidemiologic features of traumatic brain injury in King County, Washington. Pediatrics 2011; 128 (05) 946-954
    CrossrefPubMedGoogle Scholar
  • 5 Bowman SM, Bird TM, Aitken ME, Tilford JM. Trends in hospitalizations associated with pediatric traumatic brain injuries. Pediatrics 2008; 122 (05) 988-993
    CrossrefPubMedGoogle Scholar
  • 6 Lumba-Brown A, Yeates KO, Sarmiento K. et al. Centers for Disease Control and Prevention Guideline on the Diagnosis and Management of Mild Traumatic Brain Injury Among Children. JAMA Pediatr 2018; 172 (11) e182853
    CrossrefPubMedGoogle Scholar
  • 7 Kuppermann N, Holmes JF, Dayan PS. et al; Pediatric Emergency Care Applied Research Network (PECARN). Identification of children at very low risk of clinically-important brain injuries after head trauma: a prospective cohort study. Lancet 2009; 374 (9696): 1160-1170
    CrossrefPubMedGoogle Scholar
  • 8 Babl FE, Borland ML, Phillips N. et al; Paediatric Research in Emergency Departments International Collaborative (PREDICT). Accuracy of PECARN, CATCH, and CHALICE head injury decision rules in children: a prospective cohort study. Lancet 2017; 389 (10087): 2393-2402
    CrossrefPubMedGoogle Scholar
  • 9 Greenberg JK, Jeffe DB, Carpenter CR. et al. North American survey on the post-neuroimaging management of children with mild head injuries. J Neurosurg Pediatr 2018; 23 (02) 227-235
    CrossrefPubMedGoogle Scholar
  • 10 Greenberg JK, Stoev IT, Park TS. et al. Management of children with mild traumatic brain injury and intracranial hemorrhage. J Trauma Acute Care Surg 2014; 76 (04) 1089-1095
    CrossrefPubMedGoogle Scholar
  • 11 Greenberg JK, Yan Y, Carpenter CR. et al. Development and internal validation of a clinical risk score for treating children with mild head trauma and intracranial injury. JAMA Pediatr 2017; 171 (04) 342-349
    CrossrefPubMedGoogle Scholar
  • 12 Neumayer KE, Sweney J, Fenton SJ, Keenan HT, Flaherty BF. Validation of the “CHIIDA” and application for PICU triage in children with complicated mild traumatic brain injury. J Pediatr Surg 2020; 55 (07) 1255-1259
    CrossrefPubMedGoogle Scholar
  • 13 Greenberg JK, Ahluwalia R, Hill M. et al. Development and external validation of the KIIDS-TBI tool for managing children with mild traumatic brain injury and intracranial injuries. Acad Emerg Med 2021; 28 (12) 1409-1420
    CrossrefPubMedGoogle Scholar
  • 14 Stiell IG, Bennett C. Implementation of clinical decision rules in the emergency department. Acad Emerg Med 2007; 14 (11) 955-959
    CrossrefPubMedGoogle Scholar
  • 15 Green SM. When do clinical decision rules improve patient care?. Ann Emerg Med 2013; 62 (02) 132-135
    CrossrefPubMedGoogle Scholar
  • 16 Sittig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care 2010; 19 (Suppl. 03) i68-i74
    CrossrefPubMedGoogle Scholar
  • 17 Kiatchai T, Colletti AA, Lyons VH, Grant RM, Vavilala MS, Nair BG. Development and feasibility of a real-time clinical decision support system for traumatic brain injury anesthesia care. Appl Clin Inform 2017; 8 (01) 80-96
    PubMedGoogle Scholar
  • 18 Greenberg JK, Otun A, Nasraddin A. et al. Electronic clinical decision support for children with minor head trauma and intracranial injuries: a sociotechnical analysis. BMC Med Inform Decis Mak 2021; 21 (01) 161
    CrossrefPubMedGoogle Scholar
  • 19 Nielsen J, Clemmensen T, Yssing C. Getting access to what goes on in people's heads? reflections on the think-aloud technique. Proceedings of the Second Nordic Conference on Human-Computer Interaction; Aarhus, Denmark; 2002
    PubMedGoogle Scholar
  • 20 Yen PY, Bakken S. A comparison of usability evaluation methods: heuristic evaluation versus end-user think-aloud protocol—an example from a web-based communication tool for nurse scheduling. AMIA Annu Symp Proc 2009; 2009: 714-718
    PubMedGoogle Scholar
  • 21 Hartzler AL, Chaudhuri S, Fey BC, Flum DR, Lavallee D. Integrating patient-reported outcomes into spine surgical care through visual dashboards: lessons learned from human-centered design. EGEMS (Wash DC) 2015; 3 (02) 1133
    PubMedGoogle Scholar
  • 22 Horsky J, Schiff GD, Johnston D, Mercincavage L, Bell D, Middleton B. Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions. J Biomed Inform 2012; 45 (06) 1202-1216
    CrossrefPubMedGoogle Scholar
  • 23 Jaspers MW, Steen T, van den Bos C, Geenen M. The think aloud method: a guide to user interface design. Int J Med Inform 2004; 73 (11-12): 781-795
    CrossrefPubMedGoogle Scholar
  • 24 Jaspers MW. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform 2009; 78 (05) 340-353
    CrossrefPubMedGoogle Scholar
  • 25 van den Haak M, De Jong M, Jan Schellens P. Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue. Behav Inf Technol 2003; 22 (05) 339-351
    CrossrefPubMedGoogle Scholar
  • 26 Brehaut JC, Graham ID, Wood TJ. et al. Measuring acceptability of clinical decision rules: validation of the Ottawa acceptability of decision rules instrument (OADRI) in four countries. Med Decis Making 2010; 30 (03) 398-408
    CrossrefPubMedGoogle Scholar
  • 27 Yen P-Y, Sousa KH, Bakken S. Examining construct and predictive validity of the Health-IT Usability Evaluation Scale: confirmatory factor analysis and structural equation modeling results. J Am Med Inform Assoc 2014; 21 (e2): e241-e248
    CrossrefPubMedGoogle Scholar
  • 28 Peres SC, Pham T, Phillips R. Validation of the System Usability Scale (SUS):SUS in the Wild. Proc Hum Factors Ergon Soc Annu Meet 2013; 57 (01) 192-196
    CrossrefPubMedGoogle Scholar
  • 29 Yen PY, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc 2012; 19 (03) 413-422
    CrossrefPubMedGoogle Scholar
  • 30 Yen PY, Walker DM, Smith JMG, Zhou MP, Menser TL, McAlearney AS. Usability evaluation of a commercial inpatient portal. Int J Med Inform 2018; 110: 10-18
    CrossrefPubMedGoogle Scholar
  • 31 Kushniruk AW, Borycki EM. Development of a video coding scheme for analyzing the usability and usefulness of health information systems. Stud Health Technol Inform 2015; 218: 68-73
    PubMedGoogle Scholar
  • 32 Nowell LS, Norris JM, White DE, Moules NJ. Thematic analysis: striving to meet the trustworthiness criteria. Int J Qual Methods 2017; 16 (01) 1609406917733847
    CrossrefPubMedGoogle Scholar
  • 33 Curry LA, Nembhard IM, Bradley EH. Qualitative and mixed methods provide unique contributions to outcomes research. Circulation 2009; 119 (10) 1442-1452
    CrossrefPubMedGoogle Scholar
  • 34 Tham E, Swietlik M, Deakyne S. et al; Pediatric Emergency Care Applied Research Network (PECARN). Clinical decision support for a multicenter trial of pediatric head trauma: development, implementation, and lessons learned. Appl Clin Inform 2016; 7 (02) 534-542
    Article in Thieme ConnectPubMedGoogle Scholar
  • 35 Grinspan ZM, Eldar YC, Gopher D. et al. Guiding Principles for a Pediatric Neurology ICU (neuroPICU) Bedside Multimodal Monitor: findings from an International Working Group. Appl Clin Inform 2016; 7 (02) 380-398
    Article in Thieme ConnectPubMedGoogle Scholar
  • 36 Gasson S. Human-centered vs. user-centered approaches to information system design. J Inf Technol Theory Appl 2003; 5 (02) 5 (JITTA)
    PubMedGoogle Scholar
  • 37 Masterson Creber RM, Dayan PS, Kuppermann N. et al; Pediatric Emergency Care Applied Research Network (PECARN) and the Clinical Research on Emergency Services and Treatments (CREST) Network. Applying the RE-AIM framework for the evaluation of a clinical decision support tool for pediatric head trauma: a mixed-methods study. Appl Clin Inform 2018; 9 (03) 693-703
    Article in Thieme ConnectPubMedGoogle Scholar
  • 38 Cabana MD, Rand CS, Powe NR. et al. Why don't physicians follow clinical practice guidelines? A framework for improvement. JAMA 1999; 282 (15) 1458-1465
    CrossrefPubMedGoogle Scholar
  • 39 Croskerry P. Clinical cognition and diagnostic error: applications of a dual process model of reasoning. Adv Health Sci Educ Theory Pract 2009; 14 (1, Suppl 1): 27-35
    CrossrefPubMedGoogle Scholar
  • 40 Ozkaynak M, Metcalf N, Cohen DM, May LS, Dayan PS, Mistry RD. Considerations for designing EHR-embedded clinical decision support systems for antimicrobial stewardship in pediatric emergency departments. Appl Clin Inform 2020; 11 (04) 589-597
    Article in Thieme ConnectPubMedGoogle Scholar
  • 41 Kane B, Carpenter C. Cognition and decision making. In: The Washington Manual of Patient Safety and Quality Improvement; Philadelphia, PA: Wolters Kluwer; 2016: 195-209
    Google Scholar
  • 42 Stark DE, Kumar RB, Longhurst CA, Wall DP. The quantified brain: a framework for mobile device-based assessment of behavior and neurological function. Appl Clin Inform 2016; 7 (02) 290-298
    Article in Thieme ConnectPubMedGoogle Scholar
  • 43 Gimbel RW, Pirrallo RG, Lowe SC. et al. Effect of clinical decision rules, patient cost and malpractice information on clinician brain CT image ordering: a randomized controlled trial. BMC Med Inform Decis Mak 2018; 18 (01) 20
    CrossrefPubMedGoogle Scholar