AMIA 2025 Clinical Informatics Conference On Demand Bundle

Increasing STI Screening Rates in a Pediatric Ambulatory Network through Phased EHR Integration and Change Management

In response to high Chlamydia trachomatis (CT) and Gonococcal (GC) rates among young people, our large pediatric network implemented a phased STI screening approach integrating questionaries, targeted alerts, innovative EHR build, and physician incentives to arrive at universal screening. Our data-driven approach, aligned with clinical workflow and supported by adherence to change management principles, was key to the successful improvement of both screening rates and positive results identified.

Learning Objective

• Understand a phased implementation strategy for STI screening using EHR tools, financial incentives, and workflow integration in a large pediatric network.

Speaker

• Greg Lawton, MD (Children's Hospital of Philadelphia)

Multidisciplinary Creation of Best Practice Alert to Reduce Excessively High Stool Output in the NICU

This session highlights the successful implementation of a BPA in a NICU to improve the detection of high stool output episodes. Attendees will learn strategies for multidisciplinary collaboration to address patient safety concerns, develop evidence-based thresholds, and optimize CDS tools. The case study demonstrates a 60% reduction in high stool output events and emphasizes balancing alert effectiveness with minimizing fatigue. Participants will leave equipped to design, test, and implement CDS interventions that enhance patient outcomes.

Learning Objective

• Use multidisciplinary team collaboration to address patient safety concerns and develop targeted clinical decision support interventions that improve outcome

Speaker

• Alexa Gilman, MSN, APRN, NNP-BC (Lurie Children's Hospital)

Enhancing Postoperative Documentation for Procedures: Implementation and Utilization of a Standardized Postoperative Checklist for Acetabular Fracture Surgeries at a Level 1 Trauma Center

Implementation of a structured postoperative documentation checklist within Epic significantly improved consistency and quality in documenting acetabular surgeries at a Level 1 trauma center. Triggered by specific CPT codes, the checklist facilitated comprehensive recording of key surgical findings, achieving an 80-90% utilization rate over two years. This standardized approach enhanced end-user consistency, supported uniform data collection and underscored the value of systematic documentation in optimizing patient outcomes.

Learning Objective

• Implement a procedure code-driven checklist in EHR systems for improved postoperative documentation.

Speaker

• Aly Toure, BS (Emory School of Medicine)

EHR integrated clinical pathways: a rapidly iterative, experimental approach for Learning Health Systems

Penn Medicine used an existing clinical pathways program and a new alliance to establish a Learning Health System (LHS) model to integrate evidence-based clinical pathways into an EMR. Integrated pathways enabled real-time guidance, outcome assessment, and clinician feedback. In one year, 17 pathways were implemented, with 2,648 providers using them 28,456 times. This LHS model rapidly deployed evidence-based pathways, improved workflows, and increased preferred treatment orders, demonstrating a simple, scalable approach to high-quality care integration.

Learning Objective

• Define the core components and goals of a Learning Health System (LHS)

Speaker

• Joy Iocca (Penn Medicine)

Using the OMOP Common Data Model for Implementing Electronic Clinical Quality Measures

Implementing electronic Clinical Quality Measures (eCQMs) costs healthcare systems billions annually, with each institution independently developing measurement logic. We explored using the OMOP Common Data Model to standardize eCQM implementation, focusing on a Postoperative Venous Thromboembolism measure. The OMOP-based implementation achieved superior accuracy compared to traditional CQL-based methods, demonstrating potential for reducing costs while improving measurement precision. We present a practical framework for implementing eCQMs using OMOP.

Learning Objectives

• Identify challenges in implementing eCQMs using traditional methods.

Speaker

• Steven Zeck (UC Davis Health)

Improving Medication Care for Veterans: Using FHIR to Support Sharing of Complete, Correct and Consistent Medication Data at the Veterans Health Administration

Safe and effective medication decisions depend on accurate and comprehensive medication history data. The process of taking medication history varies by the history taker’s habits and education, and also by the electronic health record’s capabilities for the presentation of objective data. Gaps and inconsistencies in the resulting history may result in mistakes in medication reconciliation and treatment planning. Standardizing the information expected in this activity will support a comprehensive medication history, regardless of the EHR being used. A draft specification was validated against best practices and quality issue reports and was aligned with existing standards efforts and regulations. A roadmap is proposed for establishing standards consensus to enable multi-system support for this critical task.

Learning Objectives

• Explain how standardizing medication information management tools will reduce clinician burden and enhance safety and quality of care.

Speaker

• Laura Heermann Langford, PhD, RN, FAMIA, FHL7 (Veteran's Affairs)

Babies Aren’t Born with Cell Phones, Yet: Initiative to Improve the Accuracy of Pediatric Patient & Family Contact Information

Patient-centered pediatric care depends on effective family engagement, smooth transitions to autonomous adult care, and adolescent confidentiality protection. A key barrier is the difficulty in distinguishing patient and guardian contact information in EHRs, leading to low staff confidence and frequent adolescent portal signup errors that risk confidentiality breaches. This session presents strategies to improve pediatric EHR documentation, reducing inappropriate email and mobile phone entries and facilitating secure and confidential communication.

Learning Objectives

• Apply best practices for accurately documenting patient and guardian contact information in pediatric electronic health records (EHRs)

Speaker

• Laura Pike, BA, PMP (Stanford Medicine Children's Health)

Design and Development of a Clinical Informatics Fellowship Scheduling Dashboard Using Agile Methodologies

Fellowship training in clinical informatics present unique challenges for leaders, instructors, and trainees as graduation requirements, Accreditation Council for Graduate Medical Education (ACGME) expectations, and graduate medical education (GME) budgetary requirements can be complex. We sought to create a tool to meet the needs of fellows and fellowship leaders using Agile methodologies, and successfully created a dashboard that satisfies scheduling and program monitoring requirements for multiple stakeholders.

Learning Objectives

• Design and implement a scheduling dashboard that meets the individualized needs of Clinical Informatics fellows and program leadership.

Speaker

• Averi Wilson, MD (University of Texas Southwestern)

Onboarding Physician Builders and SmartUsers - What worked, What didn't, and What's next

In 2024, NYU Langone Health completed Version 7 of the Epic Honor Roll program. A key component involved recruiting active providers to complete Epic-led courses to become Physician Builders. NYULH developed a comprehensive recruitment strategy, including targeted outreach, presentations, and an incentive program. This session will explore the challenges of engaging providers, the effectiveness of different incentive models, and the long-term strategy for growing the certified Physician Builder cohort to drive continued EHR improvements.

Learning Objectives

• Propose an adaptable framework for recruiting clinicians into Epic Builder and SmartUser roles.

Speaker

• James Davydov, MSc (NYU Langone Health )

Results of a national survey of the clinical informatics fellowship class of 2024 on their fellowship experiences and early careers

The first clinical informatics (CI) fellows graduated in 2016. There is limited information about the careers of CI fellowship alumni. We first conducted an IRB approved survey emailed to all CI fellows graduating in 2022. This year, we surveyed CI fellows graduating in summer/fall 2024 about their background, fellowship experiences, job search, and first position after fellowship. More than half (43/80) consented to participate and met inclusion criteria (graduation date).

Learning Objectives

• Describe strategies for evaluating clinical informatics fellowship opportunities, identify approaches for enhancing fellowship curricula to support diversity and career development, and assess the qualifications of fellowship graduates for employment in clinical informatics roles."

Speaker

• Ellen Kim, MD (Harvard Medical School)

Designing an Electronic Patient Reported Outcomes Information Architecture Reinforced by the RE-AIM Implementation Framework

To improve patient care using patient-reported outcomes (PROs), we developed an electronic health record (EHR)-integrated PRO framework based on RE-AIM principles. Using Epic Systems, we implemented the MDASI-HN questionnaire in radiation oncology, achieving 13,156 submissions with 82% compliance over 12 months. Our implementation-evaluation model supports adaptive, scalable ePRO tools for key stakeholders (patients, staff, facilitators), enhances clinical decision-making by optimizing PRO visualizations and functions, and enables continuous evaluation to detect and address implementation barriers.

Learning Objectives

• Identify and define key implementation outcomes and metrics to guide the development and evaluation of ePRO programs.

Speaker

• Amy Moreno, MD (The University of Texas MD Anderson Cancer Center)

Testing a free, fast, and secure method for routing public transit from patient address to the point of care

Despite the importance of understanding transportation barriers, there is no tool integrated with the electronic health record providing timely and granular information about public transportation accessibility during clinical encounters or for research. Therefore, a framework for a public transit routing system comprised of free, publicly available data and software sources that are offline to protect patient data was created and implemented, mapping 440,000 routes from home address to University of Maryland Medical Center.

Learning Objectives

• Learn how to experiment with OpenTripPlanner2 to route transit directions offline

Speaker

• Sinan Aktay, B.S. (University of Maryland School of Medicine)

Improving Vaccine Confidence: Usability Testing of a Caregiver-Centric mHealth App

This study evaluates a beta version of a mHealth app designed to support vaccination confidence among caregivers regarding well-child checks in the first year of life. Feedback on the alpha version was gathered from rural and urban community advisory boards resulting in a beta version tested with English- and Spanish-speaking caregivers. Results indicate high app usefulness, trust, and accessible content. Key usability issues highlighted a need for improved navigation, content clarity, and functionality.

Learning Objectives

• Assess the potential impact of mHealth app on pediatric preventative care, specifically relating to well-child checks in the first year of life.

Speaker

• Elizabeth Reisher, MS (University of Nebraska Medical Center)

Refresh, Refresh, Refresh: Association of Repeated Access to the Patient Portal Awaiting Test Results with Patient Messaging

Many patients report heightened worry while awaiting test results in the patient portal. Identifying patients who exhibit behaviors associated with worry while waiting for results could allow health systems to proactively support patients and reduce message volumes. We used portal access logs to study “refresh” behavior, where patients repeatedly access the portal while awaiting new test results. We identified characteristics of patients who refresh for results and measured the association between refreshing and subsequent messaging.

Learning Objectives

• Identify opportunities to operationalize patient portal access logs to gather real-time insight into patient worry.

Speaker

• Bryan Steitz, PhD (Vanderbilt University Medical Center)

Utilization and Impact of Artificial Intelligence-Generated Draft Replies to Patient Messages in Pediatrics

AI-generated draft replies to patient messages have not been studied in pediatric clinical settings, raising concerns about the feature’s acceptability and applicability to this context. In this single-site cohort study, users from both pediatric- and adult-facing specialties were given access to AI-generated drafts. Pediatric providers reported significant reduction in task load associated with responding to patient messages and recommended the tool more highly than adult providers despite overall lower utilization of generated drafts.

Learning Objectives

• Evaluate the effectiveness and acceptability of AI-generated draft replies in reducing task load for pediatric providers, and compare provider responses across pediatric and adult clinical settings.

Speaker

• April Liang, MD (Stanford University)

Adoption and Utility: Evaluation of Usage Rate and Editing Overlap for Artificial Intelligence-Drafted Replies to Patient Messages

This session will provide insight into the implementation and evaluation of Generative Artificial Intelligence (AI) technology to respond to patient portal messages. Attendees will learn about the high variation in usage rates and editing efforts, prompting a discussion of best practices for implementation of these technologies in attendees’ own clinical settings. They will also learn data analysis and visualization strategies they can apply when implementing and evaluating the utility of AI tools in clinical practice.

Learning Objectives

• Gain insight into the implementation and evaluation of Generative Artificial Intelligence (AI) technology performed at another organization.

Speaker

• Aasf Hanish, MPH (Penn Medicine)

Enhancing Patient Communication: The Impact of LLMs on Care Team Messaging Workflows

The deployment of AI-generated draft replies in healthcare settings has introduced new dynamics in how medical assistants (MAs), nurses (RNs), and physicians (MDs) interact with patient messages. We performed inductive coding of messages sent with and without access to draft replies to investigate their influence on scope creep and error rates in message handling by MAs and found that scope creep was rare and did not appear to be adversely affected by the use of AI-generated draft messages. We also explored the utility of fine-tuned LLMs for the automated triage of incoming messages, which would have the potential to redirect incoming patient messages with clinical questions to the appropriate role with the appropriate urgency. We found that even small open-source LLMs such as Llama-3-8B was able to achieve high accuracy when compared to human labels. These results underscore the potential for increased automation through the use of LLMs in clinical settings.

Learning Objectives

• Understand how AI generated draft replies to patient messages can impact user behavior

Speaker

• Stephen Ma, MD, PhD (Stanford University School of Medicine)

Automation of Critical Lab Result Communication Improves Lab Efficiency

An academic health system designed and implemented an automated critical lab results notification system to enhance lab staff efficiency. Based on technologies from Epic, Twilio, UiPath, and Sunquest, a bot called the provider using interactive voice response and, after confirming identity, delivered the results with Epic secure chat. The automation achieved a 27.4% success rate and maintained the 30 minute turnaround time benchmark.

Learning Objectives

• Develop a workflow to automate critical lab results using Robotic Process Automation.

Speaker

• Jonathan Austrian, MD (NYU Langone Medical Center)

Transformative Role of Generative Artificial Intelligence in Inpatient Medicine: Real-World Implementations and Future Directions

This presentation explores the transformative potential of Generative Artificial Intelligence (GenAI) in inpatient medicine, focusing on real-world implementations at NYU Langone Health. Attendees will learn about successful applications, including automatic hospital course generation, patient-friendly discharge narratives, secure chat classification and using GPT to improve medication safety. The session will also address responsible use, safety measures, and end-user education, aiming to enhance patient care and clinician efficiency.

Learning Objectives

• Understand how specific applications of GenAI can be used to improve patient care, provider efficiency, and hospital operations

Speaker

• Paawan Punjabi, MD, MSc (New York University School of Medicine/NYU Langone Health System)

AI-Driven Automation of Procedural Case Log Documentation

Physician case logs are crucial for certification, credentialing, and monitoring medical training, yet their documentation in radiology remains a labor-intensive, manual process. Residents often record over 6,000 procedures manually during their training, which not only diverts time from clinical activities but also increases the likelihood of errors. This inefficiency hampers both the residents and their training programs, highlighting the need for automated solutions. To address this, we employed large language models (LLMs) to automate case log documentation by processing procedural reports and answering predefined questions across three sections: Vascular Diagnosis, Vascular Intervention, and Nonvascular Intervention. Using Meditron-70B and MedLLaMA2-7B, we evaluated their performance against gold-standard annotations from a trained physician. Meditron-70B demonstrated superior accuracy, achieving an F1-score of 72.21% compared to MedLLaMA2's 30.90%. It excelled in precision (>88%) and recall (>59%), particularly in vascular tasks, underscoring its potential for automating this critical process. Future work will focus on fine-tuning LLMs using ground truth data to enhance alignment with case log requirements. Additionally, we plan to expand the dataset beyond the initial 60 reports to thousands of cases, enabling more comprehensive evaluations and improvements. By integrating more robust architectures and scaling the analysis, we aim to create a reliable solution that reduces documentation errors, saves time, and improves the overall efficiency of medical training workflows. This approach paves the way for transforming case log documentation into an efficient and error-free process.

Learning Objectives

• Identify the limitations of manual procedural case logging in radiology training and the need for automated, scalable solutions.

Speaker

• Nafiz Imtiaz Khan, Student (University of California - Davis)

Implementing AI-Driven Patient Summarization in Electronic Health Records: Early Insights, Best Practices, and Impact Evaluation

The integration of generative artificial intelligence (AI) in healthcare promises to alleviate documentation burdens and enhance patient care. At the Children’s Hospital of Philadelphia (CHOP), we integrated an "AI Note Summarization" tool into our electronic health record to automate the extraction of patient histories from existing notes. This pilot, involving 22 clinical staff in an outpatient setting, leveraged ChatGPT-4o within a HIPAA-compliant framework to generate concise narrative summaries. Since June of 2024, our pilot group has been assisting our AI-working group in assessing the tool's effectiveness, safety, and overall experience. The tool has been live in production environments since August of 2024, with ongoing pilot user testing and feedback. Our pilot results have been compelling. Over 2,370 summaries were generated from more than 32,426 notes with a minimal cancellation rate of 0.5%. User feedback indicates a high 70% satisfaction rate, with 90% of users affirming enhanced clinical workflows and 50% reporting increased insights into their patient's histories. Continuous bi-monthly reviews by the CHOP AI team help refine the tool, focusing on the tool's reliability and utility in clinical practice.

Learning Objectives

• Evaluate the process of implementing and assessing AI tools in clinical settings, focusing on governance, pilot testing, and feedback collection.

Speaker

• Osvaldo Mercado, MD (Children's Hospital of Philadelphia)

Team-based ordering in ambulatory care: trends and impact

Collaborative ordering workflows constitute a potential area for administrative burden alleviation for physicians, with the potential to reduce burnout and increase both work satisfaction and productivity. We use nationally representative data capturing nearly 250,000 ambulatory physicians observed over three years to evaluate whether higher rates of teamwork for ordering are associated with time savings in the EHR and productivity. We find clinically significant ordering time savings (20-30% reduction) as well as substantial spillover effects and productivity gains.

Learning Objectives

• Understand national rates of team-based ordering (i.e., non-physician pending of orders for physician sign-off) in ambulatory settings and changes that follow after physicians newly adopt team-based ordering workflows.

Speaker

• Nate Apathy, PhD (University of Maryland)

Identifying trajectories of clinician engagement with the EHR among inpatient hospitalizations

This study used Electronic Health Record (EHR) audit logs, admission data, and patient characteristics to measure clinician engagement with the EHR during inpatient medicine hospitalization. Group-based trajectory modeling was used to identify encounters with similar trajectories of clinician engagement. Six distinct groups were identified, with statistically significant differences in demographic (sex, age, race, insurance) and clinical (length of stay, admission location, nurse and doctor actions per day) characteristics between groups.

Learning Objectives

• Apply hospitalization-level EHR audit logs to use cases relevant to their own research or clinical settings

Speaker

• Daphne Lew, PhD, MPH (Washington University in St. Louis)

Characterizing Primary Care Physicians’ Work Effort and Its Determinants

This cross-sectional study describes the estimated yearly work effort required per patient on a PCP’s panel, how work effort varies by clinical FTE (cFTE), and the patient characteristics influencing differential time expenditure. We identified factors such as medical comorbidities, acute care utilization, and panel composition that affect PCP workload. This knowledge will help inform decisions on panel management, workforce sustainability, and risk-adjustment strategies in primary care settings."

Speaker

• Lisa Rotenstein, MD, MBA, MSc (UCSF)

Few Family Medicine Physicians Experience Ideal Interoperability

In a survey of over 7500 family medicine physicians with a 100% response rate in 2024, only a small fraction of physicians reported ideal interoperability experiences for clinical data (for instance, just 13% reported ideal interoperability for medications), defined as often automatically obtaining documents from outside organizations, easily finding the document, and easily finding information within the document in their EHR.

Learning Objectives

• Understand how family physicians described ideal interoperability.

Speaker

• Julia Adler-Milstein, PhD (University of California, San Francisco)

Capturing the Visitome: Sociotechnical Ethnography Through Clinical Video in the Observer Repository

The Observer Repository includes video recordings of clinical encounters, electronic health record (EHR) data, and patient and provider satisfaction information. This dataset aims to address challenges in healthcare including clinician burnout, inefficient EHR workflows, and limited visit access by innovators by enabling research, testing hypotheses about improving primary care, and bridging engineering and healthcare. The Observer Repository uses a privacy- preserving pipeline, making it an accessible resource for understanding patient-provider interactions and improving care qualit

Learning Objectives

• Establish robust protocols in collaboration with their institution’s IRB to streamline data collection processes

Speaker

• Basam Alasaly, Biomedical Informatics, M.S. (Perelman School of Medicine at the University of Pennsylvania)

Evaluating the Adoption of Billing Patient Messages as ‘E-Visits’ and Impact on Physician Burnout

Clinician-patient messaging grew dramatically at the beginning of the pandemic and has persisted at high levels. In 2020, CMS allowed new reimbursement for patient medical advice requests (e-visits): secure messages that require both medical decision-making and at least five minutes of clinician time. In response, several health systems implemented clinician-initiated billing for these messages, including UCSF Health in November 2021. To understand adoption and efficacy of this intervention, we first used Epic Clarity EHR metadata to identify physician adopters of e-visits and matched these to self-reported physician wellbeing survey data. We used a difference-in-differences analysis with ordinary least squares regression to assess how e-visit adopter physicians’ responses on overall burnout and callousness towards others changed compared to those who did not adopt e-visits. We then conducted semi-structured interviews with UCSF physician adopters of e-visit billing in order to understand how they opted to employ this functionality and perceived its effect on symptoms of burnout. Our quantitative results show that e-visit adopters reported significant reductions in callousness towards others. Interviews revealed that the initial implementation did not align with how physicians perceive this work. Specifically, interviewees cited low valuation, additional clicks, and discomfort initiating billing as barriers to billing for e-visits to a fuller extent and as factors contributing to burnout. However, physicians were optimistic about its potential to assign value to an important care modality and alleviate burnout symptoms. Future efforts should focus on aligning incentives to encourage physicians to continue utilizing this care modality and adopt billing for it.

Learning Objectives

• Identify key barriers to physician adoption of e-visit billing, based on physician-reported experiences.

Speaker

• A J Holmgren, PhD (University of California, San Francisco )

The Impact of AI Scribes on Clinicians’ Electronic Health Record Time: Initial Results from the Multisite Ambient Clinical Documentation Collaborative

Time on the electronic health record (EHR) is associated with burnout among physicians. While human scribes have been shown to reduce EHR time, there are cost and person-power limitations to scaling their use. Given these limitations, healthcare organizations across the United States are now testing the ability of artificial intelligence-powered scribes to ease documentation burden while enhancing the clinician experience. The Ambient Clinical Documentation Collaborative is a consortium of healthcare organizations across the United States that have adopted AI scribes and use Epic Systems as their EHR. Sites include the Geisinger Health System, Emory Healthcare, Mass General Brigham, New York University, the University of California at San Francisco, the University of California at Davis, the University of California at San Diego, the University of Rochester, and Yale New Haven Health System/Yale Medicine. In this study of Collaborative participants, we describe the range of settings and ways in which AI scribes are being implemented, their impact on physicians’ EHR time, and the factors associated with significant time benefits of AI scribe technology. Our preliminary results suggest variability in the implementation of AI scribe technology across the large healthcare systems represented. AI scribes are most commonly being used in the outpatient setting, and early analyses suggest the benefits of AI scribe technology for reducing clinicians’ EHR time expenditure and improving documentation efficiency, with some variability across vendors. These results can guide clinical and information technology leaders in their AI scribe implementation approaches.

Learning Objectives

• Identify the impact of AI scribes on physicians’ EHR time.

Speaker

• Lisa Rotenstein, MD, MBA, MSc (UCSF)

Comparative Case Study on Implementing Generative AI in Medical Practices to Ease Documentative Overburden: A Sociotechnical Systems Perspective

This is a comparative case study of a live implementation of Generative AI solution in 5 medical practices. We shed new light on the impact of Generative AI on various aspects such as social structures, roles, organizational processes, and technical systems of medical practices. It is well known now that increasing documentation burden on physicians has led to medical errors, patient safety concerns, and physician burnout. This study investigates the adoption and implementation of a Generative AI based clinical documentation technology in medical practices over a span of 5 months. Our data consisted of interviews, participant observations, process documentation and mapping, tracking social interactions, and analyzing textual user feedback data. The results reveal a process framework that can be generalized across medical practices, categorizing changes into social, technical, organizational, and goals & outcomes. The implementation of Generative AI has led to both tangible and intangible benefits, including the creation of a new role of Scribe to provide human oversight of AI-generated clinical documentation. Resistance and apprehensions from practice staff have impacted implementation speed and decision-making. The study emphasizes the importance of considering social and organizational process changes in the adoption of new technologies and identifies role re-reforming and triadic co-creation as key concepts. The study also includes an entrepreneur’s and emerging technology product implementation team’s experiences of the co-creation with the medical practices. Overall, this research provides a processual framework to capture the nuances of the adoption and co-evolution of an emergent and uncertain technology.

Speaker

• Sri Ramesh Eevani, Doctorate in Business Administration (Healthfirst)

Learning Objectives

• At the conclusion of this presentation the learner will be able to gain insights on qualitative case study approach of Generative AI implementation at medical practices.

Clinician Personas for Ambient Artificial-Intelligence Scribing Documentation

Ambient documentation, also known as AI-based scribes, is being used by healthcare systems to address documentation burden. Understanding clinician phenotypes for ambient documentation would allow implementation teams to determine clinicians who would benefit from the technology. We will present personas for adoption of ambient documentation derived from interview and survey data of our ambient documentation users.

Learning Objectives

• Interpret qualitative feedback from clinicians regarding the use of AI scribing technology in clinical practice

Speaker

• Julie Wang, BS (Harvard Medical School)

Rejuvenating Clinician Wellbeing with Ambient Documentation

Mass General Brigham and Emory Healthcare both piloted ambient documentation, AI-assisted scribing, at their respective institutions. Preliminary results show statistically significant improvements in clinician burnout/wellbeing scores after 6 weeks of exposure to ambient documentation. Ambient documentation is a promising tool to rejuvenate clinician wellbeing at a time of high burnout in healthcare.

Learning Objectives

• Identify ambient documentation as a tool to address clinician burnout and understand lessons learned from large ambient documentation pilots.

Speaker

• Jacqueline You, MD (Mass General Brigham)

Improving Alignment Between an Infant Sepsis Prediction Model and User Expectations Using Human-Centered Design Methods

We developed a machine learning model using electronic health record data to improve neonatal sepsis recognition. In developing system designs to study the presentation of model output to clinicians, we applied two human-centered design methods: clinician interviews and rapid prototyping. Methods: The dynamic rapid prototypes, using nearly 60 patient and model data elements per hour over 72 hours, allowed us to visualize patient data, model predictions, and feature importance. Visualization of this data revealed anomalies such as shifts in model output and discrepancies between feature importance and results from the clinician interviews. A multidisciplinary team with expertise in neonatology, data science, and human-computer interaction reviewed these anomalies using clinician interview analysis, patient chart reviews, electronic health record data analysis, and model code reviews. Results: The review process resulted in identifying three categories of anomalies: feature selection, feature importance, and model stability. This process resulted in over 40 changes to the model. Conclusion: While discovered ad hoc, our experience suggests more rigorous strategies for applying human-centered design methods beyond the presentation of machine learning model output to the development and testing of models.

Learning Objectives

• Apply human-centered design (HCD) methods, such as end-user interviews and rapid prototyping with low-cost data visualizations, to identify anomalies in a NICU sepsis machine learning model’s output.

Speaker

• Alex Ruan, MD (Children's Hospital of Philadelphia)

Formative Usability Testing of Designs to Present Machine Learning Output for Improving Sepsis Recognition in Critical Infants

Background: Traditional usability testing evaluates system usability but does not address explainability, which is crucial for systems using machine learning or other forms of artificial intelligence (AI). We developed a machine learning model to improve sepsis recognition in the neonatal intensive care unit (NICU). Methods: In an ongoing study to explore model representation and use, we developed system mockups using patient and model data from four patients. The mockups utilized nearly 200 data elements per patient and were tested iteratively in a format designed to observe user-system problems, assessment of usability using the Post-Study Scenario System Usability Scale (PSSUQ), and an assessment of explainability informed by published methods extending the Technology Assessment Model (TAM) with new constructs such as trust and understandability. Previous work in interviewing 30 NICU clinicians identified NICU nurses and advance practice providers (APP) as potentially benefiting most from the model. Thirty clinicians (15 nurses, 15 APP) from two tier four NICUs participated in the test. Results: Formative testing resulted in seven iterative versions of the system. Testing revealed and addressed usability problems with format, layout, labeling, and support content. Explainability problems identified and addressed include data science terminology, model feature importance, and presenting model data over 24 hours. PSSUQ scores were positive and consistent across all seven versions and overall responses to the TAM based questionnaire indicated high agreement with the explainability of the system. Conclusion: This study demonstrates that adapting usability testing to include explainability effectively identifies and resolves issues in AI-based systems.

Learning Objectives

• Develop a system that presents machine learning output for user testing

Speaker

• Alex Ruan, MD (Children's Hospital of Philadelphia)

Usability of integrated care pathways at a freestanding children’s hospital

Integrative care pathways (ICPs) are evidence-based, structured care plans used to improve the quality of care and patient outcomes in individuals presenting with a specified clinical problem. While ICPs can potentially be valuable, the healthcare team's usability has historically been a barrier due to the lack of integration into their workflows. AgileMD is an ICP application that integrates into the electronic health record (EHR) and can be utilized to create ICPs for various conditions. In June 2023, Children’s Nebraska, a freestanding children’s hospital, began rolling out ICPs using AgileMD. However, the tool's usability was not assessed among nurses. Therefore, this project aims to describe the usability of the Junctional Ectopic Tachycardia (JET) pathway and the Chylothorax pathway by comparing outcome metrics between the AgileMD pathway and the standard workflow for nurses within the CCU. The Clinical Effectiveness (CE) team developed a nursing task-driven simulation case for the Chylothorax pathway and the JET pathway. Two test patients were developed in the EHR playground. After completing both case simulations, each nurse completed the NASA-TLX and the System Usability Scale (SUS). A total of 16 CCU nurses completed the study. Results of the NASA-TLX demonstrated a significantly lower cognitive load across all domains for the AgileMD workflow compared to the standard workflow. The SUS score of 91.85 corresponds to an A+ letter grade, indicating “Best imaginable” usability. This study will expand to assess the task completion, click burden, and eye fixation between the two modalities.

Learning Objectives

• Develop a system that presents machine learning output for user testing

Speaker

• Kelsey Zindel, DNP, APRN-NP, CPNP-AC/PC (Children's Nebraska)

Patterns in Viewing of Pediatric Portal Notes

The 21st Century Cures act mandated the sharing of all clinical notes to patients unless exempted by limited allowable exceptions. However, little is known about who is accessing notes and what types of notes are being viewed. IRB exception was granted for review of metadata from all notes for all patients < 25 yo from a multi-state health system between July 2022 – June 2023. We collected information on patient demographics, note types, and author specialties. Continuous variables were summarized using medians and categorical variables as percentages. Further statistical analysis using GEE models pending with significant p <0.05. 1,578,188 unique notes were collected from 419,136 individual patients with 1,269,828 shared on activated patient portals. <1% of notes were blocked by providers based on acceptable exemptions. Notes were more likely to be viewed if patients were younger patients (0-2 years) (24.5%) compared to older (18+) (20.4%), English speaking (24.5%) vs Spanish (14.0%) or Other (19.1%), were privately insured, or considered medically complex. Notes from outpatient specialists were viewed most often (n=156,982, 33%), followed by outpatient primary care (n=103,572, 27%), emergency department (n=11,430, 11%), and inpatient (n=28,923, 9%) notes. Viewership by outpatient note specialty ranged from 3% (Radiology) to 45% (Genetics). Viewership by inpatient note specialty ranged from 3% (Neonatology, Rehabilitation Medicine) to 22% (Anesthesia). The data suggest that demographic and clinical factors influence the viewing of notes. These findings can inform strategies to improve access to information among families of all backgrounds to address the digital divide.

Learning Objectives

• Identify practice changes related to 21st Century Cures Act Describe note blocking, sharing and viewing rates between visit types Describe Odds Ratio access rates between demographics, note type, and specialties

Speaker

• Gift Kopsombut, MD (Nemours)