Skip to main content

Large Language Models Are Less Effective at Clinical Prediction Tasks Than Locally Trained Machine Learning Models

Read the abstract


Moderator

Song Wang, MS
University of Texas at Austin

Presenter

Katherine E. Brown, PhD
Vanderbilt University Medical Center

Statement of Purpose

Over the past several decades, medicine has been increasingly aided by artificial intelligence (AI) and particularly machine learning (ML). While model development has advanced, it is widely recognized that implementing ML models successfully often requires local, representative data. However, not all institutions have sufficient resources to implement ML effectively. Large language models (LLMs) have hinted at a potential to mitigate these challenges and fundamentally change the integration of ML in medicine. Closed-source LLMs are pre-trained and can be interacted with conversationally, characteristics that reduce technical friction required to create or use ML (or more broadly, AI) for healthcare settings.

In this study, we evaluate the utility, privacy, and fairness of LLMs compared to traditional ML, using electronic health record (EHR) data from Vanderbilt University Medical Center (VUMC) to predict the likelihood of patient discharge from the hospital within 24 hours and the public-use MIMIC-IV and MIMIC-IV ED datasets from Beth Israel Deaconess Medical Center (BIDMC) to predict the likelihood of transfer to the intensive care unit (ICU) within 24 hours after triage in the emergency department (ED).

Learning Objectives

  • Design a multi-faceted LLM evaluation that includes predictive performance, output calibration, data privacy, and algorithmic fairness.
  • Describe the advantages and disadvantages of LLMs and traditional ML for clinical prediction tasks.

Additional Information

The target audience for this activity includes physicians, nurses, other healthcare providers, and medical informaticians.

No commercial support was received for this activity.

Completion of this “Other Activity (Regularly Scheduled Series – RSS)” is demonstrated by participating in the live webinar or viewing the on-demand recording, engaging with presenters during the live session by submitting questions, and completing the evaluation survey at the conclusion of the course.

Learners may claim credit and download a certificate upon submission of the evaluation. Participation in additional resources and the course forum is encouraged but optional.

The American Medical Informatics Association is accredited by the Accreditation Council for Continuing Medical Education (ACCME) to provide continuing medical education for physicians.

The American Medical Informatics Association designates this Other activity (Regularly Scheduled Series (RSS)) for a maximum of 12 AMA PRA Category 1 Credit(s)™. Physicians should claim only the credit commensurate with the extent of their participation in the activity.

The American Medical Informatics Association is accredited as a provider of nursing continuing professional development by the American Nurses Credentialing Center's Commission on Accreditation.

  • Nurse Planner (Content): Robin Austin, PhD, DNP, DC, RN, NI-BC, FAMIA, FAAN
  • Approved Contact Hours: 12 CME/CNE

It is the policy of the American Medical Informatics Association (AMIA) to ensure that Continuing Medical Education (CME) activities are independent and free of commercial bias. To ensure educational content is objective, balanced, and guarantee content presented is in the best interest of its learners and the public, the AMIA requires that everyone in a position to control educational content disclose all financial relationships with ineligible companies within the prior 24 months. An ineligible company is one whose primary business is producing, marketing, selling, re-selling or distributing healthcare products used by or on patients. Examples can be found at accme.org.

In accordance with the ACCME Standards for Integrity and Independence in Accredited Continuing Education, AMIA has implemented mechanisms prior to the planning and implementation of this CME activity to identify and mitigate all relevant financial relationships for all individuals in a position to control the content of this CME activity.

In accordance with the ACCME Standards for Integrity and Independence in Accredited Continuing Education, AMIA has implemented mechanisms prior to planning and implementation of this CME activity to identify and mitigate all relevant financial relationships for all individuals in a position to control the content of this CME activity.

Faculty and planners who refuse to disclose any financial relationships with ineligible companies will be disqualified from participating in the educational activity.

For an individual with no relevant financial relationship(s), course participants must be informed that no conflicts of interest or financial relationship(s) exist.

Disclosures

Presenter

The following presenters have no relevant financial relationship(s) with ineligible companies to disclose.

  • Katherine E. Brown, PhD

AMIA Staff

The AMIA staff have no relevant financial relationship(s) with ineligible companies to disclose.

*All of the relevant financial relationships listed for these individuals have been mitigated.

 

Dates and Times: -
Type: JAMIA Journal Club
Course Format(s): Live In Person
Credits:
1.00
CME
,
1.00
CNE
Register now
Share