Dynamic Few-Shot Prompting for Clinical Note Section Classification Using Lightweight, Open-Source Large Language Models
Read the abstract
Moderator
Presenter
Kurt Miller
Mayo Clinic Center for Digital Health
Statement of Purpose
Critical information about patient care is locked behind context-sensitive, domain-specific language of clinical note text. Detecting the structure of this clinical text has been demonstrated to enhance extraction of information embedded in this text by providing rich context, thereby improving performance of reliant downstream applications. However, due to the heterogeneity of section schema across healthcare institutions and steep demands of labelled data for supervised machine learning models, previous attempts to identify section composition have relied on large, annotated corpora or failed to generalize across diverse data.
Our work explores the application of a dynamic few-shot learning approach to section classification tasks of different section schema using relatively smaller, open-source large language models (LLMs). We compare performance across several LLMs, assess the optimal number of exemplars included in selected models’ context windows, conduct embedding training data ablation testing, and evaluate whether inserting additional context around each section imparts further performance benefits. The results exhibit the value of providing contextually relevant information to LLMs and the opportunity for generalizing across similar tasks with relatively small models and little training data.
Learning Objectives
- Differentiate the basic structural components of a clinical note.
- Develop and apply a dynamic few-shot, contextual, or similar in-context learning approach to a clinical natural language processing task.
- Analyze the significance of relevant contextual information to LLMs.
Additional Information
The target audience for this activity includes physicians, nurses, other healthcare providers, and medical informaticians.
No commercial support was received for this activity.
Completion of this “Other Activity (Regularly Scheduled Series – RSS)” is demonstrated by participating in the live webinar or viewing the on-demand recording, engaging with presenters during the live session by submitting questions, and completing the evaluation survey at the conclusion of the course.
Learners may claim credit and download a certificate upon submission of the evaluation. Participation in additional resources and the course forum is encouraged but optional.
The American Medical Informatics Association is accredited by the Accreditation Council for Continuing Medical Education (ACCME) to provide continuing medical education for physicians.
The American Medical Informatics Association designates this Other activity (Regularly Scheduled Series (RSS)) for a maximum of 12 AMA PRA Category 1 Credit(s)™. Physicians should claim only the credit commensurate with the extent of their participation in the activity.
The American Medical Informatics Association is accredited as a provider of nursing continuing professional development by the American Nurses Credentialing Center's Commission on Accreditation.
- Nurse Planner (Content): Robin Austin, PhD, DNP, DC, RN, NI-BC, FAMIA, FAAN
- Approved Contact Hours: 12 CME/CNE
It is the policy of the American Medical Informatics Association (AMIA) to ensure that Continuing Medical Education (CME) activities are independent and free of commercial bias. To ensure educational content is objective, balanced, and guarantee content presented is in the best interest of its learners and the public, the AMIA requires that everyone in a position to control educational content disclose all financial relationships with ineligible companies within the prior 24 months. An ineligible company is one whose primary business is producing, marketing, selling, re-selling or distributing healthcare products used by or on patients. Examples can be found at accme.org.
In accordance with the ACCME Standards for Integrity and Independence in Accredited Continuing Education, AMIA has implemented mechanisms prior to the planning and implementation of this CME activity to identify and mitigate all relevant financial relationships for all individuals in a position to control the content of this CME activity.
In accordance with the ACCME Standards for Integrity and Independence in Accredited Continuing Education, AMIA has implemented mechanisms prior to planning and implementation of this CME activity to identify and mitigate all relevant financial relationships for all individuals in a position to control the content of this CME activity.
Faculty and planners who refuse to disclose any financial relationships with ineligible companies will be disqualified from participating in the educational activity.
For an individual with no relevant financial relationship(s), course participants must be informed that no conflicts of interest or financial relationship(s) exist.
Disclosures
Planning Committee
The planning committee and reviewers reported that they have no relevant financial relationship(s) with ineligible companies to disclose.
- Joanna Abraham, PhD, FACMI, FAMIA
- Jifan Gao, MS
- Frances Hsu, BS, MS
- Sonish Sivarajkumar
- Song Wang, MS
- Faisal Yaseen
Presenter
The following presenters have no relevant financial relationship(s) with ineligible companies to disclose.
AMIA Staff
The AMIA staff have no relevant financial relationship(s) with ineligible companies to disclose.
*All of the relevant financial relationships listed for these individuals have been mitigated.
