Skip to main content

A Framework for Making Predictive Models Useful in Practice

This on-demand webinar does not offer CE credit.

Jung K, Kashyap S, Avati A, et al. A framework for making predictive models useful in practice [published online ahead of print, 2020 Dec 22]. J Am Med Inform Assoc. 2020; ocaa318. doi:10.1093/jamia/ocaa318.

Read the article

Watch the Recording



Nigam Shah, MBBS, PhD
Professor of Medicine (Biomedical Informatics) and of Biomedical Data Science
Stanford University

Manager and Moderator

Hannah Burkhardt
PhD candidate
University of Washington School of Medicine
Biomedical Informatics and Medical Education

Statement of Purpose

The widespread adoption of electronic health records (EHRs) has turned the record of routine clinical practice into data that can be used to learn models for classification and prediction of medical conditions. However, high predictive performance of models trained on EHR data using machine learning (ML) does not always translate into clinical gains in the form of better care or lower cost. There is increasing concern that current models are not useful, reliable or fair. It is known that the net benefit of using a model to risk-stratify is inextricably linked to the subsequent actions it triggers or prevents in a care workflow. Therefore, it is necessary to develop methods to evaluate models in a manner that is aware of the context in which they operate.

In the current work, we present a framework to evaluate the impact of factors in healthcare delivery on the net benefit of triggering an Advanced Care Planning (ACP) workflow based on predictions of 12-month mortality. We analyze the effect of workflow deviance on a model’s usefulness, showing significant impact on achieved utility. We find that the benefit of using a predictive model for identifying patients for interventions is highly dependent on the capacity to execute the workflow triggered by the model. Routine analysis of the sensitivity of the net benefit realized by model-triggered clinical workflows to various healthcare delivery factors is necessary.

Target Audience

The target audience for this activity is professionals and students interested in health informatics.

Learning Objectives

The general learning objective for all of the JAMIA Journal Club webinars is that participants will

  • Use a critical appraisal process to assess article validity and to gauge article findings' relevance to practice

After participating in this webinar, the listener should be able to:

  • Describe the role of healthcare delivery factors in realizing the net-benefit of guiding care decisions via predictive models
  • Apply workflow simulation as a way of evaluating models in a context-aware manner
  • Understand the relationship between setting classifier probability thresholds and downstream work incurred
  • Summarize the tradeoffs imposed by capacity-constrained workflows on a model's net-benefit

Commercial Support

No commercial support was received for this activity.

Disclosures for this Activity

Nigam Shah discloses that he is a stockholder in Prealize Health. 

The following presenters, planners, and staff who are in a position to control the content of this activity disclose that they and their life partners have no relevant financial relationships with commercial interests:

JAMIA Journal Club planners: Hannah Burkhardt, Kirk E. Roberts
AMIA Staff: Susanne Arnold, Pesha Rubinstein

Dates and Times: -
Type: Webinar
Course Format(s): On Demand