A Framework for Making Predictive Models Useful in Practice
This on-demand webinar does not offer CE credit.
Jung K, Kashyap S, Avati A, et al. A framework for making predictive models useful in practice [published online ahead of print, 2020 Dec 22]. J Am Med Inform Assoc. 2020; ocaa318. doi:10.1093/jamia/ocaa318.
Watch the Recording
Presenter
Nigam Shah, MBBS, PhD
Manager and Moderator
Hannah Burkhardt
Statement of Purpose
The widespread adoption of electronic health records (EHRs) has turned the record of routine clinical practice into data that can be used to learn models for classification and prediction of medical conditions. However, high predictive performance of models trained on EHR data using machine learning (ML) does not always translate into clinical gains in the form of better care or lower cost. There is increasing concern that current models are not useful, reliable or fair. It is known that the net benefit of using a model to risk-stratify is inextricably linked to the subsequent actions it triggers or prevents in a care workflow. Therefore, it is necessary to develop methods to evaluate models in a manner that is aware of the context in which they operate.
In the current work, we present a framework to evaluate the impact of factors in healthcare delivery on the net benefit of triggering an Advanced Care Planning (ACP) workflow based on predictions of 12-month mortality. We analyze the effect of workflow deviance on a model’s usefulness, showing significant impact on achieved utility. We find that the benefit of using a predictive model for identifying patients for interventions is highly dependent on the capacity to execute the workflow triggered by the model. Routine analysis of the sensitivity of the net benefit realized by model-triggered clinical workflows to various healthcare delivery factors is necessary.
Target Audience
The target audience for this activity is professionals and students interested in health informatics.
Learning Objectives
The general learning objective for all of the JAMIA Journal Club webinars is that participants will
- Use a critical appraisal process to assess article validity and to gauge article findings' relevance to practice
After participating in this webinar, the listener should be able to:
- Describe the role of healthcare delivery factors in realizing the net-benefit of guiding care decisions via predictive models
- Apply workflow simulation as a way of evaluating models in a context-aware manner
- Understand the relationship between setting classifier probability thresholds and downstream work incurred
- Summarize the tradeoffs imposed by capacity-constrained workflows on a model's net-benefit
Commercial Support
No commercial support was received for this activity.
Disclosures for this Activity
Nigam Shah discloses that he is a stockholder in Prealize Health.
The following presenters, planners, and staff who are in a position to control the content of this activity disclose that they and their life partners have no relevant financial relationships with commercial interests:
JAMIA Journal Club planners: Hannah Burkhardt, Kirk E. Roberts
AMIA Staff: Susanne Arnold, Pesha Rubinstein