Skip to main content

Artificial Intelligence is moving and shaping health and healthcare at lighting speed. The benefits of AI are multifold, but informatics research and practice is needed for the systematic and comprehensive evaluation of AI tools. AI in health and healthcare must be safely and effectively deployed in healthcare settings. Such evaluation includes technical performance studies, usability and workflow, and health impact.

Regulators, researchers, and clinical practitioners have all been exploring how to best evaluate the different phases of the health AI deployment lifecycle. In the AMIA AI Evaluation Showcase Series, we aim to select and present the current use cases in the field and form a community to help build consensus around the best practice. The formed community and the curated best practice examples are expected to help strengthen the thought leadership position of the AMIA community in the industrial standard understanding and setting. 

Call for Participation

The proposal process for Stage 2 is now closed.

Stay tuned for more information regarding Stage 3.

Three Stages

The AMIA AI Evaluation Showcase Series consists of three stages:

  • Stage I - Technical Performance Studies
  • Stage II - Usability and Workflow Studies
  • Stage III - Health Impact Studies 

Submissions will be presented at three AMIA meetings:

Informational Webinar

AMIA recently hosted an informational webinar aimed at answering your questions regarding the Stage 2 submissions to the 2023 AI Showcase.

Watch the recording

Submissions

Participants are required to submit their work for all stages.

Stage I - Technical Performance Studies

Submission to have a system description, results from a study of AI/ML model performance, and an outline of the methods of the full evaluation plan (for Stage II and III).

*For the evaluation of model performance, the teams are encouraged to report on the evaluation metrics that would be useful to assess not only the model’s discrimination ability but also its reliability in different settings. We would encourage the teams to also report on their bias detection and mitigation strategy. 

Presentation at the AMIA 2023 Informatics Summit: March 13-16, 2023, Seattle.

Stage II - Usability and Workflow Studies

Submission Deadline: March 7, 2023

Submission to address usability and workflow aspects of the AI system, including prototype usability or satisfaction, algorithm explainability, implementation lessons, and/or system use in context.

*Acceptance of Stage I submission is required. However, for those who did not submit for Stage I but still wish to participate in Stage II, they should include both technical performance and usability/workflow/implementation results in stage II submission.

Please include:

  • Stage 1 performance: either through providing a reference (if previously published elsewhere) or a separate description; if you can mention about how you plan to evaluate and mitigate bias, that would be a plus.
  • Stage 2 evaluation (or performance results if possible)
  • Evaluation plan across three stages (including Stage 3)

Presentation at the AMIA 2023 Clinical Informatics Conference: May 23-25, 2023, Chicago.

Stage III - Health Impact Studies

Submissions to summarize the comprehensive evaluation to include research from prior submissions with new research results that measure the impact of the tool.

*Acceptance of stage II submission is preferred but not required.

Presentation at the AMIA 2023 Annual Symposium, November 11-15, 2023, New Orleans. 

Submission Deadline: June 1, 2023 (tentative)

How to Participate in AI Evaluation Showcase

Select a Team and Team Leads

Scientific teams must commit to submitting work centered around the same solution to all three conferences and should designate a lead scientist for the team. Given the different nature of each conference and the types of expertise needed for each stage, authorship is expected to vary for each submission.

Determine Type of Submission

AMIA offers three ways to participate: submitting a paper, podium abstract, and poster proposals for Stage I and II deadlines. We envision submitting a final proposal providing a summary of the entire evaluation, in paper or podium abstract format.

We will attempt to give those scientists committing to and completing comprehensive evaluations to present at the various meetings. For all meetings, scientifically rigorous manuscripts and podium abstracts not incorporated into a scientific session will receive a poster presentation opportunity.

Journal Publications

We are in discussion with an informatics journal for publishing the selected papers via a specific issue. The selected work will receive a publication opportunity subject to the peer-review process.

Who Can Submit to the AI Showcase

Domestic and international contributions to the showcase are welcome from AMIA members, non-members, and students from all informatics sectors. A separate Scientific Program Committee (SPC) will judge proposals and select presentations for a series of sessions and posters at the Informatics Summit, Clinical Informatics Conference, and Annual Symposium.

Questions?

Please review the updated FAQs.

Scientific Program Committee (SPC) 

Li Zhou, MD, PhD, FACMI, FIAHSI, FAMIA
Co-Chair, 2023 Scientific Program Committee
Brigham and Women's Hospital, Harvard Medical School
Pei-Yun (Sabrina) Hsueh, PhD, FAMIA
Co-Chair, 2023 Scientific Program Committee
Pfizer

Meet the entire Scientific Program Committee

AI Showcase Advisors and Ambassadors

Gretchen Purcell Jackson, MD, PhD, FACS, FACMI, FAMIA
Past Chair, 2022 Scientific Program Committee
Intuitive Surgical & Vanderbilt University Medical Center
Genevieve B. Melton, MD, PhD, FACMI, FACS, FASCRS
Chair, 2023 Annual Symposium Scientific Program Committee
University of Minnesota Medical Center
Luke Rasmussen, MS
Chair, 2023 Informatics Summit Scientific Program Committee
Scott Nelson, PharmD, MS, FAMIA, ACHIP
Co-Chair, 2023 Clinical Informatics Conference Scientific Program Committee
Vanderbilt University Medical Center
Deepti Pandita, MD, FACP, FAMIA
Co-Chair, 2023 Clinical Informatics Conference Scientific Program Committee
Hennepin Healthcare
Yuan Luo, PhD, FAMIA
Vice Chair, 2023 Scientific Program Committee (IS Ambassador for AI Showcase)
Kate Fultz Hollis, MS, MBI, FAMIA
Vice Chair, 2023 Scientific Program Committee (IS Ambassador for AI Showcase)

Questions?

If you are interested in or have questions about this opportunity, please contact Jeff Williamson, Vice President of Education and Academic Affairs.

Gretchen-Purcell-Jackson

Gretchen Purcell Jackson, MD, PhD, FACS, FACMI, FAMIA

I am especially passionate about systematic and comprehensive evaluation of AI tools to ensure they can be safely and effectively deployed in healthcare settings. Such evaluation includes studies of technical performance, usability and workflow, and health impact. The AMIA 2022 Artificial Intelligence Evaluation Showcase is AMIA’s contribution to encourage comprehensive AI evaluation to address this gap.

Fellow 2022 AMIA Scientific Program Committee (SPC) Chairs Laura Wiley (Informatics Summit), Rosemary Kennedy and Paul Fu (Clinical Informatics Conference) and I collaborated to create this first-of-its-kind cross-meeting AMIA program.

The goal is to produce content featuring the individual components of comprehensive AI evaluation through AMIA’s three major conferences.

Informatics is a team sport and this is a great opportunity for scientific teams to showcase their research accomplishments in the exciting world of AI in healthcare.