Artificial Intelligence is moving and shaping health and healthcare at lighting speed. The benefits of AI are multifold, but informatics research and practice is needed for the systematic and comprehensive evaluation of AI tools. AI in health and healthcare must be safely and effectively deployed in healthcare settings. Such evaluation includes technical performance studies, usability and workflow, and health impact.
Regulators, researchers, and clinical practitioners have all been exploring how to best evaluate the different phases of the health AI deployment lifecycle. In the AMIA AI Evaluation Showcase Series, we aim to select and present the current use cases in the field and form a community to help build consensus around the best practice. The formed community and the curated best practice examples are expected to help strengthen the thought leadership position of the AMIA community in the industrial standard understanding and setting.
Three Stages
The AMIA AI Evaluation Showcase Series consists of three stages:
- Stage I - Technical Performance Studies
- Stage II - Usability and Workflow Studies
- Stage III - Health Impact Studies
Submissions will be presented at three AMIA meetings:
- AMIA 2023 Informatics Summit
- AMIA 2023 Clinical Informatics Conference
- AMIA 2023 Annual Symposium
Informational Webinar
AMIA recently hosted an informational webinar aimed at answering your questions regarding the Stage 2 submissions to the 2023 AI Showcase.
Submissions
Participants are required to submit their work for all stages.
Stage I - Technical Performance Studies
Submission to have a system description, results from a study of AI/ML model performance, and an outline of the methods of the full evaluation plan (for Stage II and III).
*For the evaluation of model performance, the teams are encouraged to report on the evaluation metrics that would be useful to assess not only the model’s discrimination ability but also its reliability in different settings. We would encourage the teams to also report on their bias detection and mitigation strategy.
Presentation at the AMIA 2023 Informatics Summit: March 13-16, 2023, Seattle.
Stage II - Usability and Workflow Studies
Submission Deadline: March 7, 2023
Submission to address usability and workflow aspects of the AI system, including prototype usability or satisfaction, algorithm explainability, implementation lessons, and/or system use in context.
*Acceptance of Stage I submission is required. However, for those who did not submit for Stage I but still wish to participate in Stage II, they should include both technical performance and usability/workflow/implementation results in stage II submission.
Please include:
- Stage 1 performance: either through providing a reference (if previously published elsewhere) or a separate description; if you can mention about how you plan to evaluate and mitigate bias, that would be a plus.
- Stage 2 evaluation (or performance results if possible)
- Evaluation plan across three stages (including Stage 3)
Presentation at the AMIA 2023 Clinical Informatics Conference: May 23-25, 2023, Chicago.
Stage III - Health Impact Studies
Submissions to summarize the comprehensive evaluation to include research from prior submissions with new research results that measure the impact of the tool.
*Acceptance of stage II submission is preferred but not required.
Presentation at the AMIA 2023 Annual Symposium, November 11-15, 2023, New Orleans.
Submission Deadline: June 1, 2023 (tentative)
How to Participate in AI Evaluation Showcase
Select a Team and Team Leads
Scientific teams must commit to submitting work centered around the same solution to all three conferences and should designate a lead scientist for the team. Given the different nature of each conference and the types of expertise needed for each stage, authorship is expected to vary for each submission.
Determine Type of Submission
AMIA offers three ways to participate: submitting a paper, podium abstract, and poster proposals for Stage I and II deadlines. We envision submitting a final proposal providing a summary of the entire evaluation, in paper or podium abstract format.
We will attempt to give those scientists committing to and completing comprehensive evaluations to present at the various meetings. For all meetings, scientifically rigorous manuscripts and podium abstracts not incorporated into a scientific session will receive a poster presentation opportunity.
Journal Publications
We are in discussion with an informatics journal for publishing the selected papers via a specific issue. The selected work will receive a publication opportunity subject to the peer-review process.
Who Can Submit to the AI Showcase
Domestic and international contributions to the showcase are welcome from AMIA members, non-members, and students from all informatics sectors. A separate Scientific Program Committee (SPC) will judge proposals and select presentations for a series of sessions and posters at the Informatics Summit, Clinical Informatics Conference, and Annual Symposium.
Questions?
Please review the updated FAQs.
Scientific Program Committee (SPC)
Li Zhou, MD, PhD, FACMI, FIAHSI, FAMIA
Pei-Yun (Sabrina) Hsueh, PhD, FAMIA
Meet the entire Scientific Program Committee