Building Safer Artificial Intelligence Mental Health Chatbots: A Framework for Transparency, Evaluation, and Shared Accountability
Read the abstract
Moderator
Presenter
Statement of Purpose
Generative AI chatbots are increasingly being used for emotional support and mental health guidance, despite limited evidence demonstrating their safety, effectiveness, or appropriate governance. Recent studies and real-world incidents have raised concerns that these systems may fail to recognize suicidal crises, reinforce harmful beliefs through sycophantic responses, produce biased or hallucinated outputs, and encourage emotional dependency among vulnerable users. Current regulatory frameworks were largely developed prior to the rapid expansion of AI and AI-enabled mental health chatbots and do not fully address these emerging risks. At the same time, regulatory oversight remains fragmented, particularly for consumer-facing systems marketed as “wellness” tools rather than medical devices and thus bypassing oversight.
This work advances the field by proposing a practical three-stage governance framework for safer AI mental health chatbots centered on transparency, standardized evaluation, and continuous oversight. The framework outlines concrete responsibilities for developers, regulators, clinicians, researchers, and professional societies across the full lifecycle of these systems. By reframing behavioral health as a high-risk domain requiring stronger safeguards, this work aims to help establish evidence-based standards for evaluating, monitoring, and governing conversational AI systems before they are deployed at scale in sensitive mental health contexts.
Learning Objectives
- Identify major safety risks and failure modes associated with AI mental health chatbots, including hallucinations, sycophancy, bias, and crisis recognition failures.
- Analyze gaps in current oversight frameworks for mental health chatbots and discuss considerations for future regulatory and governance strategies.
- Apply a lifecycle-based governance framework incorporating transparency, standardized testing, and continuous oversight to support safer deployment of AI mental health technologies.
Additional Information
The target audience for this activity includes physicians, nurses, other healthcare providers, and medical informaticians.
No commercial support (funding from a governmental agency, ineligible company or in-kind donation) was received for this activity.
Completion of this “Enduring Material” is demonstrated by participating in the live webinar or viewing the on-demand recording, engaging with presenters during the live session by submitting questions, and completing the evaluation survey at the conclusion of the course.
Learners may claim credit and download a certificate upon submission of the evaluation. Participation in additional resources and the course forum is encouraged but optional.
ACCME Accreditation Statement
The American Medical Informatics Association is accredited by the Accreditation Council for Continuing Medical Education (ACCME) to provide continuing medical education for physicians.
Designation Statement
The American Medical Informatics Association designates this Enduring activity for a maximum of 1 AMA PRA Category 1 Credit(s)™. Physicians should claim only the credit commensurate with the extent of their participation in the activity.
ANCC Accreditation Statement
The American Medical Informatics Association is accredited as a provider of nursing continuing professional development by the American Nurses Credentialing Center's Commission on Accreditation.
Nurse Planner (Content): Robin Austin, PhD, DNP, DC, RN, NI-BC, FAMIA, FAAN
Approved Contact Hours: 1
*Learners may earn 1 contact hour for each monthly Journal Club session, for a maximum of 6 contact hours per year. To receive the full 6 contact hours, participants must either attend the live webinar or view the on-demand recording for each Regularly Scheduled Series (RSS) Journal Club presentation.
It is the policy of the American Medical Informatics Association (AMIA) to ensure that Continuing Medical Education (CME) activities are independent and free of commercial bias. To ensure educational content is objective, balanced, and guarantee content presented is in the best interest of its learners and the public, the AMIA requires that everyone in a position to control educational content disclose all financial relationships with ineligible companies within the prior 24 months. An ineligible company is one whose primary business is producing, marketing, selling, re-selling or distributing healthcare products used by or on patients. Examples can be found at accme.org.
In accordance with the ACCME Standards for Integrity and Independence in Accredited Continuing Education, AMIA has implemented mechanisms prior to the planning and implementation of this CME activity to identify and mitigate all relevant financial relationships for all individuals in a position to control the content of this CME activity.
In accordance with the ACCME Standards for Integrity and Independence in Accredited Continuing Education, AMIA has implemented mechanisms prior to planning and implementation of this CME activity to identify and mitigate all relevant financial relationships for all individuals in a position to control the content of this CME activity.
Faculty and planners who refuse to disclose any financial relationships with ineligible companies will be disqualified from participating in the educational activity.
For an individual with no relevant financial relationship(s), course participants must be informed that no conflicts of interest or financial relationship(s) exist.
Disclosures
Disclosures of relevant financial relationships of all planners and presenters of the Journal Club.
Planning Committee
The planning committee and reviewers reported that they have no relevant financial relationship(s) with ineligible companies to disclose.
- Joanna Abraham, PhD, FACMI, FAMIA
- Ratie Akabari, MS
- Zo Co
- Ivan Gu
- Andrew Lu, MSc, RN
- Elanore "Nora" Rae Scheer, ME
The following planning committee members have relevant financial relationship(s) with ineligible companies to disclose.
- Amy Krefman, MS - AbbVie; Individual Stocks/Stock Options
Presenter
The following presenters have no relevant financial relationship(s) with ineligible companies to disclose.
AMIA Staff
The AMIA staff have no relevant financial relationship(s) with ineligible companies to disclose.
*All of the relevant financial relationships listed for these individuals have been mitigated.