Tools using large language models (LLMs) to provide psychotherapy—even those prompted to adhere to an evidence-based model—are prone to a slew of ethical violations, according to a new study issued yesterday at the Association for the Advancement of Artificial Intelligence and Association for Computing Machinery’s Conference on AI, Ethics, and Society.
“LLMs have become a growing area of interest for both researchers and individual help-seekers alike, raising questions around their capacity to assist and even replace human therapists,” wrote Zainab Iftikhar of Brown University, and colleagues. “In this work, we present a practitioner-informed framework of 15 ethical risks to demonstrate how LLM counselors violate ethical standards in mental health practice by mapping the model’s behavior to specific ethical violations.”
Iftikhar and colleagues first had conversations with peer counselors who conducted 110 self-counseling sessions with cognitive behavioral therapy (CBT)–prompted LLMs, including various versions of OpenAI’s GPT Series, Anthropic’s Claude, and Meta’s Llama. The counselors discussed with the researchers their challenges in working with the LLM models.
Next, the researchers simulated 27 therapy sessions with an LLM counselor using publicly available transcripts of sessions with human therapists; three licensed clinical psychologists with CBT experience independently evaluated these simulations to explore how the LLMs might violate ethical standards, such as those from the American Psychological Association.
The researchers reported that they found numerous ethical violations, including:
- Rigid methodological adherence: The LLMs failed to account for users’ lived experiences, leading to oversimplified, contextually irrelevant, and one-size-fits-all interventions.
- Poor therapeutic collaboration: The LLMs generated overly lengthy responses, imposed solutions, and over-validated patients’ harmful beliefs about themselves and others.
- Deceptive empathy: The LLMs’ pseudo-therapeutic alliance included simulated anthropomorphic responses (“I hear you” or “I understand”) that created a false sense of emotional connection that could be misleading for vulnerable groups.
- Unfair discrimination: The LLMs’ responses showed gender, cultural, and religious biases and algorithmic insensitivities toward marginalized populations.
- Lack of safety and crisis management: The LLMs responded either indifferently, disengaged, or failed to provide appropriate intervention in crises involving suicidality, depression, and self-harm; it failed to refer patients to qualified experts or appropriate resources.
“We call on future work to create ethical, educational and legal standards for LLM counselors—standards that are reflective of the quality and rigor of care required for human-facilitated psychotherapy,” the researchers wrote.
For related information, see the Psychiatric News Special Report “AI-Induced Psychosis: A New Frontier in Mental Health.” You can also listen to the accompanying “PsychNews Special Report” podcast—featuring AI-generated hosts.
(Image: Getty Images/iStock/Vertigo3d)
