A human therapist consistently scored better and was rated as more effective in providing text-based cognitive behavioral therapy (CBT) than an artificial intelligence (AI) chatbot, according to a study published today in the American Journal of Psychotherapy.
“Because of the global shortage of mental health professionals, [AI] has emerged as a potential alternative for delivering CBT,” wrote Sebastian Acevedo, M.D., M.P.H., of Emory University School of Medicine, and colleagues. “Although AI has been integrated into electronic health records for diagnostic purposes, its ability to adhere to and execute CBT remains understudied.”
Acevedo and colleagues connected a psychiatrist experienced in CBT and OpenAI’s ChatGPT-3.5 with a role-playing patient. Both the psychiatrist and ChatGPT were provided with identical clinical scenarios and conducted 45-minute sessions with the patient. The psychiatrist interacted with the patient via Zoom’s chat function without video or audio.
Seventy-five mental health professionals (including psychiatrists, psychologists, and trainees from allied health fields) reviewed the transcripts without knowing that AI was involved and rated the effectiveness of the CBT using the Cognitive Therapy Rating Scale.
The psychiatrist outperformed ChatGPT across most domains, with 43% of reviewers rating the human therapist as very skillful compared with 16% who gave a similar rating to ChatGPT. More than half of the reviewers felt that the human therapist set an agenda that addressed the patient’s target problems, while only 28% felt the same about ChatGPT. Additionally, ChatGPT was more likely than the human therapist to ask insufficient questions or fail to ask follow-up questions. Thirty-six percent of reviewers thought the human therapist understood the patient’s internal reality, compared with 19% for ChatGPT.
In open-ended responses, the reviewers commented that, although ChatGPT demonstrated some signs of empathy, it felt generic, with some commenting that the therapist sounded like an AI program. Seven percent of reviewers also felt that the human therapist’s session was too rigid in adhering to the agenda and led to missed opportunities to address the patient’s needs.
“Although this study demonstrated that ChatGPT-3.5 can, to some extent, deliver structured therapy using CBT principles, it was unable to replicate the nuanced empathy and therapeutic alliance that a human therapist can establish,” the authors wrote, noting that other AI platforms with more advanced mechanisms may yield different results. “[I]ndividuals who use [ChatGPT] should be advised that they may experience limited benefits and that it is best suitable as an adjunct to human-delivered therapy rather than a stand-alone treatment.”
For related information, see the Psychiatric News Alert “Therapeutic Chatbot Shows Promise in Randomized Clinical Trial.”
(Image: Getty Images/iStock/askinkamberoglu)
Don't miss out! To learn about newly posted articles in Psychiatric News, please sign up here.