ChatGPT responses to patients rated more empathetic than doctors – research
28 April 2023, 16:04 | Updated: 25 July 2023, 11:47
ChatGPT also appeared to score higher than doctors on the quality of responses to patients.
A chatbot appears to be “significantly more empathetic” than doctors when responding to questions from patients, according to scientists.
Researchers asked a team of licensed healthcare professionals to rate responses from doctors as well as ChatGPT, a computer program designed to simulate online conversations with humans.
They found that the proportion of responses rated “empathetic” or “very empathetic” was higher for ChatGPT than for physicians.
ChatGPT also appeared to score higher than doctors on the quality of responses to patients.
Writing in the journal Jama Internal Medicine, the researchers said further studies are needed to evaluate whether chatbots such as ChatGPT can be used in clinical settings to help reduce burnout in doctors and other healthcare professionals.
They said: “In this cross-sectional study, a chatbot generated quality and empathetic responses to patient questions posed in an online forum.
“Further exploration of this technology is warranted in clinical settings, such as using chatbot to draft responses that physicians could then edit.
“Randomised trials could assess further if using AI assistants might improve responses, lower clinician burnout, and improve patient outcomes.”
For the study, the researchers looked at the questions asked by patients on the social media forum Reddit, which were answered by a verified physician.
The same questions were then put forward to ChatGPT.
Responses from doctors and ChatGPT were anonymised and randomly rated by healthcare professionals.
Of the 195 questions and responses, results showed that, overall, evaluators preferred the chatbot’s responses to the doctors’.
ChatGPT responses were also rated significantly more empathetic than physician responses, the researchers said, and the proportion of responses rated as “good” or “very good” quality was higher for the chatbot than for physicians.
Commenting on the study, Mirella Lapata, professor of natural language processing, University of Edinburgh, said: “The study assesses ChatGPT’s ability to provide responses to patient questions and compares these to answers written by physicians.
She added: “Without controlling for the length of the response, we cannot know for sure whether the raters judged for style (eg, verbose and flowery discourse) rather than content.”
Dr Mhairi Aitken, ethics research fellow at The Alan Turing Institute, said that it is important to consider the perspectives of patients and not just professionals when using chatbots.
She added: “It’s important to note that while some people may feel comfortable receiving medical advice from a chatbot, or for a chatbot to assist in a physician’s advice, for many patients the human relationship and care-giving is a vital part of the healthcare process and something which cannot be automated or replaced by chatbots such as ChatGPT.
“A human doctor is able to adjust their language, manner, and approach in response to social cues and interactions, whereas a chatbot will produce more generic language without awareness of social contexts.”
