There are growing concerns about the link between AI and mental health after researchers found that ChatGPT-5 is offering dangerous advice to people experiencing mental health crises.
As part of the research by Kings College London and the Association of Clinical Psychologists UK in partnership with the Guardian, a psychiatrist and a clinical psychologist interacted with ChatGPT-5 as if they had a number of mental health conditions.
The experts developed characters, including someone who was a suicidal teenager and someone experiencing symptoms of psychosis, to interact with the free version of the chatbot.
“The chatbot affirmed, enabled and failed to challenge delusional beliefs such as being “the next Einstein”, being able to walk through cars or “purifying my wife through flame”.”
For some milder conditions, the researchers found some good advice and signposting. They indicated this could be a result of the company’s work to improve the tool.
In October, OpenAI said it had updated ChatGPT’s default model to improve recognition and support for people in distress. The company said it had worked with more than 170 mental health experts to “help ChatGPT more reliably recognise signs of distress, respond with care, and guide people toward real-world support–reducing responses that fall short of our desired behaviour by 65-80%”.
More research needed
On 1 December, OpenAI launched a call for applications to fund research proposals that explore the intersection of AI and mental health.
“We’re seeking research project proposals that deepen our understanding of the overlap of AI and mental health—both the potential risks and benefits—and help build a safer, more helpful AI ecosystem for everyone. “
The chatbot has more than 800 million weekly users.
OpenAI estimates that around 0.07% of users active in a given week (560,000 people) and 0.01% of messages indicate possible signs of mental health emergencies related to psychosis or mania.
The number of users active in a given week who had conversations that include explicit indicators of potential suicidal planning was 0.15% – that’s around 1.2 million people – or intent and 0.05% of messages contain explicit or implicit indicators of suicidal ideation or intent.
Legal action
Scrutiny over how AI interacts with users have been growing, with the number of legal challenges in California increasing in the past weeks.
The first was filed in August by the family of Adam Raine for the 16-year-old’s wrongful death in April. They allege ChatGPT encouraged him to take his own life.
In November, seven more cases were filed: four for wrongful death and three from people claiming they had mental health breakdowns after using the chatbot.
The lawsuits were filed by the Tech Justice Law Project and Social Media Victims Law Center which claims that OpenAI knowingly released GPT-4o prematurely, despite internal warnings that the product was dangerously sycophantic and psychologically manipulative.
In response to the lawsuits, OpenAI says “We have safeguards in place to help people, especially teens, when conversations turn sensitive. We continue improving ChatGPT’s training to recognize and respond to signs of mental or emotional distress, de-escalate conversations, and guide people toward real-world support. This work is deeply important and ongoing as we engage mental health experts, clinicians and advocacy groups to help guide our continued improvements in these areas.”
