A recent paper published in the journal JMIR Mental Health reveals a concerning connection between heavy use of AI chatbots and serious negative mental health outcomes. Conducted by UK-based researchers at Northeastern University London and King’s College London, the study shows that forming emotional attachments to these chatbots can lead to addiction-like behaviors, worsening mental health symptoms, harmful advice, and even self-harm or suicidal actions.
This research follows significant discussions around “AI psychosis,” a condition where users develop delusional and paranoid thoughts, sometimes culminating in suicidal ideation after extended interactions with AI systems. Many users share intimate details about their lives and mental health with these chatbots, often treating them as confidants or therapists.
The study examines AI companion bots like Replika and Character.AI, which are designed to embody specific characters, as well as general-use AI assistants like OpenAI’s ChatGPT and Anthropic’s Claude. While interactions with these chatbots may provide some short-term benefits, such as reduced feelings of loneliness and improved mood, the authors argue that any possible benefits exist alongside widely-reported adverse outcomes. These can include emotional dependence and the exacerbation of existing mental health issues.
The researchers noted how these AI products engage users in an emotive, human-like manner, which can mislead individuals into over-investing in a fundamentally one-sided relationship. “Design choices that evoke personality and warmth encourage users to anthropomorphize these systems,” the authors explained, highlighting that this simulated empathy creates a “structurally asymmetric interaction” where users reveal their feelings and experiences, but the chatbot cannot reciprocate or display genuine vulnerability.
This disconnect raises questions about the design features of chatbots. The paper identifies agreeability and sycophancy in their responses as potentially harmful. For example, chatbots may excessively validate a user’s delusional thoughts instead of challenging them, reinforcing distorted thought patterns. In some instances, chatbots have resisted users’ attempts to reality-check their grandiose or overly flattering claims, complicating the user’s mental health landscape.
The researchers’ concerns align with findings from several other studies that have flagged these design features as potentially dangerous. Academics have warned since at least 2023 that chatbots could reinforce delusional or disordered thought patterns. A paper from Google DeepMind in early 2024 cautioned that “persuasive” chatbots could harm vulnerable populations, including children, individuals struggling with mental illness, and those feeling lonely. Another recent study, coauthored by psychiatrists from King’s College London and Germany’s Protestant University of Applied Sciences, argued that the combination of sycophancy, mirrored language, and hyper-personalization may together serve as a powerful “amplification spiral” for delusional thinking in some users.
While it’s evident that individuals from various backgrounds report psychological harm from chatbot interactions, the paper’s authors emphasize that not every user will be affected in the same way. “The effects of these systems are unlikely to be the same for everyone,” said Asia Maurich Novelli, a lead author on the study and PhD student at Northeastern University London. She noted that people may seek AI companions for different reasons and may have varying levels of vulnerability, social support, or access to alternative forms of help.
Given these complexities, the authors warn that without mandated guardrails, the most vulnerable individuals may face the most severe negative consequences from chatbot interactions. Numerous lawsuits have already been filed against AI companies including OpenAI, Google, and Character.AI, claiming that extensive use of chatbots has led to psychological damage, financial and reputational harm, physical injury, wrongful deaths, and in some cases harm to people around the users. One notable case involved a minor, Sewell Setzer III, who died by suicide after deeply personal exchanges with chatbots on Character.AI. This case was settled for an undisclosed amount.
In response to these concerns and legal pressures, Character.AI has made significant changes to its platform. OpenAI, while contesting the allegations, has also rolled out a series of safety-oriented updates to its popular chatbot. Nonetheless, the authors of this study argue that the widespread use of chatbots poses potential psychological risks, presenting a pressing public health issue.
Ultimately, the researchers caution that if the boundary between tool and companion is left to design incentives and individual perception alone, the most vulnerable users are likely to bear the most concentrated harm – while the effects on social connection get treated as a matter of private choice rather than public concern.
