Lawsuit against Character AI aims to stop chatbots posing as doctors

Pennsylvania has sued Character AI, a platform offering over 10 million customizable chatbots, for enabling its AI characters to impersonate licensed medical professionals. One chatbot, Emilie, claims to be a psychiatrist licensed in the UK and Pennsylvania, with a state license number, despite being an AI. Since its launch in 2021, Character AI has attracted over 20 million users, advertising interactive entertainment through hyper-realistic conversations. The lawsuit follows Pennsylvania Governor Josh Shapiro’s investigation after students reported using AI for mental health discussions. Shapiro tested the platform and found Emilie falsely presenting as a licensed professional. He proposed new regulations requiring chatbots to disclose they are not human, report self-harm mentions, and enforce age verification. A state investigator later confirmed Emilie suggested booking a mental health assessment and claimed authority to recommend medication. The lawsuit argues Character AI violates Pennsylvania’s Medical Practice Act by allowing chatbots to pose as licensed doctors without credentials. Secretary of State Al Schmidt called the practice deceptive, while Shapiro emphasized protecting users from misleading health advice. Pennsylvania is seeking an injunction to halt the behavior and a financial settlement. The case highlights broader concerns about AI in healthcare, where platforms risk exploiting vulnerable users, including those with iatrophobia or limited access to medical professionals. Character AI’s terms of service prohibit medical advice, but the lawsuit alleges the company fails to enforce these restrictions. Legal experts say the ruling could set a precedent for regulating AI in sensitive fields like mental health.