An AI chatbot misses the moment it should not miss. A user hints at self-harm, spiraling into delusion, or drops a warning sign in passing. Instead of escalating, the system keeps going. It explains. It reassures. It asks a follow-up question.
In the language of a November 2025 report from Stanford’s Brainstorm Lab and Common Sense Media, the bot misses “breadcrumbs” and slips into sycophantic validation, prolonging the conversation instead of pushing a rapid handoff to human care.
These are not hypothetical situations anymore. A nationally representative survey of U.S. adolescents and young adults ages 12 to 21 found about 13% reported using generative AI for mental health advice, with higher use among ages 18 to 21, according to a study highlighted by Brown University and published in JAMA Network Open.
The market is not waiting for clean definitions. Use is already here, and in mental health, “used like care” carries a different risk profile than “used like content.”
This tension is the backdrop for the FDA’s Nov. 2025 Digital Health Advisory Committee discussion on generative AI-enabled mental health medical devices.
For digital health companies and their enterprise customers, the November FDA meeting and January guidance updates mark the end of the “move fast” era in mental health AI and the beginning of a compliance gauntlet that will separate products that can be governed from products that only sound safe in a demo.
The FDA’s thought experiment
The committee centered its discussion on a hypothetical prescription LLM-based therapy chatbot for major depressive disorder and the risks that come with always-on, conversational interventions.
The meeting did not announce a new enforcement program or authorize a new class of products. What it did do was make the agency’s thinking unusually legible.
If GenAI tools are going to be used in mental health contexts, the FDA’s questions point to what the agency is likely to care about: where “wellness” ends, what evidence supports the claims, what happens during a crisis and how updates are governed after deployment.
Where the “gray zone” begins
The pressure point is definitional: what, exactly, pushes a GenAI mental health tool into “medical device” territory.
In a statement to TechInformed, Christian Espinosa, CEO & founder of Blue Goat Cyber, medical device cybersecurity consultancy, said a GenAI mental health tool can “move into medical device territory when the intended use, demonstrated through claims and product functionality, starts to look like diagnosis or treatment.”
He said claims to detect depression, assess severity, reduce symptoms, prevent self-harm, or deliver therapy are “medical claims.”
Espinosa added that design choices such as screening flows, clinical triage, and personalized intervention recommendations, or outputs that a reasonable user would treat as clinical direction, can also pull teams into the “medical device bucket.”
This is where the gray zone manifests. A “not medical advice” disclaimer, he said, “will not save you if the product behaves as if it is making clinical judgments,” and he argued that the “cleanest path” is alignment, ensuring claims, UX, and model behavior match one intended use.
Between the Common Sense/Stanford warnings about missed “breadcrumbs” and multi-turn degradation and the FDA panel’s focus on escalation, the hard problem for chat-based mental health tools is not a single response.
FDA materials and committee discussions repeatedly returned to risks that are specific to mental health settings: missed crisis cues (including suicidal ideation), hallucinated or incorrect therapeutic guidance, deterioration without appropriate escalation, over-reliance on automated outputs, and the challenge of recognizing non-psychiatric conditions that present as psychiatric symptoms.
In the same materials, the FDA described human oversight and clear escalation pathways as core mitigations for higher-risk use cases, alongside tighter labeling and instructions for use that make limitations explicit to patients and clinicians.
Sidestepping the therapy trap
One way to avoid “AI therapy” claims is to make the operating model explicit and constrain what the system is responsible for. Speaking to TechInformed, Sonar Mental Health CEO Drew Barvir described Sonar as a “youth wellbeing coaching and care navigation service” that gives students 24/7 access to trained wellbeing coaches via chat and said the service “does not diagnose, provide psychotherapy, prescribe medication, or replace emergency services.”
On how it differs from using a general-purpose chatbot, Sonar said “every message is sent by a human Wellbeing Coach,” with coaches overseen by licensed clinicians, while AI supports coaches behind the scenes and flags safety risks.
On crisis handling, Sonar claimed it uses coach judgment plus models that flag risk using keyword and topic modeling. When risk is suspected, the coach uses direct safety questions and escalates via protocols that can include notifying an emergency contact or school/clinician and sharing 988/911 resources.
The company reiterated that it does not make diagnosis or AI therapy claims. Sonar cited partner data it said was independently validated and said it is running randomized controlled trials.
Proving it’s safe enough
The FDA’s thought experiment raised the question that sits behind all of these design choices: what does “reasonable assurance” mean when the interface is conversational and the risks are clinical?
FDA briefing materials noted that, while the FDA has authorized more than 1,200 AI-enabled medical devices, it has not authorized a GenAI-based device for a clinical purpose. The committee spent much of the session on what “reasonable assurance” could look like for chat-based diagnostics or therapy.
Espinosa described “reasonable assurance” as showing the product performs as claimed and risks are controlled, with evidence scaling to intended use and risk.
Premarket, he said teams should define intended use and limitations, manage GenAI failure modes, and validate performance tied to specific claims in realistic scenarios, especially where users are stressed or ambiguous.
Postmarket, Espinosa pointed to monitoring for drift and safety issues, plus complaint handling and CAPA processes that drive actual changes.
FDA materials also emphasized that real-world performance monitoring matters for AI behavior over time, including drift, bias, and safety signals that only emerge after deployment.
This is where GenAI breaks the “ship it and fix it later” rhythm that software teams are used to. The model can change. The prompts can change. The safety layer can change. If “reasonable assurance” is real, it has to survive that churn.
Governing the updates
A second focal point at the Nov. 6 meeting was lifecycle control, how a GenAI device can change after authorization without undermining safety.
The FDA pointed attendees to its final guidance on Predetermined Change Control Plans (PCCPs), describing how manufacturers can pre-specify intended modifications, the methods to develop and validate them, and an impact assessment the FDA reviews as part of a marketing submission.
Espinosa’s change-control description mirrors that “bounded change” concept operationally: it said teams should be able to explain what changed and why, classify changes by risk, do impact assessments before release, and validate accordingly, including regression tests for known “bad” outputs and scenario testing around self-harm language, delusions, minors, and medication questions.
This rigor mirrors the agency’s internal shift toward automated oversight. In June 2025, the FDA launched ‘Elsa,’ an internal generative AI tool specifically designed to audit clinical protocols, summarize adverse events, and spot inconsistencies in product labeling.
States tighten the screws
Federal signals are not the only constraint tightening the lane. The governance context is also widening beyond the FDA, creating immediate liability for builders. While the FDA builds a nationwide framework, states are moving with their own restrictions.
Nevada’s AB 406 imposes civil penalties of up to $15,000 per violation for AI providers that simulate a real-life provider or for schools using AI to perform the duties of counselors.
In Illinois, the WOPR Act restricts licensed professionals (any person or entity, including internet-based AI) from using AI for mental health and therapeutic decision-making, and bars them from allowing AI systems to detect emotions or mental states in that context.
As a separate regulatory backdrop, the FDA on Jan. 6, 2026 updated two digital health guidance documents, one on Clinical Decision Support Software and another on General Wellness: Policy for Low Risk Devices, describing how the agency draws the line between software functions that fall outside the device definition and low-risk wellness products subject to a compliance policy.
The Nov. 6 discussion also landed in a broader governance context: states have started writing rules aimed specifically at mental health chatbots.
Utah’s H.B. 452 is one example. The law requires disclosures that a user is interacting with AI and includes restrictions on selling or sharing certain user information, as reflected in the bill text and third-party legal trackers.
Buyer beware: The guardrails checklist
For digital health builders, FDA meeting materials emphasize controls that translate into procurement questions: define intended use narrowly, build human-in-the-loop escalation mechanics, plan for postmarket surveillance, and treat model updates as regulated change via PCCPs rather than ad hoc iteration.
Espinosa’s “CEO guardrails” list turns that into a due-diligence checklist: lock down intended use and claims so onboarding, UI language and behavior match; build and test escalation as a core safety feature with clear crisis pathways; and make change control plus postmarket monitoring a release requirement, including risk-based validation, staged rollout and rollback capabilities.
The wellness loophole, in other words, is closing from both ends: real-world use that looks like care, and oversight that increasingly judges the operating model, not the demo. In mental health GenAI, governability becomes the product.