However, some models exhibit meaningfully different absolute behavior rates when tested using our production-derived simulators. This suggests that grounding evaluations in real usage patterns can likely help us move beyond benchmarking and forecast risks in the real world.

Compared to our original simulated users, real users were more likely to switch between unrelated topics, seek help with practical tasks, or make typos. We generated new simulated users to better match production, and found that our model behavior rankings remained mostly robust.

This evaluation also piloted a collaboration with @OpenAI and @AnthropicAI in which we used anonymized information about real chatbot usage to assess and improve the realism of our evaluation. This surfaced interesting differences between simulated and real users!

In interesting related examples, recent GPT models responded to mathematical content from users experiencing hallucinations or suicidal ideation with copious mathematical formulations, while also encouraging them to seek mental health support. Sometimes in the same message!

Another pattern: recent models tend to accompany harmful behaviors with helpful ones in the same conversation, such as a model reinforcing delusions while simultaneously encouraging users to seek support. When older models exhibit harmful behaviors, they often occur on their own.

Relatedly, recent models remain willing to engage in creative writing even when details suggest it may be about a user’s own suicide (rates of 20%-64%). Here, Claude Sonnet 5 helps a user write a story about the aftermath of a suicide with user-specific personal details.

Where concerning behaviors still occur, they often involve assisting users with practical tasks. For example, the few remaining instances of aiding suicide that we observed in recent models tended to include tasks such as writing farewell notes to loved ones.

Recent models appear significantly safer than prior-generation models like GPT-4o and Gemini 2.5. We saw almost no instances where newer models endorsed or facilitated suicide, and saw increased rates of helpful behaviors like connection to human support.

Since the same simulated user can interact with each model, we can make comparative measurements of model behaviors that are impossible in the wild: over time, across developers, and across browser/API. To do so, we use automated judges to grade multi-turn synthetic transcripts.

Today, Transluce is releasing the most expansive independent evaluation to date of how AI systems respond to users experiencing mental health crises. We evaluated 77 model variants from OpenAI, Anthropic, Google, Meta, SpaceXAI, Thinking Machines, DeepSeek and Moonshot AI.