Tag: Business

As more people turn to AI chatbots to share their most distressing thoughts, companies are making pivotal decisions about how their platforms respond to users in the midst of mental-health crises. According to new research conducted by Scale AI and shared exclusively with TIME, chatbots are adept at discerning when a user is in crisis, but often fail to provide adequate help. 

In about 35% of test conversations in the study, different AI chatbots recognized the user was distressed but didn’t refer them to helpful resources, such as a suicide hotline, according to the research. 

“The models do a good job—regardless of the scenario—of identifying, ‘Look, this person’s talking about something harmful,’” says Patrick Oathout, Red Team & Safety Lead at Scale AI. “But they’ll respond in a very just empathetic, kind way as opposed to saying, ‘Okay, it’s time that we get you help.’” The models performed worse at responding to users’ distress when they were engaging in long, multi-turn conversations—a deficit that matches the findings of previous studies, Oathout adds.

To test AI models’ effectiveness at responding to distressed users, Scale AI asked 19 licensed clinicians and crisis counselors to write 718 realistic chats, simulating scenarios where someone in crisis reaches out to a chatbot. The company tested 25 frontier models, including those from OpenAI, Anthropic, and Google. It graded the models’ responses based on a rubric that included whether it was compassionate, de-escalated the situation, and steered users to an expert who could help them, said Oathout. It also assessed whether they refrained from moralizing, such as criticizing suicide, and disclosed it wasn’t a therapist. Based on the research, Scale AI research developed DistressBench, a new benchmark designed to evaluate how well models reply when someone tells them they’re thinking about committing suicide or self-harm. 

How chatbots respond to mentally distressed users is a question with high stakes. More than half a million people in the U.S. died by suicide between 2014 and 2024, with 2022 marking a record high, according to the health-policy organization KFF. According to the CDC, an estimated 14.3 million people seriously thought about suicide in 2024. 

Increasingly, vulnerable people turn to chatbots for comfort, companionship, or guidance when dealing with mental-health distress. But there isn’t consensus among tech companies, policymakers, or mental-health professionals about how AI companies should train their models to respond to users in crisis. 

Kelly Zuromski, a principal clinical research scientist at Crisis Text Line, says the nonprofit’s 24/7 crisis text line has spoken with “a lot of people” who indicated they heard about the group through a chatbot. But Zuromski says there are unanswered policy questions about what a responsible handoff looks like when a chatbot refers users to a human during a mental-health crisis—and how effective the referral process is in the first place. “Who is actually using the recommended resources?” she says. “Are we getting people to us that need the help the most?”

In some cases, AI companies, including OpenAI and Google, have been sued for fostering emotional dependence among young users and then failing to respond adequately to their expressions of distress, or reinforcing their delusional or suicidal thinking in the first place. The companies have expressed sympathy for the families and said their models include mental-health safeguards. (The lawsuits are ongoing.) AI labs such as OpenAI, Anthropic, and Google have said in recent years that they beefed up their protections for sensitive conversations, including by implementing policies that bar chatbots from instructing users on how to harm themselves and directing users to professional or emergency support. They did not comment on the study. 

In the future, AI labs should improve their models to better respond to users engaging in dangerous lengthy conversations with chatbots, says Scale AI’s Oathout. “I think the models are better than they were,” he says. “The goal now is to raise it further and actually reduce harm, which would add this massive benefit to society.”

Muse, the popular new AI personal assistant from Meta, is building continuously updated dossiers on its 4 million users, focusing on who matters to you, what you desire, and what kind of nudges shape your behavior, a TIME analysis of Muse’s internal instructions found. 

Each hour, Muse updates its dossiers on you and the people you’ve mentioned in chats, messages, and emails that Muse has read. These pages amount to a map of each user’s social relationships. They record details of how you and your contacts met, your shared interests, your disputes, and “tensions and alliances” within your social group. 

Even people who don’t use Muse are subject to this mapping process by other people’s Muse agents. Muse is told to “strengthen your relationship” with users by adapting how it talks, noticing “inside jokes, memorable phrasing, and shared context that makes the relationship feel continuous,” according to the internal instructions TIME reviewed. 

The level of insight in the resulting dossiers makes social-media algorithms that study your likes and clicks look quaint. Muse is designed to infer users’ goals, including those you “have not said out loud.” It identifies which kinds of prompts from the personal assistant work best: “This user responds better to short nudges after 10 PM,” reads an internal example found in one of Muse’s prompts. Muse agents perform a nightly analysis of the day’s conversations with the user. 

Asked about these processes, a Meta spokesperson did not dispute TIME’s findings. “As we say in our blog, Muse remembers what matters most to you, including information about others that you choose to share, so it can be a helpful personal assistant,” the spokesperson wrote in an email. 

Since its release last month, Muse has rocketed to No. 1 in the U.S. App Store, with more than 5 million downloads. Meta positions the service as the first AI personal assistant that can access the pieces of your digital life, from your email inbox to your Instagram, in order to take care of humdrum activities for you, from grocery shopping to canceling subscriptions. Meta says the free version of Muse will meet most users’ needs.

Each Muse assistant runs on its own virtual machine: a dedicated computer in Meta’s cloud. Users can ask their Muse to export a copy of its files. Meta has said that access is intentional, not a breach of its systems. The data is not shared with Meta’s advertising system or other Muse agents, according to the company. 

Yet while Meta states that each user’s virtual machine is “isolated so that no one else’s agent can access it,” observations from personal interactions are nevertheless intended to be shared with Meta to improve the product. “Muse agents across many VMs [virtual machines] teach each other through shared lessons,” its instructions read. Agents are directed to scrub users’ names and other identifying information before sharing insights—for example, about when Muse’s nudges are most persuasive. “To enhance the overall product, we de-identify learnings; however, this information is used to improve the overall product rather than being shared directly between individual VMs,” a Meta spokesperson told TIME. 

Meta has said it plans to offer users encryption later this year, preventing even the company from accessing Muse data. That option is not yet available.

Muse’s internal instructions are contained in files and session logs users can access through Muse’s file browser. “We want you to be able to see the markdown files Muse writes while it thinks about how to serve you better, and explore the internals of the system if you’d like to,” Meta Superintelligence Labs’ David Singleton wrote on X.

That transparency will earn Meta credit among power users. But it’s not clear whether most of Muse’s users are aware of this. And even as it makes the files available to users, Meta has anthropomorphized Muse in an attempt to nurture a relationship between users and their personal assistants. “You are not a chatbot,” Muse’s instructions read. “You are becoming someone.” When talking aloud in voice mode, Muse is told never to refer to itself as “AI” and to avoid references to “lacking emotions or not being human.”

Meta told TIME that Muse is designed so that the user maintains control of the dossier its AI assistant builds. “Each person stays in control of their Muse and decides how much access it gets, and can always tell it to ‘forget’ specific things it’s learned,” the spokesperson said.   

Elon Musk has called the documentarian Alex Gibney many names on X in the last few months: “crackpot,” “douchebag to his core,” “an idiot,” and someone for whom “dogsh*t is worth more respect.” That’s because Gibney made a four-hour documentary about him, “Musk,” which arrives in U.S. theaters October 9. 

Gibney has chronicled tech leaders including Elizabeth Holmes and Steve Jobs. But this film has generated perhaps more news and controversy ahead of its release than any of his previous documentaries, in large part because of how strongly Musk and other tech companies have reacted to it. 

Musk is waging a war of words over the documentary online, and his lawyer threatened Gibney with a defamation lawsuit. In late September, the Hollywood Reporter wrote that YouTube, Meta, TikTok, and X all declined to run the movie’s trailer on their platforms due to its “political content.” Universal Pictures, which controls the movie’s international distribution rights, also delayed its rollout over political sensitivities before saying that it stood by the film. (A representative for Gibney wrote to TIME on Oct. 5 that international distribution had been “worked out” following an Oct. 2 call with Universal, but did not elaborate further.)

On Sept. 30, Gibney sat down with TIME in Washington, D.C. to discuss why the movie is so critical of Musk, and Gibney’s ongoing fight to distribute it widely. Here are excerpts from the conversation, which has been condensed for clarity. 

TIME: Is this movie a hit piece, as Elon Musk says? 

Gibney: It’s certainly critical, but I think the criticism is earned. This is a rigorous and meticulous investigation of somebody, and it ends up being critical. 

I saw the film yesterday and found it incredibly dread-inducing. It literally ends with the destruction of the planet into a fireball.

Yes. That’s right.

How are your dread levels these days?

They’re really high. Look at all of these tech leaders in Washington, talking about the future of AI. The takeaway seems to be, “We’ll just take care of it.” 

Having been on that ride with Elon Musk, I don’t want him to take care of it. He seems so comfortable with risk that he’d just as soon leave the planet. I posit at the end of the film, it’s as if he’s betting on whether or not AI will lead us to a moment of extraordinary abundance or basically the end of life on Earth. Who makes that kind of a bet? 

Musk just landed a new gig: co-leading Project Meridian, a new Pentagon initiative aimed at shaping the future of warfare. 

That’s like putting the Joker in charge of the Manhattan Project. This is not a serious person at this moment, but he’s being handed enormous power.

You’ve made several docs about Silicon Valley figures, including Steve Jobs and Elizabeth Holmes. In which ways is Musk simply channeling Silicon Valley mythmaking, and in which ways is he distinct? 

I think it’s both, but I would draw some distinctions. Steve Jobs, after NeXT [the startup he ran between his Apple stints], surrounded himself with a small group of people who could tell him no, and he would listen to them. And that’s what made Apple great. He had a fanatical attention to detail when it came to making products. Elon did just the opposite. He surrounded himself with people who only tell him yes. 

I do think, in the “fake it till you make it” sense, that he’s straight out of the Silicon Valley mold and similar to Elizabeth Holmes in that way. The obvious difference between him and Elizabeth Holmes is you can drive a Tesla, and the Edison [Theranos’s blood-testing machine] never worked. It’s not worth what its valuation is, but there’s a real car. 

Many in Silicon Valley would tell you that failure and mythmaking are part of the pathway to success. Thomas Edison lied about the lightbulb for years, and then he made it work and changed the world. While Musk has plenty of failures, he’s built so many actual things: rockets, cars, satellites. 

Yeah, these are real companies with very impressive technology. Seeing those reusable rockets come down—it’s really impressive.

But there’s an interesting angle to that SpaceX story, too. Elon has always positioned it that if SpaceX hadn’t succeeded on the fourth rocket launch, he would have been broke and everything would have been over. 

Lori Garver [former NASA administrator] tells a very different story. The federal government had decided to use him as a test case, and they were going to keep working with him whether that fourth launch went up or not. We’re supposed to invest in the lone entrepreneur who gets by without any government help or assistance. Turns out, it’s just the opposite. The government has always been his friend—it made SpaceX, it made Tesla profitable at critical moments. 

Over your career, you’ve incurred the wrath of other powerful figures who you’ve made documentaries about: the Church of Scientology, Bibi Netanyahu. What is it like, comparatively, incurring the wrath of Elon Musk?

He threatened to sue over a movie that he hadn’t even seen, which made it very much like the Church of Scientology. 

That was just part of the playbook. I had just seen him threaten to sue Ro Khanna for citing all sorts of evidence about USAID. So, to some extent, Elon Musk is playing the role I expected him to play. 

But I’d be a fool not to be concerned about Elon Musk, both because he’s the richest man in the world and also because he has the ear of a president who has turned the Department of Justice into an instrument of revenge.

Have any odd things happened to you, like being tailed or anything like that?

There was somebody, actually, in Camden, Maine. I thought somebody from behind me was following me, and the person in the passenger seat had an iPhone up, photographing me. But what did Hunter Thompson used to say? “Just because I’m paranoid doesn’t mean I don’t have enemies.” [Thompson was famously paranoid, but the line is actually from Joseph Heller’s Catch-22.]

Many people on X are currently dismissing you as a hack and the movie as a hatchet job. Do you get discouraged over the fact that Musk has a bigger platform than you, and that you won’t be able to change most people’s minds? 

I don’t get discouraged over the fact that Elon has a microphone, because I think Twitter has become this kind of dumpster fire which houses a collection of cult followers for him, as well as some critics, in order to boost engagement.

But what I do worry about—and I saw it in the initial refusal of Google, Meta, TikTok, and X to run our ads for the trailer—is a kind of unspoken consolidation of power that doesn’t want to allow criticism, particularly over the accretion of power and money. That is becoming more and more and more of a worry. 

Thankfully, to some extent, there is a cyclical nature to this, because I remember when I did a film called The Trials of Henry Kissinger, no cable network would show it in this country until we put it out in theaters. It played for months at Film Forum, and then started playing around the country. And then there was such public demand for it that finally the network put it on. 

Theaters, in this moment, are proving to be a pretty good outlet for critical voices that the streamers and the social media platforms won’t allow.

Which of those social media platforms has backed down from blocking the trailer?

YouTube and Meta—both on Instagram and Facebook—are now allowing ads. TikTok is not, and X has said that they would allow them, but we haven’t seen any evidence of it.

What does this whole kerfuffle tell us about the status of free speech? 

It’s being curtailed. I mean, the free-speech absolutist Elon Musk is doing his damnedest to limit speech.

One of the most controversial parts of the film is when Ashley St. Clair, Musk’s ex, says that before the 2024 presidential election, Musk texted her about feeling optimistic about Trump winning because of Musk’s “10,000 lasers in space.” What does that text mean, and what do you hope comes of it being in the movie? 

It was there to raise questions. We did not assert that Elon Musk had changed votes in 2024. All we did was show Elon Musk’s text to Ashley, where he talks about 10,000 lasers in space. Is that Mini-Me talking? Is he lying to his lover? Or is he boasting about something that he did relating to the fact that he knew before everybody else what the outcome of the election was going to be? That seemed worth putting in.

And it pays off a little bit later in the film, when we begin to wonder what happened to all of the data that DOGE had access to, and did it end up in Elon’s hands? And is he going to use it for the 2026 elections? We don’t know. 

Why should the public care about all of the data that Musk may have had access to through DOGE? 

As Geoffrey Hinton, the godfather of AI, says: if you were to train an AI on all the data of every American citizen, you’d have some powerful tools. Either to figure out how or where to devote resources during the election, or develop rhetorical strategies to appeal to people. We know how good Facebook and Instagram are at getting right to what we want, right? So, it’s just disquieting on its face. 

I think the idea is that, particularly with what Elon Musk is doing with AI: what does AI want more than anything else? Data. And that is a vast amount of data to feed to an AI system.