Insider
How do you make money off people caught in a mental-health emergency?
Loris, a startup that helps companies improve their customer-service conversations, found a way. Founded in 2018, Loris used data generated from text conversations with people in distress to make “empathetic” customer-service software. The source for the data? Crisis Text Line, a nonprofit suicide-prevention hotline and the parent company of Loris.
Crisis Text Line, now in its 10th year of operations, uses artificial intelligence to respond to people experiencing emotional abuse, self-harm, and suicidal thoughts. A Politico investigation last year explained how the help line created Loris to develop and sell AI software that could guide customer-service agents through live chats with customers using proven “de-escalation techniques, emotional-intelligence strategies, and training experience.”
At the heart of the arrangement was what Crisis Text Line called the “largest mental health data set in the world” — 219 million messages from more than 6.7 million conversations over text, Facebook Messenger, and WhatsApp.
Exploiting data for profit is par for the course in the modern technology business — search engines, social-media platforms, and streaming apps all gather and monetize the data they gain from users. But commercializing the data from a crisis line is vastly different from mining data on the binge-watching or online-shopping habits of customers. The data used for Loris came from people who were at the lowest point of their lives, when they may not have been able to truly understand and consent to how it would then be used.
After the Politico story sparked outrage, the companies claimed that they used only anonymized data — though research has found that anonymization is not fool-proof. Eventually, the furor became so great that Crisis Text Line severed all ties with Loris and told the company to delete any previously transferred information. But the Loris mess was just one of a growing number of mental-health-data fiascos.
Despite being touted as the fix for the broken healthcare system, the mushrooming tech-based mental-health industry has a dark side. In the past year, a flurry of reports have found that some of the most recognizable names in the industry have repeatedly engaged in creepy and harmful data-sharing practices that treat people in need of help as prospective sources of profit instead of as patients. Taken together, the reports reveal a dangerous cocktail of tech solutionism, abuse of consumer trust, and regulatory failure that puts highly vulnerable people at risk. And they raise important questions about the future of mental-health care and the role of technology in it.
The case for online mental health
Depression is a leading cause of disability worldwide. One person dies by suicide every 40 seconds. Globally, an estimated 12 billion working days and $1 trillion are lost every year because of depression and anxiety. It’s a massive challenge that’s brought conventional mental-health care to its knees. Mental health gets less than 2% of national healthcare budgets on average, and there’s a dire shortage of trained therapists and psychiatrists. Only one in five people in high-income countries and one in 27 people in low- and lower-middle-income countries get minimally adequate treatment for depression. Add to this mix the unprecedented spike in demand for mental-health care caused by the pandemic, and the situation becomes immeasurably worse.
Enter: Silicon Valley-esque upstarts that believe tech can solve the world’s most complex problems. Venture-capital investments in mental-health startups grew from $2.3 billion in 2020 to a record $5.5 billion in 2021. Even as funding cooled in 2022 amid a broader tech-industry slowdown, mental health remained the highest-funded area in digital health.
Two types of companies have emerged in the frothy mental-health-startup scene. There are chatbot-based apps such as Woebot, which offer DIY “clinically tested tools and tactics” grounded in the principles of cognitive-behavioral therapy to help users deal with everyday stress and emotional problems. The main attraction of these apps is the promise of immediate, 24/7 support.
[…]
‘The vast majority of mental-health apps are exceptionally creepy’
BetterHelp, a poster child of online therapy founded in 2013, calls itself “the world’s largest therapy platform” and says it has over 2 million users. BetterHelp is also a poster child for the industry in another way: Last month, the US Federal Trade Commission reprimanded the company for “betraying consumers’ most personal health information for profit.”
The FTC said BetterHelp handed over sensitive user data — including emails, IP addresses, and replies to health questionnaires — to Facebook so the platform could use this information to run advertisements for BetterHelp that targeted similar users. The commission added that BetterHelp did this despite promising users their personal data would not be used or disclosed except for limited purposes, such as to provide counseling services. It also said that BetterHelp’s Facebook campaign helped it acquire tens of thousands of new paying users and millions of dollars in revenue. The FTC banned BetterHelp from sharing users’ private health data with third parties and ordered it to pay a $7.8 million fine that would go toward partial refunds to customers — the first time the commission has penalized a mental-health company for mishandling users’ private data. BetterHelp decided to settle, but it did not accept any wrongdoing and defended its methods as “industry-standard.”
[…]
BetterHelp’s claim that its data-sharing practices were “industry-standard” is more of an indictment of the industry than exculpatory for the company. Days after the FTC order, news broke that Cerebral, another splashy online mental-health company, had admitted to sharing 3.1 million American patients’ private health information, including mental-health assessments, with advertisers and social-media platforms. (The company says it has since removed the data-tracking code from its apps.)
The mental-health startups Talkspace and BetterHelp, as well as dozens of others, were also named in a damning 2022 report by the Mozilla Foundation, a nonprofit organization that describes its mission as working to keep the internet healthy. The report highlighted the unscrupulous security and privacy practices of mental-health apps, concluding that the vast majority of mental-health apps tracked, shared, and commercialized their users’ most intimate and vulnerable thoughts and feelings. The Mozilla Foundation characterized these practices as “extremely creepy.”
[…]
Earlier this year, a study by Joanne Kim at Duke University’s Technology Policy Lab exposed the frightening extent to which mental-health data had been commoditized. While it isn’t clear where brokers get their mental-health data because of a lack of regulation, the study found that these brokers traded the data in the “open market, with seemingly minimal vetting of customers and seemingly few controls on the use of purchased data.”
Some data brokerage firms charged upward of $75,000 or $100,000 a year for access to data that they claimed included information on individuals’ mental-health conditions. One broker charged $275 for 5,000 aggregated counts of Americans’ mental-health records, the study found. Another “advertised highly sensitive mental health data to the author, including names and postal addresses of individuals with depression, bipolar disorder, anxiety issues, panic disorder, cancer, PTSD, OCD, and personality disorder, as well as individuals who have had strokes and data on those people’s races and ethnicities,” the report added.
There are enough health-focused startups that have failed to keep private health data safe that, at this point, it’s clear the problem is industrywide.
[…]
Legal gray areas
Following the Mozilla report in the summer, a group of US senators sent a letter to Talkspace and BetterHelp asking them to explain their data policies. The senators expressed concerns about how these companies shared confidential user data with “third-party Big Tech firms and data brokers, who have shown remarkably little interest in protecting vulnerable consumers and users.”
But the increased congressional scrutiny may not amount to much. While these activities sound illegal, they may not be. The Health Insurance Portability and Accountability Act, created in the 1990s, extends only to “covered entities” in the US, such as your doctor office or hospital, and their “business associates.” Digital-health tools occupy a gray area under HIPAA, a flaw that can be — and is — exploited by all manner of businesses, from virtual-therapy platforms to period trackers.
“As I understand it, HIPAA does not apply to direct-to-consumer healthcare products, which would include the vast majority of mental-health apps,” Piers Gooding, a researcher at the Melbourne Law School and associate editor of the International Journal of Mental Health and Capacity Law, told me. “The FDA and FTC may play roles in evaluating these direct-to-consumer technologies and their claims,” Gooding added. “But there remain gaps. For example, the FTC doesn’t seem to cover data gathered by nonprofit organizations, which was a concern raised in the Crisis Text Line case.”
When questioned about data violations, these companies often hide behind the excuse that they had secured their users’ “legal consent” by requiring they agree to the platforms’ terms and conditions when they signed up. In the real world, however, legal consent rarely translates to meaningful consent.
“Nearly all consent forms we come across on the internet are dense texts filled with inaccessible legal jargon,” Deepa Singh, an AI-ethics researcher at the University of Delhi, wrote after the Crisis Text Line controversy. On top of that, giving consent is usually a one-time event in the life cycle of a user using a particular service. Meaningful consent is more nuanced. It requires a clear understanding of what is being asked of the user, can change over time, and “does not hide behind obfuscation,” Singh argued.
[…]
The Most Revolutionary Act 