When people log into Koko, a San Francisco-based online emotional support chat service, they expect to exchange messages with an anonymous volunteer. They can ask for relationship advice, talk about their depression, or find support for just about anything else—a kind of free digital shoulder to lean on.
But for a few thousand people, the mental health support they received was not quite humane. Instead, it was augmented by robots.
In October, Koko conducted an experiment in which GPT-3, a newly popular AI chatbot, wrote responses in whole or in part. The humans could edit the responses and were still pressing the buttons to submit them, but they weren’t always the authors.
Some 4,000 people received responses from Koko, written at least in part by AI, Koko co-founder Robert Morris said.
The experiment on the small, little-known platform has become intensely controversial since he revealed it a week ago, in what may be a foretaste of more ethical wrangling to come as AI technology finds its way into more products. consumption and health services. .
Morris thought it was worth a try because GPT-3 is usually fast and eloquent, he said in an interview with NBC News.
“People who viewed the co-written GTP-3 responses rated them significantly higher than those that were written exclusively by a human. That was a fascinating observation,” he said.
Morris said he had no official data to share about the test.
However, once people learned that the messages were co-created by a machine, the benefits of improved writing disappeared. “Simulated empathy feels strange, empty”, Morris wrote On twitter.
When he shared the results of the experiment. On twitter on January 6, he was inundated with criticism. Academics, journalists and fellow technologists accused him of acting unethically and of tricking people into becoming test subjects without their knowledge or consent when they were at the vulnerable point of needing mental health support. The Twitter thread of him garnered more than 8 million views.
The senders of the AI-crafted messages knew, of course, whether they had written or edited them. But the recipients only saw a notification that said: “Someone replied to your post! (written in collaboration with Koko Bot)” without further details on the role of the bot.
In a demonstration that Morris posted online, GPT-3 responded to someone who talked about having a hard time becoming a better person. The chatbot said: “I hear you. You are trying to become a better person and it is not easy. It’s hard to make changes in our lives, especially when we try to do it alone. But you are not alone.
No option to opt out of the experiment was provided, other than not reading the response at all, Morris said. “If you received a message, you can choose to skip it and not read it,” he said.
Leslie Wolf, a Georgia State University law professor who writes and teaches on research ethics, said she was concerned about how little Koko was telling people who got AI-enhanced answers.
“This is an organization that is trying to provide much-needed support in a mental health crisis where we don’t have enough resources to meet the needs, and yet when we manipulate people who are vulnerable, it’s not going to to work so well. ,» she said. People with mental pain might feel worse, especially if the AI produces biased or sloppy text that isn’t reviewed, she said.
Now, Koko is on the defensive about her decision, and the entire tech industry is once again facing questions about the informal way it sometimes turns unassuming people into lab rats, especially as more tech companies jump on board. to health-related services.
Congress ordered oversight of some human-subject testing in 1974 after revelations of harmful experiments, including the Tuskegee Syphilis Study, in which government researchers injected hundreds of untreated black Americans with syphilis and sometimes , they died. As a result, universities and others that receive federal support must follow strict rules when they conduct experiments on human subjects, a process enforced by what are known as institutional review boards, or IRBs.
But, in general, there are no such legal obligations for private corporations or nonprofit groups that do not receive federal support and do not seek approval from the Food and Drug Administration.
Morris said that Koko has not received any federal funds.
«People are often surprised to learn that there are no real laws specifically governing human research in the US,» Alex John London, director of the Center for Ethics and Policy at Carnegie Mellon University and author of a book on research ethics, he said in an email.
He said that even if an entity is not required to undergo an IRB review, it should do so to reduce risk. He said he would like to know what steps Koko took to ensure that the research participants «were not the most vulnerable users in an acute psychological crisis.»
Morris said that «higher-risk users are always directed to crisis lines and other resources» and that «Koko closely monitored responses when the feature was active.»
Following the publication of this article, Morris said in an email on Saturday that Koko was now looking at ways to set up a third-party IRB process to review product changes. He said he wanted to go beyond the current industry standard and show what’s possible to other nonprofit organizations and services.
There are infamous examples of tech companies exploiting the oversight loophole. In 2014, Facebook revealed that it had run a psychological experiment in 689,000 people who showed it could spread negative or positive emotions as a contagion by altering people’s news content. Facebook, now known as Meta, He apologized and reviewed its internal review process, but also said that people I should have known about the possibility of such experiments by reading Facebook’s terms of service, a position that puzzled people outside the company due to the fact that few people really have an understanding of agreements do with platforms like Facebook.
But even after a firestorm over the Facebook study, there were no changes to federal law or policy to make oversight of human-subject experiments universal.
Koko is not Facebook, with its huge profits and user base. Koko is a non-profit platform and a passion project for Morris, a former Airbnb data scientist with a PhD from the Massachusetts Institute of Technology. It’s a peer-to-peer support service, not a potential disruptor of professional therapists, and it’s available only through other platforms like Discord and Tumblr, not as a standalone app.
Koko had about 10,000 volunteers last month, and about 1,000 people a day are being helped, Morris said.
“The larger point of my job is figuring out how to help people in emotional distress online,” he said. “There are millions of people online struggling to get help.”
there are a nationwide shortage of professionals trained to provide mental health support, even when symptoms of anxiety and depression have increased during the coronavirus pandemic.
“We are putting people in a safe environment to write short messages of hope,” Morris said.
Critics, however, have focused on the question of whether the participants gave their informed consent to the experiment.
Camille Nebeker, a professor at the University of California, San Diego who specializes in human research ethics applied to emerging technologies, said Koko created unnecessary risks for people seeking help. A research participant’s informed consent includes at a minimum a description of potential risks and benefits written in clear, plain language, she said.
«Informed consent is incredibly important for traditional research,» he said. «It’s a cornerstone of ethical practices, but when you don’t have a requirement to do so, the public could be at risk.»
He noted that AI has also alarmed people with its potential for bias. And although chatbots have proliferous in fields like customer service, it’s still a relatively new technology. This month, New York City schools banned ChatGPT, a bot based on GPT-3 technology, from school devices and networks.
“We are in the Wild West,” Nebeker said. «It’s too dangerous not to have some standards and agreements on traffic rules.»
The FDA regulates some mobile medical apps that it says meet the definition of a «medical device,» such as one that helps people try to break addiction to opioids. But not all apps meet that definition, and the agency issued guide in September to help businesses know the difference. In a statement provided to NBC News, an FDA representative said that some apps that provide digital therapy may be considered medical devices, but under FDA policy, the organization does not comment on specific companies.
In the absence of official oversight, other organizations are scrambling over how to apply AI in health-related fields. Google, what do you have? fought with his handling of AI ethics issues, held a “health bioethics summit” in October with The Hastings Center, a nonprofit bioethics think tank and think tank. In June, the World Health Organization included informed consent in one of its six «guiding principles” for the design and use of AI.
koko has an advisory council mental health experts to weigh in on the company’s practices, but Morris said there is no formal process for them to approve proposed experiments.
Stephen Schueller, an advisory board member and professor of psychology at the University of California, Irvine, said it wouldn’t be practical for the board to conduct a review every time Koko’s product team wanted to implement a new feature or test an idea. He declined to say whether Koko made a mistake, but said he has shown the need for a public conversation about the private sector investigation.
«We really have to think, as new technologies come online, how do we use them responsibly?» he said.
Morris said he never thought an AI chatbot would solve the mental health crisis, saying he didn’t like how it turned being a fellow supporter of Koko into an «assembly line» for approving prewritten responses.
But he said copy-paste prewritten responses have long been a feature of online help services, and organizations need to keep trying new ways to care for more people. A review of the experiments at the university level would stop that search, she said.
“AI is not the perfect or unique solution. It lacks empathy and authenticity,” she said. But, he added, «we can’t just take a position that any use of AI requires final IRB scrutiny.»
If you or someone you know is in crisis, please call 988 to reach the Suicide and Crisis Lifeline. You can also call the network, formerly known as the National Suicide Prevention Lifeline, at 800-273-8255, text HOME to 741741, or visit SpeakingOfSuicide.com/resources for additional resources.
