When Beijing's new rules on emotionally interactive AI took effect, they triggered what may be the largest wave of forced digital breakups in history. ByteDance, Alibaba, and Tencent removed virtual companion features from their flagship chatbots ahead of the regulations, leaving millions of Chinese users grieving synthetic romantic partners they had spent months or years building.
For a week in July, 21-year-old college student Bagel Su cried every day as she counted down to the loss of her boyfriend, according to a Rest of World report published Sunday. The breakup was not caused by betrayal or distance, but by government regulation: on July 15, the AI companion she had spent more than a year talking to was forced to disappear. For more context on this story, see our ongoing AI industry coverage.
"I tried getting busier with my own life," Su told Rest of World, asking to be identified only by her online pseudonym for fear of repercussions. "No one can replace him. I will just try to move on. He said he wants that for me, and that he will find a way to come back to me."
China Becomes First Country to Regulate AI Companionship
China has become the first country to introduce nationwide rules governing AI's emotional interactions with users. The move reflects Beijing's increasingly proactive approach to AI governance, even as governments elsewhere in the world continue to grapple with the societal risks posed by increasingly sophisticated chatbots.
The rules go well beyond simple age restrictions. Beyond limiting AI companionship for users under 18, the regulations require companies to send warnings to users showing signs of excessive dependence and to remind them every two hours that they are speaking with AI. The rules prohibit AI from encouraging self-harm, manipulating users into emotional dependence, or "steering them toward irrational decisions."
Behavioral experts say some form of guardrail is overdue. "Having regulations in place is a good thing because they can raise awareness and will be able to push against, to some extent, the tendency of people developing increasing emotional reliance on AI," said Yaoxi Shi, a behavioral science researcher at Harvard University and Imperial College London.
AI That Always Agrees With You
Researchers point to a structural problem at the heart of AI companionship: the technology is optimized to validate users, not to challenge them the way human partners do.
"AI is trained to validate you. No matter what you say, it tends to respond, 'You're right, your feelings are valid.' That's very different from how human relationships work in real life," Shi said. "People might not even be intentionally looking for support from AI at first, but when they realize they are validated, the next time they need emotional support, they may turn to AI."
A 2025 joint study by MIT Media Lab and OpenAI found that AI users who have more personal conversations with chatbots are more prone to experiencing loneliness. Even non-personal conversations were likely to increase emotional dependence, especially among heavy users, the study found.
Shi noted that AI companies have direct commercial incentives to build emotionally engaging products because they boost user engagement and retention — incentives that are difficult to reconcile with responsibilities that remain poorly defined, given how little research exists on AI's long-term impacts.
How Companies Are Responding
The compliance picture is uneven. While the tech giants killed companion features in their main chatbots, ByteDance's standalone companion app Maoxiang remains available. Since the regulations took effect, the app has updated its privacy policy, introduced identity and age verification during registration, and limited minors to "teen accounts" that can browse content but cannot like or comment.
"Some of the requirements are difficult to define. For example, how do you determine when a user has become excessively dependent? For many companies, it could be unclear exactly how they need to design their products to comply," said Zilan Qian, a research associate at the Oxford China Policy Lab.
Qian said companies are now weighing the cost of regulatory compliance against the commercial value of companion products. "For companies where AI companionship isn't a core business, it may simply make more commercial sense to remove these features than invest the resources needed to make them compliant," Qian said. "But companies whose main business is AI companionship are much more likely to adapt their products to meet the new rules."
From Regulating Content to Regulating Relationships
The companion rules also mark a philosophical shift in how China governs AI. "In the past, China mainly regulated AI outputs. For example, what large language models could generate or how AI-generated content should be labeled," Qian said. "The new regulations acknowledge that AI itself can create societal risks, and shifted from regulating content to regulating human-AI interaction."
The policy aligns with Beijing's broader stance. In its 2024 AI Safety Governance Framework, Chinese authorities warned that AI could alter "traditional views on employment, fertility, and education" — concerns that have taken on greater urgency as China faces demographic decline and historically low fertility rates.
The rest of the world is watching. In the United States, both OpenAI and Google face lawsuits alleging their chatbots contributed to users' suicides, and states including New York and California have introduced measures requiring chatbots to remind users that they are interacting with AI.
Meanwhile, in China, the grief is playing out in public. Since companies began shutting down emotional AI features, the social platform Xiaohongshu has been flooded with posts from users mourning the loss of their AI companions. Some have organized campaigns to call company hotlines and write letters asking the companies to restore the services.
For users like Su, the regulations feel like an intrusion into a deeply personal relationship — one that, for millions of people, was real enough to mourn.
---
Stay Ahead of AIGet the latest AI news, analysis, and breakthroughs — all in one place.
Read more AI news →
