San Francisco's city attorney has ordered Apple and Google to remove artificial intelligence nudification apps from their app stores, warning that the platforms are violating California laws prohibiting services that create deepfake pornography. The demand escalates a growing confrontation over AI tools that transform ordinary photos of real people into explicit images, a crisis that has disproportionately victimized women and children.
The Scale of the Problem
Nudification apps use generative AI to make it trivially easy to strip clothing from photos of real people, alter their features, place them in sexualized positions, and swap faces onto other people's naked bodies. The technology has proliferated rapidly across mobile app stores, and the results are being used to bully, humiliate, and threaten victims with devastating consequences.
San Francisco City Attorney David Chiu told Wired that his office was absolutely horrified by how ubiquitous the nudifying technology has become, victimizing mostly women and children at an alarming scale as more tools flooded the market. The images, he said, are used to destroy reputations, damage mental health, and strip victims of autonomy, with some targets becoming suicidal.
What the Letters Demand
Chiu's office sent formal letters to both companies warning that their app stores were violating California's laws that prohibit supporting services which create deepfake pornography, according to Wired. He asked Google to remove five apps and Apple to remove eight. The specific apps were not named in public reporting to avoid driving people toward them, but Wired noted that one flagged app had accumulated more than a million downloads and openly advertised features to sexualize images of women or produce free and uncensored videos.
Chiu's office estimated that Apple and Google have likely made millions of dollars in fees by allowing the harmful apps to remain in their stores rather than taking stronger action or developing better detection systems. The city attorney characterized the situation as profiting off a public nuisance, arguing that allowing any such apps to remain available is unacceptable.
Google Responds, Apple Stays Silent
In a statement to Ars Technica, Google spokesperson Dan Jackson confirmed that the five apps flagged by Chiu had been suspended from the Google Play store for violating policies against harmful content. Jackson said Google Play does not allow apps that contain sexual content and that the company continually takes proactive steps to detect and remove apps with harmful content. When violations are reported, Jackson said, Google investigates and takes swift action, which has included suspending hundreds of violating apps and restricting related search terms like nudify on its store.
Apple's response has been markedly different. While the company has removed some apps flagged by researchers in the past, it has otherwise remained notably silent amid calls to police nudification apps more aggressively in its App Store. The iPhone maker did not immediately provide a detailed public response to the city attorney's demands.
The Cat-and-Mouse Game of Detection
The challenge of policing these apps is compounded by their evolving tactics. Wired reported that researchers have recently warned some harmful apps have gotten better at evading app store removals by advertising only face-swapping features while concealing their nudifying capabilities. In a May preprint paper, researchers identified 420 apps marketed as generic face-swapping tools and tested 155 of them. In over 70 percent of cases, the apps could be used to sexualize images, demonstrating how easily harmful functionality hides behind innocuous marketing.
This detection gap means that even when app stores proactively search for nudify tools, many slip through by presenting themselves as harmless entertainment or photo-editing software. The researchers' findings suggest that the true number of problematic apps in circulation is likely far higher than the thirteen flagged by Chiu's office.
The Grok Precedent
The nudify app crackdown also intersects with a broader debate over AI-generated harmful content. This week, xAI filed a lawsuit confirming that it had found instances of Grok-generated child sexual abuse materials and nonconsensual intimate imagery targeting adults. To address the misuse, xAI sued the user allegedly responsible for prompting Grok to generate the illegal content.
The Grok case raises a thornier question for app stores: whether AI chatbots that can be prompted to produce illegal content should themselves be subject to removal. Apple reportedly told senators in April that it had privately threatened to remove Grok from the App Store, according to NBC News, but the xAI app remains available today. As long as it is possible to use Grok to generate illegal content, app stores will likely continue to face pressure to take action, just as Chiu demanded for standalone nudification apps.
A Turning Point for Platform Accountability
The San Francisco order represents one of the most aggressive government interventions to date targeting app stores over AI-generated harmful content. By invoking California's deepfake pornography laws, Chiu is testing whether state statutes can compel platform operators to take responsibility for the AI tools distributed through their marketplaces.
The outcome could set a precedent for how other jurisdictions approach the problem, and whether the burden of policing AI-generated nonconsensual imagery falls on developers, platform operators, or both. For the millions of people, predominantly women and girls, who have been targeted by these tools, the stakes extend far beyond regulatory theory.
Stay Ahead of AI
For ongoing coverage of AI ethics, platform accountability, and the regulatory battles over harmful synthetic media, follow AI Buzz Wire for reporting on the issues shaping AI's impact on society.
Read more AI ethics news →
