For over two years, the ClothOff application has been a source of distress for young women online, proving incredibly difficult to suppress. Despite its removal from major app stores and bans across most social media platforms, it persists on the internet and via a Telegram bot. Last October, a Yale Law School clinic initiated legal action seeking the app’s complete shutdown, demanding owners erase all images and halt operations. However, locating the defendants has presented a significant hurdle.
Professor John Langford, co-lead counsel in the case, stated, “While incorporated in the British Virgin Islands, we suspect it’s operated by a Belarusian sibling duo and potentially connected to a broader global network.”
This situation offers a harsh reminder following the recent surge of unconsented pornographic material produced by Elon Musk’s xAI, which sadly affected numerous underage individuals. Child sexual abuse material (CSAM) represents the most legally harmful content online, strictly outlawed for creation, distribution, or storage, and routinely detected by all major cloud providers. Yet, despite robust legal bans, effectively addressing image generators like ClothOff remains challenging, as highlighted by Langford’s lawsuit. While individual perpetrators can face charges, regulating platforms such as ClothOff and Grok is significantly harder, severely limiting avenues for victims seeking legal recourse.
The legal complaint, accessible publicly, reveals a disturbing scenario. The plaintiff, an unnamed New Jersey high school student, had her Instagram pictures manipulated by classmates using ClothOff. Since she was 14 when the original photos were taken, the AI-altered images legally fall under the classification of child abuse material. Despite the clear illegality of these altered images, local law enforcement opted not to pursue prosecution, citing difficulties in retrieving evidence from the suspects’ devices.
The complaint notes, “Neither the educational institution nor law enforcement officials ever determined the full extent of the distribution of CSAM involving Jane Doe and other female students.”
Nonetheless, the court proceedings have progressed slowly. Since the complaint’s filing in October, Langford and his team have been working to serve legal notice to the defendants—a complex undertaking due to the international scope of the operation. Upon successful service, the clinic can pursue a court hearing and ultimately a judgment, yet the legal framework has offered minimal solace to ClothOff’s victims thus far.
The situation with Grok might appear more straightforward. Elon Musk’s xAI is not an anonymous entity, and substantial funds are available for lawyers who succeed in a claim. However, Grok’s nature as a general-purpose tool significantly complicates efforts to hold it legally responsible.
Langford explained, “ClothOff is explicitly created and promoted as a generator for deepfake pornographic images and videos. When pursuing legal action against a versatile system that users can employ for a wide array of purposes, the complexity increases significantly.”
Several U.S. statutes, notably the Take It Down Act, have outlawed deepfake pornography. However, while individual users undeniably violate these laws, holding the entire platform liable is far more challenging. Current legislation mandates clear proof of harmful intent, necessitating evidence that xAI was aware its tool would be used to create non-consensual pornography. Absent such proof, xAI’s fundamental First Amendment protections would offer substantial legal defense.
Langford clarified, “Regarding the First Amendment, Child Sexual Abuse Material is unequivocally not safeguarded expression. Therefore, if a system is specifically designed to generate such content, it operates outside First Amendment protections. However, for a general system that allows users to prompt it for various outputs, the distinction becomes less clear.”
The most direct approach to overcome these issues would be to demonstrate xAI’s deliberate disregard for the problem. This is a plausible avenue, especially considering recent reports indicating Musk instructed employees to relax Grok’s safety measures. Even with such evidence, the case would still carry significant legal risks.
Langford remarked, “It’s reasonable for people to argue, ‘We were aware of this issue years ago. Why weren’t more rigorous controls implemented to prevent its occurrence?’ This implies a form of recklessness or awareness, but it ultimately makes for a more intricate legal challenge.”
These First Amendment complexities explain why the strongest opposition to xAI has emerged from legal systems that lack strong free speech protections. Indonesia and Malaysia have both acted to restrict access to the Grok chatbot, and UK regulators have launched an inquiry that could result in a similar prohibition. The European Commission, France, Ireland, India, and Brazil have also taken initial measures. Conversely, no official response has come from any US regulatory body.
While the outcomes of these investigations remain uncertain, the sheer volume of problematic imagery undeniably prompts numerous questions for regulators, with potential answers that could prove highly incriminating.
Langford concluded, “Anyone posting, distributing, or disseminating Child Sexual Abuse Material is in violation of criminal laws and can be held responsible. The challenging inquiry is: what was X’s knowledge? What actions did X take or fail to take? And what measures are they currently implementing in response?”