New Decision Addresses Meta’s Rules on Non-Consensual Deepfake Intimate Images
July 25, 2024
In two cases of explicit AI images that resemble female public figures from India and the United States, the Oversight Board finds that both posts should have been removed from Meta’s platforms. Deepfake intimate images disproportionately affect women and girls – undermining their rights to privacy and protection from mental and physical harm. Restrictions on this content are legitimate to protect individuals from the creation and dissemination of sexual images made without their consent. Given the severity of harms, removing the content is the only effective way to protect the people impacted. Labeling manipulated content is not appropriate in this instance because the harms stem from the sharing and viewing of these images – and not solely from misleading people about their authenticity. The Board’s recommendations seek to make Meta’s rules on this type of content more intuitive and to make it easier for users to report non-consensual sexualized images.
About the Cases
These two cases involve AI-generated images of nude women, one resembling an Indian public figure, the other an American public figure. In the first case, an Instagram account that shared only AI-generated or manipulated images of Indian women posted a picture of the back of a nude woman with her face visible, as part of a set of images. This set also featured a similar picture of the woman in beachwear, most likely the source material for the explicit AI manipulation. The second case also involves an explicit AI-generated image resembling a female public figure, this time from the United States. In this image, posted to a Facebook group for AI creations, the nude woman is being groped. The famous figure she resembles is named in the caption.
In the first case (Indian public figure), a user reported the content to Meta for pornography but as the report was not reviewed within 48 hours, it was automatically closed. The user then appealed to Meta, but this was also automatically closed. Finally, the user appealed to the Board. As a result of the Board selecting this case, Meta determined that its original decision to leave the content on Instagram was in error and the company removed the post for violating the Bullying and Harassment Community Standard. Later, after the Board began its deliberations, Meta disabled the account that posted the content and added the explicit image to a Media Matching Service (MMS) bank.
In the second case (American public figure), the explicit image had already been added to an MMS bank for violating Meta’s Bullying and Harassment policy and so was automatically removed. These banks automatically find and remove images that already have been identified by human reviewers as breaking Meta’s rules. The user who posted the AI-generated image appealed but this was automatically closed. The user then appealed to the Board to have their post restored.
Deepfake intimate images comprise synthetic media digitally manipulated to depict real people in a sexualized way. It is becoming easier to create, with fewer pictures required to generate a realistic image. One report points to a 550% increase in online deepfake videos since 2019, the vast majority of which are sexualized depictions of real individuals and target women.
Key Findings
The Board finds that both images violated Meta’s rule that prohibits “derogatory sexualized photoshop” under the Bullying and Harassment policy. It is clear the images have been edited to show the faces of real public figures with a different (real or fictional) nude body, while contextual clues, including hashtags and where the content was posted, also indicate they are AI-generated. In the second case (American public figure), there is an additional violation of the Adult Nudity and Sexual Activity policy as the explicit image shows the woman having her breast squeezed. Removing both posts was in line with Meta’s human rights responsibilities.
The Board believes that people using Meta’s platforms should be able to understand the rules. While the term “derogatory sexualized photoshop” should have been clear enough to the two users posting in these cases, it is not sufficiently clear more generally to users. When the Board asked Meta about the meaning, the company said the term refers to “manipulated images that are sexualized in ways that are likely to be unwanted by the target and thus perceived as derogatory.” The Board notes that a different term such as “non-consensual” would be a clearer description to explain the idea of unwanted sexualized manipulations of images. Additionally, the Board finds that “photoshop” is too narrow to cover the array of media manipulation techniques available today, especially generative AI. Meta needs to specify in this rule that the prohibition on this content covers this broader range of editing techniques.
To ensure the rules prohibiting non-consensual sexualized images are more intuitive, the Board finds they should be part of the Adult Sexual Exploitation Community Standard, rather than Bullying and Harassment. In both these cases, users would have been unlikely to perceive them as an issue of Bullying and Harassment. External research shows that users post such content for many reasons besides harassment and trolling, including a desire to build an audience, monetize pages or direct users to other sites, including pornographic ones. Therefore, Meta’s rules on these images would be clearer if the focus was on the lack of consent and the harms from such content proliferating – rather than the impact of direct attacks, which is what is implied by enforcing under Bullying and Harassment. The Adult Sexual Exploitation policy would be a more logical place for these rules. This policy already prohibits non-consensual intimate images, which is a similar issue as both are examples of image-based sexual abuse. Then, Meta could also consider renaming the policy to “Non-Consensual Sexual Content.”
The Board notes the image resembling an Indian public figure was not added to an MMS bank by Meta until the Board asked why. Meta responded by saying that it relied on media reports to add the image resembling the American public figure to the bank, but there were no such media signals in the first case. This is worrying because many victims of deepfake intimate images are not in the public eye and are forced to either accept the spread of their non-consensual depictions or search for and report every instance. One of the existing signals of lack of consent under the Adult Sexual Exploitation policy is media reports of leaks of non-consensual intimate images. This can be useful when posts involve public figures but is not helpful for private individuals. Therefore, Meta should not be over-reliant on this signal. The Board also suggests that context indicating the nude or sexualized aspects of the content are AI-generated, photoshopped or otherwise manipulated be considered as a signal of non-consent.
Finally, the Board is concerned about the auto-closing of appeals for image-based sexual abuse. Even waiting 48 hours for a review can be harmful given the damage caused. The Board does not yet have sufficient information on Meta’s use of auto-closing generally but considers this an issue that could have a significant human rights impact, requiring risk assessment and mitigation.
The Oversight Board’s Decision
In the first case (Indian public figure), the Board overturns Meta’s original decision to leave up the post. In the second case (American public figure), the Board upholds Meta’s decision to take down the post.
The Board recommends that Meta:
- Move the prohibition on “derogatory sexualized photoshop” into the Adult Sexual Exploitation Community Standard.
- Change the word “derogatory” in the prohibition on “derogatory sexualized photoshop” to “non-consensual.”
- Replace the word “photoshop” in the prohibition on “derogatory sexualized photoshop” with a more generalized term for manipulated media.
- Harmonize its policies on non-consensual content by adding a new signal for lack of consent in the Adult Sexual Exploitation policy: context that content is AI-generated or manipulated. For content with this specific context, the policy should also specify that it need not be “non-commercial or produced in a private setting” to be violating.
For Further Information
To read a synopsis of public comments for this case, please click here.
