TikTok bans deepfakes of nonpublic figures and fake endorsements in rule refresh

As the prospect of a US TikTok ban continues to grow, the video app has refreshed its content moderation policies. The rules on what content can be posted and promoted are largely unchanged but include new restrictions on sharing AI deepfakes, which have become increasingly popular on the app in recent months.

The bulk of these moderation policies (or “Community Guidelines,” in TikTok’s parlance) is unchanged and unsurprising. There’s no graphic violence allowed, no hate speech, and no overtly sexual content, with gradated rules for the latter based on the subject’s age. One newly expanded section, though, covers “synthetic and manipulated media” — aka AI deepfakes, which have become increasingly popular on the app in recent months.

Previously, TikTok’s rules on deepfakes were restricted to a single line banning content that could “mislead users by distorting the truth of events [or] cause significant harm to the subject of the video.” Now, the company says all realistic AI generated and edited content must be “clearly disclosed” as such, either in the video caption or as an overlaid sticker.

TikTok says it will not allow any synthetic media “that contains the likeness of any real private figure” or that shows a public figure endorsing a product or violating the app’s other policies (i.e., its prohibitions on hate speech). The company defines public figures as anyone 18 years of age or older with “a significant public role, such as a government official, politician, business leader, or celebrity.”

AI-generated content has increased in popularity on TikTok, thanks largely to the wider availability of AI voice cloning tools that make it easy to mimic someone’s voice. These tools have created new subgenres of content, often focused on placing public figures like President Joe Biden and former President Donald Trump in unexpected scenarios, like transposing the presidents’ personalities into arguments about online gaming or Dungeons & Dragons, for example.

Other use cases are more harmful. Many AI fakes show these same figures reading transphobic or homophobic statements and have sometimes been confused for real footage. TikTok’s prohibition on deepfake endorsements, meanwhile, seems like a response to a specific video that used AI to fake Joe Rogan promoting a “libido booster for men.” Such videos have also spread on apps like Twitter and Instagram.

The update to TikTok’s policies comes at a time of increasing political pressure for parent company ByteDance, as Western governments express fear over the app’s collection of private data and its potential to sway public opinion. The US government has reportedly threatened a public ban on TikTok if owner ByteDance doesn’t sell its stake, while the app has already been banned on government devices in the US, UK, New Zealand, and Canada.

TikTok does not address these threats to its business directly with these updated policies but notes that it wants to offer “much more transparency about our rules and how we enforce them.” The company is also publishing a list of eight “Community Principles” that it says “shape our day-to-day work and guide how we approach difficult enforcement decisions.” Notably, the first two principles are “prevent harm” and “enable free expression.”