Yan Pang (University of Virginia), Aiping Xiong (Penn State University), Yang Zhang (CISPA Helmholtz Center for Information Security), Tianhao Wang (University of Virginia)

Video generation models (VGMs) have demonstrated the capability to synthesize high-quality output. It is important to understand their potential to produce unsafe content, such as violent or terrifying videos. In this work, we provide a comprehensive understanding of unsafe video generation.

First, to confirm the possibility that these models could indeed generate unsafe videos, we choose unsafe content generation prompts collected from 4chan and Lexica, and three open-source SOTA VGMs to generate unsafe videos.
After filtering out duplicates and poorly generated content, we created an initial set of $2112$ unsafe videos from an original pool of $5607$ videos. Through clustering and thematic coding analysis of these generated videos, we identify $5$ unsafe video categories: textit{Distorted/Weird}, textit{Terrifying}, textit{Pornographic}, textit{Violent/Bloody}, and textit{Political}. With IRB approval, we then recruit online participants to help label the generated videos. Based on the annotations submitted by $403$ participants, we identified $937$ unsafe videos from the initial video set. With the labeled information and the corresponding prompts, we created the first dataset of unsafe videos generated by VGMs.

We then study possible defense mechanisms to prevent the generation of unsafe videos. Existing defense methods in image generation focus on filtering either input prompt or output results. We propose a new approach called fullsysname (sysname), which works within the model’s internal sampling process. sysname can achieve $0.90$ defense accuracy while reducing time and computing resources by $10times$ when sampling a large number of unsafe prompts. Our experiment includes three open-source SOTA video diffusion models, each achieving accuracy rates of $0.99$, $0.92$, and $0.91$, respectively. Additionally, our method was tested with adversarial prompts and on image-to-video diffusion models, and achieved nearly $1.0$ accuracy on both settings. Our method also shows its interoperability by improving the performance of other defenses when combined with them.

View More Papers

Automated Expansion of Privacy Data Taxonomy for Compliant Data...

Yue Qin (Indiana University Bloomington & Central University of Finance and Economics), Yue Xiao (Indiana University Bloomington & IBM Research), Xiaojing Liao (Indiana University Bloomington)

Read More

The Discriminative Power of Cross-layer RTTs in Fingerprinting Proxy...

Diwen Xue (University of Michigan), Robert Stanley (University of Michigan), Piyush Kumar (University of Michigan), Roya Ensafi (University of Michigan)

Read More

Unleashing the Power of Generative Model in Recovering Variable...

Xiangzhe Xu (Purdue University), Zhuo Zhang (Purdue University), Zian Su (Purdue University), Ziyang Huang (Purdue University), Shiwei Feng (Purdue University), Yapeng Ye (Purdue University), Nan Jiang (Purdue University), Danning Xie (Purdue University), Siyuan Cheng (Purdue University), Lin Tan (Purdue University), Xiangyu Zhang (Purdue University)

Read More

Truman: Constructing Device Behavior Models from OS Drivers to...

Zheyu Ma (Institute for Network Sciences and Cyberspace (INSC), Tsinghua University; EPFL; JCSS, Tsinghua University (INSC) - Science City (Guangzhou) Digital Technology Group Co., Ltd.), Qiang Liu (EPFL), Zheming Li (Institute for Network Sciences and Cyberspace (INSC), Tsinghua University; JCSS, Tsinghua University (INSC) - Science City (Guangzhou) Digital Technology Group Co., Ltd.), Tingting Yin (Zhongguancun…

Read More