Mengyuan Sun (Wuhan University), Yu Li (Wuhan University), Yunjie Ge (Wuhan University), Yuchen Liu (Wuhan University), Bo Du (Wuhan University), Qian Wang (Wuhan University)

Multimodal contrastive learning models like CLIP have demonstrated remarkable vision-language alignment capabilities and now serve as foundational components in many large-scale multimodal systems. However, their vulnerability to backdoor attacks poses critical security risks. Attackers can implant latent triggers that persist through downstream tasks, enabling malicious control of model behavior upon trigger presentation. Despite great success in recent defense mechanisms, they remain impractical due to strong assumptions about attacker knowledge or excessive clean data requirements.

In this paper, we introduce InverTune, the first backdoor defense framework for multimodal models under minimal attacker assumptions, requiring neither prior knowledge of attack targets nor access to the poisoned dataset. Unlike existing defense methods that rely on the same dataset used in the poisoning stage, InverTune effectively identifies and removes backdoor artifacts through three key components, achieving robust protection against backdoor attacks. Specifically, (1) InverTune first exposes attack signatures through adversarial simulation, probabilistically identifying the target label by analyzing model response patterns. (2) Building on this, we develop a gradient inversion technique to reconstruct latent triggers through activation pattern analysis. (3) Finally, a clustering-guided fine-tuning strategy is employed to erase the backdoor function with only a small amount of arbitrary clean data, while preserving the original model capabilities. Experimental results show that InverTune reduces the average attack success rate (ASR) by 97.87% against the state-of-the-art (SOTA) attacks while limiting clean accuracy (CA) degradation to just 3.07%. This work establishes a new paradigm for securing multimodal systems, advancing security in foundation model deployment without compromising performance.

View More Papers

UsersFirst in Practice: Evaluating a User-Centric Threat Modeling Taxonomy...

Alexandra Xinran Li (Carnegie Mellon University), Tian Wang (University of Illinois Urbana-Champaign), Yu-Ju Yang (University of Illinois Urbana-Champaign), Miguel Rivera-Lanas (Carnegie Mellon University), Debeshi Ghosh (Carnegie Mellon University), Hana Habib (Carnegie Mellon University), Lorrie Cranor (Carnegie Mellon University), Norman Sadeh (Carnegie Mellon University)

Read More

Benchmarking and Understanding Safety Risks in AI Character Platforms

Yiluo Wei (The Hong Kong University of Science and Technology (Guangzhou)), Peixian Zhang (The Hong Kong University of Science and Technology (Guangzhou)), Gareth Tyson (The Hong Kong University of Science and Technology (Guangzhou))

Read More

Crack in the Armor: Underlying Infrastructure Threats to RPKI...

Yunhao Liu (Tsinghua University & Zhongguancun Laboratory), Jessie Hui Wang (Tsinghua University & Zhongguancun Laboratory), Yuedong Xu (Fudan University), Zongpeng Li (Tsinghua University), Yangyang Wang (Tsinghua University & Zhongguancun Laboratory), Jilong Wang (Tsinghua University & Zhongguancun Laboratory)

Read More