Lichao Wu (Technical University of Darmstadt), Sasha Behrouzi (Technical University of Darmstadt), Mohamadreza Rostami (Technical University of Darmstadt), Maximilian Thang (Technical University of Darmstadt), Stjepan Picek (University of Zagreb & Radboud University), Ahmad-Reza Sadeghi (Technical University of Darmstadt)

Safety alignment is critical for the ethical deployment of large language models (LLMs), guiding them to avoid generating harmful or unethical content. Current alignment techniques, such as supervised fine-tuning and reinforcement learning from human feedback, remain fragile and can be bypassed by carefully crafted adversarial prompts. Unfortunately, such attacks rely on trial and error, lack generalizability across models, and are constrained by scalability and reliability.

This paper presents NeuroStrike, a novel and generalizable attack framework that exploits a fundamental vulnerability introduced by alignment techniques: the reliance on sparse, specialized safety neurons responsible for detecting and suppressing harmful inputs. We apply NeuroStrike to both white-box and black-box settings: In the white-box setting, NeuroStrike identifies safety neurons through feedforward activation analysis and prunes them during inference to disable safety mechanisms. In the black-box setting, we propose the first LLM profiling attack, which leverages safety neuron transferability by training adversarial prompt generators on open-weight surrogate models and then deploying them against black-box and proprietary targets. We evaluate NeuroStrike on over 20 open-weight LLMs from major LLM developers. By removing less than 0.6% of neurons in targeted layers, NeuroStrike achieves an average attack success rate (ASR) of 76.9% using only vanilla malicious prompts. Moreover, Neurostrike generalizes to four multimodal LLMs with 100% ASR on unsafe image inputs. Safety neurons transfer effectively across architectures, raising ASR to 78.5% on 11 fine-tuned models and 77.7% on five distilled models. The black-box LLM profiling attack achieves an average ASR of 63.7% across five black-box models, including Google’s Gemini family.

View More Papers

Success Rates Doubled with Only One Character: Mask Password...

Yunkai Zou (Nankai University), Ding Wang (Nankai University), Fei Duan (Nankai University)

Read More

Breaking the Generative Steganography Trilemma: ANStega for Optimal Capacity,...

Yaofei Wang (Hefei University of Technology), Weilong Pang (Hefei University of Technology), Kejiang Chen (University of Science and Technology of China), Jinyang Ding (University of Science and Technology of China), Donghui Hu (Hefei University of Technology), Weiming Zhang (University of Science and Technology of China), Nenghai Yu (University of Science and Technology of China)

Read More

XR Devices Send WiFi Packets When They Should Not:...

Christopher Vattheuer (University of California, Los Angeles (UCLA)), Justin Feng (University of California, Los Angeles (UCLA)), Hossein Khalili (University of California, Los Angeles (UCLA)), Nader Sehatbakhsh (University of California, Los Angeles (UCLA)), Omid Abari (University of California, Los Angeles (UCLA))

Read More