Lujia Shen (Zhejiang University), Yuwen Pu (Zhejiang University), Shouling Ji (Zhejiang University), Changjiang Li (Penn State), Xuhong Zhang (Zhejiang University), Chunpeng Ge (Shandong University), Ting Wang (Penn State)

Transformer-based models, such as BERT and GPT, have been widely adopted in natural language processing (NLP) due to their exceptional performance. However, recent studies show their vulnerability to textual adversarial attacks where the model's output can be misled by intentionally manipulating the text inputs. Despite various methods that have been proposed to enhance the model's robustness and mitigate this vulnerability, many require heavy consumption resources (e.g., adversarial training) or only provide limited protection (e.g., defensive dropout).

In this paper, we propose a novel method called dynamic attention, tailored for the transformer architecture, to enhance the inherent robustness of the model itself against various adversarial attacks. Our method requires no downstream task knowledge and does not incur additional costs. The proposed dynamic attention consists of two modules: *(i) attention rectification*, which masks or weakens the attention value of the chosen tokens, and *(ii) dynamic modeling*, which dynamically builds the set of candidate tokens. Extensive experiments demonstrate that dynamic attention significantly mitigates the impact of adversarial attacks, improving up to 33% compared to previous methods against widely used adversarial attacks. The model-level design of dynamic attention enables it to be easily combined with other defense methods (e.g., adversarial training) to further enhance the model's robustness. Furthermore, we demonstrate that dynamic attention preserves the state-of-the-art robustness space of the original model compared to other dynamic modeling methods.

View More Papers

Exploring the Influence of Prompts in LLMs for Security-Related...

Weiheng Bai (University of Minnesota), Qiushi Wu (IBM Research), Kefu Wu, Kangjie Lu (University of Minnesota)

Read More

Vision: An Exploration of Online Toxic Content Against Refugees

Arjun Arunasalam (Purdue University), Habiba Farrukh (University of California, Irvine), Eliz Tekcan (Purdue University), Z. Berkay Celik (Purdue University)

Read More

Flow Correlation Attacks on Tor Onion Service Sessions with...

Daniela Lopes (INESC-ID / IST, Universidade de Lisboa), Jin-Dong Dong (Carnegie Mellon University), Pedro Medeiros (INESC-ID / IST, Universidade de Lisboa), Daniel Castro (INESC-ID / IST, Universidade de Lisboa), Diogo Barradas (University of Waterloo), Bernardo Portela (INESC TEC / Universidade do Porto), João Vinagre (INESC TEC / Universidade do Porto), Bernardo Ferreira (LASIGE, Faculdade de…

Read More

PriSrv: Privacy-Enhanced and Highly Usable Service Discovery in Wireless...

Yang Yang (School of Computing and Information Systems, Singapore Management University, Singapore), Robert H. Deng (School of Computing and Information Systems, Singapore Management University, Singapore), Guomin Yang (School of Computing and Information Systems, Singapore Management University, Singapore), Yingjiu Li (Department of Computer Science, University of Oregon, USA), HweeHwa Pang (School of Computing and Information Systems,…

Read More