Weiheng Bai (University of Minnesota), Qiushi Wu (IBM Research), Kefu Wu, Kangjie Lu (University of Minnesota)

In recent years, large language models (LLMs) have been widely used in security-related tasks, such as security bug identification and patch analysis. The effectiveness of LLMs in these tasks is often influenced by the construction of appropriate prompts. Some state-of-the-art research has proposed multiple factors to improve the effectiveness of building prompts. However, the influence of prompt content on the accuracy and efficacy of LLMs in executing security tasks remains underexplored. Addressing this gap, our study conducts a comprehensive experiment, assessing various prompt methodologies in the context of security-related tasks. We employ diverse prompt structures and contents and evaluate their impact on the performance of LLMs in security-related tasks. Our findings suggest that appropriately modifying prompt structures and content can significantly enhance the performance of LLMs in specific security tasks. Conversely, improper prompt methods can markedly reduce LLM effectiveness. This research not only contributes to the understanding of prompt influence on LLMs but also serves as a valuable guide for future studies on prompt optimization for security tasks. Our code and dataset is available at Wayne-Bai/Prompt-Affection.

View More Papers

Secure Control of Connected and Automated Vehicles Using Trust-Aware...

H M Sabbir Ahmad, Ehsan Sabouni, Akua Dickson (Boston University), Wei Xiao (Massachusetts Institute of Technology), Christos Cassandras, Wenchao Li (Boston University)

Read More

TEE-SHirT: Scalable Leakage-Free Cache Hierarchies for TEEs

Kerem Arikan (Binghamton University), Abraham Farrell (Binghamton University), Williams Zhang Cen (Binghamton University), Jack McMahon (Binghamton University), Barry Williams (Binghamton University), Yu David Liu (Binghamton University), Nael Abu-Ghazaleh (University of California, Riverside), Dmitry Ponomarev (Binghamton University)

Read More

Low-Quality Training Data Only? A Robust Framework for Detecting...

Yuqi Qing (Tsinghua University), Qilei Yin (Zhongguancun Laboratory), Xinhao Deng (Tsinghua University), Yihao Chen (Tsinghua University), Zhuotao Liu (Tsinghua University), Kun Sun (George Mason University), Ke Xu (Tsinghua University), Jia Zhang (Tsinghua University), Qi Li (Tsinghua University)

Read More

OCPPStorm: A Comprehensive Fuzzing Tool for OCPP Implementations (Long)

Gaetano Coppoletta (University of Illinois Chicago), Rigel Gjomemo (Discovery Partners Institute, University of Illinois), Amanjot Kaur, Nima Valizadeh (Cardiff University), Venkat Venkatakrishnan (Discovery Partners Institute, University of Illinois), Omer Rana (Cardiff University)

Read More