Omar Abusabha (Sungkyunkwan University, South Korea), Jiyong Uhm (Sungkyunkwan University, South Korea), Tamer Abuhmed (Sungkyunkwan University, South Korea), Hyungjoon Koo (Sungkyunkwan University, South Korea)

A function inlining optimization is a widely used transformation in modern compilers, which replaces a call site with the callee’s body in need. While this transformation improves performance, it significantly impacts static features such as machine instructions and control flow graphs, which are crucial to binary analysis. Yet, despite its broad impact, the security impact of function inlining remains underexplored to date. In this paper, we present the first comprehensive study of function inlining through the lens of machine learning-based binary analysis. To this end, we dissect the inlining decision pipeline within the LLVM’s cost model and explore the combinations of the compiler options that aggressively promote the function inlining ratio beyond standard optimization levels, which we term extreme inlining. We focus on five ML-assisted binary analysis tasks for security, using 20 unique models to systematically evaluate their robustness under extreme inlining scenarios. Our extensive experiments reveal several significant findings: i) function inlining, though a benign transformation in intent, can (in)directly affect ML model behaviors, being potentially exploited by evading discriminative or generative ML models; ii) ML models relying on static features can be highly sensitive to inlining; iii) subtle compiler settings can be leveraged to deliberately craft evasive binary variants; and iv) inlining ratios vary substantially across applications and build configurations, undermining assumptions of consistency in training and evaluation of ML models.

View More Papers

A Hard-Label Black-Box Evasion Attack against ML-based Malicious Traffic...

Zixuan Liu (Tsinghua University), Yi Zhao (Beijing Institute of Technology), Zhuotao Liu (Tsinghua University and Zhongguancun Lab), Qi Li (Tsinghua University and Zhongguancun Lab), Chuanpu Fu (Tsinghua University), Guangmeng Zhou (Tsinghua University), Ke Xu (Tsinghua University and Zhongguancun Lab)

Read More

VDORAM: Towards a Random Access Machine with Both Public...

Huayi Qi (School of Computer Science and Technology, Shandong University, Qingdao, Shandong, China and Institute for Network Sciences and Cyberspace, Tsinghua University, Beijing, China), Minghui Xu (School of Computer Science and Technology, Shandong University, Qingdao, Shandong, China), Xiaohua Jia (Department of Computer Science, City University of Hong Kong, Kowloon, Hong Kong SAR, China), Xiuzhen Cheng…

Read More