Evan Li (Northeastern University), Tushin Mallick (Northeastern University), Evan Rose (Northeastern University), William Robertson (Northeastern University), Alina Oprea (Northeastern University), Cristina Nita-Rotaru (Northeastern University)

LLM-integrated app systems extend the utility of Large Language Models (LLMs) with third-party apps that are invoked by a system LLM using interleaved planning and execution phases to answer user queries. These systems introduce new attack vectors where malicious apps can cause integrity violation of planning or execution, availability breakdown, or privacy compromise during execution.

In this work, we identify new attacks impacting the integrity of planning, as well as the integrity and availability of execution in LLM-integrated apps, and demonstrate them against IsolateGPT, a recent solution designed to mitigate attacks from malicious apps. We propose Abstract-Concrete-Execute (ACE), a new secure architecture for LLM-integrated app systems that provides security guarantees for system planning and execution. Specifically, ACE decouples planning into two phases by first creating an abstract execution plan using only trusted information, and then mapping the abstract plan to a concrete plan using installed system apps. We verify that the plans generated by our system satisfy user-specified secure information flow constraints via static analysis on the structured plan output. During execution, ACE enforces data and capability barriers between apps, and ensures that the execution is conducted according to the trusted abstract plan. We show experimentally that ACE is secure against attacks from the InjecAgent and Agent Security Bench benchmarks for indirect prompt injection, and our newly introduced attacks. We also evaluate the utility of ACE in realistic environments, using the Tool Usage suite from the LangChain benchmark. Our architecture represents a significant advancement towards hardening LLM-based systems using system security principles.

View More Papers

Odysseus: Jailbreaking Commercial Multimodal LLM-integrated Systems via Dual Steganography

Songze Li (Southeast University), Jiameng Cheng (Southeast University), Yiming Li (Nanyang Technological University), Xiaojun Jia (Nanyang Technological University), Dacheng Tao (Nanyang Technological University)

Read More

Fuzzilicon: A Post-Silicon Microcode-Guided x86 CPU Fuzzer

Johannes Lenzen (Technical University of Darmstadt), Mohamadreza Rostami (Technical University of Darmstadt), Lichao Wu (Technical University of Darmstadt), Ahmad-Reza Sadeghi (Technical University of Darmstadt)

Read More

SysArmor: The Practice of Integrating Provenance Analysis into Endpoint...

Shaofei Li (Peking University), Jiandong Jin (Peking University), Hanlin Jiang (Peking University), Yi Huang (Peking University), Yifei Bao (Jilin University), Yuhan Meng (Peking University), Fengwei Hong (Peking University), Zheng Huang (Peking University), Peng Jiang (Southeast University), Ding Li (Peking University)

Read More