Yukina Okazawa (Toho University), Akira Kanaoka (Toho University), Takumi Yamamoto (Mitsubishi Electric Corporation)
Security Operation Centers (SOCs) rely on security monitoring tools such as SIEM systems and IDSs, yet the usability of these tools remains insufficiently examined despite their essential role in analysts’ daily workflows. Prior research has highlighted operational burdens including overwhelming alert volume, high false positive rates, and analyst fatigue. However, existing efforts have focused mainly on technical alert reduction rather than evaluating how effectively SOC tools support analysts’ decision making in practice. This gap indicates the need for a structured and SOC specific usability evaluation methodology. This paper introduces a methodology for evaluating the usability of SOC tools that combines a heuristic walkthrough with eleven evaluation criteria derived from empirical studies of SOC operations. These criteria capture usability factors that general purpose techniques often overlook, such as context dependent interpretation, escalation reasoning, and reliance on environmental knowledge. To support controlled and reproducible evaluations, we also present a simulated operational environment that produces realistic sequences of alerts, benign events, and false positives based on representative attack scenarios. We apply the method to an open source SIEM, Prelude OSS, and demonstrate how the framework identifies recurring usability challenges such as limited contextual support, inconsistent workflow guidance, and difficulties in handling realistic alert volumes. These challenges align with previously reported issues in SOC practice, indicating that the proposed method can systematically expose usability problems inherent to many SOC tools rather than issues specific to a single system. Together, the methodology and simulated environment provide a foundation for rigorous and repeatable usability evaluations of SOC tools, complementing existing technical approaches to alert reduction and offering concrete directions for improving tool design.