Can You Do that Again? Real-World Requirements for Cybersecurity Experiment Replication

Abstract

Novel cybersecurity technologies are often prototyped and evaluated using experiments conducted by both the technology developers and 3rd party evaluators. Evaluations may rely on testbeds and experimentation entirely within the bounds of a testbed, or require datasets, devices or experimental procedures that drive non-testbed-based evaluations. In all circumstances, however, the ability to replicate results between different experiments is crucial. In this talk, I will describe experimental methods and procedures evolved through practical use at ISI on a number of security research projects, and highlight some real-world takeaways that support replication, based on building and evaluating research prototypes.

Speaker’s Biography

Stephen Schwab is currently a Research Director in the USC Information Sciences Institute’s Networking and Cybersecurity Division. He has performed research in operating systems and network security, including building and using the DETER cyber security testbed and tools at ISI for DHS, NSF and DARPA. Recent projects have focused on forecasting and attributing malicious attacks, as well as detecting malicious tampering in embedded firmware. He is currently building a new generation of testbeds for experimentation, using the MERGE testbed software, to conduct experiments for the DARPA Dispersed Computing program and other research programs.