Oliver D. Reithmaier (Leibniz University Hannover), Thorsten Thiel (Atmina Solutions), Anne Vonderheide (Leibniz University Hannover), Markus Dürmuth (Leibniz University Hannover)
Email phishing to date still is the most common attack on IT systems. While early research has focused on collective and large-scale phishing campaign studies to enquire why people fall for phishing, such studies are limited in their inference regarding individual or contextual influence of user phishing detection. Researchers tried to address this limitation using scenario-based or role-play experiments to uncover individual factors influencing user phishing detection. Studies using these methods unfortunately are also limited in their ability to generate inference due to their lack of ecological validity and experimental setups. We tackle this problem by introducing PhishyMailbox, a free and open-source research software designed to deploy mail sorting tasks in a simulated email environment. By detailing the features of our app for researchers and discussing its security and ethical implications, we demonstrate the advantages it provides over previously used paradigms for scenario-based research, especially regarding ecological validity as well as generalizability through larger possible sample sizes.We report excellent usability statistics from a preliminary sample of usable security scientists and discuss ethical implications of the app. Finally, we discuss future implementation opportunities of PhishyMailbox in research designs leveraging signal detection theory, item response theory and eye tracking applications.