Ghazal Abdollahi (University of Utah), Hamid Asadi (University of Utah), Robert Ricci (University of Utah)
Persistent, high-volume SSH brute-force activity frequently overwhelms security operations, yet current defenses often treat network telemetry as a terminal artifact for post-hoc diagnosis rather than a source for upstream investigation. These approaches focus on absolute volume suppression and binary alerts, often failing to provide population-aware rankings that are necessary to prioritize high-risk, relative outliers. This work addresses these gaps by introducing Nested Outlier Detection (NOD), a two-stage framework that transforms raw network telemetry into structured behavioral strata. By progressively filtering routine noise, NOD isolates ”outliers of outliers”; statistically extreme behaviors. NOD provides interpretability by mapping these outliers to three intuitive dimensions; volume, reach, and credential diversity; enabling population-level reasoning. This tiered approach reveals distinct attacker phenotypes characterized by high volume, broad target reach, and a variety of credentials. Evaluation on large-scale datasets demonstrates that NOD compresses millions of logs into compact, interpretable structures, shifting the defensive focus from per-source classification to the graded, population-level reasoning required for scalable triage and longitudinal threat analysis.