Ernest Akpaku (School of Computer Science and Communication Engineering, Jiangsu University), Jinfu Chen (School of Computer Science and Communication Engineering, Jiangsu University), Joshua Ofoeda (University of Professional Studies, Accra)
Detecting advanced cyber threats, particularly zero-day vulnerabilities, poses significant challenges in network security. This paper presents TIPSO-GAN, an optimized Generative Adversarial Network (GAN) for detecting malicious traffic. TIPSO-GAN addresses common GAN-based intrusion detection system (IDS) issues, such as training instability and mode collapse, by framing GAN training as a swarm optimization problem, harnessing collective intelligence for complex optimization. To enhance Particle Swarm Optimization (PSO), TIPSO- GAN employs three strategies: (1) adaptive inertia weights for a balance of exploration and exploitation, (2) a diversity preservation strategy to prevent premature convergence, and (3) a feedback loop to reinitialize stagnant particles. TIPSO-GAN integrates transfer learning with a Temporal-Decaying Multi- Head Self-Attention mechanism to prioritize recent features, aiding in unseen malicious traffic detection. A combination of reconstruction loss and focal loss in the objective function further ensures realistic normal samples while focusing on challenging malicious samples. Across CIC-IDS2018, CICAPT-IIoT2024, and CIC-DDoS2019, TIPSO-GAN achieves 99.1±0.1, 98.9±0.1, and 98.7±0.1 F1, outperforming the strongest baseline by 0.2–1.0 F1 and exceeding transformer IDS models. On CICAPT-IIoT2024, it reaches 0.999±0.002 macro PR-AUC, ahead of the next best method (0.960±0.005). Under strict zero-day evaluations, TIPSO-GAN attains 92.3 F1 in LOFO tests and 79–83 F1 in cross-dataset experiments while maintaining recall above 0.80. Despite PSO-enhanced training, TIPSO-GAN maintains 0.42 ms latency, ∼2400 flows/s throughput, and a 2.1 GB footprint, with stable performance up to $10^{8}$ flows. Our code is accessible at https://github.com/osampas27/tipsoganmod.