A deployment-efficient CNN-LSTM framework for real-time malicious traffic detection across heterogeneous, imbalanced, and encrypted network traffic

Abstract

Modern enterprise networks, including those adopting Software-Defined Wide Area Network (SD-WAN) architectures, face increasing intrusion detection challenges due to encrypted, heterogeneous, and imbalanced traffic. This paper presents a deployment-aware and robustness-oriented intrusion detection framework motivated by the security requirements of enterprise WAN and SD-WAN deployment environments. The experimental evaluation uses three heterogeneous benchmark traffic domains that are relevant to enterprise WAN monitoring but are not SD-WAN-specific datasets, and that do not cover all operational conditions. The study evaluates a deployment-efficient hybrid CNN-LSTM model across three heterogeneous traffic domains, including laboratory-generated traffic (CICIDS-2017), highly imbalanced multi-attack traffic (UNSW-NB15), and real-world encrypted traffic (CERNET-2020, with a substantial proportion of TLS/SSL-encrypted flows). A lightweight Channel Attention mechanism is incorporated, intended to enhance feature discrimination. SD-WAN is treated strictly as an intended deployment context rather than a validated setting: no SD-WAN testbed or SD-WAN traffic was used. Extensive experiments demonstrate that the proposed framework achieves 98.1% accuracy, 97.6% F1-score, and 0.99 AUC on the encrypted CERNET dataset, and 99.2% accuracy on CICIDS-2017, achieving strong within-dataset detection performance and providing cross-domain transferability insights across three heterogeneous benchmark datasets. To evaluate cross-domain robustness, a shared-feature transfer setting is introduced between CICIDS-2017 and UNSW-NB15, where the model achieves over 80% accuracy without retraining, indicating partial cross-dataset transferability for this 22-feature aligned subset only. A constrained transferability probe on CERNET (10 shared features only) indicates non-trivial discriminative capability despite substantial feature-space heterogeneity. In addition, the framework is designed for CPU-only inference, with an amortized per-flow model-inference latency of 2.3 ms (batch size 64; traffic acquisition, feature extraction, and window construction excluded), which is not end-to-end detection latency. Overall, this work contributes a systematic robustness evaluation framework for intrusion detection in heterogeneous, encrypted, and imbalanced network environments, addressing the gap between laboratory research and practical enterprise WAN deployment. The CNN-LSTM with channel attention is an established combination; the contribution lies in the evaluation protocol rather than in a new architecture. Comparisons are limited to the baselines evaluated (DNN-LSTM-GRU, PSO-D-SEM, DRL-DQN, and classical machine learning; PSO-D-SEM and DRL-DQN on CERNET only); no Transformer or TCN baseline was included, and no superiority over current methods is claimed. All experiments address the binary task (benign versus malicious); fine-grained attack-family and attack-stage information, which supports threat attribution, response prioritization, and forensics, is not assessed and is lost under this formulation.

The authors' abstract, as published at the source. Discover Artificial Intelligence, 2026 · DOI ↗