ANOMALY DETECTION IN CYBERSECURITY USING DEEP LEARNING APPROACH

ABSTRACT The increasing complexity and volume of cyber threats have exposed critical limitations in traditional signature-based and rule-driven intrusion detection systems, particularly in detecting novel and previously unseen attacks. In response, this study investigates the application of unsupervised deep learning-based anomaly detection for network security, focusing on autoencoder architectures trained exclusively on normal network traffic. The objective of this research is to design, implement, and evaluate an unsupervised anomaly detection framework capable of identifying abnormal network behavior without reliance on labeled attack signatures. The proposed approach employs three autoencoder variants: a standard feedforward Autoencoder (AE), a Long Short-Term Memory Autoencoder (LSTM AE), and a Bidirectional LSTM Autoencoder (BiLSTM-AE). All models are trained using only normal traffic samples from the NSL-KDD benchmark dataset, learning to reconstruct normal patterns; anomalies are detected when reconstruction error exceeds a threshold. Data preprocessing includes cleaning, normalization, feature encoding, and dataset balancing. Model performance is evaluated using standard metrics, including accuracy, precision, recall, F1-score, and area under the ROC curve (AUC). Experimental results demonstrate that the Standard Autoencoder achieves the highest overall performance, with an F1-score of 0.892 and an AUC-ROC of 0.940 on the NSL-KDD test set, effectively detecting network intrusions without requiring labeled attack data. In contrast, both LSTM-based models perform near randomly (F1 < 0.07, AUC ≈ 0.5), revealing that the NSL-KDD dataset lacks the temporal dependencies necessary for recurrent architectures to be effective. A critical methodological contribution is the demonstration that the 95th percentile threshold increases recall from below 1% to 88% compared to the traditional μ+3σ rule, providing a practical guideline for threshold selection in autoencoder-based anomaly detection. These findings underscore the importance of aligning model architecture with data structure and establish the Standard Autoencoder as a practical, deployable solution for network intrusion detection. While the study is limited to offline analysis on a benchmark dataset, it provides a foundation for future research on real-time deployment, adaptive learning, and explainable anomaly detection in practical cybersecurity environments. Keywords: Anomaly Detection (AD), Autoencoder (AE), Deep learning (DL), Intrusion Detection System (IDS), Unsupervised Learning, NSL-KDD