A deep learning-based data detector for fiber communication system in the presence of nonlinear noise

Abstract

In this paper, the authors introduce a data detection method for optical fiber communication systems affected by nonlinear noise and model the entire system as an end-to-end deep learning (DL) neural network, incorporating the transmitter, transmission channel, and receiver. Based on this framework, the authors propose an autoencoder-based neural network architecture for long-haul optical communication systems to enhance data detection performance. Through simulations, the authors compare and evaluate the proposed method against Maximum Likelihood (ML) detection and other autoencoder models, considering both linear and nonlinear noise scenarios with a 16-QAM signal. The simulation results demonstrate that, the proposed approach achieves superior accuracy and computational efficiency. This study hence provides a foundation for optimizing optical fiber communication systems using end-to-end deep learning techniques.