Pneumonia remains the leading infectious cause of death in children under the age of five, killing about 700,000 children each year and affecting 7% of the world’s population. X-ray images of lung become the key to the diagnosis of this disease, skilled doctors in the diagnosis of a certain degree of subjectivity, if the use of computer-aided medical diagnosis to automatically detect lung abnormalities, will improve the accuracy of diagnosis. This research aims to introduce a deep learning technology based on the combination of Xception neural network and long-term short-term memory (LSTM), which can realize automatic diagnosis of patients with pneumonia in X-ray images. First, the model uses the Xception network to extract the deep features of the data, passes the extracted features to the LSTM, and then the LSTM detects the extracted features, and finally selects the most needed features. Secondly, in the training set samples, the traditional cross-entropy loss cannot more balance the mismatch between categories. Therefore, this research combines Pearson’s feature selection ideas, fusion of the correlation between the two loss functions, and optimizes the problem. The experimental results show that the accuracy rate of this paper is 96%, the receiver operator characteristic curve accuracy rate is 99%, the precision rate is 98%, the recall rate is 91%, and the F1 score accuracy rate is 94%. Compared with the existing technical methods, the research has achieved expected results on the currently available datasets. And assist doctors to provide higher reliability in the classification task of childhood pneumonia.