Transformer Architectures for Respiratory Sound Analysis and Multimodal Diagnosis

Abstract

Abstract Breath sound analysis is one of the diagnostic approaches for various lung diseases. However, traditional auscultation is subjective and dependent on the physician’s experience. Recent studies have demonstrated that computer-assisted methods analyzing these sounds can significantly improve the accuracy of diagnosis of the patient’s condition. However, the absence of a common basis for fair comparison, limited use of patient metadata, and lack of interpretability make these approaches difficult to adopt in clinical practice. In the paper, we propose to utilize the benefits of the recent state-of-the-art sound classification models, and a multimodal approach for respiratory disease screening and diagnostics. To achieve this goal we use the Audio Spectrogram Transformer (AST) model and a vision-language model (VLM) Moondream2. We compare them with our previous work on DenseNet201 Convolutional Neural Network (CNN). Proposed methodology allows a fair comparison of different computer assisted methods for respiratory disease screening and diagnostics. The training and evaluation of the models has been conducted using the same dataset containing respiratory sound recordings and patient information. The obtained results demonstrate that these transformer-based models can serve as a basis for the development of computer-assisted methods for asthma diagnosis, as well as for continuous and remote monitoring of patient condition. In addition our approach enables interpretability of the model performance. The achieved performance is much better than the best results of the physician’s auscultation. MSC Classification: 92C55 , 68T07 , 68T10

The authors' abstract, as published at the source. Operations Research Forum, 2026 · DOI ↗