Beyond Answer Accuracy: An Audit-First Framework for Claim-Level Faithfulness Evaluation of Multimodal Retrieval-Augmented Generation in Chest X-Ray Question Answering

Abstract

Radiology vision–language models can generate fluent chest X-ray answers that contain unsupported assertions when individual claims are not verifiably grounded in retrieved evidence. Answer-level similarity and coarse label agreement cannot detect such failures because they do not preserve the evidence chain behind each output. This paper proposes an audit-first evaluation framework that logs, for each query, the retrieval trace, a per-claim citation map, schema enforcement actions, and system telemetry. Claim-level faithfulness is measured by a Claim Support Score and a Hallucination Rate that sum to one by construction over the claims actually delivered. On a de-identified, credentialed-access chest radiograph dataset with three radiology vision–language model backbones, reference-aligned Claim Support Score without retrieval ranges from 0.605 to 0.655, and evidence-linked Claim Support Score with retrieval ranges from 0.724 to 0.760. These use different reference standards and are reported as complementary measurements, not compared directly. With retrieval, Hallucination Rate falls from 0.332 to 0.388 in the lowest retrieval-support band to 0.148 to 0.210 in the highest, suggesting evidence coverage as a candidate lever. The automated support labels have not yet been validated by independent clinical expert annotation, so findings are evaluation results rather than clinical safety evidence.

The authors' abstract, as published at the source. Applied Sciences, 2026 · DOI ↗