Information quality, understandability, actionability and readability of large language models for HIV health education: A cross‐platform evaluation

Abstract

Abstract Objectives Large language models (LLMs) are increasingly used to generate health education materials, but their suitability for patient‐facing human immunodeficiency virus (HIV) education remains insufficiently evaluated. We assessed the information quality, understandability, actionability, readability, text structure and source transparency of HIV patient education materials generated by six LLMs or artificial intelligence platforms. Methods In this cross‐sectional study, 100 common HIV patient education questions across 10 domains were submitted once in English to ChatGPT 5.4‐Thinking, DeepSeek‐V3.2, Doubao, ERNIE Bot 5.0 Preview, Ant Afu and Gemini 3.1 Pro, yielding 600 responses. Responses were evaluated using Patient Education Materials Assessment Tool for Printable Materials (PEMAT‐P), EQ IP‐36, DISCERN, Global Quality Score (GQS), artificial intelligence response quality metric (AIRM), Journal of the American Medical Association (JAMA) benchmarks and seven readability indices. Platform comparisons used paired Friedman tests with Wilcoxon–Holm post hoc analyses; topic analyses used question‐level aggregation followed by Kruskal–Wallis and Dunn–Holm comparisons. Results Significant family‐adjusted differences were observed across platforms for all prespecified text‐structure, quality/understandability/actionability and readability outcomes. At the topic level, five of six text‐structure outcomes, PEMAT‐P Understandability, PEMAT‐P Actionability, the exploratory PEMAT‐P composite score and all seven readability indices differed significantly after family adjustment, whereas EQ IP‐36, DISCERN, GQS, AIRM and JAMA did not. Overall actionability was limited and readability demands were high. Conclusions LLM‐generated HIV patient education materials showed substantial cross‐platform and topic‐level heterogeneity. These findings support the use of LLMs as expert‐reviewed drafting tools rather than independent patient education sources. Direct factual accuracy, guideline concordance, hallucination and clinical safety were not assessed.

The authors' abstract, as published at the source. HIV Medicine, 2026 · DOI ↗