Quality, Readability, Understandability, and Actionability of AI Chatbots’ Responses to Dental Patient Questions: A Comparative Evaluation

Abstract

Background/Objectives: Artificial intelligence (AI) chatbots are increasingly used to obtain oral health information, yet prior dental studies have already evaluated overlapping domains such as accuracy, quality, readability, understandability, and actionability. This study applied and extended a multidimensional evaluation approach to compare AI-generated answers to dental patient questions, with emphasis on Turkish-language readability, standard PEMAT-P applicability, effect-size and uncertainty reporting, and inter-rater reliability. Methods: Thirty-three AI-assisted, expert-curated questions were submitted to ChatGPT 5.2, Gemini, DeepSeek V3, and Grok 3, yielding 132 responses. Three dental evaluators independently assessed clinical accuracy, Global Quality Score (GQS), understandability (PEMAT-U), and actionability (PEMAT-A). PEMAT-P scores were calculated using the standard item structure and item-specific applicability rules, with items coded as not applicable (N/A) excluded from the denominator. Readability was calculated using a deterministic, rule-based implementation of the Ateşman formula. Model comparisons used the Friedman test, Bonferroni-adjusted Wilcoxon signed-rank tests, effect sizes, bootstrap confidence intervals, and two-way random-effects intraclass correlation coefficients. An exploratory Pareto analysis assessed multidimensional dominance without imposing arbitrary weights. Results: Significant model differences were observed for all five dimensions (all p ≤ 0.002). Gemini had the highest mean GQS, clinical accuracy, PEMAT-U, and PEMAT-A scores, but its Ateşman readability score was significantly lower than those of ChatGPT, DeepSeek, and Grok. No significant readability differences were observed among the latter three models. Average-measure reliability was poor for GQS and clinical accuracy and moderate for PEMAT-U and PEMAT-A. No model dominated another across all five dimensions. Conclusions: The chatbots demonstrated model-specific strengths rather than uniform superiority. Their performance should be interpreted multidimensionally, together with effect magnitude, uncertainty, and inter-rater reliability, rather than through a single overall ranking.

The authors' abstract, as published at the source. Diagnostics, 2026 · DOI ↗