En yeni nesil büyük dil modellerinin endodonti uzmanlık sınavı sorularındaki performansı: karşılaştırmalı kesitsel bir çalışma

Özet (abstract)

Abstract Background This study aimed to compare the performance of Gemini 3 Pro, Gemini 3 Flash, ChatGPT 5.2, and DeepSeek V3.2 on the Endodontics section of the Turkish Dental Specialty Exam (DUS), with particular attention to question source, type, and endodontic subtopic. Methods The dataset comprised 187 endodontic questions from 2012–2025, stratified into internet-sourced from 2012–2021 ( n = 127) and textbook-sourced from 2022–2025 ( n = 60). Questions were further categorized by type as case analysis, literature knowledge, combination or visual analysis, and by 10 endodontic subtopics. Following a standardized zero-shot prompting protocol (public web interfaces, default settings, single run per item), two independent endodontic assessors scored responses on a blinded 0–3 rubric, and then the two evaluator scores were averaged for each response. Overall model performance was expressed as a percentage of the maximum possible rubric score (100%). Statistical analysis employed the Friedman test, Wilcoxon Signed-Rank tests with Bonferroni correction, Mann–Whitney U and Kruskal–Wallis tests. Results All reported percentages represent normalized rubric-based performance scores, evaluating both answer selection and reasoning quality, rather than simple correct/incorrect accuracy rates. High inter-rater agreement was achieved, with quadratic weighted kappa values ranging from 0.954 to 0.998. In overall performance, Gemini 3 Pro (90.37%) and Gemini 3 Flash (88.32%) significantly outperformed ChatGPT 5.2 (78.25%) and DeepSeek V3.2 (62.12%) (adjusted p ≤ 0.001; Kendall’s W = 0.214); corresponding answer accuracies were 94.1%, 93.0%, 86.6%, and 73.3%. No significant difference was found between the Gemini models (adjusted p = 1.000). No significant difference was observed between internet-sourced and textbook-sourced questions within any model. In literature knowledge questions, combination questions, and selected endodontic subtopics, the Gemini models generally showed higher rubric-based performance than the other models. Conclusions The findings indicate high rubric-based performance of the latest-generation LLMs, particularly the Gemini 3 series, on Turkish endodontics residency examination questions. No significant difference in performance was observed between internet-sourced and textbook-sourced questions; however, contamination of the data in both groups cannot be excluded. These models may serve as adjunctive tools in endodontic education; however, examination performance does not establish clinical diagnostic performance, and their outputs require expert validation.

Yazarların özeti; kaynağından alınmıştır. BMC Oral Health, 2026 · DOI ↗