Beyond question and answer: Practical applications of multimodal and multifunctional large language models in ophthalmology with a scoping review and future directions

Abstract

Multimodal large language models (MLLMs) connect language understanding with image-based reasoning and are increasingly explored as artificial intelligence systems for ophthalmic image analysis, workflow support, research prototyping, and education. This scoping review synthesizes 21 peer-reviewed studies that moved beyond text-only question answering to evaluate visual question answering (VQA), domain-adapted or fine-tuned ophthalmic models, retrieval-augmented generation (RAG), and low- or no-code tools for analytics and prototype development. Across studies, the evidence suggests early feasibility for image-conditioned explanations, triage support, documentation assistance, structured data extraction, and rapid tool creation. However, the evidence base remains heterogeneous and preliminary, with mixed diagnostic performance, variable prompts and model versions, limited external validation, and persistent risks including reasoning errors, hallucinations, session variability, privacy concerns, and security threats. To illustrate potential workflows not yet fully covered by the literature, we also provide clearly labeled author-built demonstrations, which are non-evidentiary, for fundus and optical coherence tomography (OCT)-based VQA, optical character recognition (OCR) tabulation, code-free analytics, automated software programming, and web-based risk calculators. Future priorities include extending image-conditioned reasoning to additional ophthalmic modalities, strengthening RAG with curated corpora and provenance, evaluating few-shot and in-context learning for rare conditions, and using controlled generative imaging for education and data augmentation. Overall, MLLMs may support supervised ophthalmic artificial intelligence applications in selected clinical, research, and educational workflows. Clinical deployment will require external validation, prospective human-in-the-loop evaluation, standards-based workflow integration, privacy-preserving infrastructure, audit logging, cybersecurity, and post-deployment monitoring.

The authors' abstract, as published at the source. Engineering Applications of Artificial Intelligence, 2026 · DOI ↗