Cloud‐Native and Distributed Systems for Efficient and Scalable Large Language Models—A Research Agenda

Abstract

ABSTRACT Objective Large Language Models (LLMs) are rapidly transforming artificial intelligence applications, but their unprecedented computational, memory, communication, and data‐management demands pose significant challenges to existing computing infrastructures. Traditional cloud and distributed systems were not designed for the scale, dynamic execution patterns, heterogeneous hardware requirements, and service‐level demands of modern LLM workloads. This paper aims to examine how cloud‐native and distributed systems can evolve to support efficient, scalable, reliable, and sustainable LLM training and inference. Methods We conduct a structured analysis of representative research advances and practical developments in distributed LLM training, inference serving, cloud‐native orchestration, resource management, heterogeneous computing, and edge‐cloud intelligence. We organize the analysis around major system challenges, including computation, distributed execution and system software, resource management and serving, security and privacy, data management, and standardization. We further examine representative resource‐management strategies and emerging paradigms, including serverless inference, federated and decentralized learning, trustworthy distributed AI, and heterogeneous computing with emerging accelerators. Results Our analysis reveals that supporting LLMs at scale requires a transition from general‐purpose cloud infrastructure toward LLM‐aware, cross‐layer, and adaptive systems. Existing approaches have improved distributed execution, elastic resource management, multi‐tenant serving, and hardware utilization, but important gaps remain in cross‐layer coordination, workload‐aware orchestration, data and context management, interoperability, security, and system‐level optimization. Based on these gaps, we develop a research agenda covering future directions in sparse and heterogeneous computation, LLM‐enhanced system software, prompt and context management, autonomous cloud management, decentralized intelligence, edge‐cloud collaboration, personalized LLM services, and emerging computing paradigms. Conclusion Cloud‐native and distributed systems will play a central role in enabling the next generation of LLM services. Future infrastructures should evolve toward more intelligent, elastic, interoperable, secure, and heterogeneous platforms that jointly optimize computation, memory, communication, data, and service objectives. Continued advances in system design, standardization, and cross‐disciplinary collaboration will be essential for realizing scalable and sustainable LLM infrastructures.

The authors' abstract, as published at the source. Software Practice and Experience, 2026 · DOI ↗