Evaluating OpenMP Offloading for Intra-Node Multi-GPU Programming Across NVIDIA, AMD, and Intel Architectures: A 3D Heat Transfer Case Study

Abstract

Currently, many of the leading supercomputers are equipped with GPUs from manufacturers such as NVIDIA, AMD, or Intel, which provide substantial parallelism and high throughput. It is common for a single compute node to host multiple GPUs (typically four or more); we refer to this as an intra-node multi-GPU setting. Therefore, effectively leveraging all these GPUs within a single compute node is essential for applications in scientific and engineering domains. However, several factors must be considered before utilizing these GPUs for scientific computing, including the implementation of data communication, the programming models available for use across these GPUs, and the level of performance that can be achieved with a single codebase across different GPU architectures and configurations within a single compute node. OpenMP Offloading is a prominent directive-based programming model that can be executed on all three GPU types: NVIDIA, AMD, and Intel. In this research, we present an analysis of the benefits and performance challenges of using OpenMP Offloading to solve the 3D heat equation, which involves interior computation, halo computation, and halo exchange. For additional comparison and a scalability study, we also consider the conjugate gradient (CG) method. We investigate how the performance of OpenMP Offloading varies in relation to native GPU programming models—CUDA for NVIDIA, HIP for AMD, and SYCL for Intel. Furthermore, we demonstrate that OpenMP Offloading can achieve performance improvements of approximately 2× for 2 GPUs and around 4× for 4 GPUs when compared to single-GPU OpenMP Offloading implementations across all three GPU types. This analysis is conducted systematically through various OpenMP Offloading implementations that utilize different low-level APIs for memory allocation and different memory transfer options (synchronous, asynchronous, and peer-to-peer), and these are compared against native implementations in CUDA (NVIDIA), HIP (AMD), and SYCL (Intel).

The authors' abstract, as published at the source. Electronics, 2026 · DOI ↗