Dynamic Load Balancing and Communication Optimization for Data-Parallel Training on Heterogeneous Clusters

Abstract

The growing adoption of artificial intelligence in space communications has imposed increasingly demanding efficiency requirements on deep learning training for tasks such as intelligent signal processing and resource allocation. However, performance disparities and workload fluctuations across heterogeneous nodes cause synchronization delays, while gradient aggregation and redundant parameter updates further increase training overhead. This paper proposes a data-parallel training method that combines dynamic load balancing with communication optimization. The Iteration-Level Dynamic Batch Adjustment algorithm (IDAT) estimates node processing rates online and dynamically assigns local batch sizes while maintaining a fixed global batch size, with a time-variation threshold, integer adjustment, and GPU memory constraints used to control overhead. The communication optimization algorithm (MRA) integrates Reduce-Scatter, sharded parameter updates, and All-Gather into a unified workflow to reduce communication startup and redundant update costs. Experiments using ResNet-50, AlexNet, and VGG-16 on a cluster of three servers and five NVIDIA V100-series GPUs show that IDAT achieves a 1.03×–1.04× speedup over the baseline, while the combined IDAT–MRA scheme achieves a 1.41×–1.82× speedup over the native PyTorch DDP 1.12.1 and Horovod 0.28.1 configurations, with an absolute accuracy difference of no more than 0.02 percentage points. These results demonstrate the effectiveness of the proposed framework in improving synchronous data-parallel training efficiency on the evaluated heterogeneous cluster and indicate its potential applicability to the training of intelligent models for space communication systems.

The authors' abstract, as published at the source. Aerospace, 2026 · DOI ↗