LoRDO: Distributed Low-Rank Optimization with Infrequent Communication

LoRDO performs distributed low-rank optimization with infrequent communication, matching full-rank quality at a fraction of the synchronisation cost.

Read the paper here.

Abstract:

Distributed training of foundation models via DDP is limited by interconnect bandwidth. While infrequent communication strategies reduce synchronization frequency, they remain bottlenecked by the memory and communication requirements of optimizer states. Low-rank optimizers can alleviate these constraints; however, in the local-update regime, workers lack access to the full-batch gradients required to compute low-rank projections, which degrades performance. We propose LoRDO, a principled framework unifying low-rank optimization with infrequent synchronization. We first demonstrate that, while global projections based on pseudo-gradients are theoretically superior, they permanently restrict the optimization trajectory to a low-rank subspace. To restore subspace exploration, we introduce a full-rank quasi-hyperbolic update. LoRDO achieves near-parity with low-rank DDP in language modeling and downstream tasks at model scales of 125M–720M, while reducing communication by ≈10×. Finally, we show that LoRDO improves performance even more in very low-memory settings with small rank/batch size.

Recommended citation: Andrej Jovanović, Alex Iacob, Mher Safaryan, Ionut-Vlad Modoranu, Lorenzo Sani, William F. Shen, Xinchi Qiu, Dan Alistarh, & Nicholas D. Lane. (2026). LoRDO: Distributed Low-Rank Optimization with Infrequent Communication. In International Conference on Machine Learning (ICML 2026).