Skip to content

Source paper

PyTorch Distributed: Experiences on Accelerating Data Parallel Training

Shen Li, Yanli Zhao, Rohan Varma and 8 more

Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, Soumith Chintala

arXiv 2006.15704 · 20205 upvotes on Hugging Face

Research ideas from this paper (4)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper details the design and implementation of PyTorch's DistributedDataParallel (DDP) module, focusing on maximizing training throughput via gradient bucketing, communication-computation overlap, and skipping synchronization. The authors evaluate DDP on varying scales (up to 256 GPUs) and backends (NCCL, Gloo), identifying communication latency as the primary bottleneck and proposing architectural optimizations to maintain mathematical equivalence with local training while achieving near-linear scalability.

How it works (AI summary)

The system employs a single-program multiple-data (SPMD) architecture where the model is replicated across devices. The core technique involves 'Gradient Bucketing,' where small gradient tensors are coalesced into larger buckets based on the reverse order of parameter registration (mimicking backward pass order) to saturate communication bandwidth. An autograd hook triggers asynchronous AllReduce operations on these buckets as soon as they are ready, allowing communication to overlap with the ongoing backward pass computation.