Runtime-Adaptive Gradient Bucketing via Graph Tracing
Replace static parameter-to-bucket mappings in PyTorch DDP with a warm-up tracing phase that records actual autograd execution order.
Source paper
Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, Soumith Chintala
Generated by AI. The paper’s authors did not write these.
Replace static parameter-to-bucket mappings in PyTorch DDP with a warm-up tracing phase that records actual autograd execution order.
Integrate on-the-fly gradient compression into the AllReduce primitive to mitigate communication bottlenecks in distributed training.
Deploy an auto-tuning feedback loop with a PID controller to dynamically adjust gradient bucket sizes at runtime based on observed network performance.
Construct transient virtual bucket maps for layer-skipping architectures to prevent the transmission of zeros for inactive parameters.
This paper details the design and implementation of PyTorch's DistributedDataParallel (DDP) module, focusing on maximizing training throughput via gradient bucketing, communication-computation overlap, and skipping synchronization. The authors evaluate DDP on varying scales (up to 256 GPUs) and backends (NCCL, Gloo), identifying communication latency as the primary bottleneck and proposing architectural optimizations to maintain mathematical equivalence with local training while achieving near-linear scalability.
The system employs a single-program multiple-data (SPMD) architecture where the model is replicated across devices. The core technique involves 'Gradient Bucketing,' where small gradient tensors are coalesced into larger buckets based on the reverse order of parameter registration (mimicking backward pass order) to saturate communication bandwidth. An autograd hook triggers asynchronous AllReduce operations on these buckets as soon as they are ready, allowing communication to overlap with the ongoing backward pass computation.