All posts
ParallelismJune 12, 20267 min read
Multi-GPU layouts that actually scale
Tensor, pipeline, and expert parallelism trade communication for memory in ways that depend on your interconnect.
Interconnect decides the answer
The layout that wins inside a single high-bandwidth node loses across nodes. Any recommendation that does not name the topology is incomplete.
Communication overlaps or it dominates
Collectives that are not overlapped with compute set a floor on latency no kernel improvement can lift.
Layout is a per-workload decision
Short-prompt, high-concurrency serving and long-context batch generation want different splits on identical hardware.