Tensor Parallelism vs Pipeline Parallelism

Plain-English answer

Plain-English answer guidance: Use tensor parallelism to split each layer across fast, closely connected GPUs; use pipeline parallelism when layers must be distributed across devices/nodes and bubble latency is acceptable. For a model that fits one GPU, avoid both unless concurrency testing proves a need.

Why it matters

Why it matters guidance: Use tensor parallel inside a high-bandwidth multi-GPU server; pipeline parallel across heterogeneous or capacity-driven layouts; replicas for models that already fit. KV cache may be replicated or partitioned differently by the runtime; include long-context memory in the topology plan.

Real StackBench examples

Common misconception

Common misconception guidance: Communication overhead, synchronization, and workload sensitivity make naive scaling unreliable. Report interconnect and topology with every multi-GPU benchmark.

Measured / example table

Decision rule

Decision rule guidance: Prefer one GPU, then independent replicas, then tensor parallel on fast interconnect, and pipeline parallel only for a model that otherwise cannot be placed efficiently.

Related benchmarks

Related benchmarks guidance: Tensor parallel often helps large batches on NVLink-class systems but can hurt single-stream latency over slow links. Pipeline parallel generally needs enough in-flight work to stay busy. Quantize the model, choose a higher-memory GPU, use expert parallelism for MoE, or serve smaller replicas.

Planner / Explore CTA

Compare this recommendation against your model, context, concurrency, latency, and budget in the ComputeSage Planner.

Sources / methodology

Recommendations combine official specifications, public model/runtime documentation, adjacent benchmark observations, and clearly labeled engineering estimates. Review the ComputeSage evidence methodology.

• vLLM installation and hardware documentation
• llama.cpp multi-GPU documentation
• StackBench controlled lab approved source radixark-qwen38-dflash2-sglang-dgx-owner-20260821