Running vLLM across multiple GPUs with tensor parallelism means passing the tensor-parallel-size argument equal to the number of GPUs each model replica should be sharded across, for example tensor-parallel-size 4 to split a 70B model's layers across four GPUs so each holds roughly a quarter of the weights and computes its share of every forward pass. vLLM handles the all-reduce communication between shards automatically using NCCL, so fast interconnect matters: NVLink within a node gives dramatically better scaling than PCIe alone, and multi-node tensor parallelism additionally needs InfiniBand or a comparably low-latency network to avoid becoming communication-bound. Tensor parallelism is generally chosen to fit a model's weights and KV cache into available GPU memory, while pipeline parallelism, set with pipeline-parallel-size, is used to scale across nodes without paying tensor parallelism's per-token communication cost. A common pattern is tensor-parallel-size matching GPUs per node and pipeline-parallel-size matching the number of nodes for very large models like DeepSeek R1. Sizing correctly requires knowing model weight size at your chosen precision plus KV cache headroom for expected context length and concurrency. Nanobase AI, a Silicon Valley infrastructure engineering company, sizes and configures these tensor-parallel topologies for customer GPU clusters.
Starting from memory, not from a guessed GPU count
The most reliable way to size tensor-parallel-size is to work backward from memory, not to guess a GPU count and hope it fits. A model's weights at a given precision are a fixed number: a 70B parameter model needs roughly 140 GB in FP16, about 70 GB in FP8, and roughly 38 GB in INT4. Add KV cache headroom on top, which typically needs an additional 20 to 50 percent of the weight footprint depending on target context length and concurrent request count, since every active sequence holds its own growing cache.
Divide that total memory requirement by the memory available per GPU, then round up to the nearest number that evenly shards the model's attention heads, and that is your starting tensor-parallel-size, not an arbitrary power of two chosen out of habit.
Working through the arithmetic
| GPU | Memory | 70B FP8 weights (~70 GB) + KV headroom | Minimum GPUs needed |
|---|---|---|---|
| H100 | 80 GB HBM3, 3.35 TB/s | Needs roughly 90–105 GB total | 2 (with headroom to spare) |
| H200 | 141 GB HBM3e, 4.8 TB/s | Needs roughly 90–105 GB total | 1, or 2 for more concurrency headroom |
| B200 | ~180 GB HBM3e, 8 TB/s | Needs roughly 90–105 GB total | 1, with substantial concurrency headroom |
| A100 | 80 GB, 2 TB/s | Needs roughly 90–105 GB total | 2 |
These figures are illustrative starting points; always validate against your actual quantization choice, target context length, and expected concurrent request count before finalizing a deployment size, since KV cache headroom scales with both context length and concurrency, not weights alone.
Why interconnect determines whether the GPU count you calculated actually delivers throughput
Tensor parallelism requires an all-reduce communication step after every layer, since each GPU only holds a shard of the computation and the results must be combined before the next layer runs. Within a single node, NVLink provides the bandwidth to make this communication overhead small relative to compute time. Across nodes, that same all-reduce traffic has to cross a network, and without InfiniBand or an equivalently low-latency, high-bandwidth interconnect, multi-node tensor parallelism becomes communication-bound: adding more GPUs stops improving throughput because they spend more time waiting on network transfers than computing.
Choosing between tensor and pipeline parallelism
Tensor parallelism shards each layer across GPUs and pays a communication cost every layer, which is why it is best kept within a single NVLink-connected node. Pipeline parallelism instead assigns different layers to different GPUs or nodes, communicating only at pipeline stage boundaries, which tolerates slower interconnects better but can introduce pipeline bubble inefficiency if not tuned carefully. The common production pattern for very large models, including reasoning models like DeepSeek R1 that exceed a single node's capacity even at reduced precision, is tensor-parallel-size matching the GPU count per node and pipeline-parallel-size matching the number of nodes.
A sizing checklist before committing hardware
- Determine model weight size at your chosen precision (FP16, FP8, or INT4).
- Estimate KV cache headroom based on target maximum context length and expected peak concurrent requests.
- Divide total memory need by per-GPU memory to get a minimum tensor-parallel-size.
- Confirm that count evenly divides the model's attention head count, since uneven sharding is not supported.
- Check whether that GPU count fits within one NVLink-connected node; if not, plan for InfiniBand and consider pipeline parallelism across nodes instead of pure tensor parallelism.
- Load test the resulting configuration under realistic concurrency before finalizing procurement, since headroom estimates are starting points, not guarantees.
For the higher-level GPU count question across model sizes, see how many GPUs for 70B and 405B models, and for the orchestration layer that runs this configuration in production, see deploying vLLM on Kubernetes.
Frequently asked questions
Does more GPUs always mean higher throughput with tensor parallelism?
No, past a certain point communication overhead between shards outweighs the benefit of more parallel compute, especially across nodes without a fast interconnect; there is a diminishing-returns point specific to each model and network topology.
Can tensor-parallel-size be an odd number?
It must evenly divide the model's number of attention heads, so valid values depend on the specific model architecture; most large open-weight models are designed with head counts divisible by common values like 2, 4, and 8.
Is pipeline parallelism slower than tensor parallelism?
Not inherently, but it introduces different overhead in the form of pipeline bubbles, periods where later stages wait for earlier ones; it tolerates slower interconnects better, which makes it the more practical choice across nodes without InfiniBand.
How much KV cache headroom should we plan for a long-context deployment?
Long context windows and high concurrency both grow KV cache demand substantially, often well beyond the 20 to 50 percent baseline; for very long context or high-concurrency use cases, model the KV cache size directly per sequence rather than relying on a general rule of thumb.
How Nanobase AI helps
Nanobase AI sizes tensor-parallel and pipeline-parallel topologies for customer GPU clusters based on actual model, quantization, and traffic requirements, then validates the configuration under load before hardware procurement is finalized. This sizing work is core to Nanobase AI's NVIDIA GPU infrastructure practice.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.