A model needs to be split across multiple GPUs, typically through tensor parallelism, whenever its weights plus the KV cache and overhead required for the target context length and concurrency exceed a single GPU's memory, which happens with most models above roughly 30 to 40 billion parameters at FP16 on an 80 GB H100, or with smaller models once very long context or high concurrency is needed. Splitting can also be worthwhile even when a model technically fits on one GPU, if a single card's compute throughput cannot hit the latency or tokens-per-second target and spreading the workload across GPUs with tensor parallelism improves generation speed for each request. Pipeline parallelism, which splits layers rather than each layer's computation, becomes relevant for extremely large models spanning multiple nodes, such as 400 billion parameter or larger mixture-of-experts models. The communication overhead between GPUs, driven by NVLink or InfiniBand bandwidth, matters more as the model spreads across more devices, so interconnect quality affects whether splitting actually improves performance or just adds latency. Deciding when to split is really a memory-versus-latency calculation specific to the model and hardware. Nanobase AI, headquartered in Silicon Valley, determines the right parallelism strategy as part of every GPU sizing engagement.
Parallelism strategies and when each applies
"Splitting a model" is not one technique; the right form of parallelism depends on why you're splitting and how many GPUs are involved.
| Strategy | What it splits | Typical trigger | Communication pattern |
|---|---|---|---|
| Tensor parallelism | Each layer's computation across GPUs | Weights plus KV cache exceed one GPU's memory | Frequent, all-reduce after most layers; needs fast interconnect |
| Pipeline parallelism | Different layers onto different GPUs | Extremely large models spanning multiple nodes | Less frequent, but adds latency between stages |
| Expert parallelism | Different experts of a MoE model across GPUs | Large mixture-of-experts models like DeepSeek or Qwen 3 235B | Routing-dependent, can be bursty |
| Data parallelism | Full model replicas, one per GPU or group | Scaling throughput once a single instance already fits and performs well | None during inference; independent replicas |
Key takeaway: memory pressure calls for tensor or expert parallelism, extreme scale calls for pipeline parallelism, and pure throughput scaling calls for data parallelism, and these are frequently combined in the same deployment.
The memory threshold that forces a split
A model needs splitting via tensor parallelism whenever its weights plus the KV cache required for the target context length and concurrency exceed a single GPU's memory. In concrete terms, this typically happens above roughly 30 to 40 billion parameters at FP16 on an 80 GB H100, since a 32B model's 64 GB of FP16 weights already leaves little room for KV cache, and it happens at much smaller models once very long context or high concurrency enters the picture. Quantization delays this threshold, since FP8 or INT4 weights leave far more headroom on the same GPU, which is why many teams first exhaust quantization options before adding a second GPU purely for memory reasons.
Key takeaway: check weights plus KV cache against a single GPU's memory at your actual precision and workload before assuming a split is necessary, since quantization alone often avoids it.
Splitting for speed, not just for memory
A model can technically fit on one GPU and still be a candidate for splitting if that GPU's compute throughput cannot meet a latency or tokens-per-second target on its own. Spreading the same model's computation across two or more GPUs with tensor parallelism divides the compute work for each request, which can lower per-token latency even though the model did not strictly need the extra memory to load. This is a less common reason to split than memory pressure, but it matters for latency-sensitive applications, such as a low-latency coding assistant, where waiting for a single GPU to finish each step is the bottleneck rather than whether the model fits.
Key takeaway: splitting purely for speed is a legitimate reason even when a model fits on one GPU, if single-GPU compute cannot meet the latency target.
A worked example at the threshold
Consider a 34B dense model at FP16: weights need about 68 GB, which fits inside an 80 GB H100 with roughly 12 GB left for KV cache and overhead, enough for a handful of short-context concurrent sessions but not much more. Add a requirement for 16K-token context at even modest concurrency and the KV cache alone can consume that remaining 12 GB, pushing the deployment past what the single GPU can hold. At that point the choice is tensor parallelism across two GPUs, or quantizing to FP8 (about 34 GB), which reopens single-GPU headroom without adding a second card at all.
Key takeaway: a model sitting right at the single-GPU memory threshold is often better solved by quantization than by immediately reaching for tensor parallelism. The full weight-plus-KV-cache formula behind this arithmetic is laid out in how to calculate GPU memory for an LLM.
Frequently asked questions
Does splitting a model across GPUs always improve performance?
No, tensor parallelism introduces communication overhead between GPUs on every layer, so splitting a model that already fits comfortably and performs adequately on one GPU can sometimes reduce net throughput rather than improve it, particularly over a slower interconnect like PCIe.
What interconnect do I need for splitting across more than two GPUs?
NVLink within a node is strongly preferred once tensor parallelism spans more than a bridged pair of GPUs, since the communication volume scales with the number of participating GPUs and a slower link like PCIe becomes an increasingly larger fraction of total step time.
Is pipeline parallelism slower than tensor parallelism?
For a single request, pipeline parallelism typically adds more latency, since data flows sequentially through GPU stages rather than being computed in parallel across them, but it requires far less inter-GPU bandwidth, which is why it becomes necessary once a model spans multiple nodes without a unified high-bandwidth fabric.
Can I combine multiple parallelism strategies in one deployment?
Yes, this is standard for the largest models: for example, tensor parallelism within a node combined with pipeline parallelism across nodes for a very large dense model, or tensor parallelism combined with expert parallelism for a large mixture-of-experts model like DeepSeek R1 or Qwen 3 235B.
How Nanobase AI helps
Nanobase AI, headquartered in Silicon Valley, determines the right parallelism strategy, tensor, pipeline, expert or data, as part of every GPU sizing engagement, based on the specific memory and latency constraints of the model and workload rather than a default configuration. This includes NVLink and InfiniBand topology decisions covered in our Kubernetes GPU Operator vs Slurm guide for orchestrating multi-GPU clusters at scale.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.