Which GPU do we need to fine-tune a 70B model?

For a 70B parameter model the right GPU choice depends on whether you use LoRA-style adapters or full fine-tuning, since the two have very different memory profiles. With QLoRA, a single H100 80 GB or an H200 141 GB is normally enough because the frozen base weights sit in 4-bit precision at roughly 38 GB and only the small adapter matrices and their optimizer states need full-precision headroom. Full fine-tuning is far heavier because Adam-style optimizers store parameters, gradients and two optimizer moments, pushing memory needs for a 70B model well past a terabyte once activations are included, which typically requires eight or more H100 or H200 GPUs sharded with DeepSpeed ZeRO-3 or FSDP. An RTX PRO 6000 96 GB can also handle QLoRA fine-tuning of a 70B model at smaller batch sizes, making it a cost-effective option for teams that already own workstation-class GPUs. Multi-GPU setups also benefit from InfiniBand interconnects when full fine-tuning spans multiple nodes. Nanobase AI, an NVIDIA Inception Program member, sizes and installs the exact GPU configuration a 70B fine-tuning workload needs.

Two completely different hardware plans for one model size

The phrase "fine-tune a 70B model" hides two very different infrastructure requirements depending on method, and pricing out the wrong one is the most common early mistake. QLoRA needs roughly 40 to 48 GB of VRAM total because the frozen base model sits in 4-bit precision, while full fine-tuning needs to store 16-bit or 32-bit parameters, gradients, and Adam optimizer moments for all 70 billion weights, which pushes total memory well past a terabyte before activations are even added. These are not two points on the same scale; they require entirely different GPU counts and, for full fine-tuning, a distributed training framework.

Deciding LoRA versus full fine-tuning first is what determines the GPU sizing question, not the other way around.

Sizing table by method and hardware option

A single H100 or H200 covers the great majority of enterprise 70B fine-tuning projects once QLoRA or full-precision LoRA is the chosen method.

Why full fine-tuning needs InfiniBand, not just more GPUs

Adding GPUs to a full fine-tuning job is not simply a matter of buying eight cards. DeepSpeed ZeRO-3 and FSDP shard model parameters, gradients, and optimizer states across GPUs, which means every training step requires frequent communication between devices to reassemble the pieces needed for computation. On standard Ethernet, this communication overhead can dominate step time and erase much of the benefit of adding hardware, which is why multi-node full fine-tuning clusters are typically built with InfiniBand or a comparable high-bandwidth, low-latency interconnect between GPUs.

Teams that provision eight H100s without matching interconnect often find training slower than expected relative to the raw compute added. GPU count alone does not determine full fine-tuning throughput; interconnect bandwidth is an equal partner in the sizing decision.

Checking whether a workstation GPU is enough

For QLoRA specifically, an RTX PRO 6000 with 96 GB of GDDR7 memory is genuinely competitive with data center hardware, since the workload is memory-bound rather than requiring the ECC memory or multi-GPU NVLink that data center GPUs add. Teams already running GPU workstations for other work, such as AI Mobile Test Lab infrastructure or internal development, can often repurpose that hardware for QLoRA fine-tuning of a 70B model without a separate GPU procurement, a point covered further in fine-tuning on a single RTX PRO 6000.

Workstation-class GPUs are a legitimate, cost-effective choice for QLoRA at 70B scale, not just a fallback for smaller models.

Frequently asked questions

Can two H100s split a 70B QLoRA job for faster training?

Yes, though it is not required for memory reasons the way full fine-tuning requires it. Splitting across two GPUs with data parallelism mainly speeds up training by processing more examples per step, rather than being necessary to fit the model in memory.

Does H200's larger memory change the calculation?

The H200's 141 GB of HBM3e gives more headroom for longer sequence lengths or larger batch sizes during QLoRA fine-tuning of a 70B model, but it does not change the fundamental method choice between QLoRA and full fine-tuning.

Is cloud rental or owned hardware better for a single 70B fine-tuning project?

For a single project, rented cloud GPU capacity usually avoids the capital cost and lead time of hardware procurement. Owned hardware becomes more economical once a team runs multiple fine-tuning or inference workloads over time on the same GPUs.

How much does sequence length affect the sizing above?

Significantly for activation memory. Longer training sequences increase memory per step even when parameter count stays fixed, so the VRAM figures above assume typical instruction-tuning sequence lengths rather than very long-context training.

How Nanobase AI helps

Nanobase AI, an NVIDIA Inception Program member, sizes and installs the exact GPU configuration a 70B fine-tuning workload needs, whether that is a single H100 for QLoRA or a multi-node H200 cluster with InfiniBand for full fine-tuning. We also handle the Kubernetes GPU Operator and interconnect configuration that keeps distributed training efficient. See how many GPUs a 70B or 405B model needs or our solutions for related sizing guidance.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.