How many GPUs are needed to fully fine-tune a 70B model?

Fully fine-tuning a 70 billion parameter model needs substantially more memory than serving it, because training has to hold the model weights, gradients, and optimizer states simultaneously, and with a common optimizer like Adam in mixed precision, a reasonable rule of thumb is roughly 16 to 20 bytes of memory per parameter once weights, gradients, optimizer moments and master copies are all accounted for, putting total memory needs for a 70B model over a terabyte before activation memory is even added. That places full fine-tuning well beyond a single node's capacity, typically requiring eight or more H100 or H200 GPUs working together with memory-sharding techniques such as ZeRO or fully sharded data parallelism, along with gradient checkpointing to control activation memory, and fast InfiniBand or NVLink interconnect to keep the sharded training efficient. Many teams find that parameter-efficient methods like LoRA or QLoRA deliver comparable results for domain adaptation at a small fraction of this hardware cost, so full fine-tuning is usually reserved for cases requiring the deepest possible behavioral change to the base model. Nanobase AI designs and provisions the multi-GPU training clusters this scale of full fine-tuning requires.

Why training memory is not the same calculation as inference memory

Inference sizing asks only how big the weights are at a given precision. Training sizing has to add three more pools on top: gradients (one value per trainable parameter), optimizer state (for Adam, typically two additional values per parameter for the moment estimates, often kept in FP32 for stability), and a master FP32 copy of the weights when training in mixed precision. A 70B model that needs 140 GB to serve in FP16 can need well over a terabyte to fully fine-tune, which is the single biggest reason full fine-tuning and inference deployment are sized on completely different hardware footprints.

The per-parameter byte breakdown

Activation memory adds further on top of this figure and scales with batch size, sequence length and whether gradient checkpointing is enabled, but the roughly 18 bytes per parameter figure already establishes that this workload cannot fit on any single GPU or even a single 8-GPU node's combined memory without sharding.

Cluster options that make this feasible

1. ZeRO Stage 3 or fully sharded data parallelism (FSDP) partitions weights, gradients and optimizer state across all GPUs in the job rather than replicating them, which is what makes multi-terabyte training memory demands tractable on a finite cluster.
2. Gradient checkpointing trades recomputation time for a substantial cut in activation memory, almost always enabled for a model this size.
3. A minimum of eight H100 or H200 GPUs working together with sharding is the realistic starting point; fewer than that generally cannot hold even the sharded state plus activations at a usable batch size.
4. Fast interconnect, NVLink within a node and InfiniBand between nodes, is not optional at this scale, since sharded training requires constant gradient and parameter synchronization across GPUs, and a slow interconnect turns communication into the bottleneck rather than compute.
5. Multi-node scaling (two or more 8-GPU nodes) becomes necessary when the target batch size or sequence length pushes even a fully sharded single node past its combined memory, or when training time needs to be shortened by adding parallel compute.

The alternative worth considering first

Because the memory gap between full fine-tuning and parameter-efficient methods is so large, many teams evaluate whether LoRA or QLoRA can reach acceptable quality before committing to a multi-node training cluster. As covered in VRAM for LoRA fine-tuning, the same techniques scale to 70B and can run on a small fraction of the GPUs full fine-tuning requires. Full fine-tuning is justified when the task requires deep behavioral change to the base model that adapter-based methods cannot reach, such as substantially altering core reasoning patterns rather than adapting tone, format or domain vocabulary; for the latter, parameter-efficient methods usually get there for far less hardware.

Frequently asked questions

Can full fine-tuning of a 70B model be done on a single 8-GPU H100 node?

It is tight but generally feasible with aggressive ZeRO Stage 3 sharding, gradient checkpointing, and a modest batch size, though many teams find a second node gives meaningfully more usable batch size and training throughput rather than running at the absolute memory ceiling.

Does H200 change this calculation significantly?

H200's 141 GB per GPU roughly doubles combined node memory over H100 at the same GPU count, which allows a larger batch size or longer sequence length within the same sharded configuration, though the fundamental terabyte-scale memory requirement for the model itself is unchanged.

Is full fine-tuning ever done on fewer than eight GPUs?

For 70B specifically, rarely in practice; below eight GPUs, even with full sharding, the remaining per-GPU memory for activations and communication buffers becomes too constrained for a workable training configuration at reasonable sequence lengths.

How much does full fine-tuning cost compared to LoRA in practice?

Exact costs vary by provider and should be verified as of 2026, but the GPU-count difference alone (eight-plus data-center GPUs versus one) typically makes full fine-tuning many times more expensive in both hardware and training time than a LoRA or QLoRA approach for the same model size.

How Nanobase AI helps

Nanobase AI designs and provisions the multi-GPU training clusters that full fine-tuning at this scale requires, including sharding strategy, interconnect topology and checkpointing configuration, and helps customers determine upfront whether a parameter-efficient method would reach the same outcome for a fraction of the hardware investment.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.