The best GPU for fine tuning LLMs on premise depends on model size and technique, but for most serious fine tuning work, H100 or H200 GPUs in an NVLink connected multi GPU server remain the strongest general purpose choice, since fine tuning benefits from both high memory bandwidth for gradient computation and fast GPU to GPU communication when a model is split across multiple cards using data or model parallelism. For parameter efficient methods like LoRA or QLoRA on models up to around 13B to 34B parameters, a single RTX PRO 6000 with 96 GB of memory or a single A100 can often handle the job without needing a full multi GPU cluster, making it a much more affordable entry point for teams fine tuning smaller or mid sized open weight models. Full fine tuning of larger models, or training with long sequence lengths and large batch sizes, benefits significantly from an 8 GPU H100 or H200 NVLink domain, where checkpoint sharding and gradient synchronization across GPUs happen efficiently. Storage and CPU to GPU data pipeline throughput also matter more for fine tuning than for inference, since training reads through datasets repeatedly. Nanobase AI sizes fine tuning infrastructure based on model size, dataset scale, and chosen technique rather than defaulting to the largest available GPU.
Technique determines hardware more than model name does
Two teams fine-tuning the same 70B open-weight model can need very different hardware depending on whether they are running full fine-tuning or a parameter-efficient method like LoRA or QLoRA. The fine-tuning technique, not just the base model size, is usually the bigger driver of GPU requirements, because full fine-tuning updates every parameter and its optimizer state, while LoRA-style methods freeze the base model and train a small number of additional parameters, dramatically reducing memory and compute needs.
Getting this mapping wrong in either direction is expensive: over-provisioning a multi-GPU H100 cluster for a QLoRA job wastes budget, while under-provisioning a single GPU for full fine-tuning of a large model simply will not run.
Matching technique, model size, and hardware
| Technique | Typical model size range | Recommended hardware | Why |
|---|---|---|---|
| LoRA / QLoRA | Up to ~13B–34B | Single RTX PRO 6000 (96 GB) or single A100 (80 GB) | Frozen base weights plus small adapter parameters fit comfortably in memory |
| LoRA / QLoRA | 70B+ | Single high-memory GPU (RTX PRO 6000, A100, H100) or light multi-GPU | Quantized base weights reduce footprint enough for fewer GPUs than full fine-tuning would need |
| Full fine-tuning | Up to ~13B | Single or dual H100/H200 | Optimizer states and gradients for all parameters need substantial memory and bandwidth |
| Full fine-tuning | 70B+ | 8-GPU H100 or H200 NVLink domain | Model, gradient, and optimizer sharding across GPUs needs fast NVLink communication |
Why full fine-tuning leans toward NVLink-connected clusters
Full fine-tuning benefits from both high memory bandwidth for gradient computation and fast GPU-to-GPU communication when a model is split across multiple cards using data or model parallelism, since every training step involves synchronizing gradients or sharded optimizer states across the participating GPUs. An 8-GPU H100 or H200 NVLink domain handles checkpoint sharding and gradient synchronization efficiently precisely because every GPU can communicate with every other GPU at high bandwidth without routing through slower PCIe or network paths. For long sequence lengths and large batch sizes, this interconnect advantage compounds, since more data moves between GPUs per training step.
Why parameter-efficient methods change the calculus
LoRA and QLoRA freeze the base model's weights and train only a small number of additional adapter parameters, and QLoRA further quantizes the frozen base weights to reduce their memory footprint. This means the dominant memory cost, storing and updating optimizer state for billions of parameters, mostly disappears, which is why a single GPU with enough memory to hold the (possibly quantized) base model plus a modest amount of adapter and activation memory is often sufficient even for models in the 13B to 34B range, and sometimes larger with sufficiently aggressive quantization. The tradeoff is that parameter-efficient methods, while effective for many domain adaptation and instruction-tuning tasks, do not always match full fine-tuning's ceiling on tasks requiring the model to learn substantially new behavior across all its parameters.
Infrastructure beyond the GPU itself
- Storage and CPU-to-GPU data pipeline throughput matter more for fine-tuning than for inference, since training reads through datasets repeatedly rather than serving one request at a time.
- Checkpoint storage should be sized for the technique: full fine-tuning checkpoints scale with total model size, while LoRA/QLoRA adapter checkpoints are dramatically smaller.
- For multi-GPU full fine-tuning, verify the framework's support for the specific parallelism strategy (data, tensor, or pipeline) being used, since this affects both GPU count and interconnect requirements.
- Consider whether RAG or fine-tuning is even the right approach before sizing hardware, since some use cases are better served by retrieval than by additional training.
Frequently asked questions
Can QLoRA fine-tune a 70B model on a single GPU?
Often yes, with a high-memory GPU like an RTX PRO 6000 or H100 and sufficiently aggressive quantization of the frozen base weights, though exact feasibility depends on sequence length, batch size, and the specific quantization scheme used.
Does full fine-tuning always require multiple GPUs?
For smaller models, roughly up to 13B parameters, a single or dual H100/H200 setup can handle full fine-tuning. Larger models generally need an 8-GPU NVLink-connected cluster to hold sharded optimizer states and gradients efficiently.
Is LoRA as effective as full fine-tuning?
For many domain adaptation and instruction-tuning tasks, LoRA and QLoRA achieve results close to full fine-tuning at a fraction of the hardware cost, though tasks requiring substantial new behavior across the full model can still favor full fine-tuning.
Does NVLink matter for LoRA/QLoRA training?
Less than for full fine-tuning, since parameter-efficient methods typically fit on a single GPU or need only light multi-GPU coordination, reducing the reliance on the fastest possible inter-GPU communication that full fine-tuning depends on.
How Nanobase AI helps
Nanobase AI, a Silicon Valley enterprise AI engineering company, sizes fine-tuning infrastructure based on model size, dataset scale, and the chosen technique rather than defaulting to the largest available GPU cluster. We also help teams decide between fine-tuning and retrieval-based approaches upfront, drawing on our RAG vs fine-tuning decision guide. Explore GPU infrastructure and fine-tuning solutions or contact us to right-size your training environment.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.