Most enterprise teams should default to LoRA and reserve full fine-tuning for cases that require deep changes to the model's internal representations. LoRA freezes the pretrained weights and injects small trainable low-rank matrices into the attention and feed-forward layers, so it typically trains under one percent of total parameters, needs far less GPU memory, and produces a lightweight adapter file that is easy to version, swap and roll back. Full fine-tuning updates every weight and can outperform LoRA on tasks that need broad shifts in style, reasoning pattern or multi-task behavior, but it demands multi-GPU setups with frameworks like DeepSpeed or FSDP even for a 7B to 13B model once optimizer states are counted. In practice, LoRA and its quantized variant QLoRA match full fine-tuning quality on most instruction following, classification and structured output tasks while cutting compute cost by an order of magnitude. Full fine-tuning still makes sense for continued pretraining on a new domain or language, or when serving one adapter per customer is not viable. Nanobase AI helps clients benchmark both approaches on their own data before committing GPU budget to one path.
Why the memory gap is so large
The practical difference between LoRA and full fine-tuning comes down to what the optimizer has to store, not just what gets updated. An Adam-style optimizer keeps the parameters themselves, the gradients, and two optimizer moment estimates for every trainable weight, which multiplies the effective memory footprint of full fine-tuning to roughly four times the raw parameter count in mixed precision, before activations are even counted. LoRA sidesteps almost all of this by freezing the base weights entirely and only computing gradients and optimizer states for the small injected adapter matrices.
For a 7B model, full fine-tuning in mixed precision typically needs well over 100 GB across parameters, gradients, and optimizer states, pushing it onto multiple GPUs, while LoRA can fit comfortably on a single 24 to 48 GB card. The memory difference is not incremental, it is the difference between a project that needs a GPU cluster and one that runs on a single workstation card.
A concrete comparison at three model sizes
| Model size | LoRA (approx. VRAM) | Full fine-tuning (approx. VRAM) | Typical hardware |
|---|---|---|---|
| 7B | 12–20 GB | 100–130 GB | LoRA: single 24 GB GPU; full: multi-GPU |
| 13B | 20–30 GB | 200–250 GB | LoRA: single 40–48 GB GPU; full: multi-GPU |
| 70B | 40–48 GB (with QLoRA) | 1+ TB | LoRA: single H100 80 GB; full: 8+ H100/H200 |
These figures assume standard sequence lengths and batch sizes with gradient checkpointing enabled; actual numbers shift with context length and batch size. At every model size in this table, LoRA collapses a multi-GPU requirement down to hardware a single team can own outright.
Where full fine-tuning still wins on quality
LoRA's frozen base weights are also its limitation: the model can only express changes reachable through a low-rank update to a subset of layers, which is usually enough for instruction following, classification, and structured output but can fall short when the task needs broad shifts across many layers simultaneously. Full fine-tuning earns its cost in three situations: continued pretraining that must reshape token-level representations across the whole network, multi-task training where the model must juggle many simultaneously conflicting objectives, and cases where LoRA has already been tried and measurably underperforms on a held-out evaluation set.
It is worth noting that a well-tuned LoRA configuration, and especially QLoRA, matches full fine-tuning quality on most enterprise instruction tuning and classification tasks in published comparisons, so the burden of proof should sit with full fine-tuning, not with LoRA. Reach for full fine-tuning only after a properly tuned LoRA run has been benchmarked and found lacking, not as a default starting point.
Deployment and iteration speed also favor LoRA
Beyond training cost, LoRA adapters are typically tens to a few hundred megabytes, which makes them trivial to version, store, and roll back compared to a full multi-gigabyte model checkpoint per experiment. This matters more than it looks on paper: a team iterating on hyperparameters or dataset versions can keep dozens of LoRA checkpoints around for comparison at negligible storage cost, while doing the same with full fine-tuning checkpoints quickly becomes an infrastructure problem in its own right. Multiple LoRA adapters can also be served from one base model copy, which full fine-tuning cannot replicate without duplicating the entire model per variant, a distinction covered in more detail when serving multiple adapters on one GPU becomes relevant.
Faster, cheaper iteration compounds over a project's lifetime, which is often a bigger practical win than the headline VRAM savings.
Frequently asked questions
Can LoRA match full fine-tuning quality?
On most instruction following, classification, and structured output tasks, yes, when rank and target modules are tuned properly. Full fine-tuning tends to pull ahead only on tasks requiring broad representational shifts, such as continued pretraining on a new domain or language.
Does LoRA work with quantized base models?
Yes, this combination is QLoRA, which quantizes the frozen base model to 4-bit precision while training LoRA adapters in higher precision, cutting memory further without materially changing LoRA's core trade-offs.
Which layers should LoRA adapters target?
Attention projection layers, commonly the query and value projections, are the traditional default, but applying LoRA to all linear layers, including feed-forward layers, often improves quality at modest additional memory cost and is increasingly the recommended default.
Is full fine-tuning ever worth it for a small model?
Sometimes, for a 1B to 3B model where the absolute memory cost of full fine-tuning is manageable on a single GPU and the task benefits from broader weight updates. At larger scales the cost-benefit tilts strongly toward LoRA.
How Nanobase AI helps
Nanobase AI benchmarks LoRA against full fine-tuning on a client's actual dataset before committing GPU budget to either path, so the decision rests on measured evaluation results rather than a general rule of thumb. We size the exact GPU configuration each approach needs, from a single card for LoRA to a sharded multi-GPU cluster for full fine-tuning, and configure the training framework accordingly. Explore our GPU infrastructure solutions or see how many GPUs a 70B model needs for related sizing guidance.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.