What learning rate, epochs and batch size should we use for LoRA?

A reasonable starting point for LoRA fine-tuning is a learning rate between one times ten to the negative four and three times ten to the negative four, since LoRA's adapter matrices tolerate higher rates than full fine-tuning would, combined with a cosine or linear decay schedule and a short warmup of around three to five percent of total steps. Most instruction tuning runs need only one to three epochs over the dataset, since more passes on a modest-size dataset quickly lead to overfitting and memorized formatting rather than generalized behavior. Batch size is usually set by available GPU memory rather than chosen freely, so an effective batch size of sixteen to sixty-four is common, achieved through gradient accumulation when the physical batch that fits in memory is much smaller. These values are starting points, not fixed rules, and the right settings shift with dataset size, task difficulty and model size, so a small hyperparameter sweep against a held-out validation set is worth the extra compute. Nanobase AI tunes these settings against client-specific validation metrics rather than applying one default configuration to every project.

Starting points are not final answers

A learning rate between roughly 1e-4 and 3e-4, one to three epochs, and an effective batch size of 16 to 64 achieved through gradient accumulation are reasonable defaults for LoRA fine-tuning, but treating them as fixed rules rather than a starting point is where many projects leave quality on the table. These values were established across a range of tasks and model sizes in published work and community practice, which means they are a safe place to begin, not a guarantee of optimal results for any specific dataset.

Default hyperparameters get a project started safely; a small sweep against your own validation set is what gets it to its actual best result.

A practical sweep grid that does not blow the budget

Because LoRA training runs are relatively cheap, a modest grid search is affordable even on a limited compute budget, unlike full fine-tuning where each run is expensive enough to discourage experimentation.

Running the full cross-product of this grid is usually unnecessary; a reasonable approach fixes two parameters at sensible defaults while sweeping the third, then narrows in on the best-performing combination. A grid search does not need to be exhaustive to be useful; even a partial sweep against a real validation set beats accepting default values unchecked.

Reading the warning signs during training

Watching the training curves, not just the final numbers, catches problems earlier and cheaper than waiting for a full run to complete. A learning rate that is too high typically shows loss spiking or failing to decrease smoothly in the first several hundred steps, which is worth stopping early rather than letting run to completion. A learning rate that is too low shows painfully slow improvement that a longer run alone will not meaningfully fix. Validation loss that starts rising while training loss keeps falling is the classic overfitting signature, and it usually appears within the second or third epoch on a modest-size dataset, which is exactly why one to three epochs is the common range rather than more.

Stopping a clearly failing run early saves compute for the next configuration in the sweep, rather than waiting out a run whose curves already show the outcome.

How dataset size and task difficulty shift these ranges

The starting-point ranges assume a moderate dataset of a few thousand examples on a mid-size model; both smaller datasets and harder tasks shift the right answer. Smaller datasets, in the low hundreds of examples, generally need fewer epochs and sometimes a slightly lower learning rate to avoid memorizing the limited data outright, connecting directly to avoiding overfitting on a small dataset. Harder tasks that require learning more complex new patterns, as opposed to simple format or tone adjustment, sometimes benefit from a higher LoRA rank paired with more conservative learning rate and epoch settings, a trade-off covered further in choosing LoRA rank and alpha.

Hyperparameter ranges shift with dataset size and task difficulty, which is why a validation-set sweep matters more than memorizing one set of numbers.

Frequently asked questions

Should we use a cosine or linear learning rate schedule?

Both work well for LoRA fine-tuning, and the difference is usually smaller than the difference from picking the right peak learning rate. Cosine decay is a common default choice in most current frameworks.

Does gradient accumulation change the effective learning rate needed?

Not directly, but it changes the effective batch size, and a larger effective batch size can sometimes tolerate a slightly higher learning rate, which is why both should be considered together rather than tuned in isolation.

How do we know when to stop a hyperparameter sweep?

Once additional configurations stop meaningfully improving the validation metric, or the improvement is smaller than the noise between repeated runs with the same settings, further sweeping has diminishing returns.

Is warmup necessary for a short LoRA training run?

Yes, even a short warmup of 3 to 5 percent of total steps typically stabilizes the first portion of training, since starting at full learning rate immediately can cause early instability regardless of run length.

How Nanobase AI helps

Nanobase AI tunes learning rate, epoch count, and batch size against client-specific validation metrics rather than applying one default configuration to every project, running the kind of targeted sweep described here as a standard part of each fine-tuning engagement. See our solutions for how this fits into a full fine-tuning delivery.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.