Fine-tuning a vision-language model on your own documents follows a similar LoRA-based workflow to text-only fine-tuning, but the training examples pair an image, such as a scanned invoice, form or diagram, with an instruction and the target text response, and the adapter is typically applied to the language model layers while the vision encoder is often kept frozen or trained with a much smaller learning rate. Current open-weight vision-language models handle document understanding tasks like layout-aware extraction, table parsing and handwriting recognition well after fine-tuning on a few hundred to a few thousand representative document images paired with the correct extracted output. Data preparation is the hardest part of these projects, since you need accurately labeled ground truth for each document, ideally verified by a human reviewer, and enough variety in document layout, image quality and scan orientation to generalize beyond the exact templates in the training set. Evaluation should measure field-level extraction accuracy on held-out documents, not just overall response similarity. Nanobase AI, a Silicon Valley enterprise AI engineering company, builds document-specific vision-language fine-tuning pipelines for clients moving off manual data entry or brittle OCR rules.
Data shape is the main thing that changes from text fine-tuning
A vision-language model fine-tuning example pairs an image with an instruction and a target text response, structurally similar to text-only instruction tuning but with an image tensor added to each training record. For document tasks specifically, the image is typically a scanned or photographed page, and the target response is the extracted field, structured JSON, or answer to a question about that page's content. Getting the image quality and diversity right in training data matters as much as getting the target text right, since a model trained only on clean, high-resolution scans will degrade sharply on the skewed, low-light or handwritten documents that show up in real production use.
Model architecture choices that affect training
Most current open-weight vision-language models (in the Qwen-VL and Llama Vision families, among others) freeze or apply a much lower learning rate to the vision encoder while applying LoRA to the language model layers, since the vision encoder has already learned general visual features and rarely needs task-specific retraining for document understanding. This keeps the trainable parameter count and GPU memory footprint close to a text-only LoRA fine-tuning job, despite the added image input. Full fine-tuning of the vision encoder itself is occasionally warranted for highly unusual visual domains (specialized medical imaging, non-standard document layouts unlike anything in the base model's pretraining), but for typical enterprise document types this is rarely necessary and adds meaningful training cost for limited benefit.
A practical data preparation sequence
Realistic document quality and explicit edge cases matter more to final accuracy than raw example count, so front-load effort there rather than on volume.
- Collect a representative sample of the actual document types and quality levels seen in production, including realistic scan artifacts, rather than only clean reference examples.
- Label the target output precisely in the format you need at inference time, whether that is structured JSON with specific field names or free-text answers to embedded questions.
- Include negative and edge cases explicitly: documents missing an expected field, ambiguous handwriting, multi-page documents where the answer spans pages, since models trained only on clean positive examples fail unpredictably on these in production.
- Split by document source or template, not just randomly, when validating, to catch overfitting to a specific document layout rather than genuine field understanding.
- Start with a few hundred to a few thousand labeled examples depending on document type diversity, then expand based on evaluation gaps rather than collecting a large dataset upfront.
Evaluating document VLM fine-tuning results
| Task type | Data format needed | Evaluation metric |
|---|---|---|
| Field extraction (invoices, forms) | Image + target JSON with field names | Field-level accuracy, exact match per field |
| Table parsing | Image + structured table representation | Cell-level accuracy, structure match |
| Free-text Q&A over a document | Image + question + answer | Answer correctness, often via edit distance or LLM-judged match |
| Handwriting recognition | Image + transcribed text | Character or word error rate |
Field-level accuracy, meaning the percentage of individual fields extracted correctly rather than requiring an entire document to be perfect, is generally the more actionable metric for extraction tasks, since it identifies which specific fields need more training data rather than giving a single pass/fail signal per document.
Where this connects to the rest of the fine-tuning pipeline
Everything about dataset format choices and structured JSON output fine-tuning for text models applies directly to the text side of a vision-language training example; the image simply becomes an additional input alongside the instruction. Teams building document AI pipelines should also weigh whether a fine-tuned VLM or a combination of OCR plus a text-only fine-tuned model better fits their accuracy and latency requirements, since both are viable architectures depending on document complexity and layout variability.
Frequently asked questions
Do we need to fine-tune the vision encoder, or just the language model layers?
For most document understanding tasks, freezing or barely touching the vision encoder while applying LoRA to the language model layers works well, since the encoder's general visual feature extraction transfers reasonably well to new document types. Full vision encoder fine-tuning is rarely necessary outside highly unusual visual domains.
How many document examples do we need to fine-tune a VLM well?
It depends heavily on document diversity, but a few hundred to a few thousand representative examples covering realistic quality variation typically produces solid results for a well-defined extraction task. Diversity of document conditions matters more than raw example count when planning collection effort.
Can a fine-tuned VLM replace a traditional OCR pipeline entirely?
For many structured extraction tasks, yes, since a fine-tuned VLM can directly output structured data without a separate OCR and parsing stage. For extremely high-volume, well-standardized documents, a hybrid pipeline combining OCR with lighter-weight parsing can still be more cost-effective at scale.
How Nanobase AI helps
Nanobase AI fine-tunes vision-language models for document understanding tasks including invoice extraction, form parsing and multilingual document Q&A, building the evaluation pipeline alongside the training data so accuracy is measured per field rather than assumed. This sits within our broader enterprise AI deployment work for document-heavy industries like insurance and finance.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.