Yes, large language models can extract data from documents without any training examples, using what is called zero-shot extraction, where the model is simply given a schema description, such as field names and expected types, along with the document text or image, and asked to return the matching values in a structured format like JSON. This works because large language models already learned general document understanding and language patterns during pretraining, so they can generalize to a new document type or field set immediately, unlike older extraction systems that needed labeled examples for every new form layout. Zero-shot accuracy is often strong for common, clearly labeled fields like dates, names and totals, but it typically improves further with few-shot prompting, where two or three annotated examples are included in the prompt to show the model the exact expected format and edge cases specific to the business. For consistently high accuracy on a narrow, high-volume document type, few-shot prompting or light fine-tuning still outperforms pure zero-shot extraction, so the choice is really about how much upfront effort a use case justifies. Zero-shot extraction is especially valuable for handling rare or one-off document types that would not justify building a dedicated model. Nanobase AI uses zero-shot and few-shot extraction to get new document types into production quickly before optimizing further.
Why zero-shot extraction works at all
Zero-shot extraction works because a large language model already learned general document structure and language patterns during pretraining, so it can generalize to a schema it has never seen labeled examples for, simply by being given field names, type definitions and the document content. This is a genuine capability shift from older extraction systems that needed labeled examples for every new form layout before they could extract anything at all, and it means a new, rare or one-off document type can go into production the same day it is first encountered rather than waiting for a labeling and training cycle. Zero-shot accuracy is strongest on common, clearly labeled fields, dates, names, totals, that follow conventional formatting, and weaker on fields that require inferring an implicit value or handling a business-specific convention the model has no way to know without an example.
Zero-shot vs few-shot vs fine-tuned
| Approach | Setup effort | Accuracy on common fields | Accuracy on business-specific conventions | Best fit |
|---|---|---|---|---|
| Zero-shot | None | High | Lower, needs explicit instruction | New or rare document types |
| Few-shot (2-5 examples in prompt) | Low | High | Improved, examples show conventions | Moderate-volume, evolving document types |
| Fine-tuned | High (needs labeled dataset) | Highest, tuned to exact format | Highest, learns from real examples | Stable, high-volume, narrow document type |
Structuring the prompt for reliable output
Reliable zero-shot extraction depends heavily on how precisely the schema is specified in the prompt: field names alone produce inconsistent results, while explicit type definitions, allowed value ranges, and a clear instruction to return a fixed structured format like JSON produce far more consistent, machine-parseable output across many documents. Using a model's native structured-output or function-calling mode, where available, rather than asking for JSON in free text and hoping the model formats it correctly, removes an entire category of parsing failure caused by malformed output. Explicitly instructing the model to return null or a defined placeholder for a field it cannot find, rather than guessing or omitting the field silently, also makes downstream validation far easier to implement correctly.
When zero-shot stops being good enough
For consistently high accuracy on a narrow, high-volume document type, such as one specific vendor's invoice format processed thousands of times a month, few-shot prompting or light fine-tuning still outperforms pure zero-shot extraction, since showing the model two or three annotated examples of that exact format teaches it edge cases and conventions no amount of schema description alone communicates. The practical decision point is volume and stability: a document type seen rarely or whose format keeps changing favors zero-shot's zero setup cost, while a stable, high-volume document type justifies the modest investment of a few labeled examples or a fine-tuning pass, since the accuracy gain compounds across every future document of that type.
Frequently asked questions
How many examples does few-shot prompting typically need?
Often just two to five well-chosen examples that cover the format's edge cases, such as a document with a missing field or an unusual layout variant, provide most of the accuracy improvement over zero-shot; adding many more examples has diminishing returns and consumes more of the context window.
Is zero-shot extraction reliable enough for financial or compliance-critical fields?
It can be a reasonable starting point, but any extraction feeding a financial posting or compliance decision should include validation rules and a human review step for low-confidence fields, since zero-shot accuracy, while often strong, is not guaranteed to be error-free on any given document.
Does zero-shot extraction cost more than a fine-tuned model per document?
Per-document inference cost is comparable if using the same base model, but a fine-tuned smaller model can often achieve similar accuracy at lower inference cost once trained, since fine-tuning lets a more efficient model reach the accuracy a larger general-purpose model achieves through prompting alone.
How Nanobase AI helps
Nanobase AI uses zero-shot and few-shot extraction to get new document types into production quickly, then moves high-volume types to fine-tuning once the accuracy gain justifies the investment. Related: fine-tuning a document AI model on your own forms.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.