For structured data extraction into JSON or similar schemas, model choice matters less than the extraction pipeline design, but among open-weight models, Qwen 3 and Llama 4 tend to produce the most reliably well-formed structured output when combined with constrained decoding techniques like grammar-based sampling or JSON schema enforcement through tools such as Outlines or vLLM's structured output support. Constrained decoding is what actually guarantees valid output format, since even a strong model prompted to return JSON will occasionally produce malformed output without it, so pairing a capable model with schema enforcement matters more than picking the single highest-benchmark model. For extracting from scanned or image-based documents, a vision-language model such as Qwen2.5-VL is needed first to get accurate text and layout understanding before structured extraction logic can run on top of it. Smaller fine-tuned models, in the 7B to 14B range, often match larger general models on narrow extraction tasks once trained on examples from specific document types, at meaningfully lower serving cost. Testing accuracy field by field against a labeled sample of real documents is the only way to know if an extraction pipeline is production-ready. Nanobase AI, a Silicon Valley enterprise AI engineering company, builds these structured extraction pipelines, including the schema enforcement layer, rather than relying on model choice alone for reliability.
Model choice is the smaller lever in structured extraction reliability
A strong model prompted to "return JSON" will occasionally produce malformed output regardless of how capable it is, because free-form generation has no hard guarantee of matching a schema, only a learned tendency to. Constrained decoding, generation techniques that make invalid output structurally impossible rather than merely unlikely, is what actually guarantees valid format, which means pairing a capable model with schema enforcement matters more than picking the single highest-benchmark model for extraction tasks.
Comparing extraction approaches
| Approach | How it works | Reliability |
|---|---|---|
| Prompt-only JSON request | Model asked to output JSON via instructions alone | No structural guarantee; occasional malformed output even from strong models |
| Function calling / tool schemas | Model output constrained to a predefined function signature | More reliable than prompting alone, depends on the model's function-calling training |
| Grammar-constrained decoding (e.g., Outlines, vLLM structured output) | Sampling restricted at each token step to only valid continuations per a schema | Structurally guarantees valid output matching the schema |
Grammar-constrained decoding works at the token sampling level, rejecting any next token that would make the output invalid against the target JSON schema, which is a fundamentally different guarantee than a model that has simply learned JSON formatting well from training data. This is why a smaller model paired with proper schema enforcement often outperforms a much larger model relying on prompting alone for pure format reliability, even if the larger model has better raw language understanding.
A structured extraction pipeline, step by step
- Define the target schema explicitly, including required fields, types, and any nested structure, before writing extraction prompts.
- Choose an extraction approach matched to the model's serving stack: Outlines or vLLM's structured output support for grammar-constrained decoding, or native function-calling for models trained specifically for tool use.
- For scanned or image-based documents, run a vision-language model first, such as Qwen2.5-VL, to get accurate text and layout understanding before any structured extraction logic runs on top of it.
- Validate extracted output against the schema post-generation as a second safety layer, even when using constrained decoding, to catch semantic errors, a correctly formatted but factually wrong field, that structural constraints alone cannot prevent.
- Score accuracy field by field against a labeled sample of real documents, since aggregate accuracy can mask a specific field that consistently extracts incorrectly.
When a smaller fine-tuned model beats a larger general one
For extraction from a narrow, consistent document type, invoices, a specific form template, a particular contract structure, a smaller model in the 7B to 14B range, fine-tuned on examples from that specific document type, often matches or exceeds a much larger general-purpose model's accuracy, at meaningfully lower serving cost. This is one of the clearer cases in enterprise LLM deployment where narrow fine-tuning plus a small model beats scale, because the extraction task's variability is genuinely bounded by the document type rather than open-ended.
Frequently asked questions
Does constrained decoding slow down generation?
It adds some overhead compared to unconstrained generation, since the sampling process checks validity at each step, but the overhead is generally modest and well worth the reliability gain for production extraction pipelines where malformed output would otherwise require manual correction or a retry loop.
Can constrained decoding guarantee factually correct extraction, not just correctly formatted?
No, format validity and factual accuracy are separate guarantees; constrained decoding ensures the output matches the schema structurally, but a field can still be correctly formatted and semantically wrong, which is why field-level accuracy testing against labeled documents remains necessary regardless of the decoding method used.
Is a vision-language model always needed for document extraction?
Only when the source is scanned, handwritten, or otherwise image-based rather than clean digital text; extraction from already-digital text documents can go directly to a text model without a vision-language step.
How Nanobase AI helps
Nanobase AI, a Silicon Valley enterprise AI engineering company, builds these structured extraction pipelines, including the schema enforcement and field-level validation layer, rather than relying on model choice alone for reliability. This connects to our document AI and vision-language work for enterprises processing scanned or image-based document types.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.