How do we handle PDFs with tables and images in RAG?

Handling PDFs with tables and images in RAG requires a parsing step that goes beyond plain text extraction, because naive text extraction typically scrambles table structure and drops images entirely, destroying the information a user actually needs. Layout-aware parsers such as Unstructured, LlamaParse, or NVIDIA's document processing tools detect tables and reconstruct them as structured markdown or HTML before chunking, which preserves row and column relationships instead of flattening a table into an unreadable stream of numbers. Images and charts are typically handled with a vision-language model that generates a text description or extracts embedded data at ingestion time, so the description becomes searchable text even though the original image is also kept and can be shown alongside the answer. Scanned or image-based PDFs need OCR before any of this, and OCR quality directly limits everything downstream, so it is worth validating OCR accuracy on a sample of the actual document set rather than assuming it works. Keeping each table or figure as its own chunk, tagged with the page number and surrounding heading, generally retrieves and answers better than merging it into surrounding paragraph text. Nanobase AI, a Silicon Valley document AI team, builds ingestion pipelines specifically tuned to a customer's PDF formats rather than using a one-size-fits-all parser.

The failure is invisible until someone asks about a table

Naive text extraction from a PDF, the kind built into many basic parsing libraries, reads a page left to right and top to bottom without understanding visual layout, which works reasonably for single-column prose and fails badly on multi-column layouts, tables, and embedded images. The failure is easy to miss during development because most test documents look fine when skimmed, and it only becomes visible in production when a user asks a question whose answer lived in a table row that extraction scrambled or dropped.

A PDF parsing failure on tables is silent during development and only surfaces when a user asks the exact question that depended on the scrambled content.

Tool comparison for layout-aware parsing

No single tool wins on every document type; testing against the actual PDF layouts in the corpus, not a generic sample, determines the right pick.

Scanned vs native PDFs need different handling

A native PDF, one generated directly from a word processor or reporting tool, contains actual text objects that a layout-aware parser can extract and structure directly. A scanned PDF is fundamentally an image, requiring optical character recognition before any text or table structure exists to parse at all, which adds an error-prone step and a meaningfully different failure surface, such as OCR misreading numbers in a financial table. Enterprise corpora are frequently a mix of both, and a pipeline needs to detect which type a given file is and route it accordingly rather than assuming one processing path handles both well.

Detecting scanned versus native PDFs and routing each to the right processing path prevents OCR errors from being silently applied to documents that never needed OCR.

A processing pipeline that preserves table structure through to retrieval

1. Detect document type: native text PDF versus scanned image PDF, routing each to the appropriate extraction path.
2. Run layout-aware parsing to identify distinct regions: prose, tables, and images, rather than treating the page as one text blob.
3. Reconstruct tables into a structured format, such as markdown tables, that preserves row and column relationships rather than flattening them into unstructured text.
4. Generate a text description for meaningful images or figures, since an embedding model cannot retrieve based on visual content directly without either a multimodal embedding model or a text caption.
5. Chunk each region type according to its own rules, keeping a full table intact in one chunk where possible rather than splitting it across chunk boundaries.
6. Attach page and section metadata to every chunk to support citation back to the original document location.

Preserving table and image structure through parsing does no good if the chunking step afterward splits that structure back apart, so the two stages need to be designed together.

Frequently asked questions

Can a language model read a table directly if it is preserved as markdown in a chunk?

Yes, most current language models handle markdown tables in context reasonably well, which is why reconstructing tables into markdown, rather than leaving them as unstructured extracted text, meaningfully improves the model's ability to answer questions about tabular data.

Do we need a multimodal embedding model for images in PDFs?

Not necessarily. Generating a text caption or description of an image and embedding that caption is a simpler and often sufficient approach; a multimodal embedding model becomes worth the added complexity when visual detail itself, not just a textual description, needs to be searchable.

How do we handle a PDF with a table that spans multiple pages?

The parsing step needs to detect table continuation across a page break and merge it back into one logical table before chunking, since treating each page's fragment as a separate table loses the relationship between rows split across the boundary.

Is OCR accuracy good enough for financial tables with precise numbers?

OCR accuracy on clean, well-scanned documents is generally strong but not perfect, and financial or numeric tables carry a higher cost when a misread digit goes unnoticed, which is a reason to spot-check OCR output on high-stakes numeric documents rather than trusting it blindly.

How Nanobase AI helps

Nanobase AI builds document parsing pipelines that detect scanned versus native PDFs, preserve table structure through to the chunking stage, and route each document type to the right extraction tool, rather than applying one generic parser across a mixed corpus. See our solutions and the related guide on Excel and tabular data in RAG for handling structured data specifically.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.