How do we build a data pipeline that feeds documents into an LLM system?

Building a data pipeline that feeds documents into an LLM system requires stages for ingestion, cleaning, chunking, embedding and indexing, each of which needs to run incrementally so new or updated documents reach the model without a full reprocessing job every time. Ingestion connects to source systems such as SharePoint, Confluence, Google Drive or a document management system through APIs or connectors, extracts text from formats like PDF, DOCX and HTML while preserving structure such as headers and tables, and captures metadata like author, date and access permissions needed for later filtering. Chunking splits documents into passages sized for the embedding model's context window, typically a few hundred tokens with some overlap, and choosing chunk boundaries that respect semantic units like paragraphs or sections meaningfully improves downstream retrieval quality. An orchestrator like Airflow, Dagster or Prefect schedules and monitors each stage, detects document changes to trigger re-embedding of only the affected chunks, and handles failures without silently dropping documents. Access control metadata must propagate all the way through to the vector index so retrieval never surfaces content a user should not see. Nanobase AI, a Silicon Valley enterprise AI engineering company, builds these ingestion-to-index pipelines for enterprise document repositories, keeping permissions and freshness intact from source system to retrieval.

Match the connector to its actual extraction problem

A single generic "PDF and HTML parser" is not a document pipeline; each source system has its own extraction failure mode that needs a dedicated handler and its own test cases.

Tables and scanned documents deserve dedicated handling

Embedded tables lose their row and column relationships when extracted as plain text, turning a clean financial table into an unreadable wall of numbers by the time it reaches the embedding model. Scanned PDFs need OCR with layout awareness, not generic OCR, since a naive text dump often merges multi-column layouts into a scrambled reading order. Both problems stay invisible until someone asks a question whose answer lives in exactly the table or page that got mangled, so they need dedicated parsing logic and dedicated test documents rather than a hope that the generic parser handles them well enough.

A pipeline that is never tested against a real scanned PDF or a real multi-column table will pass every demo and then fail on the first real question that depends on one.

An incremental pipeline architecture, not a full reprocessing job

1. A change-detection step polls or subscribes to source-system webhooks to identify new, modified or deleted documents, rather than re-scanning the entire repository each time.
2. Only the changed documents are re-parsed and re-chunked; unaffected chunks keep their existing embeddings untouched.
3. New or modified chunks are re-embedded and written to a staging area rather than directly overwriting the live index.
4. A validation step checks the new chunks parsed cleanly, with no tables collapsed and no boundary cutting a sentence in half, before promotion.
5. Staged chunks are swapped into the live index atomically, and chunks from deleted documents are removed in the same step.
6. Failed documents go to a dead-letter queue with the specific error, rather than being silently dropped, so a failure surfaces instead of quietly disappearing from the index.

Reprocessing only what changed is what makes a pipeline usable on a repository that updates daily, rather than something that only works for a one-time import.

Permissions as a pipeline stage, not an afterthought

Access-control metadata, meaning who can see a given document in the source system, has to be captured during ingestion and carried through chunking, embedding and indexing as a first-class field on every chunk, not bolted on at retrieval later. A system that filters by permission only at query time, without that metadata attached from the start, either over-restricts, hiding content that should be visible, or under-restricts and returns content a user should never see. This is one of the most common places enterprise deployments fail an internal security review.

Frequently asked questions

How often should the pipeline check source systems for changes?

It depends on how current the answers need to be. A webhook-driven approach, where the source system notifies the pipeline of a change, gives near-real-time freshness and is preferable when the source system supports it. Polling on a schedule, from every few minutes to daily, is a reasonable fallback otherwise.

What happens if OCR gets a scanned document wrong?

A dedicated validation step should flag documents with unusually low text-extraction confidence for manual review rather than silently indexing garbled text. Routing low-confidence documents to a review queue, instead of blocking the whole pipeline or indexing bad text, keeps the pipeline moving without corrupting the index.

Should permissions be enforced at ingestion or at query time?

Both, and they serve different purposes. Permission metadata must be captured during ingestion and attached to each chunk, but the actual enforcement, deciding what a specific user can see, has to happen at query time against their current access, since permissions can change after a document was indexed.

Do we need a different pipeline for structured and unstructured sources?

Not necessarily a different pipeline, but a different extraction stage. Structured exports like CSV or database tables skip OCR and layout parsing entirely, while unstructured formats like scanned PDFs need the full extraction path; both can share the same chunking, embedding and indexing stages downstream.

How Nanobase AI helps

Nanobase AI, an NVIDIA Inception Program member, builds these ingestion pipelines with source-specific handlers for SharePoint, Confluence, Drive and document management systems rather than one generic parser, and treats permission metadata as a design requirement from the first review, not a retrieval-time patch. We validate parsing against a client's actual documents, including scanned files and tables, in a working demo before the pipeline goes live as part of a full RAG deployment. For the governance side, see why data lineage matters for AI governance.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.