How do we connect a private LLM to our company documents?

Connecting a private LLM to company documents is done through retrieval-augmented generation, a pattern where documents are converted into vector embeddings, stored in a vector database, and retrieved at query time to give the model relevant context it was never trained on. The pipeline starts by ingesting files from sources like SharePoint, Google Drive, Confluence or a file server, splitting them into chunks of a few hundred to a thousand tokens, and embedding each chunk with an embedding model served locally alongside the LLM. Those embeddings go into a vector database like Qdrant, Weaviate or pgvector, and when a user asks a question, the system retrieves the most relevant chunks and passes them to the LLM as context, so its answer is grounded in the company's actual documents rather than general training data. Keeping permissions consistent matters here, since retrieval should respect the same access controls as the source system, so a document a user could not open directly should not surface in their AI answers either. Reranking and hybrid search, combining keyword and vector search, typically improve answer quality over vector search alone. Nanobase AI, a Silicon Valley enterprise AI engineering company, builds these document-connected RAG pipelines as a core part of private LLM deployments.

The pipeline has five stages, and each one has failure modes

Connecting a private LLM to company documents is usually summarized as "add RAG," which glosses over five distinct pipeline stages, ingestion, chunking, embedding, storage, and retrieval, each of which can independently produce poor answers even when the model itself is strong. The most common cause of a RAG system giving wrong or irrelevant answers is not the model; it is chunking that splits context awkwardly or a retrieval step that returns the wrong document chunks, both of which sit upstream of the model entirely.

Chunking strategy comparison

The chunking approach chosen at the start of a RAG project has more influence on answer quality than almost any other single decision in the pipeline.

Vector database choice

The right vector database choice depends far more on operational fit than on raw retrieval quality, since well-configured versions of any major option perform similarly for most enterprise document volumes. Several open-source vector databases fit an on-premise RAG pipeline well, including options like Qdrant, Weaviate, Milvus, and pgvector as a Postgres extension, and a team that already runs Postgres often finds pgvector the lowest-friction starting point.

Building the pipeline step by step

1. Inventory document sources: SharePoint, Google Drive, Confluence, file servers, and any system with an API or connector available, prioritizing the highest-value content first rather than everything at once.
2. Build or configure ingestion connectors for each source, respecting existing access permissions so retrieval does not leak documents a user could not otherwise see.
3. Choose a chunking strategy from the table above matched to the dominant document type, and test it against a sample before processing the full corpus.
4. Select an embedding model and generate embeddings for each chunk, storing them in the chosen vector database alongside metadata like source, author, and date.
5. Build the retrieval step: given a user query, embed it, search the vector database for the most relevant chunks, and inject them into the prompt sent to the model.
6. Tune retrieval parameters, chunk count returned, similarity threshold, and re-ranking if used, against real user queries, since default settings rarely perform optimally without adjustment.

Retrieval tuning is not a one-time setup task; it needs ongoing adjustment as the document corpus grows and as real usage reveals which query types the initial configuration handles poorly.

Keeping the index current

A RAG pipeline connected once and never updated slowly degrades as source documents change, since the vector index is only ever a snapshot in time, not a live mirror. Production deployments need a re-indexing schedule, or ideally an event-driven update triggered when a source document changes, so the retrieval layer does not serve outdated policy documents or superseded contract versions as if they were current.

Frequently asked questions

How many documents can a self-hosted RAG pipeline realistically handle?

Modern open-source vector databases handle millions of chunks without issue on modest hardware; the practical limit for most enterprises is document quality and permission management, not raw technical scale.

Does RAG respect existing document permissions automatically?

Not automatically; permission-aware retrieval has to be explicitly built by checking the source system's access controls at query time or by tagging chunks with permission metadata during ingestion, otherwise a naive RAG setup can expose documents to users who should not see them.

Can RAG work across multiple languages, such as English and Turkish documents together?

Yes, with a multilingual embedding model and a model like Qwen2.5 that handles multiple languages well, a single RAG pipeline can serve mixed-language document corpora, though testing retrieval quality separately for each language is worthwhile since embedding quality can vary by language.

How Nanobase AI helps

Nanobase AI, a Silicon Valley enterprise AI engineering company, builds RAG pipelines that connect private LLMs to real company document sources, with permission-aware retrieval, a chunking strategy matched to the actual document types, and a re-indexing process that keeps answers current as source documents change. This is one of the most integration-heavy parts of a private ChatGPT deployment; see our solutions for how it fits into a full engagement.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.