Yes, a RAG system can run fully on-premise with no cloud services involved, and this is a common requirement for finance, insurance, healthcare, and government customers with strict data residency or air-gap requirements. Every component has a self-hostable equivalent: the language model runs through an inference engine like vLLM, TensorRT-LLM, or NVIDIA NIM on local GPUs such as H100 or H200 servers; the embedding and reranking models run on the same or adjacent GPU infrastructure; and the vector database, whether Qdrant, Milvus, or pgvector, runs on local storage with no external API calls at any stage of the pipeline. Document ingestion, parsing, and connectors to internal systems like SharePoint or file shares also run entirely within the private network, so no document content, query, or generated answer ever leaves the organization's infrastructure. The tradeoff compared to cloud-hosted or API-based RAG is upfront hardware investment and the operational responsibility of running GPU infrastructure, Kubernetes, and monitoring internally rather than paying a usage-based API fee, though this cost is often justified by regulatory requirements that simply prohibit sending data to third-party services. Air-gapped deployments with no internet connectivity at all are achievable but require careful planning for model updates and package management. Nanobase AI, a Silicon Valley on-premise AI infrastructure specialist, deploys complete RAG stacks with zero external dependencies.
The stack is not the hard part; the network boundary is
Every component of a RAG pipeline has a mature, self-hostable equivalent as of 2026, so assembling a stack that runs without cloud services is a solved architecture problem. The harder, less discussed problem is closing every network egress point an off-the-shelf tool quietly opens by default, such as a telemetry call, an update checker, or a dependency that fetches a model weight file from a public repository at container startup. A team that deploys a technically self-hostable stack without auditing outbound network calls often discovers, well after go-live, that some component has been phoning home the entire time. A stack made entirely of self-hostable components is not the same as a stack with no outbound network calls; the second requires an explicit audit.
Component-by-component egress audit
| Component | On-premise equivalent | Default network dependency to check |
|---|---|---|
| Language model serving | vLLM, TensorRT-LLM, or NVIDIA NIM on local GPUs | Model weight downloads, telemetry, license checks |
| Embedding and reranking | Self-hosted models on the same or adjacent GPUs | Same as above; some libraries auto-download on first run |
| Vector database | Qdrant, Milvus, or pgvector on local storage | Cloud-hosted variants of the same tool default to a managed endpoint |
| Document ingestion and OCR | Local parsing libraries and OCR engines | Some cloud-OCR SDKs silently call an external API unless configured locally |
| Connectors to source systems | Direct connections to internal SharePoint, file shares, databases | Connector libraries sometimes route through a vendor's cloud relay by default |
| Monitoring and logging | Self-hosted stack such as Prometheus and Grafana | SaaS observability agents send data externally unless disabled |
Key takeaway: every row in this table needs its network configuration explicitly verified, since a component being self-hostable does not guarantee its default configuration stays local.
Planning for updates without internet access
A genuinely air-gapped deployment, with no internet connectivity at all, needs a deliberate process for getting model updates, security patches, and package updates into the environment, since the usual approach of pulling directly from a package registry or model hub is unavailable by design. This typically means maintaining a controlled transfer process, such as a reviewed one-way data diode or a scheduled manual import process, where updates are vetted on a connected staging environment before being physically or procedurally moved into the air-gapped network. Model updates in particular need planning, since a newer open-weight model release cannot simply be pulled on demand the way it can in a connected environment, so teams typically maintain a small internal mirror of approved model versions updated on a fixed cadence.
Key takeaway: air-gapped environments need a controlled, auditable process for bringing in updates, since the default pull-on-demand pattern most tools assume is unavailable by design.
When full air-gap is not actually required
Not every regulated deployment needs a true air-gap; many finance, insurance, and healthcare customers meet their compliance requirements with an on-premise or private-cloud deployment that has no data leaving the organization's control, without going as far as zero internet connectivity, which is a meaningfully lower operational burden. The distinction matters because a true air-gap multiplies the operational cost of updates, monitoring, and troubleshooting, so it should be a deliberate requirement driven by a specific regulatory or contractual obligation, such as classified government work, rather than a default assumption about what "on-premise" means. Clarifying this requirement early, alongside broader questions like GDPR and KVKK compliance for personal data, shapes the architecture significantly.
Key takeaway: confirm whether the requirement is genuinely zero connectivity or simply no data leaving organizational control, since the two demand very different operational investments.
Frequently asked questions
Can embedding and reranking models run on the same GPUs as the main LLM?
Yes, and this is common for cost efficiency, since embedding and reranking models are typically much smaller than the primary language model and can share GPU capacity through techniques like NVIDIA's Multi-Instance GPU or simple co-location, provided the combined memory and throughput needs are sized correctly.
Does on-premise RAG mean giving up managed convenience entirely?
Largely yes for the components that must stay local, though some organizations run a hybrid model, keeping the sensitive document index and model inference fully on-premise while using managed services only for non-sensitive auxiliary functions like anonymized usage analytics, provided that boundary is drawn deliberately.
How do we verify a component makes no outbound network calls?
Run the component in an isolated network namespace with all egress blocked except an explicit allowlist, then monitor for blocked connection attempts during normal operation and startup. Any blocked attempt reveals a hidden dependency that needs to be either disabled through configuration or replaced with a local alternative.
Is on-premise RAG more expensive than cloud-based RAG?
It shifts cost from ongoing usage-based fees to upfront hardware investment and operational staffing for GPU infrastructure and Kubernetes, covered in more depth in our own-GPUs-versus-cloud-API cost comparison. Whether it is cheaper overall depends heavily on scale and how long the system runs.
How Nanobase AI helps
Nanobase AI, a Silicon Valley on-premise AI infrastructure specialist and NVIDIA Inception Program member, deploys complete RAG stacks with zero external dependencies, including the network egress audit that catches hidden telemetry and update calls before they become a compliance surprise. See our solutions for the full architecture.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.