How to deploy an LLM on-premise in 2026: architecture, stack, cost

Deploy an LLM on-premise in 2026 by running an open-weight model on NVIDIA GPU servers, serving it with vLLM, TensorRT-LLM or NVIDIA NIM, and exposing it to applications only through an OpenAI-compatible gateway that enforces SSO, quotas and audit logging. Add a RAG layer with a vector database for company knowledge, and build guardrails, evaluation and observability into the first release. Budget the platform as GPU capex amortized over 3 to 5 years plus power, support and staff opex, and size GPUs on weights plus KV cache, never on weights alone.

When on-premise beats cloud

Cloud APIs remain the right default for low, spiky or exploratory usage. On-premise wins when at least one of four conditions holds.

Data sovereignty. Prompts, retrieved documents and model outputs never leave your network, so there is no vendor data processing agreement to negotiate and no cross-border transfer analysis to defend.

Steady high volume. Owned GPUs are a fixed monthly cost, so cost per token falls as utilization rises; sustained load usually beats metered APIs on price, an idle platform does not. The calculation is in own GPUs vs cloud API cost per token.

Latency. Removing the internet round trip and placing the model next to the systems it calls gives predictable p99 latency, which matters most for agents making many sequential calls per task.

Regulation. The EU AI Act, GDPR, KVKK and sector rules for banking, healthcare and defence can require processing in a specific jurisdiction on infrastructure your auditors can inspect end to end. See the EU AI Act, GDPR and KVKK checklist.

Key takeaway: choose on-premise when data, volume, latency or regulation demand it; otherwise start in the cloud and keep the option open by building on an OpenAI-compatible API from day one. Many enterprises end up hybrid: regulated workloads on-premise, burst traffic in the cloud.

Reference architecture

As of 2026 a production on-premise LLM platform has six layers; each is replaceable, but the layering should not be skipped.

• GPU servers. Nodes with 4 or 8 NVIDIA GPUs: H100 (80 GB HBM3, 3.35 TB/s, 700 W), H200 (141 GB HBM3e, 4.8 TB/s), B200 (~180 GB HBM3e, ~8 TB/s, ~1000 W) or RTX PRO 6000 (96 GB GDDR7) for smaller models. NVLink within a node; InfiniBand or RoCE only when one model spans nodes.
• Platform. Linux, NVIDIA driver, CUDA, container toolkit and Kubernetes with the NVIDIA GPU Operator; local NVMe for the model cache, shared storage for the registry and RAG corpus.
• Serving layer. vLLM, TensorRT-LLM or NIM containers, one per model, with tensor parallelism, continuous batching and paged KV cache. FP8 on Hopper and Blackwell roughly halves memory versus FP16 with little quality loss.
• OpenAI-compatible gateway. The only endpoint applications may call. It terminates SSO (OIDC), issues service keys, enforces per-team quotas, routes across models, redacts PII and writes every request and response to the audit store.
• RAG and vector database. Ingestion, chunking, an embedding model, a vector database and a reranker, with document permissions copied from the source system and enforced at query time.
• Applications and agents. Chat interfaces, copilots, workflow automations and MCP servers that connect the model to SAP, Salesforce, Microsoft 365 or Snowflake.

Beside the stack sit SSO and RBAC, immutable audit logs shipped to your SIEM, input and output guardrails, and observability for GPU health, tokens per second, time to first token and queue depth. Key takeaway: applications never talk to the serving layer directly; the gateway is where security, cost attribution and auditability live.

Software stack by layer

Key takeaway: pick one option per layer, pin its version and write down why; in 2026 discipline matters more than novelty. A detailed serving comparison is in vLLM vs TensorRT-LLM vs Ollama vs SGLang.

Step-by-step deployment plan

1. Define the workload. Use cases, concurrent users, peak requests per minute, context length and latency targets. Shortlist two or three models and record their licence terms.
2. Size the GPUs. Memory need is weights plus KV cache plus runtime overhead: a 70B model is about 140 GB in FP16, 70 GB in FP8 and ~38 GB in INT4, plus 20 to 50% for KV cache. In Nanobase AI sizing reviews, KV cache is the item most often left out. Sizing tables are in how many GPUs for 70B, 405B and DeepSeek R1.
3. Procure and prepare the facility. Confirm rack space, power (an 8-GPU H100 server draws about 10 kW under load), cooling and network before ordering. Server lead times are often the longest item on the plan.
4. Install the platform. OS, drivers, container toolkit, Kubernetes and the GPU Operator, then burn-in and DCGM diagnostics before any model touches the hardware.
5. Bring up the serving layer. Deploy one model, enable FP8 where supported, and benchmark against the step 1 targets. A minimal single-node vLLM start:

   docker run --gpus all --ipc=host -p 8000:8000 \
     -v /models:/models \
     vllm/vllm-openai:latest \
     --model /models/your-70b-instruct \
     --tensor-parallel-size 4 \
     --quantization fp8 \
     --max-model-len 32768 \
     --served-model-name llm-70b

1. Deploy the gateway. Wire it to SSO, create service keys per application, set quotas and enable request logging. From here, no application calls the serving layer directly.
2. Build RAG. Start with one high-value corpus and measure retrieval quality on a labelled question set before adding sources.
3. Harden security. Network segmentation for GPU nodes, secrets in a vault, guardrails, PII redaction, audit logs to the SIEM and a written retention policy.
4. Evaluate. Golden set, load test at peak concurrency and a red-team pass; fix or formally accept every finding before go-live.
5. Go live and operate. Pilot group first, a rollback path to the previous model version, and a named owner with runbooks for patching, model updates, capacity review and incidents.

Key takeaway: the serving layer is the easiest part; the gateway, RAG quality and operations are where deployments succeed or stall.

Cost structure: capex, opex and amortization

The table lists cost components rather than totals, because vendor quotes vary; verify current pricing for your region and configuration.

Amortize capex over 3 years if you expect to refresh to the next GPU generation quickly, or over 5 years if the workload is inference-only and stable. Monthly platform cost is capex divided by the amortization months plus monthly opex; dividing by monthly tokens gives cost per token.

Utilization is the largest lever: because monthly cost is nearly fixed, the same hardware at 20% utilization costs about three times more per token than at 60%. Key takeaway: model the platform as a fixed monthly cost and treat utilization as the variable; buy for the sustained load, not the peak, and fill idle capacity with off-peak batch jobs.

Air-gapped deployments

Air-gapped sites (defence, critical infrastructure, some banks) run the same architecture with no path to the internet. The differences are procedural.

• Offline bundles. Weights, container images and Helm charts are downloaded on a connected staging network, hashed, signed and moved by approved media into the private registry.
• Private registry and mirrors. Harbor for containers, a local model store and a mirror of OS and Python packages; NIM containers can run from a pre-populated local model cache.
• Licensing and telemetry. Confirm offline activation for NVIDIA AI Enterprise before purchase; disable usage reporting in every component and verify with egress monitoring.
• Change control. Updates follow a scheduled cadence with a full evaluation run, because there is no quick hotfix path.
• Internal PKI and time. An internal CA and internal NTP; both are common causes of failed offline installs.

Key takeaway: build and test the full stack on a connected staging replica first, then transfer signed artifacts; never debug for the first time inside the air gap.

Common mistakes

1. Sizing on weights alone. KV cache and concurrency push a 70B FP16 model past two H100s; budget headroom.
2. Buying before benchmarking. Run the candidate model on rented GPUs first, then order.
3. Applications calling vLLM directly. No authentication, no audit trail, no way to swap models without touching every application.
4. Treating RAG as a one-week task. Chunking, permissions and retrieval evaluation take longer than the model deployment.
5. No high availability. One node with one model is a demo; plan at least two replicas for production.
6. Ignoring model licences. Open weights are not all Apache 2.0; check commercial use and attribution terms per model.
7. No owner after go-live. Assign a team, an on-call rotation and a budget for model updates before the pilot ends.

Key takeaway: most failed on-premise projects fail on sizing, security or ownership, not on the model.

Frequently asked questions

How many GPUs do I need to run a 70B model on-premise?

Start from memory: a 70B model needs about 140 GB in FP16, 70 GB in FP8 or ~38 GB in INT4, plus 20 to 50% for KV cache. In FP8 that fits two H100 80 GB GPUs or one H200 141 GB with room for context. For FP16 with long contexts and many concurrent users, four H100s per replica is the practical minimum.

Should I use vLLM, TensorRT-LLM or NVIDIA NIM?

Use vLLM for broad model coverage, fast adoption of new architectures and a fully open-source stack. Use TensorRT-LLM for fixed models on NVIDIA GPUs where you want the highest throughput per GPU. Use NIM for NVIDIA-packaged, supported containers under NVIDIA AI Enterprise. All three expose OpenAI-compatible APIs, so the gateway design stays the same.

Can I deploy an on-premise LLM without Kubernetes?

Yes. A single GPU server with Docker Compose running vLLM, a gateway, a vector database and monitoring is a valid production setup for one or two models with modest traffic. Kubernetes with the GPU Operator becomes worthwhile when you run several models, need rolling updates and autoscaling across nodes, or already operate Kubernetes elsewhere.

Is on-premise LLM deployment cheaper than cloud APIs?

It depends almost entirely on sustained utilization. Owned GPUs are a fixed monthly cost, so cost per token drops as usage rises, while metered APIs scale linearly. High, steady daily volume favours on-premise; low or spiky usage favours the cloud. Compute both with your own token volumes and verify current hardware and API pricing before deciding.

Do I need NVIDIA AI Enterprise licences?

Only if you use NVIDIA NIM or want NVIDIA-supported containers, drivers and support SLAs. vLLM, TensorRT-LLM, the GPU Operator and DCGM are open source or freely available and run without a licence. Many enterprises run open-source serving with support from an integration partner; others prefer the NVIDIA-backed path for audit reasons. Verify current terms with NVIDIA.

How do I update models in an air-gapped environment?

Download and evaluate the new model on a connected staging replica, record its hash and licence, sign the bundle and transfer it by approved media into the private registry. Run the same evaluation suite inside the air gap, deploy the new version beside the old one through the gateway, shift traffic gradually and keep the previous version for rollback.

How Nanobase AI can help

Nanobase AI designs, builds and operates on-premise LLM platforms end to end: GPU sizing and procurement support for H100, H200, B200 and RTX PRO systems; installation with Kubernetes, the GPU Operator, MIG and InfiniBand; serving with vLLM, TensorRT-LLM or NIM; an OpenAI-compatible gateway with SSO and audit logging; RAG and fine-tuning; guardrails, observability and compliance mapping for the EU AI Act, GDPR and KVKK; and air-gapped delivery where required. We are headquartered in Silicon Valley and are a member of the NVIDIA Inception Program. See our solutions or book a live demo to see a reference deployment running.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.