An on-premise LLM needs GPUs with enough memory to hold the model weights plus a working buffer for the KV cache, sized to the specific model and expected concurrency. As a reference point, a 70B parameter model needs about 140 GB of GPU memory in FP16, roughly 70 GB in FP8, or around 38 GB in INT4, plus 20 to 50 percent extra headroom for the KV cache under concurrent users. That points to a single NVIDIA H100, with 80 GB of HBM3 and 3.35 TB/s of bandwidth, or an H200, with 141 GB of HBM3e and 4.8 TB/s, for a quantized 70B model, or multiple GPUs linked over NVLink or InfiniBand for larger models like a 405B parameter Llama variant. Beyond GPUs, the build needs sufficient CPU cores and system RAM for data loading, fast NVMe storage for model checkpoints and vector indexes, redundant power and cooling rated for 700W-plus GPUs, and networking capable of feeding multi-GPU workloads without bottlenecking. Smaller deployments for a department or a coding assistant can run on a single RTX PRO 6000 with 96 GB of memory. Nanobase AI sizes and installs this hardware, matching GPU class, count and interconnect to the actual models and concurrency a company plans to run.
GPUs are the headline, not the whole bill of materials
Most conversations about on-premise LLM hardware focus entirely on which GPU to buy, but the GPU is one line item in a stack that also has to include power delivery, cooling, networking, and storage sized to match. A correctly chosen GPU sitting in a rack with insufficient power circuits or cooling capacity will throttle or fail to run at full performance, so the surrounding infrastructure has to be sized alongside the GPU, not after it. A single H100 draws up to 700 W at full load, and a server with eight of them can pull well over 6 kW, which many standard office server rooms are not wired to support.
Sizing the full stack by deployment scale
| Component | Pilot (1 GPU) | Departmental (2–4 GPUs) | Company-wide (8+ GPUs, multi-node) |
|---|---|---|---|
| GPU | 1x H100 or RTX PRO 6000 | 2–4x H100/H200 | 8+ H100/H200/B200, multi-node |
| Power circuit | Standard 208V/30A often sufficient | Dedicated high-density circuit | Datacenter-grade power distribution |
| Cooling | Standard rack cooling may suffice | Enhanced airflow or in-row cooling | Liquid or rear-door cooling likely needed |
| Networking | Standard Ethernet | 25–100 GbE | InfiniBand or 100+ GbE between nodes |
| Storage | Local NVMe, hundreds of GB | NVMe + shared storage, low TB | High-throughput shared storage, tens of TB |
| Typical location | Office server room or colo | Small datacenter or colo | Dedicated datacenter or colo |
The jump from departmental to company-wide is usually where a company's own office infrastructure stops being adequate, forcing a move to a colocation facility or dedicated datacenter. That decision is covered further under hosting an LLM in a colocation datacenter.
Networking matters more than it looks
For a single-GPU deployment, standard networking is fine because there is no inter-GPU communication to worry about. Once a model spans multiple GPUs or multiple nodes, whether for a large model like a 405B parameter model or for high-availability replicas, the interconnect between GPUs becomes a real bottleneck if under-provisioned. Multi-GPU and multi-node deployments need InfiniBand or high-speed Ethernet between nodes, since standard networking becomes the bottleneck long before the GPUs themselves do. This is the same reason large training clusters use InfiniBand, though inference workloads tolerate somewhat more networking latency than training.
A practical procurement checklist
- Confirm the target model and quantization level, since this sets the GPU memory requirement (a 70B model needs about 140 GB in FP16, roughly 70 GB in FP8, or around 38 GB in INT4).
- Add 20 to 50 percent memory headroom for KV cache under expected concurrent users.
- Check the facility's available power circuit capacity against the GPU server's rated draw before ordering hardware.
- Confirm cooling capacity, since GPU servers generate substantially more heat per rack unit than typical enterprise servers.
- Size storage for model weights, checkpoints for any fine-tuning, and document embeddings if retrieval is part of the deployment.
- Plan networking for the actual topology, single-node versus multi-node, before finalizing the network hardware order.
Working through this checklist before a purchase order goes out catches the power and cooling gaps that are far cheaper to fix on paper than after a rack is already installed.
Frequently asked questions
Can existing office servers run an LLM without new hardware?
Only for very small models or CPU-based inference, which is typically too slow for interactive use; meaningful LLM inference needs a GPU, and most existing office servers do not have one, or have consumer GPUs unsuited to sustained enterprise workloads.
Do we need liquid cooling for a small deployment?
No, a single or dual-GPU deployment typically runs fine on standard rack air cooling; liquid or advanced cooling becomes relevant mainly at higher GPU density, generally eight or more GPUs in a rack.
How much storage does model serving actually need?
Model weights alone are modest, often under 500 GB even for large models, but production deployments also need space for logs, embeddings if using retrieval, and multiple model versions during upgrades, so planning several terabytes of fast storage is prudent even for a moderate deployment.
How Nanobase AI helps
Nanobase AI sizes and installs the complete hardware stack for on-premise LLM deployment, not just the GPU choice: power and cooling assessment, networking design for single- or multi-node clusters, and storage planning matched to the model and document scope. The team handles installation and validation on-site or through a colocation partner, so the infrastructure is production-ready rather than a GPU sitting in a rack that cannot run at full capacity.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.