DeepSeek R1 has about 671 billion total parameters with roughly 37 billion active per token in its mixture-of-experts design, and since inference memory scales with total parameters rather than active ones, the native FP8 weights need approximately 670 GB. That fits on a single 8x H200 server, which offers 1,128 GB of HBM3e, with headroom left for KV cache and concurrent requests, while an 8x H100 node only provides 640 GB and typically cannot hold the full FP8 model plus serving overhead, pushing teams toward two H100 nodes with tensor and pipeline parallelism over InfiniBand or toward more aggressive INT4 quantization to fit a single node. INT4 brings the weight footprint down to roughly 340 to 380 GB, which does fit comfortably on 8x H100 with room to spare. Running DeepSeek R1 on-premise also demands strong InfiniBand or high-bandwidth NVLink interconnect because of the model's expert-routing communication pattern. Distilled DeepSeek R1 variants at 7B to 70B are a far lighter alternative when full accuracy is not required. Nanobase AI, headquartered in Silicon Valley, has sized and deployed on-premise DeepSeek R1 clusters on both H100 and H200 hardware for customers requiring full data sovereignty.
Full model versus distilled variants: a sizing comparison
The full 671B DeepSeek R1 is only one point on a spectrum. DeepSeek also released distilled variants that transfer R1's reasoning behavior into much smaller dense models, and for many on-premise deployments the distilled sizes are the more realistic starting point.
| Variant | Approx. parameters | FP16 weights | INT4 weights | Realistic hardware |
|---|---|---|---|---|
| R1 (full, MoE) | 671B total, 37B active | Not typically run in FP16 | ~340-380 GB | 8x H200, or two H100 nodes |
| R1-Distill-Llama-70B | 70B dense | ~140 GB | ~38 GB | 2x H100/A100, or 1x H200 |
| R1-Distill-Qwen-32B | 32B dense | ~64 GB | ~16-18 GB | 1x H100, or 1x 48 GB card at INT4 |
| R1-Distill-Qwen-14B | 14B dense | ~28 GB | ~8 GB | 1x 24 GB GPU |
| R1-Distill-Qwen-7B | 7B dense | ~14 GB | ~4 GB | 1x 16-24 GB GPU |
Key takeaway: full R1 needs a multi-GPU node, but a distilled variant can deliver reasoning-style behavior on hardware as modest as a single 24 GB card.
Interconnect and facility requirements for the full model
Running full R1 on-premise is as much a networking and facilities decision as a GPU decision. The model's mixture-of-experts routing generates significant inter-GPU traffic, so within a node NVLink bandwidth directly affects throughput, and across nodes, if the deployment spans more than one 8-GPU server, InfiniBand at high bandwidth is effectively required rather than optional. An 8-GPU H200 or H100 server also draws roughly 10 to 11 kW at full load, which exceeds what many existing enterprise racks were provisioned for, so confirming power and cooling capacity is a prerequisite step, not an afterthought.
Two H100 nodes connected over InfiniBand can hold full R1 in FP8 by splitting the model with tensor and pipeline parallelism, but the cross-node communication adds latency that a single H200 node avoids entirely. For teams choosing between the two paths, a single H200 node is usually the simpler and faster option when the budget allows it.
Key takeaway: full on-premise R1 needs data-center-grade power, cooling and interconnect, not just enough raw GPU memory.
Why 37B active parameters don't shrink the memory footprint
R1's mixture-of-experts router activates only about 37 billion of its 671 billion parameters for any given token, which is what makes it fast relative to a dense model of similar total size, but every expert still has to sit resident in GPU memory since the router can route any token to any expert. Sizing from the active-parameter figure is the single most common mistake teams make with this model, leading directly to under-provisioned hardware that fails once real traffic exercises experts that a smaller test workload happened not to touch.
Key takeaway: budget on-premise R1 memory from the full 671B parameter count, never from the 37B active-parameter figure that describes compute cost per token.
Planning steps for an on-premise R1 build
- Decide between the full model and a distilled variant based on accuracy requirements measured against your own tasks, not assumed capability.
- Choose 8x H200 for the simplest single-node full-model path, or two H100 nodes with InfiniBand if H200 is not available.
- Confirm rack power (10-11 kW per 8-GPU server) and cooling capacity before hardware arrives.
- Select a serving engine with mixture-of-experts support, such as vLLM or SGLang, and configure expert and tensor parallelism together.
- Load-test with realistic concurrency before finalizing the InfiniBand topology for any multi-node configuration.
Key takeaway: a full-model on-premise R1 deployment is a sequence of hardware, network and software decisions, not a single GPU purchase.
Frequently asked questions
Is a distilled DeepSeek R1 variant a good substitute for the full model?
For many tasks, yes, particularly general reasoning and coding at a smaller scale, though distilled models do not fully match the full 671B model's capability on the hardest reasoning benchmarks. Evaluate a distilled variant against your own tasks before assuming it can replace the full model.
Can DeepSeek R1 run on a single H100 server at all?
Only a distilled variant, not the full 671B model. A single 8x H100 server provides 640 GB, which is tighter than the roughly 670 GB the full FP8 model needs once serving overhead is included, so full R1 on one H100 node typically requires INT4 quantization to fit comfortably.
How much InfiniBand bandwidth does an on-premise R1 deployment need?
For a single-node deployment, InfiniBand is not required since NVLink handles intra-node communication. For a multi-node deployment splitting the model across servers, high-bandwidth InfiniBand, commonly 400 Gb/s per GPU in modern builds, is needed to avoid the network becoming the throughput bottleneck.
Does running R1 on-premise avoid all data governance concerns?
On-premise deployment keeps inference and data within your own infrastructure, which addresses data residency and access-control concerns that come with a hosted API, but it does not by itself satisfy every compliance requirement; logging, access controls and monitoring still need to be designed around your specific regulatory obligations.
How Nanobase AI helps
Nanobase AI, headquartered in Silicon Valley, has sized and deployed on-premise DeepSeek R1 clusters on both H100 and H200 hardware, including the InfiniBand and power planning that full-scale deployment requires, and helps customers evaluate whether a distilled variant meets their accuracy bar before committing to a larger cluster. Our on-premise LLM deployment guide covers the full facilities checklist, and our solutions page details the end-to-end service.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.