A single 8-GPU server, providing up to 640 GB of combined VRAM on H100 or 1,128 GB on H200, covers the large majority of enterprise LLM inference workloads, including most dense models up to around 400 billion parameters at FP8 and mixture-of-experts models like DeepSeek R1 at FP8 with reasonable headroom, so multiple servers are usually not necessary purely for model size. Multiple servers become necessary when high availability with automatic failover is required, when aggregate throughput across many concurrent users exceeds what one node's compute can sustain even with the model comfortably fitting in memory, or when running multi-node training or fine-tuning workloads that require InfiniBand-connected clusters larger than a single node. For most inference-focused enterprise deployments serving a single department or company-wide chatbot, one well-specified 8-GPU node with a redundant power supply and adequate cooling is a reasonable starting architecture, with a second node added later for redundancy rather than sized in from the start. The decision should be driven by actual availability requirements and a measured throughput ceiling, not by an assumption that more servers automatically means more reliability. Nanobase AI designs single-node and multi-node architectures based on the customer's specific availability and throughput requirements.
Model size rarely forces multiple servers on its own
A single 8-GPU server provides up to 640 GB of combined VRAM on H100 or roughly 1,128 GB on H200, which covers the large majority of enterprise LLM inference workloads, including most dense models up to around 400 billion parameters at FP8 and mixture-of-experts models like DeepSeek R1 at FP8 with reasonable headroom. If the driving question is "will the model fit," a single well-specified 8-GPU node almost always answers yes for anything short of the very largest frontier models, which means the real decision usually comes down to factors other than raw model size.
The three conditions that actually require multiple servers
| Condition | Why one node isn't enough |
|---|---|
| High availability with automatic failover | A single node is a single point of failure; if it goes down for hardware fault or maintenance, service stops entirely |
| Aggregate throughput exceeds one node's compute | Even with the model comfortably fitting in memory, request volume can exceed what one node's GPUs can process, regardless of memory headroom |
| Multi-node training or fine-tuning at scale | Large training jobs need InfiniBand-connected clusters larger than a single node's GPU count to reach target batch sizes or training speed |
Notably, none of these three conditions is "the model doesn't fit," which is the assumption many teams start from when considering a multi-server buildout; memory capacity and compute throughput are separate constraints, and it is entirely possible to have abundant memory headroom on one node while still needing a second node purely for throughput or redundancy.
A practical starting architecture
For most inference-focused enterprise deployments serving a single department or a company-wide chatbot, one well-specified 8-GPU node with a redundant power supply and adequate cooling is a reasonable starting architecture. Adding a second node for redundancy afterward, once the first node's real throughput ceiling and actual uptime requirements are understood, is generally a better sequence than sizing for multiple nodes from the start based on an assumption that more servers automatically means more reliability, an assumption that is not always true if the second node is not properly load-balanced and monitored.
- Deploy the first node and measure real throughput under production-representative load to establish an actual compute ceiling, not a theoretical one.
- Define the actual availability requirement for the specific use case, an internal tool tolerating occasional downtime has very different needs than a customer-facing, revenue-linked service.
- Add a second node when either the throughput ceiling is approached or the availability requirement genuinely demands automatic failover, configuring load balancing and health checks across both nodes at that point.
- Reserve InfiniBand-connected multi-node clusters specifically for training and fine-tuning workloads that need it, rather than assuming inference and training should share the same multi-node architecture by default.
Why the reliability assumption deserves scrutiny
"More servers means more reliability" is only true if the additional server is actually configured for failover, with load balancing, health checks and traffic routing in place; simply adding a second node without that configuration adds cost without adding the resilience it was meant to provide. The decision should be driven by actual availability requirements and a measured throughput ceiling, not by an assumption, which means the architecture conversation belongs alongside a concrete SLA or uptime target, not before one exists.
Frequently asked questions
Can a single 8-GPU node serve a 671B-parameter MoE model like DeepSeek R1?
At FP8, DeepSeek R1's roughly 671 GB weight footprint fits within an 8x H100 node's 640 GB only with essentially no headroom, making an 8x H200 node, with over 1,100 GB, the more realistic single-node option; see GPUs needed to run DeepSeek R1 671B on-premise for the full breakdown.
How do I know when my throughput needs exceed one node?
Monitoring real request queue depth, latency percentiles, and GPU compute utilization under production load, not a theoretical estimate, is the reliable way to identify when a single node's aggregate throughput is becoming the bottleneck rather than model memory.
Is Kubernetes GPU Operator or Slurm needed for a single-node deployment?
Kubernetes GPU Operator adds value even on a single node for scheduling, monitoring and lifecycle management, while Slurm is more commonly reserved for multi-node batch and training workloads; see Kubernetes GPU Operator vs. Slurm for the full comparison.
Does adding a second node double infrastructure cost for the same workload?
Roughly, yes, in hardware terms, though a properly configured second node also adds redundancy value that a single node cannot provide, so the cost comparison should weigh against the cost of downtime for the specific use case, not against hardware price alone.
How Nanobase AI helps
Nanobase AI designs single-node and multi-node architectures based on the customer's specific availability and throughput requirements, configuring load balancing and failover only where the actual SLA justifies the added complexity and cost.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.