Yes, a 70 billion parameter model runs well on two RTX PRO 6000 GPUs, which together provide 192 GB of VRAM, enough for the roughly 140 GB of FP16 weights plus meaningful headroom for KV cache and several concurrent users, with even more room to spare when the model is served in FP8, about 70 GB, or INT4, about 38 GB. Tensor parallelism splits the model evenly across the two cards, and the RTX PRO 6000's 96 GB per GPU makes it one of the few workstation-class cards that can hold half of a 70B model's FP16 weights on its own. This configuration is popular for on-premise deployments that want data-center-class capacity without the power, cooling and cost profile of a full H100 or H200 server. The trade-off versus data-center GPUs is lower memory bandwidth and no NVLink between the two cards, so multi-GPU communication runs over PCIe, which can limit throughput under heavy concurrent load compared to an NVLink-connected H100 pair. For most single-department or mid-size company chatbot workloads this is not a practical bottleneck. Nanobase AI installs and tunes two and four GPU RTX PRO 6000 clusters for exactly this class of on-premise 70B deployment.
What you gain and give up versus an H100 pair
Two RTX PRO 6000 GPUs and two H100s both provide enough combined memory for a 70B model at FP16, but the similarity mostly ends there. The comparison below focuses on what actually differs for a real deployment.
| Factor | 2x RTX PRO 6000 | 2x H100 |
|---|---|---|
| Combined VRAM | 192 GB GDDR7 | 160 GB HBM3 |
| Memory bandwidth per GPU | Lower than HBM3 | 3.35 TB/s |
| GPU-to-GPU link | PCIe (no NVLink) | NVLink, 900 GB/s |
| Power draw | Meaningfully lower | Higher, ~700W per GPU |
| Typical use case fit | Department or mid-size on-premise deployment | High-concurrency, latency-sensitive serving |
| Relative capital cost | Lower | Higher |
Key takeaway: RTX PRO 6000 pairs trade some throughput ceiling for materially lower cost and power draw, which is the right trade for many on-premise deployments below very high concurrency.
Where PCIe communication actually becomes a bottleneck
Tensor parallelism splits each layer's computation across GPUs and requires an all-reduce communication step after nearly every layer, so the GPU-to-GPU link's bandwidth directly affects throughput. NVLink's 900 GB/s on H100 makes this communication nearly free relative to the compute; PCIe Gen5's roughly 64 GB/s per direction is over an order of magnitude slower, so a two-GPU RTX PRO 6000 setup pays a real, measurable tax on every forward pass.
In practice this tax is most visible at high batch sizes and high concurrency, where more communication happens per unit of time and the PCIe link saturates. At low to moderate concurrency, typical of a single department's chatbot or an internal tool with dozens rather than hundreds of simultaneous users, the GPUs' compute is usually the limiting factor before PCIe bandwidth is, which is why this configuration remains popular despite the interconnect gap.
Key takeaway: PCIe communication overhead scales with concurrency, so this configuration is well suited to moderate load and less suited to high-concurrency production serving.
Getting a clean configuration running
- Confirm both GPUs are visible and correctly enumerated with
nvidia-smi, and that driver versions match across both cards. - Launch the serving engine with tensor parallel size 2, using a framework such as vLLM that supports PCIe-connected multi-GPU serving without requiring NVLink.
- Start with FP8 or INT4 weights rather than FP16 to reduce both memory pressure and the volume of data moved across PCIe during inference.
- Benchmark at your expected concurrency before finalizing the configuration, since PCIe overhead only shows up clearly under realistic load, not in a single-request smoke test.
- Monitor GPU utilization versus PCIe link utilization during load testing to confirm which resource is actually the bottleneck for your workload.
Key takeaway: getting a PCIe-linked pair to perform well is mostly about matching quantization and concurrency expectations to the interconnect's real limits, not about exotic configuration.
How much headroom each precision actually leaves
At FP16, 70B weights (~140 GB) leave only about 52 GB of the 192 GB combined for KV cache and overhead across both cards, which is workable for moderate concurrency but not generous. At FP8, weights (~70 GB) leave roughly 120 GB free, comfortable for dozens of concurrent 4K-token sessions with room to spare. At INT4, weights (~38 GB) leave over 150 GB free, more than most department-scale deployments will ever use, at the cost of the accuracy trade-off quantization always carries. For this hardware pairing, FP8 is usually the right default: it keeps quality close to FP16 while leaving far more of the 192 GB available than FP16 does.
Key takeaway: FP8 is the sweet spot on a two-card RTX PRO 6000 setup, leaving generous KV cache headroom without INT4's accuracy trade-off.
Frequently asked questions
Do I need NVLink to run a 70B model on two GPUs?
No. Tensor parallelism works over PCIe, just with lower inter-GPU bandwidth than NVLink provides. It is a performance trade-off, not a functional blocker, and for moderate concurrency workloads the difference may not be noticeable.
Is the RTX PRO 6000 a data-center GPU?
It is a workstation-class GPU built on the Blackwell architecture with 96 GB of GDDR7 memory, positioned between consumer cards and data-center parts like the H100. It lacks some data-center features such as NVLink but offers substantially more memory than most workstation cards.
Can I add a third RTX PRO 6000 later to this setup?
Yes, tensor parallelism can extend to more GPUs, though each additional GPU without NVLink adds more PCIe communication overhead, so returns diminish faster than they would on an NVLink-connected node.
Is this configuration good enough for a customer-facing production chatbot?
For low to moderate traffic, yes, particularly for internal or department-scale deployments. For a customer-facing service with unpredictable high concurrency and strict latency requirements, benchmarking against your actual traffic pattern before committing is essential, since that is where PCIe overhead is most likely to show up.
How Nanobase AI helps
Nanobase AI installs and tunes two and four GPU RTX PRO 6000 clusters for on-premise 70B deployments, configuring tensor parallelism, quantization and serving-engine settings to get the most out of PCIe-connected hardware, and benchmarks the result against an H100 or H200 alternative so the trade-off is measured rather than assumed. See our solutions page or explore the sizing trade-offs in our H100 vs H200 vs B200 guide.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.