Can Llama 3.3 70B run on a single H100?

Llama 3.3 70B does not fit on a single H100 at FP16, since its roughly 140 GB of weights exceed the H100's 80 GB of HBM3 by a wide margin, but it does fit at FP8, where the weights shrink to about 70 GB, leaving only around 10 GB for KV cache, activation buffers and framework overhead, enough for short context and low concurrency but tight for anything more demanding. INT4 quantization brings the weight footprint down to roughly 38 GB, leaving substantially more room, around 40 GB, for KV cache and concurrent users, which is the more realistic path when the goal is specifically staying on a single H100 rather than moving to a multi-GPU or H200 setup. An H200, with 141 GB of HBM3e, comfortably runs Llama 3.3 70B at FP8 with real headroom for longer context and more concurrent sessions, making it the better single-GPU option when budget allows. The right choice depends on whether the deployment prioritizes staying on existing H100 hardware or getting more serving headroom from a single card. Nanobase AI, an NVIDIA Inception Program member, tests both the FP8 and INT4 paths on customer H100 hardware before recommending which one to run in production.

Turning "does it fit" into "how well does it serve"

Fitting on a single H100 and serving it usefully are two different questions. FP8's roughly 70 GB weight footprint technically fits inside 80 GB, but the remaining 10 GB, after CUDA and serving engine overhead, leaves very little room for KV cache, which caps both context length and concurrent users at levels that may not meet real production needs. The practical question for a single-H100 deployment of Llama 3.3 70B is not whether it loads, but how many simultaneous users and how much context it can actually serve before hitting the memory ceiling.

Estimated concurrency ceiling by precision on one H100

*Illustrative, based on roughly 1 GB of KV cache per session at 4K tokens for a 70B-class model; actual figures depend on serving engine configuration and real context length distribution.

This table is why INT4 is often the more realistic single-H100 path for Llama 3.3 70B specifically when the goal is staying on one card: FP8 leaves barely enough room for a handful of short-context sessions, while INT4 opens up meaningfully more concurrency at a modest, generally acceptable accuracy cost for most production use cases.

Deploying the INT4 path in practice

1. Choose a quantization format, AWQ or GPTQ are the most common production-ready options for this model size, and validate output quality on representative prompts before committing.
2. Set a conservative gpu_memory_utilization initially (see GPU memory overhead beyond weights) and measure real KV cache usage under test traffic rather than assuming the full remaining 40 GB is usable.
3. Cap maximum context length deliberately if the use case does not need Llama 3.3's full context window, since a small number of long-context sessions can consume a disproportionate share of the INT4 configuration's KV cache room.
4. Load-test at expected peak concurrency before launch, since the table above is a starting estimate, not a guarantee, and real prompt lengths vary by workload.
5. Plan the upgrade path to a second H100 or an H200 if concurrency or context needs grow past what INT4 on one card comfortably supports.

When to skip the single-H100 constraint entirely

If the existing hardware is already H100 and staying on a single card is a hard requirement, INT4 is the practical answer. But if the deployment is not yet locked into H100 specifically, an H200, with 141 GB of HBM3e, runs Llama 3.3 70B at FP8 with real headroom for longer context and meaningfully more concurrent sessions on a single card, avoiding the quantization trade-off altogether. The decision comes down to whether the constraint is "we already own H100 hardware" or "we want a single GPU," since those two constraints point to different answers.

Frequently asked questions

Does INT4 quantization noticeably hurt Llama 3.3 70B's output quality?

INT4 quantization on a 70B model typically shows a small, measurable quality difference from full precision on standard benchmarks, generally acceptable for most production use cases, though testing against the specific target task remains the reliable way to confirm this, as discussed in quality loss of 4-bit quantization on a 70B model.

Can I run Llama 3.3 70B on a single H100 at FP16 with any workaround?

Not without CPU or disk offloading, which reduces throughput substantially and is generally impractical for real-time multi-user serving; FP16 on a single 80 GB H100 requires more memory than the card has for the weights alone.

Is two H100s a better option than one H100 with INT4?

Two H100s running FP8 with tensor parallelism gives more total memory, better model quality than INT4, and hardware redundancy, at roughly double the GPU cost; it is generally the better choice when budget allows and the deployment needs to grow.

How much concurrency is actually needed to decide between FP8 and INT4 on one H100?

If expected peak concurrency is in the single digits with short context, FP8 may be workable; anything beyond that on a single H100 points toward INT4, a second GPU, or an H200, depending on which trade-off, precision or budget, matters more for the use case.

How Nanobase AI helps

Nanobase AI, an NVIDIA Inception Program member, tests both the FP8 and INT4 paths on customer H100 hardware before recommending which one to run in production, validating real concurrency and quality against the target workload rather than a generic benchmark.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.