Google Cloud offers H100 GPUs through the A3 and A3 Mega machine types and B200 GPUs through the newer A4 machine type, with A3 Ultra sitting in between on H200. A3 instances pair eight NVIDIA H100 GPUs with 80 GB of HBM3 each at 3.35 TB/s of bandwidth, while A3 Mega adds doubled GPU to GPU networking bandwidth for large distributed training jobs. A3 Ultra moves to H200 GPUs with 141 GB of HBM3e at about 4.8 TB/s, aimed at memory heavy inference and fine tuning. The A4 machine type introduces NVIDIA B200 GPUs with roughly 180 GB of HBM3e per GPU and about 8 TB/s of bandwidth, and Google also offers A4X built around GB200 NVL72 with Grace CPUs for the largest training clusters. Choosing between these depends on whether the workload is compute bound training, which benefits most from A4 or A4X, or memory bound inference, where A3 Ultra or even standard A3 is often sufficient and easier to get quota for. Nanobase AI, a Silicon Valley enterprise AI engineering company, sizes Google Cloud GPU machine types against actual model and throughput requirements before committing to a configuration.
The networking layer is what actually separates these machine types
It is tempting to pick a Google Cloud GPU machine type purely by GPU generation, but A3, A3 Mega, A3 Ultra and A4 differ just as much in GPU-to-GPU networking as in silicon. A3 uses standard networking suited to single-node or lightly distributed inference, A3 Mega doubles GPU-to-GPU bandwidth for larger distributed training jobs, and A3 Ultra and A4 move to even higher-bandwidth fabrics purpose-built for the largest training clusters. A workload that never spans more than one node's GPUs gains little from the networking upgrades that justify A3 Mega or A4 Ultra's higher cost, which makes networking requirements, not GPU generation alone, the first filter to apply.
A decision matrix by workload shape
| Workload | Recommended machine type | Why |
|---|---|---|
| Single-node inference, 70B or smaller in FP8 | A3 (H100) | Standard networking is sufficient; easier quota |
| Long-context or high-concurrency inference | A3 Ultra (H200) | Extra HBM3e memory per GPU reduces sharding needs |
| Multi-node distributed fine-tuning | A3 Mega (H100) | Doubled inter-GPU bandwidth supports gradient sync at scale |
| Largest model pretraining or frontier-scale training | A4 (B200) or A4X (GB200 NVL72) | Highest bandwidth and memory for compute-bound training |
The matrix's takeaway is that most production inference workloads belong on A3 or A3 Ultra, while A3 Mega, A4 and A4X earn their cost mainly in training, not serving.
Spot, reservation and committed use options on Google Cloud
Google Cloud offers the same fundamental capacity trade-offs as other hyperscalers: on-demand pricing for flexibility, committed use discounts for predictable long-term workloads, and reservations for guaranteed capacity on a specific machine type and region. Spot VMs are available for GPU machine types and can meaningfully lower cost for interruption-tolerant batch inference or evaluation jobs, but Google can reclaim spot capacity with short notice, making it unsuitable for latency-sensitive production serving. For workloads with a known future start date, reserving capacity ahead of time is generally the more reliable path during periods of tight H100 or B200 supply, since on-demand availability for the newest machine types is not guaranteed even with approved quota. Spot capacity is a cost lever for interruption-tolerant work only; it should never be the primary plan for serving live production traffic.
Getting GPU quota approved on Google Cloud
Quota for A3, A3 Mega, A3 Ultra or A4 machine types is managed through the IAM and Admin quotas page, but self-service requests for the largest allocations are frequently routed to the Google Cloud account team for manual review rather than approved automatically. A request that specifies the exact machine type, region, GPU count, and a concrete business justification tends to move faster than a generic increase request. For enterprises without an existing Google Cloud relationship, engaging a partner or the account team before submitting a large request generally improves both approval odds and turnaround time, particularly for A4 and A4X capacity, which remains tightly allocated as of 2026. A specific, well-justified request tied to a named workload consistently outperforms a generic quota increase ask, regardless of account size.
Frequently asked questions
Is A3 Ultra the same GPU as A3 Mega?
No. A3 and A3 Mega both use NVIDIA H100 GPUs, differing mainly in inter-GPU networking bandwidth. A3 Ultra moves to NVIDIA H200 GPUs with more memory per GPU, aimed at memory-bound inference and fine-tuning rather than networking-bound training.
Do I need A4 for a 70B-parameter model?
Usually not. A 70B model served in FP8 fits comfortably on A3 or A3 Ultra with standard vLLM or TensorRT-LLM serving. A4's B200 GPUs and higher bandwidth matter more for much larger models or very high-throughput training.
Can I use TPUs instead of these GPU machine types on Google Cloud?
Yes, Google Cloud also offers TPU v5e, v5p, and the newer Trillium generation as an alternative to GPU machine types, though TPUs typically require a JAX or PyTorch XLA-based workflow rather than a standard CUDA stack.
How does A4X differ from A4?
A4X pairs NVIDIA GB200 NVL72 with Grace CPUs in a tightly coupled architecture aimed at the largest training clusters, while A4 offers B200 GPUs in a more conventional node configuration better suited to a broader range of training and inference workloads.
How Nanobase AI helps
Nanobase AI, an enterprise AI engineering company operating in the Google and Microsoft ecosystems, sizes Google Cloud GPU deployments against actual workload shape rather than defaulting to the newest machine type, and helps enterprises secure A3, A3 Ultra or A4 quota with a justification that reflects real usage. Our GPU sizing guidance and cloud and hybrid solutions cover the same decision framework in more depth.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.