How do I set up shared storage for a GPU cluster?

You set up shared storage for a GPU cluster by choosing between a parallel filesystem such as Lustre or WekaFS for high-throughput training data and checkpoints, and simpler NFS for smaller clusters or lighter-weight workloads where peak throughput matters less than ease of operation. Parallel filesystems like Lustre distribute file data across many storage servers and disks so a single training job can pull aggregate bandwidth far beyond what one NFS server could deliver, becoming necessary once dozens of GPUs are simultaneously reading large datasets or writing multi-gigabyte checkpoints during distributed training. Weka offers similar parallel performance with generally simpler operations than traditional Lustre, at a higher commercial licensing cost, and is popular in newer GPU cluster deployments specifically because it reduces the specialized storage administration skill Lustre traditionally requires. NFS remains reasonable for smaller clusters, fewer than roughly eight to sixteen GPUs, or workloads dominated by inference rather than heavy checkpoint writing, since it is far simpler to deploy and maintain. Whichever system you choose, mount it identically across every compute node so job scripts do not need per-node path adjustments, and size network bandwidth to storage so it does not become the bottleneck. Nanobase AI designs and deploys shared storage sized to actual dataset and checkpoint throughput requirements for each cluster.

Start from the bandwidth math, not the product name

Teams often begin the storage conversation by debating Lustre versus WekaFS versus NFS as if it were a brand preference, but the right answer falls out of a simple calculation: total checkpoint size across all participating GPUs, divided by how long you are willing to stall training to write it, gives the minimum aggregate throughput the storage system must sustain. A cluster that never runs this calculation tends to either overpay for parallel filesystem performance it doesn't need, or discover mid-project that NFS cannot keep up with checkpoint writes from sixty-four GPUs at once.

Working the numbers

1. Estimate per-GPU checkpoint contribution: for a sharded checkpoint, total model and optimizer state size divided by GPU count.
2. Multiply by GPU count to get total checkpoint payload per save event, which for a 70B-parameter model with optimizer state can reach several hundred gigabytes to over a terabyte depending on precision and optimizer choice.
3. Divide by the maximum acceptable write stall time, for example wanting a checkpoint to complete within two minutes, to get required aggregate write throughput.
4. Repeat the same exercise for dataset read throughput during the highest-concurrency training phase, since read and write patterns often need different tuning even on the same filesystem.
5. Compare the result against each storage option's realistic sustained throughput, not its marketing peak figure, and size network bandwidth to storage to match, since an undersized network link turns a fast filesystem into a bottleneck anyway.

Storage options compared

NFS remains entirely reasonable below roughly eight to sixteen GPUs or for workloads dominated by inference rather than heavy checkpoint writing, since its simplicity outweighs the throughput ceiling at that scale. Parallel filesystems become necessary once dozens of GPUs read large datasets or write multi-gigabyte checkpoints simultaneously, since a single NFS server's throughput ceiling gets divided across every concurrent client.

A tiered approach most clusters converge on

Rather than choosing one system for everything, most production clusters end up with a tiered design: a parallel filesystem or WekaFS for hot data actively read and written during training, and object storage for cold archival of completed checkpoints, older datasets, and long-term retention where access latency matters far less than cost per terabyte. This keeps the expensive, high-performance tier sized to active working-set throughput rather than total historical data volume, which is usually a fraction of the cost of sizing one system for everything.

Frequently asked questions

Does every node need identical storage mount paths?

Yes, mounting shared storage at identical paths across every compute node avoids per-node adjustments in job scripts and prevents subtle failures when a job scheduled on a different node than expected cannot find its data at the path it was written with.

How does storage choice interact with checkpoint frequency?

A slower storage tier pushes toward longer intervals between checkpoints, since each write stalls training longer, while faster parallel storage allows more frequent checkpointing without meaningfully impacting total training time. See checkpointing strategy for how the two decisions interact.

Can cloud object storage replace on-premise parallel filesystems entirely?

Generally no for active training, since object storage's latency and throughput characteristics do not match what GPUs need for real-time dataset streaming or fast checkpoint writes, though it works well as the cold-archival tier behind a faster active-data filesystem.

How Nanobase AI helps

Nanobase AI designs shared storage sized to actual dataset and checkpoint throughput requirements, working the same bandwidth math described here against each customer's specific model sizes and training cadence before recommending Lustre, WekaFS, or NFS. That avoids both underpowered storage that bottlenecks GPUs sitting idle and overprovisioned systems that add cost without matching throughput.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.