How do we version datasets with DVC or lakeFS?

Versioning datasets with DVC or lakeFS applies Git-like version control to data files and directories that are too large for Git itself, so a team can track exactly which version of a dataset produced a given trained model and roll back or branch data the same way it branches code. DVC works alongside Git, storing lightweight pointer files in the repository while the actual data lives in remote storage such as S3, GCS or Azure Blob, and it links naturally with existing pipelines since a training run can reference an exact DVC-tracked data version through a normal Git commit. LakeFS takes a different approach, providing a Git-like versioning layer directly on object storage that supports branching, committing and merging entire data lakes without a companion pointer-file workflow, which suits a team doing large-scale data engineering with tools like Spark reading directly from a lake. DVC tends to fit a smaller, ML-focused team already comfortable with Git, while lakeFS fits a larger platform where multiple teams need isolated branches of a shared lake for experimentation without risking production data. Both solve the same reproducibility problem, ensuring an old training run can be exactly recreated. Nanobase AI sets up dataset versioning as a standard part of any training pipeline it builds, choosing the tool that matches existing infrastructure.

Three versions have to line up, not just one

Reproducing a model means reproducing three things at once: the code that trained it, the data it trained on, and the resulting model artifact itself. Git handles the first well, a model registry handles the third, and the middle one is where most teams have the weakest tooling. A training run is only truly reproducible if the exact dataset version, not just "the data as of roughly that time," is recorded alongside the code commit and model version, since even a small silent change to a source table can produce a materially different model.

DVC vs lakeFS: a decision table

DVC suits a team that wants dataset versioning to feel like an extension of a workflow they already trust, Git commits and pull requests, while lakeFS suits a larger platform where multiple teams need isolated branches of a shared lake without risking production data. LakeFS is a natural fit for a team running large-scale data engineering, often with Spark reading directly from the lake, without waiting on a central data team to coordinate every change.

A concrete DVC workflow

dvc add data/training_set_v3.parquet
git add data/training_set_v3.parquet.dvc .gitignore
git commit -m "Add training set v3, deduped and PII-scrubbed"
dvc push

The Git commit records which exact data version paired with which code, so checking out that commit later, then running dvc pull, reconstructs the precise state a training run saw, not an approximation.

What versioning does not solve on its own

Dataset versioning solves reproducibility; it does not solve storage cost or privacy risk on its own. Keeping every historical version of a large dataset indefinitely can become an expensive habit, and an old version containing personal data that has since been requested for deletion under a privacy regulation does not disappear just because a newer version exists. A retention and deletion policy for old dataset versions, tied to the same governance process covering data lineage, needs to exist alongside the versioning system itself.

Frequently asked questions

Do we need both DVC and lakeFS, or just one?

Most teams need only one; they solve the same core problem from different starting points. Choosing based on whether the workflow should feel Git-centric on top of existing repos, DVC, or lake-centric across a shared object store used by multiple teams, lakeFS, avoids running redundant tooling.

Does dataset versioning replace a data quality validation step?

No, versioning tracks what data existed at a point in time; it does not check whether that data was actually good. Data quality validation still needs to run against each version before it feeds a training pipeline.

How large can a dataset be before DVC or lakeFS becomes impractical?

Both are designed for large datasets that exceed what Git can reasonably handle, into the terabyte range and beyond, since the actual data lives in object storage rather than the version control system itself; practical limits come from storage cost and retrieval time, not from the versioning tool.

How Nanobase AI helps

Nanobase AI sets up dataset versioning as a standard part of every training pipeline it builds, choosing DVC or lakeFS based on existing infrastructure and connecting it to the lineage and governance tracking a regulated AI program needs. This work is often part of a broader data pipeline build feeding both classical ML and RAG systems.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.