Building a high-quality instruction dataset starts with mining real usage data such as support tickets, internal documentation, chat transcripts and analyst reports, then converting that raw material into clear instruction and response pairs that reflect the exact task the fine-tuned model will perform. The most reliable process combines a first pass of automated extraction, often using a larger model to draft candidate question-answer pairs from source documents, with a human review step that corrects factual errors, removes personally identifiable information and rejects examples that are ambiguous or contradictory. Diversity across phrasing, difficulty and edge cases matters more than sheer volume, so teams should deliberately include hard negatives, unusual formatting requests and multi-turn examples rather than only the easy majority case. Deduplication and a held-out evaluation set carved out before training begins are essential, since data leakage between training and evaluation silently inflates measured quality. Version controlling the dataset alongside the model checkpoint makes it possible to trace regressions back to specific data changes later. Nanobase AI, a Silicon Valley enterprise AI engineering company, runs this data pipeline for clients from raw documents through to a training-ready dataset.
Mapping source data to the pairs a model can learn from
Company data rarely arrives in instruction-response form, so the first job is mapping each raw source to the shape it needs to become. Different sources need different extraction approaches, and treating them uniformly wastes effort on sources that need a different pipeline entirely.
| Source | Extraction approach | Common issue |
|---|---|---|
| Support tickets and chat logs | Pair customer question with agent's final response | Inconsistent quality across historical agents |
| Internal documentation / wikis | Generate Q&A pairs from document sections using a larger model | Stale or contradictory content across versions |
| Analyst reports / structured records | Template-based conversion into instruction format | Requires domain expert review for correctness |
| Past model outputs (human-corrected) | Use correction as the target, original as a hard negative | Only useful once a first model version exists |
Each source needs its own extraction logic; a single generic script applied to every source type produces a lower-quality dataset than source-specific handling.
A concrete pipeline from raw documents to training-ready data
- Mine raw sources and run an automated first-pass extraction, often using a larger model to draft candidate instruction-response pairs from each document or ticket.
- Route every candidate pair through human review to correct factual errors, fix awkward phrasing, and reject ambiguous or contradictory examples.
- Scrub personally identifiable information, using automated detection tools plus manual review for the categories that matter most in your domain.
- Deduplicate near-identical pairs so the dataset is not artificially weighted toward whatever topic happened to generate the most raw source material.
- Carve out a held-out evaluation set before any further processing, so later steps cannot accidentally leak evaluation examples into training.
- Version the finished dataset alongside the model checkpoint it produces, so later quality regressions can be traced to a specific data change.
Skipping the human review step to save time is the single most common way a company-data fine-tuning project ends up training a model on its own historical mistakes.
Deliberately including the hard cases
A dataset built purely from the easiest, most common examples in company records teaches a model to handle the easy majority case well while remaining fragile on exactly the inputs that caused problems in the past. Deliberately sourcing hard negatives, unusual formatting requests, and genuinely ambiguous historical cases, alongside clean majority-case examples, produces a model that generalizes better to the full range of production inputs. This is particularly important when mining support tickets or chat logs, since historical responses to hard cases are exactly the ones worth reviewing carefully rather than filtering out as noise.
A dataset that only contains easy, well-handled historical cases will produce a model that is only reliable on easy, well-handled inputs.
Why version control matters as much as data quality
Treating a training dataset as a single static file loses the ability to answer a question every fine-tuning project eventually faces: why did quality change between two model versions? Versioning the dataset the same way source code is versioned, with a clear record of what changed between versions and why, turns that question into a quick diff rather than a guessing exercise. This becomes essential once a model enters a retraining cadence, since each refresh needs to be traceable back to the specific data changes that produced it.
Dataset versioning is what turns "the new model seems worse" into a specific, diagnosable data change rather than a mystery.
Frequently asked questions
How much human review time should we budget per raw source document?
This varies by source complexity, but budgeting review time as a real project phase, not an afterthought, is more important than the exact number, since under-budgeting review is the most common way data preparation timelines slip.
Can we skip the automated extraction step and write examples manually?
For a small dataset, yes, but automated first-pass extraction from existing documents typically produces a larger candidate pool for the same review effort, since reviewers are correcting drafts rather than writing from scratch.
What counts as a contradictory example that should be rejected?
Any pair where the same or similar question in your source data received meaningfully different correct answers across different documents or time periods, since training on both without resolution teaches the model inconsistent behavior.
Should we keep rejected examples for later reuse?
Yes, keeping rejected examples with a reason for rejection helps future data collection avoid repeating the same mistakes and can inform which source documents need updating independent of the fine-tuning project.
How Nanobase AI helps
Nanobase AI runs this data pipeline for clients from raw documents through to a training-ready, version-controlled dataset, including PII scrubbing and the human review workflow described here. We treat data preparation as its own engineering discipline rather than a preliminary step before the "real" work of training. See our solutions for more on how we structure these engagements.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.