Evaluating a fine-tuned model requires comparing it to the base model on both the target task and general capability benchmarks, since a model can improve narrowly while quietly regressing elsewhere. Task-specific evaluation should use a held-out test set that was never seen during training, scored with metrics that match the actual use case, such as exact-match or schema validity for structured output, an LLM-as-judge score for open-ended generation, or accuracy for classification. General capability regression is checked with standard benchmarks or a broad instruction-following test set to confirm the model has not forgotten reasoning, coding or general knowledge it had before fine-tuning. Human evaluation, even a small blind comparison where reviewers rate base versus fine-tuned outputs side by side without knowing which is which, catches quality issues that automated metrics miss, particularly around tone and factual correctness. Latency, output length and refusal rate are also worth tracking since fine-tuning can shift them unexpectedly. Nanobase AI builds this evaluation harness before training begins so every fine-tuning run has a clear, repeatable pass or fail bar.
Three evaluations, not one
A single accuracy or loss number cannot answer whether a fine-tuned model is actually ready for production, because it only measures one dimension of quality. A complete evaluation harness runs three distinct checks, each catching a different failure mode the others would miss.
| Evaluation type | What it catches | Method |
|---|---|---|
| Task-specific held-out test | Whether the target behavior actually improved | Exact-match, schema validity, or LLM-as-judge scoring against a never-seen test set |
| General capability regression | Whether fine-tuning damaged unrelated skills | Standard benchmark or broad instruction-following suite, base vs. fine-tuned |
| Blind human comparison | Quality issues automated metrics miss | Reviewers rate base vs. fine-tuned outputs without knowing which is which |
A model that passes only the task-specific test has been evaluated on one axis out of three that actually matter for a production decision.
Matching the metric to the task, not defaulting to one
The right metric for the task-specific evaluation depends entirely on what the model is being asked to do, and using a mismatched metric produces numbers that look precise but measure the wrong thing. Structured output tasks should be scored on schema validity and field-level accuracy, not text similarity, since a response can look textually close to correct while still failing to parse. Open-ended generation tasks, such as customer support responses, are harder to score automatically and typically benefit from LLM-as-judge scoring against a clear rubric, cross-checked periodically against human judgment to confirm the judge model itself is scoring consistently. Classification tasks use standard accuracy, precision, and recall, ideally broken down per class rather than as a single aggregate, since aggregate accuracy can hide poor performance on a minority class.
Choosing a metric that matches the actual task shape is a five-minute decision that determines whether the rest of the evaluation effort produces a meaningful answer.
Why blind comparison catches what metrics miss
Automated metrics are efficient but blind to qualities that matter in production, particularly tone, subtle factual errors within an otherwise well-formatted response, and stylistic consistency. A blind side-by-side comparison, where reviewers see two anonymized outputs for the same input and pick the better one without knowing which model produced which, surfaces these issues directly from human judgment rather than a proxy metric. This does not need to be large scale to be useful; even a few dozen carefully reviewed comparisons across a range of representative inputs often reveals patterns an automated score alone would not show.
- Sample a representative set of inputs spanning easy, typical, and hard cases.
- Generate outputs from both the base and fine-tuned model for each input.
- Anonymize and randomize the order shown to reviewers.
- Have reviewers rate or rank the pair without knowing which model produced which output.
- Aggregate results and look specifically for patterns in where the fine-tuned model loses, not just the overall win rate.
A double-digit sample of blind comparisons often reveals quality issues that a much larger automated evaluation set would miss entirely.
Tracking secondary metrics that shift unexpectedly
Beyond the primary task metric, fine-tuning can shift secondary characteristics that are easy to overlook until they cause a production issue. Latency and output length sometimes change measurably after fine-tuning, since a model trained on consistently shorter or longer target responses will tend to reproduce that pattern. Refusal rate is worth tracking explicitly, since fine-tuning can either increase unwanted refusals on legitimate requests or, more concerning, decrease appropriate refusals on requests that should be declined, connecting to the broader question of whether fine-tuning removes safety guardrails. Building these secondary checks into the same evaluation run, rather than as an afterthought, catches shifts before they reach production.
Secondary metrics like latency, length, and refusal rate deserve the same before-and-after comparison as the primary task metric, not an afterthought check.
Frequently asked questions
How large does a held-out test set need to be?
Large enough for the metric to be statistically stable, commonly 50 to a few hundred examples depending on task variance, and always drawn from the same distribution as production traffic rather than a convenient but unrepresentative sample.
Can LLM-as-judge scoring replace human evaluation entirely?
Not entirely. LLM-as-judge scoring scales well and is useful for continuous evaluation, but periodic human review is still needed to confirm the judge model's scoring aligns with actual human quality judgments, since judge models carry their own biases.
Should evaluation happen only once before deployment?
No, evaluation should run again on a periodic or triggered basis after deployment, since production traffic patterns and data drift over time can reveal issues the pre-deployment evaluation set did not anticipate.
What is an acceptable amount of general capability regression?
Ideally none, but any regression should be a deliberate, measured trade-off weighed against the target-task improvement, not an unnoticed side effect discovered after deployment.
How Nanobase AI helps
Nanobase AI builds this three-part evaluation harness before training begins, so every fine-tuning run has a clear, repeatable pass or fail bar rather than a single ambiguous quality claim. We combine task-specific metrics, general capability regression checks, and blind human comparison into one evaluation process for every project. See our solutions or book a demo to see this evaluation approach in practice.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.