Running LLM evals in CI means adding an automated evaluation step to the same pipeline that already runs unit tests, so a pull request that changes a prompt, retrieval configuration or model version cannot merge unless it passes defined quality thresholds against the golden evaluation dataset. A practical setup uses a framework like promptfoo, DeepEval or Ragas configured to run as a CI step, comparing the new prompt or model's output against expected answers or rubric-based judge scores, and failing the build if scores drop below the baseline by more than an acceptable margin. Because full evaluation runs cost money and take time due to live model calls, most teams run a fast, smaller subset of the golden dataset on every pull request, and a full run on a nightly schedule or before a release for comprehensive coverage. Results should be visible directly in the pull request, similar to a code coverage report, so a reviewer can see exactly which case regressed rather than just a pass or fail signal. Flaky results caused by non-deterministic output need managing with multiple samples or a tolerance band, the same way a flaky integration test is handled in traditional testing. Nanobase AI wires evaluation frameworks directly into client CI pipelines so prompt changes get the same merge protection as any other code change.
Treat a prompt change like a schema migration, not a copy edit
A one-line prompt edit can change model behavior as significantly as a database schema migration changes application behavior, yet most teams review prompt changes with a quick read-through rather than an automated test suite. Wiring evaluation into CI means a prompt, retrieval configuration or model version change cannot merge without passing the same golden dataset that already validates production quality, turning a manual judgment call into an automated gate with a clear pass or fail.
Deciding what blocks a merge versus what only warns
| Signal | Merge gate | Rationale |
|---|---|---|
| Score drop below the defined baseline threshold | Blocks merge | A clear regression should never ship silently |
| Latency increase beyond an acceptable margin | Blocks merge for latency-sensitive features | Correctness without acceptable speed is still a regression |
| Score improvement with increased token cost | Warns, requires reviewer acknowledgment | A genuine trade-off a human should confirm is worth it |
| Flaky individual test case (inconsistent pass/fail across runs) | Warns, flagged for review | Avoids blocking on non-deterministic noise while surfacing it |
A regression should always block; a trade-off should always require a human to see it and choose, since automating that choice away either ships a silent cost increase or blocks a genuine improvement.
A CI eval step in practice
A typical setup runs a fast subset of the golden dataset on every pull request and a full run on a nightly schedule or before release:
# .github/workflows/eval.yml
on: pull_request
jobs:
prompt-eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install promptfoo
- run: promptfoo eval --config eval/pr-subset.yaml --output results.json
- run: python scripts/check_threshold.py results.json --min-score 0.85The threshold check script fails the CI job, and therefore blocks the merge, if the subset score drops below the defined baseline, giving a reviewer the same automated confidence a code coverage gate provides for traditional code changes.
Taming non-determinism so the gate is not flaky
- Set temperature to zero or as low as the use case allows for evaluation runs specifically, even if production uses a higher temperature for variety, since deterministic evaluation output is easier to compare reliably across runs.
- Run each test case multiple times and use a majority vote or average score when full determinism is not achievable, rather than treating a single sample as ground truth.
- Apply a tolerance band around the baseline rather than an exact threshold, so normal sampling variance does not trigger a false failure on every run.
- Separate genuinely flaky cases into a monitored-but-non-blocking list, the same pattern used for flaky integration tests in traditional software, revisited periodically rather than left blocking merges indefinitely.
Making results visible where reviewers actually look
Evaluation results should surface directly in the pull request, similar to a code coverage report, showing exactly which test cases regressed rather than a single aggregate pass or fail signal. A reviewer who can see "case 14 regressed because the retrieved context changed" can make an informed call quickly; one who sees only a failing CI check has to dig through logs to find the same information.
Frequently asked questions
How large should the PR-level eval subset be compared to the full golden dataset?
Large enough to catch common regression patterns, small enough to return results within a few minutes so it does not slow down the development loop; a common approach uses a representative slice covering major query types and known edge cases, reserving the full dataset for a nightly or pre-release run.
What frameworks work well for CI-based LLM evaluation?
Promptfoo, DeepEval and Ragas all support scripted, CI-friendly evaluation runs; the right choice depends on whether the primary use case is general prompt testing, unit-test-style assertions, or RAG-specific metrics like faithfulness.
Does every prompt change need to go through the full eval gate?
Changes affecting the system prompt, retrieval configuration or model version should always go through the gate; a purely cosmetic change to user-facing copy unrelated to model behavior may not need the same evaluation overhead, though the line should be defined explicitly rather than left to individual judgment.
How Nanobase AI helps
Nanobase AI wires evaluation frameworks directly into client CI/CD pipelines, including threshold tuning and non-determinism handling, so prompt changes get the same merge protection as any other code change rather than shipping on a reviewer's read-through alone.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.