A/B testing models and prompts in production means routing a defined percentage of live traffic to a variant, whether a different model, prompt version or retrieval configuration, and comparing outcomes against a control group using both automated quality metrics and real business signals like conversion or resolution rate. Traffic splitting can happen at the LLM gateway layer, using a tool like LiteLLM or Portkey to route requests by a consistent hash of the user ID so the same user always sees the same variant, which avoids a confusing inconsistent experience within a single session. The evaluation side needs leading indicators, such as judge scores and latency, that show results within hours, and lagging indicators, such as user satisfaction or downstream task completion, that take longer to accumulate but reflect what matters to the business. Statistical significance still applies to LLM experiments the same way it applies to any product experiment, and a team should define a minimum sample size and test duration up front rather than stopping when a metric looks favorable. Feature flag systems built for traditional software, such as LaunchDarkly, extend naturally to gating prompt and model variants without a full redeploy. Nanobase AI sets up traffic-splitting and measurement infrastructure so prompt and model experiments produce statistically sound answers rather than anecdotal impressions.
LLM experiments fail in ways classic product A/B tests do not
A conversion-rate A/B test has one well-understood source of noise: user behavior variance. An LLM prompt or model experiment has two additional sources: the model's own non-determinism at any temperature above zero, and the variance of whatever judge or rubric scores the output. Running an LLM A/B test without accounting for judge-score variance and output non-determinism produces a result that looks statistically clean and is not actually reproducible if the same experiment ran again. Teams that skip this step often ship a "winning" prompt that was really just a lucky sample.
A repeatable process
- Fix everything except the variable under test. If comparing two prompts, hold the model, temperature and retrieval configuration constant; changing more than one variable at a time makes the result uninterpretable.
- Define the primary metric and the guardrail metrics before launch. A prompt that improves the primary quality score but doubles latency or token cost has not actually won, so guardrails need a threshold decided in advance, not after seeing results.
- Route consistently per user, using a stable hash of the user or session ID at the gateway layer, so the same user sees the same variant throughout the test and the experience does not feel inconsistent within a session.
- Set sample size and duration before looking at results, based on the expected effect size and the variance observed in a pilot run; stopping early because a metric looks favorable is the fastest way to ship a false positive.
- Score with multiple judge calls per output, not one, when using LLM-as-a-judge, and average or take a majority vote, since a single judge call carries its own sampling noise on top of the variance between prompt variants.
Choosing what to measure
| Metric type | Example | Time to signal | Risk if used alone |
|---|---|---|---|
| Leading, automated | Judge quality score, latency, token cost | Hours | Can miss real user-perceived quality |
| Leading, behavioral | Regeneration rate, abandonment | Hours to a day | Confounded by unrelated UX issues |
| Lagging, business | Task completion, conversion, resolution rate | Days to weeks | Slow, but reflects what actually matters |
A sound test design pairs at least one leading and one lagging metric, using the leading metric to catch an obviously broken variant early and the lagging metric to confirm the change actually helped before a full rollout.
When not to run this online at all
A prompt change touching a safety-critical path, financial calculations, medical guidance, anything with legal exposure, should not get its first real-world exposure through a live A/B test on unsuspecting users. Shadow deployment against recorded or replayed traffic, evaluated against a golden dataset first, is the safer sequence, with a live A/B test reserved for confirming an already-validated change's real-world impact rather than discovering whether it works at all.
Frequently asked questions
How long should an LLM A/B test run?
Long enough to reach the sample size calculated from the pilot variance and expected effect size, which depends heavily on the metric; a latency comparison can resolve in hours while a task-completion metric with low natural variance may need one to two weeks of traffic.
Can we A/B test more than two variants at once?
Yes, but each additional variant increases the total sample size needed to reach significance on any one comparison, so testing more than three or four variants simultaneously usually means a longer test or a less confident result on each pairwise comparison.
Do feature flag tools built for traditional software work for prompt experiments?
Yes, tools like LaunchDarkly extend naturally to gating prompt and model variants, since the routing logic, consistent per-user assignment and percentage rollout, is identical to any other feature flag use case.
How Nanobase AI helps
Nanobase AI, a Silicon Valley enterprise AI engineering company, sets up gateway-level traffic splitting, judge-score variance controls and pre-registered sample sizes so prompt and model experiments produce answers a team can actually act on. This typically pairs with the production monitoring already watching the application, and can be reviewed live in a demo.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.