Speculative decoding uses a small, fast draft model, or lightweight prediction heads such as EAGLE or Medusa attached to the main model, to propose several candidate tokens ahead of the target model, which then verifies all of them in a single forward pass and accepts the ones that match what it would have generated on its own. It does reduce per-token latency, because verifying multiple speculated tokens costs roughly the same as generating one token normally, so accepted tokens are effectively free; typical speedups range from about 1.5x to 3x depending on how well the draft model or heads predict the target model's outputs. The gain depends heavily on acceptance rate: predictable text such as code or repetitive structured output accepts more speculated tokens and benefits more than open-ended creative generation. Speculative decoding mainly helps latency-sensitive, low-concurrency scenarios; under very high concurrency, where the GPU is already compute-bound from batching many requests, the relative benefit shrinks because the extra verification compute competes with other sequences. vLLM, TensorRT-LLM, and SGLang all support speculative decoding with draft models or EAGLE-style heads. Nanobase AI evaluates speculative decoding against a customer's actual traffic concurrency before recommending it, since the benefit is workload-dependent.
Three approaches to the same idea, with different tradeoffs
Speculative decoding's core idea, propose several tokens ahead cheaply, verify them all in one forward pass of the full model, is implemented several distinct ways, and the choice between them affects setup complexity, acceptance rate, and infrastructure cost differently.
| Method | How candidate tokens are proposed | Infrastructure cost | Typical fit |
|---|---|---|---|
| Independent draft model | A smaller, separately trained model of the same family generates candidates | Needs a second model loaded and run alongside the target model | Works well when a well-matched small model in the same family already exists |
| Medusa heads | Extra prediction heads attached to and trained alongside the target model, predicting several future tokens at once | Requires fine-tuning the target model with additional heads | No separate model to serve, but needs a training step to add and align heads |
| EAGLE | Lightweight head trained to predict future hidden states rather than tokens directly, improving on Medusa's accuracy | Also requires a trained component, but typically achieves higher acceptance rates than Medusa | Preferred where available due to stronger acceptance rate for similar overhead |
| Lookahead decoding | Uses n-gram statistics or the model's own generation pattern without a separate trained component | No extra model or training needed | Simpler to deploy, generally lower acceptance rate than trained approaches |
Why acceptance rate is the number that actually matters
Every speculative decoding method's benefit comes down to one metric: what fraction of proposed tokens does the target model actually accept as what it would have generated anyway. A high acceptance rate means most speculated tokens are essentially free, verified in the same forward pass that would have produced only one token normally. A low acceptance rate means the extra compute spent proposing and verifying candidates is largely wasted, since rejected tokens have to be regenerated the normal way. This is why predictable output, such as code completion or structured data generation, tends to see stronger results than open-ended creative writing, where the target model's next token is inherently harder for any draft mechanism to anticipate.
Method choice should follow from what acceptance rate is realistically achievable on your actual output distribution, not from which method sounds most sophisticated.
Engine support as of 2026
vLLM, TensorRT-LLM, and SGLang all support speculative decoding, though the specific methods available and their maturity differ by engine and release version. vLLM supports both independent draft models and EAGLE-style speculation. TensorRT-LLM has invested in Medusa and draft-model approaches integrated into its compiled engine pipeline. SGLang has added EAGLE support with attention to keeping the verification step compatible with its RadixAttention scheduling. Checking your target engine's current release notes before committing to a specific method is worthwhile, since this is an active area of development and capability gaps close quickly.
When the benefit shrinks: the concurrency ceiling
Speculative decoding's latency benefit is largest at low concurrency, where the GPU has spare compute capacity to spend on proposing and verifying extra candidate tokens without competing against other requests. At high concurrency, when the GPU is already compute-bound serving many simultaneous sequences through continuous batching, that spare capacity shrinks, and the relative benefit of speculative decoding narrows because the verification step now competes for the same compute other users' requests need. This makes speculative decoding a better fit for latency-sensitive, lower-concurrency deployments, such as interactive coding assistants, than for high-throughput batch or high-concurrency chat services.
A decision path
- Identify whether your workload is latency-sensitive at low-to-moderate concurrency, or throughput-sensitive at high concurrency; the former benefits more.
- Estimate acceptance rate on a sample of your actual output distribution, ideally by testing a candidate method directly rather than assuming.
- If a well-matched smaller model in the same family exists, an independent draft model is the lowest-effort starting point.
- If no suitable smaller model exists, evaluate EAGLE or Medusa, weighing the training and integration effort against expected acceptance rate.
- Re-test after any target model update, since acceptance rate is tied to the specific target model's behavior and can shift after fine-tuning.
For the latency question this technique targets specifically, see how to reduce time to first token.
Frequently asked questions
Does speculative decoding change the model's output distribution?
No, when implemented correctly, speculative decoding is mathematically guaranteed to produce output statistically identical to standard decoding from the target model; it only changes how many forward passes are needed to produce that output, not what the output is.
Do we need to train anything to use an independent draft model?
Not necessarily, if a suitable smaller model from the same family already exists and is compatible with the target model's tokenizer and vocabulary; if no such model exists, training one or using head-based methods like EAGLE becomes necessary instead.
Is EAGLE always better than Medusa?
EAGLE generally achieves higher acceptance rates than Medusa in published comparisons because it predicts hidden states rather than tokens directly, but actual results depend on the specific target model and task, so testing both against your own workload before choosing is worthwhile.
Can speculative decoding be combined with quantization?
Yes, and it often is, since quantization reduces per-token compute cost independently of speculative decoding's mechanism; the two techniques address different parts of the latency problem and are commonly used together in production deployments.
How Nanobase AI helps
Nanobase AI evaluates speculative decoding methods against a customer's actual traffic concurrency and output distribution before recommending one, since the benefit is highly workload-dependent and the wrong method choice can add complexity without a matching latency gain.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.