Speculative Decoding Explained: How Draft Tokens Speed Up LLMs

What the draft and target models actually do

The target is the model whose output behaviour the system aims to retain. A draft model or another proposal mechanism supplies candidate continuations that are cheaper to generate. In a separate-model design, the draft is often smaller than the target; in other methods, proposals come from a lightweight component or from repeated text in the input. [2] [3]

A token is an item in the model’s vocabulary, not necessarily a whole word. Tokens can represent word pieces, punctuation or other encoded content. The target’s next-token distribution depends on the context already accepted. Speculation changes how candidate tokens are prepared and checked; it does not remove that dependence on preceding text.

The draft does not need to reproduce every target decision. It needs enough useful accepted candidates, at low enough cost, to make the full loop faster. A less accurate draft can still preserve the target distribution when paired with the correct sampler—but poor acceptance can make it an ineffective accelerator. [2]

Why ordinary token generation can be slow

Autoregressive decoding normally advances one output token at a time for each sequence. The next step depends on the token just produced. A powerful accelerator can therefore have spare arithmetic capacity during a small-batch decoding workload while memory access or communication dominates the time taken by each step. [1] [2]

This is not a description of every inference workload. Larger batches, long contexts, different model architectures and other hardware arrangements can change the limiting resource. Chen and colleagues explain why scoring a short continuation can take a similar time to sampling one token under their stated memory- and latency-bound conditions. The word “similar” is important: verification is not free. [2]

How the draft-and-verify loop works

The following outline describes a standard single-sequence exact-sampling approach. More advanced implementations may use trees, parallel drafting or other scheduling strategies, but candidates still need the appropriate verification rule. [2] [3]

1. Draft a short continuation. The proposal mechanism generates several candidate tokens. A small autoregressive draft model may itself generate those candidates sequentially.
2. Score the candidate positions with the target. A causal target-model pass evaluates the proposed continuation, producing the conditional information needed for verification at each position.
3. Accept a prefix using the correct rule. Check candidates from the beginning. The rule for stochastic sampling differs from a simple greedy token match.
4. Correct the first rejection. Produce a valid replacement using the sampler’s correction rule, then discard the later draft candidates because their proposed context no longer applies.
5. Advance the accepted output. If every candidate was accepted, the standard approach can also produce a bonus target token. Repeat from the new accepted prefix.

What “verified together” does—and does not—mean

The target can score several proposed positions in a batch because the candidate continuation is already available. Causal masking makes each position’s score depend on its preceding context, rather than allowing unrestricted use of later candidate tokens.

If an early candidate is rejected, scores for the later candidate path no longer describe the corrected continuation. Those later proposals cannot simply be appended. Parallel scoring reduces some sequential target work; it does not let accepted text ignore autoregressive dependencies. [2]

Five draft tokens do not guarantee a fivefold speedup

Suppose a draft proposes five tokens and verification accepts the first four before rejecting the fifth. The standard loop can advance by those four accepted tokens plus one corrected token. That is five output tokens from one verification loop. It is not five tokens for the wall-clock cost of one ordinary target step. [2]

The elapsed time also includes drafting, target verification, sampler work and scheduling overhead. If the first proposal is rejected, the loop can still produce a valid corrected token, but it has also done draft work that did not advance the accepted output. A high rejection rate can make speculation slower than the baseline.

When all K draft tokens are accepted, a standard loop may emit K plus one tokens because it can sample a bonus token from the target information already computed. K plus one is a maximum output count per loop under this scheme, not an automatic speedup factor. [2]

Why stochastic verification is more than a token match

For a classic probability-based sampler, let p(x) be the target probability of a candidate token and q(x) its draft probability at the same accepted context. A token sampled from the draft is accepted with probability min(1, p(x) ÷ q(x)). The ratio corrects for the different probabilities assigned by the two models. [2]

On rejection, the classic algorithm samples from a corrected distribution proportional to max(0, p(x) − q(x)), normalised across the vocabulary. It does not simply choose the target’s highest-probability token. Nor is drawing from the unadjusted target distribution the same correction rule for this full-probability draft-sampling algorithm. [2]

Greedy decoding uses a different, deterministic selection setting. A draft token can be checked against the target’s greedy choice, with the appropriate correction when they disagree. The familiar explanation “accept matching tokens” is useful for that case, but it should not replace the stochastic acceptance-and-correction explanation. [3]

A two-outcome example of distribution preservation

Consider one token position with only two possible outcomes, A and B. This is a deliberately simplified educational example, not a vocabulary or result from a real LLM. Suppose the target assigns probabilities 0.60 to A and 0.40 to B, while the draft assigns 0.80 to A and 0.20 to B.

A drafted A is accepted with probability 0.60 ÷ 0.80 = 0.75. A drafted B is accepted with probability min(1, 0.40 ÷ 0.20) = 1. The accepted probability mass is therefore 0.60 for A and 0.20 for B, leaving 0.20 for rejection.

The positive difference between target and draft probabilities is zero for A and 0.20 for B. The normalised correction distribution therefore chooses B. Adding the rejected mass back gives final probabilities of 0.60 for A and 0.40 for B—the target’s distribution. This shows why correcting a rejected sample matters, rather than merely accepting the draft’s most likely answer. [2]

Same distribution does not mean identical text every time

Distribution preservation means the exact sampling algorithm retains the probabilities defined by the target model and sampling settings. It does not mean two independently sampled responses must be byte-for-byte identical. Different random draws can produce different valid sequences from the same distribution. [2]

It also does not make the target factually correct. A target model can still generate inaccurate or unsuitable content. Preserving its distribution is a mathematical property of the sampling process, not a guarantee that every answer is reliable.

Implementation details matter too. vLLM documents floating-point and batching effects that can change token probabilities, and does not promise stable token log probabilities across all runs. Its documented greedy-equality tests and theoretical losslessness should not be rewritten as an unconditional cross-hardware identical-output promise. [3]

What the published speedup results actually measured

Leviathan, Kalman and Matias reported a 2×–3× acceleration for T5-XXL compared with a standard T5X implementation in their 2023 ICML paper. That is evidence from a named model and baseline, not a typical-speedup estimate for every present-day LLM service. [1]

Chen and colleagues reported a 2×–2.5× decoding speedup for the 70-billion-parameter Chinchilla model in a distributed setup in their February 2023 paper. Their reported result similarly belongs to that setup, draft and workload. [2]

These papers establish that the technique can provide substantial gains under suitable conditions. They do not establish that switching it on in an unrelated configuration will reproduce those numbers. The real question is whether saved target work outweighs drafting, verification and serving overhead on the intended traffic.

How vLLM supports speculative decoding

vLLM is an open-source inference and serving engine. Its stable documentation describes speculative decoding as a way to reduce inter-token latency for medium-to-low request-rate, memory-bound workloads, and lists several proposal methods. Compatibility and configuration depend on the installed release and model architecture. [3]

Not every method is a second independent language model. The proposal mechanism is a design choice, and target-specific methods need the matching model or trained component rather than an arbitrary interchangeable checkpoint. [3]

• Separate draft model: another model supplies candidate tokens for verification.
• N-gram or suffix proposal: continuations are proposed from text patterns without loading another draft model.
• EAGLE-style methods: a compatible learned speculator uses information associated with the target to propose continuations.
• Multi-token prediction, or MTP: supported target architectures or associated components provide additional predictions.

Which configuration choices matter?

The stable documentation uses a speculative_config object, or the CLI option --speculative-config. Important fields include method, model when the method requires one, and num_speculative_tokens. The draft model’s parallelism is configured with draft_tensor_parallel_size; temperature and top_p are sampling settings, not fields to put into the speculation configuration. [3]

Tokenizer compatibility needs deliberate checking. A shared vocabulary is the usual default for separate-model drafting, but the retrieved stable documentation also describes a version-supported token-level-intersection path for different vocabularies, with restrictions. “Every draft must always use the same tokenizer” is therefore too broad; neither should arbitrary model pairs be assumed compatible. [3]

Use the documentation that matches the installed version, including its model and parallelism limitations. This article is a conceptual guide, not a tested deployment recipe: no particular checkpoint pair, GPU allocation or executable serving command has been validated here.

Do all speculative variants preserve the target distribution?

No. The exact-sampling property depends on the verification algorithm. vLLM’s developer-preview documentation explicitly labels FLy verification as lossy: it relaxes rejection decisions and is not distribution-preserving. That is a preview feature description, not a claim that the same option exists in every stable installation. [4]

For a deployment that requires the exact-sampling guarantee, verify the chosen method and sampler rather than assuming every feature with “speculative” in its name has identical guarantees. The standard algorithm’s mathematical property does not transfer automatically to approximate verification variants. [2] [4]

When speculative decoding is most worth measuring

The opportunity is stronger when a proposal is cheap, acceptance is useful and target decoding has spare capacity to score short continuations efficiently. Low-concurrency interactive generation can fit that description, which is why vLLM highlights inter-token latency at medium-to-low request rates. [2] [3]

Prompt-lookup methods are worth considering when the desired response repeats parts of the input; other workloads may need a learned drafter to propose useful continuations. A task label such as “code” or “summarisation” alone does not establish the acceptance rate of a particular configuration.

At higher traffic, the target may already use available capacity efficiently, and extra drafting can compete for memory or compute. Some methods still offer gains at higher request rates, according to vLLM’s qualitative guidance. Measure the actual serving goal rather than treating high concurrency as an automatic failure or an automatic success. [3]

Separate first-token latency, decoding speed and throughput

Time to first token measures how long a user waits before output begins. Inter-token latency measures the spacing between generated tokens after generation starts. Total request time also includes prompt processing, queueing and the complete response. Server throughput measures aggregate work served over time.

Speculation targets the decoding stage. A long prompt can make prefill a major part of the request, so faster output-token generation does not guarantee an equally large improvement in first-token latency or total time. A per-request speedup and a server-wide throughput increase are also different results. [2] [3]

Accepted-token fraction, accepted prefix length and emitted tokens per verification loop describe proposal behaviour. Wall-clock latency describes the user-facing result. Track both: a high acceptance rate can still be an unattractive trade if the drafter is expensive.

What a fair comparison needs to hold constant

A useful baseline keeps the target model, its numerical configuration, prompts, generation settings and response-length limits comparable. Test the intended concurrency rather than only a convenient single request. Record the draft method and speculation length along with hardware, software version and memory use. [3]

Warmup, queueing and input/output length affect measurements. Report the latency or throughput metric that was actually measured, and distinguish median behaviour from slower requests. A faster short demonstration should not be relabelled as a guarantee for a public application.

For an India-based service, network and application delay can remain visible to users after GPU decoding becomes faster. This is a general measurement consideration, not a claim that a particular Indian provider has enabled speculation or achieved the paper’s speedup.

Common questions about speculative decoding

Does speculative decoding reduce answer quality?

The standard exact-sampling approach is designed to preserve the target model’s sampling distribution rather than substitute the draft’s output behaviour. Approximate variants can make different trade-offs. Neither exactness nor speculation guarantees that the target’s answer is factually correct. [2] [4]

Will it make my model three times faster?

Not necessarily. Threefold acceleration was part of the reported range for a specific T5-XXL comparison, not a universal result. Model pairing, hardware, workload, generation settings and serving load determine whether a different setup benefits. [1] [3]

Do I always need another model?

No. A separate draft model is one option. vLLM also documents text-pattern proposal methods and target-associated prediction approaches. Each method has its own compatibility and workload requirements. [3]

Can speculation make inference slower?

Yes. Drafting and verification take time and consume resources. If too few useful tokens are accepted, or the system has little spare capacity, that extra work can outweigh the saved target steps. Correct output sampling and good speed are separate requirements. [2]