Speculative Decoding in Practice: 2x–3x Speedups Without Accuracy Loss

1. Executive Summary#

When production teams need to cut LLM latency, they usually reach for aggressive quantization (e.g., 4-bit weights) or smaller models, sacrificing output quality. Speculative Decoding offers a rare free lunch in computer science: a 2x to 3x reduction in end-to-end generation latency with mathematically identical output distributions to the full target model.

2. Why It Works: Exploiting Idle Compute During Decode#

As established in our serving architecture breakdown, generating 1 token on a 70B model takes roughly the same wall-clock time as processing 5 tokens in parallel—because the bottleneck is reading the 70B weights from HBM into SRAM, not performing the arithmetic in Tensor Cores.

Speculative decoding exploits this asymmetry via a two-stage loop:

1. Drafting: A tiny, fast Draft Model (e.g., a 1B model, or lightweight auxiliary heads) autoregressively guesses the next gamma tokens (e.g., 5 tokens): [t_1, t_2, t_3, t_4, t_5].
2. Parallel Verification: The massive Target Model (e.g., 70B) processes the prompt plus all 5 drafted tokens in a single forward pass. Because causal attention computes logits for every position simultaneously, the 70B model evaluates the probability of all 5 draft tokens in the time it normally takes to generate 1 token.

3. Rejection Sampling Guarantees Zero Quality Loss#

For each drafted token t_i, we compare the target model's probability P_target(t_i) with the draft model's probability P_draft(t_i):

• If P_target(t_i) >= P_draft(t_i), the token is immediately accepted.
• If P_target(t_i) < P_draft(t_i), we accept the token with probability P_target(t_i) / P_draft(t_i).
• At the first rejected token, we discard all subsequent draft tokens and sample a corrected token directly from a residual distribution norm(max(0, P_target - P_draft)). And if all 5 tokens are accepted, the target model's final logit gives us a 6th bonus token for free.

Mathematically (Leviathan et al., 2023), this rejection sampling scheme guarantees that the generated sequence is distributed identically to sampling from the target model alone.

4. Three Production Speculative Architectures#

1. Standalone Draft Model: Pairing Llama-3.1-70B with Llama-3.2-1B. Requires both models to share the exact same tokenizer vocabulary and reside in VRAM.
2. Prompt-Lookup Decoding (PLD / N-gram Matching): Zero extra VRAM. Uses string matching over the input prompt and previous context to propose draft tokens. Ideal for RAG summarization, code refactoring, and JSON extraction where output tokens frequently copy substrings from the prompt.
3. Medusa & EAGLE (Self-Speculative Heads): Attaches lightweight MLP or single-layer autoregression heads directly onto the target model's penultimate hidden states. EAGLE conditions draft predictions on both feature vectors and token embeddings, achieving high acceptance rates (3.5+ tokens per step) with minimal memory overhead.