SlimSpec: faster speculative decoding without cutting the vocabulary

Speculative decoding only helps in production when the drafter is both fast and accepted often enough by the target model. One hidden bottleneck is the draft model’s LM head: even a lightweight drafter may still spend meaningful compute projecting hidden states into a large vocabulary.

SlimSpec is our low-rank draft LM-head architecture for speculative decoding. Instead of shrinking the vocabulary, it compresses the drafter’s hidden representation while preserving full-vocabulary support. In our EAGLE-3 experiments, SlimSpec reduces LM-head cost by 4-5x versus the standard unembedding projection while keeping acceptance quality competitive.

This research is part of the same direction behind Custom Speculator Training in Nebius Token Factory: train the draft model for the workload you actually serve, then evaluate it with the base model on real traffic. If you are running production inference with repeatable latency or throughput pressure, read the Custom Speculator docs or talk to our team about a spec decoding performance review.

SlimSpec has been submitted to NeurIPS; the preprint is available on ArXiv.

What this changes in productionWhat this changes in production

For production teams, speculative decoding is not just about making a benchmark faster. The useful question is whether the drafter can propose tokens quickly enough, and accurately enough, to improve the serving path users actually experience.

Vocabulary-reduction methods can make the LM head cheaper, but they introduce a tradeoff: once the drafter cannot propose tokens outside a reduced vocabulary, acceptance quality can suffer. SlimSpec takes a different path. It keeps full-vocabulary support and reduces the cost of the projection itself.

That matters for workloads where throughput, latency, and acceptance rate all have to move together: coding assistants, structured-output agents, enterprise copilots, and other repeated production patterns where generic speculative decoding may leave performance on the table.

Why the draft LM-head becomes a bottleneckWhy the draft LM-head becomes a bottleneck

In auxiliary-head speculative decoding methods such as MEDUSA, Hydra, and the EAGLE family, the draft-side computation naturally splits into two parts:

$$T_{\mathrm{draft}} = T_{\mathrm{backbone}} + T_{\mathrm{head}}$$

The backbone produces draft hidden states h. The LM-head maps those hidden states to vocabulary logits. The standard projection is performed by the unembedding matrix:

$$\mathrm{logits} = W_{\mathrm{full}}h, \quad W_{\mathrm{full}} \in \mathbb{R}^{V \times d}$$

where V is the full target vocabulary size and d is the drafter hidden dimension. The cost of this projection is $O(Vd)$, which, in modern LLMs, corresponds to hundreds of millions of operations per drafted token and therefore makes the drafter LM-head a natural computational bottleneck.

Why vocabulary truncation is not enoughWhy vocabulary truncation is not enough

A natural way to reduce LM-head cost is to reduce the number of logits that the drafter computes. Static methods such as FR-Spec (Zhao et al., 2025), VocabTrim (Goel et al., 2025), and BCL (Shoham, 2026) choose a smaller vocabulary subset. Dynamic methods such as CORAL (Weng et al., 2025), DynaSpec (Zhang et al., 2025), and SpecVocab (Williams et al., 2026) choose an active subset depending on the context.

This helps, but it introduces two problems.

First, if the drafter is restricted to a truncated vocabulary, tokens outside that vocabulary can never be proposed, which creates a hard ceiling on the acceptance rate.

For a target distribution p and draft distribution q, the per-token acceptance rate is governed by the distributional overlap:

$$\alpha = \sum_{v \in \mathcal{V}} \min\{p(v), q(v)\}$$

where V is the full vocabulary. If the drafter only supports a truncated vocabulary $\mathcal{V}_{\mathrm{tr}}$, then $q(v) = 0$ for every token outside $\mathcal{V}_{\mathrm{tr}}$. This means:

$$\alpha \leq \sum_{v \in \mathcal{V}_{\mathrm{tr}}} p(v)$$

No matter how well the drafter is trained, it cannot accept probability mass assigned to tokens it is unable to propose.

Second, vocabulary truncation can create a train-test mismatch. Forward KL training assumes the drafter can represent the full target distribution, but a truncated drafter assigns zero probability to many tokens — making the KL divergence ill-defined. The usual workaround is to mask and renormalize the target during training, but verification at inference time still uses the original, unmasked target. The drafter is therefore optimized for a distribution it will never actually be evaluated against.

Dynamic vocabulary selection can reduce some of these issues, but it adds inference-time machinery: routing, top-k selection, irregular indexing, and gathering context-dependent subsets of weights. These operations are not as GPU-friendly as dense matrix multiplication and can become bottlenecks themselves.

SlimSpec: compress the representation, not the vocabularySlimSpec: compress the representation, not the vocabulary

Instead of projecting the drafter hidden state directly into the vocabulary space, SlimSpec first compresses the hidden representation to rank $r$, then maps it to the full vocabulary:

$$\mathrm{logits} = W_{\mathrm{up}}W_{\mathrm{down}}h, \quad W_{\mathrm{up}} \in \mathbb{R}^{V \times r}, \quad W_{\mathrm{down}} \in \mathbb{R}^{r \times d}$$ where $r < d$. The resulting LM-head cost becomes $O(Vr + rd)$. Since modern LLM vocabularies are large and usually $V >> d$ the reduction is approximately linear in the rank ratio:

$$\frac{Vr + rd}{Vd} \approx \frac{r}{d}$$

Relative LM-head GPU time $T_{\mathrm{head}}$ across models; lower is better. The underlying $T_{\mathrm{head}}$ values are normalized with respect to the full vocabulary baseline, set to 1.0. VocabTrim reduces the draft vocabulary to 64K tokens. Both VocabTrim and SpecVocab can reduce LM-head latency by only about 60%, while SlimSpec achieves an approximately 4-5x reduction.

For example, using $r = d/8$ gives a much cheaper LM-head while still preserving full-vocabulary support. This is the key difference between SlimSpec and vocabulary-reduction methods. SlimSpec does not remove tokens from the output space. It keeps all token logits available, but computes them through a thinner intermediate representation.

LM-head latency alone is not the right objectiveLM-head latency alone is not the right objective

A cheaper LM-head does not automatically mean faster end-to-end decoding. In speculative decoding, throughput depends not only on how fast the drafter runs, but also on how many draft tokens are accepted by the target model. We can write the resulting throughput in tokens per second as:

$$\mathrm{TPS} = \frac{\tau}{T_{\text{non-head}} + T_{\mathrm{head}}}$$

where $\tau$ is the average acceptance length. To compare different LM-head acceleration methods, we can express the relative end-to-end speedup $\rho_{\mathrm{TPS}} = \mathrm{TPS}^{\mathrm{method}} / \mathrm{TPS}^{\mathrm{baseline}}$ as:

$$\rho_{\mathrm{TPS}} = \rho_\tau \frac{1 + \kappa}{1 + \nu\kappa}$$

where:

• $\rho_\tau = \tau^{\mathrm{method}} / \tau^{\mathrm{baseline}}$ is the acceptance ratio between a selected method and full projection baseline.
• $\nu = T_{\mathrm{head}}^{\mathrm{method}} / T_{\mathrm{head}}^{\mathrm{baseline}}$ is the LM-head latency ratio between a selected method and full projection baseline.
• $\kappa = T_{\mathrm{head}}^{\mathrm{baseline}} / T_{\text{non-head}}^{\mathrm{baseline}}$ measures how important the LM-head is relative to the rest of the speculative decoding pipeline, independent of the particular LM-head acceleration method.

This makes the trade-off explicit: cheaper LM-head $(\nu < 1)$ improves speedup, but lower acceptance $(\rho_\tau < 1)$ drags it back down. Therefore, the best method is not necessarily the one with the cheapest LM-head, but the one that reduces LM-head cost while preserving enough accepted tokens per speculative step.

Experimental setupExperimental setup

We evaluated SlimSpec using the EAGLE-3 drafter setup across three target models:

• Llama-3.1-8B-Instruct
• GPT-OSS-20B
• Qwen3-30B-A3B-Instruct-2507.

The training corpus was constructed from 660K prompts from Infinity-Instruct-0625, with responses generated by the corresponding target model. This matches the drafter training distribution to target-model samples encountered during inference.

All drafters were trained with 6 speculative tokens and shared weights across positions. We evaluated on three benchmarks:

• MT-Bench — instruction following
• HumanEval — code generation
• GSM8K — mathematical reasoning.

Experiments were run greedy decoding (temperature =0, using batch sizes 1 and 64 to cover latency-critical and high-throughput serving regimes. The inference setup was based on vLLM 0.17.1 with NVIDIA H200 GPUs.

ResultsResults

SlimSpec is designed to improve end-to-end speculative decoding throughput, not just reduce the cost of the draft LM-head in isolation. We therefore evaluate each method by how much LM-head latency it removes and how much acceptance quality it preserves.

End-to-end speedup decomposition in the $(\nu, \rho_\tau)$ plane for Llama-3.1-8B at batch size 1 $(\kappa = 0.25)$. Dashed lines are theoretical speedup level curves. The shaded region indicates no end-to-end improvement over the full-vocabulary baseline. SlimSpec (red stars) achieves the largest LM-head acceleration $\nu$ while keeping $\rho_\tau$ close to 1.

Static vocabulary truncation methods such as FR-Spec, VocabTrim-T, and BCL follow a clear trade-off curve: smaller vocabularies reduce LM-head cost, but they also reduce acceptance quality. This is expected — once tokens are removed from the draft vocabulary, the drafter can no longer propose them, which limits the achievable overlap with the target distribution.

The dynamic vocabulary truncation method SpecVocab improves this trade-off by preserving more acceptance at similar LM-head latency. However, it introduces context-dependent token selection and additional inference-time routing logic.

SlimSpec shifts the frontier further. Its default rank configuration substantially reduces LM-head latency while keeping acceptance close to the full-vocabulary baseline. This makes it a better operating point for end-to-end speculative decoding than methods that reduce the vocabulary itself.

We next compare SlimSpec against the strongest representative baselines across target models, benchmarks, and serving regimes.

Comparison of SlimSpec with the main baselines. Speedup is measured relative to vanilla inference without a speculator. MT, HE, and GSM denote MT-Bench, HumanEval, and GSM8K, respectively.

The table shows that SlimSpec consistently improves over the full-vocabulary draft LM-head and competitive vocabulary-truncation baselines. For Llama-3.1-8B-Instruct, SlimSpec delivers the best average speedup in both low-latency and high-throughput regimes. For GPT-OSS-20B, it again provides the strongest average result, with especially clear gains at batch size 1. For Qwen3-30B-A3B-Instruct, the improvement is smaller because non-head costs account for a larger share of total latency, leaving less room for LM-head compression to affect end-to-end throughput.

Overall, these results show that SlimSpec provides a strong acceptance-cost trade-off while preserving full-vocabulary support. It avoids the hard acceptance ceiling of static vocabulary truncation and the inference-time routing overhead of dynamic vocabulary methods, making it a simple and practical choice for draft LM-head acceleration.

Want to apply speculative decoding to your own workload?Want to apply speculative decoding to your own workload?

Custom Speculator Training is now live in Nebius Token Factory. Teams can train workload-specific draft models from their own data, then work with Nebius to evaluate and deploy those drafters against real production traffic.

Talk to your SA about a spec decoding performance review: Custom Speculator Training, Token Factory by Nebius