Multi-Token Prediction: The Biggest Free Lunch in Local LLM Inference

Every autoregressive LLM has the same bottleneck. One forward pass, one token. For a 27-billion-parameter model, generating 100 tokens means loading the entire weight matrix from VRAM 100 times — and every single pass does roughly the same amount of work, whether the next token is a predictable filler word or a complex logical continuation.

The GPU is not compute-bound. It is memory-bandwidth-bound. It spends most of its time shuttling parameters from HBM to the compute units, not doing math. This is why a 27B model on an RTX 3090 generates at ~38 tok/s even though the GPU has 35.6 TFLOPS of FP32 compute — the arithmetic units sit idle while the memory bus catches up.

Multi-Token Prediction (MTP) fixes this by predicting several tokens in a single forward pass instead of one. It does not make the model smarter. It makes the model waste less computation. And for local inference — where memory bandwidth is the dominant constraint — that distinction matters enormously.

What MTP actually is

Standard decoding is strictly sequential:

Prompt: "The capital of France is"
     ↓
Predict → " Paris"     ← 1 forward pass
Append → "The capital of France is Paris"
     ↓
Predict → "."           ← 1 forward pass
Append → "The capital of France is Paris."
     ↓
Predict → " It"         ← 1 forward pass

Each token requires a full model forward pass. For a 27B model, that is 27 billion multiply-add operations per token, most of which are just reloading the same weights.

MTP changes the architecture at the prediction head level:

Hidden State
     ↓
┌────────────────────────────────────────────┐
│  Head 0 (main) → Token +1                  │
│  Head 1 (MTP)  → Token +2                  │
│  Head 2 (MTP)  → Token +3                  │
│  Head 3 (MTP)  → Token +4                  │
└────────────────────────────────────────────┘

The main transformer produces a hidden state. Instead of one output head mapping that state to a next-token distribution, there are multiple heads — each trained to predict a different future position. The MTP heads are lightweight (a fraction of the transformer’s parameters), so the extra cost is small. The main model still verifies everything.

During inference, the process is:

1. Main model generates token N
2. MTP heads draft tokens N+1, N+2, ..., N+k  (cheap pass)
3. Main model verifies ALL k drafts in ONE forward pass
4. Accept the correct prefix, reject the first mismatch
5. Continue generation from the last accepted token

Because verification is exact, the output is bit-for-bit identical to vanilla decoding. No quality loss. No approximation. Just fewer expensive forward passes.

This is self-speculative decoding — the model drafts with its own lightweight heads, not with a separate small model. No second model to load, no doubled memory, no management complexity.

Why Qwen3.6 is special here

MTP is not something you bolt onto any model after training. The prediction heads must be learned during pre-training — the loss function changes from predicting just token t+1 to predicting t+1, t+2, t+3, and t+4 simultaneously. Every training sample teaches multiple future predictions, which also provides a richer gradient signal that can improve the model’s internal representations.

Qwen3.6 and the Qwen3.5 family were trained with MTP from scratch. The heads are baked into the checkpoint. Unsloth ships dedicated MTP GGUF files that contain both the main model and the MTP module in a single file — you do not manage two models.

This is why older models (Llama 3, Mistral, etc.) cannot use MTP. They lack the prediction heads entirely. You can still use traditional speculative decoding with a separate draft model, but the overhead is much larger.

Real numbers: what to expect

The speedup range is 1.4–2.2x. That spread exists because acceptance rate varies by hardware, prompt type, and the --spec-draft-n-max setting. Here are actual benchmarks from independent sources:

RTX 3090 — Qwen3.6 27B Q4_K_M

Tested on a single RTX 3090 24GB with nine mixed prompts, n_predict=192:

Configuration	Mean tok/s	Wall time	Speedup	Acceptance
Baseline (no MTP)	38.3	50.6s	1.00x	—
MTP n=2	63.8	34.4s	1.47x wall / 1.67x mean	80.5%
MTP n=3	61.4	27.2s	1.86x wall / 1.60x mean	71.1%

Two speedup metrics because they measure different things. Mean tok/s averages each prompt’s individual generation rate — what you see while tokens stream. Wall-clock is what users feel: total time from prompt submitted to all responses complete. For agents and chat workflows, wall-clock is what matters.

The surprise: n=2 wins on per-prompt throughput for 8 of 9 prompts, but n=3 wins on total wall time because it spends less time on prompt prefill for long inputs. The difference is entirely driven by one prompt (a long code review) where n=3 finished in 5.8s vs n=2’s 13.3s.

Apple Silicon — Qwen3.6 27B Q8_0

On M-series Mac with unified memory:

Configuration	tok/s	Acceptance
Baseline	~7	—
MTP n=2	~16	82%
MTP n=3	~18–21	72%

That is a 2.3x speedup on n=2 — remarkable for a change that requires zero retraining and no separate model.

RTX 6000 Ada — Unsloth’s reported numbers

• Qwen3.6 27B MTP: 160 tok/s
• Qwen3.6 35B-A3B MTP: 240 tok/s

RTX PRO 6000

• Qwen3.6 27B: baseline 45.76 → MTP 79.37 tok/s = 1.73x
• Qwen3.6 35B-A3B MoE: baseline 193.36 → MTP 225.48 tok/s = 1.17x

Why MoE models benefit less

The 35B-A3B model is a Mixture-of-Experts architecture: 35B total parameters but only 3B active per token. Baseline decoding is already cheap because you only load a small fraction of weights per forward pass. Speculative decoding saves target-model forward passes — when those passes are already cheap, there is less to save. You still get absolute gains, but the relative speedup shrinks from 1.7–2.2x (dense models) to 1.1–1.2x (MoE).

The RTX 3090 + MoE caveat

A rigorous 19-config benchmark tested Qwen3.6-35B-A3B on a single RTX 3090 with llama.cpp’s draft speculation. The result: no configuration achieved net speedup over baseline. MoE expert routing saturates consumer Ampere memory bandwidth in a way that makes draft speculation counterproductive on llama.cpp.

However, vLLM with native MTP (method=mtp, num_speculative_tokens=1) on the same hardware hit +27.5% faster decode rate. The negative result is engine-specific, not hardware-class-independent. If you are running MoE models, test vLLM’s MTP rather than assuming llama.cpp draft speculation will help.

Acceptance rate patterns

The draft acceptance rate determines your effective speedup. Higher acceptance means more tokens per verification cycle. Observed patterns:

Content type	Typical acceptance
Code / structured output	80–86%
Dense models (27B) on general text	65–82%
MoE models (35B-A3B)	55–76%
Open-ended creative prose	Lower end

Code and structured prompts are the sweet spot because they are highly predictable. The next token after def calculate_ is almost certainly a function name. Creative writing has more branching possibilities, so the MTP heads are wrong more often.

Hardware requirements

MTP adds roughly 1–2.5 GB of extra memory over a standard GGUF. The overhead comes from the MTP head weights and their KV cache. Here is the full picture:

Qwen3.6	3-bit	4-bit	6-bit	8-bit	BF16
27B	16 GB	19 GB	25 GB	31 GB	56 GB
35B-A3B	18 GB	24 GB	31 GB	39 GB	71 GB

Units: total memory (RAM + VRAM, or unified memory on Mac).

Practical fit:

• RTX 3090/4090 (24 GB): 27B Q4_K_M fits comfortably. Budget ~20 GB for model + MTP + KV cache at moderate context.
• RTX 6000 Ada (48 GB): 35B-A3B at 6-bit or 8-bit with room for large context.
• Mac M-series (32+ GB): 27B Q4 or Q6 runs well. M4/M5 Max for best results.
• 16 GB GPUs: Tight. The MTP overhead may push past your limit at reasonable context sizes. Test with nvidia-smi during model load.
• 12 GB GPUs: 35B-A3B with vLLM + FP8 quantization is viable (9.5 GB VRAM reported), but only with aggressive quantization and limited context.

How to run it

Path 1: llama.cpp (most control)

MTP landed in mainline llama.cpp via PR #22673, merged May 16, 2026. No fork required — a stock build from recent master has it.

Build:

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release --target llama-server -j

Run:

./build/bin/llama-server \
  -m Qwen3.6-27B-MTP-Q4_K_M.gguf \
  -ngl 99 -c 10000 -fa on -np 1 \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0

Key flags:

Flag	What it does
--spec-type draft-mtp	Enable MTP speculative decoding
--spec-draft-n-max 2	Draft 2 tokens ahead per cycle
-np 1	Required. MTP does not support parallel slots.
-fa on	Flash attention. Significant speedup.
--cache-type-k q8_0	Quantized KV cache. Saves RAM with negligible quality loss.
--cache-type-v q8_0	Same for value cache.

Auto-download from HuggingFace (no manual GGUF download needed):

./build/bin/llama-server \
  -hf unsloth/Qwen3.6-27B-MTP-GGUF:Q4_K_S \
  -ngl 99 --spec-type draft-mtp --spec-draft-n-max 2 \
  -c 8192 -fa on -np 1

Important naming change: The flag was renamed from --spec-type mtp to --spec-type draft-mtp in the merged PR. Unsloth’s docs and several tutorials still reference the old name. If you use --spec-type mtp on a current build, it silently does nothing. Use draft-mtp.

Path 2: Unsloth Studio (easiest)

1. Install and launch Unsloth Studio
2. Search for “Qwen3.6 MTP” and download your quant
3. MTP settings are auto-detected for your hardware (Mac, CPU, GPU)
4. Override in the sidebar if you want to tune --spec-draft-n-max

This is the path of least resistance. Unsloth Studio handles the flag naming, hardware detection, and optimal defaults.

Path 3: vLLM (high throughput)

vllm serve unsloth/Qwen3.6-27B-NVFP4 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'

For the 35B-A3B MoE:

vllm serve unsloth/Qwen3.6-35B-A3B-NVFP4-Fast \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'

vLLM’s MTP implementation is the one that showed +27.5% on consumer Ampere where llama.cpp’s draft speculation failed for MoE models. If you are on vLLM, this is the better path.

Path 4: Ollama

Ollama’s support depends on the version of its underlying llama.cpp integration. Since MTP merged in May 2026, it should land in Ollama as the backend updates. Check your installed version.

Tuning –spec-draft-n-max

This is the single most important MTP parameter. It controls how many tokens the draft heads propose before verification.

The general advice is “start at 2, test 1 through 6.” That is correct but incomplete. Here is the nuance:

Why higher is not always better:

Draft 1 token  → 98% acceptance → fast verification
Draft 6 tokens → 20% acceptance → most work discarded

When acceptance drops, you are spending compute on drafting tokens that get thrown away. The verification pass is cheap but not free — it still has to run a forward pass on the rejected tokens before discarding them.

Hardware-specific sweet spots:

Hardware class	Best range	Why
Mac M-series	1–2	Lower memory bandwidth than discrete GPU
Consumer GPU (3090/4090)	2–3	Good compute, decent bandwidth
Datacenter GPU (A100/H100/RTX 6000)	3–6	High bandwidth, strong parallelism
CPU-only	1	Very limited

The wall-clock vs throughput tradeoff:

n=2 gives higher per-prompt acceptance rates (~80%) but spends more time on prefill. n=3 gives lower per-prompt acceptance (~71%) but finishes total work faster because it processes long prompts more efficiently. For coding agents making many short calls, n=2 wins. For long-form generation or analysis, n=3 wins.

The practical benchmarking loop:

for n in 1 2 3 4 5 6; do
  ./llama-server -m model.gguf \
    --spec-type draft-mtp --spec-draft-n-max $n \
    -ngl 99 -fa on -np 1 --metrics
done

Check draft_tokens_accepted / draft_tokens in the llama.cpp logs to find your acceptance rate per setting. The optimal n is the one where acceptance rate times n is maximized.

The prefill tax

MTP slows down prompt processing by roughly 17% on the same hardware. This means time-to-first-token goes up. For a short chat message, this is invisible — a few extra milliseconds. For RAG pipelines processing 8K+ context, the cost is real.

The breakeven is roughly 50 generated tokens. If you are generating fewer than 50 tokens per request, the prefill overhead may exceed the generation savings. For anything longer — code generation, document drafting, analysis — MTP pays for itself many times over.

What to avoid

Do not assume n=2 is optimal everywhere. Unsloth’s docs recommend it as a starting point. Community benchmarks show n=3 wins on wall-clock time for long prompts. Test on your actual workload.

Do not use the old flag name. --spec-type mtp was renamed to --spec-type draft-mtp. The old name silently does nothing on current builds. This catches people because the documentation has not caught up everywhere.

Do not enable parallel slots. -np 1 is required. MTP does not support multi-slot inference yet. If you need concurrent users, you need separate model instances.

Do not pair MTP with multimodal on llama.cpp. Text-only for now. If you need vision encoders active (mmproj), MTP is not compatible in current llama.cpp builds.

Do not expect big gains on MoE with llama.cpp. The community has documented this: 35B-A3B on consumer Ampere with llama.cpp draft speculation shows no net speedup. Use vLLM’s MTP instead for MoE models.

Do not benchmark only mean tok/s. Wall-clock time and time-to-first-token are equally important, especially for agentic workflows. A configuration that is 10% faster on mean tok/s but 30% slower on prefill may be worse for your use case.

The broader inference stack

MTP is one piece of a larger optimization stack. Each technique attacks a different bottleneck:

Technique	What it attacks	Overhead
Quantization (GGUF, NVFP4)	Memory footprint	Minimal quality loss
Flash Attention	Attention computation	None
Paged KV Cache	Long context memory	Minimal
MTP / Speculative Decoding	Forward pass count	~1–2.5 GB VRAM
Prefix Caching	Repeated prompts	Cache memory
Continuous Batching	GPU utilization across requests	Complexity

MTP is the one that attacks the generation bottleneck — the dominant cost for local inference. Quantization lets models fit; MTP makes them fast.

What to run today

If you have a 24 GB GPU and want to run Qwen3.6 locally, here is the concrete setup:

# Download and build llama.cpp
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release --target llama-server -j

# Run with MTP
./build/bin/llama-server \
  -hf unsloth/Qwen3.6-27B-MTP-GGUF:Q4_K_S \
  -ngl 99 -c 8192 -fa on -np 1 \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  --cache-type-k q8_0 --cache-type-v q8_0

Open http://localhost:8080 in your browser. You are running a 27B model at 60+ tok/s generation on a consumer GPU, with no accuracy loss, no separate draft model, and no quality tradeoff.

That was not possible six months ago.

---

MTP support in llama.cpp: PR #22673 (merged May 16, 2026). MTP GGUFs: unsloth/Qwen3.6-27B-MTP-GGUF and unsloth/Qwen3.6-35B-A3B-MTP-GGUF. Full MTP research notes: [[MTP-Multi-Token-Prediction-Speculative-Decoding-Deep-Dive-2026]].