Speculative decoding
Speculative decoding speeds up token generation by letting a small Multi-Token-Prediction (MTP)
drafter — shipped inside the .litertlm file — propose several tokens ahead, which the main
model then verifies in a single forward pass. When the drafter is accurate, many tokens are accepted
per main-model step, so decode throughput goes up without changing the output distribution. On the
hardware we measured it has not paid off yet: it slowed gemma-4-E2B down on desktop CPU and GPU (see
Measured results).
LiteRtLmSharp exposes it as an engine-level flag with a per-conversation override, plus the native benchmark API to measure the effect.
Requirements
- A model that ships an MTP drafter. The Gemma 4 builds (
gemma-4-E2B-it,E4B,12B) do, andLiteRtModelInfo.SupportsSpeculativeDecodingtells before loading. On a model without a drafter the flag fails: engine creation throwsLiteRtException(Ministral 3 reportsNOT_FOUND: tf_lite_mtp_drafter not found in the model), and a per-conversationtruefails the first send. - The engine flag is fixed at engine creation. Since native v0.18.0 a conversation can override it:
LiteRtConversationOptions.EnableSpeculativeDecoding = trueon an engine loaded without it makes the runtime load the drafter lazily on that conversation's first send, andfalseturns it off for one conversation of a speculative engine. The runtime keeps a lazily loaded drafter on the engine and would use it for every later conversation that leaves the setting unset (on CPU those decoded at 0.80× of the plain speed); the binding passes the engine setting to them explicitly, so unset keeps meaning the engine setting. On the WebGPU GPU backend (desktop), disable the disk cache— fixed in LiteRT-LM v0.14.0. On v0.13.1 the drafter's shared weight-cache file failed to open ("Access denied") on Windows and engine creation failed unlessCache = LiteRtCache.Disabled(upstream #2572; full investigation in Root cause). The fix is in the v0.14.0 tag and re-verified end to end with our v0.14.0 natives (engine create + generate + cache readback across two loads, win-x64 WebGPU): the default disk cache is safe again, and warm loads benefit from it.
Using it
using var engine = LiteRtEngine.Load(new LiteRtEngineOptions
{
ModelPath = "gemma-4-E2B-it.litertlm",
Backend = LiteRtBackend.Gpu,
MaxNumTokens = 4096,
EnableSpeculativeDecoding = true, // MTP drafter
EnableBenchmark = true, // so GetBenchmarkInfo() reports timings
// On LiteRT-LM v0.13.1 natives, speculative + WebGPU GPU also required
// Cache = LiteRtCache.Disabled (upstream #2572); fixed in v0.14.0.
});
using var chat = engine.CreateConversation();
chat.Send("Hello!");
if (chat.GetBenchmarkInfo() is { NumDecodeTurns: > 0 } b)
Console.WriteLine($"{b.LastDecodeTokensPerSecond:F1} tok/s decode · TTFT {b.TimeToFirstTokenSeconds:F2}s");
// Native v0.18.0+: opt one conversation out of the engine setting (or in, on an engine loaded without it).
using var plain = engine.CreateConversation(new LiteRtConversationOptions { EnableSpeculativeDecoding = false });
GetBenchmarkInfo() works after both blocking (Send) and streaming
(SendStreamingAsync) generation. It returns null when benchmarking was not enabled or no
turn has completed yet; it throws EntryPointNotFoundException on native binaries that predate the
benchmark API (the samples catch this and fall back to wall-clock timing).
In the samples
- Console:
--specflag, or the Switch model / backend / speculative menu option (default off). The context gauge then shows… · N tok/s decode · TTFT ….LiteRtLmSharp.Sample gemma-4-E2B-it.litertlm "Tell me a joke" --spec - MAUI: after picking the backend, a Speculative decoding prompt (shown only for MTP-capable
models). The three tab headers show
· speculative on/offand the gauge shows decode tok/s · TTFT.
(Until v0.14.0 both samples automatically set Cache = LiteRtCache.Disabled when speculative decoding
was enabled on the GPU backend; with #2572 fixed they use the default disk cache everywhere.)
How effectiveness is measured
The model test SpeculativeDecodingBenchmarkTests does an A/B: it loads the engine with the drafter
off, runs one fixed prompt, reads decode tokens/sec from the benchmark API, disposes the engine,
then repeats with the drafter on. It asserts both runs produce coherent text and that the
benchmark reports decode throughput, and logs the speedup ratio (a Markdown row) for this doc.
Run it locally (the gate keeps it out of the fast unit-test runs):
$env:LITERTLM_TEST_MODEL = "<path>\gemma-4-E2B-it.litertlm"
$env:LITERTLM_TEST_BENCH = "1" # also set LITERTLM_TEST_BACKEND=gpu to measure GPU
dotnet test LiteRtLmSharp.Tests/LiteRtLmSharp.Tests.csproj -c Release `
--filter "FullyQualifiedName~SpeculativeDecodingBenchmarkTests" --logger "console;verbosity=detailed"
In CI it runs on the CPU leg of model-tests.yml on each push via ci.yml (linux-x64 / linux-arm64 /
win-x64 / osx-arm64); the printed row lands in that job's console log and step summary.
Sampler note: the CPU sampler factory only implements the TopP sampler (Greedy and TopK return "not implemented yet"; verified on v0.14.0 and v0.15.0). Speculative decoding preserves the output distribution, not the exact token sequence under sampling, so the A/B checks coherence + throughput rather than asserting byte-identical output.
Measured results
gemma-4-E2B-it.litertlm, single fixed prompt, 128 max output tokens. Decode throughput is the
native benchmark API's decode_tokens_per_sec for the turn.
| Platform / backend | spec OFF | spec ON | speedup | Notes |
|---|---|---|---|---|
| win-x64 · CPU (dev box, LiteRT-LM v0.18.0, 2026-10-06) | 33.0 tok/s | 25.8 tok/s | 0.78× | official prebuilt; same ratio as on v0.13.1 |
| win-x64 · GPU WebGPU/D3D12, RTX 3080 (dev box, v0.18.0, 2026-10-07, F32 activations, the default since 1.3.0) | 100.3 tok/s | 51.8 tok/s | 0.52× | official prebuilt, default disk cache, GPU sampler embedded (no CPU-sampling fallback); two runs (100.3/51.8 and 100.0/51.7) |
| win-x64 · GPU WebGPU/D3D12, RTX 3080 (dev box, v0.18.0, 2026-10-07, F16 activations, the default before 1.3.0) | 87.7 to 96.4 tok/s | 37.7 to 43.5 tok/s | 0.43× to 0.45× | default disk cache; two runs (87.7/37.7 and 96.4/43.5) |
| win-x64 · GPU WebGPU/D3D12, RTX 3080 (dev box, v0.18.0, 2026-10-06, F32) | 91.4 tok/s | 60.5 tok/s | 0.66× | disk cache disabled by the benchmark (a v0.13.1 workaround, obsolete since v0.14.0) |
| win-x64 · GPU WebGPU/D3D12, RTX 3080 (dev box, v0.18.0, 2026-10-06, F16) | 84.1 tok/s | 56.8 tok/s | 0.68× | disk cache disabled by the benchmark, as above |
| win-x64 · CPU (dev box, 2026-06-15) | 29.9 tok/s | 23.4 tok/s | 0.78× | works, but slower — see below |
| win-x64 · GPU WebGPU/D3D12, RTX 3080 (dev box, 2026-06-15) | 41.8 tok/s | 35.5 tok/s | 0.85× | A/B both with cache off; plain GPU with the disk cache ≈85 tok/s |
| linux-x64 · CPU (CI ubuntu-latest, 2026-06-16) | 16.7 tok/s | 12.2 tok/s | 0.73× | from model-tests.yml |
| osx-arm64 · CPU (CI macos-15, 2026-06-16) | 21.1 tok/s | 20.5 tok/s | 0.97× | from model-tests.yml |
| linux-x64 · CPU (CI ubuntu-latest, v0.18.0, 2026-10-07) | 25.6 and 26.1 tok/s | 12.7 and 12.8 tok/s | 0.49× to 0.50× | two CI runs |
| linux-arm64 · CPU (CI ubuntu-24.04-arm, v0.18.0, 2026-10-07) | 25.2 and 26.0 tok/s | 17.9 and 17.9 tok/s | 0.69× to 0.71× | two CI runs |
| win-x64 · CPU (CI windows-latest, v0.18.0, 2026-10-07) | 16.1 and 25.8 tok/s | 8.8 and 20.6 tok/s | 0.55× to 0.80× | two CI runs; shared runners are noisy |
| osx-arm64 · CPU (CI macos-15, v0.18.0, 2026-10-07) | 21.1 and 12.8 tok/s | 17.1 and 11.7 tok/s | 0.81× to 0.92× | two CI runs; shared runners are noisy |
| android-arm64 · GPU OpenCL, Adreno 650 (Moto G100, 2026-06-16) | 13.9 tok/s | 14.1 tok/s | ~1.01× | runs correctly on GPU (drafter on OpenCL, GPU sampler active, no fallback); ~32% draft acceptance, too low to beat the drafter overhead on this older GPU |
Findings
- CPU regresses (every platform). The drafter + verification overhead is not amortized on CPU, so
speculative decoding is a net loss-to-neutral for this model: 0.73× (linux-x64 CI), 0.78×
(win-x64 dev box), 0.97× (osx-arm64 CI). On v0.18.0 the CI runners measured 0.49× (linux-x64) to
0.92× (osx-arm64), and the dev box again 0.78×. This matches the general result that
speculative decoding helps memory-bandwidth-bound (accelerator) decode, not compute-bound CPU
decode. Both outputs were coherent, full paragraphs, and the
*.mtp_drafter.xnnpack_cache_*file produced alongside the model confirms the drafter was actually engaged. - Desktop WebGPU GPU on the official prebuilts (v0.18.0): works, still slower. With the default disk cache and the GPU sampler embedded in the library (so neither factor below applies any more), the drafter halves the decode throughput on an RTX 3080: 100.3 → 51.8 tok/s (0.52×) with the F32 activations the binding uses by default, 0.43× to 0.45× with F16. With the disk cache disabled, as the benchmark ran until October 2026, plain decoding is slower and the ratio less bad (0.66× with F32, 0.68× with F16). The drafter's overhead, not the sampling path, dominates at this acceptance rate.
- Desktop WebGPU GPU on v0.13.1 worked (with the cache off), but didn't help. With
Cache = LiteRtCache.Disabledthe engine loads and the drafter speculates on the GPU (the CLI reports ~0.32 draft-acceptance on this prompt). In a fair A/B with the cache off on both legs, spec is a slight regression (35.5 vs 41.8 tok/s, 0.85×). Two compounding factors: our package's WebGPU sampler does not load (LiteRtTopKWebGpuSampler_UpdateConfigis missing → CPU-sampling fallback), which is expensive in the tight draft/verify loop; and disabling the disk cache itself costs throughput here (plain GPU with the cache runs ~85 tok/s). So the drafter's overhead isn't recovered on this desktop config. - A real accelerator (Adreno 650) also shows no win here. Validated on a Moto G100 (Android 12,
Adreno 650, OpenCL): MTP runs correctly — logcat confirms
enable_speculative_decoding: true, the drafter compiles on the OpenCL delegate (itsmtp_draftersubgraph initializes on GPU alongside decode/prefill/verify), and the GPU sampler loads (measured with the self-built set and its Androidpatchelf; the official v0.16.0 library embeds the sampler, so there's no CPU-sampling fallback here, unlike desktop WebGPU on v0.13.1). Yet throughput is flat: 14.1 (on) vs 13.9 (off) tok/s, ~1.01×. The drafter's acceptance rate is ~32% (399 drafted, 126 verified) — essentially identical to desktop (~0.317), so acceptance is model/prompt-bound, not hardware-bound. At ~32% acceptance the per-step drafter cost roughly cancels its benefit on this older GPU. Upstream's ≈3× (LiteRT-LM#2211) presumably needs a newer/faster accelerator (cheaper verification relative to the drafter) and/or a higher-acceptance workload; a recent flagship GPU is the remaining thing to try.
Root cause: the disk cache
Our binding first appeared to fail GPU+MTP at engine creation. Running Google's official litert-lm
CLI (pip install litert-lm, v0.13.1 — the same engine version) on the same machine and model
reproduced the identical failure with the default --cache disk:
delegate_webgpu.cc] Failed to create litert::ml_drift::DelegateKernelLiteRt: INTERNAL:
Could not map file ('…gemma-4-E2B-it.litertlm_…_mldrift_weight_cache.bin'): Access denied.
serialization_weight_cache/mmap_handle.cc:147 → … → llm_litert_mtp_drafter.cc:197
With --cache no the CLI succeeds and the drafter runs. So it is not a binding bug: with MTP
the main model and the drafter share one …_mldrift_weight_cache.bin, and on Windows the second
consumer's mmap of that file fails (a file-sharing violation). Pointing the cache at a different
directory does not help (same shared file, same error) — only disabling the disk cache avoids it.
The CLI's --cache no maps to litert_lm_engine_settings_set_cache_dir(settings, ":nocache"), which
we now expose as Cache = LiteRtCache.Disabled.
Source-level (v0.13.1): CreateCompilationOptions in runtime/executor/llm_executor_settings_utils.cc
applies the .mtp_drafter cache suffix only on the CPU/XNNPACK branch, not the GPU/MlDrift branch, so
the main model and the drafter resolve to one …_mldrift_weight_cache.bin — which is why a custom
directory doesn't help. The Windows share-mode that turns the collision into "Access denied" lives in
the closed libLiteRtWebGpuAccelerator (inferred ERROR_SHARING_VIOLATION).
Fixed in v0.14.0 (commit 4aa96a019 applies the cache_suffix on the GPU/MlDrift branch too, plus
the follow-up 0a6590988; both are in the tag). Re-verified 2026-07-10 with our v0.14.0 natives on
win-x64 WebGPU: spec + default disk cache creates, generates, and reads the cache back across engine
reloads. The whole section above is kept as the v0.13.1 historical record.
Upstream tracking (researched 2026-06-15)
- Cache: no dedicated issue for the Windows/WebGPU case yet — #2503 is the iOS/Metal sibling, and a buried comment on #2461 reports the exact trace as a regression from v0.12.0 (without MTP, so the collision is broader than MTP). A new upstream issue is warranted.
- CPU-sampling fallback: this was #2073 (WebGPU sampler exports 3/7 C-ABI functions on macOS/Windows → CPU fallback), closed upstream on 2026-09-22. It no longer applies: the official prebuilts the binding ships since 1.2.0 embed the GPU samplers in the library. Per flutter_gemma #287 the steady-state cost had been small (~3%), but it weighed more on the speculative draft/verify loop.
In short: the flag works on every backend we tested with the default cache (v0.13.1 needed
Cache = LiteRtCache.Disabled on desktop WebGPU), but on desktop CPU and GPU it slows decoding down for
gemma-4-E2B (0.78× and 0.52× on v0.18.0), and an older mobile GPU (Adreno 650) shows no win at ~32%
acceptance. Measure it on your target accelerator and workload before turning it on, and use the
per-conversation override to keep it where it pays off.