Engine tuning and benchmarking
A handful of LiteRtEngineOptions settings trade off load time, memory, precision and speed, and let
you measure the result. They are all fixed at engine creation (LiteRtEngine.Load) and most are
advanced knobs — the defaults are good for typical use, so reach for these only when you have a
specific load-time, memory or throughput goal.
This guide explains what each one does, when it helps, and what it costs. Two related performance
features have their own pages: speculative decoding (the MTP drafter) and
the compiled-artifact cache (LiteRtEngineOptions.Cache).
Activation precision — ActivationDataType
The precision of the activation tensors during inference. The binding defaults to Float32 (since
1.3.0). The runtime's own default for the text executor on GPU is F16, which corrupts output on the GPUs
we measured (below) at no speed gain on a desktop GPU.
using var engine = LiteRtEngine.Load(new LiteRtEngineOptions
{
ModelPath = "gemma-4-E2B-it.litertlm",
Backend = LiteRtBackend.Gpu,
ActivationDataType = LiteRtActivationDataType.Float16, // opt in to half precision (default: Float32)
});
Only the GPU backend honors this, and only as F32 vs F16.
Float32runs activations at full precision.Float16uses half the activation memory, with a precision loss that is NOT always small (below).nulllets the runtime choose: the precision the model file declares for its text model (F16 for the Gemma 4 E-series), else F16 on GPU.It sets the text executor only. The vision and audio encoders run at the precision the model file declares for them, else F32, and no option changes that. The Gemma 4 E-series declares F16 for its vision encoder, so on GPU its images are encoded in F16 whatever this setting says.
On CPU it is a no-op: the CPU/XNNPACK path does not read it (the executor only switches the GPU delegate to FP16 when the setting is F16 and the backend is GPU).
Int16/Int8are accepted but not distinctly implemented by the shipped executors: on GPU they fold into F16, on CPU they are ignored. They exist only to mirror the native enum — do not expect 8/16-bit activation quantization from them.What F32 costs. gemma-4-E2B, LiteRT-LM v0.18.0, medians of 3 runs:
GPU Decode F16 → F32 Time to first token F16 → F32 Memory RTX 3080 (Windows, WebGPU) 113 → 117 tok/s about the same F32 commits 0.3 to 0.6 GB more Adreno 650 (Moto G100, Android 12, OpenCL) 15.2 → 14.1 to 14.8 tok/s 0.31 → 0.57 s F32 uses about 150 MB more RAM Workload: one engine, a 60-token prompt, 256-token replies, the median of three runs after a warm-up. The speculative-decoding page measures the first reply of a freshly loaded engine (128 tokens), which reads lower (100 tok/s with F32 on the same GPU).
On the phone F32 costs about 5% of decode speed and doubles the time to first token (prefill runs slower). F16 did not corrupt our quick digit, date and counting checks there, but it did corrupt structured extraction on desktop GPUs (below). If the phone's speed matters more, try
Float16and check your own outputs.
F16 corrupts structured output on desktop GPU — keep the Float32 default
If you set ActivationDataType to Float16 (or to null, which lets the runtime pick F16 on GPU) and your
GPU outputs show corrupted digit sequences (dates like 206-15-2023 or 195959-06-17, truncated or
looping numbers), degraded reasoning versus the same model on CPU, or results that vary run-to-run
at temperature 0, the cause is very likely the F16 activations, not your prompts and not the model. This failure mode is widely reported against LiteRT-LM but hard to
find the real knob for (upstream threads blame the sampler DLL, suggest repetition penalties, or go
unanswered — see google-ai-edge/LiteRT-LM#2637, #2727, #2202, and the export-gap/sampler issues
#2073/#2080), so it is documented here with what we measured:
- On win-x64 WebGPU (RTX 3080, temp 0), a 16-check benchmark of structured extraction from free text
FAILS 13/16 with rotating errors on F16 (the default before 1.3.0) — digit sequences corrupted at emission (before
any post-processing), dates resolved against the wrong reference, relations between extracted
entities inverted or attached to the wrong entity — and passes 16/16 across 3 consecutive runs
with
Float32, with clean digits in the raw output and no measurable speed cost on that GPU (~32 s either way; F32 compute was not the bottleneck). - The same F16→F32 flip reproduces on a non-Gemma model (Ministral-3-3B q4: 13/16 → 16/16), so it is a property of the GPU text executor's precision, not of one model family.
- F16 did not only corrupt digits — it degraded reasoning quality (which entity a fact belongs to, which date a relative expression resolves to), which is easy to misattribute to the model or the prompt.
- CPU is immune (the knob is GPU-only, and the CPU path never showed the corruption).
This is why the binding defaults to Float32 since 1.3.0. Before that, the default followed the
runtime (F16 on GPU) and the recommendation was to set Float32 by hand. If you opt into Float16, A/B
it once on your target GPU with your own outputs: the cost of F32 is activation memory and possibly
speed on GPUs with weak F32 throughput; on the desktop GPU we measured (RTX 3080) it cost no speed.
Prefill chunk size — PrefillChunkSize
How many prompt tokens the engine prefills per step. 0 (default) means no chunking — the whole prompt
is prefilled in one pass.
PrefillChunkSize = 256, // prefill 256 tokens at a time
- CPU + dynamic models only. It is read only by the CPU "dynamic" executor (models whose KV-cache and sequence-length tensors are dynamically sized). On GPU, or on static models, it is a no-op (the native layer silently ignores the call).
- Smaller chunks lower peak memory during prefill — intermediate activations are sized to the chunk, not the whole prompt — and allow more timely cancellation of a long prompt.
- Trade-off: more chunks mean more prefill iterations and per-chunk overhead, so total prefill can be slower. Leave it off (no chunking) for the shortest prefill latency.
- When to set it: a long prompt on a memory-constrained CPU device, or where you want responsive cancellation partway through prefill.
Parallel file-section loading — ParallelFileSectionLoading
Whether the engine loads parts of the .litertlm file concurrently during startup. null (default)
leaves it on.
ParallelFileSectionLoading = false, // serial, single-threaded load
- A
.litertlmfile is a container of typed sections (tokenizer, model graph, weights, metadata). With this on (the default), the tokenizer section is parsed on a background thread while the model/executor is built, overlapping the two and shortening cold-start init. - With it off, that load is deferred onto the calling thread and serialized after model construction — slower startup, but single-threaded.
- When to set
false: single-threaded environments (e.g. WASM without pthreads), or to avoid the brief concurrent IO/thread peak during init, accepting a slower start.
Experimental YNNPACK delegate — EnableYnnpack
Lets the experimental YNNPACK delegate take the
CPU operations it supports before XNNPACK. null (default) leaves the engine default (off). Requires native
LiteRT-LM v0.16.0+.
EnableYnnpack = true, // CPU backend, linux-arm64 only
- CPU backend only; the GPU backends ignore it.
- linux-arm64 only. Of the official libraries, only linux-arm64 (the Raspberry Pi target) carries the
YNNPACK kernels (a model test on CI's arm64 leg loads an engine with the flag on and generates). On every other platform the v0.18.0
libraries reject it at engine creation with
UNIMPLEMENTED(v0.16.0 ignored it), and the exception names the setting. Measure on the target device before assuming a speed-up.
Vision token cap — MaxVisionTokensPerImage
An upper bound on the vision tokens one image may expand to: the engine loads the vision encoder signatures up to the cap, rounded up to the next signature (native v0.18.0+). It caps the size; the per-image budget chooses it.
MaxVisionTokensPerImage = 140, // engine-wide ceiling (Gemma 4 signatures: 70, 140, 280)
// and per conversation (or per send), at or below the cap:
// new LiteRtConversationOptions { VisualTokenBudget = 140 }
- Images still expand to the model's default size (about 256 tokens on Gemma 4) unless
VisualTokenBudgetasks for less, so a cap below that default needs a budget at or below the cap on every image send. Without one the send fails withINVALID_ARGUMENT("No signature found…"), and a budget above the cap fails the same way. - A cap above the model's largest signature (280 on Gemma 4) fails
LiteRtEngine.LoadwithINVALID_ARGUMENT("Requested target capacity (300) exceeds maximum available signature length (280)"). A cap between signature sizes loads the next larger one: with a cap of 100, budgets up to 100 work (an image costs 104 tokens with a budget of 100). - To make images cheaper, the budget alone is enough:
VisualTokenBudget = 70brings an image from 260 to 68 tokens on gemma-4-E2B-it, with or without a cap. See the multimodal section.
Metal residency — EnableMetalResidencySet
Keeps the model weights and allocations resident in GPU memory through Apple's MTLResidencySet API
(native v0.18.0+). Apple platforms with the Metal GPU backend only; other platforms and backends ignore
it, which the model-backed test checks on Windows. On macOS the GPU backend runs WebGPU over Metal (see the
native ABI notes), so measure its effect on the target device rather than assume one.
CPU thread counts: NumThreads / AudioNumThreads
How many CPU threads the text and audio executors use. null (default) lets the engine pick its own
count.
NumThreads = 4, // text executor CPU threads
AudioNumThreads = 2, // audio executor CPU threads (only when audio is enabled)
- CPU backend only. These set the thread counts for the CPU/XNNPACK executors and are a no-op on
non-CPU backends (GPU and NPU manage their own parallelism).
AudioNumThreadsapplies only when an audio executor is configured (AudioBackendset); otherwise it is ignored. - Non-positive values are rejected: pass a positive count, or leave it
nullfor the default. - When to set it: pin the engine to a subset of cores on a busy machine, or tune throughput on a device where the default (all cores) contends with the rest of the app. More threads help prefill and decode up to the memory-bandwidth limit; past that they only add scheduling contention.
LoRA adapters: engine ranks and adapter paths
Loading a low-rank adaptation on top of a LoRA-enabled base model has two layers, both added in v0.14.0:
- Engine-level
LiteRtEngineOptions.LoraRank/SupportedLoraRanks(and theAudioLoraRank/SupportedAudioLoraRanksvariants) declare the adapter rank(s) the engine compiles for. The supported-ranks lists are only honored on the GPU (Artisan) backend. - Per-conversation
LiteRtConversationOptions.LoraPath/AudioLoraPathpoint at the adapter file, which is opened when the conversation is created, so a bad path fails fast withLiteRtExceptionatCreateConversation, not mid-generation.
Requires a LoRA-enabled model. What the runtime does at v0.18.0, measured on CPU with upstream's LoRA test bundle and its rank-32 adapter:
- The adapter changes generation, deterministically (the v0.14.0 and v0.15.0 runtimes rejected every text adapter with "Lora is not supported").
- It applies to the whole engine, not to its conversation. After a conversation with
LoraPathhas generated, conversations without an adapter on the same engine generate with it too. While a conversation with an adapter exists, a send on a conversation without one fails withLiteRtStatusCode.Internal("No LoRA ID is set"). - So use one adapter per engine. To switch adapters or to return to the base model, dispose the conversations and the engine and load it again.
- The published gemma-4 bundles carry no LoRA slots
(LiteRT-LM#3173): an adapter is accepted
and has no effect on them, with or without
LoraRank. - The GPU backend is not validated: the test bundle does not load on it.
Tests in LoraTextAdapterTests pin these behaviors, so a runtime that changes them shows up as a failing
test.
Benchmarking
Measuring real runs — EnableBenchmark
Turn on benchmark instrumentation and read the timings off the conversation after a turn. The overhead is timing bookkeeping only.
using var engine = LiteRtEngine.Load(new LiteRtEngineOptions
{
ModelPath = "gemma-4-E2B-it.litertlm",
Backend = LiteRtBackend.Cpu,
EnableBenchmark = true,
});
using var chat = engine.CreateConversation();
chat.Send("Write a short paragraph about the printing press.");
if (chat.GetBenchmarkInfo() is { NumDecodeTurns: > 0 } b)
Console.WriteLine(
$"prefill {b.LastPrefillTokensPerSecond:F1} tok/s · decode {b.LastDecodeTokensPerSecond:F1} tok/s · " +
$"TTFT {b.TimeToFirstTokenSeconds:F2}s · init {b.TotalInitTimeSeconds:F2}s");
GetBenchmarkInfo() returns null when benchmarking was not enabled or no turn has completed. It
exposes the latest turn's LastPrefillTokenCount / LastDecodeTokenCount, the matching
LastPrefillTokensPerSecond / LastDecodeTokensPerSecond, plus TimeToFirstTokenSeconds and
TotalInitTimeSeconds.
Synthetic, content-independent benchmark — BenchmarkPrefillTokens / BenchmarkDecodeTokens
To measure raw throughput at fixed token counts, independent of any prompt, set the synthetic token counts. A send then runs a benchmark instead of answering:
using var engine = LiteRtEngine.Load(new LiteRtEngineOptions
{
ModelPath = "gemma-4-E2B-it.litertlm",
Backend = LiteRtBackend.Cpu,
BenchmarkPrefillTokens = 512, // prefill exactly 512 tokens
BenchmarkDecodeTokens = 128, // decode exactly 128 tokens
});
using var chat = engine.CreateConversation();
chat.Send("anything"); // the text is ignored — this is a benchmark run
var b = chat.GetBenchmarkInfo()!; // reports 512 prefill / 128 decode
Console.WriteLine($"prefill {b.LastPrefillTokensPerSecond:F1} tok/s · decode {b.LastDecodeTokensPerSecond:F1} tok/s");
How it works: the prompt is padded or truncated to BenchmarkPrefillTokens for prefill, and decoding
runs exactly BenchmarkDecodeTokens tokens, ignoring the stop token. The measured prefill/decode rates
therefore depend only on the token counts and the engine settings — ideal for comparing devices, or for
measuring the effect of ActivationDataType / PrefillChunkSize reproducibly.
Two caveats:
- The reply is not a real answer — it is a synthetic run. Use a dedicated engine for benchmarking, not one you also chat with.
- Setting either count turns benchmark mode on (the same switch as
EnableBenchmark), soGetBenchmarkInfo()works even withoutEnableBenchmark.
This is the mechanism the project's CI A/B harness uses to compare speculative decoding on and off; the measured numbers and method are in speculative-decoding.md.