Class LiteRtEngineOptions
- Namespace
- LiteRtLmSharp
- Assembly
- LiteRtLmSharp.dll
Options for creating a LiteRtEngine.
public sealed record LiteRtEngineOptions : IEquatable<LiteRtEngineOptions>
- Inheritance
-
LiteRtEngineOptions
- Implements
- Inherited Members
Properties
ActivationDataType
Activation tensor precision. Defaults to Float32. Only the GPU backend honors this, and only as F32 vs F16; on CPU it is a no-op.
public LiteRtActivationDataType? ActivationDataType { get; init; }
Property Value
Remarks
The runtime's own GPU default for the text executor is F16, which corrupts structured output
(digits, dates, JSON) on the GPUs we measured, with no speed gain on a desktop GPU: gemma-4-E2B on an
RTX 3080 decodes 113 tok/s with F16 and 117 with F32, and F32 commits about 0.3 to 0.6 GB more. So
the binding asks for F32. Set Float16 to trade precision for
activation memory (for example on a mobile GPU, after checking your outputs), or null to let
the runtime choose: the precision the model file declares for its text model (F16 for the Gemma 4
E-series), else F16 on GPU. The setting reaches the text executor only: the vision and audio encoders
run at the precision the model file declares for them (F16 for the Gemma 4 E-series vision encoder),
else F32. Int16 and
Int8 are accepted by the native API but not distinctly
implemented by the shipped executors (folded to F16 on GPU). Maps to
engine_settings_set_activation_data_type. See docs/engine-tuning.md.
AudioBackend
Backend for the audio encoder, enabling audio input. null (default) leaves audio
unconfigured — audio attachments will not work. Requires a model with audio support (e.g. the
Gemma 4 E-series); a no-op on models without it.
public LiteRtBackend? AudioBackend { get; init; }
Property Value
Remarks
A model may constrain which backend its audio encoder accepts. The Gemma 4 audio sub-model
requires CPU: passing Gpu makes litert_lm_engine_create
fail with "Audio backend constraint mismatch. Model requires one of [cpu]" even when
Backend is GPU — on any platform, not a win-x64 quirk. Use
Cpu for such models; the vision encoder has no such constraint and
runs on GPU. Maps to the audio_backend_str parameter of the C API
engine_settings_create.
AudioLoraRank
LoRA rank for the audio executor. null (default) leaves the engine default. Only
applies when an audio executor is configured (AudioBackend set). Maps to
engine_settings_set_audio_lora_rank. Unlike the text path, audio LoRA has not been validated
against a real adapter in this binding.
public int? AudioLoraRank { get; init; }
Property Value
- int?
Exceptions
- ArgumentOutOfRangeException
The value is zero or negative.
AudioNumThreads
Number of CPU threads for the audio executor. null (default) leaves the engine
default. Only applies when an audio executor is configured (AudioBackend set), and
only reaches the audio backend's CPU compilation options — a no-op otherwise (verified in
engine.cc set_audio_num_threads → audio_executor CPU options). Maps to
engine_settings_set_audio_num_threads.
public int? AudioNumThreads { get; init; }
Property Value
- int?
Exceptions
- ArgumentOutOfRangeException
The value is zero or negative.
Backend
Backend to run the model on. Defaults to Cpu. Use Gpu, Npu, or Custom(string) for a backend exposed by your own native build.
public LiteRtBackend Backend { get; init; }
Property Value
BenchmarkDecodeTokens
Synthetic decode token count for benchmarking — the decode half of
BenchmarkPrefillTokens. 0 (default) = off. See BenchmarkPrefillTokens
for the full behavior. Maps to engine_settings_set_num_decode_tokens.
public int BenchmarkDecodeTokens { get; init; }
Property Value
BenchmarkPrefillTokens
Synthetic prefill token count for benchmarking. 0 (default) = off (normal inference).
public int BenchmarkPrefillTokens { get; init; }
Property Value
Remarks
When > 0 the engine runs a synthetic benchmark instead of answering: the real prompt is
truncated or padded to exactly this many tokens for prefill, and decoding runs exactly
BenchmarkDecodeTokens tokens (ignoring the stop token). So
GetBenchmarkInfo() reports prefill/decode throughput at FIXED token
counts — independent of the prompt — which is useful for device throughput benchmarking and for
measuring the effect of the tuning settings reproducibly. The reply text is not a real
answer. Setting either this or BenchmarkDecodeTokens also turns benchmark mode on
(the same switch as EnableBenchmark). Use a dedicated engine instance — do not reuse
it for real chat. Verified observable through the Conversation API on win-x64 CPU (the default
engine reads these during prefill/decode). Maps to engine_settings_set_num_prefill_tokens.
Cache
Where the engine keeps its compiled-artifact cache (GPU shaders / converted weights), which speeds up subsequent loads. Defaults to Default (written next to the model file). Use Disabled, InMemory, or Directory(string) for an explicit path.
public LiteRtCache Cache { get; init; }
Property Value
Remarks
The cache files are named after the model file (its name, modification time and size), not after the native runtime version. A newer runtime detects a GPU cache written by an older one and rebuilds it on the first load, which makes that load slower once. A Directory(string) must exist: the runtime does not create it, and since native v0.18.0 Load(LiteRtEngineOptions) fails with InvalidArgument when it is missing.
EnableBenchmark
Enable benchmark instrumentation so GetBenchmarkInfo() returns prefill/decode tokens-per-second, time-to-first-token and init time. Fixed at engine creation; the overhead is timing bookkeeping only.
public bool EnableBenchmark { get; init; }
Property Value
EnableMetalResidencySet
Whether the GPU backend keeps the model weights and allocations resident in GPU memory through
Apple's MTLResidencySet API, which prevents them from being paged out and lowers allocation
overhead. null (default) leaves the engine default. Apple platforms with the Metal GPU
backend only; other platforms and backends ignore it.
public bool? EnableMetalResidencySet { get; init; }
Property Value
- bool?
Remarks
Maps to engine_settings_set_gpu_enable_metal_residency_set (native LiteRT-LM
v0.18.0+). The desktop macOS GPU backend runs on WebGPU over Metal; measure the effect on the
target device rather than assume one.
EnableSpeculativeDecoding
Enable speculative decoding: the model drafts several tokens ahead with a small Multi-Token-Prediction (MTP) drafter and the main model verifies them in one step. It pays off only where verification is cheap relative to the drafter (memory-bound accelerators, high acceptance); measure it before turning it on.
public bool EnableSpeculativeDecoding { get; init; }
Property Value
Remarks
Requires a .litertlm that ships an MTP drafter (e.g. the Gemma 4 E2B/E4B/12B builds;
SupportsSpeculativeDecoding tells before loading). On a model without
one, engine creation fails with LiteRtException (Ministral 3 reports NOT_FOUND
"tf_lite_mtp_drafter not found in the model"). The setting is fixed at engine creation;
EnableSpeculativeDecoding overrides it per conversation
(native v0.18.0+). Pair with EnableBenchmark to measure the effect (see
GetBenchmarkInfo()).
Measured with gemma-4-E2B on LiteRT-LM v0.18.0 (see docs/speculative-decoding.md): it
slows decoding down on desktop CPU (0.78×) and on the desktop WebGPU GPU backend
(0.52× on an RTX 3080 with the default float32 activations), with the default disk cache in both
cases. On an Adreno 650 phone GPU it made no difference (measured in June 2026 on LiteRT-LM
v0.13.1).
EnableYnnpack
Lets the experimental YNNPACK delegate take the CPU operations it supports before XNNPACK.
null (default) = engine default (off). CPU backend, linux-arm64 only. Requires native
LiteRT-LM v0.16.0+.
public bool? EnableYnnpack { get; init; }
Property Value
- bool?
Remarks
Maps to engine_settings_set_enable_ynnpack. Of the official libraries, only linux-arm64 carries
the YNNPACK kernels. Elsewhere, true makes Load(LiteRtEngineOptions) fail with
Unimplemented (native v0.18.0; v0.16.0 ignored the flag). Measure any
speed-up on the target device rather than assume one.
GpuDecodeStepsPerSync
Decode steps the GPU runs between host syncs. null (default) = engine default. Only
honored by the Artisan GPU backend (the mobile GPU stack); the desktop WebGPU backend ignores
it. Requires native LiteRT-LM v0.15.0+.
public int? GpuDecodeStepsPerSync { get; init; }
Property Value
- int?
Remarks
Maps to engine_settings_set_gpu_decode_steps_per_sync. Larger values reduce
sync overhead but coarsen streaming/cancellation granularity.
Exceptions
- ArgumentOutOfRangeException
The value is zero or negative.
GpuWaitForWeightUploads
Whether engine load blocks until GPU weight uploads complete. null (default) = engine
default. Only honored by the Artisan GPU backend; the desktop WebGPU backend ignores it.
Requires native LiteRT-LM v0.15.0+.
public bool? GpuWaitForWeightUploads { get; init; }
Property Value
- bool?
Remarks
Maps to engine_settings_set_gpu_wait_for_weight_uploads.
LoraRank
LoRA rank for the text executor. null (default) leaves the engine default (LoRA
disabled). Requires a LoRA-enabled model and a matching adapter passed per conversation via
LoraPath. Maps to engine_settings_set_lora_rank.
public int? LoraRank { get; init; }
Property Value
- int?
Remarks
The adapter applies to the whole engine, not to one conversation (LiteRT-LM v0.18.0, measured on CPU with upstream's LoRA test bundle): after a conversation with LoraPath has generated, conversations without an adapter on the same engine generate with it too, and while a conversation with an adapter exists, a send on one without it fails with Internal. Use one adapter per engine, and reload the engine to switch adapters or to return to the base model.
The published Gemma 4 bundles carry no LoRA slots (google-ai-edge/LiteRT-LM#3173): an adapter is accepted and has no effect on them. The GPU backend is not validated (the test bundle does not load on it).
Exceptions
- ArgumentOutOfRangeException
The value is zero or negative.
MaxNumImages
Maximum number of images the engine accepts per turn. 0 = engine default. On the standard binaries this setting has no effect — see the remarks for the case where it applies. To bound how much of the context window images consume, use VisualTokenBudget instead.
public int MaxNumImages { get; init; }
Property Value
Remarks
The native library compiles several engine implementations into one binary and picks one per
backend. This value always reaches the engine settings, but only the legacy TFLite
implementation reads it; the standard binaries select the modern CompiledModel engines for
CPU/GPU, which ignore it. It matters only when a custom native build routes your backend
(e.g. one targeted via Custom(string)) through the legacy engine. Kept for
parity with the reference Kotlin binding's maxNumImages. Maps to
engine_settings_set_max_num_images.
MaxNumTokens
The total context window in tokens: prompt + generated replies, accumulated across every turn of a conversation. 0 = engine default.
public int MaxNumTokens { get; init; }
Property Value
Remarks
This is the hard ceiling for a conversation's KV cache. Size it for the longest exchange you expect — the system prompt, any restored History, and each user turn and reply all count against it and accumulate. As the running total (TokenCount) nears the limit, generation degrades, so manage the conversation before then: trim history, cap replies with MaxOutputTokens, or start a fresh conversation. A larger window costs more memory and a slower prefill, so prefer the smallest that fits your use case. For multimodal input leave room for the media on top of the text — an image expands to roughly 256 vision tokens — for which 4096 is a comfortable starting point.
Setting this also arms the binding's KV overflow guard. The native runtime does not
police the limit: a send that grows past it writes beyond the allocated cache and corrupts native
memory (typically a deferred 0xC0000005 process crash on a later call). With an explicit
value here, the send methods instead clamp each reply to the remaining context and throw
LiteRtContextOverflowException when the conversation is full. The clamp cannot
cover sends whose prefill cost is unmeasurable managed-side — media attachments and
SendRaw's per-send extraContext get only the conversation-full check, so keep extra
headroom under the limit on those paths (an image is roughly 256 tokens). With 0 the effective
limit is internal to the engine (the C API exposes no getter), so the guard stays off.
Keep the value at or below the model's built-in context length. The native loader sizes the KV cache from this hint only when it is smaller than the model's own maximum; a value at or above it silently falls back to the model's default sizing, leaving the real cache SMALLER than this property claims — and the overflow guard, calibrated against this value, would then stop policing before the actual edge. Values below the model's minimum prefill work group (~128) are unusable outright: every send is rejected.
Keep the value at or above the model's largest prefill signature (1024 for the published gemma conversions). The native loader accepts a smaller limit, but a send whose prefill spans more than the smallest work group then fails inside the native graph (v0.18.0 reports "Failed to invoke the compiled model Failed to allocate tensors") instead of cleanly. The C API exposes no way to query the signatures, so the binding cannot validate this up front; when a send fails that way on an engine loaded with a limit below 1024, the exception names this as the likely cause.
MaxVisionTokensPerImage
Upper bound on the vision tokens one image may expand to, which limits the vision encoder
signatures the engine loads (up to the cap, rounded up to the next signature). null (default)
leaves every signature of the model available.
public int? MaxVisionTokensPerImage { get; init; }
Property Value
- int?
Remarks
It caps the image size; it does not choose it. Images still expand to the model's default size
(about 256 tokens for the Gemma 4 E-series) unless a visual token budget asks for less, so pair a cap
below that default with VisualTokenBudget (or
VisualTokenBudget) at or below the cap. Without one, an image send
fails with InvalidArgument ("No signature found…"), and an image send
whose budget exceeds the cap fails the same way ("Visual token budget (…) cannot be larger than the
engine's max vision tokens per image (…)"). Left null, the engine checks the budget against the
model's own maximum instead (MaxVisionTokenBudget, 280 on the Gemma 4
E-series). It applies only when VisionBackend is set and the model declares a per-image
token count. A cap above the model's largest signature (280 on the Gemma 4 E-series) fails
Load(LiteRtEngineOptions) with InvalidArgument. Maps to
engine_settings_set_max_vision_tokens_per_image (native LiteRT-LM v0.18.0+).
Exceptions
- ArgumentOutOfRangeException
The value is zero or negative.
ModelPath
Path to the .litertlm (or .task) model file. Must be set —
Load(LiteRtEngineOptions) throws ArgumentException when it is empty.
(Deliberately not required: future native versions add alternate model sources — e.g.
loading from a file descriptor — which will land here as sibling properties.)
public string ModelPath { get; init; }
Property Value
NumThreads
Number of CPU threads for the text executor. null (default) leaves the engine
default. CPU-backend only: the native setter reads the executor's CpuConfig and is a
no-op when the text backend is not CPU (verified in engine.cc set_num_threads). Maps to
engine_settings_set_num_threads.
public int? NumThreads { get; init; }
Property Value
- int?
Exceptions
- ArgumentOutOfRangeException
The value is zero or negative.
ParallelFileSectionLoading
Whether the engine loads the .litertlm file's sections in parallel during startup.
null (default) leaves the engine default (on). When on, the tokenizer section is parsed on
a background thread while the model is built, shortening cold-start init; set false to load
it serially on the calling thread (single-threaded environments, or to avoid the brief concurrent
init peak, at the cost of a slower start). Maps to
engine_settings_set_parallel_file_section_loading. See docs/engine-tuning.md.
public bool? ParallelFileSectionLoading { get; init; }
Property Value
- bool?
PrefillChunkSize
Maximum prompt tokens prefilled per step. 0 (default) = no chunking (the whole prompt is
prefilled at once). CPU + dynamic models only — ignored on GPU and on static models. A
smaller chunk lowers peak memory during prefill and allows more timely cancellation of a long
prompt, at the cost of more prefill iterations (potentially slower). Maps to
engine_settings_set_prefill_chunk_size. See docs/engine-tuning.md.
public int PrefillChunkSize { get; init; }
Property Value
SupportedAudioLoraRanks
The set of LoRA ranks the audio executor should support. null (default) leaves the
engine default. Requires an audio executor (AudioBackend set); see
SupportedLoraRanks for the GPU-backend caveat. Every element must be positive and the
list non-empty. Maps to engine_settings_set_supported_audio_lora_ranks.
public IReadOnlyList<int>? SupportedAudioLoraRanks { get; init; }
Property Value
Exceptions
- ArgumentException
The list is empty.
- ArgumentOutOfRangeException
Any element is zero or negative.
SupportedLoraRanks
The set of LoRA ranks the text executor should support (for switching adapters of different
ranks). null (default) leaves the engine default. Only honored on the GPU (Artisan)
backend: on other backends the native layer accepts and silently ignores it (verified in
llm_executor_settings.h SetSupportedLoraRanks). Every element must be positive and the list
non-empty. Maps to engine_settings_set_supported_lora_ranks.
public IReadOnlyList<int>? SupportedLoraRanks { get; init; }
Property Value
Exceptions
- ArgumentException
The list is empty.
- ArgumentOutOfRangeException
Any element is zero or negative.
UseRingbuffersLocalAttention
Store the KV cache of local-attention layers in ringbuffers sized to what those layers
actually need, instead of allocating the full context length — lower memory for long
contexts, at the cost of instant rewinding. null (default) = off. Backend-agnostic in
interface but currently only the Artisan GPU backend implements it; unsupported
models/backends log a native warning and ignore it. Requires native LiteRT-LM v0.15.0+.
public bool? UseRingbuffersLocalAttention { get; init; }
Property Value
- bool?
Remarks
Maps to engine_settings_set_use_ringbuffers_local_attention (the knob the JS
API exposes as use_autosized_ringbuffers).
VisionBackend
Backend for the vision encoder, enabling image input. null (default) leaves vision
unconfigured — image attachments will not work. Requires a multimodal model (e.g. the Gemma 4
E-series); on a text-only model setting this has no effect.
public LiteRtBackend? VisionBackend { get; init; }
Property Value
Remarks
Maps to the vision_backend_str parameter of the C API engine_settings_create.
May differ from Backend (e.g. main on Gpu, vision on
Cpu). Pair with image attachments via
Image(ReadOnlySpan<byte>) and tune the image prefill budget
with VisualTokenBudget.