Table of Contents

Class LiteRtEngineOptions

Namespace
LiteRtLmSharp
Assembly
LiteRtLmSharp.dll

Options for creating a LiteRtEngine.

public sealed record LiteRtEngineOptions : IEquatable<LiteRtEngineOptions>
Inheritance
LiteRtEngineOptions
Implements
Inherited Members

Properties

ActivationDataType

Activation tensor precision. Defaults to Float32. Only the GPU backend honors this, and only as F32 vs F16; on CPU it is a no-op.

public LiteRtActivationDataType? ActivationDataType { get; init; }

Property Value

LiteRtActivationDataType?

Remarks

The runtime's own GPU default for the text executor is F16, which corrupts structured output (digits, dates, JSON) on the GPUs we measured, with no speed gain on a desktop GPU: gemma-4-E2B on an RTX 3080 decodes 113 tok/s with F16 and 117 with F32, and F32 commits about 0.3 to 0.6 GB more. So the binding asks for F32. Set Float16 to trade precision for activation memory (for example on a mobile GPU, after checking your outputs), or null to let the runtime choose: the precision the model file declares for its text model (F16 for the Gemma 4 E-series), else F16 on GPU. The setting reaches the text executor only: the vision and audio encoders run at the precision the model file declares for them (F16 for the Gemma 4 E-series vision encoder), else F32. Int16 and Int8 are accepted by the native API but not distinctly implemented by the shipped executors (folded to F16 on GPU). Maps to engine_settings_set_activation_data_type. See docs/engine-tuning.md.

AudioBackend

Backend for the audio encoder, enabling audio input. null (default) leaves audio unconfigured — audio attachments will not work. Requires a model with audio support (e.g. the Gemma 4 E-series); a no-op on models without it.

public LiteRtBackend? AudioBackend { get; init; }

Property Value

LiteRtBackend?

Remarks

A model may constrain which backend its audio encoder accepts. The Gemma 4 audio sub-model requires CPU: passing Gpu makes litert_lm_engine_create fail with "Audio backend constraint mismatch. Model requires one of [cpu]" even when Backend is GPU — on any platform, not a win-x64 quirk. Use Cpu for such models; the vision encoder has no such constraint and runs on GPU. Maps to the audio_backend_str parameter of the C API engine_settings_create.

AudioLoraRank

LoRA rank for the audio executor. null (default) leaves the engine default. Only applies when an audio executor is configured (AudioBackend set). Maps to engine_settings_set_audio_lora_rank. Unlike the text path, audio LoRA has not been validated against a real adapter in this binding.

public int? AudioLoraRank { get; init; }

Property Value

int?

Exceptions

ArgumentOutOfRangeException

The value is zero or negative.

AudioNumThreads

Number of CPU threads for the audio executor. null (default) leaves the engine default. Only applies when an audio executor is configured (AudioBackend set), and only reaches the audio backend's CPU compilation options — a no-op otherwise (verified in engine.cc set_audio_num_threads → audio_executor CPU options). Maps to engine_settings_set_audio_num_threads.

public int? AudioNumThreads { get; init; }

Property Value

int?

Exceptions

ArgumentOutOfRangeException

The value is zero or negative.

Backend

Backend to run the model on. Defaults to Cpu. Use Gpu, Npu, or Custom(string) for a backend exposed by your own native build.

public LiteRtBackend Backend { get; init; }

Property Value

LiteRtBackend

BenchmarkDecodeTokens

Synthetic decode token count for benchmarking — the decode half of BenchmarkPrefillTokens. 0 (default) = off. See BenchmarkPrefillTokens for the full behavior. Maps to engine_settings_set_num_decode_tokens.

public int BenchmarkDecodeTokens { get; init; }

Property Value

int

BenchmarkPrefillTokens

Synthetic prefill token count for benchmarking. 0 (default) = off (normal inference).

public int BenchmarkPrefillTokens { get; init; }

Property Value

int

Remarks

When > 0 the engine runs a synthetic benchmark instead of answering: the real prompt is truncated or padded to exactly this many tokens for prefill, and decoding runs exactly BenchmarkDecodeTokens tokens (ignoring the stop token). So GetBenchmarkInfo() reports prefill/decode throughput at FIXED token counts — independent of the prompt — which is useful for device throughput benchmarking and for measuring the effect of the tuning settings reproducibly. The reply text is not a real answer. Setting either this or BenchmarkDecodeTokens also turns benchmark mode on (the same switch as EnableBenchmark). Use a dedicated engine instance — do not reuse it for real chat. Verified observable through the Conversation API on win-x64 CPU (the default engine reads these during prefill/decode). Maps to engine_settings_set_num_prefill_tokens.

Cache

Where the engine keeps its compiled-artifact cache (GPU shaders / converted weights), which speeds up subsequent loads. Defaults to Default (written next to the model file). Use Disabled, InMemory, or Directory(string) for an explicit path.

public LiteRtCache Cache { get; init; }

Property Value

LiteRtCache

Remarks

The cache files are named after the model file (its name, modification time and size), not after the native runtime version. A newer runtime detects a GPU cache written by an older one and rebuilds it on the first load, which makes that load slower once. A Directory(string) must exist: the runtime does not create it, and since native v0.18.0 Load(LiteRtEngineOptions) fails with InvalidArgument when it is missing.

EnableBenchmark

Enable benchmark instrumentation so GetBenchmarkInfo() returns prefill/decode tokens-per-second, time-to-first-token and init time. Fixed at engine creation; the overhead is timing bookkeeping only.

public bool EnableBenchmark { get; init; }

Property Value

bool

EnableMetalResidencySet

Whether the GPU backend keeps the model weights and allocations resident in GPU memory through Apple's MTLResidencySet API, which prevents them from being paged out and lowers allocation overhead. null (default) leaves the engine default. Apple platforms with the Metal GPU backend only; other platforms and backends ignore it.

public bool? EnableMetalResidencySet { get; init; }

Property Value

bool?

Remarks

Maps to engine_settings_set_gpu_enable_metal_residency_set (native LiteRT-LM v0.18.0+). The desktop macOS GPU backend runs on WebGPU over Metal; measure the effect on the target device rather than assume one.

EnableSpeculativeDecoding

Enable speculative decoding: the model drafts several tokens ahead with a small Multi-Token-Prediction (MTP) drafter and the main model verifies them in one step. It pays off only where verification is cheap relative to the drafter (memory-bound accelerators, high acceptance); measure it before turning it on.

public bool EnableSpeculativeDecoding { get; init; }

Property Value

bool

Remarks

Requires a .litertlm that ships an MTP drafter (e.g. the Gemma 4 E2B/E4B/12B builds; SupportsSpeculativeDecoding tells before loading). On a model without one, engine creation fails with LiteRtException (Ministral 3 reports NOT_FOUND "tf_lite_mtp_drafter not found in the model"). The setting is fixed at engine creation; EnableSpeculativeDecoding overrides it per conversation (native v0.18.0+). Pair with EnableBenchmark to measure the effect (see GetBenchmarkInfo()).

Measured with gemma-4-E2B on LiteRT-LM v0.18.0 (see docs/speculative-decoding.md): it slows decoding down on desktop CPU (0.78×) and on the desktop WebGPU GPU backend (0.52× on an RTX 3080 with the default float32 activations), with the default disk cache in both cases. On an Adreno 650 phone GPU it made no difference (measured in June 2026 on LiteRT-LM v0.13.1).

EnableYnnpack

Lets the experimental YNNPACK delegate take the CPU operations it supports before XNNPACK. null (default) = engine default (off). CPU backend, linux-arm64 only. Requires native LiteRT-LM v0.16.0+.

public bool? EnableYnnpack { get; init; }

Property Value

bool?

Remarks

Maps to engine_settings_set_enable_ynnpack. Of the official libraries, only linux-arm64 carries the YNNPACK kernels. Elsewhere, true makes Load(LiteRtEngineOptions) fail with Unimplemented (native v0.18.0; v0.16.0 ignored the flag). Measure any speed-up on the target device rather than assume one.

GpuDecodeStepsPerSync

Decode steps the GPU runs between host syncs. null (default) = engine default. Only honored by the Artisan GPU backend (the mobile GPU stack); the desktop WebGPU backend ignores it. Requires native LiteRT-LM v0.15.0+.

public int? GpuDecodeStepsPerSync { get; init; }

Property Value

int?

Remarks

Maps to engine_settings_set_gpu_decode_steps_per_sync. Larger values reduce sync overhead but coarsen streaming/cancellation granularity.

Exceptions

ArgumentOutOfRangeException

The value is zero or negative.

GpuWaitForWeightUploads

Whether engine load blocks until GPU weight uploads complete. null (default) = engine default. Only honored by the Artisan GPU backend; the desktop WebGPU backend ignores it. Requires native LiteRT-LM v0.15.0+.

public bool? GpuWaitForWeightUploads { get; init; }

Property Value

bool?

Remarks

Maps to engine_settings_set_gpu_wait_for_weight_uploads.

LoraRank

LoRA rank for the text executor. null (default) leaves the engine default (LoRA disabled). Requires a LoRA-enabled model and a matching adapter passed per conversation via LoraPath. Maps to engine_settings_set_lora_rank.

public int? LoraRank { get; init; }

Property Value

int?

Remarks

The adapter applies to the whole engine, not to one conversation (LiteRT-LM v0.18.0, measured on CPU with upstream's LoRA test bundle): after a conversation with LoraPath has generated, conversations without an adapter on the same engine generate with it too, and while a conversation with an adapter exists, a send on one without it fails with Internal. Use one adapter per engine, and reload the engine to switch adapters or to return to the base model.

The published Gemma 4 bundles carry no LoRA slots (google-ai-edge/LiteRT-LM#3173): an adapter is accepted and has no effect on them. The GPU backend is not validated (the test bundle does not load on it).

Exceptions

ArgumentOutOfRangeException

The value is zero or negative.

MaxNumImages

Maximum number of images the engine accepts per turn. 0 = engine default. On the standard binaries this setting has no effect — see the remarks for the case where it applies. To bound how much of the context window images consume, use VisualTokenBudget instead.

public int MaxNumImages { get; init; }

Property Value

int

Remarks

The native library compiles several engine implementations into one binary and picks one per backend. This value always reaches the engine settings, but only the legacy TFLite implementation reads it; the standard binaries select the modern CompiledModel engines for CPU/GPU, which ignore it. It matters only when a custom native build routes your backend (e.g. one targeted via Custom(string)) through the legacy engine. Kept for parity with the reference Kotlin binding's maxNumImages. Maps to engine_settings_set_max_num_images.

MaxNumTokens

The total context window in tokens: prompt + generated replies, accumulated across every turn of a conversation. 0 = engine default.

public int MaxNumTokens { get; init; }

Property Value

int

Remarks

This is the hard ceiling for a conversation's KV cache. Size it for the longest exchange you expect — the system prompt, any restored History, and each user turn and reply all count against it and accumulate. As the running total (TokenCount) nears the limit, generation degrades, so manage the conversation before then: trim history, cap replies with MaxOutputTokens, or start a fresh conversation. A larger window costs more memory and a slower prefill, so prefer the smallest that fits your use case. For multimodal input leave room for the media on top of the text — an image expands to roughly 256 vision tokens — for which 4096 is a comfortable starting point.

Setting this also arms the binding's KV overflow guard. The native runtime does not police the limit: a send that grows past it writes beyond the allocated cache and corrupts native memory (typically a deferred 0xC0000005 process crash on a later call). With an explicit value here, the send methods instead clamp each reply to the remaining context and throw LiteRtContextOverflowException when the conversation is full. The clamp cannot cover sends whose prefill cost is unmeasurable managed-side — media attachments and SendRaw's per-send extraContext get only the conversation-full check, so keep extra headroom under the limit on those paths (an image is roughly 256 tokens). With 0 the effective limit is internal to the engine (the C API exposes no getter), so the guard stays off.

Keep the value at or below the model's built-in context length. The native loader sizes the KV cache from this hint only when it is smaller than the model's own maximum; a value at or above it silently falls back to the model's default sizing, leaving the real cache SMALLER than this property claims — and the overflow guard, calibrated against this value, would then stop policing before the actual edge. Values below the model's minimum prefill work group (~128) are unusable outright: every send is rejected.

Keep the value at or above the model's largest prefill signature (1024 for the published gemma conversions). The native loader accepts a smaller limit, but a send whose prefill spans more than the smallest work group then fails inside the native graph (v0.18.0 reports "Failed to invoke the compiled model Failed to allocate tensors") instead of cleanly. The C API exposes no way to query the signatures, so the binding cannot validate this up front; when a send fails that way on an engine loaded with a limit below 1024, the exception names this as the likely cause.

MaxVisionTokensPerImage

Upper bound on the vision tokens one image may expand to, which limits the vision encoder signatures the engine loads (up to the cap, rounded up to the next signature). null (default) leaves every signature of the model available.

public int? MaxVisionTokensPerImage { get; init; }

Property Value

int?

Remarks

It caps the image size; it does not choose it. Images still expand to the model's default size (about 256 tokens for the Gemma 4 E-series) unless a visual token budget asks for less, so pair a cap below that default with VisualTokenBudget (or VisualTokenBudget) at or below the cap. Without one, an image send fails with InvalidArgument ("No signature found…"), and an image send whose budget exceeds the cap fails the same way ("Visual token budget (…) cannot be larger than the engine's max vision tokens per image (…)"). Left null, the engine checks the budget against the model's own maximum instead (MaxVisionTokenBudget, 280 on the Gemma 4 E-series). It applies only when VisionBackend is set and the model declares a per-image token count. A cap above the model's largest signature (280 on the Gemma 4 E-series) fails Load(LiteRtEngineOptions) with InvalidArgument. Maps to engine_settings_set_max_vision_tokens_per_image (native LiteRT-LM v0.18.0+).

Exceptions

ArgumentOutOfRangeException

The value is zero or negative.

ModelPath

Path to the .litertlm (or .task) model file. Must be set — Load(LiteRtEngineOptions) throws ArgumentException when it is empty. (Deliberately not required: future native versions add alternate model sources — e.g. loading from a file descriptor — which will land here as sibling properties.)

public string ModelPath { get; init; }

Property Value

string

NumThreads

Number of CPU threads for the text executor. null (default) leaves the engine default. CPU-backend only: the native setter reads the executor's CpuConfig and is a no-op when the text backend is not CPU (verified in engine.cc set_num_threads). Maps to engine_settings_set_num_threads.

public int? NumThreads { get; init; }

Property Value

int?

Exceptions

ArgumentOutOfRangeException

The value is zero or negative.

ParallelFileSectionLoading

Whether the engine loads the .litertlm file's sections in parallel during startup. null (default) leaves the engine default (on). When on, the tokenizer section is parsed on a background thread while the model is built, shortening cold-start init; set false to load it serially on the calling thread (single-threaded environments, or to avoid the brief concurrent init peak, at the cost of a slower start). Maps to engine_settings_set_parallel_file_section_loading. See docs/engine-tuning.md.

public bool? ParallelFileSectionLoading { get; init; }

Property Value

bool?

PrefillChunkSize

Maximum prompt tokens prefilled per step. 0 (default) = no chunking (the whole prompt is prefilled at once). CPU + dynamic models only — ignored on GPU and on static models. A smaller chunk lowers peak memory during prefill and allows more timely cancellation of a long prompt, at the cost of more prefill iterations (potentially slower). Maps to engine_settings_set_prefill_chunk_size. See docs/engine-tuning.md.

public int PrefillChunkSize { get; init; }

Property Value

int

SupportedAudioLoraRanks

The set of LoRA ranks the audio executor should support. null (default) leaves the engine default. Requires an audio executor (AudioBackend set); see SupportedLoraRanks for the GPU-backend caveat. Every element must be positive and the list non-empty. Maps to engine_settings_set_supported_audio_lora_ranks.

public IReadOnlyList<int>? SupportedAudioLoraRanks { get; init; }

Property Value

IReadOnlyList<int>

Exceptions

ArgumentException

The list is empty.

ArgumentOutOfRangeException

Any element is zero or negative.

SupportedLoraRanks

The set of LoRA ranks the text executor should support (for switching adapters of different ranks). null (default) leaves the engine default. Only honored on the GPU (Artisan) backend: on other backends the native layer accepts and silently ignores it (verified in llm_executor_settings.h SetSupportedLoraRanks). Every element must be positive and the list non-empty. Maps to engine_settings_set_supported_lora_ranks.

public IReadOnlyList<int>? SupportedLoraRanks { get; init; }

Property Value

IReadOnlyList<int>

Exceptions

ArgumentException

The list is empty.

ArgumentOutOfRangeException

Any element is zero or negative.

UseRingbuffersLocalAttention

Store the KV cache of local-attention layers in ringbuffers sized to what those layers actually need, instead of allocating the full context length — lower memory for long contexts, at the cost of instant rewinding. null (default) = off. Backend-agnostic in interface but currently only the Artisan GPU backend implements it; unsupported models/backends log a native warning and ignore it. Requires native LiteRT-LM v0.15.0+.

public bool? UseRingbuffersLocalAttention { get; init; }

Property Value

bool?

Remarks

Maps to engine_settings_set_use_ringbuffers_local_attention (the knob the JS API exposes as use_autosized_ringbuffers).

VisionBackend

Backend for the vision encoder, enabling image input. null (default) leaves vision unconfigured — image attachments will not work. Requires a multimodal model (e.g. the Gemma 4 E-series); on a text-only model setting this has no effect.

public LiteRtBackend? VisionBackend { get; init; }

Property Value

LiteRtBackend?

Remarks

Maps to the vision_backend_str parameter of the C API engine_settings_create. May differ from Backend (e.g. main on Gpu, vision on Cpu). Pair with image attachments via Image(ReadOnlySpan<byte>) and tune the image prefill budget with VisualTokenBudget.