Class LiteRtEmbeddingEngineOptions
- Namespace
- LiteRtLmSharp
- Assembly
- LiteRtLmSharp.dll
Options for loading a LiteRtEmbeddingEngine.
public sealed record LiteRtEmbeddingEngineOptions : IEquatable<LiteRtEmbeddingEngineOptions>
- Inheritance
-
LiteRtEmbeddingEngineOptions
- Implements
- Inherited Members
Properties
ActivationDataType
Gets the activation precision. Defaults to Float32, like
ActivationDataType: EmbeddingGemma's activations exceed the float16
range, so its model card advises against float16, which is the runtime's fallback on the GPU backend.
The CPU backend runs float32 either way. Float16 measured no faster on a desktop GPU but about twice
as fast per sentence on a phone GPU (Adreno 650), with vectors slightly further from the CPU ones. Set
Float16 to make that trade, or null to let the runtime
choose (the model's preferred type, else float16 on GPU).
public LiteRtActivationDataType? ActivationDataType { get; init; }
Property Value
Backend
Gets the backend to run the model on. Defaults to Cpu.
public LiteRtBackend Backend { get; init; }
Property Value
Remarks
On the GPU backend the vectors differ slightly from the CPU ones (cosine similarity above 0.99 between the two in our measurement), so index and query on the same backend.
Cache
Gets where the engine keeps its compiled-artifact cache. Defaults to Default (next to the model file). A Directory(string) must already exist: the runtime does not create it, so Load(LiteRtEmbeddingEngineOptions) throws ArgumentException.
public LiteRtCache Cache { get; init; }
Property Value
MaxInputLength
Gets the longest input, in tokens, the engine prepares for, or null (default) for the limit the
model declares (EmbeddingGemma 2: 1024). The engine loads the smallest of the model's input
signatures that holds this many tokens, plus the shorter ones (see
EmbeddingInputLengths; EmbeddingGemma 2 has 128 to 8192), so texts up to
that signature's length are accepted (1500 loads the 2,048 signature). A longer text fails unless
OverflowStrategy truncates or chunks it. A value above the
longest signature makes Load(LiteRtEmbeddingEngineOptions) fail. Longer signatures need more
memory.
public int? MaxInputLength { get; init; }
Property Value
- int?
Exceptions
- ArgumentOutOfRangeException
The value is zero or negative.
MinInputLength
Gets the shortest input signature, in tokens, the engine loads, or null (default) for
the model's own minimum. Signatures shorter than this are not loaded, which saves memory but pads
short texts to a longer signature. It must not exceed the effective maximum:
MaxInputLength when set (checked by Load(LiteRtEmbeddingEngineOptions)), else the
limit the model declares (1024 for EmbeddingGemma 2; the runtime rejects a larger minimum).
public int? MinInputLength { get; init; }
Property Value
- int?
Exceptions
- ArgumentOutOfRangeException
The value is negative.
ModelPath
Gets the path to the embedding model's .litertlm file. Must be set.
public string ModelPath { get; init; }
Property Value
NumThreads
Gets the number of CPU threads, or null (default) for the engine default. CPU backend only.
public int? NumThreads { get; init; }
Property Value
- int?
Exceptions
- ArgumentOutOfRangeException
The value is zero or negative.