Instructions to use litert-community/LFM2.5-1.2B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT-LM
How to use litert-community/LFM2.5-1.2B-Instruct with LiteRT-LM:
# LiteRT-LM runs on various platforms (Android, iOS, Windows, Linux, macOS, IoT, Web/WASM) # and supports many APIs (C++, Python, Kotlin, Swift, JavaScript, Flutter). # For platform-specific integration guides, please refer to the official developer website: # https://ai.google.dev/edge/litert-lm # To try LiteRT-LM, the easiest way is to use our CLI tool. # 1. Install the LiteRT-LM CLI tool: pip install -U litert-lm # 2. Download and run this model locally: # See: https://ai.google.dev/edge/litert-lm/cli # A single .litertlm file in the repo is picked automatically; otherwise the CLI asks which one to run # (or pass its name right after the repo id). litert-lm run \ --from-huggingface-repo=litert-community/LFM2.5-1.2B-Instruct \ --prompt="Write me a poem"
- LiteRT
How to use litert-community/LFM2.5-1.2B-Instruct with LiteRT:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
# No code snippets available yet for this library.
# To use this model, check the repository files and the library's documentation.
# Want to help? PRs adding snippets are welcome at:
# https://github.com/huggingface/huggingface.jsLiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).
Measured on device (edge-compat, lfm2.5-1.2b-instruct-int4): Galaxy S26 Β· LiteRT-LM 0.16.0 Β· GPU Β· did not run: engine_create_failed (2026-08-23); Raspberry Pi 5 Β· LiteRT-LM 0.16.1 Β· CPU, 4 threads Β· decode 9.3 tok/s Β· prefill 54 tok/s Β· TTFT 4.84 s (2026-09-01). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/lfm2.5-1.2b-instruct-int4/CARD.md
Measured on device (edge-compat, lfm2.5-1.2b-instruct-int8): Galaxy S26 Β· LiteRT-LM 0.16.0 Β· CPU Β· decode 41.0 tok/s Β· prefill 491 tok/s Β· TTFT 440 ms (2026-09-05); Pixel 8a Β· LiteRT-LM 0.16.0 Β· CPU Β· decode 20.4 tok/s Β· prefill 143 tok/s Β· TTFT 1.48 s (2026-09-05); Raspberry Pi 5 Β· LiteRT-LM 0.16.1 Β· CPU, 4 threads Β· decode 7.0 tok/s Β· prefill 75 tok/s Β· TTFT 3.56 s (2026-09-01). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/lfm2.5-1.2b-instruct-int8/CARD.md
LFM2.5-1.2B-Instruct β LiteRT-LM
LiquidAI/LFM2.5-1.2B-Instruct converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (requires litert-lm β₯ 0.15 / a recent AI Edge Gallery).
Update (2026-08-04): the
.litertlmfiles were updated in place to add theExecutorMetadatasection that litert-lm β₯ 0.15 requires to bind the hybrid conv/attention state buffers (without it, 0.15 fails at inference withmissing some output TensorBuffers). Weights and graph are byte-identical to the original release, and the files ran on litert-lm 0.14 until the 2026-10-13 re-pack, whoselfm2model type needs 0.15.
LFM2.5-1.2B-Instruct is Liquid AI's flagship on-device model: a hybrid architecture mixing gated short-convolution blocks with grouped-query attention. Per the base model card it outperforms Qwen3-1.7B and Gemma 3 1B on knowledge and instruction-following benchmarks at a smaller footprint.
Verified on a Pixel 8a (Android 16, LiteRT-LM 0.16.1, 2026-09-07): the _int4_gpu file, downloaded from this repository and checked against its SHA-256, answered one text turn on the GPU and one on the CPU.
| File | Recipe | Size | GSM8K (n=100) |
|---|---|---|---|
LFM2.5-1.2B-Instruct_int8.litertlm |
int8 dynamic (linears + convs + embedding) | 1.25 GB | 81% (bf16 reference: 79%) |
LFM2.5-1.2B-Instruct_int4.litertlm |
int4 blockwise-32 + OCTAV linears, int8 embedding, convs float | 736 MB | 72% |
LFM2.5-1.2B-Instruct_int4_gpu.litertlm |
same int4 recipe, re-exported so it runs on the GPU | 736 MB | = int4 (A/B in notes) |
2026-09-21: chat template updated to accept the 0.18 content-parts form (string form unchanged); weights, tokenizer and executor metadata byte-identical.
| Context (KV cache) | 4096 max (see --max-num-tokens note) |
| Backend | CPU for every file; the _int4_gpu file also runs on the GPU with litert-lm β₯ 0.16.0 β Android OpenCL, macOS, and iOS Metal, all verified by generation. On iOS set maxNumTokens to 1024 (see the iPhone section β a mismatched value breaks engine creation) |
| Template | bundled β full chat template incl. tool-list support + <think> channel declaration |
| Base model | LiquidAI/LFM2.5-1.2B-Instruct (LFM Open License v1.0) |
Quality β GSM8K
GSM8K, greedy, 0-shot chain-of-thought, max-tokens 1024, n=100, same harness for all rows:
| Configuration | GSM8K |
|---|---|
| PyTorch bf16 (reference) | 79% |
| LiteRT int8 (this repo) | 81% |
| LiteRT int4-b32 OCTAV (this repo) | 72% |
The int8 file is at full parity with the PyTorch reference (+2pt, within noise). Both files pass an 8-question sanity check (7/8, zero degenerate outputs). The int4 file trades β7pt for a 41% smaller download.
Usage
litert-lm run ./LFM2.5-1.2B-Instruct_int8.litertlm --prompt "Explain what a hybrid conv-attention LLM is in two sentences."
The bundle carries the tokenizer and the full chat template (ChatML-style <|im_start|>β¦<|im_end|> with tool-calling support). Generation stops cleanly at <|im_end|>.
--max-num-tokens tip: decode speed drops as the token budget grows (int8 decode: 101 tok/s at 1024 β 77 tok/s at 4096 on an M4 Max) β set it to the smallest value your use case needs (1024 is a good chat default; the file allows up to 4096).
One Engine per conversation. On litert-lm 0.18.0 the runtime starts a new Conversation on an Engine that already ran another conversation from the earlier conversation's short-convolution state: it reuses the prompt tokens the two conversations share and does not reset the state (my open report). In a test that asks three questions (What is the capital of France? / What is 17 times 23? / What color is the sky on a clear day?) first in a pirate-persona conversation and then in a conversation told to answer in one word, on the same Engine against a fresh Engine in a new process (answers capped at 200 tokens; CPU backend, also for _int4_gpu; litert-lm 0.18.0 on a Mac), the _int8 file gave the same three one-word answers on both Engines, in its one run. Those three short questions say nothing about longer conversations. The test could not measure the _int4 and _int4_gpu files: each file's fresh-Engine baseline did not pass the test's own check (all three expected answers present: Paris, 391, blue) in either of its two runs (a test is run a second time only when its first run does not pass), so there was nothing to compare. Create a new Engine for each conversation. The first conversation on an Engine is unaffected. A fix landed on LiteRT-LM main on 2026-11-03 (e1f86b05). No release carries it yet (v0.19.1 does not); this note will be updated when one does.
Tool calls on litert-lm 0.18.0. Since the 2026-10-13 re-pack (metadata only: llm_model_type lfm2, and the chat template now renders tool calls and tool results), the runtime parses the LFM2 tool call and the tool result reaches the model; the earlier files returned the call as message text (<|tool_call_start|>[get_weather(location="Paris")]<|tool_call_end|>). The lfm2 model type needs litert-lm β₯ 0.15.0; the tool-call parser is 0.18.0's β on 0.15β0.17 the call still arrives as text (read from the runtime source, not run here). For _int4_gpu, the GPU file: the 2026-10-13 re-pack changes the model type, and with it the runtime's GPU executor defaults (every non-generic LiteRT-LM model type shares them: allow_src_quantized_fc_conv_ops on, OpenCL hint_waiting_for_completion off, kernel batch size unset); the phone GPU rows on this card were measured with the previous file and have not been re-run β the Mac GPU digit check passed on 0.18.0.
Performance
litert-lm benchmark (litert-lm 0.15.0), CPU backend, Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), warm-up run discarded, otherwise idle machine: decode depends strongly on the KV budget, so both settings are listed:
| Variant | --max-num-tokens |
Prefill (256) | Decode | TTFT |
|---|---|---|---|---|
| int8 | 1024 | 1592 tok/s | 93.1 tok/s | 0.17 s |
| int4 | 1024 | 381 tok/s | 109.4 tok/s | 0.68 s |
| int8 | 4096 | 1121 tok/s | 78.9 tok/s | 0.24 s |
| int4 | 4096 | 343 tok/s | 74.9 tok/s | 0.76 s |
On a Pixel 8a (Tensor G3, CPU backend, measured in AI Edge Gallery): int8 decodes at ~19 tok/s and int4 at ~31 tok/s β on phone-class memory bandwidth the int4 file is ~1.7Γ faster as well as 41% smaller, so prefer int4 on mid-range devices unless you need the last few GSM8K points.
GPU β the _int4_gpu file
Pixel 8a (Tensor G3), litert_lm_main built from the litert-lm v0.16.0 release tag, 263-token prompt, generation capped at 256, 3 runs per backend:
| Backend | Prefill (263 tok) | Decode | TTFT |
|---|---|---|---|
| GPU (OpenCL) | 188β193 tok/s | 21.0β21.2 tok/s | 1.41β1.44 s |
| CPU | 38β54 tok/s | 15.2β24.3 tok/s | 4.9β7.0 s |
On phone-class hardware the GPU's win is prefill and time-to-first-token (3β6Γ both); decode is bandwidth-bound and roughly a wash, so long prompts gain far more than long answers. The CPU rows spread because the phone throttles across a run series; the GPU rows repeat to within 2%.
Apple M4 Max (litert-lm benchmark --cache no, litert-lm 0.16.0, -p 256 -d 256 --runs 3, otherwise idle machine; both backends were verified to generate correct text before these numbers were quoted):
| Backend | Prefill (256) | Decode | TTFT |
|---|---|---|---|
| GPU | 3765 tok/s | 318.3 tok/s | 0.07 s |
| CPU | 337.0 tok/s | 80.5 tok/s | 0.78 s |
On a desktop GPU the gain is across the board β roughly 11Γ prefill and 4Γ decode against the same file on CPU.
The hybrid short-conv blocks are what make a 1.2B decode at ~100 tok/s on CPU β faster than same-size pure-attention models. Long prompts prefill through the large (up to 1024-token) signatures at >1500 tok/s.
GPU β Galaxy S26 (Snapdragon 8 Elite Gen 5)
Galaxy S26 (SM-S942Q / SM8850, Android 16), litert_lm_main from litert-lm 0.16.0, the _int4_gpu file, 19-token prompt, one run per backend:
| Backend | Prefill (19 tok) | Decode | TTFT | Init |
|---|---|---|---|---|
| GPU (OpenCL) | 80.3 tok/s | 24.7 tok/s | 0.28 s | 15.0 s |
| CPU (XNNPACK) | 10.3 tok/s | 24.1 tok/s | 1.89 s | 6.5 s |
The prompt was 19 tokens, so read the prefill column as a floor rather than a throughput figure β at that length the fixed per-call cost dominates. The 263-token Pixel 8a rows above are the throughput measurement. Decode is a wash (24.7 against 24.1), which is what bandwidth-bound decode looks like; the GPU's win here is time-to-first-token, 6.7Γ. The GPU took the whole graph, 542 of 542 ops on LITERT_CL; on CPU, XNNPACK took 469 of 542.
GPU β iPhone 17 Pro (iOS Metal)
The engine-creation failure previously tracked in LiteRT-LM#3129 turned out to be a context-sizing issue on the integration side, not a runtime bug. The Metal delegate compiles its kernels against the KV cache that maxNumTokens pre-allocates; a value that does not match this file's exported plan (our test app derived ~650) makes engine creation fail with a shader-compile error. On iOS Metal, set maxNumTokens to 1024 (Swift EngineConfig(maxNumTokens:), C API litert_lm_engine_settings_set_max_num_tokens).
iPhone 17 Pro, litert-lm v0.16.0 xcframework, maxNumTokens 1024, single run, device charging: decode 67.7 tok/s (the int4 file on CPU on the same phone: 52.6), TTFT 109 ms on a short prompt, prefill 1,982 tok/s at p=128, peak ~470 MB. All Metal kernels compile, generation is coherent, and the file scores the same 8-question sanity check as on CPU (7/8).
Only the _int4_gpu file loads on the GPU. LFM2.5-1.2B-Instruct_int4.litertlm delegates 536 of its 579 ops and then fails to create the engine, because the remaining 43 do not lower (GATHER_ND: Operation is not supported, plus several INT64 tensors). Use _int4_gpu on the GPU backend and _int4 or _int8 on CPU.
Run on Android
Install a recent Google AI Edge Gallery (1.0.16+ imports .litertlm directly from Hugging Face), import this repo (or adb push a file and use local import), select the CPU backend, and chat. GPU acceleration needs the _int4_gpu file and litert-lm β₯ 0.16.0 (litert_lm_main --backend=gpu, or the SDKs); the Gallery bundles its own runtime, which may lag that version β if its GPU toggle fails there, use CPU.
β Gallery's import dialog defaults to CPU-only. Scroll it to Compatible accelerators and tick GPU at import time; a model already imported has to be deleted and re-imported. Nothing in the bundle sets this. Full recipe: Running converted models on GPU (Android).
In a Flutter app, flutter_edge_ai_litertlm (1.14.0) installs this repo's bundle through FlutterEdgeAi.resolveHuggingFace('litert-community/LFM2.5-1.2B-Instruct'), which reads the repo's litertlm_manifest.json and picks the file and backend for the device; in React Native, react-native-litert-lm (0.8.0) does the same through resolveFromManifest(). Neither route is device-verified by this card; each package's README states its tested platforms.
For sample apps, see google-ai-edge/litert-samples.
Conversion notes
Converted with released litert-torch 0.9.1 (which ships the lfm2 hybrid model support) and packaged for litert-lm β₯ 0.14 (β₯ 0.15 since the 2026-10-13 re-pack), with one exporter patch: the stock LFM2 short-conv block saves its conv state from the last columns of the prefill chunk, which are padding whenever the prompt does not exactly fill a prefill signature β corrupting the first generated token of nearly every reply. The patch derives the valid length from the attention mask in-graph and gathers the conv state from the last valid columns instead; with it, engine output is token-identical to an exact per-token reference loop at every prompt length. Multi-length prefill signatures (1β1024) are exported so the runtime can pick tight chunks.
Quantization targets fully-connected + embedding ops (plus convs in the int8 file, quantized at export time); post-hoc int8 of the convs with ai-edge-quantizer breaks generation β quantize convs only at export time.
Why there is a separate _int4_gpu file. The exporter patch above reads the chunk's valid length with index_select and a mask sum, which lower to GATHER_ND and INT64 ops, and GPU delegates reject both β the CPU-only files take 536 of 579 operations on the GPU and the runtime then refuses the partial split. litert-torch 0.9.2 fixed the underlying conv-state bug upstream using valid-token masking and a one-hot matmul state select, which emits neither op. _int4_gpu is that export: litert-torch 0.9.3 + litert-converter 0.3.1, the same int4 recipe and the same weights as the int4 file, plus the executor-metadata section litert-lm β₯ 0.15 needs. It delegates fully on Android OpenCL β 501/501 and 519/519 nodes, zero rejected ops β and runs on the macOS GPU backend. Quality is unchanged: run head to head against the published int4 file on identical prompts, decoding and scoring, both score 38/50 on GSM8K (CPU, n=50), and the GPU file passes an 8-question sanity check on both backends. The CPU-only files are left exactly as they are rather than replaced, so nothing changes for anyone already using them.
Raspberry Pi 5 (CPU)
Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (minβmax in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.
| File | Prefill (tok/s) | Decode (tok/s) | TTFT | Peak RSS |
|---|---|---|---|---|
LFM2.5-1.2B-Instruct_int4.litertlm |
54.1 (53.7β54.2) | 9.3 (9.3β9.3) | 4.8 s | 1.4 GB |
LFM2.5-1.2B-Instruct_int4_gpu.litertlm |
54.6 (53.5β54.6) | 9.3 (9.3β9.3) | 4.8 s | 1.4 GB |
LFM2.5-1.2B-Instruct_int8.litertlm |
75.0 (74.5β75.7) | 7.0 (7.0β7.1) | 3.6 s | 1.9 GB |
Known runtime issues
Long greedy replies on the GPU may vary between runs. On the GPU's default path, where the sampler writes the inputs of the next decode step, a long greedy reply of LFM2.5-2.6B differed from one process to the next on a Mac (WebGPU, LiteRT-LM 0.19.1 from PyPI and built from source) and on a Pixel 11 Pro GPU (PyPI litert-lm-api 0.19.0); with --sampler_handles_input=false (litert_lm_advanced_main built from source; the Python, Kotlin and Swift APIs of 0.19.1 have no such setting) it gave one reply. The files here run the same GPU path; this was not measured on them.
From a conversation preface (the system message, initial messages, tools and extra context a conversation is created with) that has tools and no messages, LiteRT-LM renders these files' chat template with an empty message list, and the template reads messages[0] without checking that the list is non-empty, so the render fails with Failed to apply template: undefined value. This happens in two places: when the preface is prefilled at creation (prefillPrefaceOnInit = true in Kotlin, prefillPrefaceOnInit: true on the web, SetPrefillPrefaceOnInit(true) in C++; the default is false), where creating the conversation fails with that error, and in the preface render (renderPrefaceIntoString() in Kotlin and Swift, RenderPrefaceIntoString() in C++, litert_lm_conversation_render_preface_to_string in the C API). The same happens with extra context (such as enable_thinking) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2). Read in the source of LiteRT-LM 0.17.0, 0.18.1 and main; measured with LiteRT-LM 0.18.1 (the litert-lm package from PyPI) through its C API on these files' template text. The default path, where the first message is rendered together with the preface, is not affected: the Python API and the CLI of 0.18.1 expose neither the flag nor the preface render, and Google AI Edge Gallery sets neither (its source as read on 2026-10-25). A system message or any initial message in the conversation config avoids it.
On the iPhone the Metal GPU can still fail to create the _int4_gpu file's engine, on LiteRT-LM v0.18.0 too. On an iPhone 17 Pro with v0.18.0 on 2026-10-19, a test app's chat launches at maxNumTokens 672 failed at engine creation, 2 of 2: a generated Metal kernel did not compile (implicit conversions between vector types), then Failed to modify graph with delegate. About two minutes later, on the same phone and app build, benchmark() at a context of 2048 ran on the GPU, 4 of 4 (1024 prefill tokens at 2,194β2,280 tok/s, 256 decode tokens at 70.3β74.4 tok/s). Those launches differ in the context and in the API; which of the two decides was not checked, and maxNumTokens 1024 (GPU β iPhone 17 Pro, above) was not run on v0.18.0.
Structured output: LFM2.5-1.2B-Instruct_int4.litertlm does not take a JSON ResponseFormat. It carries llm_model_type lfm2, and LiteRT-LM refuses constrained decoding for that type when the conversation is created (Constrained decoding is not supported for Lfm2DataProcessor.; measured with litert-lm 0.16.0 and 0.19.1 on a Mac CPU). The _int4_gpu and _int8 files were not run with constrained decoding.
License and changes
Distributed under the LFM Open License v1.0 (see LICENSE, inherited from the base model). Note the license's commercial-use limitation for organizations above US$10M annual revenue. Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged unmodified into the .litertlm bundle; exporter conv-state fix as described in Conversion notes. This repository is a community conversion and is not affiliated with Liquid AI.
- Downloads last month
- 5,778
Model tree for litert-community/LFM2.5-1.2B-Instruct
Base model
LiquidAI/LFM2.5-1.2B-Base