[Ooooze]

Why

OpenSpec change add-litert-lm-runtime, core part: run Google's LiteRT-LM (.litertlm files such as Gemma 4 E2B/E4B) as a local engine next to llama.cpp, on CPU, GPU and, on Windows, the Intel NPU through OpenVINO.

What changes

• Engine builds. New kind litert-lm: Google's PyPI wheels, listed with sha256 in conf's manifest (ATOMIC_LITERT_LM_MANIFEST_URL). Each release carries an abi number; core installs only the ABIs it binds (SUPPORTED_LITERT_LM_ABI = [1]). A .whl unpacks as a zip, and only the native part of litert_lm/ is kept under <build>/lib/. The host probes a build before it is switched in. ADR 2026-10-21-litert-lm-builds-are-pypi-wheels-with-an-abi-number.

• Host mode litert-host probe|inspect|serve of the core binary, on bun:ffi. ffi-bun.ts is the second file allowed to import bun:*. Chunks are copied on the library's thread by a small C shim, compiled with Bun's TinyCC, into a mutex-guarded FIFO that JS drains every 2 ms. A threadsafe JSCallback read freed memory and crashed in about 1 run in 5. The host runs its own OpenAI server:

  • streaming, tool calls, JSON schema (LL_GUIDANCE), thinking, stop;
  • generations run one at a time (FIFO);
  • a pool of four warm conversations;
  • a client that hangs up cancels its generation.

  ADR 2026-10-21-litert-lm-runs-in-a-host-mode-with-its-own-openai-server.

• Runtime. LiteRtLmRuntime plus the following:

  • the litert_lm block of model.yml and the litert-lm settings schema;
  • four new error codes;
  • GET /runtimes/litert-lm/accelerators and POST …/prepare, which download OpenVINO for a supported Intel NPU;
  • POST /models/litert-lm/check, the verdict for a variant or a file;
  • client methods, routing and listing, GPU residency.

• Fixes from running it on real hardware:

  • cancel on disconnect under Bun: node:http gives no close once the body is read, so the host now uses core's requestAborted;
  • relative --lib on Windows: SetDllDirectoryW drops the working folder from the DLL search;
  • temperature below 0.01 is sent as greedy: at 0 the GPU sampler answered gibberish;
  • EcoQoS on Windows: the host opts out, since Windows kept the windowless process on efficiency cores. CPU decode went from 6 to 28 tok/s, the RTX 4070 from 68 to 100;
  • JSON schema with speculative decoding (core's default): a constrained conversation now runs without speculative decoding. Its drafts skipped the grammar.

Checked

• npm run verify steps after merging main:

  • lint, typecheck, format, build, build:bin, test:contract:bun — pass;
  • test:e2e — 217 passed, 10 skipped;
  • unit and contract — 9623 passed, 1 failed: src/diffusion/server-process.test.ts › kills a server that does not finish loading in time. It failed in both full runs after the merge and in one before it, and passes on its own (3/3 each time). The fake engine has 1.5 s to record its pid, which a loaded machine misses. This PR does not touch diffusion;
  • because of that failure, the per-file coverage floors were not evaluated after the merge. Before it they failed for 8 files that fail the same way on main, none of them a LiteRT-LM file.

• Integration on Google's real library (npm run test:integration), with the wheel installed by core from PyPI and its sha256 checked:

  Platform	Result
  macOS arm64	5/5
  Linux x64 (Ubuntu)	5/5
  Windows x64	6/6 (includes the new JSON-schema-with-speculative test)

• Hardware runs. Checked on each: correctness against CPU, streaming, tool call, JSON schema, conversation isolation, and cancel on disconnect.

  Device	Path	Gemma 4 E4B decode
  macOS Metal	—	—
  Linux NVIDIA	OpenCL	—
  Windows RTX 4070	D3D12	50 tok/s
  Windows Intel iGPU	D3D12	11 tok/s
  Windows CPU	—	14 tok/s

  Same machine, llama.cpp b11463 on E4B: CUDA 65, Vulkan on the iGPU 7.5, CPU 16. Prefill on CUDA is 7× faster than LiteRT-LM.

• Network. No sockets other than loopback, and no DNS lookups, from probe, inspect or serve on any OS.

Not checked or not done:

• Intel NPU: there is no Core Ultra 200V or series 3 machine; this moves to acceptance.
• GPU on Linux without NVIDIA: the engine fails with VK_ERROR_DEVICE_LOST on the Intel iGPU, so conf keeps GPU closed for linux-x64.
• usage.completion_tokens: it counts stream chunks, so it is under-counted with speculative decoding. Left as is by decision.
• Release 6.9: not part of this PR.

🤖 Generated with Claude Code