ort-server

A generic, self-contained ONNX Runtime model server for Lemonade. It loads one exported ONNX model + a small manifest and serves it over HTTP. Lemonade launches it as a subprocess (like moonshine-server) and forwards requests to it.

Consumer: lemonade #2592 — the router's classifier condition type. Classification is the first capability; the server is intentionally generic so embeddings and reranking can follow.

Status: initial scaffold — not yet built/released. See #2592.

Scope

ort-server serves text-modality ONNX graphs — classification today, embeddings and reranking next. Because it tokenizes from the model's own tokenizer.json, any HuggingFace text classifier runs from a standard ONNX export with no per-model work. Vision and audio models are out of scope by design: those modalities are served by other Lemonade backends (images → stable-diffusion.cpp; audio → whisper / moonshine / kokoro).

Design

The server is thin, and a model is easy to bring: a standard ONNX export, its own tokenizer.json, and a small manifest.json.

• model.onnx is a plain export (input_ids/attention_mask[/token_type_ids] → logits) — the ordinary optimum export, no in-graph baking, no custom ops.
• The server tokenizes at runtime by loading the model's tokenizer.json via mlc-ai/tokenizers-cpp — the exact HuggingFace tokenizer, so parity is guaranteed.
• manifest.json declares the task and how to shape the output.

model-dir/
  model.onnx        # plain export: input_ids/attention_mask -> logits
  tokenizer.json    # the model's HuggingFace tokenizer
  manifest.json

manifest.json:

{
  "task": "text-classification",        // or "token-classification"
  "id2label": {"0": "SAFE", "1": "INJECTION"},
  "score_normalization": "softmax",
  "token_aggregation": null              // "first-subword" | "max" for token-classification
}

HTTP contract

Method	Path	Body	Response
GET	/health	—	200 when the model is loaded and ready
POST	/classify	{"text": "...", "top_k": N?}	{"labels": {"<label>": <score in [0,1]>, ...}}

Future capabilities (same server, new endpoints): POST /embed, POST /rerank.

Execution providers

v1 ships CPU EP only. The roadmap adds providers as build variants without changing the code:

cpu → dml (Windows GPU/NPU) → rocm (AMD GPU) → vitisai (Ryzen NPU).

Build

cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release
./build/ort-server --model-path <model-dir> --port 8100

CMake fetches ONNX Runtime, tokenizers-cpp, cpp-httplib, and nlohmann/json (see CMakeLists.txt). tokenizers-cpp builds a small Rust static lib, so cargo/rustup must be on PATH to build ort-server from source (build-time only — users of the prebuilt binary need nothing).

Releases

CPU-only, so release bundles build on GitHub-hosted runners (windows-latest, ubuntu-latest, macos-latest) — no self-hosted/GPU runners for v1. Pushing a v* tag builds and attaches per-platform archives named to match Lemonade's downloader:

ort-server-<version>-windows-x64.zip
ort-server-<version>-linux-x64.tar.gz
ort-server-<version>-linux-arm64.tar.gz
ort-server-<version>-macos-arm64.tar.gz

License

TODO: adopt the lemonade-sdk project license before first public release.