A local HTTP router for two inference tiers: a fast default and a larger model for requests that need it. It provides an OpenAI-compatible endpoint, configurable routing, circuit breakers, and fallback between backends.
I built it to make trailbrake and iliria usable through one endpoint, while keeping each engine independently deployable. The router has no runtime Python dependencies.
Status: early development, version 0.1.0. The router's tests use scripted backends. Model integration and hardware benchmarks have their own recorded results.
Requires Python 3.11+:
git clone https://github.com/PhilipJohnBasile/racecontrol.git
cd racecontrol
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev]'
python -m examples.demoThe demo starts the real router and two scripted backends on temporary loopback ports. It sends a normal request, makes the fast backend fail, and verifies that the second request reaches the deep backend. It then shuts down all three servers.
Fast backend answered.
Deep backend recovered the request.
Verified: default routing, backend failure, and cross-tier fallback.
cp config/router.example.toml config/router.toml
# Set each backend's URL and model ID in config/router.toml.
racecontrol check-config --config config/router.toml
racecontrol serve --config config/router.tomlThe example configuration listens on 127.0.0.1:8100. Use explicit #deep, #reason, or reasoning_effort=high when requesting the deep tier; automatic routing quality depends on the configured policy and workload.
flowchart LR
A[Client request] --> B[Routing policy]
B --> C[Fast backend]
B --> D[Deep backend]
C --> E[Response and decision log]
D --> E
C -. failure fallback .-> D
The router communicates with backends over HTTP. That keeps runtime dependencies small and makes request translation, timeouts, streaming, rollback, and failure recovery testable without loading a model.
python -m pytestArchitecture and tradeoffs · Detailed setup · Verification notes · Release history
MIT.