Latency-based LLM routing

sturnus is a sidecar that speaks the OpenAI API and sends each request to whichever of your providers is fastest and healthy right now. One static binary. No Redis, no database, no control plane.

Install Source on GitHub

RustMIT

Requests flock to the fastest healthy provider. Slower ones keep a small share as a probe, so a recovered provider wins traffic back on its own.

Drop-in: works with the code you already have

- client = OpenAI(base_url="https://api.openai.com/v1", api_key="sk-...")
+ client = OpenAI(base_url="http://127.0.0.1:4000/v1", api_key="unused")

response = client.chat.completions.create(
    model="fast",  # alias, resolved to the fastest candidate
    messages=[{"role": "user", "content": "Hello"}],
)

The model name alias (fast in the example above) maps to a list of interchangeable provider and model pairs in a small TOML file. Traffic shifts between them as their latency changes.

How routing works

Measure. Every response feeds two exponentially weighted moving averages per candidate: time to first chunk, and success rate.
Score. Effective latency is latency divided by success rate. Slow or flaky providers score worse.
Weight. Each candidate gets a share proportional to (best / its_score)^k. The best takes most of the traffic. Worse ones keep a shrinking share, floored at 1%.
Probe by default. That floor ensures every provider gets a live measure. When a slow or erroring provider recovers, it wins traffic back automatically.

Each sidecar routes from what it observes locally, so there is no shared state to run and nothing to keep in sync across pods.

Features

Transparent passthrough: Only the model field is rewritten. The rest of the body is forwarded byte for byte, keeping key order, number precision and formatting. Streaming responses are relayed as chunks arrive.
Session affinity without state: Every response carries an x-session-affinity header. Send it back to pin a multi-turn conversation to the same provider, across any pod. The pin breaks itself if that provider starts failing.
Memory bounded: Request buffers are capped per request and in aggregate. Bursts beyond the budget are shed with 429 and Retry-After instead of an OOM kill.
Providers: OpenAI, Anthropic, Groq, Azure OpenAI, Google AI Studio, any OpenAI-compatible server, and Vertex AI with GKE Workload Identity and automatic token refresh.
Observability: Prometheus metrics for requests, time to first chunk, latency and errors, labelled by alias, provider and model. Structured JSON logs with W3C traceparent propagation, and a /status endpoint showing the live scores.
Small and auditable: A single static binary published as a scratch container for amd64 and arm64. MIT licensed. Runs entirely inside your infrastructure.

What it doesn't do

Retry or fail over a request: sturnus is transparent. Errors are returned verbatim and only feed the routing signal, so your client SDK's own retry logic decides what to do next.
Route on cost or quality: It routes on latency alone, so every model under an alias should be one suited to your use case. Spend tracking, prompt management and an admin UI are out of scope. sturnus only routes.

Install

Run it with Docker, or install it with cargo. Either way it needs a config.toml. Start from config.example.toml and add your providers.

Docker

docker run -v ./config.toml:/config.toml \
  -p 4000:4000 \
  ghcr.io/sturnus-dev/sturnus:latest

Cargo

cargo install sturnus
sturnus --config config.toml

Binary

Static builds for Linux and macOS, x86_64 and aarch64, are attached to every release.

A minimal config

listen = "127.0.0.1:4000"

[provider.openai]
base_url = "https://api.openai.com/v1"
api_key = "${OPENAI_API_KEY}"

[provider.vertex]
vertex_ai = { project_id = "my-gcp-project", location = "us-central1" }

[model]
fast = [
  { provider = "openai", model = "gpt-4o-mini" },
  { provider = "vertex", model = "google/gemini-2.5-flash" },
]

On Kubernetes, run it as a native sidecar bound to loopback. The full configuration reference, endpoint list and metrics table are in the README.