A high-performance, open-source LiteLLM-proxy alternative in Rust —
an OpenAI/Anthropic-compatible AI gateway and load balancer.

rolter proxies commercial providers and load-balances self-hosted OpenAI-compatible fleets (e.g. 20–30 vLLM instances) with cache-aware routing, full RBAC, reload-free configuration, and cost/usage tracking.

Status: active development. The gateway, Postgres-backed control plane, reload-free configuration, cost controls, reliability primitives, and core provider surfaces are implemented; remaining work is tracked in ROADMAP.md, TODO.md, and GitHub issues.

Why rolter

• Fast — a Rust data plane (Axum/Hyper/Tower on Tokio) with lock-free config reads and minimal-copy streaming.
• Cache-aware load balancing — route prefix-heavy traffic to the vLLM replica most likely to have the KV cache warm.
• Drop-in — speak the OpenAI and Anthropic APIs your clients already use.
• Operable — virtual keys, budgets, rate limits, cost tracking, RBAC, and…