LLM Router

Part of the worlds-biggest-software-project initiative.

An open, AI-native gateway that intelligently routes LLM queries across providers based on cost, quality, and speed.

LLM Router is a multi-provider gateway for teams running production AI workloads across multiple model providers. It targets platform engineers, AI/ML teams, and CTOs who need to control inference cost and reliability without locking into a single proprietary gateway. The core problem it solves: static routing rules are brittle and proprietary intelligent routers are opaque, while existing open-source gateways stop short of learned, quality-aware routing.

Why LLM Router?

Static rules are brittle: incumbent open-source routers like LiteLLM rely on manually defined cost or latency rules that require ongoing tuning and do not adapt to production quality outcomes.
Intelligent routing is locked behind proprietary SaaS: Martian demonstrates learned per-prompt routing with 20-97% cost reduction, but its Model Mapping technique is closed and may be patent-protected.
Compliance-grade gateways are heavyweight: Portkey ships 50+ guardrails and SOC2/HIPAA/GDPR certifications, but is more than many teams need for simple cost-aware routing.
Hosted gateways carry overhead at scale: OpenRouter's flat 5% fee on top of model cost makes it expensive for teams with large inference bills (>$50K/mo).
Ecosystem gateways create lock-in: Vercel AI Gateway and Cloudflare AI Gateway are tightly coupled to their host platforms and offer rule-based, not learned, routing.

Key Features

Multi-Provider Gateway

Unified OpenAI Chat Completions API across 50+ providers as the default interface
Provider fallback and failover with health monitoring and configurable retry logic
Cost-based routing that selects cheaper models for simple queries and more capable models for complex ones
Rate limiting (TPM/RPM) and spending caps to prevent runaway costs
Budget tracking per provider, request, team, project, and user

Intelligent Routing

Learned routing model trained on production quality and cost outcomes, continuously improving without manual rules
Semantic routing based on query content rather than static thresholds
Quality degradation detection that monitors for silent failures and triggers re-routing or alerts
Automatic fallback strategy optimisation that predicts which provider sequence minimises cost and latency while preserving quality

Caching & Performance

Semantic caching that deduplicates requests based on embedding similarity, not exact match
Circuit breaker with real-time provider health detection and millisecond failover
Latency-optimised streaming route selection for user-facing applications where time-to-first-token matters

Governance & Compliance

PII detection and classification with routing to PII-safe providers or geographic regions for GDPR / CCPA compliance
Hierarchical cost allocation across team, project, feature, and cost centre
Cost attribution for multi-provider chains so retries and fallbacks are correctly accounted for
Audit logs and request/response tracing via OpenTelemetry

Observability

Request/response logs, latency tracking, and error tracking emitted as OpenTelemetry traces
Cost anomaly detection that flags unexpected spend spikes per feature or team
Multi-provider cost forecasting and recommendations to minimise spend

AI-Native Advantage

Unlike rule-based incumbents, LLM Router uses a learned routing model that improves continuously from production outcomes, closing the gap with proprietary systems like Martian without the opacity. AI-driven capabilities include automatic quality degradation detection, prompt rewriting for token reduction (with consent), PII classification for residency-aware routing, and cost anomaly detection. These are areas where existing open-source gateways rely on static configuration and where managed alternatives are either closed-source or bundled with heavyweight compliance stacks.

Tech Stack & Deployment

LLM Router exposes the OpenAI Chat Completions API as its standard interface so existing client SDKs work unchanged. Deployment targets self-hosted Kubernetes or VM environments, with optional Redis for distributed rate limiting and cooldown state. Observability is emitted via OpenTelemetry for downstream consumption by Prometheus, Grafana, Datadog, Honeycomb, or Jaeger. Provider integrations cover OpenAI, Anthropic, Azure, Bedrock, Vertex AI, Cohere, Mistral, Groq, Together, DeepSeek, and others.

Market Context

LLM routers and AI gateways sit inside the broader AI infrastructure market, projected to exceed USD 200 billion by 2030. Incumbent pricing spans a wide range: open-source self-hosted (LiteLLM, free), managed gateways (Portkey from $49/mo), usage-based hosted endpoints (OpenRouter at model cost + 5%), and enterprise contracts (Kong, six figures). Primary buyers are platform engineers responsible for AI cost control, AI/ML teams running multi-model experiments, CTOs at AI-native companies with inference bills above $50K/mo, and enterprise architects standardising on a single gateway for governance.

Project Status

This project is in the research and specification phase.
Contributions, feedback, and domain expertise are welcome.

Contributing

We welcome contributions from developers, domain experts, and potential users. See CONTRIBUTING.md for guidelines.

Important: All contributions must be your own original work or clearly attributed open-source material with a compatible licence. Copyright infringement and licence violations will not be tolerated and will result in immediate removal of the offending contribution. If you are unsure whether a piece of code, text, or other material is safe to contribute, open an issue and ask before submitting.

Note: Martian's "Model Mapping" interpretability technique may be patent-pending or patent-protected. Contributions reproducing this specific approach must be preceded by independent patent research.

Licence

Licence to be determined. See discussion for context.