Your AI stack works fine with one model. The moment you add a second provider, a fallback, a rate limit, or a cost ceiling, everything breaks. Routing logic bleeds into application code, API keys get hardcoded, teams duplicate retry logic across services, and suddenly your “enterprise AI” is just a pile of half-working integrations held together by environment variables. That’s exactly the problem open-source LLM gateways solve, and in 2026, the options have matured enough to pick one seriously.
What an LLM Gateway Actually Does (and Why You Need One)
An LLM gateway sits between your application and every model provider OpenAI, Anthropic, Azure, Cohere, Mistral, local models, whatever. It normalizes the API surface, handles routing, enforces rate limits, logs requests, manages keys, and gives you failover without touching application code.
Without one, you’re solving the same problems in every service: retry logic, model fallback, token tracking, cost controls. With one, you solve it once at the infrastructure layer.
The four capabilities that matter at enterprise scale:
- Unified API surface — one endpoint regardless of which model is behind it
- Load balancing and fallback — route to backup providers when primary fails or rate-limits
- Observability — token usage, latency, cost, and errors in one place
- Access control — centralized API key management, team-level quotas, audit logs
Open-Source LLM Gateways Worth Evaluating in 2026
1. Helicone
Helicone is the one that shows up when teams need serious observability baked into the gateway layer, not wired in afterward through a separate logging pipeline. It’s built as a proxy you can drop in front of any OpenAI-compatible endpoint, and monitoring starts immediately, with no instrumentation changes in the application code.
What separates Helicone from most gateways is the depth of the request-level data it captures by default: token counts, cost per request, latency, user-level attribution, and custom properties you tag at call time. That makes it practical for teams that need to answer “which feature is driving our AI spend” rather than “how many tokens did we use this month.”
Key capabilities:
- One-line proxy setup changes the base URL; observability starts immediately.
- Request logging with cost tracking, latency, and custom metadata per request
- User-level and session-level attribution out of the box
- Prompt management and versioning with A/B testing support
- Caching layer to reduce duplicate calls and control spend
- Self-hostable or managed cloud deployment
Best for: Teams that prioritize observability and cost attribution above everything else. Particularly useful when multiple teams or product features share the same LLM budget, and you need to break down spend by owner.
Trade-offs: Helicone is observability-first, and fallback logic is less of a focus compared to LiteLLM or Kong. If your primary problem is multi-provider failover rather than usage visibility, you’ll likely combine Helicone with another routing layer or use a different tool.
2. LiteLLM
LiteLLM is the most widely adopted open-source LLM proxy in 2026, and for good reason: it supports more model providers than any other and has an enormous community contributing to it.
It exposes a single OpenAI-compatible endpoint to 100+ models. You point your app at LiteLLM, configure your providers in a YAML file, and the gateway handles translation, retries, and fallback.
Key capabilities:
- Broadest provider support: OpenAI, Anthropic, Azure, Bedrock, Cohere, Mistral, Ollama, and dozens more
- OpenAI-compatible API (minimal code changes to adopt)
- Budget controls per key, per team, per model
- Built-in spend tracking and cost dashboards
- Proxy server with Redis-backed caching
Best for: Teams that need maximum provider coverage and a battle-tested community project. If you’re standardizing across a heterogeneous model portfolio, LiteLLM has the widest support.
Trade-offs: The breadth comes with configuration complexity. At large scale, self-hosting LiteLLM requires careful tuning; it can become a performance bottleneck if not provisioned correctly. Observability is functional but not as deep as Helicone’s native offering.
3. Kong AI Gateway
Kong built its reputation on API gateways for microservices, and Kong AI Gateway extends that into LLM traffic. If your organization already runs Kong for API management, this is a natural addition; it plugs into the same control plane, uses the same plugin architecture, and fits into existing GitOps workflows.
Key capabilities:
- Semantic caching to reduce duplicate LLM calls (and cost)
- AI prompt engineering plugins (injecting system prompts at the gateway level)
- Rate limiting, authentication, and logging via existing Kong plugins
- Multi-cloud support with declarative configuration
- Enterprise tier available with dedicated support
Best for: Enterprises already on Kong for API management, or teams that need gateway-level prompt manipulation without application changes.
Trade-offs: If you’re not already in the Kong ecosystem, the setup overhead is real. It’s a heavier deployment compared to purpose-built LLM gateways. The AI-specific features are also more recent, so they’re not as mature as Kong’s core gateway capabilities.
4. Apache APISIX
APISIX is a high-performance API gateway that added LLM routing support in recent versions. It’s CNCF-hosted, runs on NGINX under the hood, and is built for teams that need raw throughput at scale.
The LLM plugin layer handles provider routing, load balancing, and basic observability. What APISIX brings is performance; it consistently outperforms other gateways in requests-per-second benchmarks at high concurrency, which matters when you’re processing thousands of simultaneous LLM calls.
Key capabilities:
- High-throughput architecture (Lua-based plugins, NGINX core)
- Dynamic routing without reloads
- Plugin ecosystem covering auth, rate limiting, observability, and transforms.
- Strong Kubernetes integration
- Active Apache Software Foundation governance
Best for: Teams where raw gateway performance is the primary constraint, high-volume inference pipelines, real-time applications where latency margins are tight.
Trade-offs: LLM-specific features (semantic caching, cost tracking, model fallback chains) are less mature than those of purpose-built LLM gateways. You’ll likely need to build or integrate additional tooling for production-grade LLM observability.
5. Envoy AI Gateway
Envoy Proxy is the infrastructure backbone for many service meshes (Istio, for one), and Envoy AI Gateway extends it with LLM-specific routing capabilities. The value proposition is clear if you’re already running Envoy: you add AI routing to an existing, hardened proxy layer rather than operating a separate gateway.
Key capabilities:
- Built on battle-tested Envoy infrastructure
- Token-based rate limiting (not just request-based, critical for LLM cost control)
- Header-based routing and model selection
- Strong observability via existing Envoy metrics pipeline
- Native integration with service mesh deployments
Best for: Teams running Istio or other Envoy-based service meshes who want to add LLM routing to existing infrastructure. Also strong for teams with strict compliance requirements that need request-level auditability.
Trade-offs: Envoy AI Gateway is earlier-stage than the others. Configuration is verbose (Envoy’s xDS API is powerful but not quick to get right). If you don’t already know Envoy, the learning curve is steep.
Side-by-Side Comparison
| Feature | Helicone | LiteLLM | Kong AI Gateway | Apache APISIX | Envoy AI Gateway |
|---|---|---|---|---|---|
| Provider Support | OpenAI-compatible endpoints | 100+ (broadest) | Major providers | Major providers | Major providers |
| API Compatibility | OpenAI-compatible | OpenAI-compatible | OpenAI-compatible | Plugin-based | xDS / HTTP |
| Fallback / Failover | Limited (observability focus) | ✅ | ✅ | ✅ | ✅ |
| Semantic Caching | ✅ | Partial | ✅ | Limited | ❌ |
| Native Observability | ✅ Deep (core feature) | ✅ Basic | ✅ Via plugins | ✅ Via plugins | ✅ Via Envoy metrics |
| Token-Level Rate Limiting | ✅ | ✅ | ✅ | Request-level | ✅ |
| Cost Tracking | ✅ Per request + per user | ✅ Budget controls | Partial | Limited | ❌ Native |
| Kubernetes Native | ✅ | ✅ | ✅ | ✅ | ✅ |
| Setup Complexity | Low | Low | Medium–High | Medium–High | High |
| Maturity | Mature | Mature | Mature (AI layer newer) | Mature (AI layer newer) | Early-stage |
| Best Fit | Observability + cost attribution | Max provider coverage | Kong-ecosystem teams | High-throughput pipelines | Envoy/service mesh teams |
How to Pick One Without Overthinking It
The gateway decision usually comes down to three questions:
1. What’s already in your infrastructure?
If you run Kong, use Kong AI Gateway. If you run Envoy/Istio, evaluate Envoy AI Gateway. If you’re greenfield, pick based on capabilities.
2. How important is deep observability?
If you need cost attribution per team, per project, or per feature, and you don't want to wire up five separate tools, Helicone’s native approach wins. The proxy setup is a single base URL change; request-level tracing, cost tracking, and user attribution work immediately with no additional instrumentation.
3. How many providers do you need to support?
If your answer is “all of them” or close to it, LiteLLM has the broadest support and the most community-contributed integrations.
What “Enterprise Scale” Actually Breaks
A gateway that works in development often fails in production for one of three reasons:
- Stateless routing at high concurrency: Most gateways handle this fine, but verify your chosen gateway can handle thousands of simultaneous streaming connections without memory bloat. Envoy and APISIX have the strongest track record here.
- Fallback logic for partial provider failures: If a provider is degraded (slow responses rather than hard failures), naive retry logic can make things worse. Look for gateways that support circuit breakers and timeout-based fallback, not just error-code-based fallback.
- Cost attribution at the team level: Token usage without team-level attribution means your finance team can’t audit spend, and engineering teams have no incentive to optimize. This is a gap across several gateways: Helicone and LiteLLM handle it natively, while APISIX requires external tooling.
Conclusion
Running enterprise AI at scale isn’t a model problem; it’s an infrastructure problem. The model is the easy part. Routing, fallback, cost controls, observability, and access management are where teams consistently underestimate complexity.
The right gateway is the one that fits your existing stack and solves your actual bottleneck, whether that’s provider breadth, raw throughput, observability depth, or the need to integrate with an existing Kong or Envoy deployment. None of these projects is a wrong choice; they have different priorities.
What matters is picking one and centralizing your LLM traffic through it before that routing logic spreads across every service in your stack and becomes impossible to audit or change.
Top comments (7)
Great breakdown, this matches what we've seen operationally. One thing I'd add to the "how to pick" section: the fallback behavior under partial failure matters more than the feature checkbox. A lot of teams pick a gateway because it has "fallback ✅" on a comparison table, then get burned in production because it only triggers on hard errors (5xx) and not on a provider that's just gotten slow. Circuit breakers with latency-based tripping are the real differentiator, and it's not something you can tell from a feature matrix, you have to load test it yourself.
Also worth flagging for anyone choosing based on this table: LiteLLM's proxy can become a genuine bottleneck at high concurrency if you don't run it with enough workers and a properly tuned Redis backend for the cache/budget layer. It's easy to underprovision it because the initial setup feels so lightweight.
Solid overview overall, especially the point that this is an infra decision, not a model decision. Curious whether you've tested any of these under simulated provider degradation (not just outage), that's usually where the real differences show up.
Really good addition, and honestly the more important one. "Fallback: ✅" on a comparison table tells you almost nothing about the condition that actually triggers it. Most implementations I've looked at are wired to hard failures, connection refused, 5xx, timeout past some generous threshold, which covers the easy case and completely misses the one that actually costs people money in production: a provider that's still returning 200s but has quietly degraded to 3x normal latency. Nothing trips, nothing fails over, and you just eat the slow responses until someone notices error budgets look fine but users are complaining.
Latency-based circuit breaking is the real signal, and you're right that it's invisible from a feature matrix. You have to actually inject degraded latency (not just kill the endpoint) and watch whether the gateway's failover logic reacts to it. A lot of them don't, by design or by oversight, because "is it up" is a much easier thing to build detection for than "is it behaving normally."
The LiteLLM point is a good one to flag explicitly. The proxy is deceptively easy to stand up, which I think causes people to treat that initial config as the steady state instead of a starting point. Under real concurrency the worker count and the Redis tuning for the cache and budget layer stop being optional details and start being the thing that determines whether the gateway itself becomes the tail latency problem you were trying to avoid in the first place.
On your question, yes, simulated degradation is where most of the differentiation actually showed up for us too. Clean outage tests make almost every gateway look fine since binary failure is the easy case to handle. Once you inject partial degradation, elevated latency without errors, intermittent slow responses, a provider that's healthy but rate limiting, you see which ones have real health-check logic versus which ones are just watching for connection failures. That gap doesn't show up until you go looking for it, which is probably the single biggest thing missing from most public comparisons of these tools.
It's been a good exchange, thanks for engaging with it in depth. Good luck with the load testing, and if you end up writing up the degraded-latency results somewhere, that'd be a genuinely useful public data point given how little of that testing gets shared.
Nice comparison, this is a topic that needs more clear-eyed writeups. I'd push back slightly on grouping Kong AI Gateway and Envoy AI Gateway with the "purpose-built" options in the takeaways section. If a team isn't already running Kong or Envoy, the operational cost of adopting either just for LLM routing is significant, you're taking on an entire API gateway or service mesh layer to get token rate limiting. That's a very different commitment than standing up LiteLLM or Helicone.
Would also add that "self-hostable" hides a lot of variance in actual maintenance burden. LiteLLM's proxy is a fairly small footprint to run yourself. Kong and Envoy are not, even before you touch the AI-specific plugins.
Appreciate that you called out the cost attribution gap for APISIX specifically, that's the kind of detail that only shows up after you've actually tried to get finance an answer, not from reading docs.
That's a fair pushback, and honestly the grouping was doing too much work in that takeaways section.
The distinction you're drawing is the right one. Kong and Envoy are infrastructure layers that happen to have AI gateway plugins now, not tools built for the AI gateway use case first. If you're already running Kong for your API estate or Envoy as your mesh data plane, adding the AI plugin is a marginal cost, you're extending something you operate anyway. But if a team's starting from zero, "just add the AI gateway plugin" undersells what they're actually taking on: config model, upgrade cadence, the whole operational surface of a gateway or mesh, before they've routed a single LLM request. That's a different category of decision than "install LiteLLM proxy and point traffic at it."
Good call on self-hostable hiding variance too. I was using it as a binary when it's really a spectrum of operational weight. LiteLLM's proxy is close to a single deployable service with a config file. Kong and Envoy come with their own control plane concerns, their own failure modes, their own people-need-to-know-this-tool cost, independent of anything AI-specific. Lumping all three under "self-hostable" in a takeaways table flattens exactly the distinction a team evaluating this needs.
On APISIX, that one came from watching someone actually try to hand finance a per-team cost breakdown and hit a wall the docs don't warn you about. Glad it landed, that kind of gap only shows up in practice, never in a feature comparison table.
Appreciate you pushing on the grouping, I'll rework that section.
This is a useful list, thanks for putting it together. Would add one nuance: "maturity" in the table is doing a lot of work for Kong and APISIX. The core gateways are rock solid, but the AI-specific plugins on both are genuinely new, so "mature (AI layer newer)" is a bit generous when you're evaluating things like semantic caching correctness or token counting accuracy under streaming responses. Worth a longer bake-off before betting production spend controls on them.
Also curious how you're defining "broadest provider support" for LiteLLM in practice, since a lot of the 100+ list is thin wrappers where things like function calling or structured outputs don't behave consistently across providers. Provider count and provider parity are two different claims.
Good writeup either way, bookmarking this for the next time someone on my team asks "just use LangChain right?" and I get to say no.
Fair pushback on both counts, thanks for pressing on them.
On Kong and APISIX, you're right that I collapsed two different maturity claims into one cell. The core proxy in both cases has years of production hardening behind it, but the AI plugins are a recent layer bolted onto that foundation, and recent means exactly what you'd expect: less battle testing on the parts that are hardest to get right. Semantic caching correctness is a good example, since a cache hit on the wrong semantic match is a silent failure mode, not a loud one, and you won't find it without deliberately testing for it. Token counting under streaming is similar, the failure is a slow drift in accuracy rather than a crash. Both deserve their own bake-off rather than inheriting the reputation of the underlying gateway. I'll fix the table to split those out instead of averaging them into one "mature" label.
On LiteLLM, that's a real distinction I should have been more careful with. Provider count is trivially true and easy to verify, you can count the entries in the docs. Provider parity is the claim that actually matters for production and it's much harder to verify, since it means digging into whether function calling, structured outputs, and streaming behavior are consistent rather than just present. A lot of that 100+ list is coverage in the sense of "a request will not immediately error out," not coverage in the sense of "the contract behaves the same way across providers." I was sloppy conflating the two and "broadest provider support" should really be read as "broadest provider count," with parity flagged as a separate, unverified claim that needs testing per feature per provider before you rely on it.
Appreciate you keeping the writeup honest. Consider both corrections noted for the next revision.