LLM gateway
Last verified: June 2026· definition
An LLM gateway (or AI gateway) centralizes calls to one or more model providers. Instead of each service calling provider APIs directly, they call the gateway, which adds cross-cutting concerns: routing, caching, retries and fallbacks, rate limiting, logging, and cost tracking.
A gateway gives you a single place to change models, enforce policy, and see spend by route — which is exactly what makes cost and quality manageable as usage grows.
Gateways can be self-hosted (for data residency and control) or managed; the right choice depends on your data and governance needs.
The value is having one place for cross-cutting concerns
Routing, caching, fallbacks, rate limiting, cost attribution, and audit logging are all things every service calling a model needs. Implemented per-service they drift apart, get implemented inconsistently, and make questions like total spend by team unanswerable. A gateway concentrates them so behaviour is uniform and observability is complete by construction. That consolidation, rather than any individual feature, is usually what justifies it.
It makes provider changes configuration, not code
Applications calling providers directly embed that choice throughout the codebase, so switching providers, adding a fallback, or trying a different model becomes a code change in many places. Behind a gateway these become configuration. This matters most during incidents — when a provider is degraded, being able to fail over centrally is the difference between a brief blip and an outage lasting as long as a deploy cycle.
It is a dependency on your critical path
Everything routes through it, so its availability becomes your AI features’ availability and its latency adds to every call. This is the real cost of the pattern and it deserves explicit treatment: the gateway needs the same operational rigour as any other critical service, and ideally a path for applications to degrade gracefully if it becomes unavailable. Teams sometimes adopt a gateway for reliability and inadvertently introduce a new single point of failure.
Start with what you need, not the full feature set
Gateways accumulate capability, and adopting all of it at once obscures which parts are earning their place. Most teams get the bulk of the value from routing, fallbacks, and per-route cost attribution. Caching, guardrail integration, and advanced governance are worth adding when a concrete need appears, since each adds configuration surface and another way for behaviour to differ from what an application author expects.
Common misconceptions
MythA gateway is only worth it at large scale.
RealityThe consolidation argument applies as soon as more than one service calls a model. What scale changes is how much the inconsistency costs, not whether it exists.
MythA gateway saves money by itself.
RealityIt is where cost controls live, not a saving in its own right. The savings come from routing, caching, and batching decisions the gateway makes convenient to implement.
MythYou must self-host the gateway to have control.
RealitySelf-hosting gives full control of the data path and adds operational burden. Managed gateways trade some control for less operational work; which is right depends on your data requirements, not on a general principle.
Frequently asked questions
Do I need an LLM gateway?
It becomes worthwhile when more than one service calls models, when you need cost attribution across teams, or when you want routing and fallbacks without changing application code. A single service with one model does not need one, and adding it early can be premature. The usual trigger is the second consumer appearing, at which point the cost of divergent implementations starts to be felt.
Should the gateway be self-hosted or managed?
It turns mainly on your data requirements. Self-hosting keeps request and response data entirely within your infrastructure, which matters for residency and sensitive content, and adds a service to operate. Managed gateways remove that burden and introduce a hop through a third party. If data may not leave your environment, the decision is already made; otherwise weigh the operational cost against the convenience.
What should a gateway handle that applications should not?
Anything cross-cutting: provider selection and routing, retries and fallbacks, rate limiting, prompt and response caching, cost and token attribution, and audit logging. Anything domain-specific — prompt construction, business validation, deciding what a feature does with a result — belongs in the application. A gateway accumulating application logic becomes a bottleneck for every team that depends on it.