What Is an AI Gateway? How LLM Gateways Work and When You Need One

An AI gateway is a proxy between your application and every LLM provider. What it does on each request, how routing and fallback work, what it costs you, and the signs you need one.

An AI gateway, also called an LLM gateway, is a proxy that sits between your application and the model providers it calls. Your code sends every request to one endpoint, usually in the OpenAI chat completions format, and the gateway decides which provider and model serve it, applies your limits and policies, and records what happened.

The idea is the same one API gateways brought to microservices: move the cross-cutting work out of every service and into one place. With LLMs that work is unusually heavy, because providers differ in auth, request shape, rate-limit behaviour, error format and price.

What happens to a request

A gateway is a pipeline. Everstack's runs each request through these stages, and most gateways have some version of each:

Rate limiting enforces request budgets per key and per tenant, using a token bucket.
Authentication validates the API key and resolves which tenant the request belongs to.
Routing turns the model field into a concrete provider and model.
Cache lookup returns a stored answer for an exact or semantically similar request.
Context compaction trims conversation history that would overflow the target model's context window.
Provider call translates the request into the provider's native API, injects that provider's credential, and translates the response back.
Response caching stores the result for the next identical or similar request, and the request's latency, tokens and cost go to the observability pipeline.

Every stage is optional. A gateway running nothing but routing is still useful; one running all of them replaces a surprising amount of application code.

Routing and fallback

Routing is the core of the job. A request for claude-sonnet-5-5 has to reach Anthropic's Messages API; a request for fast might be an alias your team defined for a cheap model with a lower max_tokens. Everstack resolves the model field in a fixed order: tenant-defined aliases first, then explicit routes from configuration, then the built-in model catalog. The first match wins, which is what lets one tenant override default routing without affecting another.

Fallback is what makes a gateway worth running in production. A fallback chain lists providers to try when the first fails, with one of three strategies:

Priority tries providers in order. Use it when you have a clear preference, such as the cheapest provider first.
Round robin spreads requests across providers with similar capabilities.
Parallel sends to all of them and takes the first success, trading cost for latency.

What triggers a fallback matters as much as the chain. Rate limits (HTTP 429), timeouts and server errors (5xx) are transient, so retrying elsewhere is right. A model-not-found error is a configuration problem, and a well-behaved gateway does not paper over it.

What a gateway is not

A gateway's scope ends where the request ends. It does not run your agent's loop, execute the code a model writes, decide whether an answer was correct, or store a user's memory between sessions. Those are runtime problems, and teams that need them either assemble separate systems around the gateway or use a platform that ships them together. Be clear which problem you are buying for: adding a platform to solve a proxy-sized problem is over-buying, and a proxy will not grow into an agent runtime.

Signs you need one

More than one provider in production. Each new provider otherwise means a new SDK, new error handling and new retry logic in every service.
Outages or rate limits are reaching users. Fallback across providers is the cheapest reliability you can add.
Nobody can say what AI costs per customer. A gateway sees every request, so per-tenant spend and budgets become a query rather than a spreadsheet.
Keys are scattered. Provider keys in application config are a leak waiting to happen; a gateway holds them and issues scoped keys instead.
Several teams build on LLMs. Rate limits, caching and observability bolted on per service drift apart quickly.

You probably do not need one yet if you call a single provider from a single service, and that provider's own dashboard answers your cost questions.

Hosted or self-hosted

Hosted gateways and aggregators are the fastest start: one key, no infrastructure, often the widest model selection. The trade is that prompts and responses transit the vendor's infrastructure, and some charge fees on top of provider prices.

Self-hosted gateways keep traffic on infrastructure you control and let you use your own provider contracts. The trade is operational: you run it, upgrade it and keep it available. Read the licence before adopting one. Many open-source gateways put the features you need by month six, such as virtual keys, budgets, SSO or high availability, behind a paid enterprise licence.