What a gateway actually is 8 min live
The clearest primary definition in the category comes from Microsoft, not from a startup landing page. Azure API Management describes its AI gateway as "a set of capabilities that help you manage your AI backends effectively. Use these capabilities to secure, scale, monitor, and govern AI models, agents, and tools." The sentence right after it is the one worth reading twice: it "extends API Management's existing API gateway; it's not a separate offering." This is not a new category of infrastructure. It is the API gateway your organization has run for fifteen years, taught to count tokens.
LiveThe seven capabilities, verified across five products4 min▶
These are not one vendor's feature list. Each row below appears in the product documentation of Azure API Management, Kong, LiteLLM, Cloudflare and MLflow. When a category converges this tightly on its feature set, the feature set is the definition.
| Capability | What it does | Why it cannot live in the app |
|---|---|---|
| Unified OpenAI-compatible API | one request shape for every provider | it can, and this is the one that does |
| Routing and failover | send traffic elsewhere when a provider degrades | needs a live view of provider health across all callers |
| Cost tracking and budgets | spend per team, per app, per key, with hard caps | needs every team's traffic in one ledger |
| Token rate limiting | caps tokens per minute per consumer | a quota is only meaningful if it is enforced centrally |
| Semantic and exact caching | returns a stored answer instead of paying again | a cache hit only pays off if it is shared |
| Observability | logs prompts, completions, latency, errors | the eval set is the whole estate, not one app |
| Key management and guardrails | rotation, scoping, content policy at the edge | keys distributed to apps cannot be rotated centrally |
Read the third column downward. Five of the seven capabilities are structurally impossible inside a single application, no matter how good its libraries are. That column is the whole argument for layer seven, and Part 2 turns it into a rule.
LiveMicrosoft's "why," which is the most concrete one any vendor gives4 min▶
Most gateway pitches open with "unified API." Microsoft's opens with an operations problem, and it is far more persuasive because it names a failure you can picture.
With one application you simply cap tokens-per-minute on the deployment and you are done. With twenty applications, in Microsoft's own words, "you need to make sure that one app doesn't use the whole TPM quota and block other apps."
That is a noisy-neighbour problem over prepaid throughput. You bought a fixed quantity of tokens per minute, and every consumer is drawing from the same tap. The first application to write a retry loop takes the capacity that the customer-facing product needed. Nobody is malicious; the architecture just has no arbiter.
This reframes the purchase honestly. You are not buying a nicer SDK. You are buying an arbiter for a shared, prepaid, finite resource, plus the ledger that says who consumed it. Every other capability follows from having somewhere to stand.
Self-studyRead the vendor stat block as a frozen marketing artifact2 min read▶
A small critical-reading habit worth carrying into every evaluation in this layer. LiteLLM says "100+ LLMs" on its documentation and "140+" on its marketing pages. Neither number is dishonest; they were simply written at different times and never resynchronised.
The same pattern shows up across this shelf: Cube's homepage claimed 18,000 GitHub stars against an actual 20,564, and Prefect advertises "50.7k+" against a repository count of 23,565 (both fetched 2026-08-07). Vendor stat blocks are snapshots someone pasted in once, not telemetry. If a number matters to your decision, fetch it yourself from the primary feed and write down the date, which is exactly what every price on these pages does.
The rule: SDK or proxy ★ 12 min · everyone classifies their own stack
If you remember one sentence from layer seven, make it this one. The SDK gives one application many providers. The proxy gives many applications one governed door. They look similar in a demo because both let you swap a model name in a string. They are not similar at all, and the difference is not a matter of degree.
List every AI application in your organization. If the answer is one, and it will stay one for a year, the SDK is the correct choice and a proxy is a component you will operate for no benefit.
Now ask who is going to be asked, at some point, "what did we spend on AI last quarter and which team spent it?" If that question has an owner, you need the proxy, and you needed it before the second application shipped.
Apply the chokepoint test from b1: what stops a team calling the provider API directly? If the answer is a policy document rather than network egress rules and key custody, your gateway covers the compliant half of your spend and your dashboards are confidently wrong.
Classify each product below yourself before reading the table. Two of them will surprise the room.
| Product | Gateway? | What the documentation actually supports |
|---|---|---|
| LiteLLM Proxy | Yes | many apps, one door, per-key budgets and attribution |
| Portkey | Yes | full capability set, now inside Palo Alto Networks |
| Cloudflare AI Gateway | Yes | caching, rate limiting and analytics at the edge |
| Kong AI Gateway | Yes | AI plugins on an existing API gateway data plane |
| Azure API Management | Yes | explicitly an extension of the existing API gateway |
| MLflow AI Gateway | Yes | plus traffic collection into evaluation datasets |
| Amazon Bedrock | No | the Converse API unifies the surface for models AWS hosts |
| OpenRouter | Partly | a gateway that also intermediates the commercial relationship |
LiveWhy Bedrock is not a gateway, precisely4 min▶
This is not a criticism of Bedrock, which is a good product doing a different job. It is a category correction, and getting it wrong produces an architecture with a hole where the ledger should be.
- Bedrock's Converse API unifies the request surface for models AWS hosts. That is genuinely useful and it is one of the seven capabilities. It is not the other six.
- It does not proxy first-party OpenAI or Anthropic APIs. Traffic your teams send directly to those providers is invisible to it, which means it cannot be the only door.
- It therefore cannot do cross-provider cost attribution, which is the capability most organizations actually bought a gateway for.
- The tell is in a competitor's documentation: Azure's own docs treat Bedrock as a backend that the gateway manages. A thing that sits behind the gateway is not the gateway.
The practical consequence: if your architecture row for layer seven says "Bedrock," you have chosen a model-hosting layer and left the governance chokepoint empty. That is a defensible choice on an AWS-only estate. It is not a defensible answer to "what are we spending across providers."
LiveWhy OpenRouter is only partly one4 min▶
OpenRouter does most of what the capability table lists, so calling it "not a gateway" would be wrong. The reason it sits in its own row is that it changes something the other gateways leave alone: it runs its own credit billing rather than routing through your provider accounts.
That single design decision moves it from an infrastructure component to a counterparty, with two consequences worth stating in a procurement meeting:
- Every prompt transits a third party that is not your model provider. For a self-hosted proxy the prompts stay inside your perimeter; here there is an extra organization holding your traffic.
- You lose direct rate-limit negotiation. When you need a quota increase for a launch, you are no longer the provider's customer of record, so the conversation you would have had is not available to you.
For prototyping and for reaching long-tail models quickly, that trade is often worth it. For the production path of a regulated workload, it usually is not. The point of the classification exercise is to make that a decision rather than an accident.
Production traffic becomes an eval set 6 min live
Here is the capability that never makes the comparison grid and is worth more than any of the seven that do. A gateway sees every prompt and every completion your organization produces. That is not a logging side effect. It is the only place where the question "how well is our AI actually working" can be answered with your own traffic instead of somebody's benchmark.
LiveTwo vendors say this out loud3 min▶
- Azure API Management logs prompts and completions to Azure Monitor. Once that stream exists, your real questions, in your real vocabulary, against your real data, are sitting in a queryable store.
- MLflow's AI Gateway states the purpose explicitly, enabling "dataset collection from real-world traffic for building evaluation benchmarks." That is a vendor telling you the gateway is an eval-set collector, in the documentation, without a marketing frame.
Why this matters more than it sounds: in b9 you will see that the gap between a public benchmark score and your warehouse is enormous, and that the only measurement worth trusting is one taken on your own conditions. The gateway is where those conditions are recorded. Skip it and every accuracy conversation you have for the next two years is about somebody else's data.
Cross-reference: the full treatment of building and running evaluations on collected traffic is in learn-ai-evals. This session's job is only to make sure the collection point exists in your architecture, because a gateway added later cannot log traffic that already happened.
LiveThe four questions, asked of a rented model3 min▶
Run b7's registry questions against layer seven and the parallel stops being an analogy and becomes an operational checklist:
| Question | Registry answer (models you train) | Gateway answer (models you rent) |
|---|---|---|
| What is running? | the artifact behind the alias | the model and version each route resolves to |
| Where did it come from? | lineage to run and training data | which provider, which deployment, which region |
| Who approved it? | the promotion gate | the route config change and who merged it |
| How do I roll it back? | reassign the alias pointer | repoint the route, no application deploy |
That last row is the underrated one. In the SDK world, changing model providers is a code change, a review, a release and a deploy per application. Behind a proxy it is a configuration change in one place, which is also precisely why the proxy is a governance surface and the SDK is not.
What the gateway costs you 7 min live
Every argument so far has been for the gateway, and the argument is sound. Now the other side, at the same volume, because an architecture row you cannot defend is not a decision. Five of these, and the fifth one has a date on it.
1 · You inserted a single point of failure into 100% of AI traffic, in the name of reliability. Sit with the shape of that. You added the component to survive one provider degrading, and in doing so you created a component whose outage makes every healthy provider unreachable. This is worth paying for, but it must be run like the tier-one dependency it now is, not like a proxy someone stood up in a sprint.
2 · Added latency on every single call. LiteLLM rewrote its core in Rust and publishes p99 overhead figures precisely because buyers interrogate this. Take the numbers as directional only: they are self-reported, unaudited, and their own published figures are internally inconsistent across pages. Measure it yourself against your own payload sizes, which takes an afternoon and settles the argument.
3 · Unified APIs are lowest-common-denominator by construction. The proof is in the products themselves: passthrough endpoints exist as escape hatches, letting you send a provider-native request straight through. That feature is an admission that the abstraction leaks. Expect your most valuable use case to be the one that needs the escape hatch.
4 · The gateway concentrates every provider key and every prompt in one system. You built a single store containing your credentials, your customers' questions and your proprietary context. Palo Alto Networks completing its acquisition of Portkey on 2026-05-29 is the market pricing exactly this - a security company buying the place where the secrets pool.
5 · Vendor risk, now on the critical path. Cloudflare deprecated its Universal Endpoint. The category is three years old and already deprecating primitives. Check for retirement notices before you design around any managed feature, and keep the routing configuration in a form you could re-express somewhere else.
Self-studyThe vendor-risk habit, applied beyond this layer2 min read▶
Cloudflare's Universal Endpoint deprecation is not an isolated event, and the habit it should build is general. Two other live examples from this course, both verbatim from vendor documentation:
- "Amazon SageMaker Model Monitor is no longer open to new customers" - a governance-critical component of a flagship service, quietly closed to new adoption.
- Vertex AI is now presented as the "Gemini Enterprise Agent Platform", which is a repositioning rather than a retirement, but it changes what you are buying and what the documentation covers.
So add one line to every vendor evaluation: search the docs for retirement, deprecation and "no longer open to new customers" before the demo, not after the contract. It takes five minutes and it is the cheapest risk work available to you.
Try it yourself - this week ◐ 30-40 min total
- Count the AI applications in your organization, then try to answer "what did each one spend last month." How you get to that answer, or fail to, tells you whether you have layer seven or only think you do.
- Apply the chokepoint test in writing: list every route by which a team could reach a model provider without passing your gateway. If the list is not empty, your cost dashboard covers a fraction you should be able to name.
- Classify whatever you run today against the eight-row table in Part 2. If the row says Bedrock or OpenRouter, write the one sentence explaining what you gave up and why it was the right trade.
- Measure the added latency yourself on a representative payload rather than quoting a vendor's p99. Bring the number to b10, because the sponsor will ask.
Official sources covered
Every definition and capability on this page comes from product documentation rather than analyst commentary, because this category is young enough that the documentation is still the most honest thing written about it. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Your team already uses an SDK that supports every major provider. Finance now wants AI spend broken down by team. What do you tell them?
Budgets, attribution, key rotation and audit all require a place where every team's traffic exists. Inside one application process there is no such place - which is why this is a structural limit, not a feature gap.
2 · A colleague says "we already have a gateway, it's Bedrock." What is the correction?
Bedrock does one of the seven capabilities well and is a sound model-hosting choice. Calling it layer seven leaves the governance chokepoint empty, so nobody can answer "what are we spending across providers."
3 · What is the uncomfortable shape of the reliability argument for a gateway?
The component added to survive one provider degrading becomes the component whose outage makes every provider unreachable. That is worth paying for, but only if it is operated as a tier-one dependency with an owner and a runbook.