If your AI application suddenly becomes 300–500ms slower after adding an AI gateway, the first question should not be “Is the gateway slow?” It should be “Which part of the gateway is actually consuming the time?” An extra network hop can add latency, but a 400ms increase is usually a sign that something more than simple request forwarding is happening. In practice, the delay can come from connection setup, DNS or TLS negotiation, authentication lookups, synchronous logging, policy checks, retries, buffering, provider selection, or simply measuring time incorrectly. Modern AI gateways generally add milliseconds, not hundreds of milliseconds, when they are warm and properly configured, so a large increase warrants a request-level trace rather than guesswork.
The first thing I checked: direct API vs gateway
The most useful test is also the simplest.
I sent the same request to the model provider in two ways:
- Directly from the application to the provider.
- Through the AI gateway to the same provider.
Everything else stayed the same: model, prompt, API key, generation settings, region, and request payload.
The important number is:
Gateway overhead = gateway request latency − direct provider latency
That distinction matters because the total response time includes model inference. If the direct request takes 900ms and the gateway request takes 1.3 seconds, the gateway did not necessarily “make the model slower.” The gateway added roughly 400ms somewhere around the provider call.
This is also why average latency can be misleading. I prefer comparing p50, p95, and p99, because a gateway can look perfectly healthy at the median while connection setup, overloaded workers, or retries create painful tail latency.
Where the 400ms usually goes
When an AI gateway adds hundreds of milliseconds, I break the request into separate stages rather than treating the gateway as a single black box.
A typical request looks roughly like this:
Client → Gateway → Authentication → Policy → Routing → Provider → Streaming response → Gateway → Client
Each stage needs its own timestamp.
For example:
| Stage | What to Measure |
|---|---|
| Client → Gateway | Network + TLS |
| Authentication | Token/key validation |
| Policy Checks | Rules, limits, classification |
| Routing | Model/provider selection |
| Gateway → Provider | Connection + network |
| Provider TTFT | Model processing |
| Streaming | First token and token delivery |
| Logging | Synchronous audit/telemetry work |
If the gateway reports only “request completed in 1.3s,” you still don't know where the 400ms went.
That was the first lesson: measure the individual stages, not just the final response time.
1. Connection setup can quietly add latency
One of the easiest problems to miss is connection reuse.
If the gateway creates a new outbound connection for every AI request, the request can incur DNS lookup, TCP setup, and TLS negotiation costs repeatedly.
That is unnecessary overhead for a high-volume AI application.
The provider connection should normally be pooled and reused. The same principle applies to the connection between your application and the gateway.
I would check:
- Are HTTP keep-alive connections enabled?
- Is the HTTP client reusing connections?
- Is connection pooling configured correctly?
- Is DNS being resolved repeatedly?
- Is TLS being negotiated for every request?
- Are idle connections being closed too aggressively?
- Is the gateway running close to the provider region?
This is especially important for short AI requests. If the model returns quickly, network setup becomes a much larger percentage of total latency.
For longer generations, provider inference usually dominates, but that does not make inefficient connection handling acceptable.
2. Authentication should not hit a database every time
Another common source of unnecessary latency is authentication.
Imagine every request entering the gateway and triggering:
API request → database lookup → user lookup → permission lookup → continue
Even a relatively fast database query becomes expensive when it happens on every request.
For high-frequency AI traffic, authentication data that rarely changes should generally be cached where appropriate.
I would measure:
- Token verification time
- User lookup time
- Permission lookup time
- Cache hit rate
- Cache miss latency
- External identity-provider calls
If a cache hit takes 2ms but a cache miss takes 80ms, you immediately have something useful to investigate.
The key is not to remove authentication. It is to avoid unnecessary synchronous work on every request.
3. Synchronous logging can become a hidden bottleneck
Logging looks harmless until the gateway starts doing too much of it.
A request may trigger:
- Request logging
- Token accounting
- Cost calculation
- Audit logging
- Trace creation
- Database writes
- Metrics
- Security events
If the gateway waits for those operations before forwarding the request, the latency adds up quickly.
For example:
Provider request → write audit record → wait for database → continue.
is very different from:
Provider request → enqueue audit event → continue
For latency-sensitive traffic, telemetry that does not affect the routing decision should generally be designed so it doesn't unnecessarily block the request path.
This does not mean turning off observability. It means separating decision-critical work from record-keeping work.
Modern gateway designs commonly expose separate gateway processing and provider timing, allowing engineers to distinguish between the two.
4. Policy checks can become surprisingly expensive
Authentication usually isn't the only gateway logic.
Production AI gateways may also check:
- Rate limits
- Model permissions
- Organization limits
- Token budgets
- Prompt policies
- Data-loss rules
- Geographic restrictions
- Model routing rules
- Content classification
One rule might take milliseconds.
Ten rules involving external services can become a different problem.
The biggest mistake is running these checks serially:
Check A → Check B → Check C → Check D
If each one takes 20ms, you've already created an 80ms delay before the model receives the request.
Independent checks should run in parallel.
Caching is also useful for decisions that do not change on every request. Recent gateway benchmarking work emphasizes measuring identity, classification, policy evaluation, and audit operations separately because their latency characteristics are different.
5. Retries can explain a “mysterious” 400ms
This is one of the first things I check when latency suddenly jumps.
Suppose the normal provider request takes 700ms.
A temporary connection failure occurs.
The gateway waits 100ms and retries.
The second request succeeds.
Now the user sees something closer to:
100ms retry delay + 700ms provider request
and possibly additional connection overhead.
The gateway may still report the request as successful.
From an uptime dashboard, everything looks fine.
From the user's perspective, the application feels slow.
That is why I track:
- Retry count
- Retry reason
- Retry delay
- Provider selected
- Fallback provider
- Total provider attempts
- Time spent before each attempt
A retry should never be invisible when debugging latency.
6. Streaming can expose another problem: buffering
For chat applications, I care much more about time to first token (TTFT) than total response time.
If the model begins generating after 500ms but the gateway buffers the response before sending anything to the browser, the user may see a blank screen for much longer.
The provider could already be producing tokens while the gateway is waiting.
So I measure two separate values:
Provider TTFT
and
Client-visible TTFT
If provider TTFT is 500ms but the browser receives the first token at 850ms, the missing 350ms is somewhere between the provider and the client.
That points toward gateway buffering, middleware, compression, transformations, or streaming configuration rather than model inference.
For interactive AI applications, this distinction is critical because users perceive responsiveness from the first visible output, not from when the server finishes generating the complete answer.
7. Don't benchmark the gateway against a fake request
Another mistake is testing the gateway against a mock provider and treating the resulting latency as production latency.
A mock upstream is useful for measuring the performance of pure proxies. It is not enough for understanding the real user experience.
Real AI requests include:
- Network distance
- Provider queueing
- Model processing
- Prompt size
- Output length
- Streaming behavior
- Provider variability
The fair comparison is:
Direct provider request vs gateway → same provider
under the same concurrency and workload.
That tells you what the gateway actually costs.
Benchmarks from current AI gateway implementations commonly put gateway-specific processing in the single-digit to low-tens-of-milliseconds range. However, the exact result depends heavily on architecture, concurrency, connection handling, and what the gateway does inline.
A practical debugging checklist
If I saw a consistent 400ms increase, this is the order I would investigate:
- First: Compare direct and gateway requests using the same provider.
- Second: Check p50, p95, and p99 rather than only averages.
- Third: Measure gateway processing time separately from provider latency.
- Fourth: Check connection reuse and TLS handshakes.
- Fifth: Measure calls to authentication and external dependencies.
- Sixth: Check synchronous database, logging, and audit operations.
- Seventh: Measure policy and classification latency.
- Eighth: Inspect retries and provider fallback behavior.
- Ninth: Compare provider TTFT with client-visible TTFT.
- Tenth: Repeat the test under realistic concurrency.
The goal is not to make the gateway “fast” in the abstract. The goal is to identify the exact operation consuming the latency budget.
What a reasonable gateway latency budget looks like
There is no universal number because the correct budget depends on the application.
A 30ms gateway overhead may be irrelevant for a request that takes 3 seconds to generate an answer.
The same 30ms can matter a lot for an application where the complete response is expected in under 100ms.
For a practical production target, I would establish a gateway-specific p95 budget and continuously measure against it, rather than relying on a one-time benchmark.
For example:
| Component | Example Target |
|---|---|
| Gateway processing | <10–20ms |
| Authentication | <5ms warm |
| Policy evaluation | <10ms |
| Audit/logging | Non-blocking |
| Connection reuse | Expected |
| Retry rate | Near zero normally |
| Client-visible TTFT | Track separately |
These are engineering targets, not universal standards. The right values depend on workload and architecture.
Conclusion
An AI gateway adding 400ms to every request is not something I would accept as “the cost of having a gateway.” A properly measured gateway should let you separate its own processing from the much higher and more variable cost of model inference. Current gateway benchmarks and implementations generally show that the proxy layer itself can operate in milliseconds, which means a persistent 400ms increase is worth investigating.
The practical fix is to stop treating the request as a single number. Trace the connection, authentication, policy checks, routing, provider call, retries, streaming, and logging independently.
Once those timestamps are visible, the missing 400ms usually stops being mysterious.
The gateway isn't necessarily the problem.
The problem is the work happening inside the gateway that you haven't measured yet.
Top comments (28)
Great breakdown. The point about testing direct API vs gateway with everything else held constant is something a lot of teams skip, they just look at total latency and assume the gateway is slow. The p50/p95/p99 distinction matters a lot too, since retries and cold connections mostly show up in the tail, not the median.
The synchronous logging section hit close to home. I've seen audit writes and token accounting block the response path more than once, and moving that to an async queue made a bigger difference than any "optimization" work. Same with policy checks running serially instead of in parallel, that's such an easy thing to overlook when each individual check feels cheap.
The provider TTFT vs client visible TTFT split is probably the most useful framing here. It's tempting to blame the model when generation actually started fine and the delay is sitting in gateway buffering or streaming config.
Solid checklist to bookmark for the next time someone says "the gateway is slow" without a trace to back it up.
Really appreciate this, you're pulling out exactly the parts I hoped would land.
The "gateway is slow" claim without a trace is such a common failure mode because it's plausible. The gateway is the new thing in the request path, so it's the easiest thing to blame, even when it's innocent. Once you force a same-conditions A/B and actually split p50/p95/p99, the story usually gets more interesting. Median might be nearly identical and the whole complaint is really a p99 problem caused by connection pooling or retry backoff, which is a totally different fix than "gateway bad."
The sync logging one still gets me. Audit and token accounting feel like they should be cheap, so nobody profiles them, and then you find a write to Postgres sitting directly in the response path adding 40-80ms per request. Moving it off-path is one of those fixes that feels almost too simple for how much it helps, which is probably why it gets skipped in favor of "optimizing" something more glamorous.
And yeah, provider TTFT vs client visible TTFT is the one I'd want people to internalize most. If you only ever look at the client number, every latency issue looks like a model problem. Splitting it forces you to ask where the gap actually is, buffering, chunk size, some overzealous middleware step, instead of just filing a complaint against the model provider.
Glad it's useful as a reference. Trace first, opinions second.
One more thing worth adding to the checklist: connection reuse gets overlooked in the same way sync logging does. If the gateway is opening a fresh TLS handshake per request instead of pooling connections to the provider, that shows up almost entirely in p99 too, and it looks identical to "gateway overhead" from the outside. You have to actually check keep-alive behavior and pool size, not just assume it's configured right because it usually is on the client side.
The other failure mode I'd flag is people trusting dashboards that already aggregate away the split. A lot of observability setups report one blended latency number by default, so the provider TTFT vs client TTFT distinction never even gets the chance to surface unless someone goes and instruments it manually. The tool you have shapes the question you're able to ask, and if the tool only gives you one number, that's the only story you'll ever tell yourself.
Good thread. This is the kind of checklist that should live next to the runbook, not in a doc nobody opens until something's already on fire.
Appreciate that, connection reuse is a great catch, and it's exactly the kind of thing that hides in plain sight because it's a client-side default nobody re-checks once it's set.
And the dashboard point might be the most important one in the whole thread. A blended number doesn't just fail to answer the TTFT question, it prevents the question from being asked at all. That's a worse problem than missing data, because it feels like you have visibility when you don't.
Good place to leave it. Trace first, opinions second, and don't trust a tool that only gives you one number to tell you where the time went.
What I found most interesting here is that the 400ms isn't really the story. The story is what happens when we give a complex outcome a single name.
“Gateway latency” sounds like one thing. Once it has a name, it becomes psychologically easy to treat it as one thing — measure it as one number, assign responsibility to one component, and start optimizing the category rather than investigating the processes hidden inside it.
But your breakdown shows that the same observed 400ms could emerge from completely different mechanisms: connection setup, authentication, sequential policy checks, synchronous logging, retries, buffering, or some combination of them.
That means two systems displaying exactly the same latency can have entirely different problems.
I think this points to a broader principle in debugging complex systems: measurement becomes much more useful when it follows mechanisms rather than labels.
The retry example illustrates this especially well. A request can succeed, the uptime dashboard can remain green, and yet the user experiences a slower system because an invisible recovery process occurred underneath the successful outcome. Similarly, provider TTFT and client-visible TTFT can tell two different stories about what the user actually experienced.
There is an important distinction hiding there between system reality and observed experience. The provider may have responded quickly. The gateway may technically be functioning correctly. The request may ultimately succeed. None of those facts guarantees that the interaction felt responsive to the person waiting on the other side.
I also liked your statement that the goal isn't to make the gateway “fast” in the abstract, but to identify the exact operation consuming the latency budget. That changes optimization from a vague pursuit of improvement into a problem of attribution.
Perhaps that is why mysterious performance problems often become less mysterious the moment we stop asking “What is slow?” and start asking “Where, exactly, is time being spent?”
Excellent breakdown. It is ostensibly an article about AI gateway latency, but the decomposition principle applies much more broadly to how we investigate complex systems.
Really appreciate this perspective. I think you captured the core idea perfectly: the moment we give a complex outcome a single label, it becomes tempting to optimize the label instead of understanding the mechanisms behind it.
The distinction between system reality and user experience is especially important. A request can succeed and every dashboard can look healthy, while retries, buffering, or other hidden steps still make the interaction feel slow.
That is also why breaking latency into specific operations is so useful. Once we know exactly where the time is being spent, optimization becomes much more actionable. And I agree that this principle goes far beyond AI gateways. It is a useful way to approach debugging almost any complex system.
The retry section stood out to me. A request can technically succeed while still delivering a poor user experience because of a hidden retry. Tracking retry count, delay, provider selection, and total attempts seems like one of those metrics that becomes extremely valuable once latency starts creeping up.
Absolutely. Hidden retries can quietly become a major latency multiplier, especially when everything looks healthy from the outside. Tracking retries and total request attempts separately made it much easier to spot where the extra time was actually going.
Exactly. Separating retries from total request attempts makes those hidden latency costs much easier to trace. Glad you found that useful, and thanks for sharing your perspective!
The provider-TTFT vs client-visible-TTFT split is the part that gets skipped most often — one gotcha to add: measuring client-visible TTFT from the browser's performance timeline instead of application code. A PerformanceObserver on 'resource' with the streaming fetch will show the actual first-byte gap; instrumenting inside the fetch callback can miss the middleware buffering entirely. Also worth logging retry reason with the stage it happened at — 'retried after policy check' vs 'retried after provider connect' usually points at two completely different fixes.
Great point. Measuring client-visible TTFT from the browser timeline gives a much more realistic picture of what the user actually experiences, especially when middleware or buffering is involved. I also really like the retry-stage logging idea. Knowing whether a retry happened after policy checks or provider connection can make debugging and fixing the right layer much easier.
The point about serial policy checks stood out to me most. Ten sequential 20ms rules quietly turning into an 80ms tax before the request even reaches the model is such an easy trap to fall into; each check looks harmless in isolation, and it's only when you actually trace the stages separately that the cumulative cost becomes obvious. It's a good reminder that "the gateway is slow" is rarely one thing; it's usually several small, reasonable-looking decisions stacking up.
The retry and TTFT sections you covered in the comments already nail the biggest offenders, so I'll add one more: I'd be curious whether you ran into policy checks that genuinely can't be parallelized because later rules depend on the output of earlier ones (like a classification result gating a routing decision). That dependency chain seems like the one case where the "just parallelize independent checks" advice gets harder to apply cleanly in practice.
Absolutely, that dependency point is important. In practice, some policy checks cannot be fully parallelized when one decision determines what gets evaluated next. That is where tracing the dependency chain becomes just as important as measuring individual latency. The goal is not to parallelize everything, but to identify which checks truly need to be sequential and minimize the critical path.
The distinction between provider TTFT and client-visible TTFT is a really useful point. It’s easy to blame the model when the actual delay is sitting in buffering, retries, TLS, or middleware. Breaking the request into stages makes debugging 400ms of “mystery latency” much more actionable.
Thanks! Exactly, that was the biggest takeaway for me too. Once I broke the request into individual stages, the “400ms latency” became much easier to reason about. TTFT especially can be misleading if you only look at the model/provider side.
The serial policy check issue was our worst offender for a while. We had four middleware steps each doing a separate auth or rate-limit call, and they ran sequentially by default because the framework made that the easy path. Collapsing them into a parallel fan-out cut our gateway overhead from around 280ms to about 60ms, and the fix took less time than the week we'd spent just figuring out which step was the culprit.
That’s a great example of how small architectural changes can have a huge impact. Going from 280ms to 60ms is a serious improvement, especially when the fix was simpler than finding the bottleneck. Parallelizing independent checks is definitely something worth looking for early when optimizing gateway performance.
Great breakdown. The point about separating provider TTFT from client-visible TTFT is something a lot of teams skip, and it's usually where the "gateway is slow" narrative falls apart once you actually measure it. I've seen the same thing happen with synchronous audit logging specifically: it looks fine in staging because volume is low, then becomes a real bottleneck under production traffic once the database writes start queueing up.
The serial vs parallel policy checks section is also worth calling out more. It's such an easy mistake to make since each individual check feels harmless in isolation, but nobody adds up the total until latency complaints start coming in.
Bookmarking this as a checklist for the next time someone says "the gateway added latency" without a trace to back it up.
Absolutely agree. The provider TTFT vs. client-visible TTFT distinction is especially important because otherwise it’s easy to blame the gateway for latency introduced by logging, policy checks, network hops, or downstream queues.
And yes, serial policy checks are one of those things that look negligible individually but become surprisingly expensive when they stack up. A proper end-to-end trace usually tells a very different story from the initial “the gateway is slow” assumption.
Appreciate the bookmark! 🙌
The "don't benchmark against a fake request" point is easy to miss but explains a lot of confusing benchmark results I've seen. A mock upstream tells you the proxy overhead, not what users actually feel, since real provider queueing and variability aren't there to hide behind.
The serial vs parallel policy checks example is a good one too. Ten checks at 20ms each isn't a rounding error, it's 80ms lost before the request even reaches the model, and it's the kind of thing that's invisible until someone actually times each stage separately.
Exactly. Benchmarking against a fake upstream can make the numbers look much cleaner than what users actually experience. The serial vs parallel checks are another great example because small delays add up quickly. Timing each stage separately really helps expose where the actual latency is coming from.
Great breakdown. The biggest takeaway for me is that “gateway latency” is often too broad a label to be useful. Breaking the request into connection setup, auth, policy checks, retries, logging, provider latency, and client-visible TTFT makes the missing time much easier to find. The point about measuring p95/p99 instead of just averages is especially important, tail latency is where these hidden bottlenecks really show up.
Absolutely agree. “Gateway latency” can hide several very different bottlenecks, so breaking it down by each stage makes troubleshooting much more actionable. And yes, p95/p99 is where the real story usually appears, averages can easily make a system look healthier than it actually is. Great point!