Speaker
Description
Modern platform team eventually on any odd day gets assigned to ticket "We need to put AI features and evolved with AI age behind the gateway". The instinct is to reuse the existing solutions in the dream of "Why re-invent wheel?" they end often end up using traditional REST based api gateways which are not capable for LLMs. We rate limit by requests per second, route by HTTP path and trust our existing dashboards to work. That instinct is where trouble starts, a LLM request is not our REST/gRPC request or any functional call. One call might cost a fraction of cents on any particular model provider, while the next might cost 10x more. Request per seconds limits stop protecting the budget moment token usage comes into picture and becomes the real unit of business costs. Streaming responses (SSE) do not fir the request-response timing our observability was build around. The result is we end up getting surprise bills, rate limits that throttle the wrong thing, and a feature pinned to one provider with no failover.
This talk reframes the problem, it walks through concrete difference between REST traffic and LLM traffic: request counts vs token counts, static cost vs variable cost, single vendor locked provider vs multi-model routing, and header based policy vs prompt-aware policies. From there it introduces "envoy AI gateway" as an extension of Envoy gateway and gateway api model many teams already run, not as a replacement for it.
A brief overview at token aware rate limiting, an OpenAI compatible interface across Anthropic, Bedrock, Azure and few local models, provider failovers, and MCP supports and future A2A protocol support pipelined.
Attendees will leave with a reusable mental model for reasoning about AI traffic, a clear sense of when an AI gateway becomes necessary rather than optional. The goal is understanding first: know what breaks, why it breaks? and know how to evaluate any AI gateway against your requirements, lastly we will look at latest version release of envoy AI gateway and project milestones along with encouraging new contributors to come forward to contribute.
Session author's bio
Atharva is currently working as Open Source Developer and Maintainer at one of the projects of CNCF OpenEverest.
Worked as Google Summer of Code 2026 Mentee at OpenScienceLabs.
Working in Open Source for over year contributing in various CNCF Projects like Harbor, Kthena, OpenEverest
Loves working with golang, AI inferencing and cloud native ecosystems.
Any other info we should know?
Benefits to the audience: Most platform teams still treat AI traffic as another HTTP API. That assumption worked when every request represented roughly equal cost and latency. Modern LLM workloads break those assumptions. Requests vary widely in token usage, provider capabilities, streaming behavior, and pricing models, yet many production platforms continue to apply REST-era gateway patterns. The community is confused because the tools look familiar and the wrong assumptions stay invisible until a bill or an outage exposes them. The missing piece is shared vocabulary: teams lack agreed terms for token-aware policy, provider abstraction, and AI-specific failover, which makes design reviews slow and inconsistent.
This talk gives attendees a practical framework for thinking about AI gateways as a new platform primitive rather than an extension of traditional API gateways. They will leave understanding where REST assumptions stop working, when AI-specific gateway capabilities become necessary, how token-aware policies change platform design, and how to evaluate different AI gateway implementations for their own organizations. The immediate wins are better design reviews, better cost reasoning, and better production readiness, because the room now shares a mental model instead of debating from different premises. The model is reusable well beyond any single project: it is a lens for evaluating AI infrastructure in general.
Atharva approaches this topic from hands-on experience exploring Envoy AI Gateway, Kubernetes-native AI infrastructure, and open source cloud native projects. Rather than presenting the technology in isolation, the session connects familiar API gateway concepts with the operational realities of deploying and governing LLM workloads, helping attendees bridge the gap between existing platform practices and emerging AI infrastructure.
| Level of Difficulty | Intermediate |
|---|---|
| In Person Attendance | Remote |
| Please confirm that there are included headshots of all speakers in their profiles | Yes |
| Social Media | https://x.com/AtharvaXDevs |
| Agree to Privacy Policy and Notice | I agree |