Envoy AI Gateway reaches 1.0 for production AI use

Envoy AI Gateway has reached version 1.0, marking the first open-source AI gateway built on the Cloud Native Computing Foundation's Envoy Gateway project to reach production maturity.

The project was developed with contributions from Tetrate, Bloomberg, Nutanix and the wider Envoy community. It is designed to handle AI traffic with routing, governance and observability features built on Envoy, the software widely used in internet-scale infrastructure.

Bloomberg initiated the effort through the Envoy Gateway community after looking for a way to route generative AI workloads with the same reliability and governance it expected from Envoy. The software's first release came in early 2025, and maintainers say 16 months of collaboration have produced a stable codebase ready for broader use.

Production use has become central to the project's case for adoption. Bloomberg is already running Envoy AI Gateway in production, while Nutanix is taking the software into production and integrating it into Nutanix Agent Gateway and Nutanix Enterprise AI.

Industry backing

The release reflects a broader effort by infrastructure suppliers and large technology users to create common tools for managing AI services, especially as companies work with several model providers at once. Envoy AI Gateway aims to provide a single interface for traffic sent to providers including OpenAI, Anthropic, Google Gemini, Azure OpenAI and AWS Bedrock, as well as a wider range of OpenAI-compatible services.

The software also adds controls designed specifically for AI workloads, including token-aware rate limiting and quotas, centralised credential management across providers, and tracing based on OpenTelemetry and OpenInference standards.

Another part of the release is support for the Model Context Protocol, a developing standard intended to help AI agents connect with tools and services. The gateway includes MCP traffic routing, tool filtering rules, authentication measures and per-tool observability.

Envoy has long been a fixture in large-scale traffic management since its creation at Lyft in 2015. Maintainers argue the AI gateway extends an established internet traffic layer into AI inference and agent-based workloads, where cost controls, provider switching and governance have become more pressing for enterprise users.

Project backers say the same public codebase is being used in production by Bloomberg and Tetrate. That point may matter for organisations weighing open-source infrastructure against proprietary AI gateways sold by cloud providers and software vendors.

Dan Sun, Envoy AI Gateway and KServe Co-founder and Maintainer, described the release as part of a push towards common practices for serving AI workloads in large organisations.

"We see the Envoy AI Gateway as a key element toward standardising how enterprises securely and reliably serve AI workloads," said Sun. "Bloomberg engineers have made hundreds of contributions to this project - in the spirit of our firm's commitment to scalable, open source AI infrastructure that brings vendor neutrality, consistency, observability and control to AI inference at scale."

Production focus

Tetrate said the move to version 1.0 reflects software tested on real workloads rather than a purely experimental project. It positions itself as a major contributor to Envoy and the AI gateway effort.

"Envoy has become the foundational layer for internet traffic at the world's most demanding organisations. With v1.0, Envoy AI Gateway brings that same trust to AI workloads," said Varun Talwar, Co-founder and Chief Technology Officer, Tetrate. "The code in the public repo is the same code running in production at Bloomberg and Tetrate. That level of transparency is rare in open source, and it's what enterprises need as they scale AI."

Nutanix's involvement highlights the commercial interest in open-source software that can sit in front of multiple AI services. As businesses seek to avoid dependence on a single model provider, gateway software has become a focal point for managing traffic, policy and spending controls across different systems.

"Nutanix is proud to be a maintainer and an active contributor in the Envoy AI Gateway community, helping bring the same transparency and production-grade reliability that powers enterprise Internet traffic to the next wave of AI workloads," said Debo Dutta, Chief AI Officer, Nutanix. "We are using the project's capabilities to bring transparent, multiprovider flexibility and production-ready AI infrastructure to our customers."

The project has also drawn users beyond its core maintainers. LY Corporation says it uses Envoy AI Gateway to manage multi-tenant, self-hosted large language model traffic.

"At LY Corporation, we utilise the Envoy AI Gateway to manage our multi-tenant, self-hosted LLM traffic," said Shingo Omura, Principal Architect of AI Infrastructure, LY Corporation. "It provides a unified API for flexible routing, token-based rate limiting, authentication and authorisation, monitoring and extensibility, aligned with open standards like the Kubernetes Gateway API Inference Extension. This allows us to maximise operational efficiency of our LLM platform. We believe version 1.0 is a major milestone for production-grade AI infrastructure engineering."