Most of us already juggle multiple LLMs without thinking twice. ChatGPT for one thing, Claude for another, Gemini if you want a second opinion. At the individual level, that feels natural.
Scale that behavior across hundreds of teams and business units, each making their own model choices, setting their own guardrails, tracking spend in their own way, and what feels productive becomes ungovernable. Enterprise AI didn’t arrive as a single coordinated decision. It arrived as thousands of small ones, made independently across every team and business unit, each with its own access patterns, its own guardrails, and its own way of tracking spend.
What you’re left with is an AI estate that nobody fully owns, where costs accumulate without accountability and spotty policy coverage.
It’s at this juncture that AI initiatives stall. Not because adoption is slow, on the contrary, the volume of AI-driven interactions across enterprise systems is growing. But it’s at a pace most governance teams weren’t built to handle and growth without infrastructure is sprawl, not scale. Without intelligent controls, every request defaults to the most expensive model available, regardless of whether the task requires it. By the time a CIO asks for a unified view of AI costs, the answer requires a manual investigation across disconnected systems with no audit trail.
Scaling AI responsibly requires the same things that scaling APIs did: a governed access layer, consistent policy enforcement, and clear accountability for usage and cost. That’s what AI Gateway’s LLM capabilities deliver, and it’s already part of the platform MuleSoft customers run today.
Introducing AI Gateway from MuleSoft
We’re announcing the general availability of AI Gateway’s LLM governancebringing intelligent routing, unified multi-provider access, and cost management and governance for Large Language Models (LLMs) into Anypoint Platform. Combined with our existing support for MCP and A2A agent governance, MuleSoft now provides a single control plane across the full spectrum of enterprise AI activity.
Our AI Gateway capabilities give enterprises a single governed access point for every LLM provider – whether that’s OpenAI, Azure, Google Gemini, Amazon Bedrock, or any combination. Three capabilities sit at its core: intelligent routing, unified access across providers, and token cost management. Let’s look at all three in greater detail.
Intelligent routing
Every model call has a cost, and most tasks don’t need your best model to get the job done. Intelligent routing enables platform teams to define topics and utterances, then automatically route each request to the most appropriate model based on prompt content. Finance queries route to models optimized for financial tasks, coding requests go where technical capability is strongest and simple queries hit cost-efficient models automatically.
The result is that enterprises stop defaulting every request to the most expensive option and start making intelligent routing decisions at scale, without engineering effort across every use case.
Unified access across providers
Managing credentials, formats, and model provider relationships across every team is exactly the kind of overhead that slows AI adoption down and creates security blind spots. Developers end up managing provider keys directly and applications get tightly coupled to specific model formats
With AI Gateway, LLM traffic is exposed via a single logical endpoint, configured by platform teams for approved models and providers only. Developers authenticate once and never manage provider keys directly, and requests are automatically translated to the format of the underlying provider, with normalized responses returned in a standardized format.
This means that new models can be onboarded without application changes and, when a request doesn’t semantically match any configured route, AI Gateway automatically directs it to a fallback model rather than returning an error.
Teams get predictable behavior across the full range of real-world prompts, including the ones that don’t fit the pattern they planned for. Together, these capabilities provide a much needed degree of flexibility when it comes to both the operational implementation of AI, and your broader strategy when optimizing for model costs.
Token cost management
By the time most teams realize they’ve overspent on AI, the bill has already landed. AI Gateway tracks token consumption at the level of granularity that FinOps and platform teams actually need, by LLM proxy (a lightweight intermediary server that sits between applications and AI models), application and business group, and at daily and monthly levels. Token budgets and rate limits are enforced at the gateway before overages occur, not discovered after the fact in a cloud bill.
Governance at the gateway layer
Every enterprise AI deployment eventually runs into the same problem: governance gets implemented differently by every team that needs it. Prompt injection defenses live in one application, content safety rules in another, and compliance requirements somewhere else entirely. The result is a patchwork that no one fully controls.
Applying governance at the gateway resolves this. Prompt injection protection, content safety filtering, and PII detection are enforced as policies on every request, regardless of which team built the application behind it. When a new use case is deployed, it inherits the same controls automatically — no per-team configuration, no security review for each new integration, no divergence between environments.
For organizations scaling AI across multiple business groups, this is the difference between governance that holds as usage grows and one that quietly breaks down as it does.
The future of AI governance
The enterprises successfully scaling AI today are moving fast not because of the inherent accelerating factor of AI; it’s because they’ve taken the time to layer a foundation of operational governance that’s now paying dividends at the strategic level.
When every LLM interaction is governed, every cost is accountable, and every policy is consistent, the operational risk that typically forces organizations to slow down or halt AI rollouts stops accumulating. That foundation is exactly what AI Gateway is built to provide, and for most MuleSoft customers, it’s already within reach.
For customers on Flex Gateway, these LLM capabilities are available today at no additional cost for Platinum, Titanium, Unlimited, and Integration Advanced tiers. The policies, identity configurations, and observability tooling already in place extend directly to LLM traffic – no new vendor, no parallel governance stack, no additional contract. We’ll continue to extend AI Gateway, agent governance, and MCP capabilities as the demands of enterprise AI operations evolve.
To learn more about the state of agentic transformation and the future of AI, download this year’s Connectivity Benchmark Report and subscribe to our newsletter, Technically Speaking.
