Your agent can reach its tools. Identity is scoped. Governance is in place. None of that matters if the model behind the agent cannot reliably call those tools.

This plays out repeatedly in practice. A team builds an agent on LangChain or CrewAI. It works in development. It passes staging. Then someone asks the obvious question: where is inference actually running? The answer, almost every time, is a third-party hosted API—OpenAI, Anthropic, or Google. Not because the team prefers it, but because open-weight models running on their own infrastructure weren’t reliable enough for agentic workloads. The framework handles orchestration. The hosted API handles inference. Neither is under the organization’s control.

That creates a sovereignty dilemma. The same enterprise that’s committed to data sovereignty—because compliance or regulations require it—is routing every prompt, every tool call, and every reasoning step through someone else’s datacenter.

In a previous article, three failures that hit a single AI agent deployment overnight were examined: 43 duplicate tickets, $4,000 charged to the wrong account, and a hallucinated refund policy that led to a $280 return the company had to honor. Those failures happened because production infrastructure was absent—the gap between a working agent in development and a production-ready deployment is an infrastructure problem. If the inference layer also sits outside your control, that gap isn’t operational, it’s jurisdictional.

BYOA—bring your own agent—is Red Hat AI’s approach. The platform provides production infrastructure for any agent framework without code changes. But that story starts to crack if the inference layer undermines the very sovereignty the platform protects. This article covers how Red Hat helps close that crack.

Chat Is Forgiving. Agents Are Not.

Most enterprises started with chat. A user asks a question. The model answers. Single turn. Stateless. If the response is slightly off, the user rephrases and tries again. Graceful degradation is built into the interaction pattern.

Agentic workloads break that assumption in every dimension. An agent making a multiturn decision—planning steps, calling tools, evaluating results, revising its approach—needs the model to format tool calls in exactly the schema the tool expects. One malformed JSON object in a 6-step tool call sequence does not produce a vaguely wrong answer. It breaks the entire workflow. The tool call fails. The agent retries (creating 43 duplicate tickets, if you recall the 6 AM incident). Or worse, the agent silently produces incorrect output—like telling a customer the refund window is 90 days when the actual policy is 30.

Think of it this way: chat is a conversation. Agents are contracts. A conversation tolerates ambiguity. A contract does not. When the model generates a tool call, it isn’t suggesting an action—it’s executing one. The precision threshold is categorically different.

This is why most enterprises default to hosted frontier models for agentic workloads. The reliability gap between the best hosted models and open-weight models running on-premise has been wide enough that the sovereignty compromise felt necessary. For a decision maker, that compromise means every agent interaction generates data in a jurisdiction you don’t control. For a developer, it means a deployment architecture that can’t survive an API deprecation notice.

What vLLM Changes

That reliability gap isn’t permanent. It’s an engineering problem—and Red Hat is systematically closing it.

vLLM (Red Hat AI’s self-hosted inference engine) is generally available on Red Hat OpenShift AI. It’s the foundation of the agentic stack. If inference doesn’t perform for agentic workloads, everything above it—identity, governance, connectivity—loses its foundation.

The unreliability of open-weight models for tool calling has had three specific causes, and Red Hat is addressing each one.

Smart defaults solve the configuration problem. Running an open-weight model for agentic workloads used to require manual tuning—parser selection, temperature settings, tool-calling format. Get any of those wrong and tool calls fail in ways that are difficult to diagnose. vLLM ships opinionated configurations for agentic workloads so tool calling works correctly on first deployment. No manual tuning required. A developer points their agent at the vLLM endpoint and gets reliable tool-calling behavior without becoming an inference configuration specialist.

Multiturn hardening solves the compounding error problem. In a multiturn agentic loop—plan, call, evaluate, revise—small token-handling inconsistencies across turns accumulate. By the 4th or 5th turn, the model is operating on degraded context. vLLM’s multiturn hardening reduces those accumulated errors across complex agent workflows. For decision makers, this is the difference between an agent that works in a demo (1–2 tool calls) and one that works in production (dozens of calls across minutes or hours).

Model stratification (matching model capability to task complexity) solves the cost and latency problem. A capable orchestrator model handles planning and coordination. Smaller specialist models handle execution tasks—tool calling, data retrieval, code generation. The key insight for decision makers: you aren’t choosing between one expensive model and one cheap model. You are running the right model for each task in the workflow. All on-premise. All under your control.

vLLM’s benchmark results track against hosted frontier models as each phase of hardening ships. The goal isn’t theoretical parity—it’s the specific reliability threshold that enterprise agentic workloads require.

Writing Code Beats Writing JSON

Smart defaults and multiturn hardening improve how models generate tool calls. But there’s a deeper fix—one that changes the format of tool calls entirely.

JSON tool calls require the model to generate a precisely formatted schema for every tool invocation. The format is unforgiving. A missing bracket, a wrong type, or an extra field breaks the call. When tool schemas change—and they do—the model must generate the updated structure perfectly. There’s no partial credit.

Code-as-Actions—an approach where agents write Python code to invoke tools rather than generating JSON tool-call schemas—sidesteps the problem entirely. Instead of asking the model to produce a rigid JSON structure, the agent writes Python code to call the tool. This plays to what open-weight models are already strong at—code generation—instead of forcing them into a format where small deviations are fatal.

Cloudflare, Anthropic, and Pydantic independently converged on Code-as-Actions. When three organizations with different code bases, different model choices, and different use cases arrive at the same architectural conclusion, that’s a signal worth taking seriously. The convergence also validates the approach for decision makers: this isn’t a Red Hat experiment. It’s an industry direction.

For developers, Code-as-Actions eliminates parser maintenance. No custom code to handle different tool schemas. No brittleness when schemas change. For the sovereignty story, Code-as-Actions is how open-weight models close the remaining reliability gap with hosted frontier APIs—without requiring organizations to route sensitive data off-premise.

Owning the API Contract, Not Just the Model Weights

Reliable inference and reliable tool calling solve the performance problem. But there’s another lock-in risk that most teams don’t see until they are committed: the API contract itself.

An agent built against the OpenAI Responses API depends on OpenAI’s schema, behavior, and availability. The developer wrote the agent to speak OpenAI’s protocol. If OpenAI changes pricing, deprecates an endpoint, or alters a behavior, the organization absorbs that change—on OpenAI’s timeline, not its own. Migrating to a different back end means rewriting the agent’s integration layer. For organizations running dozens of agents, that rewrite isn’t a weekend project.

Open Responses—the Open GenAI Stack (OGX), Red Hat AI’s server-side agentic loop formerly known as Llama Stack—implements the OpenAI Responses API specification and addresses this directly. Agents built against the OpenAI Responses API can point at an OGX endpoint with minimal code changes. Same API contract. Different back end. The model running behind the endpoint can be Llama, Granite, or any other open-weight model running on your infrastructure.

The portability principle extends across API surfaces. vLLM already provides conformant OpenAI Chat Completions endpoints. Open Responses adds state management and tool-calling in the Responses API format. Anthropic Messages API and Google Interactions API front-end support are on the roadmap—organizations that use Claude Agents or Google ADK will be able to point their agents at self-hosted infrastructure without rewriting agent code.

For teams that want managed infrastructure with built-in rate limiting and policy enforcement, Models-as-a-Service (MaaS, currently in technology preview) provides a third inference path—a managed model-serving platform on OpenShift AI with Gateway API and Connectivity Link integration. Together, vLLM (direct on-cluster), OGX (agentic loop), and MaaS (managed) give organizations a spectrum of control from fully self-managed to platform-managed, all behind OpenAI-compatible APIs. For a hands-on comparison of all three paths, see Deploying agents with Red Hat AI: The curious case of OpenClaw on Red Hat Developer.

It’s worth being precise about what this is and what it isn’t. OGX implements wire-compatible APIs from OpenAI, Anthropic, and Google—proprietary vendors, not open standards bodies. This isn’t sovereignty over the API specification itself. It’s sovereignty over the back end: your infrastructure, your models, your data. The API contract becomes portable. The organization controls when and whether to absorb upstream changes. For a decision maker evaluating vendor risk, that distinction—controlling the implementation without controlling the standard—is the practical form of technology sovereignty that matters.

What Sovereignty Actually Looks Like at the Inference Boundary

Consider what the hallucinated refund policy from the 6 AM incident looks like through the inference lens. The agent fabricated a 90-day refund window because nothing in the inference layer constrained it to the actual 30-day policy. That failure had two components: a governance gap (the agent wasn’t grounded in the correct policy data) and an inference gap (the model generated a confident, plausible, wrong answer).

Closing the inference gap with self-hosted infrastructure doesn’t automatically close the governance gap—but it makes closing the governance gap possible. When inference runs on your infrastructure, you can instrument it. You can log every prompt, every tool call, every model output. You can apply guardrails at the inference layer. You can audit what the model did and why. When inference runs through a third-party API, that instrumentation is limited to what the API exposes—which is typically less than what a production governance posture requires.

Sovereignty at the inference boundary isn’t about distrust of hosted providers. It’s about operational completeness. An enterprise AI deployment that routes inference off-premise has a gap in its observability, its compliance posture, and its ability to respond when something goes wrong. Self-hosted inference, hardened for agentic workloads, is what closes that gap and makes the rest of the production infrastructure stack coherent.