The right model for the right job

Why this conversation

Frontier models made agents take off, and their bills made every CIO look at what the agents cost. Stephen Watt, Vice President and Distinguished Engineer at Red Hat, works on the other option: open weight models, run by the company itself. In this conversation he explained why he thinks they are now good enough, what sovereignty means in the United States and in Europe, and how a team picks the right model for each job. The host recorded with him in the conference’s podcast room in San Jose on 22 October 2026, the first day of AGNTCon + MCPCon North America. The rest of that day is in AGNTCon + MCPCon North America 2026: day one.

Up to 70 percent

He grew up between South Africa and Japan, moved to the United States in December 2000, worked for IBM and HP for two years each and for a few startups, and has been at Red Hat for about 14 years, mostly in research and emerging technologies. Everything Red Hat does is open source, so he could say what excites him most: an internal pilot, on its way to a full production service, that deploys open weight models for Red Hat’s own use instead of frontier models. “It’s having radical cost savings, you know, up to as much as 70 percent.” It is rare, he said, for an emerging technology prototype to have that kind of impact on finances.

Good enough, and yours to post-train

The shift is recent. Within the last six months or so, in his view, open weight models have started performing well enough, at the level of Anthropic’s Opus 4.6, which he saw as the tipping point when agentic usage exploded. At that level, nothing really holds them back beyond the ability to deploy them successfully. Agency is a different matter with model weights: they are binaries, and few have the capital or the infrastructure to reproduce them. The agency comes from post-training them, taking a model and enriching it with your own data to create something specialised. In more targeted use cases, post-training can also raise the model’s performance in the evals.

Sovereignty, from the land up

Sovereignty, he said, is used differently in different places. In the United States it is often about the desire to use Western open weight models, not about infrastructure, because the country has few issues with land and power. In Europe it is much more about the infrastructure: running on European providers, with data hosted in Europe and operated by its own people. He drew the stack from the top: agents and inference, the platform, the accelerators, the data center, and below it power and land. Sovereignty is about self-determination, and Europe has a particular problem: the latest accelerators are liquid cooled while many of its data centers are air cooled, and power is short too. That could mean Europe “might actually have a slightly different stack”.

The agentic paradox

In the United States, he said, the question is reducing token expenditure. A CIO’s biggest concern today is cloud spend, with inference rolled into it, and that leads to what he calls the agentic paradox: “you sort of be crazy not to start with frontier models, but they become so expensive that if you’re exclusively using them, you’d sort of be crazy to stay on them”. The answer he sees taking shape is a hybrid. Agent gateways surface both the frontier models and the open weight models you host, so you can pick the right model for the right job and the right budget. Engineers are getting better at that choice: they run skills against several models and use evals against those skills to learn which model to use for what. To serve the open models at production scale there are llm-d and vLLM, and llm-d, the open source project for scale-out production inference, had just reached 1.0 that week. The host closed with: “let’s build the future open.”

When it comes out

The episode is edited and in the release order. Like every episode it goes out on YouTube first, then on Apple Podcasts, Spotify and Amazon Music.