A Week In AI: Issue #17
2,000 Customer Calls
Quick bit of news first: my talk got accepted for KubeCon NA 2026 in Salt Lake City. The title is “2,000+ Customer Calls About AI In Production: Here’s What You Need To Know.”
That number is real, and it’s the reason this newsletter exists in the form it does. Almost everything I write here comes out of those conversations. The patterns I flag as “I keep hearing this” are patterns from actual calls with actual teams implementing Agents, AI Gateways, registries, sandboxing, MCP, and AI security in production. Calls where someone says “we tried this and it broke” or “our security team won’t sign off until we can answer this question.”
If you’re going to be in Salt Lake City, come find me, and if you’ve been reading this newsletter for a while, the talk is going to feel like a greatest-hits version of the last four months of issues.
Substrate Gateways and Hardware Isolation
Substrate is maturing in a specific direction: it’s becoming configurable at every boundary.
Three things landed that all point the same way.
What runs inside. The AgentHarness and SandboxAgent objects are now the interface. SandboxAgent is what you’d expect: a standard Agent, running in a Substrate sandbox. AgentHarness is the interesting one. It lets you run popular harnesses (Hermes, OpenClaw) inside the sandbox rather than being locked to one execution model.
That’s a meaningful design choice. It’s providing the isolation boundary and letting you drop whatever harness fits your workflow inside it. This lines up exactly with the “low-opinion about the application layer, high-opinion about primitives” framing from Issue #14.
What fronts it. Substrate’s router (the ingress path that gets a client to an Actor) is now pluggable. You can point it at an gateway as an example:
./hack/install-ate.sh --atenet-router=agentgateway --deploy-ate-system
And the router choice matters more than it might look. Whatever fronts your Substrate deployment is the thing that sees every request going into every Actor. That’s the natural enforcement point for identity, rate limiting, policy, and observability. Making the router pluggable means you can bring your existing Gateway rather than running two separate enforcement layers with two separate policy models.
What sits underneath. Substrate with MicroVM isolation == hardware isolation. gVisor was already good, but MicroVMs are a harder boundary, with real hardware-level separation rather than syscall interception. Different tradeoff on the isolation-versus-overhead curve, and having both available means you can pick based on your actual threat model rather than taking whatever the platform gives you.
Substrate is settling into its role as the isolation and lifecycle layer, with clean seams in every direction: pluggable harness above, pluggable router in front, pluggable isolation below. That’s what infrastructure looks like when it’s designed to be built on rather than used directly.
Golden Evals From Traces
The most interesting idea I ran into this week is a small inversion in how you build eval sets.
The standard approach to Agent evals is “write a golden set by hand”. Define the inputs, define what good output looks like, run the Agent against it, compare. That works, and it’s what most teams do when they start taking evals seriously (Issue #16).
The problem is that writing golden sets by hand is slow, and the sets you write tend to reflect what you think the hard cases are rather than what the hard cases actually are in production.
The inversion: build the golden set from traces that already went well.
You have production traces, and some of those traces are examples of the Agent doing exactly what it should have done, with exactly the output you wanted. Those traces already contain the input, the tool calls, the reasoning path, and the output. That’s a golden eval, and you didn’t have to write it. You just have to recognize it and promote it.
This is better than hand-writing in two ways. The inputs are real production inputs, not synthetic ones you imagined. The “correct” output is one your Agent actually produced under real conditions, which means it’s achievable rather than aspirational.
The workflow becomes: run in production, observe traces, flag the ones that went well, promote them to the golden set, then use that set as your regression suite whenever you change the harness, swap the Model, or update tool definitions. Your eval coverage grows organically from real usage instead of from someone’s guess about what matters.
This is also a nice argument for taking observability seriously (Issue #14). If your traces aren’t good enough to reconstruct exactly what the Agent did, you can’t harvest golden evals from them. Good tracing pays for itself twice: once when you’re debugging, and again when you’re building your eval suite.
What You Run Locally Is The Security Insertion Point
Just about everyone in engineering is using a Harness like Codex, Claude Code, or opencode locally. People that aren’t even in tech are using them. I have friends and family members saying things like “look at what this ChatGPT thing was able to do for me”. AI is definitely something that the majority of people are using in day-to-day life. It’s also embedded into tooling that already existed (e.g - in the Microsoft Office suite for every time that uses Word, Excel, PowerPoint, etc.).
Engineering teams worry about security at the Agent level when it comes to running AI on servers, but right now, I don’t think that’s the main level where security needs to be implemented (although it’s obviously incredibly important). The main level for security is right on someone’s laptop/desktop. Securing Harnesses running individually on laptops/desktops.
Luckily, this is where something like agentgateway comes into play. You can plug agentgateway into Codex, Claude Code, Claude Desktop, codex, and whatever other harness you’d like. You need to secure the line of communication between your Harness and an LLM and/or MCP server. To do that, you need an AI Gateway sitting in the middle.
More next week.
Michael