The Agent Doesn't Want Anything (And That's the Whole Problem)

There's a particular look people get when they first watch an AI agent complete a multi-step task on its own. Book the flight, draft the email, reconcile the spreadsheet, all unprompted after the first instruction. It's the look of someone meeting a very capable stranger and not yet knowing whether to be delighted or afraid. The honest answer is: both, and for the same reason.

Here is the thing we keep getting wrong in the discourse. We ask whether the agent is "intelligent" as if that's the load-bearing question. It isn't. A thermostat is not intelligent and will still cook your house if you wire the logic backwards. The load-bearing question is about delegation - and delegation is a problem humanity has been failing to solve since the first king appointed the first minister.

When you hand a task to another human, an enormous amount of unspoken context travels with it. They know that "handle the client" doesn't mean "agree to everything." They know which rules are real and which are polite fictions. They know when to stop and ask. This tacit layer - the stuff nobody wrote down because everybody assumed it - is precisely what an agent lacks. Not intelligence. Judgement about what wasn't said.

So the frontier isn't making agents smarter. We're doing that almost by accident. The frontier is making the implicit explicit - surfacing the thousand quiet assumptions that let human delegation work, and encoding them before we hand over the keys. This is slow, unglamorous, deeply humbling work, because it forces you to articulate things you didn't know you believed.

I'll say the marginally heretical part out loud. The current panic about superintelligence is, in some ways, a flattering distraction. It lets us imagine our failure mode is a machine that's too clever. The likelier failure mode is far more embarrassing: a machine that does exactly what we literally said, having never been told what we actually meant. We won't be outsmarted. We'll be taken literally. That's not a robot uprising. That's every bad manager you've ever had, at scale, at the speed of light.