Why most agent pilots stall before production

Agent pilots rarely die because the model was not clever enough. They die because nobody defined what the agent is allowed to do, what happens when it is wrong, and how anyone would know. Those three gaps kill more projects than accuracy ever has.

The demo problem

A demo runs on a handful of cases someone picked. Production runs on the cases nobody thought about: the malformed record, the customer who replies in two languages, the invoice with a scanned signature over the total. A demo answers “can it do this?” Production answers “what does it do when it cannot?”

The move from one to the other is not a bigger model. It is a specification of failure.

Three questions to settle before writing code

• What is the agent allowed to touch? Read-only, write-with-approval, or write-freely — per system, not in general. Most useful agents are read-mostly with two or three narrow write permissions.
• What does a wrong answer cost? A mis-tagged support ticket costs a minute. A mis-posted journal entry costs an afternoon and a conversation with finance. Cost of error sets how much autonomy is sensible, and it varies wildly inside the same workflow.
• How would you find out it went wrong? If the answer involves a customer complaining, the logging is not finished.

Evaluation is the unglamorous half

An evaluation set is a boring spreadsheet of inputs and the outcomes you expect. Thirty rows drawn from real history beats a thousand synthetic ones. It gives you a number before launch, a regression test when you change the prompt, and a way to answer “is it getting better or are we guessing?”

Without one, every prompt change is a vibe. With one, you can refuse to ship until the number clears a line you set in advance.

The fallback is a feature

Every agent we ship keeps the old workflow intact underneath. If confidence drops, if a tool call fails, if the queue looks strange, work routes back to a person and the agent stops. This is not a lack of ambition. It is the thing that lets a business switch autonomy on at all, because it makes the downside bounded.

What a pilot should produce

A good pilot ends with an artefact, not an impression: a measured accuracy figure on real cases, a cost-per-run, an action log you can read, a documented scope of authority, and an honest recommendation — including the recommendation not to proceed. About a third of the scoping calls we run end with a smaller build than the client expected, and a few end with no agent at all.

That is a successful pilot too. It cost weeks rather than quarters to find out.