Your AI Pilot Did Not Fail. It Was Never Measured.

Most AI pilots I have seen fail to reach production did not fail because of the technology. They were never measured in the terms production requires. Consequently, no one could say whether they were ready, and the organization moved on to the next demo. After decades of carrying intelligent systems from prototype to production, I have come to think that pilot purgatory is a measurement problem wearing a technology costume, and that agentic AI can make the consequences of that measurement gap far more expensive. A 2024 RAND report cites estimates placing the failure rate of AI projects above 80 percent, twice that of IT projects without AI.

Pilots are judged on the wrong numbers

Pilots are judged on the metrics that are easiest to capture: time saved, tasks automated, raw efficiency. They cannot, on their own, establish production readiness, because they miss what determines whether the organization keeps the system in use. Do the people who receive the output trust it enough to act on it without redoing the work? A system whose outputs everyone quietly redoes has not automated anything; it has added a step.

Track override rates: how often a person changes what the system produced. Track escalation rates and whether each escalation was warranted. Track sustained adoption over months rather than launch-week logins. Above all, track silent opt-outs: how often users bypass the AI system and return to the manual process without reporting the switch. These measures indicate reliance, not trust. Consequently, interpret them alongside independently assessed output quality and the reasons users give for intervening or opting out. A pilot with no exit criteria and no behavioral telemetry is not an experiment. It is a subscription.

You cannot read any of these figures without a baseline. Measure the manual process before replacing it; once workflows and user behavior shift, a reliable baseline is very hard to reconstruct. In my experience, this is the step most often skipped.

A conversational system I directed for banking customers’ account inquiries shows the difference in practice. The pilot, an interactive voice response and chat assistant serving a volunteer group of callers, posted excellent numbers. Production meant standing in front of the broader caller population, and there the pilot metrics proved insufficient. Instead, we measured how often callers pressed zero to reach a person, how often an agent reopened a case the assistant had closed, and how many callers phoned back within the hour about the same issue. Those three figures told us which intents the assistant handled well enough to keep and which it should hand off immediately. We expanded in stages as the figures justified, and the assistant came to handle millions of routine interactions a year. The pilot showed that it could answer questions. Production measures showed which inquiries callers were comfortable leaving with it.

Agentic AI raises the price of not measuring

A copilot that drafts suggestions keeps a person between the system and the consequence; an agent acting without approval at each step can remove that buffer. Production readiness for agents therefore turns on questions that a pilot must answer from the first day it touches a real system: what the agent is permitted to do, where its authority ends, and who can halt it. Concrete boundaries include explicit permission scopes, limits on spending and write operations, tamper-evident audit logs, and a structured escalation path that hands a human operator the context needed to take over.

Risk, policy, and the reversibility of each action set these boundaries before any measurement exists. The baseline adds calibration. It shows where the process actually runs so that you can draw limits where they are needed rather than where they are guessed. The override and escalation figures then show whether the limits are in the right place. Autonomy is not a switch. A sound progression starts the agent in suggestion mode, moves it to acting with human approval, and widens its independence only where the task’s risk permits and the measured record justifies.

Evidence, accumulated in stages

When I architected the Intelligent Filing Manager, an AI expert system for multi-jurisdictional compliance filings, the hard part was not the engine. It was persuading owners in each new environment to run it. The system moved from client-server implementations for insurance compliance to a web platform and then to other industries, and it has remained in production for decades. What carried it from one platform to the next was evidence accumulated over successive deployments: each release was tested and accepted by the users who would depend on it, and its performance record built confidence among those responsible for adopting it next.

Organizations that escape purgatory make two commitments before the pilot starts. They define what “working” means up front, in process and human terms as well as speed, with an owner, a baseline, and the evidence that would justify expanding or stopping. And they build for change, with drift monitoring on a fixed schedule and a named owner who survives the handover from the launch team.

None of this is as exciting as the demo. That is the point. Purgatory is not a place where projects are sent. It is where they stay when no one decides what production actually requires.

About the Author

Krishna C. Mukherjee is a pioneering Microsoft technologist, software architect, and technology executive whose work in artificial intelligence (AI) and enterprise systems has helped shape widely adopted computing platforms. His book, The Microsoft AI Insider’s Playbook: Your Guide to Cloud Strategy and Digital Transformation (Apress), is now available. Learn more at kcmukherjee.com.