Why AI pilots fail: 5 signs yours won't reach production
Five signs an AI pilot is headed nowhere, from a missing baseline to an unknown cost per case, and what to do about each one.
Why do AI pilots fail? An AI pilot is cheap to start and easy to extend. Proving that it works is harder, and most pilots never reach production.
According to Fortune’s coverage of the MIT NANDA report The GenAI Divide: State of AI in Business 2025, about 5% of AI pilot programs achieve rapid revenue acceleration. The vast majority stall, with little to no measurable impact on P&L. Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value or inadequate risk controls.
All three of those causes can be spotted while a pilot is still small. Here are five signs to look for, and what to do about each one.
1. There is no baseline number to beat
The pilot started with a demo and no number. Nobody wrote down how long the task takes today, what it costs or how often it goes wrong. Every result then looks promising, and none of them settles anything.
What to do instead. Before the pilot runs, record the task’s monthly volume, minutes per case, cost per case and error rate as it works today. Then write down the number the pilot must reach to go to production, and the date you will check it.
2. There is no business owner, only IT or a vendor
If the pilot belongs to IT or to the vendor running it, nobody whose budget depends on the result is waiting for it. Pilots like that get extended, then dropped without a decision.
What to do instead. Name one owner from the business, the person whose team does the work today. They own the baseline and the target, and they make the go or no-go call. IT and the vendor build it and support it.
3. Nobody knows the cost per case
Many pilots run on a fixed fee or free credits, so the real cost stays hidden. In production, cost grows with every case. It includes model fees, the people who review the output, fixes when the system gets a case wrong and the integration that keeps it running. That is where escalating costs come from.
What to do instead. Estimate the cost per case at full volume before you approve the next phase. Ask the vendor for a quote at production volume. Put the cost per case next to what each case is worth. If the math fails at full volume, stop now, while stopping is cheap.
4. The input was structured all along
Some pilots pay a model to read data that already sits in fields, such as order numbers, dates, amounts, status codes or form answers. A model can do that, at a cost on every case. A rule does it for close to nothing and gives the same answer every time.
What to do instead. Score the use case on the six questions of our “AI or not?” scorecard. How often does it happen? What is each case worth? How messy is the input? Does it need judgment? What does a mistake cost? What does the simple version cost? If the verdict is Rule or script, build the rule and keep AI for the cases the rule cannot read.
5. Mistakes are not measured on a test set
The team judges the pilot by trying it and liking the answers. Nobody has counted how often it gets a case wrong, or what a wrong case costs. A risk nobody measures is a risk nobody controls.
What to do instead. Build a test set of a few hundred real past cases with known right answers, checked by the business owner. Measure the error rate before launch and after every change to the model, the instructions or the data. Agree in advance on the error rate you will accept and on who reviews the cases the system is unsure about.
What to do with a pilot that shows these signs
One sign can usually be fixed. Two or more mean the pilot needs a decision, and there are three options:
- Fix it. Set the baseline, the owner, the cost per case and the test set, then run it again with a deadline.
- Rebuild it lean. If a rule or a script can do most of the work, build that and keep AI for what is left.
- Stop it. If the cost per case is higher than what each case is worth, closing the pilot is the saving.
A pilot with a number to beat, an owner, a known cost per case and a measured error rate either goes to production or stops cleanly. Either outcome beats another extension.
Score your pilot’s use case with the free “AI or not?” tool.
If you have a pilot waiting on a decision, book a diagnostic.