Why AI pilots stall, and how to get them to production

Almost every company now has an AI pilot. Very few have one that pays for itself. MIT's 2025 study of enterprise generative AI found that only about 5% of pilots show measurable profit impact, and Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027, citing cost, unclear business value and weak risk controls. Those numbers are not a verdict on the models. The same models are in production at companies that did get value.

We spent several weeks reading the research and comparing it with what we see when we are asked to rescue a pilot. The pattern is consistent. A demo works on twenty hand-picked examples. Then someone asks whether it works on the next ten thousand, whether last week's model upgrade made it worse, what it did with a customer's record, and how it writes back to the ERP. Nobody can answer, and the pilot sits in a holding pattern until the budget review cancels it.

This article sets out the four blockers we see most often, how to tell which one is holding your pilot back, and what to build first. It also explains where our own accelerators fit, and where they do not.

A pilot has to pass four gates on its way to production, and each gate is owned by a different group of people. Engineering and the business need proof that answers are good enough. Data owners need to know the system reads the right data with the right permissions. Risk, security and audit need evidence of what the system did. Operations need it to work inside the systems where the work actually happens. A pilot can be excellent on one axis and still be stuck at another gate for months.

The MIT researchers put the largest share of failures down to what they call a "learning gap": tools that do not retain feedback, adapt to the workflow or integrate deeply, rather than weak models. They also found that buying from specialised vendors and building through partnerships succeeded about 67% of the time, while internal builds succeeded roughly a third as often. Menlo Ventures reported the same direction from the buyer's side: in its 2025 survey, 76% of enterprise AI use cases were purchased rather than built, up from roughly half the year before. Both are surveys with their own biases, but they point at the same conclusion. Teams that start with proven building blocks for the unglamorous parts reach production more often than teams that rebuild those parts for every pilot.

No way to prove the answers are good

This is the gate most pilots hit first. A demo proves that something is possible. Production needs proof that it is reliable, measured on representative inputs, with a number that someone outside the project team accepts. Without that number every conversation about go-live becomes an argument about anecdotes.

Take an illustrative case. Imagine a distributor piloting an agent that extracts header fields and line items from supplier invoices. The demo uses fifty clean PDFs and looks perfect. A useful evaluation set looks different: a few hundred invoices sampled from real traffic, stratified by supplier, format and language, including scanned copies, credit notes and the awkward ones with line items split across pages. Each is labelled once by an accounts payable clerk. The metric is field-level accuracy, reported separately for the fields that matter financially (totals, tax, supplier bank details) and the ones that do not.

That set does three jobs. It gives the business a number to sign off against. It becomes a regression test that runs whenever the prompt, the model or the extraction code changes, so a model upgrade that improves average accuracy but quietly breaks credit notes is caught before it ships. And it tells you where to put a human in the loop: if tax amounts are right 99.4% of the time on clean PDFs but 91% on scans, scanned invoices go to review and the rest do not. Those percentages are an example, not a benchmark. The point is that you can only make the routing decision once you measure by segment.

Model-graded scoring helps when outputs are free text, but only once you have checked the grader against people. We sample a few hundred graded outputs, have a domain expert grade them blind, and measure agreement before trusting the automated score at scale. A grader that disagrees with your experts a third of the time is not a measurement anyone should sign off against.

Data that is not ready

Gartner expects organisations to abandon 60% of AI projects unsupported by AI-ready data. "AI-ready" is vague, so we break it into four questions that can each be answered yes or no. Is the data the system reads the current version? Does the system respect the same access rules as the source, so a user cannot retrieve a document they could not open directly? Can you trace each answer back to the records it used? And is there an owner who will fix the data when the AI exposes a problem with it?

The permissions question is the one that ends pilots abruptly. A retrieval assistant indexed over a shared drive works beautifully until someone asks it about salaries and it answers from a spreadsheet that HR thought was private. The fix is to carry access control lists through ingestion and filter at query time, which is straightforward to design in and painful to retrofit. Freshness is the quieter failure: a policy assistant that cites the 2023 travel policy with full confidence does more damage than one that says it does not know.

Evidence for risk, security and audit

The third gate is owned by people who were not in the demo. Security wants to know what the agent can reach. Legal wants to know what it said to a customer. Auditors want to see who approved what, and when. Regulation is moving the same way: the EU AI Act's Article 12 requires high-risk systems to record events automatically over their lifetime, and the NIST AI Risk Management Framework and ISO/IEC 42001 both expect documented measurement, monitoring and accountability.

Most pilots log prompts and responses to a developer console, if at all. That is not evidence. Evidence is a record that links an action to an identity, a permission, an input, a model version and an outcome, kept somewhere it cannot be quietly edited, and assembled into the form a reviewer recognises.

The useful insight is that the evidence comes from things a well-built system produces anyway. Agent identity and permission checks produce an action log. A model gateway produces a record of which model and prompt version answered. The evaluation harness produces scores and release decisions. If those three streams share identifiers and land in a tamper-evident store, the evidence pack is a report. If they do not, it is a forensic project every quarter.

Integration with the systems that hold the work

MIT's other finding was that executives tend to blame regulation or model quality when the real problem is "flawed enterprise integration". In practice this means the pilot produces an answer in a chat window, and a person copies it into the ERP. The value case assumed the agent would post the journal, create the purchase order or update the shipment, and nobody built that part.

Writing to systems of record is where engineering discipline matters most. Writes must be idempotent, so a retried call does not create two purchase orders. Exceptions need a queue a person works, with enough context to resolve them without opening four systems. Partial failures in multi-step processes need compensation rather than a stack trace. And the integration needs testing against real sample files, because EDI documents and ERP exports in the wild rarely match their specifications.

Diagnosing a stalled pilot

When we are asked to look at a pilot that has stopped moving, we start with the symptom rather than the technology. The table below is the short version of the questions we ask in the first week.

Figure 3 turns the same questions into an order. We check evaluation first because it is cheapest to fix and every later gate depends on it: you cannot argue for write access to the ERP with an accuracy figure you do not have.

Where our accelerators fit

We built a set of accelerators because we kept rebuilding the same unglamorous parts. Alongside AgentGate for agent identity, permissions and audit, and the BYO-LLM gateway, these are the ones aimed directly at the four gates:

1. Monarch Evals for the evaluation gate: test sets, model-graded scoring checked against people, and a release gate that blocks changes that make answers worse.
2. Monarch Retrieve for data readiness: answers from your data that respect each user's permissions and cite their sources.
3. Monarch Assure and Ledger for evidence: agent, gateway and evaluation logs mapped to the EU AI Act, NIST AI RMF, ISO/IEC 42001, SOC 2, HIPAA and PCI, kept in a tamper-evident record.
4. Monarch Connect, Intake and Review for integration: ERP connectors and EDI mapping kits tested against real sample files, document capture, and a reviewer console for the exceptions.
5. Monarch Trace and Identity for the questions that come after go-live: AI cost per customer and feature, and fine-grained permissions for your customers' teams, sharing one decision log with AgentGate.

Chase and Explain cover two common back-office patterns: agents that follow up on missing documents, and plain-language explanations of contracts. MIT found that most budgets go to sales and marketing but the strongest returns come from back-office automation, which is why so much of the list is about documents, reconciliation and the work between companies. Legacy code modernisation is the other area with published results; Morgan Stanley reported roughly 280,000 developer hours saved in five months with an internal tool. Figures like that are reported by the organisations involved, so we treat them as direction rather than promises.

The pieces compound. Evaluation results become audit evidence. Documents captured by Intake flow into the ERP through Connect. Every agent action is already identified and logged by AgentGate. Equally deliberate is what we do not build: general chatbots, IT and HR service desks, enterprise search, code-completion tools, identity providers and payment processing are done well by large platforms, so we integrate with them.

Where this diagnosis breaks

The four gates assume the use case is worth doing. Some pilots stall because the value case was never real: the process being automated takes two people an hour a week, or the decision it supports is made by a committee that will not change how it works. No amount of evaluation or integration fixes that, and the honest recommendation is to stop.

The model also breaks when the underlying process is undocumented. If three regional teams handle supplier disputes three different ways, an agent cannot be evaluated against "the" correct answer, because there is not one. The work then starts with process design, and the AI waits. We have also seen teams over-invest in the evidence gate for low-risk internal tools, building a full audit trail for a meeting-notes summariser. Match the depth of each gate to the consequence of a wrong answer.

Finally, buying is not always right. The surveys favour buying because most buyers lack the engineering capacity to build the scaffolding. A team that already has evaluation, identity and integration patterns in place may get better results building the domain-specific part itself. The question is which parts are differentiating, and which are plumbing you should not be writing for the third time.

A decision rule for the next quarter

If you have a stalled pilot, do not start a new one. Spend the next four weeks answering these, in order, and stop at the first one you cannot answer with evidence:

1. What is the pass mark, who agreed it, and what does the system score on a few hundred real cases, by segment?
2. Which data sources does it read, who owns each one, and how are their permissions enforced at query time?
3. If audit asked tomorrow what the agent did for a named customer last Tuesday, how long would the answer take?
4. Which system of record does it write to, and what happens when a write fails halfway?
5. What does one transaction cost, and is that below the value it creates?

The first unanswered question is your plan. It is usually less exciting than a new model, and it is almost always the thing standing between the pilot and production.