Article
Why your AI proofs of concept never reach production
For two years, most of the companies I meet have launched at least one generative AI experiment: an internal assistant connected to the documentation, a contract summarisation tool, an agent that triages customer requests. The demo is convincing. The executive committee is enthusiastic. Six months later, the proof of concept still runs on the laptop of the person who built it.
This scenario is so common that it’s no longer bad luck. It has causes, and they are almost always the same. None of them is about the quality of the model.
1. The proof of concept proved the wrong thing
A proof of concept answers the question “can this work?”. Production asks a different one: “does it work well enough, often enough, for enough people, at an acceptable cost?”
The difference is huge. An assistant that gives a good answer eight times out of ten makes an excellent demo. In production, the two wrong answers reach real users, who lose confidence within days.
What to do: set the success criteria before building. What rate of correct answers is acceptable? On which set of questions, representative of real use? Who judges? Without these answers, the proof of concept proves nothing but its team’s enthusiasm.
2. The data wasn’t the real data
The proof of concept was built on a clean export, prepared by hand: a few hundred well-chosen documents, no duplicates, no contradictory versions, no access rights to manage.
In production, it has to connect to the real sources: a document management system with fifteen years of history, a SharePoint where every department has its own logic, confidential documents that an intern must not be able to query. That’s often where the project stops, because nobody had costed this work.
What to do: prototype on a sample of the real data, with its flaws, and deal with access rights from the scoping stage. An assistant that ignores permissions isn’t a system in the making: it’s a data leak waiting to happen.
3. Nobody thought about integration
The proof of concept lives in a separate application. To be used, though, it has to fit into everyday tools: single sign-on, an interface users already know, data flows with business applications, logging for audit.
Each of these is a small integration project. Together, they often represent more work than the proof of concept itself.
What to do: draw the target architecture at the scoping stage, even roughly. Where does the system fit? Which APIs does it call? Who manages identities? An hour spent on this diagram saves months of surprises.
4. The costs were never calculated
During the proof of concept, the model API bill is negligible. Multiplied by a thousand users, by long documents injected into every request, by agents chaining ten calls for a single task, it can become the project’s largest expense.
What to do: estimate the cost per use during the proof phase, then design it: models suited to each task, caching, limited context, and sometimes hosting an open model. The cost of an AI system is designed; it shouldn’t be discovered on the invoice.
5. There is no owner
The proof of concept was carried by an enthusiastic sponsor and a small team. But who will run the system? Who will monitor the quality of the answers over time? Who will decide to change models when the provider releases a new one? Who will answer the data protection officer, the CISO, and tomorrow the requirements of the AI Act?
Without answers to these questions, IT refuses, rightly, to put into production a system it won’t be able to maintain.
What to do: appoint an owner before going to production, and give them the tools: observability, continuous quality evaluation, operating procedures.
Start from production, not from the prototype
What these five causes have in common is simple: they are all predictable. They can be dealt with at the scoping stage, before writing a line of code, provided you think like an architect rather than a demo builder.
That’s the logic of the approach I offer: scope, prove, architect, industrialise, in that order, with a concrete deliverable at each stage and the option to stop if the value isn’t there. A project stopped at the right time is also a success: it cost little and taught a lot.
If you have a proof of concept waiting to go into production, or several and you don’t know which to prioritise, that’s exactly the subject of a scoping call.