Misaligned incentives, the wrong unit of work, and a probabilistic model where the process demands a reproducible one
A draft annex to European GMP guidance, out for consultation until October 2025, contains the bluntest sentence yet written about AI in regulated manufacturing. Models with a probabilistic output — ones that, given identical inputs, might not return identical outputs — “should not be used in critical GMP applications”. Generative models and large language models are excluded outright from those applications. Most people on my side of the industry read that as a door closing. I read it as the clearest product specification a regulator has ever handed a builder.
Pilots here do not fail on the science, and they do not fail because the regulator is slow. They fail for three compounding reasons, none of which is the model: the person running the pilot is often quietly rewarded for its failure; the team automates the workflow it already has instead of asking what the work looks like designed for agents; and, uniquely in regulated work, a probabilistic system is asked for an output that must be reproducible and defensible on inspection years later.
I should say where I sit. I am not a life scientist. I build AI products — I have been shipping commercial applications on large language models since 2021 — and I have watched this pattern in financial services, professional services and lending. Standing outside your sector is why I see it as a pattern rather than as your peculiar problem.
The number, stated correctly
The most-quoted statistic in enterprise AI comes from MIT’s Project NANDA, whose 2025 report The GenAI Divide: State of AI in Business found 95% of organisations getting zero return on generative-AI investment. It is routinely mangled: it is not a count of pilots that never shipped. Across 300 disclosed initiatives and 52 structured interviews, the vast majority of pilots produced no measurable P&L impact; roughly 5% extracted millions in value. Adoption was never the problem. Consequence was. The same report put healthcare and pharma at 0.5 out of 5 for structural disruption, summarised in four words: “documentation/transcription pilots; clinical models unchanged”. Meanwhile FDA’s drug evaluation centre has received over 300 regulatory submissions containing AI components. There is no shortage of AI in drug development. There is a shortage of AI that changed anything.
The sponsor is often the person it threatens
Ask who is better off if the pilot works. Frequently, not the person running it. A vice-president of procurement at a large pharmaceutical company put it to the MIT researchers: “If I buy a tool to help my team work faster, how do I quantify that impact? How do I justify it to my CEO when it won’t directly move revenue or decrease measurable costs?”
That is not resistance to change; it is someone reading their own scorecard correctly. A data manager who automates query resolution has built the argument that their team is oversized. A publishing lead who lets a model draft part of a module absorbs the risk of a deficiency letter, while the saving lands in a budget they do not own.
If the pilot’s sponsor is the person whose function it shrinks, you have not run an experiment. You have run a referendum.
Automating the workflow you already have
The second failure is quieter. A team takes the process as written — the SOP, the data management plan, the amendment workflow — and drops a model into one step. That is the existing organisation, rendered slightly faster.
Consider trial data. A 2025 Tufts Center for the Study of Drug Development study across 105 protocols found a phase III protocol now collects an average of 5.9 million datapoints, and that close to a third of its procedures and datapoints are non-core or non-essential, carrying as much as 30% of site and participant burden. Put a model on that and you have automated the collection of data nobody needed.
Or take amendments. Tufts benchmarks from 950 protocols found 76% now carry at least one, up from 57% in 2015, and that the average time from identifying the need to amend to final approval is 260 days. The pilot question is whether a model can draft the amendment faster. The agent-first question is why the amendment is a discrete 260-day event at all. When reading is free — every deviation, every query, continuously — an amendment stops being a document you write and becomes a state the trial is monitored against.
It also means abandoning the single-model habit. What works is a hierarchy: a cheap, fast model reading everything; a slower, abler one that wakes only for anomalies; deterministic code deciding which of those a human ever sees. You are managing a team, not using a tool.
Non-determinism is a specification, not a verdict
Here is where life sciences genuinely differs. Elsewhere, a model right 97% of the time is a good product. In regulated work the hit rate is not the question. The questions are whether the same input yields the same output, whether you can show why, and whether both hold in three years when an inspector asks.
The guidance is unusually explicit, which is a gift to a builder. The draft GMP annex confines itself to static, frozen models; dynamic ones that learn during use are excluded from critical applications. It requires acceptance criteria at least as high as the process being replaced, a logged confidence score for each prediction with a threshold below which the system returns “undecided” rather than a guess, and test data the developers provably never touched.
EMA’s 2024 reflection paper is equally concrete on the clinical side: in pivotal trials the pre-processing pipeline and all models must be frozen in the statistical analysis plan before database lock, and “incremental learning approaches are not accepted”. EMA adds that its requirements “may in some respects be stricter than what is considered standard practice in the field of data science”.
Read as prohibitions, that is discouraging. Read as a specification, it is the most detailed brief anyone has written for a shippable AI system. None of it bans AI. It bans an unbounded model being the final answer.
The architecture is well understood: wrap the probabilistic component in deterministic machinery. A fixed, versioned model artefact. A rules engine owning every constraint that must never be violated. A confidence gate routing uncertainty to a human. An audit trail recording inputs, version, score and decision. The model proposes; the deterministic layer disposes.
The deterministic wrapper: a frozen model artefact, a confidence gate that returns undecided rather than guessing, a rules engine holding the hard constraints, and an audit trail recording every input, version and decision. Schematic by the author.
I built that shape years ago, for a cruder reason. Generating financial projections on a 2021-era language model, hallucination was not a risk but a certainty. So the model handled language and judgment while a deterministic rules engine owned the arithmetic and the constraints. The rules engine was not a safety net bolted on afterwards; it was the product. The deterministic layer is not a tax on the AI. It is what makes the AI shippable.
FDA’s January 2025 draft guidance offers one more opening: its credibility framework scopes out AI used for “operational efficiencies … that do not impact patient safety, drug quality, or the reliability of results”. Most of what models are good at today sits on the low-burden side of that line.
The strongest objection
The honest counter is that the slowness is the point. The base rate is brutal: 13.8% of programmes entering phase I reach approval, 3.4% in oncology. Where most bets lose, the cost of a fast wrong answer is not a bad quarter. Validation, traceability and four-eyes controls exist because people were harmed when they did not. Caution here is the accumulated memory of the field, not irrationality.
I accept the premise and reject the conclusion. Rigour is a property of the evidence standard, not of the days a document spends moving between two people. Those 260 days are not 260 days of scientific deliberation. The non-core third of trial procedures protects nobody: it burdens sites and participants, and higher data volume is associated with higher error rates. Conflating rigour with latency protects processes, not patients.
The regulators have drawn that distinction themselves. ICH E6(R3) asks for a “risk-based and proportionate approach”, with mitigation scaled to the importance of the data being collected. FDA’s computer software assurance guidance, finalised in February 2026 for device manufacturing, replaces uniform validation with risk-based assurance in which “the burden of validation is no more than necessary to address the risk”. Permission to be proportionate has been granted.
The 95% figure reads as an indictment of the technology. Read the study and it says something narrower: the divide “does not seem to be driven by model quality or regulation, but seems to be determined by approach”. Regulation is the most available excuse here, and it was never the binding constraint. The annex tells you which outputs must be reproducible and which may be flagged undecided; the reflection paper tells you when a model must be frozen. That is not a wall. It is a specification: written down, in public, for free.
The teams that read it as a specification will ship. The ones that read it as a warning will run another pilot.
Author Bio
