“But what can this AI model actually do?” It’s a common question being asked across the life sciences industry. Can it classify a symptom, summarize a trial record, predict a molecule’s behavior, or monitor a manufacturing process? Those capabilities matter, but they are not where I think the conversation should start. AI models are ultimately prediction systems. They can process enormous amounts of information quickly, but they can also reach the wrong conclusion and present it convincingly, sometimes surfacing a handful of sources that support an answer while overlooking far more evidence (or stronger evidence) that contradicts it. Human-in-the-loop pharma workflows exist precisely to catch that failure.
The more consequential question is who holds the authority to act on what the AI model produces? In pharma and medical device organizations, that distinction matters because many of these decisions sit inside regulated workflows. When a decision affects patient safety, product quality, or another regulated requirement, a person still needs to own the decisive course of action. The question is: where that human authority needs to remain in the process, and when it needs to take control back from the system.
That question surfaced several times earlier this year at two different conferences: CDMO Live in Rotterdam and Pharmacovigilance USA in Boston. The conversations ranged from supply chain resilience and outsourcing partnerships to patient safety and agentic AI, but the same principle kept coming back: AI should give people time back by handling tasks like gathering, organizing, and summarizing information so they can spend more time interpreting it and making difficult decisions. I wrote about that broader lesson after the events, including why greater efficiency should not mean less human responsibility. Here, I want to take that question one step further: where, exactly, should responsibility sit as AI systems become more autonomous?
The pace of adoption makes that question hard to postpone. Over 85% of pharma companies now describe AI as an immediate priority, yet only about one in seven report anything close to a fully implemented system. That gap between intent and execution is where governance decisions are actually being made right now, often without a formal framework behind them.
Most current adoption clusters around a handful of use cases. Clinical trial optimization uses AI to select high-performing trial sites and to speed up patient recruitment. Administrative and medical writing tools automate regulatory documents, safety reports, and compliance tracking. Drug discovery applications run virtual screening of molecules and generative modeling to narrow candidate molecules.
Commercial and supply chain analytics tools forecast demand and flag manufacturing risk. Each of these sits at a different point on the authority spectrum. Treating them all the same way is where organizations can get into trouble.
That spectrum has two ends, and the industry has landed on shorthand for them: human-in-the-loop (HITL) and human-on-the-loop (HOTL). In a HITL model, a person has to approve an AI-generated action before it proceeds, the human is ‘in the loop’ and is required for steps to move forward. Nothing continues without human input. In a HOTL model, the AI operates with more autonomy inside defined boundaries, and a person supervises from above, stepping in only when something appears to be failing.
Continuous manufacturing is a useful illustration of HOTL in practice. AI agents monitor critical process parameters such as temperature, pH, fluid flow, and pressure, and make small, real-time adjustments to keep production inside validated ranges (think an AI-augmented continuous manufacturing process). A person checks a dashboard periodically rather than approving every micro-adjustment. That works well until it doesn’t, and the risk is rarely the technology itself. It is complacency, the assumption that because a system has handled the last one thousand adjustments correctly, it will handle the next one thousand the same way without anyone watching closely.
Pharmacovigilance shows the other end of that same risk. AI agents can ingest and classify unstructured safety data pulled from thousands of patient reports, scientific literature, and social media, then assign risk tiers and flags automatically. A drug safety physician stays on the loop, watching for trends and stepping in when a cluster of signals appears. That model has worked in practice.
Long before AI-driven signal detection existed, conventional pharmacovigilance already depended on this same principle. A rare clotting disorder identified in a COVID-19 vaccine, affecting a small share of recipients (and too infrequent to surface in the original trial population), was identified and acted on quickly once real-world data from more patients made the pattern visible. The system worked because a qualified person was positioned to notice it and had the authority to act.
History offers a warning about what happens when that positioning fails. In 1979, operators at Three Mile Island trusted a control panel indicator showing a relief valve as closed. It was not. Cooling water escaped, and the resulting partial meltdown became a defining case study in industrial safety, not because a machine malfunctioned, but because the humans supervising it assumed the system had things handled and stopped auditing the evidence in front of them. AI introduces a version of that same risk at far greater scale, because a model can generate a fluent, well-cited answer that is confidently wrong, drawing on a handful of sources that appear to support its conclusion while ignoring far more that do not.
That risk is not theoretical when it comes to clinical documentation and decision support. Roughly two-thirds of physicians already use AI somewhere in their workflow, and one recent framework for evaluating AI-generated clinical note summaries found a 1.47% hallucination rate and a 3.45% omission rate. In a regulated environment, even error rates that low, feeding documentation the FDA, EMA, or MHRA will eventually inspect, are the reason a qualified person still needs to sign off on anything that touches patient safety or product quality.
The right model depends on the task, not on how much automation a company would like to claim. Low-risk administrative work, drafting a first pass at a report, or tracking a compliance deadline, can support more autonomous operation. Decisions that touch clinical safety or product quality need tighter human control, with a clearly defined point at which the system hands authority back. Four questions get to the heart of it:
- Who decides which model applies to a given task?
- When does review occur?
- What level of uncertainty or deviation triggers human intervention?
- And who is accountable when the system gets it wrong?
In my experience advising organizations on this, the step most often skipped is not the last one but the first. Companies build escalation protocols for when something goes wrong well before they have done the more basic work of mapping their data governance and data provenance: where the data feeding a model came from, who owns it, and whether it can be trusted before it ever reaches an AI system. Without that foundation, every downstream framework for human oversight is built on data that has not actually been tested.
None of this is an argument against AI in pharma and med device organizations. It is an argument for sequencing. Build the data governance and provenance framework first. Decide, task by task, where a person needs to remain in the loop and where supervision on the loop is sufficient. Do that before the system has already been running for months or years, because retrofitting accountability after an AI-generated deviation surfaces during an FDA inspection is a far harder conversation than having it in advance.
The industry does not yet have standardized guardrails for any of this (and no industry does yet, by the way), and regulators are still working out how prescriptive they want to be. That will likely change only after a costly error forces the issue rather than before (the question of AI contributing to a manufacturing error or patient harm event is one of “when,” not “if”). Until then, the organizations best prepared for regulatory scrutiny will answer the harder question early: who has the authority to act, and when must that authority return to a person?
Author Bio
