Two rooms that rarely talk
I spend my working life in two rooms that almost never talk to each other. The first is the hospital floor, where I care for several thousand patients a year as an attending physician. The second is the conference room, where I evaluate health technology companies, including many that promise to capture medicine in software. Sitting in both rooms has convinced me that the space between them is the single most important thing to understand about artificial intelligence in medicine right now.
On the floor, a deteriorating patient rarely announces themselves with a number. More often it is the daughter who pulls me aside and says her father is “just not himself today.” It is the nurse who feels uneasy about a patient before any vital sign has changed. These signals are quiet, human, and easy to miss. They are also often the earliest warning we get.
In the conference room, I hear a different story. Companies tell me their models can catch deterioration before the nurse does, predict who will be readmitted, and standardize decisions that humans make inconsistently. Some of that is true. Some of it is hope dressed up as data. Here is how I see the technology honestly: where it genuinely helps, where it quietly creates new risks, and where the honest answer is simply that nobody knows yet.
This is not a story about the future
This transformation has already happened. The Food and Drug Administration keeps a public list of every AI-enabled medical device authorized for sale in the United States. Between 1995 and 2015, exactly 33 such devices made that list. As of mid-2026, the total had passed 1,500 [1]. Three quarters of them sit in radiology, reading images [1].
But here is the part that should make every reader pause. Most of these devices did not reach the market by proving in rigorous clinical trials that they make patients better off. Roughly 97 percent cleared the regulatory pathway known as 510(k), which allows a device to come to market by showing it is similar to something that already exists, rather than by generating new clinical evidence [1]. One analysis published in 2026 found that only 3 out of 1,357 cleared AI devices had ever been tested on outcomes patients actually care about, such as survival or avoiding readmission [2]. We are deploying at the speed of software and validating at the speed of a poster session.
Where AI genuinely helps
Let me start with the wins, because they are real, and because I want these tools to succeed.
Documentation. The clearest success so far is the ambient AI scribe: a tool that listens to the conversation between doctor and patient and drafts the clinical note automatically. The largest rigorous study to date, published in JAMA in 2026, followed thousands of clinicians across five academic medical centers and found that scribes saved about 16 minutes of documentation time for every 8 hours of scheduled patient care, roughly a 10 percent reduction [3]. A separate study found that after 30 days of use, the proportion of physicians experiencing burnout fell from about 52 percent to about 39 percent [4]. On my own floors I can see the difference with my eyes. Doctors look up more and type less. When a tired physician gets twenty minutes of attention back, that is not a convenience. That is a patient safety tool.
Reading images. Diagnostic imaging is the other place where the evidence is genuinely strong, because the tasks are narrow and there is usually a clear right answer. Sweden ran the first large randomized trial of AI-supported mammography, covering more than 100,000 women. The AI-supported screening detected 29 percent more cancers than standard double reading, without increasing false positives [5], while cutting radiologists’ reading workload by 44 percent [6]. Follow-up results published in early 2026 showed fewer advanced cancers in the years that followed [7]. Autonomous AI screening for diabetic eye disease was the first system authorized to work without a clinician interpreting the image, which means a diabetic patient in a small-town clinic can get retinal screening without waiting months for an appointment [8]. These are the investments I underwrite with the least hesitation: narrow jobs, measurable answers, and a human still in the loop.
Closing gaps in care. There is also promising evidence, still early, that well-built tools can reduce disparities that human judgment alone creates. In one pediatric emergency department, when doctors relied on clinical judgment alone to recognize sepsis, white children were more than three times as likely as Black children to receive timely treatment. Once an electronic sepsis alert was switched on, that racial difference disappeared [9]. A consistent algorithm can, in the right setting, be more even-handed than an exhausted human at three in the morning.
Where AI quietly creates new risks
Now the harder part. The same tools that help can fail in ways that never show up in a sales demo.
The prediction that arrives too late, or too often. Every hospitalist should know the cautionary tale of the most widely deployed commercial sepsis prediction model in American hospitals. The vendor reported strong performance. Then independent researchers validated it on their own patients and found it was barely better than a coin weighted slightly in your favor [10]. At the vendor’s recommended threshold, it missed about two thirds of the patients who actually developed sepsis while flagging nearly one in five of all hospitalized patients [10]. Meanwhile, some hospitals had scaled back bedside screening because they believed the model was watching. All models are imperfect. The harm comes when clinicians stop doing the things they used to do because they trust a tool that does not deserve that much trust.
Alert fatigue is not a side effect. It is the main failure mode. Studies stretching back twenty years find that clinicians dismiss the large majority of medication safety warnings, and in one center more than 95 percent of high-priority drug interaction alerts were overridden [11]. In monitored units, more than 90 percent of audible heart rhythm alarms turn out to be false positives [12]. If a new AI warning system cries wolf even a fraction of the time, the floor will learn to ignore it within weeks.
Over-trust in the machine. In a randomized experiment, 457 clinicians reviewed patient cases, sometimes with AI advice added in [13]. Accurate advice nudged their diagnostic accuracy up modestly. But when the AI had been deliberately made biased, the same clinicians got substantially worse, losing more accuracy than the good AI had gained. Showing them explanations did not protect them. Two thirds said they did not believe AI outputs could be systematically biased at all. A related experiment in mammography found that a single wrong AI suggestion dragged experienced radiologists down to roughly 45 percent accuracy [14]. Researchers call this automation bias. On a busy floor it has a simpler name: being too tired to double-check the confident screen.
Bias baked into the objective. Some of the most damaging failures happen before the model ever meets a patient, in the choice of what to predict. A widely used population health algorithm dissected in Science in 2019 predicted which patients needed extra care by predicting who would generate high costs, because cost was easier to measure than illness [15]. But the American health system has historically spent less on Black patients for the same amount of sickness. So the algorithm concluded Black patients were healthier than they were. Correcting the objective would have nearly tripled the share of Black patients flagged for additional care. Nothing in the code was malicious. The blind spot came straight from the data, and it hid behind respectable mathematics. Even the pulse oximeter overestimates oxygen levels more often in darker-skinned patients, tripling the rate of dangerously low readings that go unnoticed in Black patients [16]. An algorithm inherits the blind spots of its inputs, and on my floor those blind spots have names and faces.
Deskilling. A multicenter study published in 2025 tracked experienced endoscopists before and after an AI polyp-detection tool entered their routine work [17]. On standard colonoscopies without AI assistance, their adenoma detection rate fell from 28.4 percent to 22.4 percent after they had grown accustomed to the tool. After a short period of working with AI help, these doctors got measurably worse at spotting polyps on their own. If a few months of assistance can dull an expert’s skill, we need to think hard about what a forty-year career of assistance does to the physicians being trained right now.
The honest middle: where nobody knows yet
Between clear help and clear harm sits a large territory where the verdict today is simply “unproven.” The most careful census to date, published in August 2026, examined all 1,357 FDA-cleared AI devices and found that only 2.5 percent had any registered prospective clinical trial, and just three had ever been tested on patient-centered outcomes such as mortality or readmission [2]. Even ambient documentation carries a stubborn disappointment: the biggest study found no reliable reduction in after-hours charting, the so-called pajama time [3].
In 2026, researchers from Stanford, Harvard, MIT, Google, OpenAI, Anthropic, Amazon and the Advanced Research Projects Agency for Health published a commentary in Nature Medicine warning that phrases like “medical superintelligence” are being used in marketing long before anyone has agreed on what would prove such a claim [18]. A benchmark score means nothing on its own. An AI scoring 75 percent on a diagnostic test sounds unimpressive until you learn average physicians score 65 percent, and impressive until you ask whether the alternative for real patients was waiting six months for a specialist. Context decides everything. Those authors also flagged a quieter problem: tools sold as “decision support” drift toward de facto autonomy. If a busy clinician signs off on the machine’s answer ninety-nine times out of a hundred without really looking, the human review is safety theater, not safety. Their proposed fix is the one medicine has always trusted: randomized trials with real patient outcomes as the finish line. Mammography shows the template already exists [5].
What the floor knows, and what I look for as an investor
Both of my rooms teach the same lesson. AI is excellent at the measurable and blind to the contextual, and medicine lives largely in the contextual. The tools that succeed are narrow, keep a clinician in charge, and are validated where they will actually be used. The ones that fail are sold as judgment instead of input.
When I evaluate a healthcare AI company, three questions matter more than any slide deck.
First, is the tool positioned as a junior team member or as a senior consultant? The attending physician still examines the patient and makes the call. And “human in the loop” has to mean a human who is actually paying attention, not a checkbox on the way to de facto automation.
Second, is the evidence proportional to the stakes, and was it gathered where the tool will actually run? Benchmark victories on curated exam questions are marketing; randomized trials with patient outcomes are evidence.
Third, is the tool transparent at the moment of use? The clinician should know what data the model saw, how confident it is, and where it is likely to be wrong. A prediction without provenance is not decision support. It is a horoscope with a logo.
So when a founder tells me their model will catch deterioration before the nurse does, I think of the daughter who knew her father was not himself before any number on the monitor moved. The floor does not need AI that pretends to see everything. It needs AI that is honest about what it cannot see, and clinicians who never forget the difference. I have stood beside both, and I am certain the second one is the part we cannot afford to lose.
Author Bio
