Every researcher who trains medical-imaging models knows the moment. The training run finishes, the metric appears, and it is not just good, it is startlingly good. Accuracy in the high nineties. A segmentation score well beyond anything in published medical AI benchmarks. The instinct is to celebrate. The discipline is to do the opposite: a result that looks too good to be true usually is, and the size of the surprise is roughly the size of the problem hiding underneath it.
The field has already run this experiment at scale. During the first year of the COVID-19 pandemic, hundreds of machine-learning models were published claiming to detect or prognosticate the disease from chest radiographs and CT scans. When a Cambridge-led team systematically reviewed them, examining 62 studies in depth from 415 shortlisted after initial screening, they concluded that not one model was fit for clinical use, citing methodological flaws and biases throughout [1]. These were not careless teams. They were good researchers, working urgently, whose evaluation pipelines quietly let them fool themselves. The same failure modes that sank those models are still present in medical-AI papers today, and three of them account for most of the damage: leakage, friendly evaluation, and unmodelled variation in the images themselves.
Leakage: the model has seen the answer
Data leakage means that information from the test set has influenced training, so the evaluation no longer measures performance on unseen data. It sounds like a beginner’s error. It is not. Kapoor and Narayanan’s survey of the reproducibility literature found leakage in at least 294 papers across 17 scientific fields, and showed that it had produced overoptimistic conclusions in field after field [2].
In medical imaging, leakage rarely looks like copying the test set into training. It looks like bookkeeping. A whole-slide image is cut into thousands of patches, and the patches are split randomly, so patches from the same slide, the same patient and the same staining batch appear on both sides of the divide. A model can then score highly by recognising the slide rather than the disease. The same thing happens when duplicate or near-duplicate images survive in a public dataset, when augmentation is applied before the split so that transformed copies of test images enter training, or when preprocessing statistics are computed over the whole dataset at once. Each is a one-line decision in a training script. Each can add points to a score that no external dataset will ever reproduce.
Friendly numbers: single splits and best runs
The second failure is quieter: honest code, friendly arithmetic. A single random split of a single internal dataset will, by chance, sometimes be easy. Report the best of several runs rather than the average, and the published number drifts further from the truth. Neither practice is fraud; together they manufacture a result that the next laboratory cannot repeat.
The remedies are unglamorous. Split at the level of the patient or the slide, never the patch. Evaluate with cross-validation and report every fold, with the spread, not only the mean. And wherever public medical AI benchmarks exist, compare against their published reference numbers on the same data before claiming an advance. In my own recent evaluation work on nuclei segmentation in histopathology images, I re-ran a standard public benchmark (NuInsSeg, with more than 30,000 manually annotated nuclei across 31 organs [3]) under five-fold cross-validation with strict separation. The honest result settled about two Dice points above the published reference of 0.814 (a modest, believable improvement), with individual folds ranging from roughly 0.82 to 0.84. Reporting that range is less thrilling than a single headline figure. It is also what the result actually is, and it is the version another group can check.
Stain variation: the confounder nobody trained for
The third failure is specific to pathology, and it is the one I have come to respect most. Two laboratories staining the same tissue will not produce the same colours. Haematoxylin and eosin uptake varies with protocol, reagents, section thickness and scanner, so the images a model meets in a new hospital differ systematically from the ones it trained on, before any biology enters the question. Tellez and colleagues quantified this across four organs and nine laboratories, testing how stain augmentation and stain normalisation affect convolutional networks; their central finding was that colour variation is a first-order effect on generalisation, and that training with stain augmentation matters more than the choice among normalisation methods [4]. In practice, a change of laboratory can move a model’s score by more than a change of architecture.
This has a direct consequence for how models should be tested. A stain-shift stress test (deliberately perturbing the colour properties of the test images, or better, evaluating on slides from a laboratory the model has never seen) should be as routine as the primary metric. A model that collapses when the stain deepens slightly is not a model of nuclei, or of tumours. It is a model of one laboratory’s colour palette.
What honest evaluation looks like
None of this requires new mathematics. It requires reporting habits, and the field has already written them down: the CLAIM checklist for AI in medical imaging asks authors to describe their data partitions, their reference standard and their failure cases precisely so that readers can judge an evaluation for themselves [5]. To that I would add three working rules from my own practice. First, treat an implausibly high score as a bug report, not a result: find the leak before you publish the number. Second, publish the benchmark comparison and the variability alongside the claim, every time. Third, show the failures: the images the model got wrong are the most informative part of any evaluation, and the first thing a sceptical reader looks for.
A short checklist for anyone reading a medical-AI claim, whether researcher, clinician or reviewer:
- Was the data split at the patient or slide level, with no overlap of cases between training and test?
- Is performance reported across folds or runs, with variability, or as a single best number?
- Is the model compared with a published benchmark on the same dataset?
- Was it tested on data from a different site, scanner or staining protocol?
- Are failure cases shown and discussed?
A paper that answers all five has earned its headline figure. A paper that answers none is asking for trust it has not built.
The case for honest medical AI benchmarks
There is a temptation to treat rigorous evaluation as a tax on progress: extra folds, extra datasets, extra weeks. I have found the opposite. The two-point improvement that survives cross-validation is worth more than the twenty-point improvement that does not survive a change of laboratory, because the two-point improvement is real: it can be built on, cited safely, and taken toward the clinic. Honest medical AI benchmarks are not the enemy of strong results. They are how we find out which results are strong.
Author Bio















