Clinical decision support is not new, and it does not have to be complex to work. If anything, the simpler it is, the more likely clinicians are to use it. The best-known earlier example is the WHO surgical safety checklist, developed by a team led by Atul Gawande, who had spent years arguing that medicine had outgrown what memory alone can carry [1]. Between 2007 and 2008, eight hospitals from Seattle to rural Tanzania trialled the nineteen-item paper version, covering things any competent team already knew, such as confirming the patient’s identity and checking that antibiotics had been given. Deaths fell from 1.5 per cent to 0.8 per cent and inpatient complications from 11.0 per cent to 7.0 per cent [2].
Decision support earns its place in medicine on fit and trust, as the checklist did. I have kept that in view since I first built clinical decision support systems as a doctoral researcher, and it matters again now that the newest kind of decision support is in the form of large language models that retrieve clinical evidence and reason over it. The interface is now a question in plain language, which removes most of the friction that held earlier systems back, and adoption has been quick. Any capable model can produce an answer, so the difficulty is telling a sound answer from a merely plausible one, and getting a clinician to use the system at all.
Two traditions of decision support
Clinical decision support has always drawn on two traditions. The first is quantitative: statistical models that predict an outcome, such as a risk score or a survival estimate. The second is qualitative: systems that reason over clinical guidelines to lay out the options and the evidence for each.
My doctoral work built both into one tool for lung cancer treatment selection. A Bayesian network, learned from the English Lung Cancer Database, estimated how a patient’s survival would change under different treatment plans [3]. Alongside it, an ontology-based rule engine matched each patient to national and international guidelines and listed the recommended options with the arguments behind them [4].
The qualitative tradition then stalled, for a practical reason. Encoding guidelines as formal rules, and keeping them current as the guidelines changed, was slow and costly, and the resulting systems were rigid. Large language models have given qualitative decision support a second life. A model that retrieves the relevant guideline passage and reasons over it in plain language, an approach known as retrieval-augmented generation (RAG) [6], removes much of the authoring burden. Later variants index the source as a knowledge graph rather than as loose passages, so that related recommendations can be traversed together [7].
The enduring requirements
The lung cancer tool I built during my doctorate was designed to a set of requirements for clinical decision support [5]. Reading them now, against a generation of AI systems, almost nothing needs changing.
Clinical decision support should aid the clinician’s decision making rather than attempt to automate it. It should be grounded in national and international guideline rules. It should give quantified, patient-specific answers rather than generic recitals. It should not disrupt clinical workflow, which means answering fast enough to be used during the consultation. It should speak a shared clinical vocabulary, in that case SNOMED CT, so it can exchange information with other systems. And it should be reachable online, so that clinicians can get to it and send back feedback quickly. Last but not least it should be transparent. The rule engine laid out each option with the arguments for it, rather than issuing a verdict to be taken on trust [4].
Transparency is the requirement that has changed character. In that system it was a property of the design. In a language model it is not a property of the architecture, so it has to be stated as a requirement and tested for. A retrieval-based language model has to meet the same list, and the harder question is how to establish whether it does.
The new evaluation problem
Grounding a model in retrieved evidence makes these systems attractive in medicine. An answer tied to a named guideline passage can be checked. But retrieval introduces failure modes that a standard model benchmark never tests.
The system can retrieve the wrong evidence. It can retrieve the right evidence and then drift from it, giving an answer the source does not support. And it can answer in general terms that are true of the condition but not tailored to the patient. Each of these is a distinct failure, and a single headline accuracy score speaks to none of them [8].
Accuracy alone misleads in a further way. Where most patients fall into one category, a model that always gives that category’s answer will score highly while being unsafe for everyone who does not fit it.
Evaluation in practice
Detecting those failures takes more than one measurement, and one of them has to be a clinician’s judgement. In a recent evaluation my colleagues and I carried out, we compared a language model on its own against two retrieval variants: retrieval-augmented generation over the NICE chronic kidney disease guideline [12], and GraphRAG over a knowledge graph built from that guideline. All three ran over seventy synthetic patient cases, with nine questions per case written by a consultant nephrologist and scored against curated reference answers [9].
The model on its own handled questions that needed broad medical knowledge, where its training data covered the ground, but it could not supply the guideline detail that mattered, which widened the surface for hallucination. Every configuration hallucinated on the kidney failure risk equation, which the guideline carries only as an image, leaving no model able to read its coefficients.
Benchmarks at scale
Scoring answer quality, and detecting failure modes like these, has caught the attention of the large AI labs, which have begun to build benchmarks for medicine. One example is OpenAI’s HealthBench, which comprises 5,000 realistic health conversations, each graded against a rubric written by one of 262 physicians. It scores responses on accuracy, completeness, context awareness, communication quality and instruction-following [10]. Earlier work such as Med-PaLM measured models against clinicians’ answers [11].
Guidance for clinical teams
No clinical team will build a benchmark the size of HealthBench, and none needs to. A few practical steps follow for a team weighing one of these systems. Evaluate it against the guideline that governs the decision it is meant to support. Keep a clinician in the loop, because a clinically wrong answer can read perfectly well. Measure several dimensions, and treat a high accuracy figure with suspicion until you know what it conceals. Test whether a clinician can judge why the system gave an answer, and whether using it is faster than not using it. And for anything that bears on a clinical decision, treat reproducibility as a requirement, since an answer that changes between runs cannot be governed or defended.
Clinical decision support is an old idea that language models have made cheap to build and easy to use, which is why they are spreading quickly. The same machinery brings failure modes of its own, from a passage retrieved wrongly to a coefficient invented where the guideline held only an image. None of that surfaces in a headline score, and none of it is visible to the clinician reading the answer. Anyone using Gawande’s surgical checklist could read the whole of it. The retrieval systems now arriving in clinical AI are far more capable and far less transparent, and they will be adopted whether or not we have learned to evaluate them well. The systems that earn a place in practice will be the ones clinicians can see into and rely on.
Author Bio
