Artificial intelligence is no longer on the horizon of medical device regulation — it’s already on regulators’ desks. The FDA’s iterative guidance on AI/ML-based Software as a Medical Device (SaMD), the EU AI Act’s risk classification scheme, and Health Canada’s adoption of International Medical Device Regulators Forum (IMDRF) principles all send the same message: algorithmic transparency is becoming a compliance expectation, not just a design aspiration.
But a major gap remains. As the regulatory landscape is moving toward requiring AI systems to be interpretable and auditable, much of the existing guidance does not specify how that transparency should be demonstrated in a submission. The result is a documentation black hole, one that leaves developers and reviewers without a common vocabulary for assessing whether the decision-making process of an AI system is understood enough to be safely deployed in a clinical environment.
This is the AI transparency gap. And closing that gap takes a structured, submission-ready framework.
Why Transparency Matters for SaMD
The AI/ML-based SaMD presents a unique challenge from a regulatory perspective; unlike traditional software, where the behavior of the software is fully deterministic and can be traced back directly to explicit programming logic, the output of a machine learning model is based on statistical patterns learned during training. A convolutional neural network that detects a pulmonary nodule on a CT scan doesn’t follow a rule book—it runs on millions of weighted parameters that no human engineer wrote by hand.1
This opacity creates a risk of compounding. Clinicians who depend on AI-generated outputs cannot interrogate the logic behind a recommendation as they might a lab algorithm. Regulatory reviewers can’t follow a failure mode through a specific path in code. Post-market surveillance teams can’t tell the difference between a model that failed because of data drift, and one that was misconfigured at deployment.
This challenge was explicitly acknowledged in the FDA’s 2021 Action Plan for AI/ML-based SaMD that called for greater transparency around model development, validation and real-world performance monitoring.2 The agency’s proposed framework for Predetermined Change Control Plans (PCCPs) also highlights the importance of developers documenting not only what their algorithm does, but why it does it and how that behavior could evolve given the passage of time.
TRACE Framework: A Practical Structure for Algorithmic Transparency
To fill this documentation gap, the TRACE framework was developed as a structured approach to demonstrate AI/ML transparency in SaMD regulatory submissions. TRACE traces transparency requirements along five dimensions:
Traceability: With traceability, can you trace every algorithmic output to a specific training decision, source of data, or architectural choice? Traceability implies that developers need to keep clear records of the data lineage, preprocessing steps, model architecture decisions, and hyperparameter configurations that went into the final model. If not, a failure in post-market surveillance has no obvious way back to root cause.
Reproducibility: Is it possible to obtain the same outputs for the same inputs? Reproducibility is the essence of regulatory confidence. This includes not only technical reproducibility (fixed random seeds, versioned dependencies), but also contextual reproducibility: does the model perform consistently across the patient populations and clinical environments described in the indications for use?
Accountability: Is there clear documentation of who made which development decisions and when? Who’s responsible for the behavior of an AI model is often unclear — a team of data scientists, clinicians, and software engineers all contribute to a model’s behavior. Decision making ownership for regulatory submissions should be clearly mapped, especially for decisions affecting safety-critical outputs.
Comprehensibility: Can the model’s behavior be explained in a way that is meaningful to its intended users? This is more than just technical documentation. End users of AI driven outputs are clinicians and patients; comprehensibility requires that developers produce not only explanations at the algorithm level but use-case narratives that describe what the AI does, what it does not do, and under what conditions its outputs should be questioned.
Explainability: Does the submission provide evidence that the internal decision making process of the model is interpretable to the extent required for the intended risk level? High-risk SaMD – devices that autonomously determine diagnosis or treatment decisions – requires a greater level of explainability than lower risk applications. This dimension directly maps to the tiered obligations of the EU AI Act, and the FDA’s “transparency and explainability” focus in its AI assurance principles.3, 7
Regulatory Landscape
The rise of TRACE-style thinking in regulatory guidance is not by chance. In 2021, the FDA, Health Canada, and the UK’s MHRA published joint principles for good machine learning practice.4 They specifically highlighted “transparency” and “explainability” as key features of safe AI systems. The EU AI Act, which took effect in August 2024 and starts applying its broader provisions in 2026, requires that high-risk AI systems, which include most SaMD, be designed so users can understand outputs.5 It also requires that technical documentation be available for review by national authorities.
ISO/IEC 42001:20236, the new international standard for AI management systems, strengthens this trend. It requires organizations using AI to create policies for transparency, documentation, and impact assessment. This language closely matches the TRACE dimensions.
Together, these developments indicate that SaMD developers now face a different question. It is no longer about whether to invest in algorithmic transparency. Instead, it is about how to achieve this in a way that satisfies regulators, aids clinical users, and stands up to post-market evaluation.
Practical Implications for Submissions
The TRACE dimensions are most valuable when they are built into the development lifecycle from the outset, rather than assembled at submission time. Transparency is not a document to be produced at the end of a project; it is a set of practices that need to run through every stage of AI/ML SaMD development.
The starting point is data lineage. Developers should maintain a living data card throughout the project, tracking the source and size of training data, the demographic characteristics of the dataset, and the sequence of preprocessing steps applied. A continuous record of the data pipeline means that reviewers and post-market surveillance teams have direct access to the decisions that shaped the model, rather than relying on a reconstruction after the fact.
Reproducibility requires similar discipline. This means documenting software versions and dependencies, fixing random seeds, and recording the computational environment in which the model was developed and validated. For teams working toward a De Novo or PMA pathway, aligning validation experiments with those evidentiary standards from the beginning avoids costly rework later.
Documentation should also be structured by audience. Technical documentation intended for regulatory reviewers and the Instructions for Use written for clinical end users serve different purposes and should be kept separate. Conflating the two undermines both: technical depth gets lost in plain language, and clinical users are confronted with detail that does not help them use the device safely.
Finally, the depth of explainability evidence should be matched to the risk level of the device. A high-risk SaMD that autonomously informs diagnosis or treatment will require stronger interpretability evidence than a lower-risk application. Calibrating this appropriately protects lower-risk developers from disproportionate documentation burden while ensuring that high-risk systems meet the standards the EU AI Act and FDA assurance principles demand.
Together, these practices build a submission package that is traceable, reproducible, and explainable by design rather than by assertion.
What’s coming ahead
The AI transparency gap will not close through regulation alone. It will close when the field develops shared documentation standards, agreed-upon methods for testing explainability claims, and organizational cultures that treat interpretability as a design requirement rather than a post-hoc addition.
Frameworks like TRACE are a step toward that shared vocabulary. They give regulatory teams a structure for asking the right questions during development, give reviewers a basis for evaluating submissions, and give post-market surveillance teams the documentation anchors they need to investigate failures when they occur.
As AI continues to move from clinical pilots to standard of care, the ability to demonstrate — not merely assert — that an algorithm’s behavior is understood, traceable, and explainable will increasingly separate submissions that move through review from those that stall in it.
Author Bio
Srividya (Sri) Narayanan, MS, CQSP, is a Regulatory Affairs Specialist at Asahi Intecc USA, where she supports FDA and Health Canada submissions for interventional medical devices. A former clinical dentist and University Silver Medallist, she transitioned into U.S. regulatory affairs after completing her MS in Regulatory Affairs at Northeastern University. Sri developed the TRACE framework for trustworthy AI/ML medical software and is co-founder of ReguTron, an AI-powered regulatory simulation platform. She also created Synapsense, an AI diagnostic tool for Parkinson’s disease that won MIT GrandHack 2025, and has published on algorithmic transparency, AI governance, and software-as-a-medical-device risk categorisation. She speaks at international forums on AI governance and regulatory strategy in medical devices. A conflict of interest disclosure applies in connection with the author’s co-founder role at ReguTron.
