Even when studies show promising findings for imaging AI tools, inconsistencies in reporting can hinder verifying results -- an essential step in translating research to clinical use.
In a perspective published on 8 March in European Radiology, an Australian team headed by Dr. Samuel J. White of the University of Adelaide underscored the need for strong, transparent reporting standards in radiological AI studies to improve reproducibility and confidence in findings.
“Transparent reporting across the analytic pipeline is essential for independent verification, evidence synthesis, and safe implementation,” the authors wrote, calling for “strengthening the quality of reporting in radiological AI studies as an essential step toward improving reproducibility in the field.”
The importance of reporting precision lies in the ability of others to obtain consistent results using the same methodology and data, White and colleagues noted. In imaging AI studies, even small variances in protocols and practices may affect the algorithm to a degree that yields significant changes in performance and results.
And while AI has become increasingly integrated into imaging, concerns about consistently reproduced results still impede full regulatory and clinical acceptance. Although numerous guidelines and initiatives have been released -- such as the Checklist for Artificial Intelligence in Medical Imaging (CLAIM), CONSORT-AI, and TRIPOD-AI -- AI reproducibility challenges remain.
These challenges, the researchers wrote, “are not solely technical problems but structural and cultural ones, rooted in how reporting is conceptualized, operationalized, and enforced.”
White’s team identified four major interrelated impediments to transparent reporting in radiological AI research and suggested practical approaches to overcoming them.
- Reporting is “frequently treated as a final-stage manuscript requirement rather than a prospectively embedded component of study design.” That is, adherence to guidance standards within the research design may be inconsistent and incomplete, with important methodological details left out. The authors propose that these guidelines should be considered and built into the structure of the research design itself to ensure robust reproducibility.
- Inconsistent implementation of these reporting standards by journals and funding bodies hinders reproducibility of results. This may make it difficult to assess whether reporting is comprehensive. The researchers recommend that funders and publishers not merely endorse the use of guidelines but explicitly require them as a condition of publication or funding.
- A number of guidance standards exist, but they are often complex and may overlap each other in what they cover -- which can lead to inconsistency and lack of clarity in reporting across studies. Too much technical specificity and too many items to follow may create ambiguity in what amount and kind of detail is sufficient for those attempting to navigate the guidelines when reporting. The inconsistency this creates may have effects “downstream” on meta-analyses and reviews, and fragmented use of different forms of guidance introduces variability that may weaken rather than strengthen reporting. White and colleagues suggested that reporting guidelines should be coordinated by the groups developing them, and that a shared set of principles should be determined and applied to all AI research.
- Data and code used by researchers is not always accessible for independent verification. Radiological AI studies often use customized architectures, workflows, even evaluation strategies, which may be difficult to reproduce. In these instances -- or where data sharing may be restricted due to privacy concerns, for example -- the authors recommended that researchers give an explanation for the lack and offer alternative solutions for attaining consistent results such as synthetic datasets.
Addressing these concerns requires structural approaches to consistent reporting and throughout processes, in which all those with a stake have involvement, and full transparency is prioritized, the authors concluded.
Read the article here.
