A growing proportion of radiology AI products in Europe are supported by peer-reviewed evidence, but the concentration of studies at lower efficacy levels highlights a critical gap in demonstrating clinical impact and improved patient outcomes, a large Dutch-led study has found.
“This study underscores both progress and persistent challenges in the validation of CE-certified radiological AI products,” noted Dr. Noa Antonissen, a doctoral candidate at Radboud University Medical Center, Nijmegen, and colleagues in an article posted on 24 July by European Radiology. “While the increasing maturity of products and market presence suggest a movement towards higher-level validations, the stagnation in studies addressing clinical and socioeconomic efficacy points to the need for more rigorous, real-world assessments.”
The researchers tracked the annual number of new CE-certified radiology AI products entering the European market from 2010 to 2023. Their data show a steady increase in new products, with a peak of 44 in 2020, compared to 33 in 2019. This was followed by a decline, with 24 new products introduced in 2021 and only four in 2022.
Yearly count of CE-certified radiological AI products introduced to the European market from 2010 to 2023. Products introduced before 2010 and those with unknown entry dates (n = 5 each) were excluded. The dashed line indicates that data for 2023 are incomplete, reflecting entries only up to 31 March 2023. All figures courtesy of Dr. Noa Antonissen et al and European Radiology.
The new study builds on the April 2021 European Radiology article by Kicky van Leeuwen, PhD, founder of Health AI Register and managing partner of Romion Health. In the original article (referred to by Antonissen and colleagues as “the 2020 analysis”), Van Leeuwen, who is the senior author of the new study, evaluated 100 CE-marked AI products in radiology, and they received the most downloaded and most cited award from European Radiology in 2021.
The group has now expanded the analysis to 173 products available in 2023, providing an updated view of the scientific evidence supporting them. The update (referred to as “the 2023 analysis") focuses on CE-certified radiological AI products listed on Health AI Register as of 31 March 2023, along with peer-reviewed papers published between 1 January 2015 and 31 March 2023. The 2023 analysis includes vendor-neutral imaging analysis tools developed using machine-learning or deep-learning techniques.
Of the 173 AI products coming from 90 different vendors, 85 were present in both the 2020 and 2023 analyses. The 15 products from the 2020 dataset that were excluded in 2023 consisted of 13 removed due to discontinuation or their absence from Health AI Register, and smaller products that were consolidated into four expanded-feature products. Additionally, 88 newly listed products since 2020 met the inclusion criteria.
The figure below shows the distribution of the 173 products across organ-based subspecialities, modalities, and functionalities.
Characteristics of 173 CE-marked AI products categorized by organ-based subspeciality, modality, and main functionality. MSK = musculoskeletal.
The researchers identified three main changes since 2020: products with peer-reviewed evidence increased from 36% to 66% by 2023; products with evidence addressing clinical impact or cost-effectiveness rose from 18% to 31% over the same period; and the number of peer-reviewed studies nearly tripled (from 237 to 639).
Most studies still focus on diagnostic accuracy, with only about 24% evaluating whether AI actually improves clinical decisions, patient outcomes, or healthcare costs. The reliance on retrospective, vendor-dependent studies further limits the independent real-world evidence base, van Leeuwen told AuntMinnieEurope on 28 July.
“The market is maturing, and recent prospective studies like the MASAI (Mammography Screening with Artificial Intelligence) trial are promising steps forward. But to move from proof-of-concept to clinical routine, we need more independent, high-quality studies that demonstrate real-world impact and added value -- evidence that's essential for adoption, reimbursement, and inclusion in guidelines,” she noted.
Highest level of efficacy applicable per product in (a) the overall market, including 100 products in the 2020 analysis and 173 products in the 2023 analysis; and in (b) a subset including 85 products that were analyzed across both the 2020 and 2023 analyses.
Future efforts should focus on improving the quality and scope of validation studies, with an emphasis on diverse, multicenter, prospective validations that reflect real-world clinical settings, patient populations, and healthcare systems, Antonissen and colleagues wrote.
“Funding agencies have a crucial role to play by prioritizing calls specifically aimed at supporting not only the development but also the independent validation, implementation, and long-term monitoring of AI tools in clinical practice,” they stated.
Under the European Medical Device Regulation (MDR), vendors are already required to monitor the clinical performance of their solutions through post-market surveillance. Publishing these findings is not mandatory, but sharing such data could provide valuable insights into the real-world effectiveness of AI tools, they continued. “We encourage both AI solution developers and users to contribute to this collective knowledge to strengthen transparency, reliability, and support for healthcare providers, ultimately advancing the safe and effective adoption of AI into clinical workflows.”
You can read the new article in full here. The other two main contributing authors were Olga Tryfonos, scientific collaborator at European University Cyprus, and Dr. Ignas Houben, a radiology resident at Isala Zwolle in the Netherlands.
