Twilight of Radiologists? Will They Be Replaced by Artificial Intelligence?

Who Will Evaluate Imaging Data

According to a survey of 331 physicians who published in the prestigious medical journals New England Journal of Medicine and Lancet between 2010 and 2022, 87.3% of them expect an increase in the use of imaging methods in the next 10 years, while 48.9% believe that the demand for radiological diagnostics will also increase.

Only 16% of respondents believe that artificial intelligence (AI) could replace radiologists in the next decade. 60% held the opposite view, with nearly a quarter still undecided. Some believe AI could help radiologists with routine, well-defined tasks. However, most physicians think AI will not replicate the human approach to problem-solving and will struggle with complex or unusual cases.

Ingredients for a Quality Report

For an AI-generated report on imaging results to be usable, it needs more than just information about the presence or absence of abnormalities. It must provide a detailed description of the image with all subtle nuances and relevant measurement uncertainties—in other words, a radiology report contains complex diagnostic information.

Several AI model prototypes have emerged that can create detailed narrative radiology reports. Along with these tools, automatic scoring systems have been developed to continuously evaluate AI tools, track their development, and improve their performance. But how well can software assess the quality of AI outputs? This is precisely what a team from Harvard University focused on in their analysis.

Not-so-capable Evaluators

Researchers tested several scoring systems on a set of AI-generated chest X-ray reports. They compared the results of computer assessments against those of six experienced radiologists. Experts categorized errors in the reports into several types (false positive/negative result, incorrect finding location, misjudged finding severity, etc.).

It turned out that automatic scoring systems often misinterpreted AI-generated reports and sometimes overlooked errors in the clinical evaluation of the image. Therefore, the research team developed a new scoring algorithm (RadGraph F1) and a composite evaluation tool RadCliQ, which combines results from several different metrics into a single score that better matches radiologist assessments. Even with these new tools, researchers noted significant differences between their results and the best possible score.

Advancing to the Next Level

The reliability of scoring systems is crucial for the development of AI tools. The metrics tested in Harvard's analysis could not reliably identify errors in AI-generated reports, some of which were clinically significant. The newly developed scoring algorithms, provided as open-source codes by the analysis authors, should help with the development of the next generation of reporting tools so that future AI model results do not differ from those described by human experts. For now, however, it seems that AI will not replace radiologists anytime soon.

(este)

Sources:
1. Kwee T. C., Almaghrabi M. T., Kwee R. M. Diagnostic radiology and its future: what do clinicians need and think? Eur Radiol 2023 Dec; 33 (12): 9401–9410, doi: 10.1007/s00330-023-09897-2.
2. Yu F., Endo M., Krishnan R. et al. Evaluating progress in automatic chest X-ray radiology report generation. Patterns 2023 Aug 3; 4 (9): 100802, doi: 10.1016/j.patter.2023.100802.
3. Pesheva E. How good is that AI-penned radiology report? Harvard Medical School, 2023 Aug 3. Available at: https://hms.harvard.edu/news/how-good-ai-penned-radiology-report