An audit to assess the quality of artificial intelligence (AI)/computer aided detection (CAD) pulmonary nodule software in CT screening

This audit is intended to assess the quality of artificial intelligence/computer-aided detection software in assisting pulmonary nodule management in the setting of lung health check programmes.

The audit is designed to be a retrospective audit (with an experienced radiologist auditing the CAD detected nodules). It could also be done as a prospective study at the time that the reporting is taken place. This may depend on local preference.

The National Health Service Targeted Lung Health Check (TLHC) protocol recommends the implementation of standardised acquisition and reconstruction parameters for TLHC low dose CT scans, as described in the National Health Service Standard Protocol[1] and Quality Assurance Standards[2]. Management of pulmonary nodules should be based on the British Thoracic Society guidelines for the Investigation and Management of Pulmonary Nodules[3].[JH1] 

Computer-aided detection software is recommended to assist reporting radiologists in the detection of actionable nodules and, where possible, volumetric software should be used to guide clinical decisions. If computer aided detection systems are used, they should only be used in a concurrent or second reader format.

The standard: 

1) A false positive rate of <2 per case is desirable for CAD systems.

2) Performance (sensitivity/ false positives) of CAD/AI software should be within the expected range claimed by the manufacturer.

Target: 

1) 100% of scans audit to have a false positive rate <2 per case.

2) 100% of scans to be within the expected range claimed by the manufacturer.

Indicators: 

1) Percentage of scans containing radiologist-confirmed nodules ≥5mm (data calculation: b / a)  [=radiologist recall rate].

2) Percentage of radiologist-confirmed nodules which were detected by AI/CAD (d / c)  [=CAD/AI sensitivity]Percentage of scans with a false positive rate <2 per case.

3) Percentage of scans correctly characterised as solid /subsolid.

Data items to be collected: 

a) Total number of scans

b) Number of scans containing radiologist-confirmed nodules

c) Total number of radiologist-confirmed nodules

d) Is the nodule correctly identified by CAD as solid /sub-solid?

e) Number of radiologist-confirmed nodules correctly detected by CAD/AI (true positive rate or sensitivity) 

f) Number of scans containing false positives (=nodules identified by CAD/AI but reviewed by radiologist and described as false positive)

g)  Number of false positive nodules per scan

h) Name and version of AI/CAD

Suggested number: 

50 examinations

1) Present and discuss findings at local Lung Health Check meetings.

2) Report findings to the AI/CAD supplier, discuss possible solutions for quality improvement and plan for a quality improvement cycle.

1) List of eligible scans [may require support from PACS administration staff]2) Data collection from PACS (and RIS if required) [radiology trainee or equivalent]3) Identification of false positive nodules in each scan [by an experienced radiologist]4) Analysis and interpretation of data [radiology trainee and review by consultant]

1. [1] Standard Protocol prepared for the Targeted Lung Health Checks Programme, NHSE https://www.england.nhs.uk/publication/targeted-screening-for-lung-cancer/

2. [2] Quality Assurance Standards prepared for the Targeted Lung Health Checks Programme, NHSE https://www.england.nhs.uk/publication/targeted-screening-for-lung-cancer/

3. [3] British Thoracic Society guidelines for the investigation and management of pulmonary nodules: accredited by NICE https://thorax.bmj.com/content/70/Suppl_2/ii1

Peter Atiiga

Co-authors

Lizzie Barclay

Oliver Byass

Nadia Jawad