Title: Computed Tomographic Colonography: Assessment of Radiologist Performance With and Without Computer-Aided Detection
Abstract: Background & AimsIn isolation, computer-aided detection (CAD) for computed tomographic (CT) colonography is as effective as optical colonoscopy for detection of significant adenomas. However, the unavoidable interaction between CAD and the reader has not been addressed. Methods: Ten readers trained in CT but without special expertise in colonography interpreted CT colonography images of 107 patients (60 with 142 polyps), first without CAD and then with CAD after temporal separation of 2 months. Per-patient and per-polyp detection were determined by comparing responses with known patient status. Results: With CAD, 41 (68%; 95% confidence interval [CI], 55%–80%) of the 60 patients with polyps were identified more frequently by readers. Per-patient sensitivity increased significantly in 70% of readers, while specificity dropped significantly in only one. Polyp detection increased significantly with CAD; on average, 12 more polyps were detected by each reader (9.1%, 95% CI, 5.2%–12.8%). Small- (≤5 mm) and medium-sized (6–9 mm) polyps were significantly more likely to be detected when prompted correctly by CAD. However, overall performance was relatively poor; even with CAD, on average readers detected only 10 polyps (51.0%) ≥10 mm and 24 (38.2%) ≥6 mm. Interpretation time was shortened significantly with CAD: by 1.9 minutes (95% CI, 1.4–2.4 minutes) for patients with polyps and by 2.9 minutes (95% CI, 2.5–3.3 minutes) for patients without. Overall, 9 readers (90%) benefited significantly from CAD, either by increased sensitivity and/or by reduced interpretation time. Conclusions: CAD for CT colonography significantly increases per-patient and per-polyp detection and significantly reduces interpretation times but cannot substitute for adequate training. Background & AimsIn isolation, computer-aided detection (CAD) for computed tomographic (CT) colonography is as effective as optical colonoscopy for detection of significant adenomas. However, the unavoidable interaction between CAD and the reader has not been addressed. Methods: Ten readers trained in CT but without special expertise in colonography interpreted CT colonography images of 107 patients (60 with 142 polyps), first without CAD and then with CAD after temporal separation of 2 months. Per-patient and per-polyp detection were determined by comparing responses with known patient status. Results: With CAD, 41 (68%; 95% confidence interval [CI], 55%–80%) of the 60 patients with polyps were identified more frequently by readers. Per-patient sensitivity increased significantly in 70% of readers, while specificity dropped significantly in only one. Polyp detection increased significantly with CAD; on average, 12 more polyps were detected by each reader (9.1%, 95% CI, 5.2%–12.8%). Small- (≤5 mm) and medium-sized (6–9 mm) polyps were significantly more likely to be detected when prompted correctly by CAD. However, overall performance was relatively poor; even with CAD, on average readers detected only 10 polyps (51.0%) ≥10 mm and 24 (38.2%) ≥6 mm. Interpretation time was shortened significantly with CAD: by 1.9 minutes (95% CI, 1.4–2.4 minutes) for patients with polyps and by 2.9 minutes (95% CI, 2.5–3.3 minutes) for patients without. Overall, 9 readers (90%) benefited significantly from CAD, either by increased sensitivity and/or by reduced interpretation time. Conclusions: CAD for CT colonography significantly increases per-patient and per-polyp detection and significantly reduces interpretation times but cannot substitute for adequate training. See editorial on page 2006. See editorial on page 2006. Computed tomographic (CT) colonography is a relatively novel health technology that is used to examine the large bowel. Specifically, it combines helical CT scanning of the cleansed and distended colorectum with complex image-rendering techniques that simulate the view obtained at conventional endoscopy, hence the alternative term "virtual colonoscopy."1Fenlon H.M. Nunes D.P. Schroy III, P.C. Barish M.A. Clarke P.D. Ferrucci J.T. A comparison of virtual and conventional colonoscopy for the detection of colorectal polyps.N Engl J Med. 1999; 341: 1496-1503Google Scholar Some comparisons with colonoscopy have suggested equivalent sensitivity and specificity when CT colonography is used to detect adenomatous polyps and invasive cancer in both symptomatic patients1Fenlon H.M. Nunes D.P. Schroy III, P.C. Barish M.A. Clarke P.D. Ferrucci J.T. A comparison of virtual and conventional colonoscopy for the detection of colorectal polyps.N Engl J Med. 1999; 341: 1496-1503Google Scholar, 2Yee J. Akerkar G.A. Hung R.K. Steinauer-Gebauer A.M. Wall S.D. McQuaid K.R. Colorectal neoplasia: performance characteristics of CT colonography for detection in 300 patients.Radiology. 2001; 219: 685-692Google Scholar, 3Pineau B.C. Paskett E.D. Chen G.J. Espeland M.A. Phillips K. Han J.P. Mikulaninec C. Vining D.J. Virtual colonoscopy using oral contrast compared with colonoscopy for the detection of patients with colorectal polyps.Gastroenterology. 2003; 125: 304-310Google Scholar and asymptomatic screenees.4Pickhardt P.J. Choi J.R. Hwang I. Butler J.A. Puckett M.L. Hildebrandt H.A. Wong R.K. Nugent P.A. Mysliwiec P.A. Schindler W.R. Computed tomographic virtual colonoscopy to screen for colorectal neoplasia in asymptomatic adults.N Engl J Med. 2003; 349: 2191-2200Google Scholar, 5Van Gelder R.E. Nio C.Y. Florie J. Bartelsman J.F. Snel P. De Jager S.W. Van Deventer S.J. Lameris J.S. Bossuyt P.M. Stoker J. Computed tomographic colonography compared with colonoscopy in patients at increased risk for colorectal cancer.Gastroenterology. 2004; 127: 41-48Google Scholar However, other studies have found it less effective.6Cotton P.B. Durkalski V.L. Pineau B.C. Palesch Y.Y. Mauldin P.D. Hoffman B. Vining D.J. Small W.C. Affronti J. Rex D. Kopecky K.K. Ackerman S. Burdick J.S. Brewington C. Turner M.A. Zfass A. Wright A.R. Iyer R.B. Lynch P. Sivak M.V. Butler H. Computed tomographic colonography (virtual colonoscopy): a multicenter comparison with standard colonoscopy for detection of colorectal neoplasia.JAMA. 2004; 291: 1713-1719Google Scholar, 7Rockey D.C. Paulson E. Niedzwiecki D. Davis W. Bosworth H.B. Sanders L. Yee J. Henderson J. Hatten P. Burdick S. Sanyal A. Rubin D.T. Sterling M. Akerkar G. Bhutani M.S. Binmoeller K. Garvie J. Bini E.J. McQuaid K. Foster W.L. Thompson W.M. Dachman A. Halvorsen R. Analysis of air contrast barium enema, computed tomographic colonography, and colonoscopy: prospective comparison.Lancet. 2005; 365: 305-311Abstract Full Text Full Text PDF Google Scholar It is well recognized that interpretation of images obtained on CT colonography is time consuming and fatiguing because of the volume of imaging data encountered by the radiologist. This has been implicated as a major cause of poor performance; it has been estimated that more than 13,000 individual images must be interpreted to identify a single 1-cm polyp.8Johnson C.D. Harmsen W.S. Wilson L.A. MacCarty R.L. Welch T.J. Iilstrup D.M. Ahlquist D.A. Prospective blinded evaluation of computed tomographic colonography for screen detection of colorectal polyps.Gastroenterology. 2003; 125: 311-319Google Scholar It is also well established that competent interpretation requires considerable skill and training, the lack of which has also been implicated when results have been disappointing.9Ferrucci J. Barish M. Choi R. Dachman A. Fenlon H. Glick S. Laghi A. Macari M. Morrin M. Paulson E. Pickhardt P.J. Soto J. Yee J. Zalis M. Working Group on Virtual ColonoscopyVirtual colonoscopy.JAMA. 2004; 292: 431-432Google Scholar, 10Halligan S. Taylor S.A. Burling D. Virtual colonoscopy.JAMA. 2004; 292: 432Google Scholar Computer-aided detection (CAD) has proved effective in situations where radiologists must detect small lesions that occur infrequently, namely screening mammography11Warren-Burhenne L.J. Wood S.A. D'Orsi C.J. Feig S.A. Kopans D.B. O'Shaughnessy K.F. Sickles E.A. Tabar L. Vyborny C.J. Castellino R.A. Potential contribution of computer aided detection to the sensitivity of screening mammography.Radiology. 2000; 215: 554-562Google Scholar and pulmonary nodules.12Awai K. Murao K. Ozawa A. Komi M. Hayakawa H. Hori S. Nishimura Y. Pulmonary nodules at chest CT: effect of computer-aided diagnosis on radiologists' detection performance.Radiology. 2004; 230: 347-352Google Scholar Researchers using small data sets have shown that CAD may also be effective for CT colonography,13Yoshida H. Masutani Y. MacEneaney P. Rubin D.T. Dachman A.H. Computerized detection of colonic polyps at CT colonography on the basis of volumetric features: pilot study.Radiology. 2002; 222: 327-336Google Scholar, 14Summers R.M. Johnson C.D. Pusanik L.M. Malley J.D. Youssef A.M. Reed J.E. Automated polyp detection at CT colonography: feasibility assessment in a human population.Radiology. 2001; 219: 51-59Google Scholar, 15Kiss G. Van Cleynenbreugel J. Thomeer M. Suetens P. Marchal G. Computer-aided diagnosis in virtual colonoscopy via combination of surface normal and sphere fitting models.Eur Radiol. 2002; 12: 77-81Google Scholar, 16Halligan S. Taylor S.A. Dehmeshki J. Amin H. Ye X. Tsand J. Roddie M.E. Computer-assisted detection for CT colonography: external validation.Clin Radiol. 2006; 61: 758-763Google Scholar and a recent large study of 1186 screening patients found that CAD detected colonic adenomas at 8- and 10-mm diameter thresholds with an efficacy not significantly different from colonoscopy.17Summers R.M. Yao J. Pickhardt P. Franaszek M. Bitter I. Brickman D. Krishna V. Choi R. Computed tomographic virtual colonoscopy computer-aided polyp detection in a screening population.Gastroenterology. 2005; 129: 1832-1844Abstract Full Text Full Text PDF Scopus (252) Google Scholar An accompanying editorial suggested that CAD might therefore improve radiologist sensitivity for polyps while simultaneously reducing expensive interpretation time.18Bond J.H. Progress in refining virtual colonoscopy for colorectal cancer screening.Gastroenterology. 2005; 129: 2103-2106Abstract Full Text Full Text PDF Scopus (16) Google Scholar However, the editorial also noted that this interaction between CAD and the reporting radiologist had not yet been addressed.18Bond J.H. Progress in refining virtual colonoscopy for colorectal cancer screening.Gastroenterology. 2005; 129: 2103-2106Abstract Full Text Full Text PDF Scopus (16) Google Scholar While some researchers have indirectly compared the performance of CAD and unaided observers when reading the same data set,19Taylor S.A. Halligan S. Burling D. Roddie M.E. Honeyfield L. McQuillan J. Amin H. Dehmeshki J. Computer-assisted reader software versus expert reviewers for polyp detection on CT colonography.AJR Am J Roentgenol. 2006; 186: 696-702Google Scholar such assessments will likely overestimate the benefit of CAD because there is no guarantee that observers will heed true-positive CAD prompts or indeed reject false-positive prompts.20Wagner R.F. Beiden S.V. Campbell G. Metz C.E. Sacks W.M. Assessment of medical imaging and computer-assist systems: lessons from recent experience.Acad Radiol. 2002; 9: 1264-1277Google Scholar Therefore, the purpose of our study was to determine the direct effects of CAD on performance characteristics when used by radiologists to interpret images obtained on CT colonography. Seven centers (4 US centers and 3 European centers) provided clinical data for the development of CAD software for CT colonography and/or its subsequent validation (Table 1). All centers had permission from their institutional review board (research ethics committee) to share existing CT data on the condition that data were made anonymous; 2 centers had such a waiver already in place, and 5 obtained it specifically for the purposes of this study.Table 1Number of Individual Patients Contributed by 4 US and 3 European Centers to the Development and Validation Data Sets for the CT Colonography CAD SoftwareCenterDevelopment data (no. of patients)Validation data (no. of patients)Patient typeCT detector-rowCollimation (mm)Reconstruction interval (mm)mAsInsufflation gasSpasmolyticPolypsNormalUSA 1742925Screening4, 82.51100AirNoUSA 27310Screening4, 82.51100AirNoUSA 334177Symptomatic42.51.25120AirNoUSA 4030Symptomatic81.25, 2.51.2, 2.150, 100, 200AirNoEUR 14400Symptomatic41.25, 2.50.75, 1.2550, 100co2YesEUR 21440Both41, 30.8, 135AirNoEUR 30915Symptomatic831.6230AirYesTotal2396347NOTE. There were no patients without polyps in the development set. Patient type and technical parameters for CT colonography are shown. Open table in a new tab NOTE. There were no patients without polyps in the development set. Patient type and technical parameters for CT colonography are shown. Patients with inherited polyposis syndromes were excluded, as were patients with cancer (because detection was to be focused on polyps). All patients were known to have colorectal polyps, proven by intraindividual, same-day total colonoscopy, which immediately followed CT colonography. Patients were a mix of both symptomatic and asymptomatic screening patients, reflecting the clinical practice at the submitting centers (Table 1). All patients underwent full bowel preparation with the aim of completely clearing the colon of residue, and CT colonography was performed with multidetector row machines, either 4-detector or 8-detector row (Table 1). Technical parameters for CT colonography are given in Table 1. Both prone and supine scanning data were acquired. There was no attempt to select only technically optimal cases because this would not reflect normal clinical practice and to do so would limit the generalizability of our findings. Colonographic DICOM data for each patient were made anonymous, written to compact disk, labeled with a unique identifier, and sent to a data manager along with details of the clinical indication for the study, the technical parameters used, the original radiologist's report, the findings at subsequent colonoscopy, and the ultimate histologic diagnosis for each polyp removed. To arrive at a reference standard against which the performance of the CAD software could be judged during both the development and the test phase, each case was read in face-to-face consensus by 2 of a panel of 3 radiologists with a subspecialty interest in gastrointestinal radiology and experience in interpretation of CT colonography images (minimum 200 endoscopically validated cases each). Readers attempted to identify polyps on paired prone and supine studies with the assistance of the radiologic, endoscopic, and pathologic reports. These data also helped match polyps with those found at endoscopy, especially in patients with more than one polyp. Readers looked for all polyps irrespective of size. Because it is known that colonoscopy is a fallible reference standard,4Pickhardt P.J. Choi J.R. Hwang I. Butler J.A. Puckett M.L. Hildebrandt H.A. Wong R.K. Nugent P.A. Mysliwiec P.A. Schindler W.R. Computed tomographic virtual colonoscopy to screen for colorectal neoplasia in asymptomatic adults.N Engl J Med. 2003; 349: 2191-2200Google Scholar, 5Van Gelder R.E. Nio C.Y. Florie J. Bartelsman J.F. Snel P. De Jager S.W. Van Deventer S.J. Lameris J.S. Bossuyt P.M. Stoker J. Computed tomographic colonography compared with colonoscopy in patients at increased risk for colorectal cancer.Gastroenterology. 2004; 127: 41-48Google Scholar, 21Rex D.K. Cutler C.S. Lemmel G.T. Rahmani E.Y. Clark D.W. Helper D.J. Lehman G.A. Mark D.G. Colonoscopic miss rates of adenomas determined by back-to-back colonoscopies.Gastroenterology. 1997; 112: 24-28Google Scholar each observer also read each data set in its entirety twice to search for polyps that may have been missed by colonoscopy. This procedure was performed following statistical advice so that the CAD software would not be unduly penalized, especially because segmental unblinding3Pineau B.C. Paskett E.D. Chen G.J. Espeland M.A. Phillips K. Han J.P. Mikulaninec C. Vining D.J. Virtual colonoscopy using oral contrast compared with colonoscopy for the detection of patients with colorectal polyps.Gastroenterology. 2003; 125: 304-310Google Scholar, 4Pickhardt P.J. Choi J.R. Hwang I. Butler J.A. Puckett M.L. Hildebrandt H.A. Wong R.K. Nugent P.A. Mysliwiec P.A. Schindler W.R. Computed tomographic virtual colonoscopy to screen for colorectal neoplasia in asymptomatic adults.N Engl J Med. 2003; 349: 2191-2200Google Scholar had not been used by all centers submitting data. However, when such a polyp was encountered, it was only considered true positive if it demonstrated typical characteristics and both observers had no doubt. Interpretation was performed using Food and Drug Administration–approved CT colonography visualization software (MedicColon 1.2, Medicsight PLC, London, England) that displayed adjacent axial prone and supine examinations, along with multiplanar reformatting and a 3-dimensional rendered endoluminal cube for problem solving. Reading was performed using adjacent workstations in a quiet environment remote from clinical activity. When a polyp was identified, its segmental location was noted along with the CT scan coordinates (for both prone and supine studies if visible on both), the maximal transverse diameter was measured using software calipers (using the appropriate multiplanar reformat), its boundary was identified using a mouse and cursor, and the resultant image was saved as a binary image file. Observers also noted their consensus opinion of the ease of visualization of each individual polyp using a 3-point scale: easy to see, neither easy nor difficult to see, and difficult to see. Polyps identified by colonoscopy that could not be identified on CT scan were discarded because their coordinates were not available to the software developers for assessment of software performance. The data manager divided data from 4 centers into 2 sets for separate CAD development and validation (Table 1). For 3 centers, partitioning was 2:1 in favor of development data, achieved by assigning the first two thirds of chronologically consecutive data to the development set and the rest to the validation set. This procedure was used instead of randomization because randomization creates 2 sets that differ only in terms of chance variation, posing a weaker challenge for software during subsequent validation.22Altman D.G. Royston P. What do we mean by validating a prognostic model?.Stat Med. 2000; 19: 453-473Google Scholar, 23Hirsch R.P. Validation samples.Biometrics. 1991; 47: 1193-1194Google Scholar A fifth center contributed data solely to the development set and a sixth contributed only one patient to the validation set, while a seventh contributed only to the validation set, thus limiting center crossover between the 2 sets (Table 1). Ultimately, the development set contained data from 239 patients with 494 polyps (mean diameter, 6.7 mm; median, 5 mm; range, 1–50 mm). These data were used to develop the CAD software as follows. The endoluminal colonic surface was extracted from each CT acquisition using a thresholding-based method, and a mathematical algorithm was applied that aimed to detect raised endoluminal objects, all of which were regarded as potential polyps. A sphericity filter was then applied that aimed to facilitate discrimination between real polyps and false-positive prompts. To this end, every voxel on the candidate surface was analyzed to determine whether or not it and its neighbors formed part of a sphere.24Dehmeshki J, Halligan S, Taylor SA, Roddie RE, McQuillan J, Honeyfield L, Amin H. Computer assisted detection software for CT colonography: effect of sphericity filter on performance characteristics for patients with and without fecal tagging. Eur Radiol (Epub online Oct 5, 2006).Google Scholar With the sphericity filter set at 1.0, only those voxels that formed part of a perfect sphere were retained as CAD prompts; others were dismissed as likely false positives. Reducing the filter toward zero allowed retention of voxels that formed part of an increasingly imperfect sphere (eg, a flattened oval). Case-by-case comparison with the reference polyp coordinates established previously by the experienced observers allowed the number of true- and false-positive prompts at each sphericity setting to be calculated. To be true positive, the center of the CAD prompt had to lie within the polyp circumference identified on the binary image file by the experienced observers. Other prompts were assigned false positive, even when in the same colonic segment and immediately adjacent to a validated polyp. Leave-one-out analysis22Altman D.G. Royston P. What do we mean by validating a prognostic model?.Stat Med. 2000; 19: 453-473Google Scholar of the 239 patients was used to determine satisfactory CAD performance, defined by per-polyp sensitivity of 0.85 for polyps ≥5 mm (at which the mean false-positive rate was 33). When this was achieved, the development phase was terminated and validation commenced. The data manager chose 60 patients randomly from the accumulated validation data whose polyps had been validated using methodology identical to that used for development data. No data were shared between development and validation sets. Six centers contributed validation data, 4 of which had contributed development data (Table 1). There were 14 patients with a polyp ≥10 mm and 40 patients with a polyp ≥6 mm. Twenty-six patients had polyps whose maximum diameter lay between 6 and 9 mm. There were 142 polyps overall: 25 patients had 1 polyp, 13 patients had 2, 12 patients had 3, 3 patients had 4, 3 patients had 5, 3 patients had 6, and 1 patient had 10. Median maximal polyp diameter was 5 mm (mean, 6.2 mm; range, 2–25 mm). There were 62 polyps ≥6 mm and 19 polyps ≥10 mm. These patients were supplemented by a further 47 patients without polyps, obtained from 3 participating centers, 2 of which had contributed development data (Table 1). All 47 were found to be negative for polyps both by same-day colonoscopy and also by CT colonography using the same experienced readers who had evaluated the patients with polyps. Again, there was no attempt to select only technically optimal cases. We speculated that the biggest audience for CAD was likely to be fully trained radiologists familiar with independently reporting conventional abdominal CT scans in daily clinical practice but relatively unfamiliar with CT colonography. We hypothesized that experienced colonography readers were less likely to benefit from CAD, and we did not envisage a role for CT novices. With this target audience in mind, we selected 10 fully trained radiologists (defined by having passed the Fellowship of the Royal College of Radiologists and having achieved the Certificate of Completion of Specialist Training, which is equivalent to board certification). The 10 radiologists read a review article25Taylor S. Halligan S. Bartram C.I. CT colonography: methods, pathology and pitfalls.Clin Radiol. 2003; 58: 179-190Google Scholar to familiarize themselves with CT colonography and principles of interpretation (although all were aware of the technique in advance of the study). Observers were aware that the study aimed to evaluate CAD for CT colonography. They were unaware of the prevalence of abnormality in the validation data set other than being told that some patients had normal findings and some had abnormal findings. They were familiarized with the same software previously used by the experienced readers to define the reference standard and then used this to read the images for the 107 validation patients. Patient order was randomized across readers. Reading was performed over several days in a quiet environment remote from clinical activity, and a coordinator was available for technical assistance at all times. Readers were unaware of each other's responses. There was no CAD assistance. Observers were asked to identify all polyps irrespective of size and recorded their findings on a study sheet along with the unique patient identifier. When a polyp was identified, its segmental location was noted along with the CT scan coordinates and its maximal transverse diameter, estimated by software calipers. Readers also recorded their diagnostic confidence using a 100-point scale, with 100 being most confident that each documented polyp was real. Each reader assigned an overall per-patient category to each patient (either "normal" or "abnormal"), again using a 100-point scale. Interpretation time for each patient was noted. After a temporal separation of 2 months to diminish recall bias, the images for the 107 patients were interpreted again under identical circumstances and using identical software with the single addition of the CAD algorithm (ColonCAR; Medicsight PLC). Observers were told that the CAD system detected real polyps and also made false-positive prompts but were told nothing about sensitivity or specificity at different polyp diameters. Readers were told they had freedom to ignore CAD prompts if they wished. CAD annotated and unannotated images were presented concurrently. Unannotated axial images were displayed on the left of the screen with corresponding annotated images on the right, prone above supine. Prompts were small circles with their epicenter at the peak of the perceived polyp. We wished the CAD software to identify the majority of polyps, even when small. On the basis of the development phase, a sphericity value of 0.75 identified 85% of polyps ≥5 mm with an average false-positive rate of 33, and this was the sphericity value used for the validation phase. Patient order was again randomized across readers, and observers were unaware of their previous or each other's responses. Data identical to the unassisted phase were collected. Because each patient may have more than one polyp (or be normal), both per-patient and per-polyp analyses were performed. Datasheets from the unassisted and CAD-assisted reads were marked against the reference standard for each patient to determine whether polyps recorded by observers were true or false positive and the number of false-negative polyps. True-negative categorization was judged on a per-patient basis. Findings with confidence scores were tabulated on a spreadsheet (Microsoft Excel; Microsoft Corp, Redmond, WA) and imported into Stata for analysis (version 8.0; Stata Corp, College Station, TX). Each polyp was assigned a unique identifier based on the polyp and patient number according to expert consensus. True-positive polyps were matched across all readers. In per-patient analyses, patients were grouped according to their largest polyp size. Sensitivity was defined as the proportion of true positives correctly identified and specificity as the proportion of true negatives correctly identified. Specificity was only calculated on a per-patient basis. Ninety-five percent confidence intervals (CIs) were calculated using the exact binomial method or by paired proportion methods for changes in sensitivity and specificity. Summary receiver operating curves were fitted using PROPROC26Pan X. Metz C.E. The "proper" binormal model: parametric receiver operating characteristic curve estimation with degenerate data.Acad Radiol. 1997; 4: 380-389Google Scholar for each reader, with and without CAD, based on the maximum confidence score assigned to polyps identified in each patient. Partial area under the curve values for a realistic range of specificity (75%–100%),27Jiang Y. Metz C.E. Nishikawa R.M. A receiver operating characteristic partial area index for highly sensitive diagnostic tests.Radiology. 1996; 201: 745-750Crossref Scopus (240) Google Scholar calculated from PROPROC, were used to compare performance across different test thresholds, for readers with and without CAD, using a paired t test. Because of the high specificity of the readers in this study, which causes degeneracy in the data, it was not possible to fit multireader, multicase models.28Obuchowski N.A. Beiden S.V. Berbaum K.S. Hillis S.L. Ishwaran H. Song H.H. Wagner R.F. Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods.Acad Radiol. 2004; 11: 980-995Abstract Full Text Full Text PDF Scopus (82) Google Scholar CAD correctly detected at least one polyp in 45 (75%) of the 60 patients with polyps: 13 (92.9%) of the 14 patients with a polyp ≥10 mm and 37 (92.5%) of the 40 patients with a polyp ≥6 mm. Overall, CAD detected 76 (53.5%) of the 142 polyps: 17 (89.5%) of 19 polyps ≥10 mm, 49 (79%) of 62 polyps ≥6 mm, and 27 (33.8%) of 80 polyps ≤5 mm. The average false-positive rate was 11.6 per patient overall (range, 0–170): 6.7 for patients with polyps and 16.4 for patients without polyps. Per-patient categorization, sensitivity, and specificity for the 10 readers with and without CAD are shown in Table 2. Of the 60 patients with polyps, 55 (92%) were identified by at least one reader when unassisted by CAD. Of the 47 patients without polyps, all were correctly classified as negative by at least 5 readers when unassisted by CAD. Only 5 (10.6%) of the 47 patients without polyps attracted more than one false-positive categorization by the unassisted readers.Table 2Per-Patient Categorizations, Sensitivity, and Specificity (for Polyps of All Sizes Combined) for 10 Readers With and Without CAD AssistanceReader no.CADTrue positiveFalse negativeTrue negativeFalse positiveSensitivityDifference in sensitivity assisted – unassisted (95% CI)SpecificityDifference in specificity assisted – unassisted (95% CI)Large polyps (≥1 cm) (n = 14)Medium polyps (6–9 mm) (n = 26)Small polyps (≤5 mm) (n = 20)1Unassisted1017330443500 (−15, 15)944 (−5, 15)Assisted101463046150982Unassisted980434522813 (−2, 28)96−6 (−18, 5)Assisted101143542542893Unassisted10156294525222 (6, 36