Aggregate Exam Results Report Allows Residency Review Committee to Evaluate Programs
By Donald J. Flemming, MD, ABR Governor, and Brooke Houck, PhD, ABR Psychometrician
2023;16(1):5
Quality graduate medical education outcomes are tightly tied to excellent feedback for individuals and residency programs. External feedback, such as outcomes from secure, externally administered and psychometrically sound exams, provides valuable objective data about program quality to residency programs. In fact, these outcomes are so important that all ACGME-accredited core residency programs must formally assess the results of board exams on an annual basis, and underperforming programs can receive a citation from their respective Review Committee for not meeting ACGME standards.
The ABR provides residency programs a pass/fail report for all initial certification exams at the same time that residents and program graduates receive their individual results. The report for the qualifying exams is more detailed than the certification exam because of the comprehensive nature of the exam and its program accreditation implications: the Radiology Review Committee uses only the qualifying exam board pass rate in its evaluation of residency programs. In the past, program directors were provided detailed data about the exam results for individual residents. The Board of Governors voted to discontinue this practice in 2022 to meet the spirit and intent of ABR bylaws. Candidates, however, continue to receive detailed feedback about their individual performances. Residents who fail the qualifying exam may choose to provide their exam report to mentors assisting them with remediation efforts.
Aggregate exam results give programs the opportunity to improve resident education and to be compliant with ACGME program requirements. A program’s board exam pass rate must be above the bottom fifth percentile for first-time test-takers over the previous three-year period OR 80% of its graduates must have passed over the last three years to be ACGME compliant. A program can therefore determine if it is at risk for a citation by noting whether 80% or more of its graduates have passed the qualifying exam over a three-year period. Programs that do not meet this requirement may still be above the fifth percentile standard but would need to contact the ACGME for this information.
In addition to board pass rate metrics, the annual report provides outcomes by specialty and modality that are reported as a percentage variance against mean scaled score. For example, the national average is subtracted from a program’s average; the difference between the two is the percentage variance (the distance between the national average and the program average over time). A scaled score is necessary to make results equivalent across the different forms that were administered for exam security. The reported variance for a given specialty may vary from year to year, but consistently low scores in a specific domain may trigger an evaluation of the education effort in that specialty.
The intent of the annual aggregate qualifying exam report is to provide diagnostic radiology residencies with an objective external metric of program performance. It is one of many data points that a program must use annually to assess program effectiveness, but it is certainly an extremely important outcome that a residency must consider given its potential impact on recruitment and compliance with ACGME program requirements. Future changes in this report will be tailored to feedback that the ABR receives from program directors to support our mutual goal of ensuring that our diplomates provide safe and effective care to the benefit of our patients.
