Content Development for Radiation Oncology Qualifying and Certifying Exams
By Paul E. Wallner, DO, ABR Associate Executive Director for Radiation Oncology; Brian J. Davis, MD, PhD, ABR Trustee; and Anthony M. Gerdeman, PhD, ABR Director of Exam Services
2022;15(4):9
Developing credible certification assessment instruments requires adhering to three foundational elements: security, validity, and relevance. Exam security is determined by the logistical arrangements of individual exam administrations, available technology, and pragmatic cost decisions. Metrics of validity are strictly determined by established psychometric principles and policies initially promulgated in 1966, as Standards for Educational and Psychological Testing, developed jointly by the American Psychological Association, the American Educational Research Association, and the National Council on Measurement in Education.1 Issues of relevance are not so clearly defined.
The Accreditation Council for Graduate Medical Education (ACGME) Radiation Oncology Review Committee (RO RC) is responsible for developing post-graduate training requirements in RO, including the attainment by trainees of specific skills, knowledge, and competencies that they deem to be essential. As guidance for the development of relevant assessment instruments, those requirements are sometimes generally defined. The RO RC requires that, “The program must provide instruction in the following areas: three-dimensional conformal radiation therapy; intensity-modulated radiation therapy; image-guided radiation therapy; stereotactic radiosurgery; stereotactic body radiotherapy; particle therapy; and concurrent chemo-radiotherapy,” among others.2
The nature of the inclusions provides little guidance to exam developers of relative clinical importance and current practice patterns. Critical emerging issues in RO, such as hypofractionation, various modalities available for image-guided therapy, and proton beam therapy are not specifically addressed, nor is the weighting of their impact on modern patient care. The issue of impact becomes increasingly important in areas such as brachytherapy and pediatric malignancies, where radiation intervention may be declining, and metastatic disease, especially oligometastases, where it is increasing.
Development of clinical material for ABR RO exams is carried out by volunteer committee members assigned to submit written items (questions) for the computer-based qualifying exams (QEs) and clinical cases for the oral certifying exams (CEs). Members of these committees are recognized content experts in their selected sites. They are appointed with consideration of geographic, gender, and ethnic diversity to avoid entering bias or institutional and regional practices into the exams. Item and case development assignments are based on adherence to previously constructed exam blueprints.
Prior to 2006, the blueprints were created by ABR trustees, committee chairs, and staff, based on review of the contemporary literature and consensus. In 2006, the ABR initiated a series of triannual clinical practice analysis (CPA) surveys designed to provide data to better inform blueprint development. These widely distributed surveys sought information related not only to what diseases RO diplomates were managing, but also to how they were managing them. CPA survey distribution continued for RO through 2020.3,4
Since 2020, when the Continuing Certification Online Longitudinal Assessment (OLA) platform was launched for RO, information related to the importance and relevance of specific pathologies and interventions has been collected on a weekly basis from more than 1,000 diplomates who respond to OLA evaluation questions. Using this new data source, ABR trustees have altered exam blueprints in a number of ways. In recognition of the limited number of pediatric cancers seen and managed by the majority of radiation oncologists, the amount of pediatrics content on the QE and CE has been reduced. Brachytherapy content also has been reduced as volumes have declined. As the clinical penetration of image-guided radiation therapy (IGRT) and hypofractionation has increased, exam content related to these areas has increased.
ABR trustees must also determine when to include emerging interventions and technologies into the exam blueprints. In this regard, consideration of the ACGME requirements and current literature is helpful, but sometimes additional information is necessary. Because of the RO RC general reference to “particle therapy,” the ABR had little guidance related to including clinical items and case material employing proton beam radiation (PBT). Absent this information, trustees had avoided including this material in clinical exam content inventories. The RO RC has considered adding a clarifying FAQ to its website indicating that “particle therapy” specifically includes PBT (personal communication, ACGME RO RC), and a poll of training programs conducted by the ABR in March 2022 revealed that a majority of RO trainees now have both clinical and didactic exposure to PBT. Thus, additional PBT content will be added to the QE and CE content inventories.
Using guidance from scientific, educational, and diplomate sources, the ABR RO trustees continue to strive to ensure the relevance of exams. As with the other foundational elements security and validity, relevance is dynamic in nature and will be altered as appropriate.
- American Psychological Association Standards for Educational and Psychological Testing. https://www.apa.org/science/programs/testing/standards Accessed June 6, 2022.
- Accreditation Council for Graduate Medical Education program requirements for post-graduate training in radiation oncology. https://www.acgme.org/globalassets/pfassets/programrequirements/430_radiationoncology_2021.pdf Accessed June 22, 2022.
- Becker, GJ, Bosma, JL, Guiberteau, MJ et al. ABR examinations: the why, what, and how. Int J Radiat Oncol Biol Phys. 2013; 87(2): 237-245.
- Wallner, PE, McGeagh, AM, Gerdeman, AM et al. Snapshot of a specialty: results of the ABR 2016 radiation oncology clinical practice analysis. Jour Amer Coll Radiol. 2019; 16 (4): 513 – 517.
