Maintaining Relevance in RO Exam Development
By Paul E. Wallner, DO, ABR Associate Executive Director for Radiation Oncology, and Anthony M. Gerdeman, PhD, ABR Director of Exam Services
2022;15(6):8
To maintain credibility, any organization responsible for evaluating knowledge and skills of clinical or scientific domains must be prepared to assure those being tested of the relevance of the assessment instruments employed. Absent specific, granular definitions of relevant subject matter by involved educators, the final determination of relevance remains somewhat subjective. The challenge then becomes developing exam content material that represents appropriate coverage of the specific domains.
Requirements for postgraduate training in radiation oncology are defined by the Accreditation Council for Graduate Medical Education (ACGME) Radiation Oncology Review Committee (RO RC). In its program requirements for approved training in RO, the RO RC specifies only broad domains, indicating that, “Residents must have experience with lymphomas and leukemias; breast, central nervous system, gastrointestinal, genitourinary, gynecologic, head and neck, lung, pediatric, skin, and soft tissue and bone tumors; and treatment of benign diseases for which radiation is utilized.”
For procedural requirements, the RO RC specifies only that, “The program must provide instruction in the following areas: three-dimensional conformal radiation therapy; intensity-modulated radiation therapy; image-guided radiation therapy; stereotactic radiosurgery; stereotactic body radiotherapy; concurrent chemo-radiotherapy; intra-operative radiation therapy; radioimmunotherapy; unsealed sources; total body irradiation therapy as used in stem-cell transplantation; total skin radiation therapy; high- and low-dose rate brachytherapy; and, particle therapy.”1
Although arguably all-inclusive of the discipline, these requirements provide no guidance related to weighting of clinical importance or addition/deletion of content, necessary details for the development of assessment instruments for Initial Certification or Continuing Certification (formerly MOC).
Development of questions for each of the three computer-based Qualifying Exam parts, the oral Certifying Exam, and Online Longitudinal Assessment (OLA) is based on blueprints created specifically for each instrument. These blueprints guide exam developers in the specific questions needed for each exam, and the weighting (specific number of questions) to be included.
Prior to 2006, clinical exam blueprints were developed by consensus among RO trustees, committee chairs, and senior ABR staff based on their understanding of current practice and literature. In 2006, the ABR initiated a series of triannual clinical practice analysis (CPA) surveys designed to better inform exam developers about what diseases radiation oncologists were managing in the field, and what techniques they were employing. This information provided real-world insight into the contemporary practice of RO.2 In 2020, the implementation of OLA for RO diplomates replaced the CPA surveys and now provides dynamic insight into similar information. Each week, more than 3,700 RO diplomates provide responses related to relevance of the subject. Since OLA implementation in 2020, more than 600,000 of these data points have been accumulated. This data source has led to a reduction in exam content related to the care of pediatric cancers and brachytherapy, as it became apparent that radiation oncologists were less involved in these areas. Similarly, content has been added regarding emerging areas of treatment management such as image-guided radiation therapy (IGRT), adaptive radiation therapy (ART), and the management of metastatic disease.
The information obtained from OLA provides excellent insight into the reduction in utilization of existing technologies and techniques but offers only limited information regarding new and emerging interventions such as proton beam therapy (PBT). IC exams are designed to assess the knowledge and skills attained by residents during their training. Absent more granular ACGME guidance, the ABR uses alternative means to determine when to begin adding exam content related to emerging practices. To aid in making this decision regarding PBT, the ABR collaborated with the Association of Directors of Radiation Oncology Programs (ADROP) in spring 2020 to assess current resident training in PBT. That survey indicated that > 65% of RO trainees now have required clinical PBT rotations and > 83% have didactic exposure. Based on these findings, PBT questions will be added to the IC clinical exam inventory in 2023. PBT material had previously been approved for inclusion in the Radiation and Cancer Biology (RCB) and Medical Physics for Radiation Oncology (MP) Qualifying Exam parts.
In 2009, the American Society for Radiation Oncology (ASTRO) promulgated a physics core curriculum that significantly aided MP blueprint development. ASTRO withdrew from support of this activity in 2018. At that time, the ABR assumed the responsibility of convening annual work groups of stakeholder organization representatives to assist in development of IC exam blueprints and study guides in MP and RCB.
ABR RO trustees and clinical exam content providers must be active in clinical practice, and the physicists and physician-scientists who submit material for the MP and RCB Qualifying Exam parts must remain active in teaching and clinical activities. These dedicated ABR volunteers form an additional level of review of the continued relevance of exam content and weighting.
References
- Accreditation Council for Graduate Medical Education Program Requirements for Training in Radiation Oncology. https://www.acgme.org/globalassets/pfassets/programrequirements/430_radiationoncology_2022v2.pdf Accessed November 8, 2022.
- Wallner, PE, McGeagh, AM, Gerdeman, AM et al. Snapshot of a specialty: results of the ABR 2016 radiation oncology clinical practice analysis. Jour Amer Coll Radiol. 2019; 16 (4): 513 – 517.
- Burmeister, J, Chen, Z, Chetty, IJ et al. The American Society for Radiation Oncology’s 2015 core physics curriculum for radiation oncology residents. Int J Radiation Oncol Biol Phys, 2016; 95 (4): 1298-1303
- Burmeister, J, Buatti, J, Jimenez, R, et al. Exam preparation and performance reporting changes for the American Board of Radiology radiation oncology physics examination: results from the ASTRO workgroup. Int J Radiation Oncol Biol Phys, 2020; 106 (1): 43-44
