Focus on RO

Focus on RO

Maintaining Relevance of Radiation Oncology Clinical Exams

By Paul E. Wallner, DO, ABR Associate Executive Director for Radiation Oncology, and Anthony Gerdeman, PhD, ABR Director of Exam Services


Paul Wallner, DO, Associate Executive Director for Radiation Oncology

A critical element in maintaining the credibility and validity of clinical certification exams is that they are deemed to be relevant by those involved, including developers and examinees. In a continuously evolving clinical care delivery milieu, maintenance of relevance can be challenging.

Prior to implementation of the ABR Continuing Certification Online Longitudinal Assessment (OLA) instrument in 2019, radiation oncology (RO) had relied primarily on tri-annual clinical practice analyses (CPAs) (1,2) and input from volunteers to inform decision-making related to the relevance of existing items, timeliness of adding emerging interventions, and elimination of those deemed to be outdated. The CPAs provided timely data related to the frequency that various tumor sites were seen within the clinical environment and how those disease sites were being treated. Thus, questions about what and how were answered. Although the CPAs provided useful and actionable information, the data was limited by the number of respondents to the optional surveys, the potential for information bias of respondents, and the infrequency of the surveys.

The OLA instrument for RO was introduced in January 2020. Each week, more than 3,000 individual items (questions) are answered by diplomates. The collated, disease-related information available from these responses will be provided to stakeholder organizations to assist in continuing medical education program development and is important to the ABR in identifying areas of knowledge gaps. In addition to providing responses to items, on average, 30% of respondents provide weekly answers to optional questions related to the relevance of the item to their practice, the importance of the topic, and the likelihood that a minimally competent radiation oncologist would get the item correct. The responses to relevance and importance provide guidance in the development of annual certification exam blueprints. The responses to whether the item would be answered correctly or not establish an Angoff score for the item by which all respondents are assessed (3,4,5,6).

The ABR also relies on requirements for residency training established by the Accreditation Council for Graduate Medical Education (ACGME) Radiation Oncology Review Committee  for guidance on what topics should be included in exam blueprints. Topics not considered essential to residency training, or as ongoing research, are generally excluded from candidate assessment (7).

  1. Wallner, PE, Yang, J, and Gerdeman, A. Clinical Practice Analysis and Radiation Oncology MOC Examination Development. Beam 7 (1). Spring 2014.
  2. Wallner, PE, McGeagh, AM, Gerdeman, AM et al. Snapshot of a specialty: results of the ABR 2016 radiation oncology clinical practice analysis. Jour Amer Coll Radiol.
  3. Wallner, PE, Gerdeman, AM and McGeagh, AM, Data-driven initial certification and maintenance of certification examination development. ASTROnews, Annual Meeting Issue Summer 2016, pp 64-65.
  4. Wallner, PE, and Ng, AK. OLA: Radiation oncology dips a toe in the water and likes the temperature. Beam. 2020;13(5):7.
  5. Wallner, PE, Segal, S, Laszakovits, DJ et al. The American Board of Radiology online longitudinal assessment Part 3 maintenance of certification instrument: rationale and summary of 6-month experience. Pract Radiat Oncol. 2020;10(6):386-388.
  6. Wallner, PE, Gerdeman, AM and McGeagh, AM, Data-driven initial certification and maintenance of certification examination development. ASTROnews, Annual Meeting Issue Summer 2016, pp 64-65.
  7. Wallner, PE and Giaccia, A. Integration of “new science” into ABR certification examination process. ASTROnews, Winter 2016; 19 (4): 36-37.

Return to The Beam


Help spread the word
© 2023 American Board of Radiology    |   
Privacy & Legal    |   
Americans with Disabilities Act (ADA) Site Map
The American Board of Radiology does not and shall not discriminate based on race, color, religion (creed), gender, gender expression, age, national origin (ancestry), disability, marital status, sexual orientation, or military status in any of its activities or operations. These activities include but are not limited to hiring and firing of staff, selection of volunteers, conducting committee meetings, and administering exams. We are committed to an environment free from discrimination, sexual harassment, and other unlawful forms of harassment. To report any actions of discrimination, sexual harassment, or other unlawful harassment, please contact Karyn Howard, Managing Director, at 520-790-2900 extension 2171 or you can call our confidential hotline at 844-280-0005.
Version: 3.0.57
The American Board of Radiology