OLA: Radiation Oncology Dips a Toe in the Water and Likes the Temperature
by Paul E. Wallner, DO, FASTRO, Associate Executive Director for Radiation Oncology, and Andrea K. Ng, MD, MPH, ABR Trustee
2020;13(5):7
In 1994, after awarding the final non-time-limited (aka lifetime) certificates in radiation oncology (RO), the ABR enrolled all new RO diplomates in its Maintenance of Certification (MOC) program. It was anticipated that this novel educational assessment program would evolve over time, and such was the case. Although the basic elements of the program remain unchanged, each of its four essential parts has been updated to better synchronize with diplomates’ practice patterns and to avoid unnecessary duplication of effort or documentation. Documentation changes for Part 1 (Professionalism and Professional Standing) and participation options for Part 2 (Lifelong Learning and Self-Assessment) were implemented early in the programs. Alternative means to complete Part 4 (Improvement in Medical Practice) were also adopted.1,2
Throughout implementation of MOC, the concern most frequently cited by diplomates was related to Part 3 (Assessment of Knowledge, Judgment, and Skills). Because the initial assessment exam was administered every 10 years, it was perceived to be high risk and high stakes, requiring significant preparation. Because there are no RO subspecialties recognized by the Accreditation Council on Graduate Medical Education (ACGME), RO certification covers all aspects of the specialty, and the Part 3 exam included content across the spectrum of the discipline. Many diplomates no longer practiced all areas of RO and perceived this broad-based content to be worrisome. Sensitive to this issue, the ABR developed a modular MOC Part 3 exam in 2016. While maintaining some general content, this instrument allowed diplomates to select modules more consistent with their areas of interest or expertise. Exam questions continued to be prepared by the same category content experts who prepared material for IC exams.3
In 2017, in keeping with expectations of legislators and regulators, and consistent with evolving American Board of Medical Specialties (ABMS) policies, the ABR Governors and Trustees announced a decision to eliminate the 10-year exam cycle and transition to a more continuous tool that was to be web based and accessible to diplomates on a variety of hardware platforms. No MOC exams were administered while the new instrument was being developed. In the absence of a satisfactory commercially available assessment instrument that met all the ABR requirements, a decision was made to develop the new tool in-house. Designated as the ABR Online Longitudinal Assessment (ABR OLA) platform, the process encountered numerous development questions. To address these issues, stakeholder organizations were consulted regarding content level, frequency distribution of questions, and information to be provided with responses; web developers were queried to optimize displays; psychometricians weighed in on question rating and performance assessment methods; and input from clinical category committees was sought to develop required content.
RO category committee volunteers represent nationally recognized content experts in eight general clinical subject areas. To create an inventory of the more than 200 items needed to launch the new instrument, cohorts of these experts were assigned to review the existing MOC question pool, repurpose some previously used material, and develop new content. It soon became apparent that the available content was often more esoteric than was appropriate for many RO diplomates whose practices were general in nature. Thus, prior to the actual launch of OLA for RO, a new content development committee consisting almost entirely of early and mid-career volunteer diplomates with general RO practices was empaneled. The target for item difficulty level became “walking-around knowledge,” loosely defined as material that should generally be known to all diplomates without the need to consult reference sources.
After almost three years of development and beta testing, OLA was launched for diagnostic radiology (DR) in January 2019, and for RO, interventional radiology/diagnostic radiology (IR/DR), and medical physics (MP) in January 2020. As had been anticipated, a majority of RO MOC participants started using the platform within the first month (2697/81%), and by the second month of availability, 3,114 RO diplomates were responding to questions on a regular basis. Participants receive immediate feedback on their responses, as well as a rationale for the correct answer and a reference for further review, as appropriate. If a response is incorrect, a variant testing the same issue is sent out within three to six weeks.
Between January 6 and June 30, 2020, 3,329 diplomates completed 141,379 individual questions. In addition to evaluations of the importance and relevance of the material to personal practice, 2,030 free-text responses were received. A majority of responses related to the specific item, but 416 (20.5%) were determined to represent sentiment about the platform, and of these, 305 (73.3%) were interpreted as positive.
After six months of experience, RO diplomates have overwhelmingly accepted, participated in, and are generally pleased with the ABR OLA instrument. As additional data is available, the Board anticipates sharing further details about performance and knowledge gaps with diplomates and stakeholder organizations.
References:
- American Board of Radiology Maintenance of Certification Program. https://www.theabr.org/wp-content/uploads/2019/01/MOC_Brochure_DR_IR-DR_RO_2019.pdf Availability verified June 30, 2020.
- Wallner, PE, Shrieve, DC, Kachnic, LA et al. American Board of radiology maintenance of certification program: evolution to better serve stakeholders. Int Jour Radiat Oncol Biol Phys. 2016; 94(1): 16-18.
- Wallner, PE, Gerdeman, A, Willis, JM et al. The American Board of Radiology radiation oncology maintenance of certification part 3 modular examination: evaluation of the first examination. Pract. Radiat. Oncol. 2016:6, 436-438.
