Testing, Pilots Set the Stage for Remote Exam Success in 2021
By Paul E. Wallner, DO, ABR Associate Executive Director for Radiation Oncology; and Brian J. Davis, MD, PhD, ABR Trustee
On March 11, 2020, the World Health Organization (WHO) declared that the SARS-CoV-2 (coronavirus disease 2019) outbreak, subsequently called COVID-19, had reached global pandemic status. Within several weeks, ABR staff and senior volunteers recognized that exam administrations planned for the spring and summer of 2020 would require postponement. As the pandemic rapidly spread across the United States, it became apparent that planning for remote administration of all ABR computer-based qualifying and oral certifying exams would be necessary if residents and candidates scheduled to take exams in 2020 were to remain on track with their certification cycles. In 2013, the ABR had converted the Diagnostic Radiology Certifying Exam to a computer-based platform, so adaptation of the various discipline’s qualifying exams to remote delivery engendered fewer significant obstacles. Here, we describe development of the Radiation Oncology Remote Oral Certifying Exam. Development of the remote oral certifying exams for interventional radiology/diagnostic radiology and medical physics followed generally similar pathways.
Conversion of the RO Oral Certifying Exam to a remote platform presented more daunting challenges than converting the computer-based exams did. With a goal of assuring the best possible exam experience for candidates and examiners, the focal point of development resided with the ABR’s Information Technology (IT) group. Under their general supervision, work groups with specific assigned tasks operated independently, with almost daily coordination. These individual tasks included: development of the primary exam delivery platform and hardware specifications; work with external consultants on exam security and user interface issues; creation of a scoring instrument consistent with longstanding practice; logistics of session timing, breaks, and time zone management; management of category and panel meetings; and training of staff, examiners, and candidates. Throughout the process, the ABR shared ideas and lessons learned with colleagues in other American Board of Medical Specialties (ABMS) Member Boards.
Early in the project, the group determined that uncertainties of internet connectivity required significant redundancy. ABR navigators would remain with candidates throughout their sessions, beginning with initial sign-on and technical, security, and identity checks. Secondary examiners were to be paired with primary examiners in each of the RO eight clinical categories. Secondary examiners would monitor the entire exam session and would score candidates in parallel with the primary examiner but would only take over the exam if a primary examiner lost connectivity. Secondary examiner scores could be considered in post-exam category meetings. Score sheets were developed to allow simple point and click comments using “radio buttons,” with additional space for free-text comments. To prevent loss of data, entries were saved automatically. Examiners and candidates were trained with recorded web programs, manuals, FAQs, question and answer sessions, and hands-on software training sessions with an ABR navigator.
At each step of the development process, the full work group reviewed progress and, when necessary, performed internal testing. As development proceeded, outreach to volunteers for external evaluation, testing, and critique was undertaken.
In late 2020, an internal beta test of the entire product was carried out with ABR staff and trustees acting as navigators, examiners, and candidates. Following subsequent modifications, a “pilot” exam was administered to 21 volunteer candidates in February 2021. The pilot represented a complete exam administration, but at no risk to candidates, such that any failures or conditions would not appear in their initial certification records, but successful completion of the exam would lead to an award of initial certification. All candidates completed the exam without difficulty, and, following minor modifications, a full exam administration was carried out successfully in May 2021 involving 192 candidates. The pilot and May exams were available only to those candidates eligible for 2020 exams.
Using secondary examiners produced significant challenges to recruitment of sufficient category committee members. The successful exam administrations in February and May 2021 supported the lack of need for continuation of that exam element, and in September 2021, 257 RO candidates were examined successfully without secondary examiner backup. Planning for 2022 exams includes reducing reliance on navigator support and eliminating parallel security and web-based exam delivery platforms. Bandwidth specifications may be reviewed to reduce minor connectivity problems that had been encountered by examiners and candidates. ABR RO trustees believe that the transition from in-person to remote exams was well managed and well received by candidates. From the candidates’ perspective, the process will remain virtually unchanged for 2022.