Remote Exam Retrospective: Balancing Flexibility with High Security Standards
By Brent Wagner, MD, MBA, ABR Executive Director, and Vincent P. Mathews, MD, ABR President
In June 2020, the ABR decided to substantially modify its exam platforms. Uncertainty regarding the length and depth of the pandemic had created an imperative to develop a plan that would allow reliable administration of a valid and timely exam program, regardless of travel restrictions and capacity constraints.
Early in the process, we asked various groups to share both ideas and concerns. Most of our focus was on program directors and candidates, and we solicited input from leadership in the respective organizations for all four radiologic disciplines, as well as our advisory committees. Subsequently, a large number of internal discussions revolved around a simple concept: we wanted to maintain consistency in the scope of the exams (e.g., number of questions, degree of difficulty, and coverage of the domain) while designing a platform that would add as little disruption as possible to an experience that is inherently stressful for exam candidates. The biggest systemic unknowns were the environmental changes that included a shift to the local user of the technology element (i.e., the computer) and the testing location (home or office). The biggest challenge was the complexity of a process that had to work more than 99% of the time in these unpredictable and variable environments. We set ambitious timelines and, along the way, learned from the successes and failures of other medical specialty boards.
For written (computer-based) exams, we looked to the remote testing experience of colleges and universities over the past several years and attempted to integrate this with basic elements of our exam software. This software had been refined over several years in our physical exam centers for diagnostic radiology (DR) and interventional radiology (IR). It was also used in a slightly different form in Pearson VUE® for medical physics (MP) and radiation oncology (RO). The remote portion of the proctoring, in the model under development, involved several compromises that attempted to balance the user experience with the necessary security to ensure a valid result.
The oral exams for IR, MP, and RO were developed using an internally developed exam platform coupled with a third-party video conference product. Linking up the correct examiners for the different portions of the exams with the candidates required complicated integration. There was also a need for redundancy based on the uncommon but inevitable interruptions in internet connectivity for the candidates.
The major points along the timeline included a pilot DR Certifying Exam in early January, a DR and IR/DR Qualifying (Core) Exam in early February, pilot oral exams for MP, RO, and IR in March, several computer-based exams across all four disciplines in April, and the main oral exams in May. At that point, we were caught up for 2020.
Two weeks later, in early June, we started administering the 2021 remote exams with the DR and IR/DR Qualifying (Core) Exam and will continue with the rest of the exams in late summer through fall. Enhancements and modifications continue to be made, based on lessons learned thus far. Our goal is to be able to continue offering remote exams indefinitely, but that will require a willingness to balance flexibility with a firm commitment to maintaining the high security standards required for high stakes exams.