Pilot Exams Demonstrate Strength of Remote Oral Exam Platform
by Anne M. Covey, MD, Brian J. Davis, MD, PhD, and Matthew B. Podgorsak, PhD, ABR Trustees
2021;14(2):5
ABR staff, with input from many volunteers, have been working since the summer of 2020 to develop remote oral exam software. To test the functionality of the platform, a “dry run” was performed in February with staff, trustees, and volunteers playing the roles of candidates and examiners. After a few adjustments, successful pilot exams were conducted last month involving 83 candidates seeking board certification in interventional radiology/diagnostic radiology (IR/DR), medical physics (MP), or radiation oncology (RO).
All candidates eligible to take a certifying oral exam were given the opportunity to volunteer for the pilot. From this pool, a limited number were selected by lottery: 33 for IR/DR, 28 for MP, and 22 for RO. There was no fee to participate and no risk to the candidates who volunteered. Pilot participants meeting all other certification requirements who passed the exam were issued certification. Those who did not pass are eligible to take the oral exam in May 2021 or a subsequent administration. The results of the pilot exam for candidates who failed will not be included in their ABR record.
In developing the oral exam platform, the ABR focused on creating a satisfactory candidate experience without sacrificing exam security, validity, or credibility. We have tried to mimic as closely as possible the face-to-face oral exam experience. One thing that has changed is that candidates now choose their preferred exam location, subject to certain technical and logistical requirements. This allows candidates to take the exam at home, if they wish, ensuring their safety during the pandemic and saving the cost and time of travel.
Other changes involve steps that were taken to protect against technical failures. For instance, there is a 10-minute break after each exam period to allow make-up time for candidates who experience minor disruptions. In addition, two new roles have been established: a navigator and a secondary examiner.
All candidates are “accompanied” by an ABR exam navigator throughout their exam day. The navigator helps the candidate sign in, checks their ID, and performs a room and workspace scan. Once the exam begins, the navigator turns off their camera and microphone, but they are available and ready to assist if the candidate or examiner has a technical problem.
In addition to the candidate and the navigator, there are two examiners present in each session — primary and secondary. All examiners work in pairs. The primary examiner presents cases, and interacts with and scores the candidate. The secondary examiner observes the exam, scores candidates as if they were conducting the exam, and is ready to take over in the event the primary examiner has an internet outage.
The role of the secondary examiner was established to ensure continuity in the candidate’s exam experience if the primary examiner loses their connection or has other technical problems that prevent them from continuing the exam.
A total of 89 examiners participated in the pilot oral exams: 27 for IR/DR, 22 for MP, and 40 for RO. The full exams in May will require more than 400 examiners: 141 for IR/DR, 120 for MP, and 147 for RO. Examiners are experts in their field who volunteer to perform this integral task in the candidate certification process. We would be unable to conduct these exams without the participation of our examiners.
Overall, the pilot exams were largely successful. As expected, they did reveal some minor areas for improvement. Based on the pilot exams, modifications are being made to the exam software that will be ready for the May exam administrations to enhance the experience for candidates and examiners. We greatly appreciate all the volunteers who gave of their time to help make this final step in exam development possible.
