Hard Work Delivers Success in the Remote Environment
by James B. Spies, MD, MPH, ABR Associate Executive Director for Interventional Radiology
With the administration of the oral certifying exams last month, the ABR has successfully completed its first remote IR/DR certification cycle, an achievement that seemed a nearly insurmountable task a year ago, when it became apparent that conducting exams remotely was the only path forward in the midst of the COVID-19 pandemic. With many technical hurdles to overcome, rethinking and re-engineering board certification exam platforms commanded the full effort of ABR staff and volunteers to get the job done in time. The successful administration of the exams is a testament to their tireless efforts.
The IR/DR Certifying Exam has two components: computer-based and oral. The computer-based component has approximately 180 questions, one-third pertaining to interventional radiology and two-thirds covering the essentials of radiology material common to all radiology certifying exams. The oral component consists of four 30-minute sections: Imaging of IR; Core and Miscellaneous Interventional (including portal interventions, biopsy, drainage, etc); Interventional Oncology, Hepatobiliary, Genitourinary, and Pancreatic Interventions; and Arterial and Venous Interventions. This has been the structure since the IR/DR pathway was approved.
Transforming the computer-based exam to a remote experience was the more straightforward project. The ABR had spent two years developing the software platform used for Online Longitudinal Assessment (OLA), and this was adapted to present the certifying case material. The project also required an exam security system, and we were fortunate to have several commercially available systems available. So, although not an easy task, the work needed was less complicated and that platform was successfully deployed in January.
The oral exam was more challenging. ABR IT staff had to create a platform that would allow presentation of the case material, showing images to both the candidate and examiner and allowing the images to be scrolled, magnified, or annotated by either. Examiners also needed access to case details and a tool for scoring in real time. A second platform for videoconferencing and discussion that could run in parallel was also needed.
Based on other specialties’ initial attempts using simple videoconference platforms, we knew the challenge was much more complicated than using a Zoom conference. As the components came together, we were confident we could deploy the needed software platform, but we still believed it essential to add redundancy to ensure connection issues that might interfere with completing an exam could be overcome.
These redundancies included 10-minute breaks between sections to make up for brief connection problems and a recovery session at the end of the day for candidates who had major disruptions to make up a section. We also left time to allow for rescheduling candidates on another day if they had a complete connection failure. A support staff navigator was assigned to each candidate to guide them through the entry process and stay with them through the exam to respond to connection issues. Finally, two examiners were assigned to every session. This ensured we could respond to examiner connection issues but also doubled the number of examiners needed. The protocol assigned one primary examiner to give the exam and score, with a secondary examiner connected in the background. If the primary examiner lost connection, the secondary examiner would take over and complete the exam. Every participant – candidates and examiners – underwent training to make sure they knew how to operate in this environment and individual technical checks to ensure their computer systems would function properly on exam day.
Thanks to the planning, training, and staff effort, as well as the dedication and patience of our 138 examiners, the exam was a great success: 351 candidates completed the exam with 1,404 individual exam sections. Only a handful of candidates had to have part of their exam completed in a recovery period or scheduled on another day. Fewer than 4% of exams required a switch from the primary to secondary examiner.
We learned much about the remote exam environment from this initial exam, and we are using that knowledge to improve the platform and refine the user experience for the October administration. Although there are always things we can improve, it is nice to take a moment to reflect on the success of this initial remote oral exam and to thank the volunteers and staff for making it happen.
For more information on remote exams, please visit our website.