Staff, Candidate Efforts Make Remote Exams Successful
By Kinsey SmithWright, ABR Administrative and Exam Support Manager, and Brent Wagner, MD, MBA, ABR Executive Director
2023;16(6):3
The ABR stewarded 11,853 (computer-based) and 2,440 (oral) candidates through 52 exam administrations from when our exams went fully remote in early 2021 through the end of 2022. To say we have learned a lot understates the steep curve we continually met to ensure each administration was a success.
Concept and design investment, as well as careful planning, has enabled the ABR to build robust platforms that measure a candidate’s competency in their designated specialty.
Every exam delivery has been different, building on opportunities to refine while also presenting new challenges to our operating procedures. One of the key takeaways is that even if we (ABR staff and candidates) do everything correctly, that does not ensure a smooth exam experience. Our data indicates that the factors that commonly cause unanticipated interruptions are internet outages, third-party software updates/provider issues, database resource strain, unsupported operating system/browser issues, and institutional network firewalls.
Despite the various issues encountered, total technical failures where the candidate was unable to complete the exam have been minimal: computer-based exams had 81 candidates (0.7%) and oral exams had five (0.2%). Oral exam recovery periods, which are built into each exam schedule and used only if needed, gave flexibility when candidates or examiners experienced technical issues that could be resolved but caused enough disruption to necessitate additional time to complete the exam. A total of 42 (1.7%) recovery periods were administered for candidates and allowed them to complete their exams on the same day despite the initial technical issues.
The deployment of our live monitoring software in July 2022 allowed us to communicate through exam software to assist candidates in real-time with issues that previously might have been grounds for an invalidated result. Frequently corrected issues included bad camera angles, video outages, and prohibited items in the workspace.
Often, high-stress events reveal areas that could be improved. For the ABR, the September 2022 DR Certifying Exam offered a challenge that showed us how we needed to reshape some of our technical and customer service processes for future exams. During the exam, we encountered a platform outage that lasted 40 minutes and halted the progress of 1,068 candidates. While we were able to recover and successfully steward all candidates through the exam, this disruption will not soon be forgotten by the staff who worked the exam nor the candidates who took it.
As a result of this close call, we got nearer to understanding the maximum number of candidates to whom we feel comfortable administering an exam without unnecessary technical risk and undue stress on staff. This resulted in several changes in the administration of the 2023 DR Certifying Exam:
- It was administered over two days instead of one.
- We improved internal crisis communication platforms for when outages happen, with clear roles and responsibilities.
- New phone messages were developed that could be quickly pushed out through our Exam Day Help Desk.
- A text-based communication process was implemented for rapid and informative updates to candidates.
Other notable improvements included the development of a separate IT support line to improve technical assistance customer service during all exams, and a designated examiner support line during oral exams to address their specific needs. These changes, in addition to the numerous external and internal software updates, have allowed our team to quickly respond and resolve the issues our candidates and examiners encounter.
To date in 2023, the ABR has administered 19 exams to 5,446 candidates (4,474 computer-based candidates and 974 oral candidates), and three additional administrations will occur before the end of the year. While the incidence and severity of technical issues have markedly decreased in 2023 (only two candidate technical failures), better systems and optimal candidate experience continue to be top priorities.
Our advice to candidates to reduce stress during any ABR testing experience is to engage with the tools created by the ABR to help prepare for exam day. These are made available to candidates ahead of time and include the exam guides, targeted email messages, Exam Readiness Check (for computer-based exams), and self-guided candidate/examiner technical check (for oral exams). Engage early, and if you encounter issues, please call our knowledgeable certification managers or IT Support Team before exam day. A second check just before the exam is also a good idea.
While a smooth exam for hundreds of candidates with unavoidable variability in hardware and network connections is impossible to guarantee, advance preparation and mutual respect for the high stakes of the process go a long way toward making every exam a success.
