Rigorous Development Processes Ensured Remote Exam Platform Success

By Michael Yunes, MD, ABR Associate Executive Director for Radiation Oncology; Mary S. Newell, MD, ABR Associate Executive Director for Diagnostic Radiology; Geoffrey S. Ibbott, PhD, ABR Associate Executive Director for Medical Physics; James B. Spies, MD, MPH, ABR Associate Executive Director for Interventional Radiology; and Matthew B. Podgorsak, PhD, ABR Board of Trustees Chair

2025;18(3):4

The mission of the ABR is to “certify that our diplomates demonstrate the requisite knowledge, skill, and understanding of their disciplines to the benefit of patients.” In the spring of 2020, when the extent of the global pandemic became clear, it was apparent that many of our longstanding mission-critical activities, including administering computer-based exams at testing centers as well as conducting face-to-face oral exams, were no longer acceptable. As did many other organizations, we developed remote processes to keep our candidates, volunteers, and staff safe.

Recent news of a high-profile credentialing organization experiencing problems with the development and implementation of their remote exams underscores the robust nature of the ABR’s processes and contrasts with our successful management of the transition to remote exams.

In studying options for remote exam administration, many potential weaknesses were anticipated and largely prevented through strategic and collaborative efforts by the ABR Board of Governors, Board of Trustees, staff, and volunteers. No implementation can be perfect; however, it was found that limiting the number of modifications at each step and maintaining proven processes minimized the risks associated with reactive and injudicious changes.

The ABR used a roadmap to manage each identified potential problem with a remote exam platform:

1. The ABR was transparent with the intent and process of transitioning to remote exams and the development of a software platform.
2. Adequate time was planned for the development and testing of the platform prior to implementation.
3. The ABR did not rely on external entities for content creation, validation, or software development.
4. Feedback from all stakeholder groups was taken into consideration at each step of exam development.
5. Exam questions continued to undergo a consistent and robust multistep process of development, editing, validation, and assessment prior to selection for use on an exam. This process has continued to improve and continues to rely on practicing subject matter experts in their disciplines.
6. The exam platform was designed to mimic the previous candidate experience of the in-person Pearson VUE proctored exams and the face-to-face oral exams.
7. Extensive testing of the software and security was performed prior to going live with an actual exam.

The ABR is dedicated to continued exam development with the following ongoing processes:

1. Changes in content or emphasis are publicized well in advance of their inclusion in any exam.
2. Exam scoring has remained criterion referenced, not norm referenced. There is no set passing threshold, but rather the passing threshold is set by the item difficulty using a process called the Angoff method. This is performed by a group of volunteers who have a clear understanding of the knowledge base expected at each level of training.
3. Best psychometric practices continue to evolve to improve the consistency and validity of each exam.
4. Continued software development improves the candidate experience and the quality of the exams.
5. Content standards are reviewed and enhanced annually to ensure that the exams remain an appropriate assessment of relevant knowledge.
6. Exam security and content confidentiality are essential components to ensure public trust in the validity of ABR certification.

The adaptability of exam candidates significantly contributed to the success of the transition to the remote processes, and their individual feedback contributed to innovations over time. It is expected that infrequent exam day technical issues will randomly arise when connecting to thousands of individual examiner and examinee networks and computers, but managing the systems on the development and delivery side minimized the risk as much as possible.

Responding to an international crisis was difficult to navigate in every facet of life. However, one strong indication of a successful response is the durability of the resultant process once the emergency has eased. The ABR’s achievements benefited from the resiliency, patience, and input of certification candidates and countless volunteers.

Return to The Beam