Candidate Feedback on Core Exam Is Positive
by N. Reed Dunnick, MD, ABR Executive Associate Director for Diagnostic Radiology, and Brent Wagner, MD, MBA, ABR Executive Director
2021;14(2):6
We distribute an exit survey after each of our exams. For the DR and IR/DR Qualifying (Core) Exam in February, we were fortunate to see a 50% response rate from the nearly 1,400 candidates. We sincerely value the input, both the digital responses and the free-form comments, as we attempt to improve the experience and validity of our exams.
Some of the responses allowed us to determine trends in the local environment used by the candidates. For example, the group was nearly evenly split between laptop vs desktop, wired vs WiFi, and “in my home” vs “in my program.” (The way we collected the data was designed to be easy for the respondent, and correlations among the above variables were not established.)
The practice exam was deemed helpful by 97% of respondents. This confirmed that candidates should develop facility with the platform, including the log-in process and the image tools, prior to exam day. We want the exam to assess knowledge, not the ability of someone to navigate an unfamiliar system.
The technical aspects of the exam were, in general, favorably reviewed by the candidates. (Unless otherwise specified, a “favorable” response included “agree” or “strongly agree” with a positive statement about the exam.) Of 30% of respondents who contacted the ABR Help Desk on exam day, over 90% favorably responded on the ABR staff’s ability to resolve the issue. Most of these requests for assistance were based on brief interruptions in connectivity.
Over 90% offered favorable responses to the following:
- It was easy for me figure out how to use the exam interface.
- If I didn’t answer a question, I was always reminded to answer before navigating away from the question.
- I understood that my viewed questions would be locked after a break.
Greater than 85% of candidates offered favorable responses regarding connectivity during the exam, including the Proctorio plug in.
What we can improve:
- Many candidates asked for a strike-through feature to eliminate “wrong” choices when going through the exam. We’re working on this, but we may not be able to implement it in 2021.
- Some candidates didn’t realize that completing one block of items automatically started the next block. If the candidate chose to take a break at that time, the first question in the subsequent block was locked because it had already been viewed by the candidate. We’re working on making it more obvious when a candidate is leaving a block or entering a break – and what that means. The blocks are merely part of a reminder to the candidate to answer questions periodically rather than letting them “build up”; this is because previously viewed content must be answered before an interruption in the exam. The best strategy is for candidates to answer questions as they go, so that an interruption in connectivity doesn’t prevent them from answering all the items.
- Lastly, approximately 15% of candidates raised concerns regarding image quality. Although we have received similar comments after past exam administrations and have implemented formal processes to improve the quality of the images, this was a slightly larger percentage than in recent years. The new remote model is challenging because of two factors beyond our control: (1) the remote device that the candidate chooses and, more importantly, (2) variable bandwidth in the candidate’s location. We are actively looking at opportunities to improve not only the technical features of the platform but also the intrinsic image quality that is part of the exam content.
