From the Board of Trustees: Candidate Feedback in Post-Exam Surveys Informs Improvements
By ABR Trustees Anne M. Covey, MD; Kate Maturen, MD, MS; Jennifer Stickel, PhD; and John H. Suh, MD
June 2026;19(3):4
After every ABR computer-based exam, candidates are invited to participate in voluntary anonymous surveys. ABR staff and committee volunteers use the survey data to improve the examinee experience and exam content.
Staff use the feedback to make the exam interface more user friendly. In response to user comments, multiple iterative changes have been made to the ABR’s remote testing platform, resulting in candidates viewing the current version more favorably than the pilot version released in 2021.
ABR committee volunteers use survey results to improve the quality and relevance of the exam questions. Unsurprisingly, there is variation — sometimes including direct contradictions — among user responses. In each exam administration, some survey respondents may compliment the overall relevance of the content, while others may indicate that questions were esoteric or not part of standard practice.
Residency programs vary in the breadth of pathology and imaging technology available. For that matter, not every resident within a specific institution will have exposure to an identical distribution of cases. The committees that prepare exam content are acutely aware of this. It may be helpful for candidates to understand how the exam is created.
Each question-writing committee consists of approximately 10 ABR certified subject matter experts. Each member submits draft exam questions that are evaluated by the committee. During committee meetings, members respectfully debate and collaborate to revise or delete questions that are either potentially misleading or outside the acceptable boundaries of clinical practice. In this manner, committee members come to a consensus on every question submitted, to ensure it is accurate and appropriate to be included in an exam. The content is reviewed again at the time of test assembly to ensure the exam is balanced in terms of the breadth and difficulty of the material. These multiple reviews are important to make sure obscure diagnoses or “trick” questions are not included.
“The exam was too hard” is a comment we typically receive from a small fraction of survey respondents. It is impossible to create multiple exams with the exact same level of difficulty, but there are methods to address this disparity. The ABR has rigorous standard-setting processes to account for question difficulty. The most common is the Angoff method, which adjusts the passing threshold (number of correct responses needed to pass) for every exam. The threshold is based on an assessment of independent volunteer subject matter experts who determine the aggregate estimated difficulty of each question prior to the exam administration. Simply put, fewer correct answers are needed to achieve a passing score on an exam that includes more difficult questions.
The ABR appreciates the candidates who take the time to complete post-exam surveys. The feedback provides valuable insight that helps improve both the exam interface and content.
