Lessons Learned on Both Sides During First Remote Core Exam
by Brent Wagner, MD, MBA, ABR Executive Director
2021;14(2):3
In February, we delivered our first remote DR and IR/DR Qualifying (Core) Exam. Technically, the exam was a success. Although several individuals struggled with connection interruptions as a result of variable bandwidth, all were given enough time to complete the exam, and there was no evidence of significant security issues.
The informal feedback we received as we attempted to provide real-time help desk functions was encouraging – this was a group that lived up to expectations of professionals who, despite the inherent stress of a board exam, were both respectful of the process and patient with us as we worked to resolve individual problems as quickly as possible. We have plans to address the challenges, for candidates and for the board, in future exam administrations. Some of these can be accomplished for the computer-based exams this month, while others will be implemented later in 2021.
What we learned from the exam was not limited to the technical parts of the process. Data we collected on physics content within the exams prompted an examination (and subsequent revision) of the scoring model. Additionally, multiple responses to the survey revealed a lack of understanding of the rationale for content selection. Acknowledging that the responsibility to address any confusion rests with us, the following paragraphs attempt to provide clarity in this context.
Core Exam Scoring Model for Physics
An understanding of physics as it relates to the radiologic sciences is one of the things that sets ABR diplomates apart from other medical professionals. The sequencing and format of the exams have evolved over the years, but physics currently makes up slightly over 20% of the total exam content of more than 650 questions. In recent years, the ABR has focused on significantly improving the relevance of the physics content, including the integration of concepts directly related to the clinical practice of radiology.
Since the introduction of the current Core Exam model in 2013, a passing score in physics has been required in addition to an overall passing score. This meant that poor performance in physics could contribute to a failing score on the exam overall (which is scored as an aggregate of organ-based and physics questions). It also meant that exceptional performance in organ-based areas would not be able to completely compensate for a poor performance in physics. This latter possibility has resulted in less than 1% of individuals over the past several years having to return for a targeted exam in physics (because they had “conditioned” physics, despite having an overall passing score on the exam).
At its meeting in March, the Board of Governors decided to do away with the conditioning category for physics effective with the 2021 exams. This change offers an advantage to a small number of candidates; for the remaining candidates, it has no impact because the overall scoring standard method is unchanged. It is important to remember that, as before, candidates and training programs should be mindful of the significant physics portion (more than 20%) of the exam, and its importance as part of the measurement of overall performance.
Difficulty Level of Questions for the Core Exam
The difficulty level was a frequent criticism of the exam content, partly based on the ABR’s intent to develop processes built around establishing “competence” (as opposed to, for example, “excellence”). We have pointed out in previous communications that our exams are “criterion referenced.” This means that there is a passing threshold that is determined in advance, based on the aggregate difficulty level of the questions (items). This is in contrast to a “norm-reference” exam (e.g., the SAT), which does not have a passing score but instead generates a percentile rank or other comparison to fellow examinees.
During exam development and assembly, the relevance and quality of items are repeatedly challenged by content experts in a diverse discussion setting in order to eliminate esoteric or other suboptimal items. A subsequent group, primarily made up of teaching faculty in the various radiologic disciplines, sets a “difficulty rating.” It’s not surprising that a candidate might encounter challenging items, perhaps because they are based on concepts that are unfamiliar or, much less commonly, not part of the candidate’s experience. The difficulty rating accounts for this because it represents, across all items as a unit, an adjustment for these challenging questions.
In addition, a detailed statistical analysis after the test reveals “problem items.” There may be ambiguities in specific items that were not recognized prior to exam delivery or other outliers; many of these items are removed before scoring for those who gave an incorrect answer. This is part of a specific ongoing effort on the part of the ABR to reach the “highest level of fairness” for our exam candidates; unfortunately, it is also part of the reason that releasing exam scores takes longer than expected.
In summary, candidates should not be discouraged if they are unable to answer every question correctly. The test is difficult, and the processes we use attempt to address variations in performance while still assessing competence.
