The Value of Diplomate Feedback in OLA
By Robert M. Barr, MD, ABR President; Mary S. Newell, MD, ABR Associate Executive Director for Diagnostic Radiology; and Jennifer Knoll, ABR Exam Development Manager
2024;17(1):2
In the dynamic landscape of ABR certification, the adoption of Online Longitudinal Assessment (OLA) reshaped how diplomates engage with continuing certification. Instead of the static unidirectional relationship that a periodic high-stakes exam entails, OLA is designed to encourage much greater interplay between those who create and those who answer OLA questions. This shift has spurred a change in how we manage and curate our question banks. A comment feature was adopted to allow diplomates an opportunity to provide insights, voice concerns, and highlight alternative interpretations regarding the questions. Understanding the significance of diplomate comments or responses to questions received through this platform has proven to be crucial for fostering the goal of assessing walking-around knowledge and optimizing the continuous learning function.
The life cycle of an exam question is lengthy and complex, starting with multiple rounds of revision and review by subject matter experts and editors. The creation process takes approximately nine months. Once a question is deployed, the committee’s review of the question shifts to consider its performance and diplomate feedback.
At the close of 2023, the ABR had collected approximately 100,000 comments across more than 9 million OLA question responses. To best use this feedback, we monitored each question using predetermined psychometric parameters, looking for things such as question performance (high or low), performance deviation between variants, decline rates, relevance ratings, and, most relevant to this discussion, comments on the question. When a question underperforms, it is flagged for review and the question curation process begins. Of note, questions may be flagged simply because they garnered a large number of comments, indicating diplomate concern.
When the system marks questions for review, they are triaged first by the respective associate executive director (AED), who reads all the comments and provides a summary for the committee to consider. Diplomate feedback gives an invaluable perspective for those determining how questions can be improved. For example, commenters might note that the images do not optimally illustrate the case or enable diplomates to differentiate among the answer choices well enough. They might point out that recent literature suggests that a different response may now be correct. On occasion, people suggest that a question might be more nuanced than anticipated by the authors due to geographic practice variability. Although the question may be well written and the diagnosis clearly shown, an additional finding may be present on the image causing distraction and confusion. Lastly, a topic that the committee agreed was walking-around knowledge may prove not to be once a larger audience responds to the question.
The AEDs analyze diplomate comments as well as other performance metrics for questions that have been flagged for review. Most such questions are returned to the writing committee to see if the question can be modified in a way that addresses the submitted comments. For example, a problematic distractor could be replaced or better images could be inserted. If the committee thinks that the question cannot be optimized through committee re-work, it is removed from the pool.
Ultimately, the work to improve questions based on diplomate comments is only as good as the feedback received. We encourage our diplomates to continue offering specific and detailed feedback and commentary; we will keep listening. Please include salient new references, as well as views on question relevance and image quality. Diverse perspectives, ideas, and experiences collectively enhance the quality and value of our exams. This iterative process highlights the symbiotic relationship between our exam development committees and our diplomates, and the importance of ongoing engagement on both sides to refine this assessment tool.
