OLA Comments: We are Listening!
by James Spies, MD, MPH, Associate Executive Director for Interventional Radiology
2020;13(6):6
Interventional radiologists have been participating in OLA for nearly a year, and diagnostic radiologists who have chosen interventional as one of their practice areas are nearly two years into the process. Tens of thousands of questions have been answered, and the process is maturing.
One benefit of OLA is the opportunity for participants to comment on individual questions, providing an unprecedented opportunity for the diplomate to guide the process.
Diplomate comments also provide a huge benefit to the ABR and our committee members, offering an invaluable source of perspectives that might not have been considered and a chance to review questions in a new light.
Our goal is for every question to be clear, have a single correct answer, and reflect knowledge that the average interventional radiologist should know without having to use reference material (i.e., “walking-around knowledge”). Not surprisingly, this leads some diplomates to believe that their questions should test only what they do in their practice. While understandable, someone who only does biopsies and drainages can’t only be tested in those areas and still be considered up to date on current IR practice. Even if a diplomate does not do uterine embolization, interventional oncology, or any other practice area in IR, they should have some general knowledge about these procedures. Hospitals, insurers, and the public rely on board certification as a standard of training and knowledge and, therefore, assessment must reflect at least basic knowledge of the range of IR practice.
The ABR has established a process for regularly reviewing comments, classifying them, and providing direct feedback to the writing committees. There is a broad range of comments. Some praise a question or say that it was a good question that they should have known. There are also those explaining why they ran out of time – their dog knocked over a vase, someone came to the door or, amazingly, that they did not have enough time to look up the answer! (For the record, these questions must be answered without assistance from others or using written or internet resources.) The comments also occasionally report technical problems – an image loaded slowly or did not load at all. Thankfully, these are rare.
The feedback has already had an impact. For example, a frequent comment is that the diplomate did not realize there was a second image. To address this, now when a question is presented, a message notes the time allowed for the question and how many images, videos, or image stacks are included.
Many commenters do not agree with a question’s answer, and they present an argument to support their view. These are among the most important comments. Despite a two-level initial review of each question, with final additional review by the entire committee, the process is not perfect. Some comments point out ambiguity in wording, disagreement with the “correct” answer, or a different way to look at a question. These incredibly valuable comments are collated, summarized, and provided to the writing committees. We review comments on an ongoing basis for glaring issues (wrong gender or other obvious errors) and then quarterly for all other issues. The associate executive director (AED) for each discipline coordinates the process ̶ reviewing the comments, preparing the report to the committee, and participating in committee calls to review and revise questions. When a question is discarded from the pool or significantly revised, all candidates who previously answered the question – correctly or incorrectly – receive full credit for the question in their assessment.
So, we do listen to you. Your comments are making this process better, with clearer questions that better reflect current practice and a fairer assessment of each diplomate’s knowledge. Keep the comments coming!
