OLA Questions Go Through Extensive Development and Review
2022;15(5):11
Before they appear in participants’ email inboxes every Monday, Online Longitudinal Assessment questions go through the wringer. The question writing process works the same for all 17 OLA volunteer committees. It includes extensive development and review by diplomates who are experts in their fields. Questions that are eventually offered to participants go through the following steps:
- A volunteer submits a question.
- ABR staff review the question for grammar and image quality.
- Further input is received from experienced volunteers.
- The appropriate committee reviews the question.
- The question becomes part of the set assembly by the committee.
- The question goes into the OLA deployment pool.
Committees typically consist of one ABR trustee, one committee chair, five senior reviewers, and 10 question writers. Members serve three-year terms with options for a second and, with trustee approval, even a third. Committees meet in person in Chicago or Tucson or by video call once or twice a year.
Joshua Nickerson, MD, an associate professor of radiology and division chief of neuroradiology at Oregon Health & Sciences University in Portland, and his neuroradiology committee teammates recently had a productive two-day meeting. It was the first time they had met in person since the pandemic started.
“Our group reviewed over 200 items to be used in the next year of OLA,” he said.
Shana Coplowitz, MD, serves as medical director at Montefiore St. Luke’s Hospital in Cornwall, New York. She provides a valuable perspective as a private practitioner on the radiation oncology team.
“I’m somewhat removed from teaching and research,” said Dr. Coplowitz, who has been writing OLA questions for two years. “I felt that this would be a great way of giving back to the field and being involved with the more ‘academic’ aspects of radiation oncology.”
Naveen Subhas, MD, has been part of the musculoskeletal radiology committee for four years. Dr. Subhas appreciates the care that goes into ensuring that material is accurate and appropriate before diplomates see it.
“By having each question reviewed and edited first by a senior reviewer, then by a committee of other writers and reviewers, and once again by a panel prior to approval, it ensures that all approved questions are adequately vetted and appropriate for the diplomates,” said Dr. Subhas, a radiologist in the Imaging Institute at the Cleveland Clinic and professor of radiology at the Lerner College of Medicine. “This process also allows writers and reviewers to learn from best practices and is transparent.”
Like all ABR volunteers, Martha Jensen-Menchaca, MD, is an OLA participant. A member of the genitourinary radiology committee for five years, she enjoys seeing the process from both sides.
“I get to see how questions are developed at multiple steps,” said Dr. Jensen-Menchaca, a diagnostic radiologist at the University of Illinois at Chicago. “There is a very thorough evaluation that each question undergoes to make it of high yield quality, as well as fair and unambiguous. A lot of thought and discussion goes into approving each and every question. We evaluate if each question can be answered with ‘walking-around knowledge’ by a competent radiologist who has selected GU questions. Our committee always reviews and, if necessary, makes modifications to ensure that all the images are optimal to ensure a correct answer.”
Some questions are eliminated or altered before they reach the deployment pool.
“We evaluate the question structure and the answer choices and make changes to eliminate any potential ambiguous items that may cause confusion,” Dr. Jensen-Menchaca said. “Our committee works hard to make the questions as perfect as possible.”
Dr. Nickerson, who has been an OLA question writer since 2019, said that when developing a question, the visual component must be considered.
“My goal when constructing a question is to find really high-quality images on which to base the item,” Dr. Nickerson said. “One of the frequent complaints I have heard over the years is that people are frustrated when a question is written based on suboptimal or archaic images. I do my best to find examples of anatomy or pathology that are depicted using the most modern techniques that we would expect to interpret in our daily practices.”
The most convenient and flexible way to satisfy Part 3 of Continuing Certification (MOC) for most diplomates since 2019, OLA also gives participants control of the process. For example, if they have an issue with a question, they may submit feedback. Those thoughts are shared with the appropriate committee and the associate executive director for the specialty.
“The process is actually fun,” Dr. Coplowitz said. “I really enjoy the meetings where we review everyone’s questions and give feedback. It’s so fascinating to me that some things which I think are really basic may not be to someone else whose practice is super specialized.”
Dr. Nickerson is proud to be part of a system that reflects the work that diplomates perform.
“By participating in the process, I feel like I am doing my part to keep the house of radiology looking after itself rather than being subject to scrutiny by those who do not understand what we do every day to take care of our patients,” he said.
