by Valerie P. Jackson, MD Online Longitudinal Assessment launched for diagnostic radiology and diagnostic radiology subspecialty diplomates this past January. Four months after implementation of the system, ABR Executive Director Valerie Jackson took time to answer a few questions about OLA, the American Board of Medical Specialties’ Vision for the Future Commission, and ABR exams in general. OLA has been active for four months. What are some of the emerging trends? More than 16,000 DR diplomates and subspecialties, almost 90 percent of the total enrolled in MOC, have answered at least one question in OLA. More than 380,000 questions have been answered, and more than 46 percent of participants have answered at least 20 questions. At that pace, this group will complete its yearly requirement of 52 questions by July. Of course, diplomates may complete up to 104 questions per year if desired. Every OLA participant has the opportunity to participate as an OLA “rater,” which helps to set the passing standard for each question. Almost one-third of participants have been involved in the rating process. Diplomates may add comments to each OLA question, and to date we have received more than 10,000 comments. Each comment is reviewed so that we can improve our processes. Many comments have been very positive about the OLA experience; others have been extremely helpful in alerting us to potential problems with an item. Feedback is provided directly to volunteers who create the content for OLA. We are very pleased with the level of engagement of our diplomates with OLA and look forward to rolling this out to interventional radiology, radiation oncology, and medical physics in January, 2020. We have received more than 10,000 comments from diplomates using OLA. Upon which suggestions is the ABR looking to act? Based on feedback from our diplomates, the ABR is working on a number of enhancements for OLA.

• We are assessing the amount of time required to answer an individual OLA question. While the vast majority of diplomates successfully answer both one- and three-minute questions in less than one minute, there have been comments about wanting more time. The time limits were set up based upon our experience with our exams and the fact that OLA is meant to assess “walking-around knowledge” that diplomates should know “off the top of their head.” The board will be reviewing all data and comments about timing for the questions to decide if changes are necessary.
• The board is evaluating the feasibility of adding a clinical practice area for emergency radiology.
• We are developing a mechanism to count OLA participation as part of ABR’s Self-Assessment CME (SA-CME) requirement.
• Eventually, we plan to allow diplomates to customize their OLA notification experience, letting them change the frequency of notifications of new questions. However, this will require extensive software development, so in the meantime, we will continue to send weekly reminders. Each question has a four-week shelf life and there is no requirement to answer questions every week.

The American Board of Medical Specialties’ Vision for the Future Commission recently released its report about continuing board certification. How do those findings impact the ABR? The ABR is grateful for the effort put into the ABMS Vision for the Future Commission report. Input from a large number of stakeholders led to production of a valuable document to guide the ABMS member boards community in the future. The ABR already had implemented many of the Commission’s recommendations, indicating that we are on the right track. Some of those initiatives go back to 2012. We still welcome feedback and recommendations for improvement. Some diplomates wonder how exams are compiled. What goes into that process? All our exams and OLA questions are compiled by unpaid volunteers who are actively practicing their specialty. Our volunteers come from both academic and private practices. These volunteers write and review questions, and attend webinars as well as face-to-face meetings to assemble the 33 exams the ABR administers each year. Approximately 1,300 volunteers write more than 8,000 questions annually. Question writers work from blueprints of exam content that are updated regularly to be relevant to current practice. Each volunteer is trained to construct questions appropriately and their questions are reviewed by more senior volunteers and our editing staff. Test assembly meetings, where the content is reviewed by volunteers from a number of different subspecialty areas, help to ensure that the questions are clear and relevant. We are often asked why it takes so long to get results after administration of our computer-based exams. They are scored relatively quickly, but after that we do a detailed analysis of the results for each item. While we believe that the questions are good at the time of administration, sometimes we find items that are unexpectedly problematic. In such cases, we need to modify the scoring of the exam. We then check and recheck every result to be certain we are correct. This process takes several weeks, with longer times required for our largest exam, the DR Core Exam.