Online Longitudinal Assessment (OLA)

Online Longitudinal Assessment (OLA)

Last verified on October 1, 2021
 
 

Online Longitudinal Assessment (OLA) is a convenient way to satisfy Part 3 of the Maintenance of Certification (MOC) program. It has started for all diagnostic radiology (DR), diagnostic radiology subspecialty (neuroradiology, nuclear radiology, and pediatrics), radiation oncology, medical physics, and interventional radiology diplomates.

 

We have made available several options to help diplomates as you start OLA:

 

Please whitelist the ABR domain (@theabr.org) in your email to receive important communication about this ongoing assessment. If you have questions, please contact us at information@theabr.org or  (520) 790-2900.

 

We recommend using a desktop computer, laptop, or tablet when answering OLA questions. Because the assessment is image-driven, cell phones don’t provide optimal results.

 

 

Background

OLA is a progressive online assessment that will replace the previous proctored MOC Exam required every 10 years. OLA is aligned with the other three parts of MOC that remain the same—Part 1: state licensure; Part 2: CME and self-assessment; and Part 4: participation in practice quality improvement activities.
 

 

Benefits of OLA

  • Flexibility in Participation – You choose when to answer the questions.
  • Immediate Feedback – You will know if you answer correctly or incorrectly, and what the correct answer is for each question. Brief discussion and reference information is also included.
  • No Travel Required – Questions are available for you to answer on your own time at work or at home with no travel required.
 

 

OLA Questions

The OLA questions are written and reviewed by your radiology peers who volunteer their time on a number of committees. To understand how exam questions are written and learn more about different types of exam questions, please see the ABR Item Writers’ Guide. For a look at the extensive QA process that each question goes through, please see the Illustrated Life Cycle of an ABR Exam Item.

 

OLA Process

OLA enables diplomates to demonstrate what they know. For diagnostic radiologists, questions will be tailored to a self-selected practice profile. The OLA dashboard will enable you to monitor your progress and performance. ABR board members and staff are committed to ensuring that OLA is reliable, reasonable, relevant, and meaningful. The ABR has a sustained responsibility to the public to certify that diplomates demonstrate the requisite knowledge, skill, and understanding of their disciplines to benefit patients.

 

OLA FAQs

 
  • + Getting Started

    How long do I need to participate in OLA?
     

    OLA is a continuous assessment in which diplomates involved in MOC will participate every year throughout their careers.

     
    When do I begin using OLA?
     

    Most diplomates participating in MOC are eligible for OLA. It is not available for a hospice and palliative or a pain medicine certificate. Those diplomates must take a traditional exam.

     
    How does OLA fit within the MOC program?
     

    OLA will fulfill the Part 3 requirement, replacing the 10-year MOC exam. Diplomates will need to continue to attest to meeting Part 1, 2, and 4 requirements each year between January 1 and March 1.

     
    Will I get to choose the clinical practice areas included in my exam?
     

    Yes, practice profile rules for diagnostic radiologists will not change. Diagnostic radiology diplomates without a subspecialty will be able to choose up to three practice areas in which to receive questions. Radiation oncology, medical physics, and interventional radiology/diagnostic radiology diplomates will not have the ability to select clinical practice areas.

     
    If I’m a diagnostic radiologist who sets my practice profile as General Diagnostic Radiology, what kind of OLA questions can I expect?
     

    You will receive questions that cover GI, GU, Interventional, MSK, Neuroradiology, Nuclear, Pediatrics, and Thoracic.

     
    Will CME credit be given for participation in OLA?
     

    No. The ABR is not accredited by the Accreditation Council for Continuing Medical Education (ACCME) to provide CME. However, as soon as someone completes their OLA progress requirement for a particular year, their self-assessment CME (SA-CME) obligations will decrease from 25 to 15 for the previous three-year period. The total number of CME credits required (75) will remain the same; only the percentage that need to be SA-CME will change.

     
    I recently passed an MOC exam. Will I be required to participate in OLA?
     

    Diplomates must participate in OLA when it becomes available for their specialty unless they choose to take the five-year traditional exam.

     
    How should I study?
     

    The goal with all OLA content is that diplomates won’t have to study. All content is intended to be “walking-around knowledge,” or answers the average diplomate should know.

     
    How will OLA affect initial certification?
     

    This change will not impact the ABR’s initial certification exams for diagnostic radiology, interventional radiology/diagnostic radiology, radiation oncology, medical physics, or any subspecialties.

     
  • + Answering OLA Questions

    How many questions do I have to answer?
     
    • For a DR certificate only, or for a DR certificate and one subspecialty: a minimum of 52 question per year
    • For a DR certificate and two or more subspecialties: a minimum of 52 questions per year for each subspecialty certificate
    • For IR/DR only: a minimum of 52 questions per year
    • For IR/DR and any subspecialty certificate(s): a minimum of 52 questions for your IR/DR certificate, and 52 per year for any subspecialty certificate you are maintaining
    • For RO: a minimum of 52 questions per year
    • For MP: a minimum of 52 questions per year for each specialty certificate
     
    How will I receive my questions?
     

    You will be notified once a week via email that you have two question opportunities available. You may choose to answer the questions immediately or at a more convenient time. Each question opportunity will be available for four weeks. When you choose a question, you will have a limited amount of time to answer it.

     
    Will I know if my answer is correct?
     

    Yes, you will receive immediate feedback. Each question will have a brief explanation of the correct answer and at least one reference will be given.

     
    What if my answer is incorrect?

    You will receive a similar question on the same topic in the future.

     
    Do I need to answer the included survey questions before I get my answer?
     

    No. You can skip them from the top of the page. The “confidence” and “relevant” survey questions are designed to be a gauge and learning tool for OLA users. Users will have the ability to review their confidence and question relevance information in the history tab and compare it to their correct and incorrect answers.

     
    What happens if I don’t answer a question and the timer runs out?
     

    If you don’t select an answer and your timer runs out, the question will be counted as incorrect. If you selected an answer and your timer runs out, the system will use the answer you selected to score that question.

     
    How often may I decline a question?
     

    You will receive 10 declines per year for each set of 104 question opportunities you receive. These are intended to be used when a question arises that isn’t a part of your clinical work. Unused declines expire at the end of the calendar year.

     
    Will I have 10 declines throughout the year if I do not use them?
     

    You will start with 10 in January. Unused declines expire at the end of the calendar year.

     
    How long will each question be available?
     

    Each opportunity will be available for four weeks.

     
    How many questions do I need to answer correctly?
     

    Each question is rated individually, and each diplomate receives a different set of questions, so each passing standard will vary slightly. You will have a dashboard showing your performance against the passing standard well in advance of answering 200 questions, so you’ll know how you’re doing.

     
    What do I do once I answer 52 questions per certificate maintained?
     

    You have two options regarding OLA participation for the rest of the year.

     

    Take a break: You may wait until next year to resume answering questions.

     

    Keep going: You may choose to answer more than 52 questions a year. If you do, you will reach the 200-question OLA evaluation threshold faster and have your OLA performance evaluation conducted sooner. There is no penalty to continue participating.

     
    What if I don’t answer 52 questions during the year?
     

    If you do not answer 52 questions a year, the remaining questions will be counted as incorrect. Please remember that the questions are based on “walking-around knowledge,” not something that requires studying.

     
    Why does it take some time for a question to load?
     

    Questions may take longer because OLA loads the entire set of images so that there is no delay once the timer starts. The load time is also dependent on your internet connection speed. For example, if there are three images to load, and you are using a public access Wi-Fi, you will likely experience a longer load time. If you are unable to wait for a question to load, you can select cancel and your question opportunity still will be available.

     
    Why do some questions allow three minutes and others only one minute to answer?
     

    The timing depends on the complexity of the question. Question timing is determined by the volunteer committees, which consist of content experts who write, vet, and compile OLA questions. In January 2020, the time allotted for many OLA questions was extended. Please see here for a chart showing the time allotted for different question types.

     
    I selected the “Answer Now” button on my dashboard page, but after the question loaded, I clicked “Cancel.” What happens? Does this count as an opportunity?
     

    Selecting “Cancel” before the question appears on the screen does not count as a question opportunity. A question opportunity is taken only when the question is seen. At that point, diplomates should either answer the question or decline it. Choosing “Cancel” at that point counts as an incorrect response.

     
    If I decline a question, does the system load a different question or bring me back to the dashboard?
     

    Declining a question will bring the diplomate back to the dashboard and use a question opportunity.

     
    How do I use the window/level item tool (looks like a sun or a gear)?
     

    After you select the window/level tool, place your cursor over the image and left-click on the mouse. Drag up and down or left and right to adjust your image. You will see the change to brightness and contrast. To reset the image, select the “Refresh” icon.

     
  • + Scoring and Performance Evaluation

    Definitions

     

    Annual Progress Requirement

    The annual progress requirement is the number of OLA questions that a diplomate needs to answer each year. For most, this is 52 questions per year, but it will vary depending on how many certificates you are maintaining. Not completing the annual progress requirement leads to forfeited questions.

     

    Scorable Question

    To be deemed “scorable,” and therefore included in your current cumulative score and annual performance evaluation, a question needs at least 50 diplomate responses, 10 question ratings, and acceptable psychometric statistics.

     

    Passing Standard

    The OLA passing standard is criterion referenced; OLA is not graded on a curve, nor do we compare your performance to that of your peers. The passing standard for each OLA question is established individually and the aggregate rating for your 200 questions defines your passing standard. The passing standard for each question is established by OLA participants who volunteer as question raters. This is an important characteristic of OLA; the radiology community sets the passing standard for each question.

     

    Current Cumulative Score

    Your current cumulative score will be available after you have answered at least 52 scorable questions and will include up to 200 of your most recent scorable questions. Your current cumulative score is updated every Sunday and does not impact your Part 3 status.

     

    Annual Performance Evaluation

    The annual performance evaluation occurs on January 1 after 200 scorable questions have been answered and will impact Part 3 status. After your first evaluation, the annual performance evaluation will occur on January 1 each year for your most recently answered 200 scorable questions.

     

    Forfeited Questions

    Questions that are required but unanswered for a given year are called “forfeited questions,” and they count as incorrect responses. For example, if you are required to answer 52 questions and only answer 50, the two unanswered questions will be considered “forfeited” and counted as incorrect.

     

    Scoring Basics

     

    How is the passing standard set for OLA?

    The passing standard for OLA is criterion referenced; OLA is not graded on a curve, nor do we compare your performance to that of your peers. The passing standard for each OLA question is established individually and the aggregate rating for your 200 questions defines your passing standard.

     

    Who sets the passing standard for OLA questions?

    The passing standard for each question is established by OLA participants who volunteer as question raters. This is an important characteristic of OLA: the radiology community sets the passing standard for each question.

     

    Who can participate as a question rater?

    OLA participants who watch a short question-rating video on the OLA site and complete a four-question quiz may participate. If you watch the video and take the quiz, you will then have the ability to rate each question after you answer it and help establish the passing standard.

     

    How much above the standard do I need to be to pass?

    A passing score only requires that you meet or exceed the OLA passing standard.

     

    How many questions do I have to answer correctly?

    In OLA, the passing standard is unique to each diplomate based on the questions they have answered. Therefore, we can’t anticipate an exact percentage needed to pass. Your current cumulative score is updated weekly, so you can closely monitor your performance. Once you have answered 52 scorable questions, you will receive weekly information on how you are performing against the OLA passing standard for your available scorable questions.

     

    How can I tell which questions are included in my annual performance evaluation?

    You may review your question history, which will include up to your latest 250 responses, but it will not indicate which questions are included in your annual performance evaluation. The questions included in your performance evaluation will change over time as different questions become “scorable.”

     

    What makes a question scorable?

    To be deemed “scorable,” and therefore included in your current cumulative score and annual performance evaluation, a question needs at least 50 diplomate responses, 10 question ratings, and acceptable psychometric statistics.

     

    What are forfeited questions?

    Questions that are required but unanswered for a given year are called “forfeited questions,” and they count as incorrect responses. For example, if you are required to answer 52 questions and only answer 50, the two unanswered questions will be considered “forfeited” and counted as incorrect.

     

    How can I see if I have forfeited questions?

    You can identify if you have forfeited questions included in your score in two places:

    1. Annual progress dashboard for the current and previous year.
    2. My OLA History.

    We recommend contacting a certification manager at information@theabr.org or (520) 790-2900 to discuss forfeited questions and how these impact your current cumulative score.

     

    Performance Evaluation Timing

     

    How many more questions do I need to answer to see my score?

    You will see your current cumulative score after you have answered 52 scorable questions.

     

    I see my OLA performance dashboard. Why isn’t there score data?

    All diplomates in OLA will see their performance dashboard; however, the dashboard may not be populated with performance data yet. Current cumulative scores require 52 scorable questions to be answered before data is presented. Annual performance evaluations occur on January 1 after 200 scorable questions have been answered. The annual performance evaluation section will provide an estimated date at which the next evaluation will be performed based on your question response frequency.

     

    My colleagues have their current cumulative score, but I don’t have mine yet. When will I receive my current cumulative score?

    Your current cumulative score will be available after you have answered 52 scorable questions. For a question to be deemed scorable, and included in your assessment, it needs to have at least 50 diplomate responses, 10 question ratings, and acceptable psychometric statistics. Therefore, there may be a delay between answering 52 questions and when your current cumulative score is provided. After you have answered 52 scorable questions, your current cumulative score will be updated every Sunday on your OLA performance dashboard to provide up-to-date information on your OLA performance.

     

    How often are my scores updated?

    Your current cumulative score will be updated every Sunday after you have answered a minimum of 52 scorable questions. Annual performance evaluations are updated every January 1 based on your latest 200 scorable questions.

     

    Why does my current cumulative score go down when I’m answering questions correctly?

    There are two primary factors that might impact your current cumulative score from week to week. First, the OLA passing standard for each question is continuously being updated based on questions ratings from additional participants as a question continues to be delivered after you’ve answered it. Because the number of question ratings for each question is quite large, the changes to the passing standard for each question are quite small. However, as you answer more OLA questions, these small changes can add up to a slight change in your score.

     

    Second, if you have answered more than 200 questions, your current cumulative score is continuously being based on a different “set” of questions as you answer more questions. Remember, your current cumulative score is based on the most recently answered 200 scoreable questions. For example, if you have answered 250 questions, your current cumulative score is likely represented by your performance on OLA questions 51 to 250. If you answer two more questions the following week, the next Sunday your current cumulative score will likely be based on your performance on OLA questions 53 to 252. We say “likely” because this assumes all the questions in that set were scoreable. To be deemed “scorable,” and therefore included in your current cumulative score, a question needs at least 50 diplomate responses, 10 question ratings, and acceptable psychometric statistics.

     

    When will I receive my annual performance evaluation?

    Annual performance evaluations occur on January 1 after 200 scorable questions have been answered. The annual performance evaluation section on your OLA performance dashboard will provide an estimated date at which the next evaluation will be performed based on your question response frequency.

     

    Why do annual performance evaluations occur on January 1?

    OLA annual performance evaluations occur on January 1 to give you time to complete any outstanding items or answer additional questions, if needed, to raise your performance before the MOC annual review on March 2. After you receive your first evaluation, you will receive an annual performance evaluation every January 1 based on your latest 200 scorable questions.

     

    Results and Next Steps

     

    When will my OLA performance impact my certificate(s)?

    OLA is one way to meet the Part 3 MOC requirement. However, because OLA requires continuous participation, the time at which your OLA performance will impact your certificate will depend on your specific timeline and certificate. We recommend contacting a certification manager at information@theabr.org or (520) 790-2900 to discuss this issue.

     

    How will I know if my OLA score is above the passing standard?

    Your current cumulative score will appear on your OLA performance dashboard with a graph indicating your performance as it relates to the passing standard for the scorable questions you have answered. This information is updated every Sunday.

     

    What will happen if I’m below the passing standard after answering 52 scorable questions?

    Your current cumulative score will be updated to show your current performance. However, your OLA performance will not impact your MOC status until you receive an annual performance evaluation, which occurs on January 1 after you have answered 200 scorable questions. You can see the estimated date for your first annual performance evaluation on your OLA performance dashboard.

     

    How do I remediate my OLA performance if I’m below the passing standard?

    There are two ways:

    • If you answer additional OLA questions and your performance meets or exceeds the passing standard at any time during the 12 months between annual reviews, your Part 3 status will immediately change from “Fail” to “Pass.”
    • If you prefer, you can take and pass a traditional exam. Doing so will ensure your MOC Part 3 status will remain “Meeting” for five years. Also, after passing an exam, OLA will be re-set and any previous scoring deficit will be removed.

    For additional details on how to remediate your performance, contact a certification manager at information@theabr.org or (520) 790-2900.

     

    I’m seeing data in my current cumulative score details for previously selected clinical practice areas. I changed my practice profile a long time ago. Why are those questions included?

    Performance evaluations include the most recent 200 scorable questions. Although updating your practice profile will immediately modify the questions you receive, changing your practice profile will not exclude previously answered questions from your performance evaluation if they fall within the 200 most recent scorable questions answered. Eventually, as you answer additional questions beyond 200, they will fall out of your subsequent performance evaluations.

     

    If I fail OLA, will I lose my certificate?

    Successful OLA performance is one of two ways to satisfy your Part 3 requirement for MOC. The other option is to take and pass a traditional five-year exam.

    We recommend contacting a certification manager at information@theabr.org or (520) 790-2900 to discuss your OLA performance, when/how performance will impact your certificate, and steps that can be taken to remediate your performance.

     

    What if I am failing on January 1 when my OLA performance evaluation is done?

    If you are failing on your annual performance evaluation, you will receive a supplemental evaluation the moment you meet or exceed the OLA passing standard at any time; that is, we will replace your fail with your passing score.

    We recommend contacting a certification manager at information@theabr.org or (520) 790-2900 to discuss your OLA performance, when/how performance will impact your certificate, and steps that can be taken to remediate your performance.

     
  • + OLA and Subspecialty Certificates (CAQs)

    When does OLA begin for my subspecialty certificate (CAQ)?
     

    Most diplomates participating in MOC are eligible for OLA. It is not available for a hospice and palliative or a pain medicine certificate. Those diplomates must take a traditional exam.

     
    I hold a subspecialty certificate. How will my practice profile be set?
     

    Since you are maintaining a subspecialty certificate, your practice profile will be set for that subspecialty and can’t be modified.

     
  • + Technical Requirements

    What devices, browsers, and operating systems are compatible with OLA?
     

    We have optimized the OLA experience for desktop computers, laptops, and tablets. Preferred operating systems include the latest version of Windows, iOS, MacOS and Android.

     
    What about mobile support?
     

    OLA is designed to work with iPads and similar tablet devices. Preferred operating systems include the latest version of Windows, iOS, MacOS and Android.

     

 

OLA Needs Assessment Data

The ABR has administered more than 4.5 million items as part of Online Longitudinal Assessment (OLA). The tool is intended to allow each individual to track their own question history and identify gaps in knowledge. In addition, when viewed in aggregate, the relative rate of correct responses may provide guidance for developers of education courses and materials to target topics that represent areas of greatest need.

 

The content committees consist of experts who develop the questions and attempt to cover a domain that corresponds to the discipline or specialty selected. Within each domain, a range of responses is not unexpected. The results are presented in quartiles: Quartile 4 represents items that were typically answered satisfactorily, whereas Quartile 1 topics represented a challenge for the group in aggregate. Click here to download the full report.

 

This is the first iteration of the presentation of this data, and we expect that it will undergo refinement over time; individuals who have suggestions on the formatting are invited to submit them on our website. We plan to release quarterly updates based on a rolling average of responses collected over the preceding six months.