Online Longitudinal Assessment (OLA)
On this page
Introduction
Online Longitudinal Assessment (OLA) is a convenient way to satisfy Part 3 of the Continuing Certification (MOC) program. It is an option for all diagnostic radiology, interventional radiology/diagnostic radiology, medical physics, radiation oncology, and subspecialty (neuroradiology, nuclear radiology, and pediatric radiology) diplomates.
Within OLA, most diplomates are provided two question opportunities each week per certificate maintained, a limited amount of time to answer the questions after opening them, and immediate feedback. In addition, diplomates receive a brief rationale for the correct answer and a reference. Questions, which are available for four weeks to allow flexibility in participation, are designed to test walking-around knowledge and don’t require studying.
OLA is designed to have minimal impact on a diplomate’s workday and requires neither time away from work nor travel expenses. The potential for retesting areas of weakness provides a further opportunity for diplomates’ self-assessment of their professional growth.
Please whitelist the ABR domain (@theabr.org) in your email to receive important communications about this ongoing assessment. If you have questions, please contact us at information@theabr.org or 520-790-2900.
Quick Tips for OLA Success
- OLA Login: New questions are posted weekly; they expire in four weeks.
- Device: Desktop, laptop, or large tablet using Chrome as the browser. (Cell phones don’t provide optimal results.)
- Answer at least 52 questions yearly for each required certificate. Answering fewer leads to “forfeited” (scored incorrect) questions.
- Forfeits, especially in years four and five of your Part 3 cycle, pull down your score.
- You can decline some questions; they will not count in your score.
- Your score compares your performance with a criterion-referenced minimum passing standard that is unique to your set of questions.
- You pass OLA by being at or above the standard anytime in the last year of your five-year Continuing Certification Part 3 cycle.
- Don’t consult other people, AI, or any outside materials (e.g., internet, textbooks) when answering OLA questions.
- Don’t stop answering questions until you meet the yearly minimum for each required certificate.
Continuing Certification Part 3 Five-Year Evaluation Cycle
Each maintained certificate has a Part 3 evaluation date on a rolling five-year cycle. To satisfy a Part 3 evaluation, a diplomate must meet or exceed their standard in a current cumulative (i.e., weekly) OLA score in year five or pass a Continuing Certification Exam in year four or five of their Part 3 cycle. Your specific Part 3 deadline is displayed in myABR and on your OLA dashboard.
If you need more information about the Continuing Certification Exam, please click on one of the following links:
- Diagnostic Radiology Continuing Certification Exam
- Interventional Radiology Continuing Certification Exam
- Medical Physics Continuing Certification Exams
- Radiation Oncology Continuing Certification Exam
OLA FAQs
-
+ Getting Started
Who is eligible for OLA?
Diplomates participating in Continuing Certification (MOC) are eligible for OLA.
I just became certified. When do I begin using OLA?
You will begin OLA in January of the year following your certification.
How long do I need to participate in OLA?
OLA is a continuous assessment in which diplomates involved in Continuing Certification (MOC) will participate every year throughout their careers.
How does OLA fit within the Continuing Certification (MOC) program?
OLA is one way to fulfill the Part 3 requirement. Diplomates need to continue to complete requirements for Parts 1, 2, and 4 as well.
Can I choose the clinical practice areas included in OLA?
Diagnostic radiologists and interventional radiologists can choose their clinical practice areas. For more information, please see your OLA dashboard, this article for DR, or this article for IR.
If I’m a diagnostic radiologist who sets my practice profile as General Diagnostic Radiology, what kind of OLA questions can I expect?
You will receive questions that cover GI, GU, Interventional, MSK, Neuroradiology, Nuclear, Pediatrics, and Thoracic.
Will CME credit be given for participation in OLA?
No. The ABR is not accredited by the Accreditation Council for Continuing Medical Education (ACCME) to provide CME. However, diplomates who meet their OLA progress requirement for a particular year do not need to complete self-assessment CME (SA-CME). All diplomates need to complete 75 Category 1 CME credits in the previous three-year period. For diplomates not completing their OLA annual progress requirement, 25 of the 75 credits must be SA-CME. Diplomates completing their OLA annual progress requirement can forego SA-CME.
I recently passed a Continuing Certification Exam. Will I be required to participate in OLA?
There are two ways to satisfy the Part 3 requirement: participate in OLA or pass the Continuing Certification Exam every five years.
How should I study?
The goal with all OLA content is that diplomates won’t have to study. All content is intended to be “walking-around knowledge,” or answers the average diplomate should know.
-
+ Answering OLA Questions
How many questions do I have to answer?
- For a DR certificate only, or for a DR certificate and one subspecialty: a minimum of 52 question per year
- For a DR certificate and two or more subspecialties: a minimum of 52 questions per year for each subspecialty certificate
- For IR/DR only: a minimum of 52 questions per year
- For IR/DR and any subspecialty certificate(s): a minimum of 52 questions for your IR/DR certificate, and 52 per year for any subspecialty certificate
- For RO: a minimum of 52 questions per year
- For MP: a minimum of 52 questions per year for each specialty certificate
How will I receive my questions?
Two questions are made available every Monday morning in OLA. Most diplomates are notified once a week via email when the two question opportunities are available. You may choose to answer the questions immediately or at a more convenient time. Each question opportunity will be available for four weeks. When you choose a question, you will have a limited amount of time to answer it. You may modify how often you receive OLA email notifications in your OLA Preferences, but this will not change when the questions become available or when they expire.
Will I know if my answer is correct?
Yes, you will receive immediate feedback. Each question will have a brief explanation of the correct answer and at least one reference.
What if my answer is incorrect?
The incorrect answer will be counted against your cumulative score. To assess whether you learn from the feedback, you will receive a similar question on the same topic in the future.
Do I need to answer the included survey questions before I get my answer?
No. You can skip them from the top of the page. The “confidence” and “relevant” survey questions are designed to be a gauge and learning tool. You have the ability to review your confidence rating information in the history tab and compare it to your correct and incorrect answers.
Does the confidence rating have an impact on my score?
No. The confidence rating is solely for your use as a learning tool/reference point. It is not taken into account at all for scoring purposes.
What happens if I don’t answer a question and the timer runs out?
If you don’t select an answer and your timer runs out, the question will be counted as incorrect. If you selected an answer and your timer runs out, the system will use the answer you selected to score that question.
How often may I decline a question?
You can use 10 declines per year for each set of 104 question opportunities you receive. Declines are intended to be used when a question arises about something that isn’t part of your clinical work. Declined questions do not appear in your question history and have no impact on the topics of future questions. Diplomates with a certificate in Nuclear Medical Physics are provided five declines per year.
If I don’t use all my declines in one year, can I carry them over to the next year?
No. Unused declines expire at the end of the calendar year.
How long will each question opportunity be available?
Each question opportunity is available for four weeks.
Why am I seeing a question that I’ve already answered?
It is common assessment practice to continue using well-performing questions – items that have good statistical properties and are well received by diplomates. As a result, you may receive a question you have previously answered. However, you will not receive a question that you’ve already answered more frequently than every two years. Reusing questions decreases the time between when you answer a question and when it is incorporated into your cumulative score (because it is already a scorable question). In addition, reusing well-performing questions makes question development more efficient, allowing us to maintain a robust pool.
How many questions do I need to answer correctly?
Each question is rated individually, and each diplomate receives a different set of questions, so each person’s passing standard will vary slightly. You will have a dashboard showing your performance against the passing standard well in advance of the Part 3 deadline, so you’ll know how you’re doing.
What do I do once I answer 52 questions per certificate maintained?
You have two options regarding OLA participation for the rest of the year:
- Take a break: You may wait until next year to resume answering questions.
- Keep going: You may choose to answer more than 52 questions a year. Questions answered beyond the annual requirement will continue to count toward your cumulative score. Correct answers may positively affect your score, whereas incorrect answers may negatively affect your score.
What if I don’t answer 52 questions during the year?
The remaining questions will be counted as incorrect. Please remember that the questions are based on “walking-around knowledge” and shouldn’t require studying.
Why does it take some time for a question to load?
Some questions may take longer because OLA loads the entire set of images so that there is no delay once the timer starts. The load time is also dependent on your internet connection speed. For example, if there are three images to load, and you are using a public access Wi-Fi, you will likely experience a longer load time. If you are unable to wait for a question to load, you can select “cancel” and your question opportunity still will be available.
Why do some questions allow three minutes and others only one minute to answer?
The timing depends on the complexity of the question. Question timing is determined by the volunteer committees, which consist of content experts who write, vet, and compile OLA questions. Please see here for a chart showing the time allowed for different question types.
I selected the “Answer Now” button on my dashboard page, but after the question loaded, I clicked “Cancel.” Does this count as an opportunity?
Selecting “Cancel” before the question appears on the screen does not count as a question opportunity. A question opportunity is taken only when the question is seen. At that point, diplomates should either answer the question or decline it. Choosing “Cancel” at that point counts as an incorrect response.
If I decline a question, does the system load a different question or bring me back to the dashboard?
Declining a question will bring you to a survey. After responding, you can elect to answer another question to go back to the dashboard.
How do I use the window/level item tool (looks like a sun or a gear)?
After you select the window/level tool, place your cursor over the image and left-click on the mouse. Drag up and down or left and right to adjust your image. You will see the change in brightness and contrast. To reset the image, select the “Refresh” icon.
-
+ Scoring and Performance Evaluation
Definitions
Annual Progress Requirement
The annual progress requirement is the number of OLA questions that a diplomate needs to answer each year. For most, this is 52 questions per year, but it will vary depending on how many certificates you are maintaining. Not completing the annual progress requirement leads to forfeited questions.
Scorable Question
To be deemed “scorable,” and therefore included in your current cumulative score, a question needs at least 50 diplomate responses, 10 question ratings, and acceptable psychometric statistics.
Passing Standard
The OLA passing standard is criterion referenced; OLA is not graded on a curve, nor do we compare your performance to that of your peers. The passing standard for each OLA question is established individually and the aggregate rating for your 200 questions defines your passing standard. The passing standard for each question is established by OLA participants who volunteer as question raters. This is an important characteristic of OLA; the radiology community sets the passing standard for each question.
Current Cumulative Score
Your current cumulative score will be available after you have answered at least 52 scorable questions and will include up to 200 of your most recent scorable questions. Your current cumulative score is updated every Monday morning.
Continuing Certification Part 3 Deadline
Part 3 requires passing an OLA performance evaluation or a Continuing Certification Exam for each maintained certificate every five years. Each maintained certificate has a certification evaluation date at the end of a five-year cycle. An OLA current cumulative score meeting or exceeding the standard in year five or a Continuing Certification Exam with a passing result in year four or five is required to continue maintenance and start the next five-year cycle. Not all diplomates have the same Part 3 evaluation deadline.
Forfeited Questions
Questions that are required but unanswered for a given year are called “forfeited questions,” and they count as incorrect responses. For example, if you are required to answer 52 questions and only answer 50, the two unanswered questions will be considered “forfeited” and counted as incorrect.
Scoring Basics
How is the passing standard set for OLA?
The passing standard for OLA is criterion referenced; OLA is not graded on a curve, nor do we compare your performance to that of your peers. The passing standard for each OLA question is established individually and the aggregate rating for your latest 200 questions defines your passing standard.
Who sets the passing standard for OLA questions?
The passing standard for each question is established by OLA participants who volunteer as question raters. This is an important characteristic of OLA: the radiology community sets the passing standard for each question.
Who can participate as a question rater?
OLA participants who watch a short question-rating video on the OLA site and complete a four-question quiz may participate. If you watch the video and take the quiz, you will then have the ability to rate each question after you answer it and help establish the passing standard.
How much above the standard do I need to be to pass?
A passing score only requires that you meet or exceed your OLA passing standard.
How many questions do I have to answer correctly?
In OLA, the passing standard is unique to each diplomate based on the questions they have answered. Therefore, we can’t anticipate an exact percentage needed to pass. Your current cumulative score is updated weekly, so you can closely monitor your performance. Once you have answered 52 scorable questions, you will receive weekly information on how you are performing against the OLA passing standard for your available scorable questions.
How can I tell which questions are included in my current cumulative score?
You may review your question history, which will include up to your latest 250 responses, but it will not indicate which questions are included in your current cumulative score. The questions included in your current cumulative score will change over time as different questions become “scorable.”
What makes a question scorable?
To be deemed “scorable,” and therefore included in your current cumulative score, a question needs at least 50 diplomate responses, 10 question ratings, and acceptable psychometric statistics.
What are forfeited questions?
Questions that are required but unanswered for a given year are called “forfeited questions,” and they count as incorrect responses. For example, if you are required to answer 52 questions and only answer 50, the two unanswered questions will be considered “forfeited” and counted as incorrect.
How can I see if I have forfeited questions?
You can identify if you have forfeited questions included in your score on the annual progress dashboard for the current and previous year. We recommend contacting a certification manager at information@theabr.org or 520-790-2900 to discuss forfeited questions and how these impact your current cumulative score.
Current Cumulative Score Timing
How many questions do I need to answer to see my score?
You will see your current cumulative score after you have answered 52 scorable questions.
I see my OLA performance dashboard. Why isn’t there score data?
All diplomates in OLA will see their performance dashboard; however, the dashboard may not be populated with performance data yet. Current cumulative scores require 52 scorable questions to be answered before data is presented.
My colleagues have their current cumulative score, but I don’t have mine yet. When will I receive my current cumulative score?
Your current cumulative score will be available after you have answered 52 scorable questions. For a question to be deemed scorable, and included in your assessment, it needs to have at least 50 diplomate responses, 10 question ratings, and acceptable psychometric statistics. Therefore, there may be a delay between answering 52 questions and when your current cumulative score is provided. After you have answered 52 scorable questions, your current cumulative score will be updated every Monday morning on your OLA performance dashboard to provide up-to-date information on your OLA performance.
How often are my scores updated?
Your current cumulative score will be updated every Monday morning after you have answered a minimum of 52 scorable questions.
Why does my current cumulative score go down when I’m answering questions correctly?
This is most likely occurring if you have answered more than 200 questions, since your current cumulative score is continuously being based on a different “set” of questions as you answer more questions. Remember, your current cumulative score is based on the most recently answered 200 scoreable questions. For example, if you have answered 250 questions, your current cumulative score is likely represented by your performance on OLA questions 51 to 250. If you answer two more questions the following week, the next Monday your current cumulative score will likely be based on your performance on OLA questions 53 to 252. We say “likely” because this assumes all the questions in that set were scoreable. To be deemed “scorable,” and therefore included in your current cumulative score, a question needs at least 50 diplomate responses, 10 question ratings, and acceptable psychometric statistics.
Results and Next Steps
When will my OLA performance impact my certificate(s)?
OLA is one way to meet the Part 3 Continuing Certification (MOC) requirement. An OLA current cumulative score meeting or exceeding the standard in year five is required to continue maintenance and start the next five-year cycle. We recommend contacting a certification manager at information@theabr.org or 520-790-2900 to discuss these details.
How will I know if my OLA score is above the passing standard?
Your current cumulative score will appear on your OLA performance dashboard with a graph indicating your performance as it relates to the passing standard for the scorable questions you have answered. This information is updated every Monday.
What will happen if I’m below the passing standard after answering 52 scorable questions?
Your current cumulative score will be updated to show your current performance. However, your OLA performance will not impact your certificate until you reach your Part 3 deadline.
I’m seeing data in my current cumulative score details for previously selected clinical practice areas. I changed my practice profile a long time ago. Why are those questions included?
Current cumulative scores include the most recent 200 scorable questions. Although updating your practice profile will immediately modify the questions you receive, changing your practice profile will not exclude previously answered questions from your current cumulative score if they fall within the 200 most recent scorable questions answered. Eventually, as you answer additional questions beyond 200, they will fall out of your subsequent current cumulative scores.
If I fail OLA, will I lose my certificate?
Successful OLA performance is one of two ways to satisfy your Part 3 requirement for Continuing Certification (MOC). The other option is to take and pass the Continuing Certification Exam. We recommend contacting a certification manager at information@theabr.org or 520-790-2900 to discuss your OLA performance, when/how performance will impact your certificate, and steps that can be taken to remediate your performance.
-
+ OLA and Subspecialty Certificates (CAQs)
When does OLA begin for my subspecialty certificate (CAQ)?
You will begin receiving OLA questions in January of the year following your certification.
I hold a subspecialty certificate. How will my practice profile be set?
Diagnostic radiologists, interventional radiologists, and those with only one DR subspecialty certificate can choose their clinical practice areas (CPAs). For more information on practice profiles for subspecialty certificates, please see your OLA dashboard, this article for DR, or this article for IR.
-
+ Technical Requirements
What devices, browsers, and operating systems are compatible with OLA?
We have optimized the OLA experience for desktop computers, laptops, and tablets. We recommend that you use one of the following supported browsers with OLA:
Older web browsers can have problems with unexpected behavior; these issues are less common in modern web browsers. In general, the two most recent major releases of Chrome, Firefox, Edge, and Safari (on a rolling basis) are supported.
Preferred operating systems include the latest version of Windows, iOS, MacOS, and Android.
What about mobile support?
OLA is designed to work with iPads and similar tablet devices. For the best user experience, we recommend using a desktop or laptop computer.
OLA Needs Assessment Data
The ABR has administered millions of questions as part of Online Longitudinal Assessment (OLA). The tool is intended to allow each individual to track their own question history and identify gaps in knowledge. In addition, when viewed in aggregate, the relative rate of correct responses may provide guidance for developers of education courses and materials to target topics that represent areas of greatest need.
The content committees consist of experts who develop the questions and attempt to cover a domain that corresponds to the discipline or specialty selected. Within each domain, a range of responses is not unexpected. The results are presented in quartiles: Quartile 4 represents questions that were typically answered satisfactorily, whereas Quartile 1 topics represented a challenge for the group in aggregate. Click here to download the full report.
This is the first iteration of the presentation of this data, and we expect that it will undergo refinement over time; individuals who have suggestions on the formatting are invited to submit them on our website. We plan to release quarterly updates based on a rolling average of responses collected over the preceding six months.
