Frequently Asked Questions About OLA Scoring

By Kalpana M. Kanal, PhD, ABR Trustee; Ben Babcock, PhD, ABR Associate Director of Psychometrics; Geoffrey S. Ibbott, PhD, ABR Associate Executive Director for Medical Physics; Jennifer Stickel, PhD, ABR Trustee; and Matthew B. Podgorsak, PhD, ABR Board of Trustees Chair

2025;18(3):7

In a recent Beam article, ABR Associate Director of Assessments Research and Strategy Brooke Houck, PhD, described how OLA scoring provides an accurate representation of diplomates’ knowledge. Her article summarized the scoring process and explained how a diplomate’s score changes depending on the question asked and its rating.

On March 11, 2025, Drs. Babcock, Kanal, and Ibbott presented an ABR webinar on OLA scoring to 210 members of the American Association of Physicists in Medicine (AAPM). While we answered some questions live and in the session chat, we could not cover all 63 questions in the allotted time. A recording of the webinar together with a compilation of the questions and our responses are posted on the AAPM website.¹ In follow-up to Dr. Houck’s article, below is a subset of the questions from the webinar that may help diplomates understand the OLA scoring process better.

As a reminder, the OLA question scoring formula is:

1. Questions are going to be much harder for people who do not practice in certain areas. How does the ABR account for varying levels of question difficulty? 

It is very important to have OLA question raters from a variety of medical physics practices and job roles. The more diverse the pool of raters, the more representation of all practice types we have in setting the OLA passing standard. We have diversity in raters and encourage increased diversity in thought by having more diplomates take action to be OLA question raters at the time an OLA item is administered to them.

2. Is the denominator ‘A’ a weighted average to account for the raters’ years of certified participation in the system?

The mean question rating is not weighted based on any rater demographic characteristics. Every rater has an equal voice in the process by design.

3. Do you feel that the binary choice for question raters biases the denominator high?

Psychometric literature has demonstrated that every method of rating questions has advantages and disadvantages. The ABR has picked a dichotomous rating system for a variety of reasons. Among those is that boiling the task down to a yes/no choice is cognitively simpler than estimating a proportion or percentage from a population. Also, systems where raters estimate a percentage or proportion consistently show a strong inward conditional bias toward the mean (see Wyse & Babcock, 2019). Dichotomous ratings tend to show much less of this conditional bias.

4. If you perform below average, do you fail OLA?

No, performing below average does not mean you fail OLA. The ABR uses criterion-referenced standards, meaning each question is evaluated based on whether the participant meets a set standard, not relative to others’ performances. Therefore, it is possible to pass OLA even if your performance is below the average of other participants.

5. If my OLA current cumulative score does not change for a while, does that mean I am scoring similarly to my peers? 

Your OLA current cumulative score level or change in your score level is not relative to other people in OLA. The ABR uses criterion-referenced standards, not norm-referenced standards.

6. If my OLA current cumulative score does not change for a while, does that mean my questions have not been calibrated yet?

There are a variety of reasons why a score may not change over a long period of time. It could be because many of your questions are not yet calibrated. It could also mean that your performance over that time has been consistent compared with your past performance.

7. As question scores are calibrated by additional question raters, are individual scores also adjusted?

Currently we recalculate every question’s average rating every scoring period (i.e., each week). This means that as new ratings are given, your OLA current cumulative score could change.

8. How soon after you have answered a question is your OLA current cumulative score updated? 

Your current cumulative score is updated weekly.

9. Is it better to take the Continuing Certification Exam or engage in OLA?

Presenting these as either/or options or pathways is not the best way to think about fulfilling your Part 3 requirement. You may want to consider the following approach:

• Answer OLA questions. Consider answering more than the minimum per year so you will have an idea of your OLA current cumulative score before year four of your five-year cycle for Part 3.
• If your OLA current cumulative score is below the standard during years four and five of your cycle, consider signing up for every opportunity to take the Continuing Certification Exam.
• Continue answering OLA questions if you are waiting for a Continuing Certification Exam opportunity. It may still be possible to pass OLA during the final year of your five-year cycle. If you do pass this way, you can cancel your Continuing Certification Exam appointment.

The ABR is not making public exact passing statistics for the Continuing Certification exams due to their small sample size and a population of test takers that is quite different from Initial Certification exams. The performance statistics that the ABR has examined, however, indicate a higher success rate for fulfilling the Part 3 requirement via OLA compared with the Continuing Certification exams. This makes forgoing OLA completely and relying solely on the Continuing Certification Exam a risky proposition.

10. Do I pass OLA if at any single time in years one through four of my Part 3 cycle my OLA current cumulative score is above zero? 

No. Your OLA current cumulative score must be equal to or greater than zero sometime during the fifth year of your Part 3 cycle. The fifth year is the OLA evaluation year.  

11. Do I have to maintain a score above zero all the time in order to pass? 

No. Your OLA current cumulative score must be equal to or greater than zero at least one time during the fifth year of your Part 3 cycle to meet the Part 3 requirement. 

More information can be found at https://www.theabr.org/online-longitudinal-assessment-ola, as well as from articles published in The Beam and other posts on www.theabr.org. 

¹The recording and the Q&A are available only to AAPM members.

Return to The Beam