OLA Questions for MP Participants Continually Being Improved

By Kalpana M. Kanal, PhD, ABR Trustee

2021;14(6):8

It has been almost two years since the ABR implemented Online Longitudinal Assessment (OLA) for medical physicists. OLA is based on the concept of walking around knowledge. This is knowledge that a typical medical physicist would need in order to provide general clinical medical physics (MP) services without having to access reference texts or consult with a colleague.

At the time of this writing, the following MP OLA statistics are available:

• Over 3,400 diplomates (99.5%) have answered at least one OLA question and 466,000 OLA questions have been answered.
• 49% of participants have agreed to serve as OLA question raters.
• Fewer than 5% of OLA questions have been declined.
• The average response time for 60-second questions is 29 seconds and for 180-second questions is 60 seconds.
• 94% of participants have completed their 2021 OLA annual progress requirement.

Each year, committees made up of MP diplomates write OLA questions that undergo extensive review before being used. Opportunities for diplomates to provide feedback and rate questions are built into the OLA software. The passing standard is criterion referenced; OLA is not graded on a curve, nor is a diplomate’s performance compared to that of their peers. The passing standard for each OLA question is established by participants who volunteer as question raters, and the aggregate rating of the question defines the passing standard, known as the Angoff value. Each diplomate participating in OLA can become a question rater and assist in setting the OLA passing standard.

In order to continually improve OLA questions, our review process was recently modified to include a statistical analysis of each question and its variant (except for nuclear medical physics where no variants are given). In the near future, this information will be reviewed by the MP associate executive director (AED) with the discipline-specific MP trustees and the OLA question writing committee for appropriate action. All steps in the process are discussed below.

Data Collection:

For each question, the following statistics are collected:

Here is a screenshot of an example OLA item and its variant:

Review Process:

General trends on the statistics are examined, and problem items are identified and reviewed. Problem items are identified by the p-value of the question, the Angoff value, the number of comments received, the frequency of the responses that differ from the key, and the relevance of the question. If the p-value is too high, that may indicate the question is too easy, and if the p-value is too low or below the Angoff value, that may indicate the question is challenging or not phrased correctly, leading to difficulty in interpretation. Problem items are discussed with the committee and question changes are made as appropriate for future use. Question changes may include modifying the question or removing it from the OLA pool. If the feedback received is about a diplomate’s practice – e.g., a diplomate saying “I don’t do this in my practice” – the original question would not be modified. MP OLA assessment is not offered at a granular practice level; i.e., it is not customized to a diplomate’s clinical practice (see AAPM newsletter article on OLA granularity). This review process is in the early stages and still being fine-tuned.

The ABR is continually improving the process of reviewing the statistics and feedback on each OLA question and using it to improve the questions. This process is done annually to introduce questions in the OLA administration. OLA committees are almost at the end of the 2021 review cycle, preparing questions for 2022.

The MP trustees, governor, and AED welcome comments and questions about OLA and thank all participants for their feedback. Please contact us at information@theabr.org.

Return to The Beam