By Ben Babcock
When the ABR administers Initial Certification exams, some people pass and others don’t. How does the ABR decide where to draw the line between a passing and failing performance?
The answer is a psychometric process called standard setting. ABR Initial Certification exams use a criterion-referenced standard setting process (Angoff) that involves a volunteer panel of certified diplomates. Those subject matter experts (Angoff participants) make judgments about the difficulty level of questions on an exam to decide where the pass/fail point should be.
How does the process work? The Angoff participants first define what it means to be a qualified candidate by identifying the knowledge and skills of a candidate who would just barely pass the exam. The definition varies based on the specifics of the discipline, but it often includes statements like:
- Practices safely.
- Does well with typical cases.
- Can tell that something is wrong but may not know the precise reason why.
- May struggle with unusual cases or rare presentations.
- May not have great knowledge of cutting-edge equipment or techniques.
Most candidates for certification exceed those levels of ability, but the Angoff method attempts to quantify the dividing line between the qualified and not qualified examinees. After defining the qualified candidate, each Angoff participant takes the exam. Having to answer every question places them in the same proverbial seat as an exam candidate. After an Angoff participant has read and answered a question, they decide whether a qualified candidate would answer the question correctly.
When Angoff participants are finished with the first round of ratings, a psychometrician guides them through a second round that focuses on a group review of selected questions. These may be questions the participants wanted to discuss with the group. This discussion allows them to voice rationales for their ratings and consider alternative viewpoints. Participants then have the option to submit a new rating.
Once the Angoff participants are finished with the activity, ABR staff have the data they need to recommend a pass/fail point for the exam being evaluated. This process is how the ABR uses the judgment of certified diplomates to determine the knowledge required to pass an ABR Initial Certification exam.
It is ABR volunteers who drive the process and make it successful. The volunteers who participate in standard setting are critical to the process of creating a fair and valid exam. Thank you, volunteers! We could not do it without your expert judgments.
Ben is the associate director of psychometrics at the ABR.
