From the Board of Trustees: The Life Cycle of an ABR Exam Question
By ABR Board of Trustees Chair Matthew B. Podgorsak, PhD; and ABR Trustees Anne M. Covey, MD; Kate Maturen, MD, MS; and Jennifer Stickel, PhD
June 2026;19(3):4
ABR exams1 rely on relevant and valid content that reasonably covers the knowledge domain of the specialty. Hundreds of volunteers contribute their efforts and expertise to develop and compile quality items2 that will accurately assess candidates’ knowledge of the subject matter. Although individuals draft most of the items, committees of subject matter experts are ultimately responsible for refining and assembling the content. While the number varies from year to year based on the needs of each exam, the ABR produces approximately 5,000 new items a year.
The creation of an exam starts with a blueprint, which is determined with input from the item-writing committee responsible for a specific portion of the domain (e.g., musculoskeletal radiology) within a specific exam (e.g., the DR and IR/DR Qualifying [Core] Exam). The blueprint represents a balance of coverage within the domain, prioritizes relevance, and avoids esoterica, trivia, and data that physicians or physicists would typically access from a web search.
Volunteers draft items to assess the elements included in the blueprint. A draft item typically includes a stem, which presents background information (when it is relevant to the question) and poses a question to assess knowledge. Often, an item includes a set of images that are the focus of the question. The draft item includes not only the correct answer (“key”) but also two to four incorrect responses (“distractors”); the latter are carefully written to be plausible but not appropriate answers to the question. As part of the draft, item writers submit an appropriate reference to allow the committee to validate the answer.
Volunteers upload items, along with relevant images, into our proprietary data management system, where they are formatted. The images are reviewed by staff for protected health information and quality, and the wording is reviewed by ABR editors. The items are then shared with the item-writing committee for detailed online review and subsequent in-depth discussion via real-time videoconferencing among members. This part of the process involves respectfully challenging one’s colleagues while refining items, and the goal is mitigating ambiguity that might confuse the examinee. Specifically, no “trick questions” are permitted.
At several steps along the way, professional editing staff carefully apply established standards for structure and language to reduce variability in items and minimize distraction for the candidate when the proposed item is eventually included on an exam.
Once the committee gives final approval of an item, it is eligible to be included in an exam. Each subspecialty committee selects items for a given exam, balancing content according to the blueprint. The entire exam undergoes a final review by committee members from each domain. This step is important to ensure the validity and relevance of content and to consider the rigor and relevance of the finished product.
All these quality improvement steps lead to an exam that is suitable for release. Another important quality improvement step takes place after the exam. A careful statistical analysis looks for items that are ambiguous or confusing (for example, the presence of two reasonably correct answers). This post hoc review by psychometricians and subject matter experts may result in a scoring modification, if they conclude that an item is sufficiently flawed to warrant its removal from scoring. This final QA step takes place before candidates receive their exam results.
Three of the themes mentioned above are especially important to the production and administration of exam content. First, the final product is the work of dozens of subject matter experts who collectively create and adjust content to be fair and relevant. Staff members from exam development, psychometrics, and editing also review each exam item and the final exam at multiple stages. Second, the ABR intentionally avoids including items that are potentially confusing or ambiguous. Finally, the value of thoughtful and informed volunteer contributions to the process cannot be overstated.
- “Exam” in this context refers to any cognitive assessment offered by the ABR, including point-in-time exams and Online Longitudinal Assessment (OLA). Although most of this discussion focuses on point-in-time computer-based exams, many of the principles and processes also apply to content development and administration for oral exams and OLA.
- “Item” is the preferred term in exam development and will be easily recognized by our volunteers. The term is essentially interchangeable with “question” in this discussion.
