To better ensure the health and safety of our candidates, the ABR has committed to a remote exam platform starting in 2021. The decision was made after weeks of consultation with key stakeholders, including candidates, programs, associations, and societies. We are continuing those discussions as we move forward in our exam development process.
Transitioning from in-person to remote exams will take substantial effort and we want to keep our stakeholders informed. Please bookmark this page or follow us on Twitter to check for updates. We appreciate everyone’s interest as we work to build a new way of delivering exams.
Examinees find many positives in first remote exam administration
Our Diagnostic Radiology Certifying Pilot Exam earlier this month provided an opportunity for us to gain a real-life look at a computer-based remote administration with a limited group of candidates. It was the first time we had administered a remote computer-based exam and the experience was largely a success.
The pilot was taken by 161 candidates, all of whom were given the chance to respond to a post-exam survey. We appreciate the 127 who shared their opinions about the experience; we will use their feedback to improve our platforms and processes.
Some interesting details emerged from the survey:
- More than 95% of respondents found the practice exam useful in helping them prepare.
- More than 80% took the exam at home rather than in a hospital or private practice office.
- More than 95% thought* that installation of the Proctorio®extension was easy to complete.
- More than 95% thought* that check-in through the Proctorio®platform was easy.
- More than 95% thought* that the exam platform (imaging tools, load times, answer entry, etc.) worked well.
- More than 90% thought* that “the image quality was adequate for testing purposes.”
- Survey respondents rated the overall experience a 4.5 out of 5 stars.
* “Agreed” or “Strongly Agreed”
We appreciate everyone who provided feedback. It’s instrumental in helping us ensure the continued delivery of a secure and valid exam, while keeping the exam day experience as pleasant as possible for the new remote administrations.
IT resource information available
During a pilot administration earlier this month, we gained additional insight into how security systems employed by many training programs/health systems may impact candidates taking ABR remote computer-based exams.
As outlined in the Remote Exam Basics Guide, candidates may take their exam in a location of their choosing. We acknowledge that some may pick a room in their training program or health system, raising potential network issues. As a result, we created a webpage with remote computer-based exam technical specifications for institutions.
This new page details the systems and processes used to deliver our remote computer-based exams, including specific domains, component requirement, web services, and website security parameters. Interested programs are encouraged to direct their IT departments to this page for details that may help them mitigate technical issues for candidates during their exam.
What we learned from our remote pilot exam administration
ABR Executive Director Brent Wagner, MD, MBA, discusses what worked and what will be improved upon after the DR Remote Certifying Pilot Exam on January 9, 2021.
Remote exam preparation update
ABR Executive Director Brent Wagner, MD, MBA, provides an update as the organization prepares to start administering remote exams.
More information on exam proctoring
The ABR is working with Proctorio to provide remote proctoring services for oral and computer-based exams in 2021. Here are further details about how the system will be incorporated.
The ABR respects candidates and knows they are overwhelmingly honest and professional. We all want a secure and meaningful exam. ABR feedback will be focused on preventing potential issues, not implementing punitive measures.
The ABR is using Proctorio for remote proctoring. Only ABR staff will be able to review ID checks, room scans, webcam video, and microphone audio of exams. Candidate personal data (i.e., email address and phone number) is never shared with Proctorio.
Proctorio artificial intelligence (AI) will be used by ABR staff to determine the priority by which remote exam sessions are reviewed. Nothing identified by AI will be used directly to give feedback to candidates. AI is a tool to help prioritize ABR staff work.
Only ABR staff will review remote exam sessions. If questionable behavior is detected by staff, it will be reviewed by a team of ABR leadership (including certified diplomates) before feedback is given to candidates.
Feedback to Candidates
Our goal is to prevent candidates from unknowingly violating exam security rules. If there is a major exam security violation, feedback about problematic behavior will be communicated to the candidate as soon as possible. If there is an obviously harmless infraction (like kids running into the room during an exam), don’t expect feedback from the ABR. We will not contact a candidate unless it is absolutely necessary.
Remote exam guide available
A few questions have come up about our transition to remote exams next year. In response, we have compiled the following Remote Exam Basics guide that will help prepare candidates for the new oral and computer-based exam experiences.
Included in the guide is important information such as where exams may be taken, computer requirements, and what is and is not allowed on exam day.
To complement the Remote Exam Basics guide, comprehensive exam guides are being compiled for each remote exam administration. These guides will be emailed to candidates well in advance of their exam days.
If you have questions after reading the basics guide, please contact an ABR certification manager at email@example.com or (520) 790-2900.
Dr. Wagner Presentations on Remote Exam Development Progress
Earlier this month, ABR Executive Director Brent Wagner, MD, MBA, hosted remote exam updates with multiple stakeholder groups representing the four specialties we certify. The following are recordings from those collaborative sessions. Anyone with questions should contact a certification manager at firstname.lastname@example.org or (520) 790-2900.
2021 Remote Exam Calendar Updated
The ABR Board of Governors recently announced remote exam dates for the second half of 2021. For registration dates, please visit the specialty calendar pages on our website.
Secure Exams Benefit the Candidates Who Take Them
By ABR Executive Director Brent Wagner, MD, MBA
Over the past few months, the ABR has initiated a complex set of projects to provide a series of modified exams for 2021. As we have announced, we will not require travel to our exam centers or commercial test centers beginning next year. Although the subject matter and scoring model of our exams will be essentially unchanged, remote exam delivery seeks to remove the uncertainty and anxiety surrounding the potential impact of the pandemic on this portion of the certification process.
The path to board certification includes satisfactory completion in two major areas: a residency and a series of standardized exams. The combination of these two elements and the significant accomplishment they represent are an important part of the social contract with our patients, their families, and the communities that medical professionals are privileged to serve.
Several compromises are necessary when designing high stakes exams; this is true not only with respect to ABR certification but also to the MCAT, LSAT, NMLE, bar exam, etc. Practicality (e.g., the length of the exam), fairness, and the candidate experience historically have been extremely important considerations for the ABR; the currently evolving remote exams are allowing us to look for novel opportunities to improve in these areas.
However, beyond these parameters, two specific factors require a careful balance: security and validity. Over the past few months, we have been reminded in meetings with stakeholder groups of faculty and trainees that remote exam platforms are inherently less secure than the traditional alternative of directly proctored exams that one might encounter in a classroom or test center. Specifically, the testing experience in higher education, from which we are borrowing our fundamental model (remote proctoring in one’s home or office using a webcam), has been repeatedly challenged by the creative application of “alternative methods of success” (e.g., searching for answers via the Internet) from isolated individuals lacking academic integrity.
To be clear, the ABR is confident that nearly all trainees in the radiology disciplines are well-educated, are committed to excellence, and epitomize the highest standards in academic integrity. The paradox is that we have an obligation to the candidates to develop a reasonably secure process – even as we admit that it is unnecessary for almost all residents. This process is still being defined but will likely require recorded webcam surveillance during the exam and a “room scan” to confirm the absence of prohibited resources (e.g., a cell phone, books, or notes).
If we fail to maintain security, we will have sacrificed not only fairness but also, just as importantly, the validity of the exam. The value of certification for the patient and, by extension, the candidate, depends on the credibility of the process as an indicator of a high level of accomplishment and ability.
Exam Delivery is Changing, but Content will Remain Consistent
With the best interests of our candidates in mind, we are moving to remote computer-based and oral exams in 2021. Despite the changes, one thing will stay the same: exams will continue to be developed by radiology and medical physics professionals who will ensure consistency of content.
Hundreds of volunteers help develop content for ABR exams. All give their time to ensure that our exams are fair and relevant to the candidates who take them. They have continued working remotely with our staff to make sure we have sufficient content even as we offer an expanded exam schedule next year.
Exam format and delivery logistics must change to accommodate remote exam locations and inevitable technical challenges. New break schedules and nursing mother accommodations are being discussed in addition to many other exam logistics. None of these developments will impact the amount or quality of content included in our exams.
Anyone with questions may contact our certification services team at email@example.com or (520) 790-2900.
Dr. Wagner Presentations on Remote Exam Development Progress
ABR Executive Director Brent Wagner, MD, MBA, recently hosted remote exam updates with multiple stakeholder groups representing the four specialties we certify. The following are recordings from those collaborative sessions. Anyone with questions should contact our certification services team at firstname.lastname@example.org or (520) 790-2900.
Update on Preparations for Remote Computer-Based Exams
The Value of Working with Stakeholders on Remote Exams
ABR Executive Director Talks About Planning Exams
Partner Chosen to Provide Insight on Establishing Remote Exams
In an effort to make remote oral and computer-based exams optimal experiences for candidates, we are partnering with Gartner, one of the world’s leading research and advisory companies.
Gartner is providing business insights, advice, and tools to help us meet the challenges of delivering remote exams. The areas in which we are consulting with Gartner include vendor selection, cyber security, purchasing, compliance, and development strategies.
We are confident that Gartner’s vast experience, which includes serving more than 14,000 businesses and nonprofits across 100 countries, will assist us in developing remote exams that exceed the expectations of our candidates.
Core Exam Will Be Held Over Three Days in 2021
One of the changes for the DR and IR/DR Core Exam remote administrations in 2021 will be extending the delivery from two days to three days.
The revision is primarily being made to account for factors beyond our control — Internet connectivity, power issues, etc. — which could result in exam interruptions. Additionally, a remote delivery will require modification to when breaks are offered and how exam content is accessed following a break. We don’t have all the details worked out on how the three-day exam will be structured; we will share that information as soon as it’s available.
The Core Exam is scheduled for February 1-3 and 8-10; June 2-4 and 7-9; and on November dates that will be determined and announced soon.
Q&A with ABR IT Director Scott Segal About Remote Exam Development
Work is going on behind the scenes to set the stage for us to begin delivering remote exams next February. Scott Segal, our IT director, is the point person for many of those efforts. He took a few moments this week to answer questions about the process.
What is the biggest challenge in switching to remote exams?
I think the hot-button issue is going to be delivering an exam that meets the high standards of the ABR for security, content, and, most importantly, examinee experience.
What are you doing to ensure that examinees feel confident using the new system?
There are some very specific challenges to giving a remote exam, so we are taking extra time and effort to mitigate risks as much as possible. For both computer and oral exams, we will give examinees advance access to the system to verify compatibility and get comfortable with the new interface.
Exam security is always a concern. How will that be handled?
Security is being addressed at multiple levels. Both ABR content as well as any information about examinees that they provide during the exam must be protected. We are working to increase the security of the ABR network and applications. This is a constant struggle and we supplement our internal staff with outside security experts.
How will the new exam delivery platforms be developed?
For computer-based exams, we are talking to a wide range of vendors and conducting extensive research on existing platforms. This should allow the Board of Governors to make the best decision possible to optimize the examinee experience. For oral exams, we are building new software that will allow the ABR to deliver an exam that replicates as closely as possible the experience examiners and examinees had when the exam was in person. We also seek to minimize, as much as we can, the stress for the candidates as they go through this process.
Stakeholder Calls Inform Exam Planning
Last week, the ABR convened meetings with initial certification stakeholder groups to solicit feedback about the move to remote exams. During these meetings, we provided a high-level overview of the remote approach for both our computer-based and oral exams.
ABR computer-based initial certification exams will take place in a remote location of the candidate’s choosing, provided that place meets a few basic requirements. Remote computer-based exams are not likely to be given at commercial testing centers (e.g., Pearson VUE) or ABR centers. The exams will use an ABR-developed exam interface similar to what has previously been used for computer-based exams. In addition, we will likely use a third-party vendor to handle exam-day security and remote monitoring. We will provide additional details about the requirements when we know more.
The oral exam will use an ABR-developed platform that will combine remote proctoring with video conferencing. As with the computer-based exams, candidates will have the freedom to select a location, but it must meet a few basic requirements. The details about exam-day location and other logistics are still in development and will be communicated when we have more information.
Many thanks to the leadership and staff of all the organizations that so thoughtfully engaged with us (listed below). We truly appreciate the collaboration as we begin this important endeavor and look forward to continuing the dialogue as we move forward.
As mentioned in previous communications, we’ll be sharing more information on exam dates for 2021 by August 1.
Association of Program Directors in Radiology (APDR)
Association of Program Directors in Interventional Radiology (APDIR)
Society of Chairs of Academic Radiology Departments (SCARD)
American Alliance of Academic Chief Residents in Radiology (A3CR2)
Radiological Society of North America (RSNA)
American College of Radiology, Resident and Fellow Section (ACR RFS)
Accreditation Council for Graduate Medical Education, Review Committee for Radiology (ACGME-RC)
IC Advisory Committee for Diagnostic Radiology
Association of Directors of Radiation Oncology Programs (ADROP)
Society of Chairs of Academic Radiation Oncology Programs (SCAROP)
Association of Residents in Radiation Oncology (ARRO)
Society of Directors of Academic Medical Physics Programs (SDAMPP)
IC Advisory Committee for Medical Physics
Society for Interventional Radiology (SIR)
Society for Interventional Radiology, Early Career Section (SIR ECS)
Society for Interventional Radiology, Residents, Fellows and Students (SIR RFS)