To better ensure the health and safety of our candidates, the ABR has committed to a remote exam platform starting in 2021. The decision was made after weeks of consultation with key stakeholders, including candidates, programs, associations, and societies. We are continuing those discussions as we move forward in our exam development process.
Transitioning from in-person to remote exams will take substantial effort and we want to keep our stakeholders informed. Please bookmark this page or follow us on Twitter to check for updates. We appreciate everyone’s interest as we work to build a new way of delivering exams.
Candidate: Completing practice exam is a must
Omar Chohan, DO, is a vascular and interventional radiologist in Buffalo who recently took the ABR IR/DR Certifying Exam. He found the ABR practice exam to be helpful for many reasons, which he writes about here. Dr. Chohan works at the University at Buffalo and Great Lakes Medical Imaging. You can follow him on Twitter: @drochohan.
By Omar Chohan, DO
Anyone taking an ABR computer-based remote exam should first complete the practice exam and technical self-check using Proctorio. The major reason is to test whether your internet connection and computer will provide a smooth testing experience. Problems may occur and it’s better for you to discover this in a stress-free environment and make adjustments as needed.
The practice exam also gives you a feel for the actual test day. It’s timed, lasts approximately 1.5 hours, and simulates exam day with the same interface, buttons, and feel. You may take it as many times as you need.
Knowing your Wi-Fi is fast enough to run the exam is critical. Having slow or inconsistent internet will make exam day much more stressful.
The practice exam will also help you figure out if your screen is adequate for viewing and interpreting images. For example, I took the practice exam on my laptop and again on my desktop. I found it was easier for me to use my desktop because the screen was significantly larger.
Make sure your computer and Chrome web browser are updated. This exam comes with its own stress and the practice exam definitely helped to calm my nerves in the sense that I knew my computer could run the exam without hiccups.
Using a side-view camera during remote exams
This video includes important information about room/workspace scans and use of side-view cameras, which will be required for computer-based exams starting April 15.
Side-view cameras necessary for future computer-based exams
Over the past two months, more than 1,500 candidates have taken remote computer-based exams offered by the ABR. Our goal was to create a platform that would not be subject to the unpredictable impact of the pandemic and would represent an improved experience for candidates. Maintaining the security of the process was important to ensure this element of the path to board certification would continue to serve as a valid indicator of knowledge and competence.
We have learned a lot in the past two months. Generally, exam candidates were respectful of the process and provided useful feedback. They approached a task that was both unfamiliar and stressful with exceptional patience and resourcefulness.
We learned that internet connectivity was imperfect but not so disruptive that the exam could not be completed by nearly all who logged in. The exam software platform itself functioned well. The vast majority of the content and images were adequate for the intended purpose.
One stressor frequently cited was the webcam recording. We borrowed online testing tools from higher education that have been used for many years, even more so over the past year. It became clear that these tools, while very reasonable for colleges and universities, create stress in a high-stakes environment (because of, for example, standard methods for tracking eye movement).
For this reason, we will be requiring a side–view (instead of the standard front-facing webcam view) for future computer-based exams beginning April 15, 2021. The new side-view webcam placement details are outlined here and a video can be found on our YouTube page.
In addition, to help offset the potential cost of this for our candidates, we will credit candidates $40 toward future ABR fees. These credits will be applied in early 2022.
We appreciate your patience as we attempt to optimize the exam experience.
If you have questions, contact a certification manager at email@example.com or (520) 790-2900.
Sara Hunter, MD, talks about taking the remote Core Exam
The chief diagnostic radiology resident at the Cleveland Clinic gives feedback and tips on the remote exam platform.
Q&A with ABR Oral Examiner Jessica Clements, MS
Jessica Clements, MS, is the chief medical physicist, regional radiation safety officer, and medical physics residency program director for the Southern California Region of Kaiser Permanente Medical Group. She’s also an oral examiner for the ABR, with experience that goes back to when those exams were held in Louisville, Kentucky.
Ms. Clements and other examiners are training and preparing for remote oral pilot exams in March. These exams will mark the first time the ABR has used a remote platform for orals.
Ms. Clements recently answered a few questions from an examiner’s perspective about changing to remote exams.
When the ABR was holding oral exams in Louisville, did you picture the process one day going remote?
I never imagined the exams transitioning to a remote format. So many things have changed around the world in the past year, making some things previously unimaginable completely common. I am very impressed that the ABR was able to adapt to those rapid changes so quickly and provide a high-quality, secure exam to candidates!
How much has your experience as an examiner helped you prepare for remote exams?
The solutions developed by the ABR for a remote exam should allow an easy transition for seasoned examiners as well as new examiners, especially after a little practice. Following a recent test of the system with other examiners and ABR staff, I felt very comfortable with the system.
Aside from the platform, do you think much else will change for candidates?
I am optimistic that the remote platform will be a good experience for candidates. When I took the oral exams in diagnostic and nuclear medical physics, I always had to prepare contingency plans such that if bad weather canceled flights, I could drive and still make it to Louisville in time. Candidates won’t have that layer of complexity adding stress to the exam process. They can identify a secure and comfortable location where they can tune in for the remote exam. Additionally, the examiner and examinee will both have digital tools to help point to specific items and communicate visually from their own screens.
How confident in the system should candidates feel as they head into remote oral exams?
I think candidates should be confident that they will receive a high-quality and secure exam. The platform has been tested extensively. One-on-one technical checks have occurred with examiners and will be conducted with examinees. The schedule has been adapted to prepare for unforeseen technical issues and an emergency technical support phone number has been established for use during exams. We are all familiar with continuous quality improvement in our clinical practices, and we should expect that the remote exam will only get better with age.
Looking back on the first round of the Core Exam
For most of the more than 1,300 candidates who took the Core Exam over the past two weeks, the new ABR remote platform was a success. Our volunteers, stakeholders, and staff have spent thousands of hours developing the system, and we will continue making improvements as we gain experience and receive user feedback.
While we’re pleased that most issues candidates had during the exam were solvable, we apologize to anyone who had difficulties. We had significant support staff available on exam days to help correct problems, most of which were related to connectivity, and will follow suit in future exams.
We will be using a similar platform for oral exams, which begin with pilots in March. The success of the Certifying Pilot Exam in January and the Core Exam this month assured us that our system for oral exams also will be reliable. Just as we did during the Certifying and Core exams, we will have support available for anyone who experiences issues.
We feel confident we have a reliable remote platform that will enable candidates to take this important step in their careers. We thank everyone for their patience and understanding as we begin a new era of exam delivery.
Candidate discusses his Core Exam experience
Ryan Adams, MD, was in the first group of candidates who took the initial remote Core Exam last week. He is chief diagnostic radiology resident at Vanderbilt University Medical Center and a graduate of Vanderbilt University Medical School. He will be completing an independent residency in interventional radiology at Massachusetts General Hospital, and his academic interests include medical technology, medical education, and quality improvement.
After the exam, he answered a few questions about how things went.
How much experience did you have with remote exams heading into the Core Exam?
In my medical school, all our exams were online using testing software installed on our personal computers, so I’ve taken dozens of remote exams.
What were you expecting going into the exam, since it was the first Core the ABR has delivered remotely?
To be honest, I was expecting a very cumbersome check-in process where I would have to video chat with a proctor and show them my ID/exam room. I was fully prepared to have the exam software itself to be a relic from the ’90s – block design, clunky, and slow. Most importantly, I was concerned we would not have adequate image manipulation tools, specifically window/level, brightness, color inversion, zoom, etc.
How did the experience compare with what you expected?
My first thought on the practice exam, especially as someone who has written software and developed websites in the past, was, “Wow, this is actually effective testing software!” As I was taking the exam, I never felt the software was holding me back from performing my best. The check-in process was smooth and automated, no glitches. It was quick to move from question to question – no long load times. It had all the image manipulation features I needed: smooth functioning for the ultrasound cine clips, easy scrolling and manipulating a stack of CT images. Overall, I was very impressed with the testing software chosen to administer this exam.
What was the best part about the exam?
The best part was submitting the last question!
What should be improved upon?
While I overall thought the exam software and administration were very good, especially for the first virtual admission of this exam, I did feel like improvements could be made. Firstly, I think a strikeout feature to mark answers you know are incorrect would be a vital revision prior to administering the exam again. Secondly, I got multiple popups with feedback during the exam – e.g., keeping face in the center of the camera, improvements to the room scan, etc. I think those expectations could be better spelled out before the exam and honestly condensed to one brief, personalized message at the end of a block would be adequate. Otherwise, I think there is very little to improve!
What advice would you give people who will be taking computer-based exams later in the year?
My advice would be to review the room scanning requirements, make sure you have your ID with you, bring some water and a snack, and, finally, relax!
Remote Exam Effects on Candidate Fees
With the introduction of remote exams, ABR Executive Director Brent Wagner, MD, MBA, discusses the impact on candidate fees.
Typical remote exam questions answered
With the DR and IR/DR Core Exam administrations coming in the next two weeks, we have been answering candidate questions on our Twitter page. Here are a couple of questions submitted by two of the more than 1,400 candidates who will be taking the exam next month.
In light of the recent massive geographic ISP-related service outage, do you have formal guidance on how examinees are to proceed if some calamitous connectivity issues arise during the February administrations of the Core Exam?
We have planned contingencies for accommodating such situations. Those experiencing a prolonged internet outage during their exam should contact our Exam Day Help Desk. The Help Desk information will be emailed to candidates approximately 48 hours before the exam.
Should we lock our computers when we take a break? What if it auto logs off when we take a long break?
If you lock your computer, it will terminate the connection with the Proctorio® remote proctoring software and automatically log you out of the exam. When taking a break, select “start break.” You will not be logged off.
If for some reason you are logged off, you can re-enter the exam using the exam access link, which will be emailed to you prior to the exam, or use the “Enter Exam Lobby” button on myABR. You will go through Proctorio again to re-enter the exam, but once in, you will be able to pick up where you left off. Our exam software collects your responses as they are given so none of your responses will be lost.
Anyone with questions before the exam should contact our certification services team at (520) 790-2900 or firstname.lastname@example.org.
Examinees find many positives in first remote exam administration
Our Diagnostic Radiology Certifying Pilot Exam earlier this month provided an opportunity for us to gain a real-life look at a computer-based remote administration with a limited group of candidates. It was the first time we had administered a remote computer-based exam and the experience was largely a success.
The pilot was taken by 161 candidates, all of whom were given the chance to respond to a post-exam survey. We appreciate the 127 who shared their opinions about the experience; we will use their feedback to improve our platforms and processes.
Some interesting details emerged from the survey:
- More than 95% of respondents found the practice exam useful in helping them prepare.
- More than 80% took the exam at home rather than in a hospital or private practice office.
- More than 95% thought* that installation of the Proctorio®extension was easy to complete.
- More than 95% thought* that check-in through the Proctorio®platform was easy.
- More than 95% thought* that the exam platform (imaging tools, load times, answer entry, etc.) worked well.
- More than 90% thought* that “the image quality was adequate for testing purposes.”
- Survey respondents rated the overall experience a 4.5 out of 5 stars.
* “Agreed” or “Strongly Agreed”
We appreciate everyone who provided feedback. It’s instrumental in helping us ensure the continued delivery of a secure and valid exam, while keeping the exam day experience as pleasant as possible for the new remote administrations.
IT resource information available
During a pilot administration earlier this month, we gained additional insight into how security systems employed by many training programs/health systems may impact candidates taking ABR remote computer-based exams.
As outlined in the Remote Exam Basics Guide, candidates may take their exam in a location of their choosing. We acknowledge that some may pick a room in their training program or health system, raising potential network issues. As a result, we created a webpage with remote computer-based exam technical specifications for institutions.
This new page details the systems and processes used to deliver our remote computer-based exams, including specific domains, component requirement, web services, and website security parameters. Interested programs are encouraged to direct their IT departments to this page for details that may help them mitigate technical issues for candidates during their exam.
What we learned from our remote pilot exam administration
ABR Executive Director Brent Wagner, MD, MBA, discusses what worked and what will be improved upon after the DR Remote Certifying Pilot Exam on January 9, 2021.
Remote exam preparation update
ABR Executive Director Brent Wagner, MD, MBA, provides an update as the organization prepares to start administering remote exams.
More information on exam proctoring
The ABR is working with Proctorio to provide remote proctoring services for oral and computer-based exams in 2021. Here are further details about how the system will be incorporated.
The ABR respects candidates and knows they are overwhelmingly honest and professional. We all want a secure and meaningful exam. ABR feedback will be focused on preventing potential issues, not implementing punitive measures.
The ABR is using Proctorio for remote proctoring. Only ABR staff will be able to review ID checks, room scans, webcam video, and microphone audio of exams. Candidate personal data (i.e., email address and phone number) is never shared with Proctorio.
Proctorio artificial intelligence (AI) will be used by ABR staff to determine the priority by which remote exam sessions are reviewed. Nothing identified by AI will be used directly to give feedback to candidates. AI is a tool to help prioritize ABR staff work.
Only ABR staff will review remote exam sessions. If questionable behavior is detected by staff, it will be reviewed by a team of ABR leadership (including certified diplomates) before feedback is given to candidates.
Feedback to Candidates
Our goal is to prevent candidates from unknowingly violating exam security rules. If there is a major exam security violation, feedback about problematic behavior will be communicated to the candidate as soon as possible. If there is an obviously harmless infraction (like kids running into the room during an exam), don’t expect feedback from the ABR. We will not contact a candidate unless it is absolutely necessary.
Remote exam guide available
A few questions have come up about our transition to remote exams next year. In response, we have compiled the following Remote Exam Basics guide that will help prepare candidates for the new oral and computer-based exam experiences.
Included in the guide is important information such as where exams may be taken, computer requirements, and what is and is not allowed on exam day.
To complement the Remote Exam Basics guide, comprehensive exam guides are being compiled for each remote exam administration. These guides will be emailed to candidates well in advance of their exam days.
If you have questions after reading the basics guide, please contact an ABR certification manager at email@example.com or (520) 790-2900.
Dr. Wagner Presentations on Remote Exam Development Progress
Earlier this month, ABR Executive Director Brent Wagner, MD, MBA, hosted remote exam updates with multiple stakeholder groups representing the four specialties we certify. The following are recordings from those collaborative sessions. Anyone with questions should contact a certification manager at firstname.lastname@example.org or (520) 790-2900.
2021 Remote Exam Calendar Updated
The ABR Board of Governors recently announced remote exam dates for the second half of 2021. For registration dates, please visit the specialty calendar pages on our website.
Secure Exams Benefit the Candidates Who Take Them
By ABR Executive Director Brent Wagner, MD, MBA
Over the past few months, the ABR has initiated a complex set of projects to provide a series of modified exams for 2021. As we have announced, we will not require travel to our exam centers or commercial test centers beginning next year. Although the subject matter and scoring model of our exams will be essentially unchanged, remote exam delivery seeks to remove the uncertainty and anxiety surrounding the potential impact of the pandemic on this portion of the certification process.
The path to board certification includes satisfactory completion in two major areas: a residency and a series of standardized exams. The combination of these two elements and the significant accomplishment they represent are an important part of the social contract with our patients, their families, and the communities that medical professionals are privileged to serve.
Several compromises are necessary when designing high stakes exams; this is true not only with respect to ABR certification but also to the MCAT, LSAT, NMLE, bar exam, etc. Practicality (e.g., the length of the exam), fairness, and the candidate experience historically have been extremely important considerations for the ABR; the currently evolving remote exams are allowing us to look for novel opportunities to improve in these areas.
However, beyond these parameters, two specific factors require a careful balance: security and validity. Over the past few months, we have been reminded in meetings with stakeholder groups of faculty and trainees that remote exam platforms are inherently less secure than the traditional alternative of directly proctored exams that one might encounter in a classroom or test center. Specifically, the testing experience in higher education, from which we are borrowing our fundamental model (remote proctoring in one’s home or office using a webcam), has been repeatedly challenged by the creative application of “alternative methods of success” (e.g., searching for answers via the Internet) from isolated individuals lacking academic integrity.
To be clear, the ABR is confident that nearly all trainees in the radiology disciplines are well-educated, are committed to excellence, and epitomize the highest standards in academic integrity. The paradox is that we have an obligation to the candidates to develop a reasonably secure process – even as we admit that it is unnecessary for almost all residents. This process is still being defined but will likely require recorded webcam surveillance during the exam and a “room scan” to confirm the absence of prohibited resources (e.g., a cell phone, books, or notes).
If we fail to maintain security, we will have sacrificed not only fairness but also, just as importantly, the validity of the exam. The value of certification for the patient and, by extension, the candidate, depends on the credibility of the process as an indicator of a high level of accomplishment and ability.
Exam Delivery is Changing, but Content will Remain Consistent
With the best interests of our candidates in mind, we are moving to remote computer-based and oral exams in 2021. Despite the changes, one thing will stay the same: exams will continue to be developed by radiology and medical physics professionals who will ensure consistency of content.
Hundreds of volunteers help develop content for ABR exams. All give their time to ensure that our exams are fair and relevant to the candidates who take them. They have continued working remotely with our staff to make sure we have sufficient content even as we offer an expanded exam schedule next year.
Exam format and delivery logistics must change to accommodate remote exam locations and inevitable technical challenges. New break schedules and nursing mother accommodations are being discussed in addition to many other exam logistics. None of these developments will impact the amount or quality of content included in our exams.
Anyone with questions may contact our certification services team at email@example.com or (520) 790-2900.
Dr. Wagner Presentations on Remote Exam Development Progress
ABR Executive Director Brent Wagner, MD, MBA, recently hosted remote exam updates with multiple stakeholder groups representing the four specialties we certify. The following are recordings from those collaborative sessions. Anyone with questions should contact our certification services team at firstname.lastname@example.org or (520) 790-2900.
Update on Preparations for Remote Computer-Based Exams
The Value of Working with Stakeholders on Remote Exams
ABR Executive Director Talks About Planning Exams
Partner Chosen to Provide Insight on Establishing Remote Exams
In an effort to make remote oral and computer-based exams optimal experiences for candidates, we are partnering with Gartner, one of the world’s leading research and advisory companies.
Gartner is providing business insights, advice, and tools to help us meet the challenges of delivering remote exams. The areas in which we are consulting with Gartner include vendor selection, cyber security, purchasing, compliance, and development strategies.
We are confident that Gartner’s vast experience, which includes serving more than 14,000 businesses and nonprofits across 100 countries, will assist us in developing remote exams that exceed the expectations of our candidates.
Core Exam Will Be Held Over Three Days in 2021
One of the changes for the DR and IR/DR Core Exam remote administrations in 2021 will be extending the delivery from two days to three days.
The revision is primarily being made to account for factors beyond our control — Internet connectivity, power issues, etc. — which could result in exam interruptions. Additionally, a remote delivery will require modification to when breaks are offered and how exam content is accessed following a break. We don’t have all the details worked out on how the three-day exam will be structured; we will share that information as soon as it’s available.
The Core Exam is scheduled for February 1-3 and 8-10; June 2-4 and 7-9; and on November dates that will be determined and announced soon.
Q&A with ABR IT Director Scott Segal About Remote Exam Development
Work is going on behind the scenes to set the stage for us to begin delivering remote exams next February. Scott Segal, our IT director, is the point person for many of those efforts. He took a few moments this week to answer questions about the process.
What is the biggest challenge in switching to remote exams?
I think the hot-button issue is going to be delivering an exam that meets the high standards of the ABR for security, content, and, most importantly, examinee experience.
What are you doing to ensure that examinees feel confident using the new system?
There are some very specific challenges to giving a remote exam, so we are taking extra time and effort to mitigate risks as much as possible. For both computer and oral exams, we will give examinees advance access to the system to verify compatibility and get comfortable with the new interface.
Exam security is always a concern. How will that be handled?
Security is being addressed at multiple levels. Both ABR content as well as any information about examinees that they provide during the exam must be protected. We are working to increase the security of the ABR network and applications. This is a constant struggle and we supplement our internal staff with outside security experts.
How will the new exam delivery platforms be developed?
For computer-based exams, we are talking to a wide range of vendors and conducting extensive research on existing platforms. This should allow the Board of Governors to make the best decision possible to optimize the examinee experience. For oral exams, we are building new software that will allow the ABR to deliver an exam that replicates as closely as possible the experience examiners and examinees had when the exam was in person. We also seek to minimize, as much as we can, the stress for the candidates as they go through this process.
Stakeholder Calls Inform Exam Planning
Last week, the ABR convened meetings with initial certification stakeholder groups to solicit feedback about the move to remote exams. During these meetings, we provided a high-level overview of the remote approach for both our computer-based and oral exams.
ABR computer-based initial certification exams will take place in a remote location of the candidate’s choosing, provided that place meets a few basic requirements. Remote computer-based exams are not likely to be given at commercial testing centers (e.g., Pearson VUE) or ABR centers. The exams will use an ABR-developed exam interface similar to what has previously been used for computer-based exams. In addition, we will likely use a third-party vendor to handle exam-day security and remote monitoring. We will provide additional details about the requirements when we know more.
The oral exam will use an ABR-developed platform that will combine remote proctoring with video conferencing. As with the computer-based exams, candidates will have the freedom to select a location, but it must meet a few basic requirements. The details about exam-day location and other logistics are still in development and will be communicated when we have more information.
Many thanks to the leadership and staff of all the organizations that so thoughtfully engaged with us (listed below). We truly appreciate the collaboration as we begin this important endeavor and look forward to continuing the dialogue as we move forward.
As mentioned in previous communications, we’ll be sharing more information on exam dates for 2021 by August 1.
Association of Program Directors in Radiology (APDR)
Association of Program Directors in Interventional Radiology (APDIR)
Society of Chairs of Academic Radiology Departments (SCARD)
American Alliance of Academic Chief Residents in Radiology (A3CR2)
Radiological Society of North America (RSNA)
American College of Radiology, Resident and Fellow Section (ACR RFS)
Accreditation Council for Graduate Medical Education, Review Committee for Radiology (ACGME-RC)
IC Advisory Committee for Diagnostic Radiology
Association of Directors of Radiation Oncology Programs (ADROP)
Society of Chairs of Academic Radiation Oncology Programs (SCAROP)
Association of Residents in Radiation Oncology (ARRO)
Society of Directors of Academic Medical Physics Programs (SDAMPP)
IC Advisory Committee for Medical Physics
Society for Interventional Radiology (SIR)
Society for Interventional Radiology, Early Career Section (SIR ECS)
Society for Interventional Radiology, Residents, Fellows and Students (SIR RFS)