To better ensure the health and safety of our candidates, the ABR has committed to a remote exam platform starting in 2021. The decision was made after weeks of consultation with key stakeholders, including candidates, programs, associations, and societies. We are continuing those discussions as we move forward in our exam development process.
Transitioning from in-person to remote exams will take substantial effort and we want to keep our stakeholders informed. Please bookmark this page or follow us on Twitter to check for updates. We appreciate everyone’s interest as we work to build a new way of delivering exams.
Exam schedule for 2023 updated
The updated computer-based and oral exam schedule for next year is available. MP Part 3 Oral Certifying has been changed from the week of April 1 to the week of April 16, and a second RISE administration has been added on December 5. All exams will again be held remotely.
Lessons learned during remote exam delivery
When the ABR announced that all 2021 exams would be given remotely, the ABR IT department had a single focus: create a platform that would allow candidates and examiners to concentrate on taking and delivering the exam, not battling new software. But it was only after that platform was complete that we learned one of the most important lessons.
Everything changed with remotely proctored exams. Candidates supplied their own computers (with their own unique configurations of operating system, memory, software, etc.). Internet access was required, and candidates had their own providers. In addition, the exams required software from a third party, Proctorio®. To meet the challenges of administering a remote exam, the ABR moved hosting of the exam software from a private data center to a public cloud provider.
All combined, this made administering a successful exam much more difficult. When a problem occured, there were many more places to investigate. During one exam, a public cloud provider experienced a systemic problem that caused more than a dozen users to temporarily lose the ability to connect to the ABR exam software. During another exam, one of the nation’s largest internet providers had a major system failure during the first day of the exam, causing many candidates to lose connectivity.
We learned that support is both crucial and much more challenging when administering remote exams. In the days of in-person exams, two IT members were sufficient. For remote exams, the ABR needed four IT support staff at headquarters and four providing support remotely, and anywhere from 10 to 15 staff on the Exam Day Help Desk. Some IT staff were dedicated to helping other staff and investigating reports of potential system problems. Most IT staff provided direct support to candidates who had technical problems that could not be resolved by the Help Desk staff, the first point of contact for candidates when they call for help. This was especially true in the mornings when candidates first logged in to the exam system.
Providing this support was challenging, but it was also rewarding. Often, just a few minutes of technical help enabled a candidate to return to their exam. We are looking forward to continuing to improve the ABR remote exam platform in 2022, but we will also be ready to support staff and candidates to make sure everyone has a successful exam experience.
Plenty of milestones along our path to remote exams
Our road to remote exams was necessary but not easy. The change took countless hours of work from staff and volunteers plus assistance from candidates to become reality. Fortunately, the new platforms have been well-received and have enabled us to administer exams to thousands of candidates in 2021. Here’s a look at milestones along the way as we moved from in-person computer-based and oral exams to a remote platform.
New format, same relief over passing Core Exam
By Hala Mazin, MD
I was a first-year radiology resident in the reading room when I heard two of my fourth-year co-residents screaming for joy after passing their ABR Core Exam. I remember thinking, “Gosh, radiologists are so weird. I mean, so you passed a board exam? What’s the big deal?”
For junior residents reading this, I can assure you after four years of residency and studying for the boards myself, it is not just a big deal. It is the biggest deal. No clinical correlation necessary.
I knew studying for my boards was going to be a stressful and hard process. However, one of the hidden positives about being in the middle of a pandemic during all of it was taking the boards remotely. Yes, I was in a sweatshirt just two feet away from my bed taking the biggest exam of my life. The only challenge I had, besides the exam of course, was actually making the bed in the morning.
My setup consisted of a 24-inch desktop screen, an 8-inch white board, two chairs (one to sit on and one to rest my feet), a tripod with a web camera off to my side, lots of beige carbs and water, and, most important, a huge piece of paper with the ABR help desk number taped to the wall next to me.
Aside from a little anxiety, my first remote exam experience was very positive, and I predict that the ease of taking remote core exams will only continue to improve. The format allowed me to be more comfortable and relaxed on the day of the exam without the stress inherent to testing centers (e.g. driving, parking, public restrooms). Considering how nervous and anxious I felt in my own home, I cannot imagine having to put myself together and get on a plane to take my boards. I did not encounter technical problems, but even colleagues who had connection issues (for example) expressed that they were able to resolve their problems quickly without losing test time.
To those of you taking the exam this year, here are my five tips to best prepare for the remote exam.
- Testing Area: Make sure you are in a quiet place without any possibility of others distracting you. I took the test at home where I knew I would be alone for the day and had a piece of paper outside my apartment door that read: “Testing in progress. Please do not knock.”
- Monitors: You can take the test on any screen larger than 12 inches. However, if you are like me and don’t like to zoom in and out of images, consider using a larger monitor.
- Lighting: Make sure you’re not in an area that is too sunny; the glare may affect your interpretation of the images. There’s a reason radiologists read in the dark.
- Check, double check, and triple check your internet connection. Have an ethernet cord handy just in case your Wi-Fi goes out. Also, have a backup location at which to take the test, such as your institution or a friend’s house, in case all else fails.
- Don’t forget to breathe, and smile for the camera.
Hala Mazin, MD, is a radiology resident at Advocate Illinois Masonic in Chicago and president of the Chicago Radiological Society Resident/Fellow Section. Her interests in radiology include breast imaging, diversity and inclusion, and mentorship. When she’s not in the reading room, you can find her eating her way through Chicago, then quickly regretting the decision. You can chat with Dr. Mazin on Twitter.
Oral exam dates complete 2022 schedule
The following chart details next year’s exam schedule, which has been updated to include oral exam dates. Please note that secondary oral exam administrations have been added for each specialty. While these additional dates are intended for candidates who must miss the first exam, or conditioned or failed a prior attempt, anyone who is eligible to sit for an oral exam has the ability to choose the corresponding secondary date. As we announced in May, all 2022 exams will be held remotely.
Candidates should put tech checks on their to-do lists
To have the best possible exam-day experience, candidates should complete their technical checks before exam day in the location and with the hardware they plan to use.
The technical check helps candidates determine that their computer setup is reliable enough to support one of our exams. Candidates assume full responsibility for ensuring their setup meets the requirements, including integrity of equipment, internet connectivity, system performance, and other issues related but not limited to the above that are outside the control of the ABR or our exam security vendor, Proctorio®. ABR staff are not able to provide technical support regarding individual setups for remote exams.
Here are more details on technical checks for our exams.
For computer-based exams, the technical check is completed by candidates at a time that is convenient for them. The technical check is initiated when candidates launch the practice exam through myABR. The technical check is used to ensure that the candidate’s microphone, webcam, internet connection, and computer hardware and software are optimized for the exam. Candidates may need to download an updated version of Google Chrome or the Proctorio® extension for Google Chrome to complete the technical self-check.
Oral (for Medical Physics, Radiation Oncology, and Interventional Radiology)
A navigator-led technical check must be performed three to four weeks before an exam. Candidates will receive an email notice to schedule a one-on-one with an ABR team member. The technical check is scheduled for 30 minutes and will provide an overview of what to expect on exam day. A whiteboard (maximum size 8.5” x 11”) is required for medical physics (MP) candidates; it is optional for radiation oncology and interventional radiology candidates. MP candidates who don’t have a whiteboard should wait to schedule their technical checks until they can obtain one. Candidates will be required to log in to a WebEx® call and complete the Proctorio® check-in process during their navigator-led technical check. These activities simulate candidate actions on exam day.
Approximately 98% of respondents to the survey after our May exams said the technical check was useful. Candidate remarks after the May exams included:
- “I think this was a very necessary component of getting ready for the exam.”
- “I was pleased with the tech check before exam day, which was well run.”
Anyone with questions is encouraged to contact one of our certification services managers at email@example.com or (520) 790-2900.
Computer-based exams require side-view cameras
This video includes important information about room/workspace scans and using side-view cameras, which are required for computer-based exams. Candidates who don’t use a side-view camera will have their exam event invalidated.
To help offset the potential cost for our candidates, we will credit candidates $40 toward future ABR fees. These credits will be applied in early 2022.
RO candidate: Nervousness gives way to relief during oral exam
Mark Sittig, MD, is a radiation oncologist practicing with Tennessee Oncology in Nashville. He completed his training at Cedars-Sinai Medical Center in Los Angeles in June 2019.
Dr. Sittig recently passed the ABR Radiation Oncology Certifying Oral Exam. He was among an initial wave of candidates to take the exam remotely. Dr. Sittig shared a few thoughts with us about his experience.
What did you think about the platform going into the exam?
“Like most examinees, I was somewhat nervous about the experience but was reassured by the favorable reviews shared by a couple of colleagues who were involved in the pilot exam. I was also pleased with the tech check before exam day, which was well run.”
How did it work for you?
“Overall, the platform worked very well. The system of a navigator, examiner, and a backup examiner worked flawlessly. I utilized a wired internet connection, wired headphones, and a PC with two screens. The audio and video quality were excellent on my end.”
How were your interactions with the examiners?
“Other than normal exam-day jitters, it was easy to communicate with the examiners using the exam software. During one section, my primary examiner experienced an internet connection issue. The exam navigator promptly switched over to the secondary examiner, who was able to pick up right where we left off.”
What have some of your cohorts been saying about the platform?
“I was fortunate to have a great study group, and they all appreciated using the online platform as well as avoiding the expense, inconvenience, and time away from family associated with in-person exams.”
Passing remote oral exam a relief after a year of waiting
Izabella Barreto, PhD, DABR, is a clinical assistant professor of medical physics at the University of Florida who passed the ABR Remote Medical Physics Certifying Oral Exam in May 2021. You can follow her on Twitter: @BarretoPhd
By Izabella Barreto, PhD, DABR
It was January 2020 and I was starting to prepare for my final oral board exam. I had booked an international flight to celebrate certification in May, and thought the 40 rolls of toilet paper Amazon delivered was excessive. I was so wrong.
One month later, air travel was restricted and the U.S. declared a public health emergency due to the COVID-19 pandemic. Candidate studying ramped up during nights and weekends until we all received the dreaded (yet expected) email one month before the exam: “Urgent Notice: MP Oral Exam Postponed.” I called other examinees and the ABR to ask, “Should we keep studying? Do you think it’s postponed one, six, or 12 months?” Nobody knew, of course. During the summer came endless Zoom, learning how to teach my courses online, and continued studying. Stress, anxiety, and worry grew – we were ordering groceries, cleaning everything with Clorox wipes, and trying to stay safe. My family members contracted COVID-19, requiring prolonged hospitalization, and my friends were losing loved ones to the virus. It was a very emotional time, yet we continued to study. In July, we were notified the exam was rescheduled to May 2021, but now in a virtual format. We were very grateful to the ABR for undertaking the complex task of offering virtual exams, as we now had clarity and could plan for a fixed date.
A few weeks before the exam, the ABR sent reassuring emails detailing how far they had gone to ensure the delivery would be smooth. The technical checks were important: I learned my network’s privacy settings were incompatible with the Proctorio system, and my program’s IT department fixed it in a second technical check. The ABR also held a helpful orientation a few days before the exam.
My experience on exam day was extremely positive. I was paired with a kind navigator who reassured me by explaining all the options for handling technical glitches, although I had no problems. I also loved the virtual platform. Given the stress of the exam, I’m glad I didn’t have to travel or worry about flight delays. I was able to take the exam in my comfortable office and came home to relax right after instead of heading to the airport. This was a better experience for the examinees compared with the former hotel option.
A few days later, I found out I had passed! I jumped, screamed, and cried. While I had seen this result twice before from the ABR, this one meant overcoming a year of stress and uncertainty. In addition to the exam, we faced many personal challenges in 2020 during the pandemic. We were also the first group to study twice for this exam, to postpone our certification by one year, and to take it in a new virtual format. We are a special kind of DABR.
The following chart lists the 2022 dates for ABR computer-based exams. Dates for oral exams will be announced this summer. All 2022 exams will be held remotely.
Candidate shares tips for remote exam preparation
Dr. Mona Arbab is performing her residency at the Indiana University School of Medicine. She graduated from Tehran University of Medical Sciences before moving to Boston and working as a postdoctoral research fellow at Massachusetts General Hospital and continuing her research at Beth Israel Deaconess Medical Center. She did a surgery intern year at Brigham and Women’s Hospital before moving to Indianapolis for her residency.
By Mona Arbab
I am a PGY4 radiation oncology resident who took the biology and physics qualifying exam last month. A few weeks before the exam, we received information about the side-view webcam requirement and a video with details about the check-in process.
Based on the information provided, I ordered a webcam and a stand. The most important part of the side-view webcam is to show the entire workspace, so I used a small table next to my desk for the webcam. The practice exam is helpful to learn about the interface and become familiar with the calculator. To prevent accidentally opening other programs, I unpinned everything from the taskbar except for Google Chrome.
On exam day, the system did not allow me to log in early, so I waited for the exam to launch at my scheduled time. The ABR help desk phone number was available on the exam screen. I had no technical issues; check-in was the same as the practice exam without a live person involved, which prevented delays. I took the exam at home on my laptop and did not have connection issues. I was allowed to have water, snacks, and a small whiteboard.
I highly recommend that all residents complete the practice exam. Doing so helped me become familiar with the check-in process and interface and I solved the technical issues prior to the real exam.
Examiners say candidates adjusting well to remote platform
After successful pilot exams using our remote oral platform earlier this year, we hosted 285 remote candidates this week for the Medical Physics Part 3 Certifying Oral Exam. The exams started Saturday and wrapped up Tuesday.
Our volunteer examiners said their experiences this week show that candidates are adapting well to the new platform. Jean Moran, PhD, a professor and co-director of the physics division and associate chair of clinical physics at the University of Michigan, served as an examiner during the MP pilot and this week’s exams.
“So many of us have had to adjust to using virtual platforms over the past year,” she said. “I have been impressed by the poise exhibited by many candidates throughout the exam process.”
Fellow examiner Jon Kruse, PhD, is a consultant in the department of radiation oncology and associate professor of medical physics at the Mayo Clinic in Rochester, Minnesota. Dr. Kruse also served during the MP pilot and this week’s exams. He said candidates appear more at ease taking the exam remotely.
“It seemed like the candidates were as comfortable as they could possibly be, given the gravity and stress level associated with an oral exam,” he said. “Compared to previous years where this was done in person in a hotel room, I think the candidates seemed much more comfortable in their own homes, which was nice to see.”
We will feature new and returning examiners’ thoughts about the remote process in the June issue of The Beam.
MP candidate shares her thoughts on the remote experience
The following is an excerpt from an item written by Tyler Blackwell, MS, DABR, for radformation.com. Here, he speaks with Xenia Ray, PhD, DABR, about taking a recent Medical Physics Remote Certifying Oral Exam.
Tyler: Tell me about your experience with the virtual oral board exam.
Dr. Ray: My experience was overwhelmingly positive. There are a lot of advantages to the virtual platform in terms of time and cost savings. Not having to travel also decreased my stress from things outside of my control that might affect the exam. The virtual format does introduce new potentially stressful factors, particularly the worry that the internet connection may fail mid-exam.
Tyler: Do you think the ABR had the right protocols to address those concerns?
Dr. Ray: The ABR and Proctorio® did a lot to address those by:
- Having a pre-check of my computer system weeks before the actual exam and allowing a dry run of the exam software and interactive tools.
- Assigning a technical navigator to each examinee. There is always someone on the line, ready to handle any glitches.
- Including a backup examiner for each exam session if there’s an issue with the primary examiner.
- Giving very clear steps about what to do if I were to lose connection.
Tyler: Was the virtual setup easy to use?
Dr. Ray: Yes. I installed and tested it out during the pre-exam computer test. There was another opportunity to get comfortable with it on the morning of the exam before getting started. It has very little functionality, which I think is a good thing. You can draw and zoom, and that’s about it.
The one thing that was a little tricky was that most of the questions have an image associated with them and then a bulleted list of questions underneath. It might be nice if you could zoom in on just the image itself while leaving the questions viewable at the same scale. But it wasn’t a big deal.
Tyler: Did you experience technical issues?
Dr. Ray: My first examiner had a glitch on his end where the questions wouldn’t show up. The technical navigator walked him through the normal steps to resolve the issue, and as soon as those didn’t work, she switched control to the backup examiner instead. The whole thing took less than two minutes, and the ABR built in plenty of the time with the breaks between examiners (10 minutes) that I didn’t have to worry about running out of time.
Tyler: So overall, it was a success? Something the ABR should consider in the future?
Dr. Ray: Very successful! I hope they never go back to the hotel format, which is more expensive for candidates. It’s especially inconvenient for applicants with families who have to be gone for two days just to take a three-hour exam. It was also awesome that I could just enjoy the rest of my day right after the exam instead of heading to an airport.
Tyler: Any advice for those on deck to take the exam?
Dr. Ray: They can focus on reviewing the materials and trust that the technical setup is being taken care of for them. The only advice would be to take their time playing with the tools when they do the technical check with Proctorio®, but honestly, it’s very intuitive, so it is more about comfort level than really needing to practice.
Candidate: Completing practice exam is a must
Omar Chohan, DO, is a vascular and interventional radiologist in Buffalo who recently took the ABR IR/DR Certifying Exam. He found the ABR practice exam to be helpful for many reasons, which he writes about here. Dr. Chohan works at the University at Buffalo and Great Lakes Medical Imaging. You can follow him on Twitter: @drochohan.
By Omar Chohan, DO
Anyone taking an ABR computer-based remote exam should first complete the practice exam and technical self-check using Proctorio. The major reason is to test whether your internet connection and computer will provide a smooth testing experience. Problems may occur and it’s better for you to discover this in a stress-free environment and make adjustments as needed.
The practice exam also gives you a feel for the actual test day. It’s timed, lasts approximately 1.5 hours, and simulates exam day with the same interface, buttons, and feel. You may take it as many times as you need.
Knowing your Wi-Fi is fast enough to run the exam is critical. Having slow or inconsistent internet will make exam day much more stressful.
The practice exam will also help you figure out if your screen is adequate for viewing and interpreting images. For example, I took the practice exam on my laptop and again on my desktop. I found it was easier for me to use my desktop because the screen was significantly larger.
Make sure your computer and Chrome web browser are updated. This exam comes with its own stress and the practice exam definitely helped to calm my nerves in the sense that I knew my computer could run the exam without hiccups.
Using a side-view camera during remote exams
This video includes important information about room/workspace scans and use of side-view cameras, which will be required for computer-based exams starting April 15.
Side-view cameras necessary for future computer-based exams
Over the past two months, more than 1,500 candidates have taken remote computer-based exams offered by the ABR. Our goal was to create a platform that would not be subject to the unpredictable impact of the pandemic and would represent an improved experience for candidates. Maintaining the security of the process was important to ensure this element of the path to board certification would continue to serve as a valid indicator of knowledge and competence.
We have learned a lot in the past two months. Generally, exam candidates were respectful of the process and provided useful feedback. They approached a task that was both unfamiliar and stressful with exceptional patience and resourcefulness.
We learned that internet connectivity was imperfect but not so disruptive that the exam could not be completed by nearly all who logged in. The exam software platform itself functioned well. The vast majority of the content and images were adequate for the intended purpose.
One stressor frequently cited was the webcam recording. We borrowed online testing tools from higher education that have been used for many years, even more so over the past year. It became clear that these tools, while very reasonable for colleges and universities, create stress in a high-stakes environment (because of, for example, standard methods for tracking eye movement).
For this reason, we will be requiring a side–view (instead of the standard front-facing webcam view) for future computer-based exams beginning April 15, 2021. The new side-view webcam placement details are outlined here and a video can be found on our YouTube page.
In addition, to help offset the potential cost of this for our candidates, we will credit candidates $40 toward future ABR fees. These credits will be applied in early 2022.
We appreciate your patience as we attempt to optimize the exam experience.
If you have questions, contact a certification manager at firstname.lastname@example.org or (520) 790-2900.
Sara Hunter, MD, talks about taking the remote Core Exam
The chief diagnostic radiology resident at the Cleveland Clinic gives feedback and tips on the remote exam platform.
Q&A with ABR Oral Examiner Jessica Clements, MS
Jessica Clements, MS, is the chief medical physicist, regional radiation safety officer, and medical physics residency program director for the Southern California Region of Kaiser Permanente Medical Group. She’s also an oral examiner for the ABR, with experience that goes back to when those exams were held in Louisville, Kentucky.
Ms. Clements and other examiners are training and preparing for remote oral pilot exams in March. These exams will mark the first time the ABR has used a remote platform for orals.
Ms. Clements recently answered a few questions from an examiner’s perspective about changing to remote exams.
When the ABR was holding oral exams in Louisville, did you picture the process one day going remote?
I never imagined the exams transitioning to a remote format. So many things have changed around the world in the past year, making some things previously unimaginable completely common. I am very impressed that the ABR was able to adapt to those rapid changes so quickly and provide a high-quality, secure exam to candidates!
How much has your experience as an examiner helped you prepare for remote exams?
The solutions developed by the ABR for a remote exam should allow an easy transition for seasoned examiners as well as new examiners, especially after a little practice. Following a recent test of the system with other examiners and ABR staff, I felt very comfortable with the system.
Aside from the platform, do you think much else will change for candidates?
I am optimistic that the remote platform will be a good experience for candidates. When I took the oral exams in diagnostic and nuclear medical physics, I always had to prepare contingency plans such that if bad weather canceled flights, I could drive and still make it to Louisville in time. Candidates won’t have that layer of complexity adding stress to the exam process. They can identify a secure and comfortable location where they can tune in for the remote exam. Additionally, the examiner and examinee will both have digital tools to help point to specific items and communicate visually from their own screens.
How confident in the system should candidates feel as they head into remote oral exams?
I think candidates should be confident that they will receive a high-quality and secure exam. The platform has been tested extensively. One-on-one technical checks have occurred with examiners and will be conducted with examinees. The schedule has been adapted to prepare for unforeseen technical issues and an emergency technical support phone number has been established for use during exams. We are all familiar with continuous quality improvement in our clinical practices, and we should expect that the remote exam will only get better with age.
Looking back on the first round of the Core Exam
For most of the more than 1,300 candidates who took the Core Exam over the past two weeks, the new ABR remote platform was a success. Our volunteers, stakeholders, and staff have spent thousands of hours developing the system, and we will continue making improvements as we gain experience and receive user feedback.
While we’re pleased that most issues candidates had during the exam were solvable, we apologize to anyone who had difficulties. We had significant support staff available on exam days to help correct problems, most of which were related to connectivity, and will follow suit in future exams.
We will be using a similar platform for oral exams, which begin with pilots in March. The success of the Certifying Pilot Exam in January and the Core Exam this month assured us that our system for oral exams also will be reliable. Just as we did during the Certifying and Core exams, we will have support available for anyone who experiences issues.
We feel confident we have a reliable remote platform that will enable candidates to take this important step in their careers. We thank everyone for their patience and understanding as we begin a new era of exam delivery.
Candidate Discusses Core Exam Experience
Ryan Adams, MD, was in the first group of candidates who took the initial remote Core Exam last week. He is chief diagnostic radiology resident at Vanderbilt University Medical Center and a graduate of Vanderbilt University Medical School. He will be completing an independent residency in interventional radiology at Massachusetts General Hospital, and his academic interests include medical technology, medical education, and quality improvement.
After the exam, he answered a few questions about how things went.
How much experience did you have with remote exams heading into the Core Exam?
In my medical school, all our exams were online using testing software installed on our personal computers, so I’ve taken dozens of remote exams.
What were you expecting going into the exam, since it was the first Core the ABR has delivered remotely?
To be honest, I was expecting a very cumbersome check-in process where I would have to video chat with a proctor and show them my ID/exam room. I was fully prepared to have the exam software itself to be a relic from the ’90s – block design, clunky, and slow. Most importantly, I was concerned we would not have adequate image manipulation tools, specifically window/level, brightness, color inversion, zoom, etc.
How did the experience compare with what you expected?
My first thought on the practice exam, especially as someone who has written software and developed websites in the past, was, “Wow, this is actually effective testing software!” As I was taking the exam, I never felt the software was holding me back from performing my best. The check-in process was smooth and automated, no glitches. It was quick to move from question to question – no long load times. It had all the image manipulation features I needed: smooth functioning for the ultrasound cine clips, easy scrolling and manipulating a stack of CT images. Overall, I was very impressed with the testing software chosen to administer this exam.
What was the best part about the exam?
The best part was submitting the last question!
What should be improved upon?
While I overall thought the exam software and administration were very good, especially for the first virtual admission of this exam, I did feel like improvements could be made. Firstly, I think a strikeout feature to mark answers you know are incorrect would be a vital revision prior to administering the exam again. Secondly, I got multiple popups with feedback during the exam – e.g., keeping face in the center of the camera, improvements to the room scan, etc. I think those expectations could be better spelled out before the exam and honestly condensed to one brief, personalized message at the end of a block would be adequate. Otherwise, I think there is very little to improve!
What advice would you give people who will be taking computer-based exams later in the year?
My advice would be to review the room scanning requirements, make sure you have your ID with you, bring some water and a snack, and, finally, relax!
Remote Exam Effects on Candidate Fees
With the introduction of remote exams, ABR Executive Director Brent Wagner, MD, MBA, discusses the impact on candidate fees.
Typical remote exam questions answered
With the DR and IR/DR Core Exam administrations coming in the next two weeks, we have been answering candidate questions on our Twitter page. Here are a couple of questions submitted by two of the more than 1,400 candidates who will be taking the exam next month.
In light of the recent massive geographic ISP-related service outage, do you have formal guidance on how examinees are to proceed if some calamitous connectivity issues arise during the February administrations of the Core Exam?
We have planned contingencies for accommodating such situations. Those experiencing a prolonged internet outage during their exam should contact our Exam Day Help Desk. The Help Desk information will be emailed to candidates approximately 48 hours before the exam.
Should we lock our computers when we take a break? What if it auto logs off when we take a long break?
If you lock your computer, it will terminate the connection with the Proctorio® remote proctoring software and automatically log you out of the exam. When taking a break, select “start break.” You will not be logged off.
If for some reason you are logged off, you can re-enter the exam using the exam access link, which will be emailed to you prior to the exam, or use the “Enter Exam Lobby” button on myABR. You will go through Proctorio again to re-enter the exam, but once in, you will be able to pick up where you left off. Our exam software collects your responses as they are given so none of your responses will be lost.
Anyone with questions before the exam should contact our certification services team at (520) 790-2900 or email@example.com.
Examinees find many positives in first remote exam administration
Our Diagnostic Radiology Certifying Pilot Exam earlier this month provided an opportunity for us to gain a real-life look at a computer-based remote administration with a limited group of candidates. It was the first time we had administered a remote computer-based exam and the experience was largely a success.
The pilot was taken by 161 candidates, all of whom were given the chance to respond to a post-exam survey. We appreciate the 127 who shared their opinions about the experience; we will use their feedback to improve our platforms and processes.
Some interesting details emerged from the survey:
- More than 95% of respondents found the practice exam useful in helping them prepare.
- More than 80% took the exam at home rather than in a hospital or private practice office.
- More than 95% thought* that installation of the Proctorio®extension was easy to complete.
- More than 95% thought* that check-in through the Proctorio®platform was easy.
- More than 95% thought* that the exam platform (imaging tools, load times, answer entry, etc.) worked well.
- More than 90% thought* that “the image quality was adequate for testing purposes.”
- Survey respondents rated the overall experience a 4.5 out of 5 stars.
* “Agreed” or “Strongly Agreed”
We appreciate everyone who provided feedback. It’s instrumental in helping us ensure the continued delivery of a secure and valid exam, while keeping the exam day experience as pleasant as possible for the new remote administrations.
IT resource information available
During a pilot administration earlier this month, we gained additional insight into how security systems employed by many training programs/health systems may impact candidates taking ABR remote computer-based exams.
Candidates may take their exam in a location of their choosing. We acknowledge that some may pick a room in their training program or health system, raising potential network issues. As a result, we created a webpage with remote computer-based exam technical specifications for institutions.
This new page details the systems and processes used to deliver our remote computer-based exams, including specific domains, component requirement, web services, and website security parameters. Interested programs are encouraged to direct their IT departments to this page for details that may help them mitigate technical issues for candidates during their exam.
What we learned from our remote pilot exam administration
ABR Executive Director Brent Wagner, MD, MBA, discusses what worked and what will be improved upon after the DR Remote Certifying Pilot Exam on January 9, 2021.
Remote exam preparation update
ABR Executive Director Brent Wagner, MD, MBA, provides an update as the organization prepares to start administering remote exams.
More information on proctoring services
The ABR is working with Proctorio to provide remote proctoring services for oral and computer-based exams in 2021. Here are further details about how the system will be incorporated.
The ABR respects candidates and knows they are overwhelmingly honest and professional. We all want a secure and meaningful exam. ABR feedback will be focused on preventing potential issues, not implementing punitive measures.
The ABR is using Proctorio for remote proctoring. Only ABR staff will be able to review ID checks, room scans, webcam video, and microphone audio of exams. Candidate personal data (i.e., email address and phone number) is never shared with Proctorio.
Proctorio artificial intelligence (AI) will be used by ABR staff to determine the priority by which remote exam sessions are reviewed. Nothing identified by AI will be used directly to give feedback to candidates. AI is a tool to help prioritize ABR staff work.
Only ABR staff will review remote exam sessions. If questionable behavior is detected by staff, it will be reviewed by a team of ABR leadership (including certified diplomates) before feedback is given to candidates.
Feedback to Candidates
Our goal is to prevent candidates from unknowingly violating exam security rules. If there is a major exam security violation, feedback about problematic behavior will be communicated to the candidate as soon as possible. If there is an obviously harmless infraction (like kids running into the room during an exam), don’t expect feedback from the ABR. We will not contact a candidate unless it is absolutely necessary.
Dr. Wagner Presentations on Remote Exam Development Progress
Earlier this month, ABR Executive Director Brent Wagner, MD, MBA, hosted remote exam updates with multiple stakeholder groups representing the four specialties we certify. The following are recordings from those collaborative sessions. Anyone with questions should contact a certification manager at firstname.lastname@example.org or (520) 790-2900.
2021 Remote Exam Calendar Updated
The ABR Board of Governors recently announced remote exam dates for the second half of 2021. For registration dates, please visit the specialty calendar pages on our website.
Secure Exams Benefit the Candidates Who Take Them
By ABR Executive Director Brent Wagner, MD, MBA
Over the past few months, the ABR has initiated a complex set of projects to provide a series of modified exams for 2021. As we have announced, we will not require travel to our exam centers or commercial test centers beginning next year. Although the subject matter and scoring model of our exams will be essentially unchanged, remote exam delivery seeks to remove the uncertainty and anxiety surrounding the potential impact of the pandemic on this portion of the certification process.
The path to board certification includes satisfactory completion in two major areas: a residency and a series of standardized exams. The combination of these two elements and the significant accomplishment they represent are an important part of the social contract with our patients, their families, and the communities that medical professionals are privileged to serve.
Several compromises are necessary when designing high stakes exams; this is true not only with respect to ABR certification but also to the MCAT, LSAT, NMLE, bar exam, etc. Practicality (e.g., the length of the exam), fairness, and the candidate experience historically have been extremely important considerations for the ABR; the currently evolving remote exams are allowing us to look for novel opportunities to improve in these areas.
However, beyond these parameters, two specific factors require a careful balance: security and validity. Over the past few months, we have been reminded in meetings with stakeholder groups of faculty and trainees that remote exam platforms are inherently less secure than the traditional alternative of directly proctored exams that one might encounter in a classroom or test center. Specifically, the testing experience in higher education, from which we are borrowing our fundamental model (remote proctoring in one’s home or office using a webcam), has been repeatedly challenged by the creative application of “alternative methods of success” (e.g., searching for answers via the Internet) from isolated individuals lacking academic integrity.
To be clear, the ABR is confident that nearly all trainees in the radiology disciplines are well-educated, are committed to excellence, and epitomize the highest standards in academic integrity. The paradox is that we have an obligation to the candidates to develop a reasonably secure process – even as we admit that it is unnecessary for almost all residents. This process is still being defined but will likely require recorded webcam surveillance during the exam and a “room scan” to confirm the absence of prohibited resources (e.g., a cell phone, books, or notes).
If we fail to maintain security, we will have sacrificed not only fairness but also, just as importantly, the validity of the exam. The value of certification for the patient and, by extension, the candidate, depends on the credibility of the process as an indicator of a high level of accomplishment and ability.
Exam Delivery is Changing, but Content will Remain Consistent
With the best interests of our candidates in mind, we are moving to remote computer-based and oral exams in 2021. Despite the changes, one thing will stay the same: exams will continue to be developed by radiology and medical physics professionals who will ensure consistency of content.
Hundreds of volunteers help develop content for ABR exams. All give their time to ensure that our exams are fair and relevant to the candidates who take them. They have continued working remotely with our staff to make sure we have sufficient content even as we offer an expanded exam schedule next year.
Exam format and delivery logistics must change to accommodate remote exam locations and inevitable technical challenges. New break schedules and nursing mother accommodations are being discussed in addition to many other exam logistics. None of these developments will impact the amount or quality of content included in our exams.
Anyone with questions may contact our certification services team at email@example.com or (520) 790-2900.
Dr. Wagner Presentations on Remote Exam Development Progress
ABR Executive Director Brent Wagner, MD, MBA, recently hosted remote exam updates with multiple stakeholder groups representing the four specialties we certify. The following are recordings from those collaborative sessions. Anyone with questions should contact our certification services team at firstname.lastname@example.org or (520) 790-2900.
Update on Preparations for Remote Computer-Based Exams
The Value of Working with Stakeholders on Remote Exams
ABR Executive Director Talks About Planning Exams
Partner Chosen to Provide Insight on Establishing Remote Exams
In an effort to make remote oral and computer-based exams optimal experiences for candidates, we are partnering with Gartner, one of the world’s leading research and advisory companies.
Gartner is providing business insights, advice, and tools to help us meet the challenges of delivering remote exams. The areas in which we are consulting with Gartner include vendor selection, cyber security, purchasing, compliance, and development strategies.
We are confident that Gartner’s vast experience, which includes serving more than 14,000 businesses and nonprofits across 100 countries, will assist us in developing remote exams that exceed the expectations of our candidates.
Core Exam Will Be Held Over Three Days in 2021
One of the changes for the DR and IR/DR Core Exam remote administrations in 2021 will be extending the delivery from two days to three days.
The revision is primarily being made to account for factors beyond our control — Internet connectivity, power issues, etc. — which could result in exam interruptions. Additionally, a remote delivery will require modification to when breaks are offered and how exam content is accessed following a break. We don’t have all the details worked out on how the three-day exam will be structured; we will share that information as soon as it’s available.
The Core Exam is scheduled for February 1-3 and 8-10; June 2-4 and 7-9; and on November dates that will be determined and announced soon.
Q&A with ABR IT Director Scott Segal About Remote Exam Development
Work is going on behind the scenes to set the stage for us to begin delivering remote exams next February. Scott Segal, our IT director, is the point person for many of those efforts. He took a few moments this week to answer questions about the process.
What is the biggest challenge in switching to remote exams?
I think the hot-button issue is going to be delivering an exam that meets the high standards of the ABR for security, content, and, most importantly, examinee experience.
What are you doing to ensure that examinees feel confident using the new system?
There are some very specific challenges to giving a remote exam, so we are taking extra time and effort to mitigate risks as much as possible. For both computer and oral exams, we will give examinees advance access to the system to verify compatibility and get comfortable with the new interface.
Exam security is always a concern. How will that be handled?
Security is being addressed at multiple levels. Both ABR content as well as any information about examinees that they provide during the exam must be protected. We are working to increase the security of the ABR network and applications. This is a constant struggle and we supplement our internal staff with outside security experts.
How will the new exam delivery platforms be developed?
For computer-based exams, we are talking to a wide range of vendors and conducting extensive research on existing platforms. This should allow the Board of Governors to make the best decision possible to optimize the examinee experience. For oral exams, we are building new software that will allow the ABR to deliver an exam that replicates as closely as possible the experience examiners and examinees had when the exam was in person. We also seek to minimize, as much as we can, the stress for the candidates as they go through this process.
Stakeholder Calls Inform Exam Planning
Last week, the ABR convened meetings with initial certification stakeholder groups to solicit feedback about the move to remote exams. During these meetings, we provided a high-level overview of the remote approach for both our computer-based and oral exams.
ABR computer-based initial certification exams will take place in a remote location of the candidate’s choosing, provided that place meets a few basic requirements. Remote computer-based exams are not likely to be given at commercial testing centers (e.g., Pearson VUE) or ABR centers. The exams will use an ABR-developed exam interface similar to what has previously been used for computer-based exams. In addition, we will likely use a third-party vendor to handle exam-day security and remote monitoring. We will provide additional details about the requirements when we know more.
The oral exam will use an ABR-developed platform that will combine remote proctoring with video conferencing. As with the computer-based exams, candidates will have the freedom to select a location, but it must meet a few basic requirements. The details about exam-day location and other logistics are still in development and will be communicated when we have more information.
Many thanks to the leadership and staff of all the organizations that so thoughtfully engaged with us (listed below). We truly appreciate the collaboration as we begin this important endeavor and look forward to continuing the dialogue as we move forward.
As mentioned in previous communications, we’ll be sharing more information on exam dates for 2021 by August 1.
Association of Program Directors in Radiology (APDR)
Association of Program Directors in Interventional Radiology (APDIR)
Society of Chairs of Academic Radiology Departments (SCARD)
American Alliance of Academic Chief Residents in Radiology (A3CR2)
Radiological Society of North America (RSNA)
American College of Radiology, Resident and Fellow Section (ACR RFS)
Accreditation Council for Graduate Medical Education, Review Committee for Radiology (ACGME-RC)
IC Advisory Committee for Diagnostic Radiology
Association of Directors of Radiation Oncology Programs (ADROP)
Society of Chairs of Academic Radiation Oncology Programs (SCAROP)
Association of Residents in Radiation Oncology (ARRO)
Society of Directors of Academic Medical Physics Programs (SDAMPP)
IC Advisory Committee for Medical Physics
Society for Interventional Radiology (SIR)
Society for Interventional Radiology, Early Career Section (SIR ECS)
Society for Interventional Radiology, Residents, Fellows and Students (SIR RFS)