Key Challenges to Remote Proctoring: Communications and Technology, Security, and Privacy
Prior to 2020, many of us might have had an academic interest in remote proctored online testing, focused on attending sessions on the topic at industry conferences or held abstract discussions about the pros and cons. But in 2020, organizations need in-depth expertise for urgent decision making.
Many reading this article have quickly gotten savvy about the basics of remote proctored online testing and there is a great deal of reference material available on the topic. Here, instead, I am going to focus on three key challenges my organization, National Board of Certification and Recertification (NBCRNA), has encountered as a test sponsor — communications and technology, security, and privacy — along with suggestions to manage such challenges. We have used both live online proctoring (LOP) and record and review, and have experience with several different vendors.
Terminology and Background
One basic area I want to cover first is terminology. Is the proper term online, remote or internet-based testing? Industry consensus is forming around online, so I will use that term in this article.
The key distinction in online testing is proctoring. There are two prime modalities that are of interest to credentialing organizations ― LOP and record and review. Both will most often incorporate a lockdown browser. While there can be some overlap in these modalities, it comes down to:
- LOP is an exam session where a remote proctor synchronously oversees the check-in process and monitors the candidate throughout the exam.
- Record and review is an exam session where the entire testing session, including check-in, is recorded and reviewed later.
There are also bells and whistles that can be incorporated into either of these, such as artificial intelligence to flag suspicious behavior, use of camera equipment (smartphone or equipment other than just the computer video), kiosk-based testing and more.1,2
Communications and Technology
While this may sound like two topics, they are really intertwined and organizations will be well served by considering this as one element. In the best of all worlds, candidates will click on a link, proceed to follow software prompts, authenticate identity, scan the environment and complete the test. Most candidates navigate the online test experience without apparent incident, not always thanks to their preparation but instead to good fortune and well-designed vendor software. A small, but significant number, will encounter technology difficulties, which can cause disruption and anxiety, constituting a challenge to the validity of inferences from the exam results.
Giving up a brick and mortar live-proctored experience does come at a price. Candidates need to understand that they must take on responsibility for setting up their own test environment and checking their own equipment prior to testing if they want a smooth experience. There is a lengthy list of potential concerns including, but not limited to:
- Having a walled room with a closed door and a clear desk
- Disconnecting and/or moving any additional monitors
- Webcam/microphone/speaker requirements
- Operating system and local administrative permissions
- Screen resolution, internet speed, browser, firewalls, cookies, pop-up blocking and antivirus interference
The dilemma here is the amount of information required to assure that all contingencies are covered. It is common knowledge in our business that candidates tend not to read instructions, so we often try to limit what we provide.
The best online experience for the candidate and for the test sponsor demands that candidates do read and prepare. You need to create urgency about reading these instructions — strong language with a warning that 10 minutes of reading now may save you a considerable headache and even possible test invalidation later. Varied approaches to information presentation ― narratives, checklists and videos ― should also be part of your preparation offerings. Most vendors offer a practice run opportunity, which can be very useful.
Customer service communication offered by the vendor is the second key component of a strategy to deal with technology issues. Despite the best preparation, these issues will occur. It is critical to understand the vendor’s capability to respond quickly and responsively to a candidate’s difficulty in navigating the online testing process. Contractual metrics and consequences relative to customer service may be the best way to ensure that the vendor and test sponsor have the same expectations.
Security
Most test sponsors have delayed a move to online testing for as long as they have due to the dual security concerns of intellectual property (IP) theft and cheating. In the continuum of approaches to test administration, LOP provides more safeguards than record and review, but not quite the same measure of security as an in-person administration. Considering the prime threat of theft of IP, use of small cameras is a possibility in any type of testing administration today, but is more likely to occur with online proctoring than in-person.
A key element, again, is the candidate’s understanding of the rules; beyond the instructional material that must be read, candidates need to consent to an agreement. Don’t hold back on an admonition that the agreement must be read before a candidate can test. Make sure the agreement covers the possibilities ― cameras/recording equipment, prohibited reference materials, unauthorized devices and additional monitors, reading items out loud or talking to others, glancing away from the screen and leaving the camera frame.
Furthermore, protocols need to be in place for handling violations. What should the proctor do if a candidate is glancing away from the screen for longer than established as acceptable? Does your candidate agreement make the consequence clear? Are you willing to follow through? And, since what is seen on video may not be sufficient evidence to establish that there has been IP theft or cheating, data forensics should be used to establish testing patterns that might support an allegation of a security breach.3
Managing breaks is a key factor in the security bundle. Ideally the candidate will not take a break during an online exam. Given exigent circumstances or a long exam, however, this may not be practical. Precautions an organization can take, that would allow for breaks, include forward only navigation, a break “button,” which can only be activated when the current question has been answered, and a limit on break time and frequency.
Privacy
Some candidates express surprise that their computer is locked down, that their ID is scanned or that portions of their home are videotaped. We have heard from candidates that they are angry at exposing themselves to so much potential for ID theft or stating that the 360-degree view of the testing area was an invasion of privacy and personal security. A wide range of potential privacy concerns can be present, from inadvertent inclusion of individuals other than the candidate included in the video, to use and retention of biometric data.
Again, the instructional materials and candidate agreement are key. Legal requirements concerning privacy are evolving and can vary by jurisdiction. There is industry guidance from the Association of Test Publishers4, and test sponsors should look to the online vendor for assistance in explaining what is involved in computer lockdown and how video material is used and retained. The range of rapidly changing country and jurisdiction level laws and regulations regarding privacy make it difficult, if not impossible, for the average test sponsor to navigate this arena on its own.
Summary
It is not easy to make decisions on modifying testing programs to meet the challenges we face today. The considerations discussed above are not the only factors to consider. In determining if and how you will use online proctoring, you might best view pros and cons through three lenses — stakes, practicality and impact.5 Evidence needed to support a testing program, or more specifically, the use of scores from that program should be weighed. Limitations resulting from a strict adherence to the most secure test administration modality may jeopardize the ability of a test sponsor to provide accessible testing opportunities to qualified candidates. A final judgement — and it is a judgement — rests on which approach to overall testing validity, reliability and fairness has the most beneficial impact.
References
- Langenfeld, T. (2020). Internet‐Based Proctored Assessment: Security and Fairness Issues. Educational Measurement: Issues and Practice, emip.12359. https://doi.org/10.1111/emip.12359
- Weiner, J. A., & Hurtz, G. M. (2017). A comparative Study of Online Remote Proctored Vs Onsite Proctored. Journal of Applied Testing Technology, 18(1), 13–20.
- Edited Henderson, J. (2019). Certification: The ICE Handbook, 3rd Edition. Institute for Credentialing Excellence.
- International Privacy Subcommittee. (2020). Privacy In Practice Bulletin Series: An ATP Security Committee Initiative. Association of Test Publishers.
- Tannenbaum, R. J., & Kane, M. T. (2019). Stakes in Testing: Not a Simple Dichotomy but a Profile of Consequences That Guides Needed Evidence of Measurement Quality. ETS Research Report Series, 2019(1), 1–16. https://doi.org/10.1002/ets2.12255