How to Identify, Address and Prevent Test Irregularities

A portion of this information was first presented at the 2023 Council on Licensure, Enforcement and Regulation (CLEAR) conference in Salt Lake City, Utah, and includes survey data provided by CLEAR members.

Test irregularities refer to any deviation from established rules, protocols or procedures during the administration of a test. Test irregularities are inevitable in any test administration context and are also evolving over time as both testing methods and environments are changing. In the past, in-person test administrations may have been disrupted by issues like power outages and noise distractions. While these issues still occur, remote proctored tests (RPT) and novel ways of cheating introduce additional concerns.

Test administrators must become more vigilant about issues that compromise validity and security in modern testing. As such, there need to be well-developed policies and procedures in place before an irregularity occurs to ensure that candidates are treated in an objective and consistent manner. This article will address the kinds of test irregularities, their potential impact on the testing process, the results of a survey to collect data on irregularities experienced by credentialing organizations and possible policies and procedures that can address them.

Kinds of Irregularities and Their Potential Impact

Irregularities can be intentional on the candidate’s part, or unintentional due to factors beyond the control of the candidate, some which may happen in-person or during RPT. Intentional irregularities include the use of proxy test takers, false identification, item harvesting or unauthorized communication with others. Unintentional irregularities can include leaving the testing area or exiting the exam before it is completed, medical or other emergencies and power outages.

Test irregularities can impact both test security and test validity. For example, item harvesting will result in the exposure of confidential materials, compromising the security of future tests. If test takers have access to the test content in advance, they may gain an unfair advantage, leading to an unequal and biased testing environment. Cheating or interruptions during test administration can lead to inaccurate measurement of the intended construct. This measurement error reduces the reliability and validity of the test results.

Test irregularities can also give rise to legal challenges and ethical concerns. Test-takers who are affected by irregularities may challenge the validity of their results or pursue legal action against the testing organization. It is critical for organizations to be aware of how to recognize irregularities when they occur and to have well-defined policies and procedures in place for addressing them.

How Testing Organizations Experience Irregularities: A Survey

Professional Testing Corporation conducted a survey of the CLEAR membership to collect information on test irregularities in the credentialing community. Data were collected on characteristics of over 100 programs.

Respondents indicated working with programs of varied annual candidate volumes as shown in Figure 1.

Figure 1

The majority of the members surveyed offer licensure exams, as can be seen in  Figure 2.

Figure 2

Respondents indicated the test modality/modalities for their testing program as indicated in Figure 3. The majority of respondents use in-person proctored testing at test centers while 33% use both in-person and remote proctored testing.

Figure 3

Survey respondents were asked to provide input on several main categories of test irregularities encountered in the last year. Here are a few of the most common answers:

• Using a proxy test taker/misrepresenting identity/impersonating another candidate/using artificial intelligence
• Item harvesting
• Communicating with others during the exam
• Using unauthorized devices, such as a phone, camera or recording device
• Exiting exam software before the exam is finished
• Leaving the test center or view of the RPT proctor without proctor permission for the allowed break during the exam administration
• Medical issues causing a test interruption
• Interruptions during testing that are outside a candidate’s control, such as a workstation issue, power outage, test center evacuation or RPT proctor/vendor technical issue
• For RPT testing only — candidate technical issues, such as power, internet or equipment failure

The surveyed irregularities are listed in rank of prevalence from low to high in Figure 4. Less than 10% of the respondents indicated they experienced irregularities such as misrepresented identities, while over 40% experienced exam interruption for both in-person and RPT exams. Close to 50% of the respondents experienced RPT technical issues.

Figure 4

Test irregularities have a significant impact on various stakeholders. Ninety-eight percent of the respondents experienced exam administration irregularities in at least one of the above categories in the past year. However, only 75% of the respondents have established policies to handle such irregularities. Developing a test irregularity policy is crucial for maintaining the integrity, fairness and reputation of the program and ensures that exam security measures are consistently applied.

Reducing Irregularities Through Clear Policies

Clear and transparent policies provide candidates with a thorough understanding of what is considered acceptable behavior during an exam and what constitutes a violation. This guidance helps prevent inadvertent violations and ensures that candidates have a fair opportunity to succeed based on their knowledge and skills.

A few considerations and benefits of a well-structured irregularity policy include:

• Outlining a standardized process for addressing violations, reducing the potential for bias or subjective judgments, and promoting fairness and accountability. This also protects both the institution and the candidates, helping avoid legal disputes that might arise from mishandling.
• Identifying types of irregularities that could occur during exam administration assists organizations to proactively have policies in place prior to the occurrence of an irregularity. It is important to consider the specific context of the exams being administered.
• Defining responsibilities upfront ensures everyone knows who needs to be included when an irregularity occurs. A clear process for reporting irregularities ensures all parties, including staff and vendors, comply with the institution's standards for test security and integrity, maintaining a consistent approach to irregularity management.
• Allowing various stakeholders within an institution to bring diverse expertise can contribute to a more comprehensive and effective testing process.

An organization should develop a communication plan, so candidates have a clear understanding of the testing process — including before, during and after exam administration. The plan should include information on who sends out the communication and how it is sent out. New or updated policies must be communicated to candidates so they are aware of current policies and the penalties for violating them. For best results, an organization should send out communication messages via multiple methods, multiple times.

A thoughtful decision-making process will need to occur to determine what action is appropriate for each irregularity. Violations can vary in severity, and the degree of severity and impact on exam security and candidate performance must be weighed and policies updated or created to address the irregularity going forward. Prevention, detection and resolution all need to be considered.

It may be helpful to set up flow charts with the various potential irregularities and the decision making that occurs for each throughout the process. See Figure 5 for an example:

Figure 5

Maintaining Irregularity Policies

Rather than merely reacting to reported irregularities, an organization should take preventive measures to minimize the occurrence of such incidents, present recommended policy updates to internal and external stakeholders for review and create plans to roll out new policy to staff, vendors and candidates.

The irregularity policy should be reviewed and evaluated on a regular basis. This allows organizations to stay current with emerging challenges and advancements in test security and ensures the policy remains relevant and effective in addressing current and potential threats to test integrity.