Evaluating Declining Pass Rates: Looking Beyond the Numbers

Editor’s note: I.C.E. members hold a variety of perspectives on the reasons behind declining pass rates and the strategies needed to effectively deal with them. This article follows one that showcases an additional point of view by Jenny Banek.

Over the past few years, some credentialing programs, specifically in certification and licensure, have observed declines in pass rates. In many of these programs, the effect has been attributed generically to disruptions that were caused by the pandemic. Although not universal, these programs have been subject to concerns raised by various stakeholder groups including candidates, training programs and policymakers about the declining performance of candidates. In some instances, this may be a longer-term trend for programs while for others, the effects of the pandemic have contributed to these outcomes.

Critics of credentialing programs often focus on the tests as a culprit for the decline. They may allege that programs increase barriers to entry for diverse candidate populations. Or they allege programs make decisions about minimally qualified performance that don’t align with decisions education or training programs make. So, a decline in pass rates increases the number of individuals concerned about the test’s role.

It’s true that credentialing programs have a legal and professional obligation to ensure substantive and procedural fairness for candidates as part of a program’s validity argument, so I am not minimizing that responsibility. However, when exam blueprints, minimally qualified performance definitions, exam forms and equating practices have often remained consistent for a program, it is essential to go beyond the pass rates; we must evaluate contributing factors to better understand why we see these trends. This article briefly discusses some of the avenues programs can pursue and proposes how to address two frequent questions in ways that preserve the value and credibility of a program.

1. Why are we seeing these trends for some programs?

An important, and perhaps primary, factor that has contributed to a reduction in pass rates is the changes in learning opportunities for candidates that can be linked to the pandemic. These changes were multidimensional and likely interacted with other factors. For example, training programs may have had to modify curriculum and instruction during the pandemic to accommodate changes in physical location, mode (e.g., in-person, hybrid, synchronous or asynchronous online), or simply volume of information that could reasonably be delivered. Any of these alone or in combination could influence candidate performance.

As a related example, opportunities to gain practical or clinical experiences may have been curtailed for similar reasons. It is reasonable to then anticipate that these limiting modifications to candidates’ learning opportunities resulted in a differently prepared candidate population. Even though we recognize eligibility requirements for certification and licensure programs are intended to be training program neutral by not preferencing certain curricular choices or instructional approaches, it does not mean they are immune from effects that broadly influenced candidate preparation.

A potential shift in the candidate ability distribution does not necessarily mean the candidate population changed. This means the characteristics of the candidate population may have remained randomly equivalent (technical phrasing for “roughly equal”), but they may not have performed as well as previous cohorts for some of the reasons discussed earlier. In comparison, some programs that had fewer eligibility requirements did see a more meaningful change in their candidate population that is more than just an effect of diminished learning opportunities. Specifically, the pandemic presented an opportunity for many people to seek alternative career pathways through credentialing. For programs that saw an uptick in candidate volumes, evaluating the applications to compare education, training and experience backgrounds with historical data may help explain changes. The outcome of that review could uncover a true change in the population that may have also shifted the ability distribution in a direction that was lower than that of a typical cohort. If programs are seeing increased numbers of eligible candidates seeking credentials without the preparation to succeed, a decline in pass rates should be expected.

It should also be noted that credentialing programs may have observed a decline in candidate volumes that could be an extension of declining enrollments in training programs that may have resorted to admitting students who were not as qualified as previous cohorts. In these instances, the candidate population may not have nominally changed, but the ability distribution for that population may have shifted downward.

It's important for readers to recognize that limiting learning opportunities and a potential change in the candidate population are only two of potentially many factors that could contribute to an observed reduction in pass rates. Programs are encouraged to pursue additional factors that may be appropriate for their context, considering the comprehensive nature of the credentialing program — eligibility, demonstration of minimally qualified performance for the purpose of the credential and maintenance of the credential. After determining which factors have contributed most to a decline in pass rates, programs can communicate these reasons to stakeholders, but may also need to provide a response to concerned stakeholders.

2. What can be done to respond to the observed changes?

In response to a program that has experienced declining pass rates, my first suggestion to policymakers is to exercise patience as they evaluate root causes. Asking the program to better understand the decline can help to provide an appropriate response. Is it a pandemic-related blip or a sustained trend? Which factors are having the greatest influence? The answer to these questions can lead to different evaluation approaches as well as solutions and stakeholder communications.

Perhaps most important is for the program fight the urge to reflexively modify passing score requirements to achieve a particular pass rate. This approach changes the criterion-referenced interpretation of the passing score (i.e., candidate performance relative to a defined standard) to a normative one (i.e., candidate performance relative to other candidates’ performance). The latter approach suggests that the credentialing body is seeking to admit or pass a certain percentage of candidates, regardless of their performance relative to the definition of minimal qualification for the credential. This strategy would also run counter to expectations in professional and accreditation standards for establishing passing standards.

If the program is observing a sustained trend, evaluating other components of the program may be needed. This may involve examining eligibility requirements, offering supplemental learning opportunities or revisiting a program’s job analysis if there have been significant changes in practice. Another lever that programs can use to potentially respond to this trend is to use its recertification or maintenance of competency (MOC) program to be more directive in helping credential holders catch up to current practice. If the changes to the profession are significant, it may require establishing a new baseline for the program which may involve full redevelopment that includes standard setting.

Determining a Path Forward

Determining why a credentialing program’s pass rate has declined can lead to conclusions informed by multiple factors. Although more prevalent because of the effects of the pandemic, some programs started observing these declining trends prior to these more recent years. The potential explanations for these changes that were discussed here serve as a jumping-off point to guide further exploration of a program’s context. Understanding why pass rates have declined can help programs better evaluate stakeholder concerns, determine appropriate mitigation approaches and communicate how these strategies help maintain or enhance the value of the credential for candidates and existing credential holders.