Declining Pass Rates: It’s Not the Test, It’s the Candidates

Although inspired by true events, the following story is entirely fictional.

It’s August 2021, and the Board of Trustees for the National Board of American Medical People (NBAMP) gather for their annual meeting. The COVID-19 pandemic has been ravaging the globe and the need for medical personnel in the field has never been more urgent. Their eyes grow wide with disbelieving horror when it is revealed that for the fourth straight quarter, the pass rate for their entry-level certification exam remained lower than before the pandemic. Dismay quickly turns to outrage as they turn to the director of testing and cry out, “What’s wrong with the test?”

Though hopefully an overdramatization, this scenario may be uncomfortably familiar to many credentialing specialists. An I.C.E. study examining the effect of the COVID-19 pandemic revealed 46% of credentialing programs saw a decline in their pass rates from 2019 to 2020. The following year, 66% of programs saw a decline. For many health care-related programs, like the fictional NBAMP certification program, the consequences of a decrease in credentialed personnel entering the workforce during a pandemic were cause for alarm.

It is understandable to ask whether a lower pass rate is the result of variance in content or statistical properties between test forms. However, for organizations that follow the Standards for Educational and Psychological Testing and the National Commission for Certifying Agencies accreditation standards, the answer should be an emphatic “no.” With the exception of a change in the test design or content (a scenario avoided by many organizations during the pandemic), rigorous practices are followed during form assembly and equating to ensure each form maintains an equivalent passing standard.

The next logical question to ask is whether a change in test administration can explain lower pass rates. This was a particularly important question during the COVID-19 pandemic, when many credentialing programs either partially or fully transitioned from administrations in testing centers to administrations of live remote proctored exams. And though it is vital for each organization to examine their own candidate data, multiple studies, including a recent article in the Journal of Applied Testing Technology, revealed most credentialing programs saw no significant differences in pass rates between the testing administration routes.

Now, back to our fictional story.

“Okay,” the NBAMP Board reluctantly concedes, their voices brimming with skepticism. “But, if the test hasn’t changed, and the changes in test administration don’t explain the falling pass rates, then what is left? Where did it all go wrong?”

The director of testing takes a deep breath and braces herself before replying, “The explanation lies not with the test, but with the candidates.”

Once changes to the test and changes in test administration are eliminated as explanations for declining candidate pass rates, it is time for organizations to acknowledge the explanation falls with the candidates. For health care-related programs, the possible sources of differences between the candidate population before, during and after the pandemic are numerous.

Change in Clinical Instruction

One explanation for the initial decline in pass rates may lie in the change in clinical instruction students received. Clinical instruction often occurs in the latter portion of an education program and is designed to provide students an opportunity to practice what they’ve learned in the classroom, as well as gain supervised first-hand experience treating a diverse population of patients.

During the pandemic, some hospitals were described as “resembling war zones.” Students experienced the stresses of surging patient censuses, lack of proper PPE, growing public distrust of medical personnel and an overworked and irregular rotation of instructors. On top of this, students had fewer opportunities for exposure to a variety of patients. A 2020 CDC study showed that an estimated 41% of U.S. adults delayed or avoided medical care because of concerns of contracting COVID-19. Limited opportunities to treat diverse patients prevented many students from making important connections between classroom and clinical instruction. This may have also skewed their perception of the prevalence of, and relationship between, certain symptoms and conditions.

Changes in Classroom Instruction

While changes in clinical instruction may explain the initial decrease in pass rates, changes in classroom instruction may have resulted in more lingering reverberations. Classroom instruction is where students learn vital fundamental knowledge needed for future practice. During the pandemic, educational programs quickly transitioned online, with many retaining some elements of virtual learning even as the pandemic eased.

Without passing judgment on the effectiveness of online instruction, it is obvious this represents a significant difference from most students’ pre-pandemic educational experience. Technology issues, lack of face-to-face interactions with their instructors and classmates and increased distractions outside of the classroom were just some of the difficulties students faced during this time.

Student Population Composition

For some educational programs, it was not just the type of instruction students received that changed, but the composition of the student population itself. A report from the National Student Clearinghouse Research Center revealed a 5.7% decrease in undergraduate enrollment for health professions and related clinical sciences from January 2020 to December 2022. 

To combat decreasing enrollments, some educational programs attempted to reduce barriers to admission by eliminating standardized testing requirements at the pre-admission stage. Between the spring of 2020 and the fall of 2021, the number of four-year universities and colleges with undergraduate health professional programs that transitioned to test-optional admission policies nearly doubled from 713 to 1,350. And by the end of 2021, a large number of educational programs for doctoral health professions, including 47.2% of occupational therapy programs and 52.6% of pharmacy programs, no longer required a standardized entrance test score.

Returning to our story:

“You have convinced us,” the NBAMP Board at last declares. “But what do we do? We can’t let this devastation continue.” Their faces are resolute with conviction as they avow, “We must take decisive action.”

Faced with lower pass rates, it’s natural for organizations to look for corrective actions. Some may consider “showing grace” to their candidates affected by the pandemic by easing their eligibility requirements and/or retake policy. Some may want to go a step further by lowering the passing score.

However, the time to reevaluate eligibility and retake policies — and to reset the passing score for an examination — is not during a time of crisis, but in conjunction with a planned reevaluation of the test design and content. Taking any of these actions rashly, in reflexive response to what are possibly transient data, would result in the lowering of the standard that had been established and validated as necessary. To do so would not only violate the principle of fairness for past candidates, but is also likely to result in less-able practitioners in the field, a negative outcome at any point in time.

“We must stay the course,” the director of testing wisely advises the Board. “Take no hasty actions. We are trusted by our certificants and by the public to maintain the standard for qualified practice. We must not betray them.”

End Scene