By Ben Babcock, PhD, and Zachary Siegel, PhD, The American Registry of Radiologic Technologists, and Greg Applegate, PhD, MBA, National Registry of Emergency Medical Technicians

Publicly reporting key information about credentialing exam performance is a requirement for National Commission for Certifying Agencies (NCCA) accreditation. There is, however, little research or guidance about what information organizations are currently publishing or should aspire to publish. This article summarizes what we presented during our session, “Givin’ the People What They Want: Public Exam Statistics Reporting for Your Stakeholders,” at the 2020 I.C.E. Exchange about public exam reporting.

Current Public Exam Reporting

We first want to outline what credentialing organizations are currently doing. To this end, we documented the public exam reporting of all 133 organizations with NCCA-accredited programs as of January 2020. We found that most organizations reported exam pass rates and not much else. Pass rates included the number and percentage of passing exam candidates. Variations included all candidates, first-time candidates only, repeat candidates and different eligibility pathway pass rates. For those reporting more than just a pass rate, the most common statistics were the mean or median total score, the standard deviation and statistics indicating measurement quality (e.g., reliability estimate).

Proactively Serving Key Stakeholder Needs

Given that so few organizations reported more than pass rates, we offer our experiences with reporting that may help an organization proactively serve the needs of key stakeholders.

1. Candidates: The people taking the exams regularly ask how to use the scaled scores on their reports, especially after failing their first attempt. By reporting section scores and individual percentiles, these candidates can add context to their results and guide both the scope and intensity of preparation for their next attempts. It could also be helpful to provide information about measurement error near the cut score or to caution candidates about the potential unreliability of subscores.
2. Educators: These stakeholders may also benefit from state-level reporting by identifying opportunities for growth or creating goals to ensure they remain competitive locally. It is also helpful to break down scores by section to help educators tailor the components of their programs to better serve their students. Finally, educators may wish to know how their programs rank as a percentile to market themselves as a quality program.
3. State or provincial regulators: Regulators may ask for performance local to their states rather than a national average. By publicly reporting exam volume and pass rates for each state, an organization can simultaneously serve all states ― even states that had not thought to ask for local data ― while saving the time it takes to create each individual state's report.
4. Researchers: Independent researchers may be interested in any of the reporting above. Historical trends often serve as either a core component of a study or as context for unexpected and unusual results. By publishing a wealth of public data, credentialing organizations can further the profession in ways they may not be equipped to handle on their own.

Providing this information to the public strengthens the argument for the effectiveness of your examination program and could be as simple as publishing the technical reports for the examinations. However, organizations should be cautious when deciding what information to share publicly. Issues such as disparate impact in pass rates, Americans With Disabilities Act (ADA) compliance and statistical analysis based on small test volumes can put an organization into a legally challenging position if conducted and presented poorly. A psychometrician ― and possibly legal counsel ― should review the information before sharing it publicly.

Presenting Information to the Public

When providing the public with examination analysis, include a clear description of the source and nature of the data used for the analysis. The methods used for doing all analyses and rationale or explanations for these methods are also extremely important. These explanations could include how to interpret the analysis with references to literature. Chapters 12, 22 and 23 of “Certification: The I.C.E. Handbook” are excellent references for what should be considered when creating and presenting examination technical information.

With this in mind, a key question to ask is, "What can my organization do to improve its public score reporting?" Below we recommend four possible steps to improve an organization’s public score reporting and explain how high up on the public reporting ladder each step will take your organization.

Step 1: Post exam statistics on a dedicated page of your website with an obvious and easy-to-search title (e.g., "Exam Statistics," "Exam Performance") ― 60^th percentile

Note: Most organizations' exam statistics were difficult for us to find during our research period. By taking this step, your stats will be easier to find and will make your statistics reporting webpage as good as, if not better than, 60% of the webpages from organizations with programs having NCCA accreditation.

Step 2: Report basic, descriptive statistics about the exam's overall score (e.g., median and midspread) ― 80^th percentile

Step 3: Report a statistic estimating the quality of your measurements (e.g., reliability, decision consistency) ― 90^th percentile

Step 4: Report a statistic that reflects something unique to your program (e.g., section score performance, performance by state) ― 95^th percentile

Finding so many organizations' exam statistics was a difficult task because many had the information in difficult-to-find parts of their websites. Creating a dedicated statistics page will make your organization above average even if you decide to report nothing different. If you do want to report more statistics and want to learn more, you can access our presentation on the I.C.E. Exchange website. We provide a free, downloadable spreadsheet that calculates key statistics simply by pasting in your exam data (no psychometrician required).

Conclusion

It takes bravery for an organization to publicly publish more than just exam pass rates. We encourage you to muster your courage and be proud of your high-quality measurements by publicly posting more exam performance information. You can simultaneously give your stakeholders what they want, increase your organization's transparency and save your staff valuable time.