Certifying the Certifiers: An ICE-CCP® Exam Performance Update

It is important for all certification programs to demonstrate that the exam(s) used to make certification decisions are reliable and valid for their intended purpose, but this is particularly true for a certification program for credentialing professionals. We now have enough data to analyze and evaluate the I.C.E. Certified Credentialing Professional® (ICE-CCP®) exam, and results indicate a high-quality exam that is meeting its intended purpose.

The new information provided in this article is an analysis of the ICE-CCP exam as a measurement instrument, and the analysis results indicate the exam is reliable and performing well within psychometric expectations for high-stakes certification exams such as this.

As outlined in the Credentialing Insights article “I.C.E. Launches Certification Program for the Credentialing Professional,” the I.C.E. Board decided in February 2020 to move forward with a certification program for credentialing professionals. A Steering Committee, representing expertise in credentialing across a variety of occupations and professions, was formed and worked alongside an external consultant to oversee the process. The I.C.E. Board wanted to ensure adherence to National Commission for Certifying Agencies (NCCA) Standards and industry best practices in the development of the I.C.E. Certified Credentialing Professional® (ICE-CCP®) credential. More than 100 credentialing professionals volunteered to serve as subject matter experts (SMEs) and participated in various exam development activities beginning in September 2020, including job analysis, item writing and review, form review and standard setting.

Development of the Initial (Beta) Certification Exam

The content validity of the exam was established through the job analysis conducted in early 2021. In short, the job analysis ensures that the exam is measuring the intended construct, namely the knowledge needed of a credentialing professional.

After reviewing the results of the job analysis, the Steering Committee decided to develop one credential reflecting generalist-level ability and understanding across three domains — governance, program operations and assessment development and validation. Furthermore, it was also decided that the target candidate population would be practitioners with a minimum of two to three years of professional credentialing experience. In 2021, governance of the credential was transitioned to the newly formed Certification Services Council, which first met in December 2021.

A beta exam was delivered between October 25, 2021 and November 9, 2021 via live remote proctoring (LRP).

The beta form included a total of 200 exam items, exceeding the number of scored items required (130). This was a deliberate decision to allow for selection of the best-performing items to be scored and key finalization prior to standard setting. To encourage a sufficient number of applicants, the registration fee was deeply discounted for the initial administration.

As is often common for the initial administration of a new certification exam, a relatively large number of applicants responded, with a total of 152 candidates testing. It is likely that many of these “early adopters” were seasoned professionals, which may help explain why they were the highest-performing cohort to date. Since the delivery of the initial exam through 2023, a total of 233 individuals have earned the ICE-CCP credential. 

Psychometric Properties of the ICE-CCP Exam

The psychometric evidence that follows indicates that the ICE-CCP exam is reliable and a useful measurement tool for making valid certification decisions. Prospective candidates can be assured that they will be completing a high-quality assessment.

Content Validity

Each form of the ICE-CCP is built based on the published examination content outline (ECO) document published in 2021. The ECO stipulates that 32% of operational items must map to the governance and resources area, 37% of operational items must map to the credentialing program operations area and that 32% must map to the assessment development and validation area. Additional targets are given at the sub-domain level. Each form of the ICE-CCP published since 2021 has met these targets.

Table 1: ICE-CCP Exam Content Outline Subsection Item Count

Test Characteristics

To evaluate the psychometric quality of certification exams, measurement professionals often compute estimates of reliability (i.e., the consistency of test scores) and decision consistency (i.e., the proportion of candidates who would receive the same pass/fail decision if they were to repeat the exam without influencing their ability). Cronbach’s (1951) alpha is a commonly used measure of test reliability with values ranging from 0 to 1, and a general guidance for high-stakes tests is that it should be at least 0.80, but preferably 0.90 or higher (Nunnally & Bernstein, 1994). Similarly, Livingston & Lewis’s (1995) decision consistency index also ranges from 0 to 1, and values above 0.8 or 0.9 are generally desired for high-stakes certification exams. All forms of the ICE-CCP exam have displayed strong properties with respect to reliability and decision consistency (see Table 2).    

Table 2: ICE-CCP Psychometric Characteristics (2021-2023)

Score Distribution

Based on all 283 candidates who took the exam from 2021 to 2023, the following score distribution is presented.

Figure 1: Raw score distributions for all candidates taking the ICE-CCP Examination (2021-2023)

The score distribution pattern is typical of high-stakes assessments with stringent eligibility criteria. The average score is 93.5 out of 130 (standard deviation = 14.9) with a median score of 95. 

Raw scores on the ICE-CCP exam are converted to scaled scores ranging from 200 to 800 where 600 represents the passing score. Candidates whose scores are below passing receive their scaled score so that they can make an informed decision about retesting. 

The original cut score was set in late 2021 using the modified-Angoff method, a well-established standard-setting method for certifying exams (Cizek, 2012). For all subsequent forms, the passing standard has been maintained through circle-arc equating (Livingston & Kim, 2009). As a result, even though examination content may change across different forms, 1) all content is linked to the same examination content outline and 2) all candidates are being held to the same standard, regardless of whether they receive a slightly easier or harder form.

The ICE-CCP Is a Valuable Credential

The information provided in this article helps demonstrate that the ICE-CCP examination is psychometrically sound and meets generally accepted standards in the certification industry. While having a valid and reliable certifying exam serves as an important foundation, the ICE-CCP program will continue to evolve and improve. For example, the option to take the exam at a test center was added to LRP for the May 2024 test window in response to candidate feedback. One-third of the applicants for that testing window (14 out of 42) chose the test center option.

As with any new credential, adoption may take time. Data regarding employer and credential holder perceptions of the credential are not currently available; however, some employers have started including the credential as preferred hiring criteria, and a majority of respondents to the post-exam survey indicated that their employers have provided support in both obtaining and maintaining the ICE-CCP credential.