By Elisa Kahn, PMP, DMA

For many organizations new to certification, establishing a credential program is all about the test — and for good reason, as it will become the organization’s key product. Often overlooked, however, is the infrastructure needed to realize the entire credential program.

Before diving into the infrastructure of a program, let’s assume the organization has the following in place:

• Needs analysis
• Strategic, sales and marketing plans
• Mission and vision statements 
• Governance (including policies, procedures, legal considerations, corporate structure, standards, administration and resources such as an advisory board)
• The exam side is well under way with key components in progress (job-task analysis, test specifications, eligibility requirements, item development, plans for standard setting, etc.)

With this in mind, let’s look at a high-level overview of what every organization new to credentialing should think about when it comes to a program’s infrastructure.

One Versus Multiple Vendors

When creating a program, it can be beneficial to consider multiple vendors rather than one vendor handling everything. Though it may seem counterintuitive at first, this allows for greater diversification, ensures one vendor does not control everything and takes into consideration that some vendors have strengths in certain areas of a program, but not all areas.

In looking to branch out to multiple vendors, consider a four-legged stool where each leg represents a part of the entire credential program:

• Application: Including demonstrating eligibility
• Exam: Testing the standards
• Recertification: Maintaining the credential
• Fulfillment: Distributing hard copy certificates and digital badges

 

Application

Tracking applicants’ activity as they become candidates, and ultimately credential holders, falls under the candidate management system (CMS). A CMS is the gateway to the credential in which individuals complete the exam application and demonstrate eligibility requirements to move forward. The CMS gathers and retains (for an agreed duration) applicant demographic information. In many cases, the majority of communication comes from the CMS to the candidate in predesigned automated messaging. Application status is tracked (approved, denied, flagged for audit, missing information), test results are stored and the credential holder’s recertification journey is done completely in the CMS. Think of the CMS as the candidate’s bucket, where application information, test results and recertification information are all held.

 

The CMS is also responsible for communicating to the test delivery vendor who is eligible to test.

Concepts to consider with the CMS vendor regarding the application process include but are not limited to:

• Designing the application and guide for applicants  
• Determining what data is collected
• Requesting special accommodations
• Document uploading capabilities
• Length of time an incomplete application remains active
• Length of the eligibility to test period
• How to handle statuses such as “Audit,” “Approve,” “Deny” and “Appeal”
• How to handle duplicate applications
• How fraud is handled
  • Banning applicants
  • Multiple applications for one person (in the case where one person has multiple applications to test more frequently than permitted in an application period, i.e., Jonathan Doe/John Doe, Stephen Smith/Steve Smith)
  • Revoking credential
• Retaining scores
• Re-exam rules/communications
• Monitoring re-examination attempts
• Communication to the applicant, candidate and credential holder
• Recertification (discussed in greater detail below)

Exam

When it comes to the exam, consider one vendor who develops the items with subject matter experts (SMEs), holds the item bank where all test items reside, builds the exam forms, delivers the forms, keeps the psychometric data to monitor the items and forms, and creates client specific reports. You can also consider having two vendors with the tasks split, where the test development vendor develops the items with SMEs, holds the item bank and builds the exam forms, and the test delivery vendor delivers the exam and collects item and exam-level data (which is then shared with the test development vendor). If an organization chooses two vendors, it is imperative to determine how data will be transferred between the two vendors and clearly designate who will be providing psychometric data analysis and client specific reports.

Concepts to consider with test delivery/test development vendor(s) include, but are not limited to:

• Item types that will be created and delivered that best test the subject material
  • Hot spots
  • Fill in the blanks
  • Essay
  • Matching
  • Multiple response multiple choice
  • Single response multiple choice
  • Video
  • Observation
  • Drag and drop
•  Form assembly
  • Fixed¹
  • Adaptive²
  • Sections³
• A delivery platform that best supports item types, form assembly and meets program policies for exam standardization and security measures such as:
  • Computer-based testing at a test center
  • Computer-based testing via remote proctor
  • Paper and pencil
  • Observation
  • Combination of any of the above
• Statistical analysis
  • Item level
  • Exam level
• Statistical analysis methodology
  • Classical test theory⁴
  • Item response theory⁵
• Scoring
  • Immediate results
  • Delayed results
  • Partial scoring
  • Special requirements such as a minimum score in each domain to pass
  • Section scoring if the exam has sections
  • Individual domain scores
  • Raw score
  • Scaled score
  • Equating
• Length of exam
  • Duration
  • Number of test items
• Testing and retesting rules
  • Number of times one can test during an eligibility period
  • Waiting period between retests
  • Waiting period between eligibility periods (if any)
  • Price charged for first test
  • Price charged for retest
• Testing accommodations
  • Legal requirements
• Security and forensics
  • Risk analysis — threats and vulnerabilities
  • Prevention plan
  • Determining item overexposure
  • Determining cheating or theft of items
  • Reaction plan if security has been breached

Recertification

The CMS supports the credential holders as they maintain their credential. With regards to recertification, organizations should consider developing an approved education provider (AEP) network that can sync with their CMS.

Requiring individuals to manually enter all the different educational venues that they attend for credential maintenance puts a heavy burden on the credential holder. It is up to the individual to upload supporting material that will prove their attendance and demonstrate that the event is appropriate for continuing education (CE).

Creating an AEP network allows approved AEPs to list their courses, seminars and conferences in the CMS and upload the attendance roster so CE credits can be automatically applied toward the attendee’s recertification application, at best. At worst, the credential holder need only select the event they attended and then upload proof of completion. While not a necessity for recertification programs, it is a great way to make the process of maintaining the credential simple for credential holders and for organizations to market their educational offerings and attract new learners.

Concepts to consider with the CMS vendor regarding the recertification process include but are not limited to:

• Designing the application and guide for certificants
• Determining what data is collected
• Document uploading capabilities
• Tracking the individual certification cycle (renew, expire, extend)
• Tracking the number of CEs submitted per credential holder
• Communicating to credential holders
• Determining the reminder schedule
• Depository for all approved education providers
• Depository for all approved education events

Fulfillment

Depending on the infrastructure, the credentialing organization may create hard copy certificates in-house and then mail them to the credential holder or will work with a printing house that will take care of this activity. Similarly, with the adoption of digital badges, if outside vendors are used, the credentialing organization will need to transfer data to these vendors.

Concepts to consider with the fulfillment vendor include but are not limited to:

• Transferring address and email information
• Proofing hard copy certificates
• Paper stock
• Frames
• Communication
• Mailing method
• Undeliverable mail
• Mailing to other countries
• Tracking packages
• Lost packages
• Types of badges
• Badge usage
• Undelivered badges

Putting It All Together

If the organization has concluded that working with multiple vendors is the direction they would like to go, integration will need to occur. It is important to realize that vendors that have been around for a while may have older technology. This may put a burden on the younger vendor to be more flexible and provide data that meets the older vendor’s particular specifications.

The diagram below illustrates the infrastructure for the entire credential program. While some tasks can be worked on simultaneously, such as building the item bank and creating the CMS for candidate and AEP use (indicated in yellow in the chart below), it becomes evident that once a submitted application is approved (indicated in green in the chart below), the inter-vendor integration process begins in earnest and the infrastructure that supports the credential program has to be ready to go.

Chart provided by author.

 

In conclusion, by understanding the infrastructure of a credential program and how the different functions integrate, organizations will realize that the program is so much more than the test. Equal emphasis, time and development should be given to the application, recertification and even fulfillment as finding the right vendor(s) is key to running a smooth program that will support all stakeholders. 

Footnotes

1. Fixed forms: Depending on the test version (form), candidates who take the same form receive the same test questions – even if the test items are randomly presented.
2. Computerized adaptive form: Every candidate receives a unique set of test questions which are dependent on their responses to the previous questions.
3. Sections: Exams may be presented to the test takers in groups of questions (sections) that must be completed before proceeding to the next section. In many cases, sections are impermeable, meaning that test takers cannot review questions from a prior section.
4. Classical Test Theory (CTT): A traditional psychometric method of developing and evaluating examination responses based on candidates’ raw scores. The two most commonly used classical item statistics are the difficulty index (p value) and discrimination index (typically the point-biserial correlation). (I.C.E. Terminology Task Force 8).
5.  Item Response Theory (IRT): A mathematical model of measurement in which candidates’ ability and item difficulty are converted to a single common scale that provides a standardized way of comparing candidates’ performance and item difficulty over multiple versions of an examination. (I.C.E. Terminology Task Force 13)

Sources

1. Albano, T. Introduction to Educational and Psychological Measurement using R. https://creativecommons.org/licenses/by/4.0/. 2020.
2. Bong, D. Fixed Form vs Adaptive Test Design in Language Proficiency Testing, Avast, https://avantassessment.com/blog/fixed-form-vs-adaptive-test-design-in-language-proficiency-testing.
3. Frohn, S. Increase Security And Efficiency With Linear On The Fly Testing (LOFT), PSI, 25 May 2021, https://blog.psionline.com/certification/increase-security-efficiency-linear-on-the-fly-testing-loft.
4. Henderson, J, editor. Certification The ICE Handbook. 3rd ed., Institute for Credentialing Excellence, 2019.
5. I.C.E. Basic Guide to Credentialing Terminology. 2nd ed., Institute for Credentialing Excellence, 2020
6. Knapp, L and Knapp, J. The Business of Certification: Creating and Sustaining a Successful Program. Association Management Press, 2017.