I.C.E. Virtual Exchange Preview: Debunking Myths About Linear-On-the-Fly Testing

By Robert C. Shaw, Jr., PhD

11.5.20

Among those who operate a certification examination, recent challenges or curiosity could prompt a question like, "If there is a smarter way to operate, will those changes fit our program?"

Presenters Kelly Zhao, a psychometrician from PSI Services, and Jessica Rapey, senior manager of professional assessment for the Commission on Dietetic Registration, plan to answer this question among others during the 2020 I.C.E. Virtual Exchange. During a session entitled, “Debunking Myths About Linear-On-the-Fly Testing: What to Prepare When Moving Into a “’LOFT,’” these presenters will describe how a program's fixed-form operations were assessed and how those results affected linear-on-the-fly (LOFT) implementation. If you share concerns about subject matter expert performance, hitting project milestones, budgetary impact or item exposure, then this session could be a good fit for you.

We spoke with Zhao and Rapey to learn more about their session.

Why is this topic important to those who run credentialing programs?

Adopting an unfamiliar method like LOFT can be a complex, labor-intensive and scary undertaking. These risks can be minimized after an assessment of LOFT readiness and evaluating the implementation roadmap. This session will describe the psychometric assessment of fit between LOFT and a credentialing exam program. A credentialing program manager will describe the experience of switching from fixed-forms to LOFT, including key differences. Session attendees will be invited to share their questions and their own potential myths about LOFT.

What highlights do you hope attendees will take way from your session?

We hope attendees will get the specific information they need, or questions they should ask their psychometrician, about their credentialing program while considering LOFT. We also plan to address areas a credentialing program may have to change to implement LOFT.

What characteristics help LOFT fit well with a program?

Higher candidate volume
Higher number of fixed forms developed each year
Limited number of pretest items approved after each testing cycle
Type of items
Details on the exam blueprint and other specifications
Higher number of items in the bank with known statistical performance
The policy on retakes and recertification
Comfort level with item/exam exposure or test security concerns
Opportunities in a typical budget and project timeline
Adaptability of subject matter experts to the training and workload

Robert C. Shaw, Jr., PhD

National Board for Respiratory Care (NBRC)

Robert Shaw worked as a respiratory therapist and respiratory therapy educator between 1982 and 1995. After earning a doctorate degree in Higher Education with emphases in measurement and statistics from the University of Missouri in 1994, Robert began advising occupational credentialing programs in 1995. Robert worked for Applied Measurement Professionals, Inc. (AMP), from 1995 through 2015 while providing psychometric direction to several programs including the National Board for Respiratory Care (NBRC). The NBRC owned AMP until the company was sold to PSI at the end of 2015. Robert is the Vice President of the Examinations division for the NBRC. Author of "I.C.E. Virtual Exchange Preview: Debunking Myths About Linear-On-the-Fly Testing," "Spotlight on Jim Henderson: 2019 ICE Lifetime Achievement Award Winner," "Three Short Reliability Stories," "Adapting to the Unexpected: Joy Schneck, Executive Director, Certification Board for Music Therapists," "Your Organization’s Examination Cannot Be Valid," and "New Credential Maintenance Program Features to be Implemented by the National Board for Respiratory Care."