Case Study: Developing an ABMS 3C-Approved Longitudinal Learning Assessment for a Continuing Certification Program

Many Member Boards are exploring unique approaches to longitudinal learning assessment (LLA) as part of their continuing certification programs. In this case study, we examine how the American Board of Plastic Surgery (ABPS) worked with a psychometrician to develop an innovative LLA that met the approval standards of the American Board of Medical Specialties (ABMS) Committee on Continuing Certification (3C). This collaboration between ABPS and their psychometrician may serve as a model for other Member Boards seeking to develop and implement a Board-approved LLA, or organizations that are rethinking their maintenance of certification offerings.

Longitudinal Learning Assessment Goals

ABMS introduced LLA, also referred to as an assessment for learning, to its Member Boards in 2017 as a way to reshape continuing certification programs for their diplomates (Board-certified physicians and medical specialists). LLAs are designed to support learning, identify gaps in knowledge and assess diplomate knowledge. The concept of an LLA is to administer shorter, more frequent assessments, as opposed to a lengthy multiple-choice assessment given every five to 10 years.

The goal of the LLA is “to assure the public that the diplomate continues to meet the standards of the specialty, and to assist diplomates in keeping up with evolving standards of practice in the specialty.”¹ Feedback is a key component of the LLA. This ensures that practitioners are receiving regular feedback as it relates to the performance standard to identify any gaps in their knowledge as it relates to specific content areas, and to paint a clear picture of what successful performance looks like. LLAs are meant to help keep diplomates aware of new or updated information in their field with a keen focus on learning emerging trends or urgent issues.

At the end of each LLA administration, diplomates receive performance reports indicating their current LLA status and their overall and subtest scores for the administration. Performance on the LLA is tracked for both initial responses (baseline knowledge) and final responses (submitted after the presentation of educational content supporting the item). The LLA status is meant to provide diplomates with ample time to address performance deficits prior to the end of their LLA cycle. Mean scores for all diplomates who complete the same subspecialty assessment module are also reported for comparison.

Assessment Research and Design

ABPS worked with a psychometrician from Data Recognition Corporation (DRC) on the design and development of their LLA. ABPS began to pilot their LLA in 2019, and following the 3C guidelines and recommendations from the psychometrician, introduced clone items into their LLA design to track evidence of learning and retention. (Clone items test a similar previous topic, but the items are worded differently. Leveraging clone items can result in increased efficiencies during the item writing process because they measure the same construct without the effort of drafting a brand-new item.)

One important decision point was determining which standard setting method would be most appropriate for an exam that was extensively customized, as ABPS diplomates have the opportunity to select a module that aligns with their practice. DRC psychometrician Katie Hanrahan, MS, MEd, assisted ABPS with this process, helping to define cohorts, handle key validations in the way that worked best for ABPS and facilitate the standard setting process, which focused on both performance and participation of the diplomates.

Hanrahan also supported ABPS on regression analysis to compare the relationship of the performance on the LLA and other ABPS assessments, g-theory to support the reproducibility or reliability of diplomate outcomes, statistical comparison of candidate cohorts (subspecialties, certification duration, point in the continuing certification cycle), new statistics, and assistance with time-sensitive reporting and presentations. After reviewing the performance of the clone items, ABPS and Hanrahan discussed topics such as the use of clones versus duplicates.

Conclusion

In March 2022, ABPS received approval from the 3C for their LLA. Going forward, the feedback provided by the LLA will help diplomates assess their baseline knowledge, identify strengths and weaknesses in their knowledge and stay abreast of new information and emerging trends in plastic surgery and their specific subspecialties.

The collaborative effort between ABPS and their psychometrician helped them navigate the unknowns of developing this new assessment with recommendations, data analysis and guidance. As Member Boards continue to explore longitudinal learning assessment for their continuing certification programs, we strongly recommend they include a psychometrician at the beginning of the process, during the design phase and then throughout the pilot in order to guide, advise and support a successful outcome for committee review.

Reference

1. ABMS (2021, October 29). American Board of Medical Specialties Standards for Continuing Certification. Retrieved from https://www.abms.org/wp-content/ uploads/2021/11/ABMS-Standards-for-Continuing-Certification-20211029.pdf