Lessons Learned From Virtual Test Development Meetings

Responses collected by Steven Garner, PMP, CAE, California-Nevada Section, American Water Works Association, Vicki Gremelsbacker, MSEd, Professional Testing Corporation, and Jeff Kelley, PhD, Alpine Testing Solutions.

With the COVID-19 pandemic this year, credentialing organizations have had to adapt to new business practices, including transitioning to virtual test development meetings. We talked to numerous organizations, subject matter experts (SMEs) and meeting facilitators with the intent to advance knowledge/understanding and provide lessons learned that may guide others through their own exam development processes. A consensus seemed to surface that cost savings for online sessions were evident (some indicated 100% due to lack of travel, catering and conference room reservations). Additional key takeaways are included below.

All interviewees are referred to by initials. A guide with full names, credentials and titles is shared at the end of the article.

What was the single biggest disadvantage of online exam development compared to in-person?

SW: I think the biggest disadvantage by far is the committee’s missed opportunity to meet face-to-face, and to start building that camaraderie before really diving into the work together. We’ve got a group of SMEs who are all passionate about their profession, and coming together to contribute and give back to that profession. In doing this type of work, I think there’s a synergy that forms within and even across teams, and it does make a difference in how they collaborate. 

What was the single biggest advantage of online exam development compared to in-person?

KA: Being able to engage a large and geographically diverse group of SMEs quickly, easily and with little or no cost. I think especially for item creation and item review, the benefits of virtual sessions far exceed any drawbacks. I think virtual sessions are also less intimidating for new volunteers; they certainly take less time, require no travel and allow even very busy professionals to help in ways they might otherwise be unable to. It’s often easier to get someone to commit to a two-hour virtual item review session than a full-day, in-person session. 

Have your online exam development meeting results been as good as in-person meeting results?

SG: Absolutely. The results were as good as, or better than, the in-person meetings. SMEs actually worked after hours to review and improve upon the accomplishments of the day, and that fed into the next day’s progress. The online sessions were conducted in two, three-hour sessions (morning and afternoon), which left the SMEs less drained from the process than when the in-person meetings ran eight hours. 

SW: The dynamics in a virtual meeting are so different from in-person meetings, but the quality of work still remains high while we’re in this virtual environment. What we’ve noticed is that productivity on work between meetings has lowered. And we completely understand that, currently, we’re just in a different time. A lot of people are still adjusting to working from home, while balancing family and just everyday life in general.

Do you feel you, the SME, had the same level of input into the exam process when you participated in an online meeting versus the in-person meeting?

BB: Yes, I believe the level of input was equivalent in both settings, but I could see how others may not feel the same. I have seen how others can easily be distracted from their points by well-placed interruptions or distractions. The same could be true for in-person meetings, but it is not as easily accomplished.

DS: The potential for an individual’s input level is consistent with in-person meetings; however, when a member is new, it is dependent upon the individual’s personality as to how much they will participate. In a virtual environment, it is more difficult to be heard if a member is less familiar with the team and/or process.

How has the security of materials been managed in your online exam development meetings?

SR: This is more of a client-by-client approach. Some clients prefer to house/exchange content via file transfer protocol (FTP) sites, third-party hosted sites, cloud-based drives or the use of password-protected item authoring tools.

JML: It depends on the test development task being conducted. When working with SMEs to develop content, I train everyone at the same time via a virtual meeting. I follow up with specific item writing assignments for each SME. Once raw content has been produced and loaded into the content management system we are using, no one other than me has access to the items. I schedule specific SMEs for remote meetings to review, edit and approve the items, again, limiting access to content. If the task is to approve an exam form, I project items to a controlled set of SMEs. I do not release intact exam forms. For standard setting workshops, I also control access to intact forms. When the meeting is completely virtual, then I project items in sets of five (that fit on my screen). SMEs are provided with rating sheets and can populate them as we work through the exam form.

MD: Limited materials are sent to participants. Highly secure documents are viewed on screen and controlled by the facilitator. Non-disclosure agreements (NDAs) must be signed before work begins. Confidentiality is discussed with the participants. 

Can you share any promising alternative methods of online exam development?

SR: In terms of process methods, we have a menu of options for virtual exam development projects that span a spectrum based on flexibility --> fidelity to the in-person dynamic. At the “flexibility” end of the spectrum, projects are conducted in a more distributed fashion, and tend to have more asynchronous components. For example, for an item development project, SMEs can watch pre-recorded training, draft items in an item authoring tool at their convenience and even engage in check listed asynchronous review. These projects might span the course of several weeks, but this can work well for clients who are unable to get SMEs to block off days at a time for the project.    

At the “fidelity to the in-person” end of the spectrum, projects tend to get completed within a tighter timeline and rely on a more synchronized meeting approach, similar to the way in-person meetings tend to function. As an example, for an item development project, SMEs and the facilitator often meet on web-based platforms all day for several days at a time. We maintain our connection through the platform while we engage in alternating independent work and review and discuss cycles. In this way, SMEs and facilitators can talk to each other as needed, via voice or chat, and remain on mute when they don’t need to. This allows us to foster a team dynamic that’s similar to in-person meetings. It also allows SMEs the ability to avoid feeling like they have to be glued to the meeting all day.

Currently, we’re using web-based meeting platforms, along with internet-based tools to conduct virtual test development projects. We will likely pilot the use of mixed reality applications in the near future in order to get closer to the in-person meeting dynamic. These technologies rely on relatively new technologies and have their challenges, but they also allow for outside computer screen and application sharing, just like a real, in-person meeting.

JML: I prepare item writing assignments proactively so the SMEs know exactly what their writing load will be (both the number of items and assigned content areas). I then staff tech reviews of the raw items in a way that minimizes exposure of the content. For example, the author of an item rarely (if ever) is also a reviewer of that item, so in the end, the author is unaware of what the final, approved item looks like. If possible, it is very helpful to use a good content management system that allows the test developer to manage who has access to content, when and for how long. 

How can one manage SME engagement while online?

JML: Tape a list of SME names to the side of the computer monitor and send at least one pre-meeting email to introduce yourself, and to outline the goals of the upcoming meeting. 

Managing a large group of SMEs in a virtual environment can be difficult. Inevitably, there is an SME that is outspoken and there is someone that is shy or reserved. The trick is to get everyone involved and to keep everyone involved. Calling people by name always seems to help. 

Try to create a comfortable tone in your virtual environments, and encourage SMEs to say when they need a break.

MD: Present a clear agenda and meeting schedule at the beginning. Provide an opportunity for all to participate. Show respect for all opinions, but tactfully cut short digressions with a reference to the clock and amount of time left to cover the materials/agenda. It’s important to keep a fast pace to retain participant engagement.

How many hours can an online meeting go before you think participants begin to lose focus?

The consensus was that a two-hour session seems to work best. If a longer meeting is needed, include a break.

Conclusion

When SMEs were asked what they missed most about meeting in person and what worked out better by meeting online, most said they missed the personal connections, networking and in-depth conversations about exam development. Most appreciated saving time and money by not having to travel.

Many organizations will be using online exam development during the COVID-19 pandemic and some of them will continue long after, given the benefits, perhaps using a blend of in-person and online methods.

Article Development Method

The three authors of this article reached out to their respective networks to gather responses to an agreed upon set of questions around the use of online exam development processes compared to the use of in-person processes. The various respondents come from a range of organizations and represent three distinct functions. Each respondent was offered questions common to all participants and unique questions related to their role or function (exam development hosts, facilitator/psychometrician, SME).

Below is a list of the respondents that are referenced in the article using only their initials.

BB: Ben Bennett, owner, Backflow Prevention Specialists, Inc.; SME

DS: Deborah Suzan Huff, AIA, NCARB, LEED AP BD+C, SSOE Group; SME

JML: Joy L. Matthews-Lopez, PhD, founder and president, JML Measurement and Testing Services, LLC; Facilitator

KA: Kate Anders, CBCC-KA, CPDT-KSA, CDBC, exam chair, Certification Council for Professional Dog Trainers; Host

MD: Marlene Dunham, program manager, Professional Testing Corporation; Facilitator

SG: Steven Garner, certification manager, California-Nevada Section of the American Water Works Association (CA-NV AWWA); Host

SR: Scott Russell, senior test development professional and manager, test development, Alpine Testing Solutions; Facilitator

SW: Sheronne Wilson, manager, examination, NCARB; All