Job Analysis: Beyond the Basics

Job analysis is the cornerstone of any sound, legally defensible credentialing program. Although a job analysis study is required by all of the major accrediting bodies, it is essential even for unaccredited programs that are designing or updating credentialing exams. In fact, the results of a job analysis study are the single most critical piece of evidence supporting the interpretation and use of credentialing test scores. In other words, the results of a well-conducted job analysis study are what allow a credentialing program to make the claim that the content of their examination reflects the necessary knowledge, skills, or other competencies required by practitioners in that field. Without a job analysis, there is no basis for the claim that individuals who possess a particular credential have job-related knowledge or skills in that field. Simply put, a job analysis is the primary method for establishing the validity of credentialing test scores.

Considering that there are about as many different ways of conducting a job analysis as there are job analysts, credentialing bodies may feel overwhelmed by the number of difficult decisions that need to be made throughout the course of a job analysis study. This article will discuss some key decisions involved in conducting a job analysis for a credentialing program.

What Are the First Steps In a Job Analysis Study?

A job analysis study is a large and resource-intensive undertaking that requires a significant amount of planning and preparation. Like any large project, conducting a job analysis requires identifying key project personnel, deciding on a methodology, and creating a detailed project schedule. The project schedule should include ample time at the outset of the project for the job analyst to become familiar with the job or role being analyzed. In addition to the previous job analysis study or existing test blueprint, the job analyst must also be supplied with background information about the profession, which may include books, job descriptions, videos, articles, standards, information about recent developments or trends in the field, and any other relevant information. While a psychometrician or other consultant hired to conduct a job analysis will have an excellent understanding of job analysis techniques, the quality of the study will likely be vastly improved if the job analyst is also well-versed in the profession being studied. It is crucial not to skip this step out of eagerness to “get started” — this is an important part of the job analysis.

Although established credentialing programs will generally have an established purpose and target audience statement, the development of these statements is often overlooked by new credentialing programs or programs conducting a job analysis for the first time. Prior to beginning the job analysis, credentialing programs should have identified and documented the objective of the credential, the nature and scope of the work being performed in the job or role, and operationally define the level of professional who will pursue the credential (e.g., entry-level, mid-career, advanced practice). In addition, at least rudimentary eligibility requirements should be established prior to beginning the job analysis study. Even if the intention is to refine these statements as part of the job analysis study, beginning the process with at least a general sense of the credential’s purpose and target audience will help inform key decisions throughout the study, including the composition of the job analysis panel and the most appropriate methodology for the study, and will also help maximize time at the job analysis panel meeting.

What is the Most Appropriate Job Analysis Methodology?

The wide variety of job analysis techniques available can make choosing an approach seem challenging. While there is no single correct way to conduct a job analysis study, it is essential that the methodology supports the desired outcomes of the job analysis study. The primary objectives of a job analysis for a credentialing program are to determine the content of the credentialing examination, establish the job-relatedness of that content, and provide documentation of the linkage between the job and the credentialing examination. A job analysis study may also provide other useful information that can be used to inform various aspects of a credentialing program, and these goals may also be a factor in selecting the most appropriate methodology.

Most credentialing programs conduct a two-part job analysis study. The first step involves convening a job analysis panel for a face-to-face meeting (generally lasting one to three days, depending on the complexity of the job or profession and the specific goals of the meeting). The panelists must be knowledgeable about the job or role in question and should be representative of the credential’s target audience in terms of roles, practice settings, demographic characteristics, geographic location, and experience.

There are a number of different approaches to facilitating the panel meeting, with the choice of approach tending to depend on the time allotted for the meeting, the goals of the meeting, and the preference of the job analyst. Regardless of the process used, the main objectives of the panel meeting generally focus on identifying some or all of the following elements:

• Major areas of job content or responsibility (often called content domains or performance domains)
• The essential job tasks performed at the specified level of practice
• The knowledge and/or skills (or other competencies) required for competent job performance in the job or profession

The second step in the study involves using a large-scale survey to validate the job tasks and/or job knowledge delineated by the job analysis panel. Conducting a validation survey is considered an essential part of the job analysis process in the field of credentialing, as professional testing standards require evidence to support the claim that the knowledge and skills being assessed by the credentialing examination are required for competent job performance across the job or role.

Should Your Framework Be Content-Oriented or Performance-Oriented?

The decision regarding whether to focus the job analysis (and, hence, organize the test content outline) on the tasks performed on the job versus on the knowledge required for successful job performance can be difficult. Creating a process-oriented framework that is organized around job performance domains has the advantage of providing a clear and direct link between the key job activities and the content of the examination, as required by professional testing standards. However, for some programs, such as many medical credentials, a test content outline that focuses on the knowledge required for successful job performance may simply make the most sense. Some programs choose to use a framework that incorporates both performance and content domains, often in the form of a matrix. While this approach has the advantage of being thorough, a matrix that is too complex will pose challenges when it comes to developing test content for each cell of the matrix. Ultimately, credentialing bodies must choose an approach that makes the most sense for their program. This decision should be made in consultation with the job analysis panel and a rationale for the approach should be documented as part of the credentialing organization’s published validity statement.

What Should the Validation Survey Include?

A well-designed validation survey includes sufficient information to support the objectives of the job analysis study while being sufficiently concise and straightforward to result in a good response rate and high quality responses. Most commonly, credentialing programs will validate the job tasks that have been identified as essential for competent performance in a particular job or role. Some programs will also choose to validate the knowledge and/or skills required for successful performance of those tasks, and while there may be advantages to doing so, it often makes for an unreasonably long and complex survey that may result in a poor response rate.

Although numerous validation rating scales exist and may be used, the most widely-used scales include:

• Criticality: To what extent will harm result if a newly certified individual is unable to perform this task? Or, to what extent will harm result if a newly certifed individual does not possess this knowledge?
• Frequency: How frequently is this job task performed by a newly certifed individual? Or, how frequently does the newly certified individual rely on this knowledge?
• Need when frst certifed: Is a newly certifed individual expected to be able to perform this task? Or, is a newly certifed individual expected to possess this knowledge?

Precisely how many and which rating scales to use is a complex issue, and has been the subject of a considerable amount of research. The decision depends on a number of factors, and should be made in consultation with an experienced job analyst and a job analysis panel or advisory committee. However, a few general guidelines include:

• Avoid using more than two or three rating scales or the survey will likely become too long and complex.
• Use rating scales that make the most sense for the content of the survey and goals of the job analysis study, and pilot test it to ensure your assumptions are correct.
• Do not use redundant rating scales that will yield similar responses; for example, do not use a scale that measures criticality in conjunction with one that measures importance.

Can Anything Be Done to Improve the Response Rate?

Validation survey response rates generally range between 20 and 35 percent, but there can be a great deal of variability, depending on factors such as the characteristics of the sample and the length and the format of the survey, to name a few. Some research suggests that mailed surveys elicit better response rates than online surveys, but practical considerations (including data entry accuracy) often make online surveys more desirable. Providing guidance on how to improve response rates is challenging, simply because what is enticing for one population may not be enticing for another population, and the matter of survey response rates is rather complex.

In brief, it is important to consider what kind of incentive is most appealing to the population at hand. Incentives such as professional development credits or discounts for training, conferences, or other professional development opportunities tend to be more appealing than drawings for prizes or gift cards. Depending on how valuable a professional’s time is relative to the time required for the survey, offering a small, token incentive may not only be ineffective, it may even be perceived as insulting. In addition, sending reminders helps increase response rates substantially, but care must be taken to limit the number of reminders that are sent. Finally, careful communication about the importance of the survey from the credentialing body in newsletters, on websites, or in emails can be helpful in encouraging participation.

On-Demand Webinar

Job analysis, a starting point for your program Presented by Sarah Carroll, PhD, Castle Worldwide, Inc.

Job analysis is the cornerstone of any sound, legally defensible licensure or certification program. Hear an overview of different methods for conducting a job analysis, review best practices, and discuss the value and importance of job analysis for credentialing programs. Learn how job analysis is used to delineate the key tasks performed in a job or role, how to identify the knowledge and skills required to perform those tasks, and how to ensure that an examination reflects the critical elements of the job or role. Listen to a discussion of common challenges and decisions faced by organizations when conducting job analyses. Learn more here.