Needs Analysis for Assessment-Based Certificate Programs
Imagine that you have an excellent idea for an assessment-based certificate program. You may be the subject matter expert (SME), or a SME has asked for your help in creating the program. How do you verify the need for your program? You must go beyond your own opinion or your inner circle of SMEs’ opinions to ensure your program is structured in a defensible manner.
A proper needs analysis is imperative to the success of an assessment-based certificate program. Begin with the end in mind. A needs analysis helps identify the target audience, the gaps in current performance and the content necessary to support success. The needs analysis results help you create learning objectives and inform the program's design, including the content you will include and how you will assess the learning. The needs analysis results can also inform your marketing strategy, extending your reach to serve more individuals and organizations.
To determine success, define and align these various perspectives:
|
Organizational Outcomes |
What are the benefits of the certificate to organizations and the profession as a whole? Tip: Even if you are developing a program for individuals who serve a broad range of organizations, consider talking with representatives of these organizations who can describe their desired results and how your target audience could support those results. |
|
Ideal Performance |
What is the necessary performance level to achieve the organizational impact? Tip: The definition of ideal performance in the role will help you create learning objectives and assessments that are in alignment. |
|
Targeted Learners |
Who is the target audience that will need to demonstrate the ideal performance? Tip: You will likely discover that the target audience does not have the knowledge and skills necessary to support success; hence, the need for the assessment-based certificate program. |
Information about organizational outcomes, ideal performance and targeted learners provides details regarding the content to include and how to design the program, including the assessment. Be sure to confirm alignment among these perspectives as you create the assessment-based certificate program. Confirming alignment among these perspectives provides validation that your certificate program has the value it intends. If you do not have clarity, ask more questions to ensure that the program meets the needs of organizations, roles and individuals.
So, how do you get this information? Collect data from qualified individuals and stakeholders who are knowledgeable about the intention of the program. Also, make sure these individuals understand the “use, purpose, scope and stakes of the certificate program.”1 There are myriad ways to approach data collection, such as interviews (individual and group), work observation, surveys and extant data. These methods yield different information, so be intentional with your choice of approaches. Consider using a combination of these approaches to get the information you need.
- With individual interviews, identify the people you would like to interview and create questions intended to draw out sufficient information. For example, to define organizational outcomes, interview leaders within organizations that hire the participants in the assessment-based certificate program. Information about the organizational outcomes helps target content, activities and assessment approaches that support organizational success.
- With work observation, identify who and what you will observe to gather information about ideal performance. For example, you may select representatives of the target audience and request time to watch what they do to achieve success. Not only can an observation inform the learning objectives and content to include, but it can also reveal important details to include in the assessment as well. Observation may also help determine where the target audience is today and where they need to be — the performance gap.
- With group interviews or focus groups, identify a group of people who will collaborate to offer valuable information. For example, you may gather representatives of the target learners and their supervisors. These people may provide opinions about what they need to do on the job after the program and recommend approaches for the learning opportunity, such as methods and media that appeal to them and why. They can also provide input on effective assessment measures so that the assessment matches the program’s intention. Be sure to keep the group interviews or focus groups on track, using facilitation techniques that honor participants’ contributions while adhering to an efficient process.
- With surveys, identify the audience from whom you would like more information and create questions that are easy to answer. For example, question types may include rating scales to indicate importance and selected-response questions versus open-ended questions. We are interested in receiving data that are easy to analyze rather than taking a lot of time to make sense of written comments. Responses to survey questions may provide specific information such as demographic details of the target learners.
- With extant data, review material related to the content associated with the assessment-based certificate program, the organizations that will employ individuals who take the program, past performance of those in the positions related to the program and individuals who are doing the work. A published report from an outside entity would not replace a needs analysis. Examples include competency models, industry or field statistics, strategic plans, benchmarking reports, job descriptions, meeting minutes, certificate recipients’ feedback, and performance management processes and results.
IMPORTANT: Is Your Program High Stakes?High-stakes programs require a job analysis. As stated in the I.C.E. 1100 Standards for Assessment-Based Certificate Programs Terms and Definitions, “High stakes are associated with substantial potential for adverse consequences for the public, clients, patients, etc. Programs may also be high stakes when participants’ careers hinge on their success in accomplishing the intended learning outcomes of an assessment-based certificate program.” A job analysis is “any of several methods used singly or in combination to identify the performance domains and associated tasks, knowledge and/or skills relating to the purpose of the credential and providing the foundation for program validation. Also known as task or practice analysis, job-task analysis or role delineation study.” Applicants for I.C.E. 1100 accreditation for assessment-based certificate programs would be well served to closely assess their program’s stakes level before submission. If conducted, a job analysis is used to specify the domain of testable content on the assessment blueprint and determine the weighting (i.e., how many items) given to each section and/or objective. For lower stakes programs without a job analysis, the assessment blueprint and weighting should be derived from the needs analysis. Note that a traditional job analysis does not typically provide all of the data that a needs analysis does, and so additional steps will be necessary to determine the organizational outcomes, ideal performance and targeted learners. Regardless of whether the exam blueprint is derived from a job analysis or the needs analysis, it is imperative that the domain of testable content and the weightings are derived through one of these processes, and not from simply tradition or the opinions of only a few people. |
After collecting the data from the needs analysis, analyze it to develop or confirm the organizational outcomes, ideal job performance and needs of the target audience. Create a needs analysis report that outlines the information you uncovered and the way you obtained it. Include who was involved and how you involved them. Also, confirm the qualifications of those involved to ensure that you have consulted individuals and stakeholders that add value to the program. Use your needs analysis report to develop learning objectives, create the assessment and determine the content to include to serve the need or fill the gap identified.
What should you do if you already created an assessment-based certificate program and didn’t complete a needs analysis? In this case, the best course of action is to use these recommended needs analysis approaches to either confirm your program's current structure or undergo a restructuring based on the findings. For example, collect data about desired organizational outcomes from those organizations for whom your participants work. Ask qualified individuals and stakeholders questions about ideal performance and compare the responses to the content in your program. Create a learner profile based on input from qualified individuals and stakeholders. Compare the learner profile to the characteristics of those who have experienced your program. Document the results of the needs analysis and revise your program as necessary. Be prepared to revise the program. A cursory review labeled as a needs analysis will probably not be sufficient justification to leave the program unchanged.
Ultimately, the needs analysis should tell the story of why the assessment-based certificate program was created. It should provide critical information about goals, performance and learners. A comprehensive needs analysis, relying on input from qualified individuals and stakeholders, serves as a solid foundation for delivering a sound program, meeting the needs of individuals and organizations.