Innovative Assessment Strategies: Are They Accessible to Test-takers With Disabilities?

The assessment industry has undergone several exciting changes over the past decade. One of the most significant is the increase in utilization of “innovative item types,” or technology-enhanced items.  Rather than relying entirely on multiple-choice items and simple graphs and charts, some test publishers have opted to develop interactive test items that may better measure critical skills while being more realistic to test-takers.

Early in this revolution, “innovative” meant items that included “drag-and-drop” (e.g. move graphics to targets on the screen), “hotspot” items (e.g. click on the graphic) and items that included reference materials or other supplementary information that could be shown via tabs on the screen. Newer innovative item types include video-based items (e.g., watching a practitioner-client interaction), video simulations and items including multiple additional charts, graphs, reference materials and case-study materials. While all these changes and enhancements may be positive developments from the perspective of test-taker engagement, they present new barriers to test-takers with disabilities. How can we ensure tests are still accessible to everyone?

What Is Accessibility and Why Does It Matter?

From a legal perspective, accessibility is about ensuring the test measures what it is intended to and is not impacted by a person’s disability. This perspective has been supported in multiple court cases (Mattinson, 2012) and also makes sense from a measurement perspective, as it is important to ensure the test measures what it purports to, without bias or other unintended effects (Reynolds, Altman, & Allen, 2021).

There may also be valid ethical and business-related reasons for ensuring the accessibility of tests. For example, there may be qualified test-takers who would pass the test and make a positive contribution to their field, if the test is accessible. While many businesses have focused on improving the diversity, equity and inclusivity of their goods, services and hiring practices, disability inclusion is — or should be — part of this conversation. Accessibility as part of a larger initiative to improve diversity makes good business sense; incorporating accessibility into products, services, devices and interactions is good for customers, partners and employees.

How Does Accessibility Relate to Universal Design?

The Universal Design for Learning (UDL) approach values diversity and inclusivity (Kearney, 2022), and is achieved through the intentional design of an inclusive curriculum or assessment in order to eliminate or reduce barriers. The origins of universal design come from building and public works construction projects such as mandating “curb cuts” and ramps to allow wheelchair riders to safely access sidewalks and buildings. As it turns out, these construction enhancements also benefitted non-disabled individuals such as travellers pulling wheeled luggage and pushing baby carriages. In the academic space, UDL was initially proposed as a means for including students with disabilities in the general education classroom, but is now better understood as a general education initiative that improves access and outcomes for all learners (Ralabate, 2011). 

Current concepts of universal design go far beyond the classroom and sidewalks, to include designing products, services and environments to be used by all people, to the greatest extent possible, without the need for requesting adaptation or customization. Many computers and smart devices today include color and contrast options (e.g. changing black text on white background to white-on-black or black-on-yellow), various visibility enhancements (e.g. enlarged cursors or fonts or color filters), audio enhancements (e.g. live captioning, reducing ambient noise, louder ringtones and alerts). Thus, it is not surprising that test-takers — with or without disabilities, are requesting some of these same features when taking exams.

From a practical perspective, we know many accessibility features of computer-based tests enhance usability for non-disabled test-takers as well, which is the essence of universal design (Rose, 2000). For example, some test design includes “self-service” magnification so any test-taker can increase the font size or magnify the test content to whatever degree is desired without needing to request formal accommodations. As more high-stakes exams are provided in digital formats, the industry should anticipate more tests will be available with self-service accessibility enhancements. 

Accessible Testing: Considerations and Strategies

Consider this scenario: An eligible applicant with a significant vision disability requests accommodations on your exam. This applicant requires screen reading software, text-to-speech software, computer dictation software, adaptive devices or other technologies for the computer-based test.

If your exam contains images, charts, graphs or video content, you recognize that making your test accessible for the test-taker with low vision will be challenging. You may engage a psychometrician to consider alternate item or form options that purportedly measure the same constructs. If these strategies are not possible, you may need to work with an accessibility expert and content experts to tag images with alt-text. This text, not visible to the test-taker, describes the images so a screen reading software program can “read” the image to the test-taker.

Test sponsors and test content experts may be alarmed by the idea of substituting images or describing them. How can we describe an image without giving away the answer? Moreover, if this type of item is fundamental to the test and skill required for a job holder with this credential, how can we eliminate or substitute these items without undermining the credential itself? Balancing these potentially-competing goals — full accessibility of the exam or test-takers with disabilities, while also protecting the validity and integrity of the exam and the public-protection mandate of the organization — can seem almost impossible at times.

Other tests are more complex and include items such as case studies. These test items may involve a stem — an initial question or clinical vignette — and various tables, charts and reference materials, which could be shown as tabs to different screens. All of this is available to test-takers who can see the tabs and click on the appropriate places to access the materials. For test-takers without vision, none of these ancillary materials are accessible. Moreover, even if a sighted proctor or assistant is available to help the test-taker navigate through these various tabs and reference materials, how will they keep all this information in mind? The working memory load of such a test item would be overwhelming, presenting an impossible barrier.

The newest alternative test item-types use videos, simulations and video-game-like content. Historically, many exams in the health professions had two components: a written exam and an Objective Structured Clinical Exam (OSCE), often with live, standardized patients. For example, an optometry exam may include assessment of a candidate’s ability to conduct eye exams using the actual equipment and live patients. Newer OSCE-like assessments may include standardized simulations of these interactions, thereby eliminating the need for live, on-site patients. Likewise, other hands-on tasks in health-related fields, construction trades, or real estate and appraisal fields could use standardized video simulations rather than live, on-site performance tests. On the one hand, these performance-based tests and test-items may be most representative of the types of actual job tasks that will be required of the credential holder. On the other, they will be challenging to make accessible to some test-takers with disabilities.

Accommodation and accessibility requests also involve individuals with cognitive impairments such as learning or attention disorders, or mental health conditions. Though less apparent, individuals who have disabilities in these areas may also struggle with barriers on exams. For example, some test-takers with attention disorders may find barriers with exams that have numerous item-types and busy content. An individual experiencing post-traumatic stress disorder (PTSD) may require extra breaks during the exam. Each time a break is completed, the individual may have to essentially “start over again” if the test design supports complex and multi-faceted content. Simply providing this test-taker with extra time to take the test may not reduce the barriers faced. A different accessibility strategy, such as breaking the test into several parts and taking it over the course of different days, may be an option.

Some test sponsors have approached accessibility by focusing exclusively on the Web Content Accessibility Guidelines (currently, WCAG 2.1), published by the Web Accessibility Initiative of the World Wide Web Consortium, the main international standards organization for the internet. These guidelines offer extensive technical standards. Governmental agencies have also published guidelines for accessibility of websites and services, including Section 508 of the US Rehabilitation Act, and the very comprehensive Accessibility for Ontarians with Disabilities Act (Canada).

Tests may be technically compliant with WCAG (or other governmental standards), but practically speaking, may not be fully accessible to test-takers with various disabilities. Publishing a fully accessible test usually means going beyond simply ensuring legal or technical compliance, but also investigating functional accessibility. This can mean having accessibility experts test your content with software such as ZoomText, JAWS and Mac’s Voiceover programs, and having people with disabilities taking your test (or a sample test) to look for accessibility difficulties from a practical standpoint. Even with extra measures taken, there is susceptibility that end users perceive the test as not fully accessible. The key to ensuring full accessibility, especially with newer, innovative item-types, is to involve not only accessibility technical compliance experts, but also disability experts, experts in the use of assistive technology software tools and end-users with disabilities. 

Psychometric Considerations

Test sponsors typically work with psychometricians to develop assessment models that ensure the standardization of the test, validity of the test and test items and appropriate interpretation of scores. To ensure all test takers’ experiences provide for equity and comparability, test sponsors are tasked with focusing on fairness as a validity issue throughout test design and test development activities.

It’s important that a test measures the factors that are intended to be, rather than construct irrelevant factors that introduce bias, such as race, gender or disability. Psychometricians can help guide the test development processes to minimize these irrelevant factors. Training and educating the key players in content and test design (test developers, subject matter experts, editors) to be aware and sensitive of potential barriers that could impart unfair testing conditions to certain populations is a simply executed, proactive measure all test sponsors should implement.

Thinking about the psychometric implications of innovative test design and newer item types, maintaining open dialogue between product owners, test developers, psychometricians and accessibility experts will support a healthy landscape for productivity. Occasionally, test adaptation is necessary to maximize accessibility to all members of the testing population, and psychometricians can help ensure comparable outcomes that represent the intended construct.

Test Delivery and Security Considerations

In addition to ensuring equal access to candidates with disabilities, and ensuring a psychometrically sound test, test sponsors need to balance potential security risks and operational factors. When test sponsors think about innovative item types and the possible accommodations or modifications that might be needed to make the tests accessible, they naturally think about security risks. For example, some third-party software could enable or require internet access. Likewise, assistive personnel, if not properly trained and vetted, could introduce exam security risks such as item harvesting.

While there are many excellent software packages available to assist candidates with disabilities, such as screen reading and speech-to-text programs, some delivery platforms cannot accommodate all options. For example, a candidate with low vision might be most familiar with the screen of a particular reading software program, but the test-delivery vendor’s delivery platform might not be compatible with the same program. In those cases, an alternative software program the vendor can support and lock down would be appropriate to ensure accessibility and support security protocol.

In any case, it is incumbent upon the test sponsor and test-delivery vendor to fully investigate the technical, operational and security implications of such software, while staying coordinated on design and what can be accommodated. Two common concerns are 1) the lack of comprehensive quality assurance review in the exam development or delivery process of any modifications that are made for candidates who need assistive technology or significant modifications to the test, and 2) TP approving accommodations the vendor is not ready to support, resulting candidate frustration and delays.

Takeaways

Test sponsors must understand the importance of accessibility and incorporate accessible design principles early in the test development process. When developing a new test, test sponsors should think carefully about how it will be accessible to candidates with a range of different disabilities, not only visual conditions. A valuable strategy is to involve experts from different disciplines into the conversations about test development. This could include accessibility and disability experts, assistive technology experts and actual end users with disabilities. It is always easier to consider accessibility when developing a new test, as opposed to trying to retrofit an existing test to be accessible. As test developers plan for innovative assessment strategies, it’s important to keep in mind the fundamental tenants of accommodations: equal access, fairness, test taker experience and security.