SAP DESIGN GUILD

Evaluating Accessibility

By Joelle Carignan, ACC, SAP Labs, Palo Alto – 02/18/2002 • updated 01/20/2004

Abstract

This article reviews some of the basic factors to consider when planning an accessibility evaluation for applications and web content. These factors relate to the definition of accessibility, the goal of an accessibility initiative, the planning of an evaluation, the inspection methods available, the use of evaluation software or "automated testing tools", and when to conduct usability testing.

 

Goals of an Accessibility Initiative

When faced with making an accessible application or web page, many people instantly wonder what this will mean in terms of design and development. Thus, the first question that comes to mind is, "What does accessibility mean?"

Defining Accessibility

Accessibility can be interpreted as a set of technical requirements that, if followed, will result in applications and web pages that are readable by assistive technologies, such as screen readers or screen magnifiers. Those technical requirements generally prescribe that an application should provide a logical keyboard navigation option and useful information about the interface, including status of an object, associated labels, and required actions. A common example is the need to include ALT text attributes for images and input fields so that screen readers can announce content that is more meaningful than the words "image" or "input field".

Accessibility can also be interpreted as meeting a set of functional performance criteria that, if achieved, will result in applications and web pages that are usable by people with disabilities. Those functional performance criteria generally ensure that people with different disabilities can complete a task and have a user experience comparable to that of users who do not have a disability.

Progressively, accessibility is defined as meeting both technical requirements and performance criteria. For instance, in the United States, Section 508 of the Rehabilitation Act contains subpart B on Technical Standards and subpart C on Functional Performance Criteria--recognizing that it is possible to meet all of the technical standards and still be faced with products that are not accessible. Conversely, situations may arise where guidelines are not strictly followed, but the resulting software will still be accessible.

Defining what accessibility means for an organization will serve as a baseline for the accessibility evaluation and affect how recommendations will be prioritized in an evaluation report.

Adopting Existing Standards

There are several standards and design guidelines on accessibility around the world. Therefore, before starting an evaluation, it is recommended to determine whether or not the organization is required to meet an existing set of standards. This will affect which evaluation software can be used to diagnose the accessibility of applications or web content. For instance, there are a few diagnostic tools that specifically assess compliance of Web applications with the requirements of Section 508. Most tools can be used for evaluating web pages and are based on the Web Content Accessibility Guidelines 1.0 (WCAG) from the W3C Web Accessibility Initiative (WAI). Still, other products now measure accessibility against both sets of standards.

Acknowledging Users' Profiles

An equally important factor to consider when defining the goals of an accessibility evaluation is the profile of user groups. The intention behind an accessibility evaluation might be to create products specifically aimed at computer users with disabilities, or it might be to meet one or more sets of basic accessibility requirements to ensure that everyone can use an application or a web page. The objective will influence design recommendations, since accessibility issues may force some usability improvement to different degrees. Also, while many people recognize that "good usability is good accessibility" [1], unfortunately, there are some cases where compromises are required. A typical example involves designing to increase movement efficiency and therefore, reducing space between elements, or designing for error prevention and having more space between targeted elements. Interfaces with more space between elements are easier for people with motor impairments to use. Such designs decrease the likelihood that these and other users will make targeting errors. However, it creates a dilemma when other design goals come into play, and having a clear view of accessibility-related goals will help to solve difficult design issues.

 

The Evaluation Plan

Once the basic goals are identified, it may be helpful to make a plan, which will define the evaluation process and methodology. A plan can be informal, or it can involve more preparation in the case of large projects that require:

 

Inspection Methods

Several usability inspection methods described by Nielson & Mack [3] can be applied in accessibility evaluations, including the following: heuristic evaluations, guideline reviews, standard inspections, cognitive walkthroughs, and formal usability inspections.

Heuristic Evaluation

In a heuristic evaluation, "specialists judge whether each dialog element conforms to established usability principles [3]". This method can be applied to an accessibility evaluation by using accessibility principles. An example of an accessibility principle is the need to provide redundant feedback (e.g., an error message which is indicated by both an auditory beep and a visual flash) [5]. In the case of web sites, a very simple set of usability heuristics that can apply to accessibility evaluations involves the questions: "Can I tell where I am?", "Where can I go?", and "What's on the current page?" [6].

Guideline Reviews

Guideline reviews consist of assessing an interface for conformance to a list of comprehensive guidelines. It can require a high degree of expertise to know about hundreds of usability guidelines [3] and, in the case of an accessibility evaluation, a specialist has to know about additional accessibility guidelines. However, there are well-known legal and industry standards for web accessibility that can be used as a basis (e.g., WCAG 1.0). Usability reports have also been published to establish design guidelines for people with disabilities (e.g., the Nielson Norman Group recently published a report based on a usability study of 44 users with disabilities) [4].

Standard Inspections

In a standard inspection, an expert checks an interface for compliance with a standard to diagnose the degree to which it is consistent with comparable systems on the market [3]. This method is typically used in accessibility evaluations of software applications and web pages, because legal and industry standards are available and very practical to use (e.g., Section 508 Standards and Checkpoints for WCAG 1.0). This type of inspection can be conducted in conjunction with the use of automated checks, which we will discuss in a moment.

Cognitive Walkthroughs

Cognitive walkthroughs simulate a user's problem-solving process. The purpose of this process is to check that the user's goals can be achieved and to identify and prioritize problems [3]. In accessibility evaluations, cognitive walkthroughs would involve the use of assistive technologies.

Usability Inspections

Formal usability inspections involve having 3 inspectors review an application individually and then conduct an inspection review meeting where the lists of the problems they found are merged [3]. Applying this technique to an accessibility evaluation requires some basic steps to simulate the user experiences of people with disabilities. For instance:

The choice of an evaluation method depends upon several factors, including: the goals of the accessibility initiative, the type of product being evaluated (e.g., an application or a web site), the state of a product (e.g., whether it is a prototype or an existing product which needs to be retrofitted), and the level of previous accessibility awareness (e.g., Does the application or web page meet some of the basic technical requirements that would allow it to work with assistive technologies?).

Apart from conducting some type of manual inspection, the evaluation can be facilitated by the use of automated checks. Some accessibility requirements are easy to check reliably with evaluation software, and several evaluation and repair tools are commercially available for web page accessibility diagnostics. In the case of applications, an option is to integrate automated checks into the development environment.

 

Accessibility Evaluation Software

The best-known evaluation software for web pages is Bobby by the Centre for Applied Special Technology. Up until 2001, Bobby was diagnosing pages based on WCAG 1.0. The latest release diagnoses Section 508 checkpoints compliance as well. A web-based version is available for free, and a downloadable version can be purchased. The downloadable version allows for diagnosis of HTML files saved on a hard drive, which is the only way to deal with dynamically generated URLs.

Similar tools include The Wave by Pennsylvania's Initiative on Assistive Technology, and Lift by UsableNet. Some other tools, such as InSight & InFocus by SSB Technologies, and ACCVerify/ACC Repair from HiSoftware offer interesting functionalities for a more substantive price. For example, InSight & InFocus offer the following features:

To perform the same tests manually, someone would have to:

Accessibility evaluation software is a convenient solution for retrofitting thousands of web pages. However, it is only a starting point for correcting accessibility violations. Interpreting the output requires knowledge of HTML and accessibility checkpoints. Moreover, some checkpoints require human judgments [2]. Evaluation software will not detect the following information:

Other limitations of accessibility software are that no tool can cover all languages, and they are not convenient to use when evaluating the accessibility of web-based applications. As mentioned earlier, some software can diagnose an HTML file if a dynamically generated page is saved on a hard drive manually. However, saving every page and running file-by-file diagnostics is a tedious process.

 

Usability Testing

After an initial accessibility evaluation has been conducted and some changes have been made, it is recommended that formal usability testing is conducted with a representative sample of users and a predefined set of tasks. This study could be done in an accessibility lab, which contains standard assistive technology products or in the workplace while using the equipment of recruited users.

As mentioned, in a formal usability test, users are asked to conduct a predefined set of tasks. The evaluation provides insight into problems through a combination of observing users and collecting their feedback. It can identify weaknesses in the interface, such as information design, structure, content, visual design, and interactive features. It can also measure subjective user satisfaction.

The use of a functional testing scenario is not recommended at this stage. The scenario should give the user a task to accomplish without providing detailed instructions on how to do it. It is also recommended that a formal usability testing protocol be followed.

Although end-user tests are the best way to gain information about the usability of an interface, a minimal amount of functional accessibility is required in the design before a usability test will be effective and worthwhile. Hence, adhering to accessibility guidelines from the start and incorporating user feedback as early as possible is the best way to ensure that design problems are discovered at a stage when they are easier to correct.

 

Conclusion

In evaluating accessibility, no size fits all. The methods chosen can be different depending on an organization's accessibility goals, the complexity of the product's design, the accessibility experience of the testing team, and the openness of development teams to make accessibility changes. This article has provided a review of existing usability inspection methods that can be used as a basis to improve the accessibility of both new and existing software and web content.

Our principle recommendation is to have an accessibility expert involved in the evaluation process or to have someone who specializes in accessibility on a product team. While evaluation software can facilitate the laborious process of retrofitting a large web site, some manual checks as well as knowledge of issues faced by users with different types of disabilities will still be required. In cases where it is necessary to retrofit complex web-based applications, conducting a usability inspection is the most effective method to use in the initial evaluation phase. Also, a formal usability test may help designers and developers in making some important decisions based on mental models, expectations, and user experiences of people with disabilities. Hence, the best time to conduct such testing is during the design phase of a new product, when there is more opportunity to introduce design changes and improve its overall usability.

 

Acknowledgements

The author would like to sincerely thank Dena Shumila, who greatly contributed to the clarity of this article, and Linda Bortolus for her review.

 

References

  1. Frontend Usability InfoCenter. Available online at www.frontend.com/accessibility_paper.html
  2. Human Judgments Required for accurate Section 508 Evaluations (no longer available online, astro.temple.edu/~kasday/talks/wave508/judge.html)
  3. Nielson, J., Mack, R., 1994, Usability Inspection Methods, Wiley
  4. Neilson Norman Group, 2001, Beyond ALT Text: Making the Web Easy to Use for Users with Disabilities.
  5. TIA Access, 1996, Resource Guide for Accessible Design of Consumer Electronics. Available online at www.tiaonline.org/access/guide.html
  6. Veen, J., 2001, The Art and Science of Web Design, New Riders, p.48

 

top top