B: Assessment Methodologies
- Introduction
- B-1. How do we determine when the requirements have been met?
- B-2. How do we determine if a product is truly accessible?
Introduction
Current assessment approaches have used summative evaluation, most frequently some form of checklist pass/fail rating with non-disabled evaluators simulating the disability in terms of sensory, cognitive and physical performance during the screening. Indications of more maturing methodologies are possible as guidelines for accessible design principles are being developed and other approaches such as performance-based evaluation and focus group techniques are being considered. Adoption of user-centered design and evaluation approaches could contribute further methods suitable for formative testing of evolving designs (See 3.15 in the bibliography for a discussion of these methods). Testing methodologies can be automated; they can involve users without disabilities simulating disabilities or tests with actual disabled users; they may involve simple inspection or evaluation of compatibility with assistive technologies (ATIA, n.d.).
There are a number of checklists for evaluating whether requirements have been met. IBM (n.d.) provides checklists for hardware, Java, software, and web accessibility. Checklists, guidelines, and techniques for ensuring accessibility of web content are also provided by WAI (n.d). WAI also has a working group dedicated to evaluation and repair tools. Checklist evaluations prove to be problematic since they rely on a well-derived set of guidelines or performance expectations.
Law and Vanderheiden (1999) propose inexpensive screening tests to apply to product design for accessibility evaluation. These tests are designed to impose functional limitations on individuals who may not actually have an impairment. Sensory screening tests include use without vision, use with low vision, use without the ability to hear, and use with reduced ability to hear. Physical screening tests include use with one hand, use with one finger, use with a mouthstick, use with a low manipulation capability, and use with a tremor/poor coordination. Cognitive screening tests include use without the ability to read and use with limited cognitive capability.
Telecommunications Industry Association (TIA) (1996) provides a guidebook that discusses federal initiatives necessitating accessible design, general characteristics of the population of people with disabilities, industrial response to the call for accessible design, general principles of accessible design, tools and tips for design evaluation, and general guidelines for accessible design. Montoya-Weiss, et al (n.d.) provide a performance measure evaluation survey for product testing. Focus group discussions provide another way to determine accessibility, but they must be used with caution, as discussed by Nielsen (1997).
B-1. How do we determine when the requirements have been met?
There are several ways to evaluate product design, including heuristic evaluation, usability testing, direct observation, and focus groups (TIA Access, 1996). Further, evaluation and testing should occur throughout the design process (Nielson, 1997; TIA Access, 1996). There are several online resources for conducting heuristic evaluation using checklists (e.g., IBM, n.d.; TIA Access, 1996, WAI, n.d.). It is important to note that one barrier to producing compliant products has been the exclusion of individuals with disabilities from the target market. Evaluations of compliance should include these individuals through focus groups or should include simulations of disability (e.g., Law & Vanderheiden, 2000). Nielson (1997) notes however, that focus groups must only be supplemental to usability testing, as inferences drawn from focus group responses are sometimes incorrect. Further, special testing considerations should be noted when conducting usability tests with persons with disabilities (Law & Vanderheiden, 1999). ATIA (n.d.) provides some guidelines for determining compatibility with and designing for compatibility with assistive technologies. Finally, some measure of accessibility can be inferred from design awards given by disability advocate groups, such as the RNIB and AFB (Montoya-Weiss, et al, 2000.).
ATIA. (n.d.). AT-IT Compatibility Guidelines, Version 1.05. Retrieved September 18, 2003, from the World Wide Web: http://www.atia.org/AT_Compatibility_Guidelines_v1.05.pdf
IBM. (n.d.). Hardware accessibility. Retrieved January 2, 2001, from the World Wide Web (link updated September 22, 2003):
Click here to go to this resource. (http://www.ibm.com/able/guidelines/hardware/
accesshardware.html)
IBM. (n.d.). Java accessibility. Retrieved January 2, 2001, from the World Wide Web (link updated September 22, 2003): http://www.ibm.com/able/guidelines/java/accessjava.html
IBM. (n.d.). Software accessibility. Retrieved January 2, 2001, from the World Wide Web (link updated September 22, 2003): http://www.ibm.com/able/guidelines/software/accesssoftware.html
IBM. (n.d.). Web accessibility. Retrieved January 2, 2001, from the World Wide Web (link updated September 22, 2003): http://www.ibm.com/able/guidelines/web/accessweb.html
Law, C. M., & Vanderheiden, G. C. (1999). Tests for screening product designs prior to user testing by people with functional limitations. Presented at the Human Factors and Ergonomics Society Conference.
Law, C. M., & Vanderheiden, G. C. (2000). Reducing sample sizes when user testing with people who have, and how are simulating disabilities - experiences with blindness and public information kiosks. Presented at the joint conference of the International Ergonomics Association and Human Factors and Ergonomics Society.
TIA Access. (1996, November). Resource guide for accessible design of consumer electronics. Electronic Industries Alliance/Electronic Industries Foundation. Retrieved January 9, 2001, from the World Wide Web: http://www.tiaonline.org/access/guide.html
WAI. (n.d.). Retrieved September 18, 2003, from the World Wide Web: http://www.w3.org/WAI/
B-2. How do we determine if a product is truly accessible?
Montoya-Weiss, M., Mueller, J., & Story, M. (n.d.). Measuring universal design. Proceedings, Designing for the 21st Century II: An International Conference on Universal Design. Boston: Adaptive Environments Center.
Nielsen, J. (1997). The use and misuse of focus groups. Retrieved January 2, 2001 from the World Wide Web: http://www.useit.com/papers/focusgroups.html