Accessible Design: Problems and Solutions
A Literature Review to Support the ITTATC Needs Assessment

Section 2: Literature Review Results

M: User-Centered Design

 Introduction



The three components of the user-centered human factors design process are analysis, design, and evaluation. These three components are interrelated and must be executed in an iterative manner. Analysis forms the basis of design requirements, thus, design reflects requirements identified through analysis. Evaluation produces validation of design. Evaluation results feed back into analysis and design. Subsequent design reflects input from analysis and evaluation.

Analysis involves identification and examination of the missions, functions, and tasks that the user must perform. Analysis may be thought of as a formal, thorough, thinking-through of the requirements that a system must meet.

Mission (or scenario) analysis is used to identify system-level (mission-level) requirements that impact human performance requirements. Mission analysis includes identification of the user population of interest, with focus on identification of characteristics of that population that will translate into design requirements. For example, anthropometric properties of the population will drive selection and arrangement of hardware components. The presence of users for which English is a second language (or who do not have any proficiency in English) will drive requirements for software interfaces and documentation. The presence of users with various impairments (e.g., visual impairment, mobility impairment, hearing impairment) will drive the selection of interface modes and/or provision of alternative methods of access. Mission analysis includes the development of explicit design reference scenarios that define the scope of functionality required and illustrate that functionality in operation. The scenarios should include normal operations as well as unusual conditions such as failure modes or environmental extremes.

Function analysis is used to identify the proper roles for human and machine components of the system, and thereby further identify machine-related requirements that translate into human performance requirements. Function analysis involves decisions about what functions to automate (partially or fully), and how the human will interact with automated functions. The decision to not automate a function, or to only partially automate it, is a decision to require the human to perform the function in part or in whole. The two major activities in function analysis are function identification (identifying all the functions that the system must perform to meet its mission requirements) and function allocation (deciding the level of automation for each function.)

Task analysis is used to identify the specific behaviors that will be required of the human operators, and to estimate operator workload and error rates on the various tasks. Task analysis outputs form the basis for design by establishing the complete set of information requirements, and by creating descriptions of temporal relationships (operational sequences) that must be supported by the design. The task analysis outputs also drive the design by identifying the task sequences that must be streamlined by the design to alleviate workload problems, and the errors that must be prevented or mitigated.

Design involves the identification and development of specific techniques to represent, and thereby transmit, information to and from the human operator. Each information element must be represented in the design. The methods of representing each information element, and of grouping and coordinating these elements, are the essential components of design. Selection of the information representation methods is based on the following factors: Note that information representation involves provision of information to the human (i.e., display), and of receipt of information from the human (i.e., control.) The selection of display techniques and control mechanisms should take into account the capabilities, limitations, and past experiences of the population of intended users.

Evaluation involves the creation of task performance conditions in which representatives of the intended population of users can perform representative tasks using the information elements produced by design. The performance of the users can be measured and evaluated, and the subjective opinions and preferences of the users can be obtained. Although there is a small role for examination of static illustrations and descriptions of information elements in the evaluation process, relatively little value is obtained from such evaluations. Much more value is obtained from creating task performance conditions by using interactive prototypes and dynamic simulation.

There are two primary types of evaluations used in the development and validation of controls and displays. The first type is formative evaluation, which refers to evaluations that solicit qualitative inputs from evaluators. These inputs may be in the form of suggestions for improvement, ideas on alternatives, and expressions of preferences. The second type is summative evaluation, which refers to evaluation procedures that generate quantitative performance measurements and pass/fail outcomes.

Formative evaluation is an important part of the design process. Formative evaluations provide an opportunity to consider two or more design alternatives and to identify potential enhancements to an existing design. Formative evaluations may be conducted using incomplete representations of the design, and may examine only a subset of the tasks of interest. As the design process progresses, the formative evaluations may become increasingly more complete and the supporting simulation may become of increasingly higher fidelity.

It is very important to begin formative evaluation in the early stages of design, before too many design decisions are made. Formative evaluation should continue, iteratively, until the final design of the operator interfaces is established. Each iteration should address progressively more detailed issues (unless a second iteration is needed on a particular problem discovered in a previous iteration).

Summative evaluation is the process by which formal pass/fail evaluative judgments are obtained. It is sometimes solely conducted as acceptance testing. It can be structured so that it produces a single pass/fail judgment on the overall design of the controls and displays. Usually, however, it will be more constructive to render pass/fail judgments on various design features (or modes) individually.

Simulation is an important part of both types of evaluations. Fidelity of simulator is a more strict concern in summative evaluation than in formative evaluation. In general, simulation fidelity requirements are low in the beginning, for early formative evaluations, and become progressively more stringent in formative evaluation. Fidelity requirements for summative evaluation are relatively high - at least as high as the final level of fidelity used in formative evaluation.



 M-1. What are the current best practices in user-centered design?







 M-2. What user-centered design tools are available to designers?






 M-3. What is the state-of-the-art in analysis of information requirements and user needs?



There are several ways for the interested designer to gather data regarding user needs and information requirements, including surveys, questionnaires, panel studies, brainstorming sessions, structured interviews, observation, diaries, attribution analysis, and focus groups (KADO, n.d.). For carefully collected data from many individuals, statistical analyses such as factor analysis, cluster analysis, and scaling analysis may be used to identify trends and patterns underlying user behaviors and responses (KADO, n.d.). The nature of the data and the questions they are expected to answer should be carefully considered before choosing a particular analysis technique (Goode, 2001). The choice of the information gathering method depends on several factors, such as the geographic distance that must be covered (questionnaires or other at-home methods would be preferable over focus groups) and the degree to which the characteristics of the population of interest matches those of the designers (brainstorming might be more cost-effective when characteristics are similar).

Conducting focus groups appears to be a common method for gathering extensive information about user needs (e.g., Pacific Bell, 1996; Perlman, 1993). For successfully conducting focus groups several factors must be addressed, such as the quality of the moderator, the size of the group, and the social/cultural nature of the group, which can have a profound effect on the outcome of the effort (Garson, 2000). Additionally, when conducting focus groups with individuals with disabilities, the specific needs of these individuals must be recognized (IBM, n.d.). Finally, supplemental methods have been recommended for eliciting tacit knowledge where current needs assessment methods focus mainly on eliciting explicit knowledge, which may or may not be accurate or complete (Ko, 1999; Nielson, 1997).



 M-4. What is the state-of-the-art in rapid prototyping and iterative design?



There are several types of prototypes available for the purposes of conducting design evaluation and guiding iterative design (Greenberg, 2000; Lamancusa, 2000). Prototypes vary in the degree to which they are similar in functionality and appearance to the actual product design. High-fidelity prototypes most precisely represent the actual product design, using computers to simulate a great deal of the product's functionality and even actually performing some tasks. In contrast, low-fidelity prototypes, including sketches and storyboards, are paper-based, and cannot simulate product functions but instead represent general design appearance and layout. Medium-fidelity prototypes are not necessarily paper-based, but do not have full capacity to simulate product functions either.

Functionality in medium-fidelity prototypes is limited in three ways, through vertical prototyping, horizontal prototyping, or scenarios (Mankoff, n.d.; Greenberg, 2000). Vertical prototyping allows a particular design function to be evaluated in depth. In contrast, horizontal prototyping allows only surface evaluation of multiple design functions. In scenarios, in which the user must use the prototype to complete a particular function, both breadth and depth of design functionality are limited.

The type of prototype (high-, medium-, or low-fidelity) selected depends on several factors, including the amount of time available to construct the prototype (high-fidelity prototypes take longer to build), monetary constraints (high-fidelity prototypes cost more to make), and the degree to which the design idea is already developed (low-fidelity prototypes are better for less well-developed design ideas; Mankoff, n.d.; Greenberg, 2000). Other considerations include the degree to which quantitative vs. qualitative data collection is desired, the amount of specialized equipment or personnel available, the degree to which independent user exploration of design functionality is desired, and the degree to which controlled study is desired.


 M-5. What is the current thinking in the field of error analysis?



A critical finding in the field of error analysis is that characteristics of system design may be more responsible for errors than the operators themselves (Moray, 1994). Another critical finding is that organizational pressures may serve to repress error reporting, which could lead to improved technology design (Linda, et al, 2000). Together, findings such as these indicate that the errors occurring within evaluations of system designs should be taken seriously as potential problems in the system. Further, if usability testing occurs within the organization in which the tested product is being developed, test subjects' desire to minimize reports of errors to avoid criticism must be considered. With regards to testing for accessibility, it has been demonstrated that users simulating disability produce similar errors resulting from design defects, particularly gross design defects, to those errors of people who are actually disabled (Law & Vanderheiden, 1999, 2000).


 M-6. What is the state-of-the-art in human performance testing and evaluation?



Performance testing and evaluation have changed a great deal as the clear demarcation between human and machine performance has blurred (Meister, 1996). In addition, as the human role moves toward supervising and monitoring, performance is not necessarily reflected in an observable behavior, which has implications for how performance is assessed (Meister, 1996). With regards to assessing the performance effects of system design, however, usability testing is still currently valued and practiced (Conyer, 1995; Lee, 1999; Nielsen, 1997). There exist several other methods for examining human-machine interaction through usability testing, such as heuristic evaluation (Nielsen, n.d.) design reference scenarios (Folds, 1998). Each has been shown to be effective, though careful consideration of verbal protocol and rating results must be made (Ericsson & Simon, 1984; Goode, 2001).


 M-7. What is the current thinking in the field of usability testing and evaluation?



As evidenced by the volume of documentation on methods and procedures (e.g., Folds, 1998, 2000; Law, et al, 2000; Nielsen, 1997, n.d.), usability testing is currently an effective approach to conducting design evaluation, although heuristic evaluation has also been shown to be effective (Nielsen, n.d.). In addition, several tools (e.g., WebCAT) exist, however, to aid designers in creating prototypes for usability testing and conducting evaluations (Meyers, 1997; NIST, n.d.). There are, however, special needs that must be considered when conducting usability tests with participants with disabilities (Law, et al, 2000), and the results of verbal protocols and ratings must be carefully evaluated in order to prevent misinterpretation of the data (Ericsson & Simon, 1984; Goode, 2001; Nielsen, n.d.). Unfortunately, recent work indicates that usability testing (in fact, usability engineering in general) is not a common practice among industrial developers (Nielsen, n.d.). Misperceptions of the cost and time associated with usability engineering appear to be the major roadblock to adopting the practice of user-centered design (Nielsen, n.d.).