Go back to Section 2: Literature Review Results

Go forward to Section 3: Annotated Bibliography

Print this page.

• Introduction
  
• M-1. What are the current best practices in user-centered design?
  
• M-2. What user-centered design tools are available to designers?
  
• M-3. What is the state-of-the-art in analysis of information requirements and user needs?
  
• M-4. What is the state-of-the-art in rapid prototyping and iterative design?
  
• M-5. What is the current thinking in the field of error analysis?
  
• M-6. What is the state-of-the-art in human performance testing and evaluation?
  
• M-7. What is the current thinking in the field of usability testing and evaluation?
  

 Introduction

The three components of the user-centered human factors design process are analysis, design, and evaluation. These three components are interrelated and must be executed in an iterative manner. Analysis forms the basis of design requirements, thus, design reflects requirements identified through analysis. Evaluation produces validation of design. Evaluation results feed back into analysis and design. Subsequent design reflects input from analysis and evaluation.

Analysis involves identification and examination of the missions, functions, and tasks that the user must perform. Analysis may be thought of as a formal, thorough, thinking-through of the requirements that a system must meet.

Mission (or scenario) analysis is used to identify system-level (mission-level) requirements that impact human performance requirements. Mission analysis includes identification of the user population of interest, with focus on identification of characteristics of that population that will translate into design requirements. For example, anthropometric properties of the population will drive selection and arrangement of hardware components. The presence of users for which English is a second language (or who do not have any proficiency in English) will drive requirements for software interfaces and documentation. The presence of users with various impairments (e.g., visual impairment, mobility impairment, hearing impairment) will drive the selection of interface modes and/or provision of alternative methods of access. Mission analysis includes the development of explicit design reference scenarios that define the scope of functionality required and illustrate that functionality in operation. The scenarios should include normal operations as well as unusual conditions such as failure modes or environmental extremes.

Function analysis is used to identify the proper roles for human and machine components of the system, and thereby further identify machine-related requirements that translate into human performance requirements. Function analysis involves decisions about what functions to automate (partially or fully), and how the human will interact with automated functions. The decision to not automate a function, or to only partially automate it, is a decision to require the human to perform the function in part or in whole. The two major activities in function analysis are function identification (identifying all the functions that the system must perform to meet its mission requirements) and function allocation (deciding the level of automation for each function.)

Task analysis is used to identify the specific behaviors that will be required of the human operators, and to estimate operator workload and error rates on the various tasks. Task analysis outputs form the basis for design by establishing the complete set of information requirements, and by creating descriptions of temporal relationships (operational sequences) that must be supported by the design. The task analysis outputs also drive the design by identifying the task sequences that must be streamlined by the design to alleviate workload problems, and the errors that must be prevented or mitigated.

Design involves the identification and development of specific techniques to represent, and thereby transmit, information to and from the human operator. Each information element must be represented in the design. The methods of representing each information element, and of grouping and coordinating these elements, are the essential components of design. Selection of the information representation methods is based on the following factors:

• Methods and conventions used in predecessor systems or comparable systems that are familiar to the population of intended users.
  
• Known capabilities and limitations of humans in the population of intended users to respond to attributes of physical stimuli such as color, brightness, shape, loudness, directionality, etc.
  
• Known cognitive capacities and limitations in the population of intended users, for example, short-term memory capacity or restrictions on literacy.
  
• Established design principles for information representation, such as control-display compatibility.
  
• Lessons learned in related systems regarding good and bad designs, especially known pitfalls associated with significant human errors.

Note that information representation involves provision of information to the human (i.e., display), and of receipt of information from the human (i.e., control.) The selection of display techniques and control mechanisms should take into account the capabilities, limitations, and past experiences of the population of intended users.

Evaluation involves the creation of task performance conditions in which representatives of the intended population of users can perform representative tasks using the information elements produced by design. The performance of the users can be measured and evaluated, and the subjective opinions and preferences of the users can be obtained. Although there is a small role for examination of static illustrations and descriptions of information elements in the evaluation process, relatively little value is obtained from such evaluations. Much more value is obtained from creating task performance conditions by using interactive prototypes and dynamic simulation.

There are two primary types of evaluations used in the development and validation of controls and displays. The first type is formative evaluation, which refers to evaluations that solicit qualitative inputs from evaluators. These inputs may be in the form of suggestions for improvement, ideas on alternatives, and expressions of preferences. The second type is summative evaluation, which refers to evaluation procedures that generate quantitative performance measurements and pass/fail outcomes.

Formative evaluation is an important part of the design process. Formative evaluations provide an opportunity to consider two or more design alternatives and to identify potential enhancements to an existing design. Formative evaluations may be conducted using incomplete representations of the design, and may examine only a subset of the tasks of interest. As the design process progresses, the formative evaluations may become increasingly more complete and the supporting simulation may become of increasingly higher fidelity.

It is very important to begin formative evaluation in the early stages of design, before too many design decisions are made. Formative evaluation should continue, iteratively, until the final design of the operator interfaces is established. Each iteration should address progressively more detailed issues (unless a second iteration is needed on a particular problem discovered in a previous iteration).

Summative evaluation is the process by which formal pass/fail evaluative judgments are obtained. It is sometimes solely conducted as acceptance testing. It can be structured so that it produces a single pass/fail judgment on the overall design of the controls and displays. Usually, however, it will be more constructive to render pass/fail judgments on various design features (or modes) individually.

Simulation is an important part of both types of evaluations. Fidelity of simulator is a more strict concern in summative evaluation than in formative evaluation. In general, simulation fidelity requirements are low in the beginning, for early formative evaluations, and become progressively more stringent in formative evaluation. Fidelity requirements for summative evaluation are relatively high - at least as high as the final level of fidelity used in formative evaluation.

Go back to the top of this page.

 M-1. What are the current best practices in user-centered design?

American Foundation for the Blind. (2000). AFB Presents 2000 Access Awards.
Retrieved January 9, 2001, from the World Wide Web: http://www.afb.org/afbnews_2000accessawards_summer.asp

Bradley, J. (1998). Human-computer interaction and the growing role of social context. ASIS Bulletin, American Society for Information Science. Retrieved January 25, 2001 from the World Wide Web: http://www.asis.org/Bulletin/Feb-98/Bradley.html

Davied, D. J., Fisher, J. E., Arnold, M., & Johnsen, D. (1999). Usage profiles of users of interactive communication technology: An empirical investigation into the significance of selected individual attributes. Retrieved January 25, 2001 from the World Wide Web: http://www.bc.edu/bc_org/avp/law/st_org/iptf/commentary/content/1999060510.html

Fichman, R. G., & Kemerer, C. F. (1995). The illusory diffusion of innovation: An examination of assimilation gaps.Retrieved January 25, 2001 from the World Wide Web: http://www.pitt.edu/~ckemerer/illusory.htm

Folds, D. J. (2000). Three crucial components of an aircrew-centered design process (AIAA-2000-1061). American Institute of Aeronautics and Astronautics.

Folds, D. J. (1998). Use of design reference scenarios in a user-centered design process (AIAA-98-1039). American Institute of Aeronautics and Astronautics.

Hackos, J. T., & Redish, J. C. (1998). User and task analysis for interface design. New York: John Wiley & Sons.

Hoffman, R. R., Crandall, B., Shadbolt, N. (1998). Use of the Critical Decision Method to Elicit Expert Knowledge: A Case Study in the Methodology of Cognitive Task Analysis. Human Factors, 40 (2), 254-276.

Kelley, J. F., Spraragen, S. L., Jones, L., Greene, S. L., & Boies, S. (1996). Extending user-centered methods beyond interface design to functional definition. Proceedings of the Human Factors and Ergonomics Society Annual Meeting . Santa Monica, CA: Human Factors and Ergonomics Society. Retrieved January 18, 2001 from the World Wide Web: http://www.musicman.net/itshfs69.html

Kelly, M. J. (1999). Principles and methods of user-centered design. In Preliminary Human Factors Guidelines for Traffic Management Centers (FHWA-JPO-99-042). Mclean, VA: Federal Highway Administration.

Lund, A. M., & Tschirgi, J. E. Designing for people: Integrating human factors into the product realization process. Retrieved January 23, 2001, from the World Wide Web: http://www.ameritech.com/corporate/testtown/library/articles/design.html

McNeese, M., Zaff, B., Peio, K., Snyder, D., Duncan, J., McFarren, M. (1992). Concept mapping: A pilot’s view of the mission. In An Advanced Knowledge and Design Methodology: Application for the Pilot’s Associate. (pp. 21-60) Wright-Patterson Air Force Base, Ohio: Armstrong Aerospace Medical Research Laboratory. (NTIS No. AAMRL-TR-90-060).

Microsoft. Accessibility Homepage. Retrieved, December 15, 2000, from the World Wide Web: http://www.microsoft.com/enable/

NCDDR. (n.d.). Improving links between research and practice: Approaches to the effective dissemination of disability research. Retrieved January 25, 2001 from the World Wide Web: http://www.ncddr.org/du/products/guide1.html

NCDDR. (n.d.). Improving the usefulness of disability research: A toolbox of dissemination strategies. Retrieved January 25, 2001 from the World Wide Web (link updated September 23, 2003): http://www.ncddr.org/du/products/guide2.html

NCDDR. (1996). A review of the literature on dissemination and knowledge utilization. National Center for the Dissemination of Disability Research. Retrieved January 25, 2001 from the World Wide Web: http://www.ncddr.org/du/products/litreview.pdf

Nielsen, J. (1994). Guerilla HCI: Using discount usability engineering to penetrate the intimidation barrier. Retrieved January 2, 2001 from the World Wide Web: http://www.useit.com/papers/guerrilla_hci.html

Nielsen, J. (1997). The use and misuse of focus groups. Retrieved January 2, 2001 from the World Wide Web: http://www.useit.com/papers/focusgroups.html

Nielsen, J. (n.d.). How to conduct a heuristic evaluation. Retrieved January 2, 2001 from the World Wide Web: http://www.useit.com/papers/heuristic/heuristic_evaluation.html

Nielsen, J. (n.d.). Ten usability heuristics. Retrieved January 2, 2001 from the World Wide Web: http://www.useit.com/papers/heuristic/heuristic_list.html

Norman, D. A. (1998). The invisible computer: Why good products can fail, the personal computer is so complex, and information appliances are the solution. Cambridge, MA: MIT Press.

Poulson, D, Ashby, M., & Richardson, S. (Eds.). (1996). USERfit: A Practical Handbook on User-Centered Design for Assistive Technology. Brussels-Luxembourg: ECSC-EC-EAEC.

Preece, J., Rogers, Y., Sharp, H., Benyon, D., Holland, S., & Carey, T. (n.d.). Methods for user-centered design. Retrieved January 18, 2001 from the World Wide Web: http://iea.fmi.uni-sofia.bh/hci/Book/c18/

Go back to the top of this page.

 M-2. What user-centered design tools are available to designers?

Meyers, B. A. (1997). User interface software tools. Retrieved January 16, 2001, from the World Wide Web: http://www.cs.cmu.edu/afs/cs.cmu.edu/user/bam/www/toolnames.html

Poulson, D, Ashby, M., & Richardson, S. (Eds.). (1996). USERfit: A Practical Handbook on User-Centered Design for Assistive Technology. Brussels-Luxembourg: ECSC-EC-EAEC.

Go back to the top of this page.

 M-3. What is the state-of-the-art in analysis of information requirements and user needs?

There are several ways for the interested designer to gather data regarding user needs and information requirements, including surveys, questionnaires, panel studies, brainstorming sessions, structured interviews, observation, diaries, attribution analysis, and focus groups (KADO, n.d.). For carefully collected data from many individuals, statistical analyses such as factor analysis, cluster analysis, and scaling analysis may be used to identify trends and patterns underlying user behaviors and responses (KADO, n.d.). The nature of the data and the questions they are expected to answer should be carefully considered before choosing a particular analysis technique (Goode, 2001). The choice of the information gathering method depends on several factors, such as the geographic distance that must be covered (questionnaires or other at-home methods would be preferable over focus groups) and the degree to which the characteristics of the population of interest matches those of the designers (brainstorming might be more cost-effective when characteristics are similar).

Conducting focus groups appears to be a common method for gathering extensive information about user needs (e.g., Pacific Bell, 1996; Perlman, 1993). For successfully conducting focus groups several factors must be addressed, such as the quality of the moderator, the size of the group, and the social/cultural nature of the group, which can have a profound effect on the outcome of the effort (Garson, 2000). Additionally, when conducting focus groups with individuals with disabilities, the specific needs of these individuals must be recognized (IBM, n.d.). Finally, supplemental methods have been recommended for eliciting tacit knowledge where current needs assessment methods focus mainly on eliciting explicit knowledge, which may or may not be accurate or complete (Ko, 1999; Nielson, 1997).

Garson, G. D. (2000). Focus Group Research. Retrieved January 23, 2001, from the World Wide Web: http://www2.chass.ncsu.edu/garson/pa765/focusgroups.htm

Goode, E. (January 2, 2001). Researcher challenges a host of psychological studied. The New York Times. Retrieved January 2, 2001, from the World Wide Web: http://www.nytimes.com/2001/01/02/science/02ERRO.html?printpage=yes

IBM. Interacting with people that have disabilities. Retrieved, December 15, 2000, from the World Wide Web: http://www-3.ibm.com/able/hr/interact.html

KADO. User needs methods. Retrieved January 23, 2001, from the World Wide Web: http://www.kado.co.uk/unmeths.htm

Ko, D. (1999). Information requirements analysis and multiple knowledge elicitation techniques: Experience with the pricing scenario system. Proceedings of the 32nd Hawaii International Conference on System Sciences.

Pacific Bell Network. (1996, June). Universal design policy. Retrieved January 4, 2001, from the World Wide Web: http://trace.wisc.edu/docs/pacbell_ud/agpd.htm

Perlman, L. G. (Electronics Industries Foundation). (1993, August). Making technology useable: The views of consumers with learning disabilities, mental retardation, and their caregivers (H133E80029) ). Retrieved January 4, 2001, from the World Wide Web: http://codi.buffalo.edu/graph_based/.universal/.kiss

Go back to the top of this page.

 M-4. What is the state-of-the-art in rapid prototyping and iterative design?

There are several types of prototypes available for the purposes of conducting design evaluation and guiding iterative design (Greenberg, 2000; Lamancusa, 2000). Prototypes vary in the degree to which they are similar in functionality and appearance to the actual product design. High-fidelity prototypes most precisely represent the actual product design, using computers to simulate a great deal of the product's functionality and even actually performing some tasks. In contrast, low-fidelity prototypes, including sketches and storyboards, are paper-based, and cannot simulate product functions but instead represent general design appearance and layout. Medium-fidelity prototypes are not necessarily paper-based, but do not have full capacity to simulate product functions either.

Functionality in medium-fidelity prototypes is limited in three ways, through vertical prototyping, horizontal prototyping, or scenarios (Mankoff, n.d.; Greenberg, 2000). Vertical prototyping allows a particular design function to be evaluated in depth. In contrast, horizontal prototyping allows only surface evaluation of multiple design functions. In scenarios, in which the user must use the prototype to complete a particular function, both breadth and depth of design functionality are limited.

The type of prototype (high-, medium-, or low-fidelity) selected depends on several factors, including the amount of time available to construct the prototype (high-fidelity prototypes take longer to build), monetary constraints (high-fidelity prototypes cost more to make), and the degree to which the design idea is already developed (low-fidelity prototypes are better for less well-developed design ideas; Mankoff, n.d.; Greenberg, 2000). Other considerations include the degree to which quantitative vs. qualitative data collection is desired, the amount of specialized equipment or personnel available, the degree to which independent user exploration of design functionality is desired, and the degree to which controlled study is desired.

Greenberg, S. (2000, March 7). Prototyping for design and evaluation. Retrieved January 18, 2001, from the World Wide Web: http://www.cpsc.ucalgary.ca/~saul/681/1998/prototyping/survey.html

Lamancusa, J. S. (2000, November 16). Rapid Prototyping Primer. Retrieved January 18, 2001, from the World Wide Web: http://www.me.psu.edu/lamancusa/rapidpro/primer/chapter2.htm

Mankoff, J. Rapid Prototyping. Retrieved January 16, 2001, from the World Wide Web: http://www.cc.gatech.edu/people/home/jmankoff/hci/rapid-prototyping.html

Go back to the top of this page.

 M-5. What is the current thinking in the field of error analysis?

A critical finding in the field of error analysis is that characteristics of system design may be more responsible for errors than the operators themselves (Moray, 1994). Another critical finding is that organizational pressures may serve to repress error reporting, which could lead to improved technology design (Linda, et al, 2000). Together, findings such as these indicate that the errors occurring within evaluations of system designs should be taken seriously as potential problems in the system. Further, if usability testing occurs within the organization in which the tested product is being developed, test subjects' desire to minimize reports of errors to avoid criticism must be considered. With regards to testing for accessibility, it has been demonstrated that users simulating disability produce similar errors resulting from design defects, particularly gross design defects, to those errors of people who are actually disabled (Law & Vanderheiden, 1999, 2000).

Law, C. M., & Vanderheiden, G. C. (1999). Tests for screening product designs prior to user testing by people with functional limitations. Presented at the Human Factors and Ergonomics Society Conference.

Law, C. M., & Vanderheiden, G. C. (2000). Reducing sample sizes when user testing with people who have, and how are simulating disabilities - experiences with blindness and public information kiosks. Presented at the joint conference of the International Ergonomics Association and Human Factors and Ergonomics Society.

Linda, T. K., Corrigan, J.M., and Donaldson, M.S. (Eds.). (2000). To Err is Human: Building a Safer Health System. Washington, D.C.: National Academy Press.

Moray, N. (1994). Error reduction as a systems problem. In M.S. Bogner (Ed.), Human Error in Medicine (pp. 67-91). Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers.

Go back to the top of this page.

 M-6. What is the state-of-the-art in human performance testing and evaluation?

Performance testing and evaluation have changed a great deal as the clear demarcation between human and machine performance has blurred (Meister, 1996). In addition, as the human role moves toward supervising and monitoring, performance is not necessarily reflected in an observable behavior, which has implications for how performance is assessed (Meister, 1996). With regards to assessing the performance effects of system design, however, usability testing is still currently valued and practiced (Conyer, 1995; Lee, 1999; Nielsen, 1997). There exist several other methods for examining human-machine interaction through usability testing, such as heuristic evaluation (Nielsen, n.d.) design reference scenarios (Folds, 1998). Each has been shown to be effective, though careful consideration of verbal protocol and rating results must be made (Ericsson & Simon, 1984; Goode, 2001).

Bandura, A. (1977). Social Learning Theory. Englewoods Cliffs, N.J.: Prentice-Hall, Inc.

Boose, J. (1985). A knowledge acquisition program for expert systems based on personal construct psychology. International Journal of Man-Machine Studies, 23, 495-525.

Conyer, M. (1995). User and usability testing: How should it be undertaken? Australian Journal of Educational Technology, 11 (2), 38-51. Retrieved January 25, 2001, from the World Wide Web (link updated September 23, 2003): http://www.ascilite.org.au/ajet/ajet11/conyer.html

Lee, S. H. (1999). Usability testing for developing effective interactive multimedia software: Concepts, dimensions, and procedures. Educational Technology & Society 2 (2). Retrieved January 25, 2001, from the World Wide Web (link updated September 23, 2003): http://ifets.ieee.org/periodical/vol_2_99/sung_heum_lee.html

Meister, D. Human Factors Test and Evaluation in the Twenty First Century. (1996) In O’Brien, T.G. , Charlton, S.G. (Eds.). Handbook of Human Factors Testing and Evaluation. (pp 313-322) Mahwah, NJ: Lawrence Erlbaum Associates, Publishers, Inc.

Nielson, J. (1997). Usability testing. In G. Salvendy (Ed.), Handbook of human factors and ergonomics (2nd Ed., pp. 1543-1568). New York, NY: John Wiley & Sons, Inc.

Go back to the top of this page.

 M-7. What is the current thinking in the field of usability testing and evaluation?

As evidenced by the volume of documentation on methods and procedures (e.g., Folds, 1998, 2000; Law, et al, 2000; Nielsen, 1997, n.d.), usability testing is currently an effective approach to conducting design evaluation, although heuristic evaluation has also been shown to be effective (Nielsen, n.d.). In addition, several tools (e.g., WebCAT) exist, however, to aid designers in creating prototypes for usability testing and conducting evaluations (Meyers, 1997; NIST, n.d.). There are, however, special needs that must be considered when conducting usability tests with participants with disabilities (Law, et al, 2000), and the results of verbal protocols and ratings must be carefully evaluated in order to prevent misinterpretation of the data (Ericsson & Simon, 1984; Goode, 2001; Nielsen, n.d.). Unfortunately, recent work indicates that usability testing (in fact, usability engineering in general) is not a common practice among industrial developers (Nielsen, n.d.). Misperceptions of the cost and time associated with usability engineering appear to be the major roadblock to adopting the practice of user-centered design (Nielsen, n.d.).

Ericsson, K.A. and Simon, H.A. (1984). Protocol Analysis: Verbal Reports as Data. "Cambridge, MA: The MIT Press. Folds, D. J. (2000)"

Folds, D. J. (2000). Three crucial components of an aircrew-centered design process (AIAA-2000-1061. American Institute of Aeronautics and Astronautics.

Folds, D. J. (1998). Use of design reference scenarios in a user-centered design process (AIAA-98-1039). American Institute of Aeronautics and Astronautics.

Goode, E. (January 2, 2001). Researcher challenges a host of psychological studied. The New York Times. Retrieved January 2, 2001, from the World Wide Web: http://www.nytimes.com/2001/01/02/science/02ERRO.html?printpage=yes

Law, C. M., Barnicle, K., & Vanderheiden, G. C. (n.d.). Usability testing of people with disabilities: Where do you begin? Proceedings, Designing for the 21st Century II: An International Conference on Universal Design. Boston: Adaptive Environments Center.

Nielsen, J. (1994). Guerilla HCI: Using discount usability engineering to penetrate the intimidation barrier. Retrieved January 2, 2001 from the World Wide Web: http://www.useit.com/papers/guerrilla_hci.html

Nielsen, J. (n.d.). Characteristics of usability problems found by heuristic evaluation. Retrieved January 2, 2001 from the World Wide Web: http://www.useit.com/papers/heuristic/usability_problems.html

Nielsen, J. (n.d.). How to conduct a heuristic evaluation. Retrieved January 2, 2001 from the World Wide Web: http://www.useit.com/papers/heuristic/heuristic_evaluation.html

Nielsen, J. (n.d.). Ten usability heuristics. Retrieved January 2, 2001 from the World Wide Web: http://www.useit.com/papers/heuristic/heuristic_list.html

NIST. (n.d.) WebCAT Category Analysis Tool. Retrieved January 2, 2001, from the World Wide Web: http://zing.ncsl.nist.gov/webmet/cat/webcat-process.html

Go back to the top of this page.

---

Go back to Section 2: Literature Review Results



Go forward to Section 3: Annotated Bibliography


Warning: Unknown: open(/home/ittatc/ittatc/sessions/sess_9i1npc3aejc1bov8raua379bb5, O_RDWR) failed: No such file or directory (2) in Unknown on line 0

Warning: Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/tmp) in Unknown on line 0