System Usability Testing

Definition

System usability testing is the system testing of an integrated, blackbox application against its usability requirements requirements.

Goals

The typical goals of system usability testing are to:

• Cause the application to fail to meet its usability requirements so that the underlying defects can be identified, analyzed, fixed, and prevented in the future.

Objectives

The typical objectives of system usability testing are to:

• Partially validate the application by determining if it fulfills its quantitative and qualitative usability requirements concerning ease of:
  • Installation by the environments team.
  • Usage by the user organization.
  • Operations by the operations organization.
• Identify usability defects involving the application’s human interface such as:
  • Difficulty of orientation and navigation (e.g., navigation defects such as broken links and anchors within a website)
  • Efficiency of interactions (based on user task analysis)
  • Information consistency and presentation
  • Appropriate use of language and metaphors
  • Conformance to the:
    • Digital Brand Description Document
    • Website Design Guidelines
  • Programming defects (e.g., incorrectly functioning tab key, accelerator keys, and mouse actions)
• Determine if the application's human interfaces should be iterated to make them more usable.
• Determine how well the human interfaces enable the different categories of users to efficiently perform their business tasks.
• Report these failures to the development teams so that the associated defects can be fixed.
• Help determine the extent to which the application is ready for launch.
• Provide input to the defect trend analysis effort.

Examples

Typical examples include system usability testing of an application that is to be used by:

• Well-trained professionals.
• Members of the public.

Preconditions

System usability testing typically can begin when the following preconditions hold:

• The usability requirements to be tested have been specified and implemented.
• The user experience team is adequately staffed and trained in usability testing.
• The application under test exists.
• Test participants are available.

Completion Criteria

System usability testing is typically complete when the following postconditions hold:

• The usability team has tested the user interface or document. For example:
  • All links and anchors navigate to the proper parts of the proper webpages.
  • All user inputs produce proper results.
  • Webpages are consistent with branding guidelines.
  • Screens open and close properly.
• A group of test participants representing users from each type of user have been identified and have evaluated the user interfaces.
• The results of the evaluation have been analyzed.
• Associated test reports with recommendations have been presented back to the development team.

Tasks

System usability testing typically includes the user experience team performing the following testing tasks:

• Test Planning:
  • The user experience team plans system usability testing.
  • The user experience team enters plans for document usability testing in the project test plan by entering the following topics in the system usability testing subsection of the project test plan:
    • Goals
    • Objectives
    • Testing work products (see below).
    • Roles and responsibilities (e.g., test monitor and test participant including user profile and sources)
    • Tasks
    • Test environment including required equipment
    • Completion and success criteria
  • The user experience team summarizes its plans for system usability testing in the master test list.
• Test Reuse:
  • Where practical, the usability team reuses test work products.
• Test Design:
  • The user experience team characterizes and profiles the users of the system to identify the questions to be used to screen potential test participants.
  • The user experience team determines:
    • The appropriate number of each kind of test participant.
    • What to say to orient each test participant.
    • What to to ask each test participant prior to the test.
    • What observations to make during testing.
    • The test suite of test cases to develop.
    • The associated test procedure.
    • What to ask each test participant following the test.
    • What to pay each test participant.
• Test Implementation:
  • The user experience team develops the following work products:
    • Usability Test Screening Questionnaire
    • Usability Test Orientation Script
    • Usability Test Observations Form
    • Usability Pretest Questionnaire
    • Test Procedure
    • Test Suite
    • Test Case
    • Usability Posttest Questionnaire
  • The user experience team uses the system test screening questionnaire to recruit the prototype usability test participants, if possible from the actual target user communities.
  • The usability team prepares the test environment, equipment, and system.
• Test Execution:
  • The test monitor dry runs the test procedure and the associated test suite of test cases.
  • The test monitor greets the test participant.
  • The test monitor reads the usability test orientation script to the test participant.
  • The test monitor has the test participant sign a nondisclosure agreement and a consent to be recorded form.
  • The test monitor has the test participant fill out the usability pretest questionnaire.
  • The test monitor starts any recording devices.
  • The test participant tests the system by following the test procedure and performing the associated test suite of test cases.
  • The test monitor observes the test participants during the test and records his/her observations on the usability test observations form.
  • The test monitor has the test participant fill out the usability posttest questionnaire.
  • The test monitor debriefs the test participant and records his/her observations on the usability test observations form.
  • The test monitor thanks and compensates the test participant.
  • The test monitor analyzes the usability test results.
• Test Reporting:
  • The user experience team prepares a test report including recommendations.
  • The user experience team delivers the test report to the development team so that they can fix the defects.
  • The user experience team updates the current test summary report including recommendations.

Environments

System usability testing is typically performed on the following environments using the following tools:

• Usability Lab production environments including ( client environment, contact center environment, and data center environment):
  • A protocol of tasks and preference questions that the participants will run through.
  • Equipment including, but not limited to:
    • Computer and monitor
    • Video camera(s)
    • Video mixer
    • Tape recorder(s)
    • Screen capturing tool (e.g., Lotus ScreenCam)
    • User action capturing tool (e.g., Winrunner)
    • Scan converter

Work Products

System usability testing typically results in the production of all or part of the following work products from the test work product set:

• Documents:
  • Project Test Plan
  • Master Test List
  • Usability Test Screening Questionnaire
  • Usability Test Orientation Script
  • Usability Test Observations Form
  • Usability Pretest Questionnaire
  • Test Procedure (a.k.a., test protocol, test script)
  • Usability Posttest Questionnaire
  • Test Report
  • Test Summary Report
• Software and Data:
  • Test Suite
  • Test Case
  • Test Data

Phases

System usability testing typically consists of the following tasks being performed during the following phases:

Guidelines

• System usability testing should be done with representative users, either in a usability lab or in the production environment.
• The following are suggested techniques for measuring usability:
  • Average time and number of steps required to install and configure the application.
  • Average time to complete a given user task.
  • Average percentage of tasks completed in a given time.
  • Average number or percentage of user errors.
  • Number of user interactions required to perform a specific user task.
  • Average number of times the user uses either inline help features and documentation.
  • Percentage of times that users expresse frustration or satisfaction.
  • Average time required to obtain proficiency using the application.
  • Success rate at finding the desired feature and using it optimally.