Stress Testing

Definition

Stress testing is the system testing of an integrated, blackbox application that attempts to cause failures involving how its performance varies under extreme but valid conditions (e.g., extreme utilization, insufficient memory inadequate hardware, and dependency on over-utilized shared resources).

Goals

The typical goals of stress testing are to:

• Cause the application to fail to scale gracefully under extreme conditions so that the underlying defects can be identified, analyzed, fixed, and prevented in the future.

Objectives

The typical objectives of stress testing are to:

• Partially validate the application (i.e., to determine if it fulfills its scalability requirements).
• Determine how an application degrades and eventually fails, as conditions become extreme. For example, stress testing could involve an extreme number of simultaneous users, extreme numbers of transactions, queries that return the entire contents of a database, queries with an extreme number of restrictions, or an entry at the maximum amount of data in a field.
• Report these failures to the development teams so that the associated defects can be fixed.
• Determine if the application will support "worst case" production load conditions.
• Provide data that will assist systems engineers in making intelligent decisions regarding future scaling needs.
• Help determine the extent to which the application is ready for launch.
• Provide input to the defect trend analysis effort.

Examples

Typical examples include stress testing of an application that is:

• Software only.
• A system including software, hardware, and data components.
• Huge (e.g., number of users, number of transactions, amount of data).
• Batch with no realtime requirements.
• Soft realtime (i.e., human reaction times).
• Hard realtime (e.g., avionics, radar, automotive engine control).
• Embeded within another system (e.g., flight-control software, cruise-control software).
• Client/server or n-tier distributed.
• A research prototype that will not be placed into service.
• Business-critical or safety-critical.

Preconditions

Stress test execution can typically begin when the following preconditions hold:

• The scalability requirements to be tested have been specified.
• The relevant software components have passed unit testing.
• Software integration has occurred (i.e., load testing can begin prior to the distribution of the software components onto the hardware components).
• The relevant system components have passed system integration testing.
• The independent test team is adequately staffed and trained in stress testing.
• The test environment is ready.

Completion Criteria

Stress testing is typically complete when the following postconditions hold:

• At least one stress test suite exists for each scalability requirement.
• The test suites for every scheduled scalability requirement execute successfully.

Tasks

Stress testing typically involves the independent test team performing the following testing tasks using the following techniques:

• Test Planning
• Test Reuse
• Test Design:
  • Use Case Based Testing
  • Workload analysis to determine the maximum production workloads.
• Test Implementation:
  • Develop test scripts simulating extreme workloads.
• Test Execution:
  • Regression Testing
  • Profiling
• Test Reporting

Environments

Load testing is typically performed on the following environments using the following tools:

• Test Environment:
  • Test Harness
  • Use case modeling tool
  • Performance analyzer
  • Profiler

Work Products

Stress testing typically results in the production of all or part of the following work products from the test work product set:

• Documents:
  • Project Test Plan
  • Master Test List
  • Test Procedures
  • Test Report
  • Test Summary Report
• Software and Data:
  • Test Harness
  • Test Scripts
  • Test Suites
  • Test Cases
  • Test Data

Phases

Stress testing typically consists of the following tasks being performed during the following phases:

Guidelines

• A system can fulfil its operational requirements and still be a failure if it does not scale.
• The iterative and incremental development cycle implies that stress testing is regularly performed in an iterative and incremental manner.
• Stress testing must be automated if adequate regression testing is to occur.
• Stress testing can elicit failures prior to launch.
• Develop test scripts simulating exceptional workloads.
• Perform stress testing for several minutes to several hours.
• To the extent practical, reuse functional test cases as stress test cases.