The Art of Software Testing
Chapter 6: Higher-Order Testing
Glen Myers
Wiley
Third Edition, 2012
A software error occurs when the program does not do what its end
user reasonably expects it to do
Even if you could perform absolutely-perfect Module and Incremental
Testing, there is still a good chance that you have not found all
software errors
To complete testing, Higher-Order Testing is necessary
Software development is largely a process of communicating information
about the eventual product and translating this information from one
form to another
1. Translate the software user's needs into a set of written requirements (Product Backlog in Scrum)
2. Design -- partitions the system into individual programs, components, or subsystems,
and defines their interfaces
3. Translate the Design into source code
Most software errors stem from breakdowns in information
communication
One solution is to orient distinct testing processes toward
particular classes of errors
Need for Higher-Order Testing increases along with the size and
complexity of the program
Function Testing
Function testing is the process of attempting to
find discrepancies between the program and the functional requirements
A functional requirement is a precise description of the program's behavior
from the end-user point of view
Higher-Order Testing
Higher-Order testing is not a process of testing the functional
requirements of the complete system or program
Higher-Order testing has a
particular purpose: to compare the system or program to its original objectives
Several Higher-Order Test categories can be used
Many of these are related to non-functional Software Quality Attributes
Some of these will be important in CS 40800 (e.g., Stress Testing, Performance Testing, Usability Testing, ....)
You will do as many of these as possible in part 2 of your project
Sanity Testing
Sanity test is a very brief run-through of the functionality of a program
to assure that the software works roughly as expected
Sanity Testing is testing the major aspects of the program
This is often done prior to a more exhaustive round of testing
Smoke Testing
Origin -- physical tests using smoke made with closed
systems of pipes to detect cracks or breaks
Subset of test cases that cover the most important functionality of a component or system are selected and run
to ascertain if the most crucial functions of a program work correctly
Purpose is to determine whether the application is so badly-broken
that further testing is even possible
Performance Testing
Many programs have specific performance or efficiency objectives, stating
such properties as response times and throughput rates under certain
workload and configuration conditions
Test cases should be designed to show that the program does not satisfy its
performance objectives
Volume Testing
Subject the program to heavy volume of data
Purpose of volume testing is to
show that the program cannot handle the volume of data specified in its
objectives
Example: The system is supposed to be able to store, retrieve, and modify information concerning 1.5
million customers
Stress Testing
Stress testing subjects the program to heavy loads or stresses
A heavy stress is a peak volume of
data, or activity, encountered over a short span of time
If an air traffic control system is supposed to keep track of up to
200 planes in its sector at any one time, you could stress-test it by
simulating the presence of 200 planes ... or more
If an operating system is supposed to support a maximum of 150
concurrent jobs at any one time, the system could be stressed by
attempting to run 150 jobs simultaneously ... or more
Web-based applications are common subjects of stress testing (Chapter 10)
You could stress a mobile device application -- a mobile phone
operating system, for example -- by launching multiple applications that
run and stay resident, then try making or receiving one or more telephone
calls ... while texting
Usability Testing
Tasking the ultimate end user of an application with testing the software in a real-world environment
Covered in next chapter (Chapter 7)
Security Testing
Most software now has specific security objectives
Security testing is the process of attempting
to devise test cases that subvert the program's security checks
One way to devise
such test cases is to study known security problems in similar systems and
generate test cases that attempt to demonstrate comparable problems in
the system you are testing
Web-based applications often need a higher level of security testing
than do most applications (Chapter 10)
Storage Testing
Programs occasionally have storage objectives that state, for
example, the amount of system memory the program uses and the size of
temporary or log files
Configuration Testing
Some software must support a variety of hardware configurations, including
various types and numbers of I/0 devices and communications lines,
or different memory sizes
Often, the number of possible configurations is too large to test each one
Test a representative subset
Reliability Testing
If the program's objectives contain specific statements
about reliability, specific reliability tests might be devised
For example, medical monitoring software must perform for 100 days without need to restart
This is often difficult to test in the short run
May be possible to simulate long periods of use
Recovery Testing
Recovery objectives state how the
system is to recover from programming errors, hardware failures, and data
errors
Programming errors can be purposely injected
into a system to determine whether it can recover from them
Hardware failures can be simulated
Data errors such as noise on a communications line or an invalid
pointer in a database can be created purposely or simulated to analyze the
system's reaction
Installation Testing
Some types of software systems have complicated installation procedures
Testing the installation procedure is an important part of the system testing
process
Acceptance Testing
Usually is performed by
the program's customer or end user and normally is not considered the
responsibility of the development organization
But, development organization should simulate this!
Alpha testing is simulated or actual operational testing by potential users/customers or
an independent test team at the developers' site
Beta testing comes after alpha testing
Versions of the software, known as beta versions, are released to a limited audience outside of the programming team
This audience can include selected end users
Performing the Higher-Order Test
One of the most vital considerations in implementing the Higher-Order test is
determining who should do it
1. Programmers should not perform a Higher-Order test on their own software
2. Of all the testing
phases, this is the one that the organization responsible for developing the
programs definitely should not perform
An ideal Higher-Order test team might be
composed of a few professional Higher-Order test experts (people who spend
their lives performing Higher-Order tests) and a representative end user or two