The Art of Software Testing

Chapter 6: Higher-Order Testing


Glen Myers
Wiley
Third Edition, 2012





  • A software error occurs when the program does not do what its end user reasonably expects it to do

  • Even if you could perform absolutely-perfect Module and Incremental Testing, there is still a good chance that you have not found all software errors
    To complete testing, Higher-Order Testing is necessary

  • Software development is largely a process of communicating information about the eventual product and translating this information from one form to another

    1. Translate the software user's needs into a set of written requirements (Product Backlog in Scrum)

    2. Design -- partitions the system into individual programs, components, or subsystems, and defines their interfaces

    3. Translate the Design into source code

  • Most software errors stem from breakdowns in information communication

  • One solution is to orient distinct testing processes toward particular classes of errors

  • Need for Higher-Order Testing increases along with the size and complexity of the program

    Function Testing

  • Function testing is the process of attempting to find discrepancies between the program and the functional requirements
    A functional requirement is a precise description of the program's behavior from the end-user point of view

    Higher-Order Testing

  • Higher-Order testing is not a process of testing the functional requirements of the complete system or program

  • Higher-Order testing has a particular purpose: to compare the system or program to its original objectives

  • Several Higher-Order Test categories can be used

  • Many of these are related to non-functional Software Quality Attributes

  • Some of these will be important in CS 40800 (e.g., Stress Testing, Performance Testing, Usability Testing, ....)
    You will do as many of these as possible in part 2 of your project

    Sanity Testing

  • Sanity test is a very brief run-through of the functionality of a program to assure that the software works roughly as expected

  • Sanity Testing is testing the major aspects of the program

  • This is often done prior to a more exhaustive round of testing

    Smoke Testing

  • Origin -- physical tests using smoke made with closed systems of pipes to detect cracks or breaks

  • Subset of test cases that cover the most important functionality of a component or system are selected and run to ascertain if the most crucial functions of a program work correctly

  • Purpose is to determine whether the application is so badly-broken that further testing is even possible

    Performance Testing

  • Many programs have specific performance or efficiency objectives, stating such properties as response times and throughput rates under certain workload and configuration conditions

  • Test cases should be designed to show that the program does not satisfy its performance objectives

    Volume Testing

  • Subject the program to heavy volume of data

  • Purpose of volume testing is to show that the program cannot handle the volume of data specified in its objectives

  • Example: The system is supposed to be able to store, retrieve, and modify information concerning 1.5 million customers

    Stress Testing

  • Stress testing subjects the program to heavy loads or stresses

  • A heavy stress is a peak volume of data, or activity, encountered over a short span of time

  • If an air traffic control system is supposed to keep track of up to 200 planes in its sector at any one time, you could stress-test it by simulating the presence of 200 planes ... or more

  • If an operating system is supposed to support a maximum of 150 concurrent jobs at any one time, the system could be stressed by attempting to run 150 jobs simultaneously ... or more

  • Web-based applications are common subjects of stress testing (Chapter 10)

  • You could stress a mobile device application -- a mobile phone operating system, for example -- by launching multiple applications that run and stay resident, then try making or receiving one or more telephone calls ... while texting

    Usability Testing

  • Tasking the ultimate end user of an application with testing the software in a real-world environment

  • Covered in next chapter (Chapter 7)

    Security Testing

  • Most software now has specific security objectives

  • Security testing is the process of attempting to devise test cases that subvert the program's security checks

  • One way to devise such test cases is to study known security problems in similar systems and generate test cases that attempt to demonstrate comparable problems in the system you are testing

  • Web-based applications often need a higher level of security testing than do most applications (Chapter 10)

    Storage Testing

  • Programs occasionally have storage objectives that state, for example, the amount of system memory the program uses and the size of temporary or log files

    Configuration Testing

  • Some software must support a variety of hardware configurations, including various types and numbers of I/0 devices and communications lines, or different memory sizes

  • Often, the number of possible configurations is too large to test each one

  • Test a representative subset

    Reliability Testing

  • If the program's objectives contain specific statements about reliability, specific reliability tests might be devised

  • For example, medical monitoring software must perform for 100 days without need to restart

  • This is often difficult to test in the short run
    May be possible to simulate long periods of use

    Recovery Testing

  • Recovery objectives state how the system is to recover from programming errors, hardware failures, and data errors

  • Programming errors can be purposely injected into a system to determine whether it can recover from them

  • Hardware failures can be simulated

  • Data errors such as noise on a communications line or an invalid pointer in a database can be created purposely or simulated to analyze the system's reaction

    Installation Testing

  • Some types of software systems have complicated installation procedures

  • Testing the installation procedure is an important part of the system testing process

    Acceptance Testing

  • Usually is performed by the program's customer or end user and normally is not considered the responsibility of the development organization

  • But, development organization should simulate this!

  • Alpha testing is simulated or actual operational testing by potential users/customers or an independent test team at the developers' site

  • Beta testing comes after alpha testing
    Versions of the software, known as beta versions, are released to a limited audience outside of the programming team
    This audience can include selected end users

    Performing the Higher-Order Test

  • One of the most vital considerations in implementing the Higher-Order test is determining who should do it

    1. Programmers should not perform a Higher-Order test on their own software

    2. Of all the testing phases, this is the one that the organization responsible for developing the programs definitely should not perform

  • An ideal Higher-Order test team might be composed of a few professional Higher-Order test experts (people who spend their lives performing Higher-Order tests) and a representative end user or two