The Art of Software Testing

Chapter 4: Test-Case Design


Glen Myers
Wiley
Third Edition, 2012





  • The most important consideration in program testing is the design and creation of effective test cases

  • Test case design is important because complete testing is impossible

  • Given constraints on time and cost, the key issue of testing becomes:
    What subset of all possible test cases has the highest probability of detecting the most errors?

  • Least effective methodology of all is random-input testing

  • We have to develop a set of thought processes that enable us to select test data more intelligently

  • Chapter 2 tells us that exhaustive Black Box and White Box testing are, in general, impossible

  • You can develop a reasonably-rigorous process by...
    developing test cases for Black Box testing based on requirements and
    examining the logic of the program to develop test cases for White Box testing

  • Recommend that you use a combination of most, if not all, of the test-case design methods to design a rigorous test of a program, since each method has distinct strengths and weaknesses

    Black Box Testing

  • Black Box Testing is based on program specifications

  • Goal is to find areas where the program does not behave according to its specifications

    Equivalence Partitioning

  • Partition the input domain of a program into a finite number of equivalence classes such that you can reasonably assume that a test of a representative value of each class is equivalent to a test of any other value

  • Each test case should invoke as many different input considerations as possible to minimize the total number of test cases necessary

    Input 1 - Equivalence classes 1, 2, 3
    Input 2 - Equivalence classes 4, 5, 6, 7
    Input 3 - Equivalence classes 8, 9

    Test case number -- Input 1 / Input 2 / Input 3

    Test case 1 -- 1 / 4 / 8
    Test case 2 -- 2 / 5 / 9
    Test case 3 -- 3 / 6 / 8
    Test case 4 -- 2 / 7 / 9

  • Avoid using 3 x 4 x 2 = 24 test cases

  • Test case design by Equivalence Partitioning proceeds in two steps:
    (1) Identify the equivalence classes
    (2) Define the test cases

    Identify the Equivalence Classes

  • The Equivalence Classes are identified by taking each input condition (usually a sentence or phrase in the specification) and partitioning it into two or more groups

  • Valid Equivalence Classes represent valid inputs to the program

  • Invalid Equivalence Classes represent all other possible inputs

  • If an input condition specifies a range of values, identify one valid equivalence class in that range and two invalid equivalence classes on either side of it

    value must be between 10 and 20
    valid (10-20)
    invalid (less than 10)
    invalid (greater than 20)

  • If an input condition specifies a "must-be" situation, such as "first character of the value must be a letter," identify one valid equivalence class (it is a letter) and one invalid equivalence class (it is not a letter)

  • If there is any reason to believe that the program does not handle elements in an equivalence class identically, split the equivalence class into multiple equivalence classes

    Define the Test Cases

    1. Assign a unique number to each equivalence class

    2. Until all valid equivalence classes have been covered by (incorporated into) test cases, write a new test case covering as many of the uncovered valid equivalence classes as possible

    Input 1 - Equivalence classes 1, 2, 3
    Input 2 - Equivalence classes 4, 5, 6, 7
    Input 3 - Equivalence classes 8, 9

    Test case number -- Input 1 / Input 2 / Input 3

    Test case 1 -- 1 / 4 / 8
    Test case 2 -- 2 / 5 / 9
    Test case 3 -- 3 / 6 / 8
    Test case 4 -- 2 / 7 / 9

    3. Until your test cases have covered all invalid equivalence classes, write a test case that covers one, and only one, of the uncovered invalid equivalence classes

    Test case 104 -- 35 (invalid) / 7 / 9
    don't use...
    Test case 104 -- 35 (invalid) / 7 / 46 (invalid)

  • The reason that individual test cases should include only one invalid case is that certain erroneous input checks mask or supersede other erroneous input checks

    Boundary Value Analysis

  • Test cases that explore boundary conditions have a higher payoff than test cases that do not

  • Boundary conditions are those situations directly on, above, and beneath the edges of input equivalence classes and output equivalence classes

    value must be between 10 and 20
    valid (be sure to use both 10 and 20)
    invalid (be sure to use both 9 and 21)

  • Rather than just focusing attention on the input conditions (input space), test cases are also derived by considering the result space (output equivalence classes)

    1. If an input condition specifies a range of values, write test cases for the ends of the range, and invalid-input test cases for situations just beyond the ends

    2. If an input condition specifies a number of values, write test cases for the minimum and maximum number of values and one beneath and beyond these values

    Use guidelines 1 and 2 for output conditions as well

  • Boundary value analysis, if practiced correctly, is one of the most useful test-case design methods

  • However, it often is used ineffectively because the technique, on the surface, sounds simple

  • Boundary conditions may be very subtle and, hence, identification of them requires a lot of thought

    Error Guessing

  • It has often been noted that some people seem to be naturally adept at program testing

  • Without using any particular methodology such as boundary value analysis, these people seem to have a knack for sniffing out errors

  • One explanation for this is that these people are practicing -- subconsciously more often than not -- a test-case design technique that could be termed error guessing

  • Given a particular program, they surmise -- both by intuition and experience -- certain probable types of errors and then write test cases to expose those errors

  • It is difficult to give a procedure for the Error Guessing technique since it is largely an intuitive and ad hoc process

  • The basic idea is to enumerate a list of possible errors or error-prone situations and then write test cases based on the list

  • The presence of the value 0 in a program's input is an error-prone situation
    Therefore, you might write test cases for which particular input values have a 0 value and for which particular output values are forced to O

  • Where a variable number of inputs or outputs can be present (e.g., the number of entries in a list to be searched), the cases of "none" and "one" (e.g. , empty list, list containing just one entry) are error-prone situations

  • Another idea is to identify test cases associated with assumptions that the programmer might have made when reading the specification (i.e., factors that were omitted from the specification, either by accident or because the writer felt them to be obvious)

    If the specification does not say "A student ID must be a 10-digit number", the software engineer might not check that it is exactly 10 digits or even that it is a number!

    White Box Testing

  • White Box testing is concerned with the degree to which test cases exercise or cover the logic of the source code of the program

  • Ultimate white box test is the execution of every path in the program
    But, complete path testing is not a realistic goal for a program with loops

  • Statement Coverage criterion (execute every statement at least once) is so weak that it generally is useless

  • Decision Coverage (write enough test cases that exercise each possible outcome of all decisions at least once) usually satisfies Statement Coverage

  • Decision Coverage is a stronger criterion than Statement Coverage, but it still is rather weak

  • A criterion that is stronger than Decision Coverage is Multiple Condition Coverage

  • Multiple Condition Coverage -- Write enough test cases to ensure that each condition in a decision takes on all possible outcomes at least once

  • Multiple Condition Coverage is superior to Decision Coverage in that it causes every individual condition in a decision to be executed with both outcomes, whereas Decision Coverage does not

    if (x AND y)

    Decision Coverage -- x FALSE leads to FALSE no matter whether y is TRUE or FALSE

    Multiple Condition Coverage -- requires four test cases
    (x TRUE, y TRUE), (x FALSE, y TRUE),
    (x TRUE, y FALSE), (x FALSE, y FALSE)

  • A set of test cases satisfying the Multiple Condition Coverage criterion also satisfies the Decision Coverage and Statement Coverage

  • For White Box Testing use Multiple Condition Coverage

    The Strategy

  • The test-case design methodologies discussed in this chapter can be combined into an overall strategy

  • The reason for combining them should be obvious
    Each contributes a particular set of useful test cases, but none of them by itself contributes a thorough set of test cases

  • The use of this strategy will not guarantee that all errors will be found, but it has been found to represent a reasonable compromise

  • Also, it represents a considerable amount of hard work

  • Developers willing to go beyond cursory testing, who will dedicate sufficient time to test case design, analyze carefully the test results, and act decisively on the findings, will be rewarded with functional, reliable software that is reasonably error free