The Art of Software Testing

Chapter 2: The Psychology And Economics Of Software Testing


Glen Myers
Wiley
Third Edition, 2012





  • Complete testing of a complex application would take too long and require too many human resources to be economically feasible

    The Psychology of Testing

  • Do NOT test a program to show that it works
  • Rather, start with the assumption that the program contains errors
  • A valid assumption for almost any program
  • Test the program to find as many of the errors as possible

  • If our goal is to demonstrate that a program has errors, our test data will have a higher probability of finding errors

  • A well-constructed and executed software test is SUCCESSFUL when it finds errors that can be fixed

  • Program testing is properly viewed as the process of trying to find the errors in a program (whose presence is assumed)
  • A successful test case is one that furthers progress in this direction by causing the program to fail

    The Economics of Testing

  • In general, it is impractical, often impossible, to find all the errors in a program

    Black-Box Testing

  • Test data are derived solely from the requirements without taking advantage of knowledge of the internal structure of the program

  • If you want to use this approach to find all errors in the program, the criterion is exhaustive input testing
  • Make use of every possible input condition as a test case
  • You have to test using not only all valid inputs, but all possible inputs
  • To test the triangle program exhaustively, you would have to produce virtually an infinite number of test cases

  • Exhaustive input testing of larger programs is even more of a problem

  • In a database application such as an airline reservation system, the execution of a transaction (such as a database query or a reservation for a plane flight) is dependent upon what happened in previous transactions

  • Exhaustive input testing is impossible!

    (l) You cannot test a program to guarantee that it is error free
    (2) A fundamental consideration in program testing is one of economics

  • Objective should be to maximize the yield on the testing investment by maximizing the number of errors found by a finite number of test cases

    White-Box Testing

  • Permits you to examine the internal structure of the program

  • Causing every statement in the program to execute at least once might appear to be the answer
  • But this is highly inadequate

  • If you execute, via test cases, all possible paths of control flow through the program, then possibly the program has been completely tested

  • The number of unique logic paths through a program could be astronomically large


    Figure 2.1 Control-Flow Graph of a Small Program

  • Determining the number of unique logic paths is the same as determining the total number of unique ways of moving from point a to point b

  • This number is approximately 10**14 or 100 trillion

  • If you could write, execute, and verify a test case every five minutes, it would take approximately l billion years to try every path.

  • Exhaustive path testing, like exhaustive input testing, appears to be impractical, if not impossible

    Software Testing Principles

    1 A necessary part of a test case is a definition of the expected output or result

  • If the expected result of a test case has not been predefined, chances are that a plausible, but erroneous, result will be interpreted as a correct result because of the phenomenon of "the eye seeing what it wants to see"

  • A test case must contain these two components
    i. A description of the input data to the program
    ii. A precise description of the correct output of the program for that set of input data

    2 A programmer should avoid testing his or her own program (whenever possible)

  • You know what the code is supposed to do, hence you may not recognize when it does otherwise
  • And you really do not want to find errors in your own code anyway

  • The program may contain errors due to the programmer's misunderstanding of the problem statement or requirements
  • If this is the case, it is likely that the programmer will carry the same misunderstanding into tests of his or her own program

  • On the other hand, debugging is more efficiently performed by the original programmer

    3 A programming organization should not test its own programs (whenever possible)

  • A programming organization is, in many senses, a living organization
  • Psychological problems similar to those of individual programmers

  • More effective for testing to be performed by an objective, independent party

    4 Any testing process should include a thorough inspection of the results of each test

  • Errors that are found in later tests were often missed in the results from earlier tests

    5 Test cases must be written for input conditions that are invalid and unexpected, as well as for those that are valid and expected

  • There is a natural tendency when testing a program to concentrate on the valid and expected input conditions, to the neglect of the invalid and unexpected conditions

    6 Examining a program to see if it does not do what it is supposed to do is only half the battle; the other half is seeing whether the program does what it is not supposed to do

  • Programs must be examined for unwanted side effects

  • For instance, a payroll program that produces the correct paychecks is still an erroneous program if it also produces extra checks for nonexistent employees

    7 Avoid throwaway test cases unless the program is truly a throwaway program

  • A common practice is to sit at a workstation and invent test cases on the fly
  • Then send these test cases through the program
  • The major issue is that test cases represent a valuable investment that, in this environment, disappears after the testing has been completed

    8 Do not plan a testing effort under the tacit assumption that no errors will be found

  • Even after extensive testing and error correction, it is safe to assume that errors still exist
  • They simply have not yet been found

    9 The probability of the existence of more errors in a section of a program is proportional to the number of errors already found in that section

  • Errors tend to come in clusters
  • In the typical program, some sections seem to be much more prone to errors than other sections

    10 Testing is an extremely creative and intellectually challenging task

  • It is probably true that the creativity required in testing a large program exceeds the creativity required in designing that program