The Art of Software Testing
Chapter 2: The Psychology And Economics Of Software Testing
Glen Myers
Wiley
Third Edition, 2012
Complete testing of a complex application would take too long and require too many human resources to be economically feasible
The Psychology of Testing
Do NOT test a program to show that it works
Rather, start with the assumption that the program contains errors
A valid assumption for almost any program
Test the program to find as many of the errors as possible
If our goal is to demonstrate that a program has errors, our test data will have a higher probability of finding errors
A well-constructed and executed software test is SUCCESSFUL when it finds errors that can be fixed
Program testing is properly viewed as the process of
trying to find the errors in a program (whose presence is assumed)
A successful test case is one that furthers progress in this
direction by causing the program to fail
The Economics of Testing
In general, it is impractical, often impossible, to find all the errors in a program
Black-Box Testing
Test data are derived solely from the requirements
without taking advantage of knowledge of the internal structure of the program
If you want to use this approach to find all errors in the program, the criterion is exhaustive input testing
Make use of every possible input condition as a test case
You have to test using not only all valid inputs, but all possible inputs
To test the triangle program exhaustively, you would have to produce virtually an infinite number of test cases
Exhaustive input testing of larger programs is even more of a problem
In a database application such as an airline reservation system, the execution of a transaction
(such as a database query or a reservation for a plane flight) is dependent upon what happened in previous transactions
Exhaustive input testing is impossible!
(l) You cannot test a program to guarantee that it is error free
(2) A fundamental consideration in program testing is one of economics
Objective should be to maximize the yield on the testing investment
by maximizing the number of errors found by a finite number of test cases
White-Box Testing
Permits you to examine the internal structure of the program
Causing every statement in the program to execute at least once might appear to be the answer
But this is highly inadequate
If you execute, via test cases, all possible paths of control flow through the program,
then possibly the program has been completely tested
The number of unique logic paths through a program could be astronomically large
Figure 2.1 Control-Flow Graph of a Small Program
Determining the number of unique logic paths is the same as determining
the total number of unique ways of moving from point a to point b
This number is approximately 10**14 or 100 trillion
If you could write, execute, and verify a test case every five minutes,
it would take approximately l billion years to try every path.
Exhaustive path testing, like exhaustive input testing, appears to be impractical, if not impossible
Software Testing Principles
1 A necessary part of a test case is a definition of the expected output or result
If the expected result of a test case has not been predefined, chances are that a plausible, but erroneous, result will be interpreted as a correct result because of the phenomenon of "the eye seeing what it wants to see"
A test case must contain these two components
i. A description of the input data to the program
ii. A precise description of the correct output of the program for that set of input data
2 A programmer should avoid testing his or her own program (whenever possible)
You know what the code is supposed to do, hence you may not recognize
when it does otherwise
And you really do not want to find errors in your own code anyway
The program may contain errors due to the programmer's
misunderstanding of the problem statement or requirements
If this is the case, it is likely that the programmer will carry the same misunderstanding
into tests of his or her own program
On the other hand, debugging is more efficiently performed by the original programmer
3 A programming organization should not test its own programs (whenever possible)
A programming organization is, in many senses, a living organization
Psychological problems similar to those of individual programmers
More effective for testing to be performed by an objective, independent party
4 Any testing process should include a thorough inspection of the results of each test
Errors that are found in later tests were often missed in the results from earlier tests
5 Test cases must be written for input conditions that are invalid and unexpected,
as well as for those that are valid and expected
There is a natural tendency when testing a program to concentrate on the valid and expected input conditions, to the neglect of the invalid and unexpected conditions
6 Examining a program to see if it does not do what it is supposed to do is only half the battle; the
other half is seeing whether the program does what it is not supposed to do
Programs must be examined for unwanted side effects
For instance, a payroll program that produces the correct paychecks is still an erroneous program if it also produces extra checks for nonexistent employees
7 Avoid throwaway test cases unless the program is truly a throwaway program
A common practice is to sit at a workstation and invent test cases on the fly
Then send these test cases through the program
The major issue is that test cases represent a valuable investment that, in this environment,
disappears after the testing has been completed
8 Do not plan a testing effort under the tacit assumption that no errors will be found
Even after extensive testing and error correction, it is safe to assume that errors still exist
They simply have not yet been found
9 The probability of the existence of more errors in a section of a program is proportional to the number of errors already found in that section
Errors tend to come in clusters
In the typical program, some sections seem to be much more prone to errors than other sections
10 Testing is an extremely creative and intellectually challenging task
It is probably true that the creativity required in testing a large program exceeds the creativity
required in designing that program