The Art of Software Testing
Chapter 8: Debugging
Glen Myers
Wiley
Third Edition, 2012
A successful test case is one that shows that a program does not do what it was designed to do
Debugging is what you do after you have executed one or more successful test cases
Debugging is a two-step process
Step l is the determination of the exact nature and location of the suspected error within the program (95%
of the problem)
Step 2 consists of fixing the error (the other 5%)
Debugging is the one aspect of the software production process that programmers enjoy the least --
Your ego may get in the way
Debugging confirms that
programmers are not perfect
They have commited an error in either the design
or the coding of the program
You may run out of steam
Of all the software development activities,
debugging is the most mentally-taxing activity
Debugging usually is performed under a tremendous amount of organizational or
self-induced pressure to fix the problem as quickly as possible
You may lose your way
Debugging can be mentally taxing because the
error you have found could occur in virtually any statement within the
program
Debugging by Induction
Induction: Possible causes <-- Results of the error
Deduction: Possible causes --> Results of the error
Induction -- You move from the particulars of a situation to the whole
l. Locate the pertinent data
First step is the enumeration of all you know about
what the program did correctly and what it did incorrectly -- the
symptoms that led you to believe there was an error
2. Organize the data
Induction implies that you are
processing from the particulars to the general
Structure the pertinent data to let you observe the patterns
Search for contradictions, events such as
the error occurs only when the customer has no outstanding balance
in his or her account
3. Devise a hypothesis
Study the relationships among the clues, and devise
one or more hypotheses about the cause of the error
If you cannot
devise a theory, more data are needed, perhaps from new test cases
If multiple theories seem possible, select the more probable one first
4. Prove the hypothesis
Compare it to the original clues or data, making sure that
this hypothesis completely explains the existence of the clues
If it does not, the hypothesis is invalid, the hypothesis is incomplete, or
multiple errors are present
5. Fix the problem
Then ... perform regression testing to ensure your bug fix did not create problems in other program areas
As the application grows larger, so does the likelihood that your fix will cause problems elsewhere
Debugging by Deduction
Induction: Possible causes <-- Results of the error
Deduction: Possible causes --> Results of the error
Deduction -- Proceed from some general theories or premises,
using the processes of elimination and refinement, to arrive at a conclusion
(the location of the error)
Murder case
Induction -- Start with set of clues and induce who is the suspect
Deduction -- Start with set of suspects and, by process of elimination using the clues, deduce who is the suspect
1. Enumerate the possible causes or hypotheses
Develop a list of all conceivable causes of the error
2. Use the data to eliminate possible causes
Carefully examine all of the data, particularly by looking for contradictions,
and try to eliminate all but one of the possible causes
If all are eliminated, you need more data gained from additional test cases to devise new theories
If more than one possible cause remains, select the most probable cause -- the prime hypothesis -- first
3. Refine the remaining hypothesis
Possible cause at this point might be correct, but it is unlikely to be specific enough
to pinpoint the error
Use the available clues to refine the theory
For example, you might start with the idea that "there is an
error in handling the last transaction in the file" and refine it to "the
last transaction in the buffer is overlaid with the end-of-file indicator"
4. Prove the remaining hypothesis
Compare it to the original clues or data, making sure that
this hypothesis completely explains the existence of the clues
If it does not, the hypothesis is invalid, the hypothesis is incomplete, or
multiple errors are present
5. Fix the problem
Then ... perform regression testing to ensure your bug fix did not create problems in other program areas
Note that steps 4 and 5 are identical for Induction and Deduction
Debugging by Backtracking
An effective method for locating errors in small programs is to backtrack
the incorrect results through the logic of the program until you find
the point where the logic went astray
You are looking for the location in the program
between the point where the state of the program was what it was expected
to be and the first point where the state of the program was not what it was
expected to be
Debugging Principles
Error-Locating Principles
Think
The most effective method of debugging involves a mental
analysis of the information associated with the error's symptoms
Put yourself in a quiet place
Review how the
program is designed, how the software should be performing within
the area that is performing incorrectly
Imagine ways in which the code may be incorrectly designed
If You Reach an Impasse, "Sleep on It"
If you cannot locate an error in a reasonable amount of time, drop it and turn your attention to something else
After putting aside the problem for a while, you will be clear for a fresh examination of its symptoms
If You Reach an Impasse, Describe the Problem to Someone Else
Talking about the problem with someone else may help you discover
something new
Simply by describing the problem, you may suddenly see the solution without any assistance
from the person
Use Debugging Tools Only as a Second Resort
Turn to debugging tools
only after you have tried other methods, and then only as an adjunct to, not a
substitute for, thinking
Avoid Experimentation -- Use It Only as a Last Resort
The most common
mistake novice debuggers make is to try to solve a problem by making
experimental changes to the program
"I'll change this and see what happens!"
Not only does it have a minuscule chance
of success, but you will often compound the problem by adding new errors
to the program
Error-Repairing Techniques
Where There Is One Bug, There Is Likely to Be Another
When repairing an error, examine its immediate vicinity for anything else that looks suspicious
Fix the Error, Not Just a Symptom of It
If the proposed correction does not match all the clues about the defect, you may be fixing only a part of the defect
The Probability of the Fix Being Correct Is Not 100%
Never assume that code added to a program to fix an error is correct
Corrections are much more error-prone than the original code in the program
A solid regression testing plan can help ensure that correcting an
error does not introduce another error somewhere else in the application
Beware of the Possibility That an Error Correction Creates a New Error
Not only do you have to worry about incorrect corrections, you
also have to worry about a seemingly-valid correction having an undesirable
side effect, thus introducing a new error
This why regression testing works like this...
Re-run the test case(s) that revealed the defect(s)
Re-run test cases that are related to the defect(s)
Re-run test cases that might logically be affected by the changes that were made to fix the defect(s)
Re-run a random subset of all test cases ... which will include many that should have no relation to the defect(s) or the changes that were made