The Art of Software Testing

Chapter 8: Debugging


Glen Myers
Wiley
Third Edition, 2012





  • A successful test case is one that shows that a program does not do what it was designed to do

  • Debugging is what you do after you have executed one or more successful test cases

  • Debugging is a two-step process

    Step l is the determination of the exact nature and location of the suspected error within the program (95% of the problem)

    Step 2 consists of fixing the error (the other 5%)

  • Debugging is the one aspect of the software production process that programmers enjoy the least --

  • Your ego may get in the way
    Debugging confirms that programmers are not perfect
    They have commited an error in either the design or the coding of the program

  • You may run out of steam
    Of all the software development activities, debugging is the most mentally-taxing activity
    Debugging usually is performed under a tremendous amount of organizational or self-induced pressure to fix the problem as quickly as possible

  • You may lose your way
    Debugging can be mentally taxing because the error you have found could occur in virtually any statement within the program

    Debugging by Induction

    Induction: Possible causes <-- Results of the error
    Deduction: Possible causes --> Results of the error

  • Induction -- You move from the particulars of a situation to the whole

    l. Locate the pertinent data
    First step is the enumeration of all you know about what the program did correctly and what it did incorrectly -- the symptoms that led you to believe there was an error

    2. Organize the data
    Induction implies that you are processing from the particulars to the general
    Structure the pertinent data to let you observe the patterns
    Search for contradictions, events such as the error occurs only when the customer has no outstanding balance in his or her account

    3. Devise a hypothesis
    Study the relationships among the clues, and devise one or more hypotheses about the cause of the error
    If you cannot devise a theory, more data are needed, perhaps from new test cases
    If multiple theories seem possible, select the more probable one first

    4. Prove the hypothesis
    Compare it to the original clues or data, making sure that this hypothesis completely explains the existence of the clues
    If it does not, the hypothesis is invalid, the hypothesis is incomplete, or multiple errors are present

    5. Fix the problem
    Then ... perform regression testing to ensure your bug fix did not create problems in other program areas
    As the application grows larger, so does the likelihood that your fix will cause problems elsewhere

    Debugging by Deduction

    Induction: Possible causes <-- Results of the error
    Deduction: Possible causes --> Results of the error

  • Deduction -- Proceed from some general theories or premises, using the processes of elimination and refinement, to arrive at a conclusion (the location of the error)

  • Murder case
    Induction -- Start with set of clues and induce who is the suspect
    Deduction -- Start with set of suspects and, by process of elimination using the clues, deduce who is the suspect

    1. Enumerate the possible causes or hypotheses
    Develop a list of all conceivable causes of the error

    2. Use the data to eliminate possible causes
    Carefully examine all of the data, particularly by looking for contradictions, and try to eliminate all but one of the possible causes
    If all are eliminated, you need more data gained from additional test cases to devise new theories
    If more than one possible cause remains, select the most probable cause -- the prime hypothesis -- first

    3. Refine the remaining hypothesis
    Possible cause at this point might be correct, but it is unlikely to be specific enough to pinpoint the error
    Use the available clues to refine the theory
    For example, you might start with the idea that "there is an error in handling the last transaction in the file" and refine it to "the last transaction in the buffer is overlaid with the end-of-file indicator"

    4. Prove the remaining hypothesis
    Compare it to the original clues or data, making sure that this hypothesis completely explains the existence of the clues
    If it does not, the hypothesis is invalid, the hypothesis is incomplete, or multiple errors are present

    5. Fix the problem
    Then ... perform regression testing to ensure your bug fix did not create problems in other program areas

  • Note that steps 4 and 5 are identical for Induction and Deduction

    Debugging by Backtracking

  • An effective method for locating errors in small programs is to backtrack the incorrect results through the logic of the program until you find the point where the logic went astray

  • You are looking for the location in the program between the point where the state of the program was what it was expected to be and the first point where the state of the program was not what it was expected to be

    Debugging Principles

    Error-Locating Principles

    Think

  • The most effective method of debugging involves a mental analysis of the information associated with the error's symptoms

    Put yourself in a quiet place
    Review how the program is designed, how the software should be performing within the area that is performing incorrectly
    Imagine ways in which the code may be incorrectly designed

    If You Reach an Impasse, "Sleep on It"

  • If you cannot locate an error in a reasonable amount of time, drop it and turn your attention to something else

  • After putting aside the problem for a while, you will be clear for a fresh examination of its symptoms

    If You Reach an Impasse, Describe the Problem to Someone Else

  • Talking about the problem with someone else may help you discover something new

  • Simply by describing the problem, you may suddenly see the solution without any assistance from the person

    Use Debugging Tools Only as a Second Resort

  • Turn to debugging tools only after you have tried other methods, and then only as an adjunct to, not a substitute for, thinking

    Avoid Experimentation -- Use It Only as a Last Resort

  • The most common mistake novice debuggers make is to try to solve a problem by making experimental changes to the program

  • "I'll change this and see what happens!"

  • Not only does it have a minuscule chance of success, but you will often compound the problem by adding new errors to the program

    Error-Repairing Techniques

    Where There Is One Bug, There Is Likely to Be Another

  • When repairing an error, examine its immediate vicinity for anything else that looks suspicious

    Fix the Error, Not Just a Symptom of It

  • If the proposed correction does not match all the clues about the defect, you may be fixing only a part of the defect

    The Probability of the Fix Being Correct Is Not 100%

  • Never assume that code added to a program to fix an error is correct

  • Corrections are much more error-prone than the original code in the program

  • A solid regression testing plan can help ensure that correcting an error does not introduce another error somewhere else in the application

    Beware of the Possibility That an Error Correction Creates a New Error

  • Not only do you have to worry about incorrect corrections, you also have to worry about a seemingly-valid correction having an undesirable side effect, thus introducing a new error

  • This why regression testing works like this...
    Re-run the test case(s) that revealed the defect(s)
    Re-run test cases that are related to the defect(s)
    Re-run test cases that might logically be affected by the changes that were made to fix the defect(s)
    Re-run a random subset of all test cases ... which will include many that should have no relation to the defect(s) or the changes that were made