Debugging via the Scientific Method
I recall a time early in my career, when we spent weeks struggling in the lab to resolve a particularly nasty problem that was plaguing the serial port and drivers that we were developing. We would repeatedly think of something new to try, certain that this would be the end of our woes, only to find time and time again that we were headed back to the drawing board. Looking back, it is embarrassing that with a Physics background, I failed to apply the Scientific Method to solve the problem more quickly.
There are four simple steps to the Scientific Method that can help you get through those nasty bugs. First, observe and describe the phenomenon – sounds straightforward, and is implemented in most software shops as a problem report. So far, so good – if this step is news to you, we’re in more trouble than we thought…
Second, formulate a hypothesis to explain the phenomenon. In any reasonably sized software system, this can be a tough challenge, and this is where many software developers fall from grace by stopping here (usually in late-night debugging sessions). It is easy to mistake the hypothesis for an explanation of the phenomenon, without the diligence of following through the remainder of the method. “Aha! That’s got to be it – let’s make the change and ship it…”
The third step is to use the hypothesis to predict other phenomena. In software, this step is often missed, and is critical to determine the side-effects around the system. What other things are happening because of the current situation? If we make changes based on the current hypothesis, what else can we expect to happen? This important step will often lead you to further investigation before leaping to the solution space, and will reduce the risk that you break something else in the process. Be careful to turn over any corner for data, and don’t jealously guard your hypothesis against data that doesn’t support it.
Finally, the fourth step is to run the tests of your predictions to confirm your hypothesis. It’s important to note that a rigorous application of the Scientific Method would demand independent testers running properly formed tests. All too often, the person that wrote the code will hastily test to confirm that the fix works, rather than to try to break it, and will fail to run an adequate suite of tests.
Repeat as necessary of course, but it’s a safe assumption that the number of cycles of the complete method will be far fewer than the cycles required if only the first two steps are applied.
There are few, if any, best-practices in software development that are new discoveries – they’ve all been around for quite a while, they are just seldom applied. Using the Scientific Method to systematically debug can be one of the most beneficial best-practices in software development, and it’s been around longer than any of us.
Are you depending on a combination of luck and brute force to resolve your most difficult debugging problems, or are you using one of the oldest best-practices in the book? – JB