A Great Bug Report

I just read Ward's blog entry on 'Best of Bugzilla' and remembered that I had a draft on this topic and never got around to publish it. Here it is:

(October 26, 2005)
Bug 113659, filed by Peter Kullmann, made my day. I don't mean the bug itself, it is the bug report that I like. If there was a textbook on how to report bugs (is there?), this one should be in it. If you haven't done so already, I would recommend you take the time to click on the link to the bug report and read it.

The best part of the bug report was the JUnit test case. I was able to drop it into the appropriate test class, and reproduce the problem just by running the test. Believe it or not, I spend more time trying to reproduce and understand bugs than on actually fixing them. I also like the warm fuzzy feeling of a failing test case that turns green after I put the bugfix in.

Having reproducible steps or even a test case is not enough though. There are cases when the observed behaviour might seem wrong but is not actually a bug. In bug 113659, Peter was very clear about why the test failure is a bug - he cites the relevant API spec that was violated by the bug.

On top of that, Peter provided a patch, attached to the bug report. I should say that this was helpful, but I think people overestimate the value of patches. I would almost always prefer a JUnit test case over a patch.

So here is my advice for reporting bugs:

• Make it easy for the bug assignee to reproduce the bug. A bug report without reproducible steps has little value. Ideally, write a JUnit test case, ready to run.
• List your reasons why you think the observed behaviour is a bug. If the assignee does not agree with you on this, it will not be fixed.
• If you already know how to fix the bug, attach a patch, but keep in mind that your time would be better spent on a JUnit test case.

Peter, thank you for the great bug report, and Ward (with John and Kai), thank you for raising this issue.