Integration Watch: When 100% code coverage is not enough

By Andrew Binstock

February 15, 2009 —

In my previous discussions of code coverage, I explained the reasons why 100% code coverage from tests is not a desirable goal. To recap, there is a lot of code that does not need to be tested, and writing code specifically to test it is simply a waste of time. For example, testing setters and getters is a recognized waste of time. Surely you have better things to do with your time.

Another common form of conceptually incorrect and equally wasteful testing is testing in a way that violates encapsulation, or more specifically, object integrity. Suppose that you have a linked list for holding tokens, and suppose that rather than wrapping the list in its own class, as you should, you just use the bare data structure as it’s provided by the library. Then, in your testing, you write a test that adds an item to the list and immediately retrieves it. Then it tests the two versions for equality.

What are you doing? You’re actually testing the data structure provided by the system library. You are not testing your code, but rather a pair of methods encapsulated inside a data structure in a different library. That is never right.

The desirability of 100% code coverage comes from an admirable but misdirected sense that unit testing the entire codebase improves the code’s reliability. This, however, is not the case. Simply because unit tests bring value in one setting does not mean that an excess of tests is a valuable thing.

However, for people inclined in this direction, there is a testing need that’s rarely discussed and rarely done, but which should gratify the desire to write tests: writing more than 100% coverage tests for certain methods.

Greater than 100% is an easily achievable goal. For example, attaining 100% branch coverage often means that a given line needs to be tested multiple ways, resulting in more than 100% line coverage for that line. One hundred percent branch coverage is a desirable goal. You definitely want to make sure that every path your logic can take has been tested. If you aim for 100% branch coverage, you’ll get very high line coverage and you’ll probably be close to the maximum of the test coverage you need.

There is an exception in certain routines where the logic is very complex. Let’s take an example that in working code always generates high complexity values for its constituent routines: a parser. Parsers, like code generators, are not well served by unit tests. The way you gauge the quality of parsers is typically with integration tests. Many dyed-in-the-wool testers will insist that mixing unit tests with integration tests is an error. It’s not an error. At worst, it offends some testers’ purity or sense of order.

Regardless, writing tests that span several methods is frequently the only way to verify parser routines. And if you want to test your code thoroughly, you will need to shove a lot of meat into the parser grinder to see what kind of sausages you get on the other end.

If you do this and carefully include the many edge cases, you will surely reach far, far more than 100% test coverage for the routines. You’ll likely be in the thousands of percent. This is good and desirable. Even if the total application does not reach 100% coverage, this piling on in areas of high complexity is exactly what you want to be doing.

Identification of complex code suitable for super-testing relies on the common complexity metric called cyclomatic complexity (CMC), also called McCabe. It's not an ideal metric, but it is sufficient for our purposes.

Many tools today can tell you the CMC of each method. (CMC should only be measured at the method level.) A project of the now-defunct Agitar Corp. used to suggest how much of a method’s code must be tested at minimum in relation to its CMC. For CMC of 0–5, 0% code is the minimum, which strikes me at too low for CMC greater than 2. CMC 6–10 needs 42% coverage (still low). And so on, all the way to CMC >30, in which case no amount of testing coverage would meet minimum criteria.

This is the key point. At CMC 30, you have to refactor. But at CMC 25–30, you want to super-test the code once you’re certain there is no further useful refactoring. (For the curious, between CMC 11–25, Agitar suggests a range of 57% to 80%. Still too low, in my view.)

So don't shoot for 100% code coverage; go for less overall, and in selected places, go for more. Much more.

Andrew Binstock is the principal analyst at Pacific Data Works. Read his blog at binstock.blogspot.com.

Related Search Term(s): testing, Agitar

Share this link: http://www.sdtimes.com/link/33267

McCabe Software acquires Agitar's assets Two months after Agitar Software ceased operations, McCabe Software swooped in and purchased its assets. McCabe says that it will continue customer and product support for Agitar's unit test tools.

Integration Watch: A new approach to testing: Crowdsourcing Gathering many testers together is uTest's model, and it is increasingly becoming a viable and sound option for developers

ThoughtWorks puts new twist on testing The latest release of this test automation software supports behavior-driven development and allows for common languages

Add comment

Name*
Email*
Country

Comment
Preview

News on Monday
more>>
SharePoint Tech Report
more>>

Download Current Issue
ISSUE 3/15/2010 PDF

Need Back Issues?
DOWNLOAD HERE

Receive the print Edition?

Google Code turns 5
Google Code Turns 5, and adds a Paxos Algorithm to make the system more stable and reliable.
03/17/2010 11:16 AM EST

Test your Visual Studio 2010 know-how
Microsoft is offering free beta certification exams for Visual Studio 2010.
03/17/2010 11:08 AM EST

Microsoft lifts the hood on IE9
Microsoft is previewing IE9.
03/16/2010 01:10 PM EST