Normal 0 21 false false false FR X-NONE X-NONE

I heard several times thathigh test coverage ratio, and ultimately 100%test coverage, is an illusion of quality. The underlying reason is that when aportion of code is executed by tests, it doesn’t mean that the validity of the results produced by thisportion of code is verified by tests. In other words, some code can becovered by tests and still contains bugs.

I totally agree with thisassertion. However, I don’t agree when one claims that high test coverage ratiois an illusion of quality. I can see several major benefits resulting from hightest coverage ratio.



Contract + High Test Coverage Ratio =Correctness checked anyway

If you apply correctly Design by Contract (DbC) high test coverage pays off. DbCmeans that your code contains side-effect free code whose only purpose is tocheck the states correctness at runtime as often as possible. If the state correctnessis violated, a contract fails abruptly. To implement the DbC principle, we (the team working on NDepend) usedso far my good old friend System.Diagnostics.Debug.Assert(…) and we are in the process ofswitching to the System.Contract API.

The trick is that ifcontract validity verifications are performed during tests execution, thentests are failing when a contract is broken. And the higher the test coverageratio, the more contracts are verified at test-time. By extension, one canactually consider test code dedicated to verification (I mean lines in yourtest like Assert.IsTrue(…)) ascontracts residing outside the code itself. Also, on NDepend development, weobserve that regression bugs are discovered much more often thanks to broken contracts,than thanks to broken test. It is because contracts actually live in the code, they are much moreclose to what code is doing than test themselves. Also contracts are more oftenverified than tests (imagine a contract in a loop vs. a test verificationoutside the loop).

One technical detail isthat TestDriven.NET discards Debug.Assert(…) failure windows by default and I explain here how to activate this behavior(thanks to Jamie Cansdale for the tip).



Bug discovered in covered code = Easier fix

When a bug is discoveredin a portion of code, fixing it is easier if the potion of code is alreadycovered by tests. Indeed, if the buggy code is already covered by at least onetest, often one just needs to copy/paste this test and tweak its input toreproduce the conditions where the bug appear. This extra test is then usefulboth for:
  • Reproducing the bug atwhim, to understand its conditions and fix it
  • Keeping verification thatin the future will check that the bug won’t re-appear.
Having high test coverageratio increases the chance that most of the bug reside actually in codecovered. Ultimately, 100% test coverage means that if the code contains somebugs, they are necessarily covered anyway by some tests.



High coverage leads to relevant investigation

Numerous times, while climbing the full coverage mountain, Istumbled on dead code or better said dead condition. Imagine that in the codebelow, despite plenty of tests the returnstatement is never covered, meaning that objis never InAParticularState.If(obj.IsInAParticularState) { return; }
It is a good hint toinvestigate. Very often in such situation, the investigation will lead to thefact that for some reasons, objcannot be InAParticularState at thatpoint. The best thing to do is then to turn the test condition into a contract.This contract asserts that obj cannotindeed be InAParticularState.

Debug.Assert(!obj.IsInAParticularState,
“document the reason why object cannot be InAParticularState here”);
As a result:


  • you eliminated dead code,
  • the contract is coveredby tests (and then verified at test time),
  • the contract constitutesa documentation for future developers for something that was not obvious atfirst glance and that required investigations.

100% Test Coverage protects from CoverageErosion

100% coverage ratio doesn’thappen by chance. Writing a few tests can often cover up to 80% of the testedcode. However developers often have to struggle hard to reach the 100% coveragegoal. By struggling hard, I mean that developers need to write non-trivial teststo cover non-mainstream scenarios. The 80/20 rule regularly applies here: theseextra 20% of code to cover can require up to 80% of the time spent in writingtests. So when a class or a namespace is 100% covered it means one thing:developers worked hard to reach this 100% value.



100% coverage thenbecomes a requirement for future evolutions and refactoring: because the coderefactored is originally 100% covered, one has to make sure that the newversion of the code is also 100% covered. In other words, when code is 100%covered it is easier to prevent what I usually name coverage erosion. The phenomenon of coverage erosion happens whencode gets refactored with poor care for writing new tests. Unfortunately, inreal-world development shop where turn-over and urgent evolutions are therules, coverage erosion is often a reality. It is what Jeff Atwood presented asThe theory of the broken window. Basically


  • if one develops in aclean environment, then one will struggle to keep the environment clean andavoid erosion.
  • if one develops in adirty environment, then one won’t even try to make the environment cleaner and theoverall entropy will increase.




More...