The most important thing about test coverage is consistency

If we can’t, for some reason, have 100% test coverage, which parts of the system should we cover and to what degree?

What will give us more value:

  • to cover only main positive scenarios, but for all features
  • to cover only a few most important features, but in 100%
  • to strive for as high coverage in as many features as possible, deciding what to cover with tests on a case-by-case basis

Many teams new to TDD or working on legacy projects struggle to achieve full test coverage for all features, so questions like this are not uncommon.

Let me share my opinion on this topic.

The biggest value of a test suite is that it gives us confidence

First of all, we need to understand what information a test suite gives us.

An intuitive answer is that when a test suite fails, it gives us information that the system is not correct. If tests are well written, it even gives us precise information what exactly is incorrect and why, which allows us to quickly fix the problem. This is of course valuable, but it’s not the most important information a test suite can give us.

A test suite gives us the most valuable information when it passes. A green bar tells us that the system is correct to a known degree 1. It gives us confidence. It allows us to consider the feature as done. It allows us to deploy to production without fear. But this is true only if we can trust our suite.

A consistent test suite, even if incomplete, gives us some degree of confidence

Can we trust an incomplete test suite? Yes, but only if it is incomplete in a consistent way.

Of course, if test coverage is only partial, we probably won’t be able to automatically deploy to production without additional manual tests. But if the coverage is consistent, e.g. if we know for sure that all the main positive paths are covered, then at least we can focus our manual testing efforts only to the negative paths. Or we may even decide that alternative paths are not critical for us and that having all positive paths covered gives us enough confidence to risk production deployment.

An incomplete test suite doesn’t give us full information, but at least the information it gives us is reliable and we can be confident in decisions we make on it.

An inconsistent test suite gives us no confidence at all

If a test suite is inconsistent, we can’t trust it, even if test coverage is relatively high. If features are covered by tests randomly, on a case-by-case basis, it’s impossible to keep track of what is covered and what isn’t. In such a case, a passing test suite gives us no useful information at all.

We can’t reduce the amount of manual testing, because we’re not sure which parts of the system can be left to the automated verification. We can’t consciously decide about the production deployment, because we’re not sure which features can we be confident about and which not. We lose all the value the green state of the test suite normally gives us.

What to cover with tests to gain maximum confidence if we can’t have a full coverage?

I’m using the term “consistent” a lot – but what exactly do I mean by consistency?

I have a very simple rule for it: Test coverage is consistent if everybody knows, with full confidence, which parts of the system are covered.

We may agree that each feature must have at least the main positive path fully covered; or that a strictly defined set of critical features must have full coverage, and other, less critical features may not be covered; or that all new features must have 100% coverage and the legacy ones don’t have to. There’s no one hard rule, we should choose whichever is most valuable in our particular case. But our choice can’t be ambiguous. If we have to check the code or to ask a colleague to make sure that some part of the system is covered, we will lose confidence – and in effect, we will lose most of the value a passing test suite gives us.

What are your experiences with incomplete test suites? Did they give you any value or were only a maintenance burden? What do you do to get the most out of such partial coverage? Please share below in the comments!

  1. we can never be sure that a system is 100% correct, no matter how high test coverage we have 

What do you think?

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s