Confidence over Coverage

A few years ago, I was in a call with a client on the topic of test automation. They were looking to add more tests to replace the manual testing they did for every release.

Someone asked “What minimum code coverage threshold should we enforce?”

My Thoughtworks colleague Nick chimed in immediately: “0%”.

Everyone on the call laughed.

“I’m actually serious”, said Nick.

He went on to explain:

“The team don’t write tests today because they don’t see the value in it. If you set a coverage threshold, they’ll write the code the same as they currently do, then write the laziest tests they can think up to meet the coverage check. You’ll get better coverage, but it will be with bad tests and nobody will trust them. So it won’t reduce your manual testing and you’ll also lose the ability to look at coverage and see if people are writing tests or not.”

He was talking about Goodhart’s law. By setting a threshold you turn coverage into a target, and once it’s a target it’s useless as a signal.

But there’s a bigger problem too.

Coverage vs Quality

Code coverage is one of those one-way metrics. Low coverage is a sign you’ve got a problem. High coverage, doesn’t tell you much by itself. It also only tells part of the story. It can tell you if there are tests that execute the code, but it can’t tell you what that code is doing, or if there’s a gap between what it does and what it’s supposed to.

That gap includes things like a lack of error handling, incorrect assumptions about user expectations, or things which you think are edge cases that end up happening all the time. Those are things your customers notice and your testers look for, and code coverage doesn’t even help to find them, let alone fix them.

I’ve seen lots of teams fall into this trap. They start enforcing a minimum code coverage threshold, and coverage goes up, but the quality issues remain. The new tests don’t prevent any bugs, so the slow and expensive manual testing continues anyway. The tests are a pain to write, make refactoring harder, and are complained about constantly. New features end up needing more time to deliver than before because the tests are slowing everything down.

What makes the tests bad?

In teams like this you can often spot the same problematic patterns in the tests.

• Coupling to implementation details, with lots of mocks and “x was called with y” style assertions; rather than verifying the observable behaviour of the system.
• Noisy, copy-pasted 20-line blocks of test setup with minor changes which make reviewing them feel like a game of spot the difference.
• Tests that only exist to meet the threshold, and nobody really knows what they’re testing or whether it’s important.
• Heavily skewed towards unit tests (where coverage is enforced) with very little of anything else. Instead of a testing pyramid, it’s a chimney sitting on a concrete slab.

The commit history tells a similarly frustrating story: "fix test" commits, test code churning with every change, and minor refactors bringing a few dozen seemingly unrelated tests along for the ride.

It’s no wonder these teams hate tests; their lived experience is that the tests cause more problems than they solve.

It’s not coverage you want

Code coverage a bit like the drunk man searching for his keys under the streetlight. It’s so easy to measure, that it feels like a reasonable place to start. But a lack of coverage isn’t the reason these teams rely on manual testing. It’s a lack of confidence.

Kent Beck calls this property Inspiring. Your tests should inspire confidence that if they pass, it’s safe to release.

When they don’t, you look for that confidence elsewhere. And the path of least resistance is usually to pay someone to click a bunch of buttons every time you release, and try not to release unless you have to.

So instead of focussing on coverage, think about why your manual testing gives you confidence where your automated tests don’t.

Then focus on that.