Code Coverage. You Don’t Need It!

What was wrong with our code coverage?

Our SDET team measured code coverage for end-to-end UI tests (those that would click on buttons in the app on users’ behalf). Some time ago management asked SDETs to measure the coverage and to achieve a certain number in it. So they did. In fact, to maintain a high number they have to update the tests every time the UI changes (and it changes with every release for a mobile app) so they spent more time (and money) fixing tests and reacting on false-positives than on increasing the actual test coverage (management even went that far to recommend measuring coverage for manual tests, which fortunately never happened, and we have different guidelines now so I can safely tell this story).

Why exactly high-coverage-test doesn’t detect regressions?

Let’s take a small library I’m working on as an example. Without the test for the root-class CadabraImpl the coverage looks like that

What the coverage is all about then?

It’s been a lot written on the topic (e.g). In short, similarly to Dijkstra’s famous phrase “Testing shows the presence, not the absence of bugs” we can say that code coverage shows the absence, not the presence of tests, and it definitely says nothing about their quality.

Can we still have coverage?

There may still be a reason or two to measure code coverage for non-unit tests or even strive for a high coverage number and modern tools can alleviate the problem by showing not only the number and covered lines/branched but also the “source” of coverage (e.g) but measuring a single number without understanding and properly communicating the meaning of it most likely will produce yet another “dashboard promoting ignorance”.

--

--

Software developer. Most recently Android Java/Kotlin engineer. Former manager, desktop and embedded software creator. https://github.com/fo2rist/

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Dmitry Si

Dmitry Si

Software developer. Most recently Android Java/Kotlin engineer. Former manager, desktop and embedded software creator. https://github.com/fo2rist/