A while back, a nagging question popped up in the little "related questions" bar on StackOverflow. 1

Is there such a thing as too much unit testing?

Why is the question nagging?

Based on the way the question was phrased, it wasn't really cut out for StackOverflow. The site's goal is specific - to answer programming questions, not touch off religious debates. The left-brained creators of the site wanted it to serve as a knowledge base, and were careful to model it such that it wouldn't turn into a history of flame wars on various programming topics. (I also think there's an element of programmer ego/bias at play on the site: the "I'm not a tester" mentality. But this is purely my speculation.)

At any rate, on the surface this is a "subjective" question; at least in the eyes of StackOverflow admins. But how subjective is it, really? Surely it's a question that deserves an answer, because it's important. The exercise of testing your own code, and developing better tests for your code most definitely provides value; I won't go into all of the benefits here but assume the reader has already heard and (hopefully experienced) many of them. Let's take a closer look at the question itself, first.

The question "Is there such a thing as too much unit testing?" resembles a few others that hinge on a central theme:

  • Is there such a thing as too much debt?
  • Is there such a thing as too much unemployment?
  • Is there such a thing as too many taxes?

It's a question about econonmics. And while it is debatable, certainly there are things we can say with some level of confidence about what may constitute a healthy level of unit testing, or provide a helpful heuristic, rather than ignoring the question entirely. Is there such a thing as too much (unit testing)? Undoubtedly. There is a point of diminishing returns on your time investment in any economic arena. But the key is to understand where tradeoffs exist and how they can be optimized.

Knowing When You're Done

Testing can only prove the presence of bugs, not their absence. –E. W. Dijkstra

Dijkstra came to this conclusion many years ago, and it's an important concept when testing; one must determine exactly what needs to be tested and how to go about doing that. Frugality is important when deciding what tests to write.

A statement that's essentially the contrapositive of Dijkstra's statement is interesting; there are always more tests you could write when implementing any given feature. Usually we determine the set of tests needed to call a feature "complete,"" and most often stop once these have been met. This is commonly referred to as a feature's acceptance criteria. You could write many more tests, but few will usually provide added benefit when competing for your time with new feature work.

But other than determining when your feature is complete, there is also an ongoing reason on why frugality matters; maintenance. Tests are code too and need to be maintained; the more tests you have around means the more dynamic a change to your codebase will be both internally and to your test suite.

Some tests are more valuable than others.

There are parts of your code that change a lot more frequently, are more prone to break, etc. It's important to have tests around assumptions, especially assumptions that are not well grounded or well understood. These are the most economical tests.

You need to balance out the amount of testing you agree to take on as a developer. You can easily overburden yourself with unmaintainable tests. In my opinion, unmaintainable tests are worse than no tests because they:

  1. Turn others off from trying to maintain a test suite or write new tests.
  2. Detract from you adding new, meaningful functionality. If automated testing is not a net-positive result, you should ditch it like other engineering practices.

J. B. Rainsberger has some interesting thoughts on these topics. He comes to the conculsion that Integrated Tests are a Scam; I tend to agree that a test will become harder to maintain, and less economical over time the more complex it is. (Rainsberger also has an interesting talk on the Economics of Software Design.)

While Rainsberger discusses working toward covering your code with focused unit tests and contract tests as a goal, Kent Beck is skeptical of writing an exhaustive suite of tests as a developer, for economical reasons. Beck's points about confidence are important. But the problem is that metrics we usually define for confidence are fuzzy at best. Even so, a 1–10 scale on the level of confidence a test gives you may become a valuable metric for judging whether or not it is worth it to keep. It's very tough to discuss this in any meaningful, measurable fashion, that can be applied across many different projects, situations and skill sets.

To Rainsberger's point, small, isolated and focused tests are much easier to maintain and cheap to keep around. When plugging in test coverage and static analysis tools, these techniques can definitely allow an experienced practicioner to come in and provide meaningful recommendations as to which areas of the system have too little or too many tests without a ton of background or domain knowledge.

What should I test?

Pragmatic Unit Testing recommends you use Right-BICEP to figure out what to test. "Right" for the happy path, then Boundary conditions, check any Inverse relationships, use another method (if it exists) to Cross-check results, force Error conditions, and finally take into account any Performance considerations that should be verified. I'd say if you are thinking about tests to write in this way, you're most likely figure out how to get to an adequate level of testing. You'll be able to figure out which ones are more useful and when. See the book for much more info.

It's important to understand that this is just a heuristic, though. Draconian application of this without analysis as to which tests are needed can leave you in the situation where you are boosting coverage metrics for the sake of the metrics. Metrics are not helpful unless you understand how they affect your bottom line.

Test at the right level

Unit tests are not the only way to write automated tests. Other types of frameworks may be built off of unit tests, but provide mechanisms to do package level, system or integration tests. The best bang for the buck may be at a higher level. I've heard some excellent developers say "Unit testing doesn't seem to catch the kinds of bugs that I tend to write." If you're on that evil genius level, you may think that there's not much benefit in writing tests at so low a level.

Small, focused tests do turn out to be valuable ways to document and specify the system and aid in its maintenance going forward, and run orders of magnitude faster. Be mindful that the more complex and far removed your tests are, the more difficult the automation harness will be to maintain, and slower to run over time. These are the hidden costs that don't become obvious until you look at the cost of your test suite over time.

Exploring other levels of tests may be helpful, given your domain and situation. But they should augument where you know they will definitely catch bugs and integration issues.

Conclusion: The Key to the Right Amount of Testing – Pay As You Go

You may have heard the addage "test a little, code a little." If not, it is the most important way to keep the amount of time and effort you spend on testing focused. This ensures you're paying for your test suite as you go; and maintaining it in small incremental cycles back and forth with the code that it is testing. You'll be less likely to question whether or not a given test will be useful, because it will tend to be small and focused, and directly drive out a feature that is being worked on. When you do this, it eases the burden of maintaining either one in isolation and keeps things in a working and deployable state that you can roll back to if needed.