A 16-Line Test Interface

I am skeptical of software “tests”

As I said, until a week or so ago, I had never really written a “test.” Having never worked as a professional programmer, I was never forced to, and writing tests as such is not a natural part of my own development process most of the time. Of course, I test my code constantly, both as a user and with sanity-check print statements or the like, frequent but transient concessions to expedience over rigor that rarely live long enough to touch a git commit; but never had I committed a function whose sole purpose was to test my own code.

The testless approach has worked fine for me thus far, and I am finally satisfied enough with my own programs, one of which contains some thirty-thousand lines of C code, to claim with some confidence that even in fairly large, complex pieces of software, tests are optional.

Still, I expect to work in the software industry someday soon, and to understand the kernels of truth undergirding its orthodoxies would behoove me. To that end, a few weeks ago I made a good-faith¹ effort to learn more about tests, what they’re for, and why they’re important.

add_two_numbers

I focused my research on unit tests, because that’s the type of test I have seen most commonly cited in the admonishments² of the professional community. What I hoped to find were explanations and examples of unit tests that would persuade me that they are valuable. What I actually found was the opposite: a range of beginner resources that repeat the common credos about unit tests but fail to convincingly justify them, and in some cases actually make a compelling argument for their futility.

A majority of these resources use essentially the same example of “testable unit”: add_two_numbers, a function or class method that takes two integers as arguments and returns their sum. Sources vary in their choice of arithmetic operator, their test cases, and the profuseness of their apologies for using such an obviously bad example; but the same basic problems are present in every instance I found. Of course add_two_numbers is meant to be a toy example and not practically useful. Still, one would hope that such an example would model the thought processes required to write useful tests. It should be possible to draw a line from the toy example to some intuition for real world usefulness, and here no such line can be drawn.

Here’s the full example provided by AWS, in python:

def add_two_numbers(x, y):
    return x + y
    
# Corresponding unit tests

def test_add_positives():
    result = add_two_numbers(5, 40)
    assert result == 45

def test_add_negatives():
    result = add_two_numbers(-4, -50)
    assert result == -54

def test_add_mixed():
    result = add_two_numbers(5, -5)
    assert result == 0

These three tests can offer us certainty that exactly three of the theoretically infinite³ possible calls to the function give an expected result. But don’t be cheeky, we identified these test cases because we presume that they represent in microcosm the full range of possible inputs – that all instances of a specific case behave more or less the same.

I’m not a math person, but even I am struck by the arbitrariness of this selection of test cases. The author apparently feels squirrely about signs, but (for example) are they really satisfied that they’ve captured the mixed case adequately by testing one of the few instances where the two values sum to zero? That’s a vanishingly small⁴ subcase. Even worse, addition with zero is not tested. In fact, zero was excluded from all of the comparable examples I found online.

Testing something as fundamental as addition does not strike me as completely absurd, not because I have ever had to implement an addition procedure myself, but because I program in C and it’s very easy to write bugs that result from unhandled type overflow. When adding two numbers, overflow is the likeliest source of problems. Maybe a comparable resource on unit tests tailored to the C programmer would give special attention to overflow? On a whim, I googled “C unit testing” and the third result delivered yet another implementation of add_two_numbers (here called my_sum) and accompanying tests that do not consider overflow.

Again, the article does not pretend that their example is useful or complete, but the fact that a practically useful test case is so near at hand but unmentioned is conspicuous. The point is that the overflow case is the test case that’s easiest to forget, and is exactly the bug that we’re likeliest to write. In fact, the test case that’s easiest to forget is always the bug that we’re likeliest to write. To write good tests, we need to identify a set of test cases that describe the behavior of the function as completely as possible. Once we’ve done that, we’ve already done the work required to anticipate and fix those bugs. So why bother with the tests?

Regressions

One commonly-cited reason to bother with the tests is to catch regressions. If we modify the tested unit, we might create a new bug, and that new bug might be caught by one of our existing tests. The alluring and potentially harmful illusion is that a full suite of passing tests implies correct code. At best, the existence of unit tests helps us catch some trivial bugs early, but at worst, the false confidence they confer inhibits the scrutiny and care required to write fewer bugs, including those that will not be caught by our existing tests.

“Test coverage”

We can begin to glean from the add_two_numbers example the gross inadequacy of almost any contrivable “test coverage” metric. Even if our tests hit every single line of code, they may nevertheless cover only a meager fraction of possible program states and behaviors. We can thoroughly trawl ever cubic inch of the ocean with our magnificent fishing net, but if our aim is to capture aquatic bacteria we will have achieved only the pretense of thoroughness. On the other hand, if we take the time to write good tests, we have already put in the careful thought required to simply improve our code and handle edge and error cases correctly. The quality of our tests, like that of our code, depends on the quality of our thought and attention. A viable method for measuring that quality has not been discovered.

The real value of tests

I’m writing critically about software tests not because I think they shouldn’t be written, but because of the unfortunate ubiquity of the view that they must be written – that writing tests is an integral part of software development, that tests imply reliability. In my view, the dogma surrounding tests merely obscures their actual value as tools for thought.

I am no Dijkstra.⁵ I could try to thoroughly convince myself of the correctness of every line of code I write, but the cognitive cost of such rigor is often too great to be feasible, so I cheat, lie, and steal, engaging in practices more at home in physical engineering – where mathematical models are approximations, and never the whole truth – than the theoretical approach nominally warranted by a formal system. This is an irony inherent in all computer programming: that we humans wrangle theoretically perfect machines⁶ not through direct transmission of pure theory, but by means of an elaborate, sometimes histrionic dance of half-baked thoughts, transient images, remembered patterns, and externalized mnemonics. The sophistication of human cognition is profound, yet our conscious information containers are of comparitively minute size and duration. It’s why we count on our fingers, why we do arithmetic on paper, why we diagram, why we litter our code with debug print statements, and why, if we so choose, we write tests. Writing tests is a cheat, and a valid one; but it is still a cheat.