Testing Habit - Better error message
Have you ever heard of Larry Wall’s Three Virtues?
Larry, creator of the Perl Programming Language, said they were:
- Laziness
- Impatience
- Hubris
Here, I want to focus on the first - Laziness. This isn’t the “sloth” kind of laziness; it’s about strategic efficiency. It’s the drive to reduce future effort for your collegues (and future you).
As humans, thinking takes energy (sometimes vast amounts). That energy isn’t an infinite resource, and so we should conserve our thinking energy for thinking about the things that matter.
As developers, we have a lot of things competing for our thinking. I’d argue we should reserve our energy for important things and we should demand more help from our systems and tools.
One area that I find is dramatically under-served is in error messaging. When reading an error message, you want as much detail as possible, at the right level of abstraction, to understand what went wrong.
Example
How many times have you gotten a test failure that looks like this?
expected: true
got: false
tests/MyClass.kt:91
Can you guess which of these tests generated that failure message?
@Test
fun `returns success`() {
val resp = HttpResponse(isSuccess = false)
assertTrue(resp.isSuccess)
}
@Test
fun `returns true when even`() {
assertTrue(isEven(3))
}
@Test
fun `returns currently active admins`() {
val users = listOf(
User(id = 1, isAdmin = false, isActive = true),
User(id = 2, isAdmin = true, isActive = false),
)
val activeAdmins = findActiveAdmins(users);
assertTrue(activeAdmins.all { it.isActive && it.isAdmin })
}
All these tests operate at varying levels of abstraction and in different domains. And yet the error messages we see when these tests fail are identical. Imagine if there were multiple assertions in the test. Where in the testing flow did we fail?
Now, imagine a world that looks more like this:
Note: It’s worth mentioning that there are several libraries that provide matchers like these, and if you find one you like, by all means use it. In this context, these are intended to be illustrative of how straightforward it can be to do it yourself.
fun assertSuccess(resp: HttpResponse) = assertTrue(
resp.isSuccess,
"Expected HttpResponse(${resp.req.method} ${resp.req.url}) to be successful, but it was not: ${resp.statusCode}"
)
@Test
fun `returns success`() {
val resp = HttpResponse(isSuccess = false)
assertSuccess(resp)
}
fun assertEven(num: Int) = assertTrue(
isEven(num),
"Expected ${num} to be even, but it is odd"
)
@Test
fun `returns true when even`() {
assertEven(3)
}
fun assertAll<T>(data: List<T>, pred: (T) -> Boolean) {
val failures = data.filterNot(pred)
assertTrue(
failures.isEmpty,
"Expected all records to match, but ${failures.count()} did not. They were: ${failures}"
)
}
@Test
fun `returns currently active admins`() {
val users = listOf(
User(id = 1, isAdmin = false, isActive = true),
User(id = 2, isAdmin = true, isActive = false),
)
val activeAdmins = findActiveAdmins(users);
assertAll(activeAdmins) { it.isActive && it.isAdmin }
}
These custom assertions generate much better error messages and provide way more context. I’d be willing to bet they’re more readable to a fresh eye, too.
Now is it worth creating these kinds of assertions for everything?
No, almost certainly not.
Arguably, examples assertEven are quite contrived and provide little value.
However, in larger contexts with more rich domains, having custom domain-specific errors can help developers diagnose much more quickly.
The sweet spot you’re looking for is when you find yourself repeatedly checking specific, non-obvious properties of your domain.
Conclusion
Next time you have a test failure that’s unhelpful, or unsatisfying, pause. Take a moment to demand more from this miracle of technology in front of you. You deserve to be a little lazy every now and then.