In the last few blog posts I wrote about my experiments with Minitest in the last few years. For today’s blog post I thought I’d write how some of Minitest’s design decision affected how I write tests these days and in general how my approach to testing have changed.
As I wrote in Sharing Examples in Minitest, in Minitest we usually write test cases by inheriting from base classes like
In my projects I often needed to add some customizations to the tests, e.g. using a Minitest plugin or custom assertions. I used to add these extensions to
test/test_helper.rb like this:
1 2 3 4 5 6 7 8 9
These days I’d usually write custom test cases instead:
1 2 3 4 5 6 7 8 9
This has a few benefits:
- no monkey patching.
- better code organization. By creating an actual class for each test type it’s pretty natural to eventually move it to a separate file (with the name corresponding to the class, somwhere in
test/support/) if it grows to be too big.
- encourages to have even more custom test cases. It’s very useful when there’s a specific subsystem of the app that deserves it’s own test case (e.g.
BillingTest) that contains helper methods, custom assertions & perhaps extra docs. Again,
ctagsfriendliness really helps here.
Stubbing & Mocking
I’m speculating here, but I wouldn’t be suprised if there would be as many resources praising stubbing (or more broadly, test isolation) as critizing it.
For me, the biggest problems with stubbing are as follows:
- “stubbing at a distance” in a
setup/beforeblock somewhere in base test case/mixin can lead to very unexpected and hard to track bugs.
- “stub leaking”, when stubbing global objects, can too lead to unexpected and hard to track bugs.
- depending on testing/mocking framework it’s sometimes very easy to stub multiple things at once, leading to complex and coupled tests that are hard to maintain.
Note, none of the issues above is intrisic to a particular testing/mocking library – it boils down to how it’s being used.
Minitest ships with a mocking/stubbing support in the
minitest/mock package. Here’s an example of stubbing from the docs:
1 2 3 4 5 6 7 8 9
Note the comment, “stub goes away once the block is done”.
This is really important and it actually solves, or rather provides a constraint, for all of the problems I listed above.
Since the stubbing happens locally, within the given block, we solve the problem of “stubbing at a distance” as well as “leaking”.
In order stub multiple things at once we’d need to nest calls to stub which really stands out (as it should) and actually looks pretty terrible. That was the “aha” moment for me.
This one time, I needed to stub a couple of dependencies and started nesting calls to
stub but looking at it, “listening to the tests”, gave me a pause and I reconsidered my design to have at most one stub in that test and thus having it be more decoupled. In fact, I think I ended up with no stubs at all in that case.
As a matter of fact, I seldom write stubs these days. I tend to prefer writing a “fake” object instead:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
What’s great about this is that I can re-use the fake in many tests as well as in development. I can make the fake more real (but not too real!) by handling important edge case which I can then trivially invoke in, again, both tests & development.
As far as mocking, I seldom do mocking nowadays. Instead of expecting that some object (usually at the boundary of the system) received some message, I’d inspect how the (deterministic) return value of my fake object is being used. In other cases, especially in larger methods when there’s some data being calculated and then sent over to an object, I’d split the method in two: 1 method that does the calculation (unit tested) and the 2nd method that invokes some other object (usually untested, integration tested instead).
Tooling & Practices
I’m constantly impressed how Minitest community is pushing for better tooling and practices. Perhaps the tools/ideas below have not originated in Minitest, but it’s where I first heard about them. For each item I added the year when it was added/considered in Minitest:
- heckle – mutation testing support for minitest. (2006)
- Randomize tests by default. (2008)
- assert_nothing_tested – Minitest doesn’t have
assert_nothing_raised. Instead of asserting that nothing failed, we should assert what the code is actually doing. (2012)
- minitest-bisect – finds the smallest amount of tests to run (in a particular order) to reproduce an order-dependant test failure. (2014)
- minitest-proveit – forces all tests to prove success (via at least one assertion) rather than rely on the absence of failure. (2016)
assert_equalshould not allow
nilfor “expected” – if your code expects a
nilthen explicitly use
assert_nil, otherwise you might have a
nilvalue and not even know about it. (2016)
- updated “Tooling & Practices” section by adding description, mentioning heckle, and a link to