OddThinking

Dear Youse Bastards,

I recently grumbled about letting myself get stuck in a unit-testing quandary.

You responded in the comments with a calvalcade of recommended best-practices, which I proceeded to both agree with in principle in many situations, but also stridently defend that I shouldn’t use them in my situation – the payback wasn’t warranted.

I then proceeded to get back to work on changing a mid-level unit. Suddenly, the Refactoring Fairy came and whispered in my ear a suggested way of abstracting away one part of the ugly, third-party I/O layer.

It meant abandoning some of the progress I had made, but it was The Right Thing™, so I added a new class. Of course, I had your silly ideas still ringing in my head, so the new layer was written to support Dependency Injection (DI) to replace the layers below, and I also subclassed a Mock version so I could test its clients. When I unit-tested the new layer, I was careful not to over-test.

Then I went back to the mid-level unit and refactored to use the new layer. The code became 43.0% more beautiful.

While I was updating the mid-level unit’s unit-tests, I could use the new mock objects, and it went very smoothly.

Now, I could work with the exactly same API from the high-level unit to the mid-level unit, but I could now simplify the handles to I/O objects, so I went ahead and did that… and while I did, I added a few more interface features to allow mock objects to be more easily used.

While changing the high-level unit – the one that starts “# TO DO: Refactor into smaller parts, that can be tested faster.” – to use the new mid-level API, I got the Refactoring Fairy to keep look-out to make sure no Project Manager was watching, and I hacked out the complete chunk that dealt with the lower-level unit, wrapped it in a pretty – and mockable – API, and tested it separately.

Finally, I stitched together the wounds of the top-level object, and looked at its unit-tests… the horrible ones that had hard-dependencies on the lower-level objects and were the cause of the original complaint. With your fancy-pants concepts at top of mind, I could now see several problems with them:

• Of course, they had horrible dependency problems, which is why I originally complained.
• They were slow (dealing with timeouts, and the like, meant they were filled with sleep statements), which triggered the To Do comment.
• They were over-testing – spending effort on checking that the sub-layers were working correctly rather than focussing on the current unit being examined.
• Finally, they also tested items that were no longer part of the unit, because they had been cut away.

I substantially rewrote these tests, using the mock layers I now had waiting. The results were much shorter and much easier to understand. The number of test-cases requiring sleep statements was substantially reduced, and I also made the timeouts overrideable, so I could force timeouts by sleeping for 50 milliseconds rather than 5 seconds.

So, to summarise. Because youse bastards put your ideas about better testability in my head, and got both my internal Unit Test Advocate andthe Refactoring Fairy all excited, I went ahead and wasted several days on refactoring – removing over-testing, adding two more layers of abstraction, adding mocks to pretend to be those new abstraction layers. That was development time that I swore would not be worth it.

Benefits

The result is:

• cleaner code, on the whole. The increased complexity of DI was outweighed by the other simplifications.
• cleaner – and less fragile – tests. While the amount of code in each test-cases was generally much shorter, that was outweighed by the new Mock implementations. There is more test code now, but it is easier to understand.
• a unit-test suite that runs about 2 minutes faster. I run the suite at least half-a-dozen times a day – often more, but I rarely sit around waiting for the result.

Limitations

Improved Bug Detection

I would have liked to say “And I found 5 serious bugs in my original code I had never caught before!”

However, that wasn’t the case. I only discovered one bug:

if number_of_unprocessed_items > 0:
    slowly_prepare_system_for_processing()
    process_remaining_items()

The idea of the condition here was to avoid the slow system preparation if there was nothing to do.

However, number_of_unprocessed_items was a function, not a number. I omitted the parenthesis. It should have been number_of_unprocessed_items() > 0. Python should know better than to define a greater-than operation on function-pointers. That is the kind of mistake C makes, not Python.

The result was the slow system preparation was always run – a small unnecessary performance hit that my unit-tests could never detect… until I mocked out the slowly_prepare_system_for_processing and noticed it was being called more often than expected.

No Dependency Injection Framework

I haven’t taken Sunny’s suggestion and found a DI Framework for Python. I suspect the demand for such a beast is lower in a language like Python where classes can be passed as parameters. I did take a look at a major one, and particularly its advertised benefits. There was a large list, and all but one of them (testability) was of zero interest to me on this project.

Poorer Integration Tests

If Class A depends on B depends on C depends on D, then I am now testing:

• C works against mock D.
• B works against mock C.
• A works against mock B.

That’s good unit-test style, right?

Before I was testing:

• C works against mock D.
• B works against C running against mock D.
• A works against B running against C running against mock D.

I was encountering problems that A’s tests were affected when C changed.

But now I have opened myself to another risk. What if one of the many mocks doesn’t match the real classes very well. I am never performing integration tests that make sure my units fit together. Before, I was getting at least some reassurance that most of my code was working together.

I guess I now need to write some systems tests that exercise the entire system (or at least, as much as I can manage without causing the transfer of real money) with an understanding that these tests will be pretty fragile.

More bloody best practice that I would insist upon in any enterprise project, but have avoided on a single-developer, I-am-the-customer-so-no-acceptance-tests-required project.

I blame you.