Luke Plant: Test smarter, not harder

“Smarter, not harder” is a saying used in many contexts, but rowing is the context I think I first heard it in, and I still associate it with rowing many years later.

When you look at novice and more experienced rowing crews, it seems particularly appropriate, because the primary difference is not the amount of effort that goes in, nor even the strength of the rowers, but technique. Poor rowers still finish a race absolutely exhausted, but they’ve moved at a fraction of the speed of better crews. Sometimes the effort they put in actually slows the boat down. They tend to make a lot of noise, splash a huge amount of water in every direction, and pull a lot of faces. (I did a lot of all those things when I tried rowing!).

Expert crews, however, do none of these things, because they don’t make you go faster. These rowers do a huge amount of training, and exercise massive amounts of concentration, to ensure that every bit of the (very large) effort they put in is actually contributing to speed.

The “smarter not harder” mindset is also essential for writing good automated software tests.

It’s in this context that religious devotion to things like TDD can be really unhelpful. For many religions, the more painful an activity, and the more you do it, the more meritorious it is — and it may even atone for past misdeeds. If you take that mindset with you into writing tests, you will do a rather bad job.

If writing tests is extremely painful, it may be a sign that something is wrong. Huge and unnecessary quantities of tests are not meritorious, they are a massive maintenance burden. Many of the things that make tests hard to write are also going to make them hard (and therefore expensive) to maintain. I’ve seen far too many examples where it looks like people have just sat back and accepted their painful fate.

For example, good ol’ Uncle Bob seems to have this attitude. He wrote:

you’d better get used to writing lots and lots of tests, no matter what language you are using!

Don’t listen to Uncle Bob! (at least, not on this subject).

“Test smarter, not harder” means:

• Only write necessary tests — specifically, tests whose estimated value is greater than their estimated cost. This is a hard judgement call, of course, but it does mean that at least some of the time you should be saying “it’s not worth it”.

• Write your test code with the functions/methods/classes you wish existed, not the ones you’ve been given. For example, don’t write this:

  self.driver.get(self.live_server_url + reverse("contact_form"))
  self.driver.find_element_by_css_selector('#id_email').send_keys('my@email.com')
  self.driver.find_element_by_css_selector('#id_message').send_keys('Hello')
  self.driver.find_element_by_css_selector('input[type=submit]').click()
  WebDriverWait(self.driver, 10).until(lambda driver: driver.find_element_by_css_selector('body'))
  

  That looks very tedious! Write this instead:

  self.get_url("contact_form")
  self.fill({'#id_email': 'my@email.com',
             '#id_message': 'Hello'})
  self.submit('input[type=submit]')
  

  (Like you can with django-functest, but it’s the principle, not the library, that’s important. If the API you want to use doesn’t exist yet, you still use it, and then make it exist.)

• Don’t write tests for things that can be more effectively tested in other ways, and lean on other correctness methodologies and much as possible. These include:

  • code review
  • static type checking (especially in languages with sound and powerful type systems, with type inference everywhere, giving you a very good cost-benefit ratio)
  • linters like flake8
  • formal methods
  • introspection (like Django’s checks framework)
  • property based testing like hypothesis.

• Move the burden onto the computer. “Push the loop in”.

  Take, for example, a requirement that every entry point to your web app (i.e. a page or HTTP API), apart from a few exceptions like login and reset password, should require authentication.

  The “test harder” religion interprets this as:

  • For every entry point
    • Write a test that
      • Ensures non-authenticated requests return 403

  That’s a lot of tests, and even worse is that you have to remember to write them.

  “Test smarter” says:

  • Write a test that
    • For every entry point
      • Ensures non-authenticated requests return 403

  That’s one test. “Write a test” is executed in developer time, so in the first example the loop (“For every entry point”) is also executed in developer time. Push the loop inside the test, and it gets executed in computer time instead.

  Already mentioned, but hypothesis is a great way to push the loop in. Also, the implementation of the requirements can benefit from the same techniques that the tests do.

• Cheat on your homework. It’s smart to get help, and hard work is for suckers. If you have a good idea, but don’t know the techniques or tools you need to implement it, or whether it is even possible (for example, in the example above you don’t know how to introspect your system to get a list of all entry points), there are a lot of smart people on StackOverflow who will revel in the challenge.

  (Level up: loudly claim on Twitter that “it appears to be impossible to X with tool Y” and mansplainers like me will magically appear with solutions).

Of course, there are still times when hard work is required for writing tests — times when it will be tedious, and times when our instincts to skimp are actually misplaced laziness that will cost more in the long run. But you should hustle and cheat your way out of unnecessary effort as much as you possibly can. You should feel like “I fooled that computer into doing so much work for me!”, not ”My RSI and bleeding fingers have hopefully appeased the testing gods and atoned for my previous omissions”.

from Planet Python
via read more