Controversies in Software Testing

It can be both exciting and frightening to work in a young field like Software Testing. Exciting, yes, in that our field is so new. We live in an age of adventure, of discovery, of experimentation. Like the early days of the personal computer, we grapple with the best way to do things, with achieving differentiation, while also realizing that if we cost too much, the customer might just go down the street to a cheaper alternative.
Frightening for just the same reason. We work in a field with a real responsibility for life — at least some of us do. Those working on embedded medical, weapons and avionics systems, for example. If we expand that responsibility to mission critical systems, software that if it goes down, we go out of business, the percentage of testers suddenly shoots up. To paraphrase an old poem:

“Because of a bug, the software was lost.
Because of the software, the website was lost.
Because of the website, the company was lost.
Because of the company, jobs were lost.
Because of the jobs, the economy was lost.”

Serious responsibility indeed. Yet we will make mistakes. We know this. Our whole job, our vocation, exists because somewhere else in the chain, someone else will make a mistake. We exist to catch those mistakes.

Oh, frame it as you will — “opportunities for improvement”, “correction of usability issues”, “review and inspection of work products to prevent errors”, wrap it in special paper and put a bow on top, the bottom line is that we exist because imperfect, fallible humans make mistakes. So we are going to make mistakes too. This should not be a shocker.

Now most of our mistakes will be that we forgot to test a feature with a certain kind of input. Oh, the mistake might not be ours; the documentation, the team, and the developers may not have given us any reason to believe the input condition was special. Perhaps we came up with the test condition, but didn’t have enough time, and made a correct judgement call that the risk was minor. We certainly won’t be able to test everything.

In any event, most of the time when we make these mistakes, we hope the software works anyway, because the programmer coded it correctly. In that case, there is no harm and no foul. Sometimes, though, the circle of our mistakes over-lap with the circle of programmer mistakes. Yes, perhaps mistakes is too hard a word. Perhaps we made a risk management call. Perhaps we didn’t triage well. The point is, sometimes a bug slips through.

Working in a new field, a field without legally enforced standards, I can see how some people would want standards; I really can. They defend you in court. You pull out your certifications, your best practices, you say “we did what everyone else said we should do”, and you avoid criminal or civil penalties. Of course, the helicopter still crashed, the website still went down, and the company still failed. I’m not cool with that.

Pre-Mature Standardization

The great advantage to standardization is that we are all doing things the same way. Notice, however, the implied trade off: By doing things the same way, we are saying those benefits outweigh any and all potential learning from experimentation and trying to do things differently.

Sometimes the advantages really do outweigh the costs. Thirty years ago I could walk into any grocery store or gas station, get Double-A batteries, and know they could fit into any electronic device that needed double-A. For the most part these batteries were interchangeable; I could buy a twenty-pack and know I could use them sooner or later on some device or other. When devices went from physical tape to compact disk, to my portable GPS, I could keep pulling from that pack of AA Batteries. And yet …

And yet, as DeMarco and Lister point out in “PeopleWare”, the Double-A battery is an interface standard. When metal touches both ends, the battery needs to discharge 1.5 Volts of Current, but exactly how that works is up to the manufacturer. Double-A is not a product standard, and thanks to that, we have need a steady flow of new technologies, from Zinc-Chloride, to Alkaline, to Lithium. There is additional differentiation in the form of rechargeable batteries, and those have slowly improved over the years.

Notice in software testing, we are not talking about an interface standard, or even a product standard, but instead a process standard, a defined way to do the work. That means we can’t benefit from further exploration in the field.

Let’s be honest: Give a dozen testers the same training on the same test standard, then give them the same piece of mid-complexity software and ask them to generate test ideas, and you’ll get a dozen (or more!) different lists. In reality, the test work isn’t standardized at all, or, at least, it’s standardized only in the most superficial way. People may be filling in the same templates and using the same documents, but the work they are doing is fundamentally different.

Not only do premature standards retard the field, they can’t even keep the promises they seem to make. How should we respond to this challenge? Physicists agree on using some rules (such as Newton’s Laws) to measure and predict at typical ‘human’ sizes. Sure, they fall down at the atomic or celestial level, but rules like:

Distance = OriginalSpeed*Time +
((1/2 Acceleration*Time)^2.)
And
Acceleration = Force/Mass

These are good enough to use to calculate the acceleration for, say, an automobile. At least it’s a start; you can improve that by adding friction and air resistance.

Some folks in our field want testing to be more like Physics, to use formulas like Defect Detection Percentage and Trend Lines to predict performance. Others believe in a more narrative, systems-thinking approach. Brett Pettichord first classified these lines of thinking in his 2003 presentation on “Schools of Software Testing”, which has been revised a time or two since. (Read it online: http://www.io.com/~wazmo/papers/four_schools.pdf )

We aren’t the only field to do this. You are likely familiar with the schools concept in Psychology, but I find the way the Macro-Economists are handling this issue to be most fascinating. It turns out they have two major schools; the Keynesians and the Austrian School.

The Keynesians also want their field to be more like physics, and have mathematical models they use to measure, control, and predict economic performance. Keynesians view interest rates, lending, and central bank policies as dials or levers to use to control economic performance. At least, the Keynesians think they can control performance.

The Austrians believe that taking the economy as an aggregate doesn’t make any sense, that there is a difference between a dollar spent on a government program to dig a ditch and a dollar spent to produce food, valuable material, or that would otherwise benefit society.

If you’ve ever seen a measurement scheme that counts small usability bugs with the same weight as a show-stopping login bug in the same category, you know what I’m talking about.

One thing the Keynesians and the Austrians are doing is dialogue about these differences. Believe it or not, they have a rap video on the subject that manages to be both entertaining and information (http://www.youtube.com/watch?v=GTQnarzmTOc). Yes, it does begin with a joke that is just a little bit crude, but keep watching.

One thing I have learned over the years, is that if I surround myself entirely with people who agree with me, no one is learning or growing.

We have our differences in the software test community. For the most part, people who self-identify in a school generally pick the same school, and you could argue that the classification scheme is unfair. I know what it’s like to have a scheme forced on a larger community by one influential subset, and it ain’t great … but I digress.

The point here is that we are in a young field. We want to experiment with lots of different approaches, but then share, dialogue, debate, and grow.

The word “Controversy” only means that there are things we disagree on. To advance the field, we need to not shy away from these things, but instead embrace them and dive in. Who’s with me?

Author Profile – Matt Heusser

Matt Heusser is a software process naturalist and lead editor of the “How to Reduce the Cost of Software Testing” anthology (Taylor & Francis, September 2011). You can follow Matt on twitter @mheusser or read his blog, Creative Chaos, at http://xndev.blogspot.com