An Interview with Stefan Butlin – Founder of TestPad

We recently came across a new test management tool, TestPad, which takes a different approach to test management.  Along our journey we met and spoke to the founder of TestPad, Stefan Butlin.

What is Testpad?

Testpad is an online tool to plan and execute manual testing, whether that’s making ad-hoc testing less ad-hoc, steering exploratory testing or maintaining detailed lists of test cases.

How is Testpad different?

Testpad is a new take on test management. It moves away from managing test cases one at a time, and instead towards a more flexible checklist approach that makes it much faster to create and maintain scripts.

Test scripts in Testpad are quite free-form and can be used in a wide variety of ways. For example:

• outline guides to steer exploratory testing
• charters for session-based testing
• traditional test cases with detailed steps
• regression tests to cover previous failures
• tests which should be automated but which haven’t been given the time yet
• even just lists of favourite things to check before making a release

Testpad has a unique user-friendly interface. It’s designed to feel and behave like a native keyboard-driven text editor with powerful outlining controls; nothing like a traditional test case manager with the typical one form at a time click-to-edit-click-to-save model. It also has a mobile UI so you can run your tests from the convenience of a touchscreen and with very separate context to the product under test.

Who is Testpad for?

At its most general, Testpad is for anyone who needs to maintain and run a list of tests or checks by hand.

With its script editing speed, it’s especially suitable for use in rapid (or agile) development environments where it’s otherwise hard to keep tests up to date with a continuously evolving product.

Testpad is great for adding control and measurability to exploratory testing without losing the value that an experienced tester can bring.

Currently, Testpad is aimed at smaller software companies that are big enough to have dedicated testers. Most customers to date are Test Managers at companies with 30-100 employees.

Who is Testpad not for?

Currently, Testpad is less suited for use in enterprise. Larger organisations (500+) tend to have the budgets, and emphasis on the mechanics of the process, to justify more expensive and ostensibly rigorous tools. Further, Testpad’s UI doesn’t support groups/departments of users and larger organisations are simply harder to sell to. Having said that, since starting Testpad, I have heard stories from inside two big companies of their having paid big money for the likes of HP’s Quality Center, only to hardly use it in favour of making do with Excel. So, never say never!

At the other end of the scale, Testpad’s subscription pricing model is less suited to tiny teams of 1 or 2 developers whose usage of a test tool will be intermittent. I’m still thinking about this one, so watch this space.

You’re a developer, why create a Test Tool?

My background is more than pure development. In my previous job, I was the CTO of a mobile web business and before that, a designer/manager of a platform for mobile user interfaces. I have therefore experienced first hand the problem of organising test teams and trying to procure the right tools. In every instance, we couldn’t find appropriate tooling and resorted to cobbling together a process built on top of spreadsheets; complete with custom formatting, archiving in wikis, faffing with emailed copies and writing custom VB to layer on some workflow.

For me these frustrations clearly highlighted the gap in the market for a simpler solution to manual testing, one that can keep up with agile product development, and the great opportunity to build a product to help make people’s testing lives easier.

What challenges have you faced creating the tool?

My biggest challenge by far is getting the word out about Testpad – a very different challenge to engineering a product. From talking to customers that have become fans there seems to be an aha! moment when evaluating Testpad. This typically comes when someone tries editing a script themselves and realises how fast it is just to keep typing (hitting enter for new cases, tab/shift-tab for outline structure) and further realising that expected outcomes don’t have to be separate to the steps to execute. The challenge is getting people to that point…

Testpad’s native-like interface is very javascript heavy and relies on a modern browser. During development, most day-to-day testing was on one of Chrome, Safari or Firefox. At least that was true until someone at the BBC who was helping evaluate Testpad first tried it on IE8 and reported it didn’t work at all – doh! Rather than try to support IE8 directly, which would have taken a lot of porting work, Testpad instead prompts for the install of ChromeFrame for users of IE8 and earlier. ChromeFrame is a great solution to get the features of a modern browser into older versions of IE. IE9 worked fine with very little to modify.

In fact, IE8 notwithstanding, after years of cross-browser headaches it is remarkable how consistent the latest crop of browsers are (IE9, Firefox, Chrome and Safari).

What have you learned about testing along the way?

Test practises are evolving all the time. There seems to be a lot of discussion currently about automated testing versus manual testing and the role of ‘traditional’ test teams in ‘modern’ product development. A common theme is the demise of manual testing in the face of increasing popularity and reliance on automation.

From my point of view, I don’t see why it has to be so either-or. Both automated testing and manual testing have a valuable role to play in most projects. Indeed, all of the teams I have talked to over the last year use a mixture of both, with increasing numbers embracing automation at the same time as moving away from fully-scripted manual testing in favour of exploratory testing.

However, whilst the quality of automated testing is rising with increasing sophistication, I think the claimed ‘exploratory testing’ is often closer to being just ad-hoc testing with very variable documentation or process to control it.

When asked by new teams how to go about their testing, my advice is to a) automate as much as possible/cost-effective and b) use exploratory testing for the rest – but where that exploratory testing is guided by a list of feature points and/or more specific scenarios if required.

Using a checklist of aspects/features/scenarios that you don’t want to forget is incredibly re-assuring to testers and managers alike. Teams can go all the way to Session Based Test Management if they like, however, I think the bulk of the value lies in the simple concept of just writing down a list of things that shouldn’t be forgotten.

Obviously I would say that as it’s an ideal use case for Testpad, however, even if teams don’t want to use Testpad, I’d still advocate steering exploratory testing with a simple checklist. I certainly wouldn’t recommend investing in the effort of creating and maintaining fully scripted test cases unless mandated by external requirements. Like automated testing, fully scripted testing will only uncover bugs you predicted might occur – Rumsfeld’s known unknowns if you like, whereas exploratory testing can, and often does, uncover unknown unknowns.

Further, having a simple checklist with a bunch of pass/fails (and bug reports) makes an effective report on the activities of exploratory testing. It gives you and your stakeholders confidence that everything you planned to check has been looked at.

Testpad looks simple, why would someone want to use it?

Well, thanks! It was a key design goal for Testpad to look (and be) simple to use, yet also flexible enough to adapt to the variety of ways people go about their testing; as a friend put it, Testpad is simple in function yet open in purpose. This is being borne out in practise too; I get a lot of feedback saying how much more productive people are finding their testing with Testpad.

Having said that, I do get asked quite often why wouldn’t someone just use a spreadsheet. Indeed, I’ve seen many companies do exactly that, even those that have already paid for expensive test case management tools.

Spreadsheets can be pressed into service for documenting test scripts, but they need a lot of customisation and convention to be useful (cell formatting, document structure, reporting, stats, test assignment, file locations, master copies, bug tracking etc).

In this light, Testpad is the ‘buy’ alternative to the spreadsheet ‘make’ option. Testpad is like a ready-to-go spreadsheet specialised for the purpose of testing. However, if you then consider features like recording test results via mobile touchscreens, delegating tests to other users and guests, multiple report formats and easy linkage to bug trackers, Testpad exceeds what can be achieved with a spreadsheet.

Do you use Testpad to test your own software?

Absolutely!

Testpad uses a mixture of automated and manual testing. The automated tests are based on Selenium and are great for catching regressions in the javascript and server-side code.

Testpad’s manual testing uses a collection of Testpad scripts as detailed guides to steer exploratory testing. If it’s a small change with low risk, just the section headers are used as the guide for a quick smoke test on one browser. If it’s a significant change with higher risk, the whole script is run on each of the four main browsers. An earlier version of one the Testpad self-test scripts is included in the website’s fledgling example library.

Testpad’s mobile UI comes in very handy here. I can load up the test script on an iPad and use it to prompt for tests to run on the browser running on my laptop. The iPad is logged into Testpad’s own account and the laptop browser is logged into a scratch account on the system under test – imagine how annoying it would be to try to test Testpad if the tests kept deleting and editing themselves! So the mobile UI makes for a very convenient way to separate these contexts and keep me sane.

Visit TestPad online – www.ontestpad.com