The concept of Exploratory Testing, a term coined by Cem Kaner in 1983, has been an important development in the field of software testing. By approaching unscripted testing as a discipline with a certain mindset, it can be studied, improved, and taught. (One of the benefits of applying a discipline to an otherwise nebulous subject is the ability for those naturally skilled to share those skills with the rest of us.)
I am relatively new to these concepts and am studying this area in an attempt to improve my skills. One topic I find interesting is the comparison of exploratory testing with its natural antithesis, scripted testing. I have seen comments that some testers have been so enamoured by the benefits of exploratory testing that they planned to abandon scripted testing altogether. This seems like an overreaction to me.
I would like to explore what I see as some of the main reasons for scripted testing, and the potential applicability of exploratory testing.
Reasons for Scripted Testing:
1. A complex new application (or new feature within an existing application) where the complete functionality is not intuitive from the interface alone.
You need a thorough reading of the spec in order to document the procedures required to exercise the new functionality. Exploring the tool interactively is not likely to achieve sufficient coverage.
2. A specialized application which requires domain-specific knowledge to fully exercise.
3. A tester who is new or simply unfamiliar with the application.
In each of these cases a scripted test, created by a knowledgeable tester, allows a less experienced tester to be effective – while gaining valuable experience with the tool and the subject matter. I find executing a test script and stepping through the usage scenarios a more effective way to learn a tool or feature than simply reading documentation. As the lightbulbs of cognition brighten, it will be natural to want to explore further and try your own examples – a good thing! Thus even the less experienced tester may use the test scripts as jumping off points for fertile excursions into extended test scenarios.
4. A new version of a mature product.
This script walks through the existing functionality in a reliable way to confirm that the new development has not broken the legacy code. Note that this script is not testing the new functionality, but rather regression tests the functionality that existed in the last release.
Exploratory testing does not seem like an effective substitute for scripted testing when considering legacy functionality. The likelihood of finding as yet undiscovered defects in released code is lower, and care must be taken not to lose coverage offered by the scripts. Expounding on these scenarios into newly added functionality may be fruitful – but then that no longer fits into this category. However, regression testing like this could really benefit from automation.
5. Specific customer usage scenarios, especially where problems were reported in earlier versions.
This is a special case of the regression testing of legacy code discussed above, with the added gravity that they were supplied by or derived from a customer, possibly when a previous new release failed with serious consequences for them.
If the removal of one of these tests allowed a similar problem to slip by, you might as well stick an apple in your mouth, as the customer and your VP of Engineering will take turns roasting you on a spit.
6. The project schedule does not allow time to wait until “Dev complete” to begin testing.
Ideally, you work from the spec to create a test script which you will execute when the code is released for testing.
It’s a natural tendency to want to see and explore the application as you read about a feature. And sometimes it can be a little difficult determining exactly how the software will react based only on a spec.
But the realities of compressed product cycles and tight schedules often make it necessary to overlap development and testing. A benefit is that you are more likely to get a clean, well formatted test script with good coverage of the documented requirements. Drawbacks include:
- The application may not function exactly as you interpreted from the spec.
- Design decisions may have been made during development that deviated from the spec.
The tester will need to work these out with the developer – and encourage a spec update.
Finally, when a test is developed from a spec you really should plan on a period of exploratory testing. No matter how detailed the spec, once you get your hands on the new feature you will likely see areas worth exploring further, which were not clear to you from a document with some screen shots and some text.
7. To facilitate outsourced testing.
As discussed earlier, testers unfamiliar with the product can be immediately productive and reasonably thorough, while also learning the application, if provided with a script. The next step in development of both internal and outsourced testers will be getting them writing their own test plans, helped by having already seen examples of good style and form in the test plans you supplied them. And again, this can serve as an excellent preamble to their own exploratory testing.
8. A test script that performs a walkthrough of basic functionality of a new feature can be useful as an Acceptance Test.
This may be executed at various stages during the development cycle, later serving as the basis for a regression test.
Conclusion
Exploratory Testing is built on the premise that there is much to be gained by venturing out of the sanctuary of pre-scripted tests. But can exploratory testing replace scripted testing? I believe that we have presented several cases where:
- It is useful to have a test script.
- It is beneficial to test following a script.
- Those same benefits cannot be achieved with exploratory testing.
And yet in each case exploratory testing offers synergistic benefits. It seems clear to me, then, that neither technique can completely replace the other. In fact, they are complementary, and both deserve a place in your arsenal of software testing tools.
Author Profile – Sean Morley
Sean Morley is a Principle Quality Engineer for Dassault Systemes. Having worked in a variety of roles on software tools for electronic design and test, he is currently focused on the testing of engineering software. Sean participates in test related forums and blogs at www.testyengineer.com in a concerted effort to improve his knowledge of the field and share his experiences. He appreciates all feedback, recommendations, and challenges as opportunities for growth, and can be contacted at seansync@gmail.com.
No comments yet.