When we’re testing a mobile application, most of our focus iscentered directly on testing the application. Butthere are other considerations surrounding our applications to consider such as the installation of the application, ongoing phone and application upgrades, as well as how our application interacts with other variables in the mobile environment. There are still more variables including firmware upgrades, memory card space issues and device permissions and settings. If you’ve been in software testing for a long time, these ideas (install, upgrade and interaction testing) seem familiar, even a bit like déjà vu. How would something new such as mobile seem old and familiar?
Back in the days of client server application testing, we had to think about installation testing. In the past decade, we’ve been primarily focused on web testing where installation testing is rarely needed. But in the 1990’s we were focused on desktop software and how gracefully applications could live and perform in a PC environment. For a stretch of time, I worked on a Windows application and I spent a fair amount of time concerned about Microsoft DLL’s, I wrote an article about installation testing and even wrote a blog post entirely focused onDLL hell. DLL hell is an actual term (not just an experience); take a look on Wikipedia for a good description of DLL hell.
The point of my digression to DLL hell is we can learn from the past. In short, DLL versions can conflict, andversion control issues can beencountered on first installations as well assubsequent upgrades of software.Issues arise when different versions of a needed DLL are loaded into memory result inhavoc. DLL hell is messy. I recall spending hours tracing DLL versions and investigating and detecting version issues across a series of commonly-used DLLs.
Now with mobile devices we’re in a new environment but like most computing environments, how long do we believe our environment will remain pristine as we download applications, install widgets, add games and install assorted applications on our portable little computers? It’s time toacknowledge our phones are computers and not just devices. I know I’m currently carrying a phone that has a faster processor speed and more memory than the first laptop I bought. Our phones are powerful. And we are increasingly spending time using applications on our phones.
As testers, it’s time once again to think about our computing “environment.” I use the term pristine environment to encapsulate thinking of a computing environment as clean, untouched – where no installation has gone before – analogous to fresh snow on the ground. But as we know, the land does not remain untouched for long. We take pictures (and fill the phone memory card).We install fun little games we might never install on our laptop computers and in doing so we alter the very computing environment we are using. Somehow on a PC, these installations are much more noticeable – we might need to insert a CD or if we download an application directly from the web, we see message boxes asking our permission to make changes. We are aware (hopefully) that we are changing the environment on our PCs when we perform these activities. But we are not so aware we are altering the environment on our phones as we snap photos and synchronize calendars and store contact data.
I propose in the coming months and years, application and permission conflicts as well as the age-old issues of processor speed and memory will return as variables to be mindful of as our phones become a primary computing environment. We need to think about testing on devices that are “pristine” as well as on devices that are in fact more “normal” meaning the opposite of pristine. The latter aredevices where many applications are loaded, the memory card is nearly full and data synchronization is happening every few minutes or even seconds, GPS software is continually chugging in the background and connectivity is jumping from network to network as we move about expecting everything to work seamlessly and flawlessly. There are numerous variables to consider – a tester’s paradise and nightmare at the same time.
So where do we begin? Start with learning what our mobile applications include. What are the contents of the installation package?By this, I mean file contents and dependencies our mobile application may have; for example, a dependency might be that space is available on a memory card. This is the type of test condition I would test with one or just a couple of devices to see how an application handles a space limit or lack of space entirely (by removing a memory card). Think through and be aware of a mobile application’s dependencies such as the chronic need for Wi-Fi connectivity, and test “what happens if” these dependencies cannot be fulfilled.
It’s important to know what permissions our mobile applications require. Let’s think through an example – imagine we have a mobile application that helps users find store locations and that very functionality – find location – may be dependent on GPS being turned on. What if the user accepts the permission request to use location services but does not actually have location services or GPS turned on? What happens when the user attempts to access the find location feature? What if location services are turned on during the installation of a mobile app but are later disabled?
Thinking through installation and permission dependencies is different than thinking about functional testing conditions. In functional testing, I focus on finding conditions that challenge the logic of an application, whereas in installation, interface and permission testing I’m thinking about how my application interacts with other software or other outside dependencies (such as device settings).
We can download a mobile app incredibly quickly compared to the olden days of client server applications, and yet we cannot easily return to the same test environment because, once an application has been downloaded, we have altered the environment. Since the state of a test environment can be changed so quickly , we should think through these conditions carefully and then mindfully step through these test scenarios. To go back to my fresh snow analogy, we can only walk down an untouched snowy path once before the condition has been changed. I suspect that someday soon, we will have software that allows us to restore a mobile environment (beyond a factory reset) much like we have had software such as Ghost that enabled testers to restore test environments relatively painlessly, quickly and yet thoughtfully.
As a tester, a question to consider is: What are the points of intersection between your app and other apps or other native phone features? Does this sound like interface/interaction testing? This is another form of testing that returns in the mobile environment. It is so easy for me to become entirely focused on testing an application that I forget about the rest of the world, but in mobile testing, while I’m focused on a mobile app, the phone itself may receive a phone call and that incoming call can quickly change the state of the environment. Yes, incoming phone calls and text messages disrupt my train of thought but do these activities disrupt the mobile application I am testing? Can I multitask? Can my application multitask? Can I resume what I was doing with a mobile app when I’m done using the phonefor the old-fashioned need of making or receiving a phone call?
I suggest, at least at times, we put aside our concerns about having hundreds of phone devices on which to test and instead test some conditions at least once or a few times on a smallmost-frequently used set of devices so that we know the answer to an assortment of “what if this happens” conditions. After all, if an application crashes or misbehaves badly enough on a primary environment, we can report the test condition and resulting issue and move onto our next test condition. It may depend on the type of application you are testing or the team you are working on or your own responsibilities within your team, but I would rather test a greater variety of test conditions on a small set of devices than test fewerconditions on a larger variety of devices.
As often is the case with software testing, there are more conditions to test than there may ever be time in the day when the next thing you know it’s time to test the next release or the operating system or another essential variable has made a change. The same is true in the world of mobile; there are firmware upgrades, device upgrades and our own software applications upgrades. Installation testing expands from testing a first-time install versus testing an upgrade. I recently encountered a critical defect when I upgraded a mobile application. So I went “backwards” by removing the software entirely, then downloaded and installed the latest version of the app without going through the upgrade process – the defect did not occur. Some additional testing time and investigation proved the issue only occurred when the app was upgradedversus a first time install.
The good news for a long-time tester like me is I have experience in install, upgrade and interface/interaction testing and I can rapidly bring this background with me in a new frontier – the mobile environment. For testers who are new to the field, this is an opportunity to learn from history as we move forward to yet another new computing platform, the mobile environment.
References
- Better Software September 2007. “Navigating the Installation: A Charter for Effective Install Testing” by Karen N. Johnson: http://karennicolejohnson.com/wp-content/uploads/2009/02/knjohnson-better-software-fall-2007-navigating-the-install.pdf
- DLL Hell from Karen N. Johnson’s blog on the Gold CD see: http://www.testingreflections.com/node/view/6475
- Ghost software: http://en.wikipedia.org/wiki/Ghost_(software)
Author Profile – Karen N. Johnson
Karen N. Johnson is a software test consultant. She is frequent speaker at conferences. Karen is a contributing author to the book, Beautiful Testing by O’Reilly publishers. She has published numerous articles and blogs about her experiences with software testing. She is the co-founder of the WREST workshop, more information on WREST can be found at: http://www.wrestworkshop.com/Home.html Visit her website at: http://www.karennjohnson.com.
No comments yet.