Testing the Installer

The installer: A crucially important cog and the the first piece of your organisation’s software the user sees

By Brian J. Noggle

If your organization builds desktop applications, especially ones for consumers to install on their own computers, you probably provide an InstallShield or other wizard designed to unpack and install the application. Your organisation takes great pride in its application, or at least it hopes the application is good enough to earn its keep; hence, your company might dedicate minutes or even whole hours to testing the application. However, somewhere at the end of the process, your company, as an afterthought, creates the installer, an application in itself, and probably releases that application, the first piece of your organisation’s software the user sees, with little or no testing from Quality Assurance professionals.

I realize that this is the second decade of the 21st century. Some of you think that the desktop application is akin to the Model A Ford, and your organisation’s Web-based solutions are the revolutionary Model T. Regardless, many of those advanced Tin Lizzies also use an installer, albeit one that’s designed to unpack onto a Web server. Although this article will talk in the metaphor of the desktop world, you can take some of its ideas, translate them into Web 2.4.1 lingo to sell them to your organisation, and apply the teachings to your situation.

Installers Qua Applications

Commonly, installers perform some or all of the following steps:

• Check the system resources, particularly the hard drive but also the operating system and other dependencies, to ensure the computer can run the installed application.
• Offer the user the chance to choose the location for the installation and to make any customizations to the installed software.
• Ask the user to enter a validation number or unique identifier.
• Ask the user to register the software online so that he or she has access to the latest marketing materials your company offers.
• Tries to trick the user into installing the Yahoo! Toolbar by either hiding that option as the only customization and/or opting to do it by default.

Even though the installer does limited tasks, just by looking through the bullet points above you can begin to imagine tests designed to confirm the installer does what it should and tests designed to see what the installer does when the user steps off the happy path.

Testing System Resources

Within this category, I include not only the hardware specifications your application might require, but also software dependencies such as the operating system or other applications that your application extends (for example, if you’re offering an Eclipse plug-in, the existence of Eclipse itself). Of course, if your installer does not check for dependencies, hey, good luck to your users. Otherwise, you might want to test:

• What if the user installs on an operating system not supported, such as Windows ME?
• What happens if your user tries to install to a hard disk without enough space available?
• What happens if your user tries to install to a hard disk that has enough space available when your installer checks it, but then does not have enough space available when the installer actually tries to write the files to disk? (This test is especially devious.)
• Does the service pack level matter when the user tries to install the application? Will it work in Windows XP without any service packs, or does it expect SP2? Worse, what happens when your application is written to work on SP2, but Microsoft releases SP3?

If you ask your dev team what should happen in many of these circumstances, they might manufacture an answer on the spot. However, you might find unexpected results in many of the cases regardless of what a developer might tell you just to make you go away. You should test.

Testing System Customization

The larger your application is, undoubtedly, the more the user can customize the installed components. This will provide a very diverse set of pathways to test and might extend beyond what you have time to test. At the very least, you should check the behaviour of your application when the user alters the default file path. Some items to look at include:

• What happens if the user tries to install to a mapped network drive? How about a thumb drive or portable hard drive?
• Does the installer handle the file system naming conventions of the system to which you’re installing? Whereas in the olden days of yore, eldritch installers had to accommodate 8.3 filenames, these days this can be limited to only making sure that the user can or cannot enter spaces and other potentially prohibited characters in paths according to operating system limitations.
• Does the application make sure that the folder/directory exists and appropriately behave if it does not? Proper behaviour can include offering to make the folder or alerting the user.
• What happens if the chosen folder is deleted by a user before the installer writes the files? (Do you see how my nefarious mind works?)
• Does the installation or omission of a selected component ensure that its files are and are not installed? You might need to get a list of the expected files that the installer writes to the computer for each check box the wizard offers and check, or at least spot check, to ensure the installer writes only the appropriate files for each option you select.
• If you allow the user to type a filename and path instead of forcing him to use a file chooser, does the edit box handle bad strings? Is the maximum length enforced?

Other Things To Consider

The preceding sections discussed various tasks that the installer performs. However, as part of testng the installer, you should consider basic application behaviour and appearance as well.

• Make sure that the text is spelled correctly and the grammar is appropriate and consistent.
• Review the actual terms of the license agreement. Too often, lazy developers swipe and paste this text from other sources and just change the company name in them. I’m not asking you to make a full legal analysis of this—if you can, you should pressure the organization to do that—but you might spot something worth complaining about within the text such as incorrect addresses or wrong version numbers. If the developer swiped and pasted, strange characters might display. Knock those out.
• Make sure that the Back and Next buttons enable properly and that the steps of the wizard retain information you enter where possible.Web-based installers probably will not support this behaviour well.
• Find out what happens if you install the application over an old version of the software. Does it retain customizations made to the old version and the data from the old version?
• Find out what happens if you install the application over incomplete installations.
• Hopefully your installer tidies up if the user aborts the installation with the cancel button, too.

Reconfiguring and Removing the Software

The installer application often allows the user, after installation, the chance to add/remove components of the software and/or to remove the software from the system. It’s easier to overlook testing out these functions of your installer because, face it, in many cases your organization does basic testing on the installer when installing to test environments, demonstration sites, evaluation customers, and beta testers.

You find these uninstall functions either in the program group for your installed software or in the Windows Add/Remove Software applet (or equivalent). You might want to make a run through these functions to check:

• Make sure the text on these panels is correct.
• If you can add components of the software, ensure that the installer correctly adds the components you select.
• If you remove components, make sure that the components no longer appear and that the other components still work as expected, that is, that the installer did not take away too much.
• Make sure that removing the software removes the software, replaces any changed DLLs, and removes folders and data from the system.
• Make sure that other applications on the machine work as expected, that is, that your uninstallation has not removed core files required by other applications on the system.
• Discover what happens when you install your application to the system after you have removed the application from the same system.

Conclusion

In most organizations, the installation application is a mere coda to the great symphony of the base application development. However, your users will use that coda first. If the installer does not work, works incorrectly or contains design flaws or misspellings that will cause the user to distrust your application, its failure might discredit your software and your organization.

If you have any time at all, you should look at the installer with an eye to its quality and test it as completely as you can. It is, after all, an application, and often one designed and developed with less forethought than your organization’s flagship software