A Little Track History that goes a Long Way

This article will show you how collecting a little information during each project can provide a better understanding and visualization of how your current project progresses.

Be careful now, “It’s hard to predict – especially about the future” as the Danish proverb goes (Piet Hein) [1]. This approach doesn’t provide the truth or the one-and-only expected outcome. It’s a way of keeping track of your project progress, similar to the GPS tracking in your car. It tells you where you are at this moment now, suggests directions and perhaps even an estimate of when you will reach your destination. In short, don’t follow it blindly.

I use the following properties to guide my tracking of the test progress on waterfall projects. With a little modification it may also apply to you and maybe even to those in agile contexts. It works well for both a project on its own and a program/portfolio of projects with the same end milestone – and even projects with subparts having multiple end dates.

Percentage of completed testcases

In my organization the acceptance criteria for launch is to pass all prepared testcases [2] – or to have an agreed deferral of them. A simpler measure is the exact number of testcases, but one project may have 50 another 300. In order to compare across projects the percentage of completed is used.

The comparison has an expression for the current progress of work through the test cases. Some would say that the best pattern is an s-curve [3], but it really depends on the organization doing the work. Some agile settings use burn-down charts [4] and track either the number of active stories or the number of “done” user stories – that’s more or less the same image.

Testday before launch

Initially I used a “chronological” test day (day 1, day 2…) but then projects have various lengths. By comparing the projects based on “how many days to launch” I can start to see patterns of progress across projects even with different durations. This is also helpful if you are tracking a number of projects, which do not have the same calendar end date – you can use the “test day before launch”.

The number of active open critical defects

Another target is that all critical defects must be closed before launch [2]. Here I use the number of open defects. There is not necessarily a connection between the size of the project (e.g. in testcases or even testdays) and the number of defects found. Large projects may find few, and small projects many – either way each and every one of those pesky bugs must be gone. Tracking the number of open defects can also help to identify if the development organizations are handling the defects sufficiently swiftly. Usually the pattern is that they are piling up in the start, and work their way down in the end, with some average bumps here & there.

The average track

As soon as you have daily data of three or more projects, you probably have a good illustration of typical patterns. I generally use the average values of all projects tracked, perhaps even the three to four latest projects would suffice. One could argue that the optiomal curve would be the “best” – ie 100% passed and 0 defects. Under that rule, any project reaching 100% passed (or 0 defects) any given testday would set the bar for all coming projects. It’s probably somewhere inbetween the “best” and the average, but to be on the safe side I prefer to track new projects against the average of the former projects.

In the samples we can see that even if project C is completing testcases every day, that project is not on track compared to the average, and needs to catch up. Looking at the defect tracking we can see that project C follows the average pattern nicely – “Yes we do have a spike of defects at this time in a project”. Comparing the two graphs though tells us that now that the higest spike of defects is over, an increase in completed testcases should be around the corner…

It’s not a scientific instrument

This is the time to stop – collecting more data! Do be carefull getting carried away with collecting and analyzing the tracking of your projects. The purpose of this tracking tool is to collect just enough data to answer the frequent question “Will we finish on time” [5].

Michael Bolton

“As a tester you are acting as the eyes, ears, nose and fingertips of the client. You are a sensory instrument for your client.”[6]

Links

• http://en.wikipedia.org/wiki/Piet_Hein_(Denmark)
• “When do we stop test” by Michael Bolton – http://www.developsense.com/blog/2009/09/when-do-we-stop-test/
• s-curve http://en.wikipedia.org/wiki/Sigmoid_function
• http://en.wikipedia.org/wiki/Burn_down_chart
• “Will we finish on time” EuroStar2010 guest blog by Jesper L. Ottosen: http://www.eurostarconferences.com/blog-posts/2010/9/14/will-we-finish-on-time—jesper-ottosen.aspx
• attributed to Michael Bolton on the blog by Lynn McKee “Do You Indulge Your Curiosity?” http://www.qualityperspectives.ca/blog/1102

Author Profile – Jesper Lindholt Ottosen

Jesper Lindholt Ottosen is a senior test manager with CSC Denmark. He has 10 years experience of test management in large scale telecommunication enterprise environments. Jesper blogs internally in CSC on testing topics and was a guest blogger for EuroStar2010. Jesper tweets as @jlottosen. This article is his personal recommendation, and do not necessarily reflect those of CSC. The graph data are not real project data, but similar sufficient to make a point.