Pick a Number
More than once in discussion or during a training session, someone has asked me for a number. What is the right number of testers for a development group of this size? What is the right number of levels of management in our organization? What productivity number should I use for this estimate? For each of these questions (and most others of this type), it is possible to quote industry statistics as a response, but the result would usually be worse than to not provide any answer at all. With the diversity of software development organizations, projects and products, it is a rare question that would fit any average number the industry could cough up. In all these cases it is important to dig deeper to find the real questions, and then look internally for the answers.
Rather than looking for the right ratio of testers for your project, consider the degree to which you need to have a product with low defects, the resources you have available, your flexibility for changing the mix of staff, and other factors. A large, high quality product can be produced with little or no dedicated test staff if the development team is diligent in ensuring the product works as expected. Conversely, a relatively small application can suffer from poor coordination between a sloppy development team and hordes of testers. The question to ask is whether the current mix can be expected to generate a product that meets your quality needs or not.
For an organizational hierarchy, the number of levels is usually computed based on the old adage that one person can reasonably manage 7 direct reports (plus or minus 2). In the real world, there are managers that can successfully wrangle a much larger number of direct reports and an incredibly huge hierarchy, while there are others that have trouble managing their own time. If the people at the top of the organizational hierarchy are getting the right information to make appropriate business decisions and the people at the bottom are getting the information that allows them to understand the big picture and where they fit in that picture, chances are that the org-chart is reasonably arranged. If not, you are probably better served to revisit your communication channels rather than to shuffle the pack to try to solve the problem.
Using industry average productivity data is probably the most common form of reliance on numbers to bolster your position in an undefendable way. Your productivity is unlikely to mirror the industry average, and can also vary dramatically from your company’s, or even your own historical performance for a number of reasons. The new project being proposed may be in a new domain, you may be developing additional features on top of an unstable baseline that will haunt you, or you are developing a production unit after years of prototypes. More important than productivity in the early stages of a project is risk – identification of the drivers of uncertainty with any estimate you may come up with. The line in the sand is one thing, but you need to appreciate the huge magnitude of uncertainty and the associated approaches for mitigation to understanding if you are comfortable in moving forward with the project at all.
In all these cases, simply coming up with a number will answer a question but rarely solves a problem. Chances are that with a little thought, you’ll be able to drill down to the root of the problem, identify the real issues and avoid getting caught up in the cheap parlour tricks of industry averages. – JB