How Much is Enough?
Quite often in working with clients, questions will pop up in a vein that fits into the general theme of “How much of this should I do, how many of these should I have?”
The questions cover a wide a range of topics. How much work in requirements analysis is enough, what’s the right number of testers to hire, how many documents should we develop, how do I know when I have a reasonable amount of regression tests in place?
In each case, it would be easy to hunt down some industry study that provides what is suggested is the right answer, but this would be a huge disservice to my clients.
It would also be easy to fall back on the standard consultant right answer of ‘it depends’, but that doesn’t help much, either (and you would have to have an insane billable rate to make that statement worth your time…).
If you simply ask an architect how much it would cost to build a 4-bedroom house, you would get ‘it depends’, and it would feel very appropriate, but then you would enter into a conversation geared towards understanding your situation.
Software teams and projects, like homes, come in all different shapes and sizes. Varying product spaces, a wide range of mission criticality, and different cultures, skill sets and collaboration approaches all drive different responses to the questions above.
Is there a way we can go further down the path of understanding how much is enough without the handholding of a consultant or architect?
Perhaps a better way to look at the problem is from the other direction. Rather than asking whether I have enough of this or should we do more of that, ask yourself if you are satisfied with the current results of what you are doing. If you are like most shops, you will probably respond that there may be some minor room for improvement. Perhaps you might respond that there is no way but up from the current situation. Likely somewhere in between.
Assuming there are opportunities to be had, then ask yourself if the particular practice you are considering can add value, or reduce the friction, of the current environment. If so, there is potential value in doing more of it than you are now.
Remember, though, that there are a number of things you could do more of at any point in time, and things you could do less of as well (which may give you the time to do more of the desirable stuff). Rather than forging ahead with the first idea that pops up, take some time to consider the breadth of things you could be doing differently. Pick the low-hanging fruit, do a little more of it than you are doing now, and reap the benefits.
Let’s take an example. Regression testing is a topic that popped up with a client last week. They don’t do any to speak of at the moment, so the question was asked: “What is the right amount of regression testing to perform”?
Questions back to them, in order:
- In the grand scheme of things, are they plagued by some areas of the application breaking when they make fixes to the code, particularly in areas that don’t appear to be related?
- How painful is this compared to other issues such as troublesome integration, unpredictable schedules, incomplete functionality, or the wide range of other issues that hinder any software organization today?
- In that context, is this the biggest issue? Can the magnitude of this issue be quantified in some way?
- Are there certain areas of the system that are hit by this issue more than other areas?
- Is there an understanding in the group of how to go about building regression tests and automating them?
- Can someone be committed to tackling the problem, in this project cycle, for the specific areas of most concern, in a way that the benefits of the effort can be clearly identified (in terms of time savings, reduction of escaped issues, schedule predictability, and so on…)?
If the answers are yes, huge, yes, yes, yes, yes and yes, then it seems pretty clear that building some regression tests for that particular area will likely be beneficial.
Note that we haven’t answered the original question of how much is enough (where the right answer is likely “more than you are doing now”), but we have recast the question to determine where we would most benefit from a change from the status quo.
To try to define in advance what the right mix of practices should be is dangerous and error prone. If I was to say that we needed a set of 100 regression tests to cover the system adequately (or 1000, or 10,000), there would be some serious pushback: we can’t do all that, the schedule won’t allow it! Chances are nothing at all would be done.
If, on the other hand, we see value in doing at least a little of something different, we get less pushback, we identify something that can be accommodated by any project, and we reap the benefits within that project cycle.
Think incrementally – add a little bit of what looks promising, see if there is the benefit you expect. If there is, but there’s still room for more benefit, add a little more. It’s not “How much is enough?” that you should be asking, but rather “Is the current mix of activities giving me the results I am after?” – JB