Is It Worth Not Doing It?
Usually, when I’m talking about different requirements practices, I use traceability as a practice that adds quite a bit of value for things like impact analysis. In most shops, though, the cost of developing and maintaining exhaustive traceability, whether real or perceived, far exceeds any potential value. That’s not always the case, though, and sometimes it takes a different approach, risk analysis, to understand how that value reaches the tipping point.
There’s no doubt that developing a complete traceability matrix from the top level vision and scope and business rules down through the analysis models and detailed requirements, then fanning out through the design and implementation and over to the test artifacts at various levels is daunting. Heck, even typing out the span of traceability in the previous sentence boggles the mind. Given the complex relationships that drives us away from a simple spreadsheet for managing this and the massive fan-out from the top to the bottom, we can build a mountain of links. I have heard that the full printed-out traceability matrix for the Boeing 777 flight controls generated 6 feet of paper (though I often say the same about the printed specifications for the Air Traffic Control system I was involved with in the ’90’s, so maybe that’s just a myth – though it is quite feasible).
Even if your life-cycle doesn’t have all these explicit artifacts and stages, there will still be a lot of links. Layer on top of this the complexity of who’s responsible for generating all of these links – basically everyone on the team gets involved at some point – and both development and maintenance of this stuff becomes expensive.
One last thing to add to the fray is a behavioral issue. As you have likely experienced at some point, the minute you go to what should be a trusted source of information to learn something (it might be a dictionary, a requirements specification, a traceability matrix) and find that information to be incorrect or incomplete, it’s credibility is shot. Fool me once, I can live with it, but find a problem a second time and I’ll probably decide to not go back for a third time. All that work, for nothing. If you are going down this rabbit hole, you had better be committed to going all the way. Keep it real and up to date, keep it useful and complete, or keep it away from the team.
All that stated, a well managed traceability matrix does add some value. As you are developing your system, it’s a great way of making sure that nothing is missing, and can be used as a way of measuring progress that can be balanced against that Gantt chart (and we all know how often those are up to date, or even credible!). When a change is proposed or a defect is found, a good matrix will make impact analysis much more straightforward, reducing the likelihood of introducing problems late in the schedule. All good value, but still virtually impossible to convince most shops it’s worth the effort. Again, I rarely even try.
Throw into the fray the issue of risk, and things can get interesting. Certainly on flight control systems, air traffic control systems, many medical systems, or any system where there is potential for loss of life, and traceability is a requirement to be applied to the lifecycle. Even with that as a stated imperative, it can feel as bothersome as flossing teeth and continue to not be an effectively applied practice. What are the risks here?
Well, let’s identify a risk to the project as the risk that the auditor comes to town, sees we are not following the prescribed approach for developing our products, and takes appropriate action. We can even break this down before continuing.
If you are developing products that are under stringent standards such as the FDA, it’s a given that the auditor will come to town for a peek every couple of years or so. If they are swamped and miss a cycle, that’s still every 4 years. Give this a probability of 25% on any given year.
What’s the likelihood that the auditor will catch you in your errant ways? I know from experience that a dental hygienist is 100% guaranteed to know when you are not flossing, but I also know that’s not the case for software systems. I have seen the games that get played, and I know that there are some auditors that are less diligent than others. I also know that the emphasis on software for auditors is increasing, so this portion of the likelihood is increasing as well. Let’s call it 50% and rising (we can’t hide forever).
The taking appropriate action, well, should be a no-brainer. If they see the problem and don’t do anything about it, they’re not being too objective, are they? As long as it is an independent body performing the audit, assume they will take that action. Total likelihood for this risk in a given year is at least 12.5% – and rising.
With risk analysis, our next step is to identify the cost if this risk were to occur. This is where the wide variation occurs, and likely determines whether it is worth driving a traceability matrix for your group. If your audit is from an internal QA representative that has been known to be overridden when push comes to shove, the cost here is zero. If they have the power to shut down your development program until you have demonstrated you have pulled up your traceability socks, we might be looking at anywhere between a week and a few months – do the math. If they decide to take it up a notch and prevent you from selling your product, or prevent your existing client base from using the product they already have, then you, my friend, have a problem.
Take these costs and multiply by that 12.5%, that’s your exposure to this one risk over the next year. When we are talking about stopped sales or shutting down usage, we are easily talking $100k or more, likely into the millions. If you are a real glutton for punishment, there is a good chance that if you are in this market, it is a small one where all the players are well known. Factor in the lost credibility, and there is no longer a decision that even has to be made. You might get away with it this year, or maybe even through the next audit, but this is serious borrowed time.
Of all the practices available in requirements engineering seems to be the most boolean as to whether it makes sense to dive down that path for a given project. If you are asked to do it by an authority with serious teeth, it is a safe bet that the intrinsic value it brings for progress tracking and change management plus the reduced risk of not doing it and dealing with the consequences will decide the issue for you. – JB