Analysis, Design Tips

There's no doubt that developing a complete traceability matrix from the top level vision and scope and business rules down through the analysis models and detailed requirements, then fanning out through the design and implementation and over to the test artifacts at various levels is daunting. Heck, even typing out the span of traceability in the previous sentence boggles the mind. Given the complex relationships that drives us away from a simple spreadsheet for managing this and the massive fan-out from the top to the bottom, we can build a mountain of links. I have heard that the full printed-out traceability matrix for the Boeing 777 flight controls generated 6 feet of paper (though I often say the same about the printed specifications for the Air Traffic Control system I was involved with in the '90's, so maybe that's just a myth - though it is quite feasible).

Even if your life-cycle doesn't have all these explicit artifacts and stages, there will still be a lot of links. Layer on top of this the complexity of who's responsible for generating all of these links - basically everyone on the team gets involved at some point - and both development and maintenance of this stuff becomes expensive.

One last thing to add to the fray is a behavioral issue. As you have likely experienced at some point, the minute you go to what should be a trusted source of information to learn something (it might be a dictionary, a requirements specification, a traceability matrix) and find that information to be incorrect or incomplete, it's credibility is shot. Fool me once, I can live with it, but find a problem a second time and I'll probably decide to not go back for a third time. All that work, for nothing. If you are going down this rabbit hole, you had better be committed to going all the way. Keep it real and up to date, keep it useful and complete, or keep it away from the team.

All that stated, a well managed traceability matrix does add some value. As you are developing your system, it's a great way of making sure that nothing is missing, and can be used as a way of measuring progress that can be balanced against that Gantt chart (and we all know how often those are up to date, or even credible!). When a change is proposed or a defect is found, a good matrix will make impact analysis much more straightforward, reducing the likelihood of introducing problems late in the schedule. All good value, but still virtually impossible to convince most shops it's worth the effort. Again, I rarely even try.

Throw into the fray the issue of risk, and things can get interesting. Certainly on flight control systems, air traffic control systems, many medical systems, or any system where there is potential for loss of life, and traceability is a requirement to be applied to the lifecycle. Even with that as a stated imperative, it can feel as bothersome as flossing teeth and continue to not be an effectively applied practice. What are the risks here?

Well, let's identify a risk to the project as the risk that the auditor comes to town, sees we are not following the prescribed approach for developing our products, and takes appropriate action. We can even break this down before continuing.

If you are developing products that are under stringent standards such as the FDA, it's a given that the auditor will come to town for a peek every couple of years or so. If they are swamped and miss a cycle, that's still every 4 years. Give this a probability of 25% on any given year.

What's the likelihood that the auditor will catch you in your errant ways? I know from experience that a dental hygienist is 100% guaranteed to know when you are not flossing, but I also know that's not the case for software systems. I have seen the games that get played, and I know that there are some auditors that are less diligent than others. I also know that the emphasis on software for auditors is increasing, so this portion of the likelihood is increasing as well. Let's call it 50% and rising (we can't hide forever).

The taking appropriate action, well, should be a no-brainer. If they see the problem and don't do anything about it, they're not being too objective, are they? As long as it is an independent body performing the audit, assume they will take that action. Total likelihood for this risk in a given year is at least 12.5% - and rising.

With risk analysis, our next step is to identify the cost if this risk were to occur. This is where the wide variation occurs, and likely determines whether it is worth driving a traceability matrix for your group. If your audit is from an internal QA representative that has been known to be overridden when push comes to shove, the cost here is zero. If they have the power to shut down your development program until you have demonstrated you have pulled up your traceability socks, we might be looking at anywhere between a week and a few months - do the math. If they decide to take it up a notch and prevent you from selling your product, or prevent your existing client base from using the product they already have, then you, my friend, have a problem.

Take these costs and multiply by that 12.5%, that's your exposure to this one risk over the next year. When we are talking about stopped sales or shutting down usage, we are easily talking $100k or more, likely into the millions. If you are a real glutton for punishment, there is a good chance that if you are in this market, it is a small one where all the players are well known. Factor in the lost credibility, and there is no longer a decision that even has to be made. You might get away with it this year, or maybe even through the next audit, but this is serious borrowed time.

I had lunch with a couple of people driving the technology side of a successful business a few days ago, and it was a familiar story. They had challenges getting everything done that they were being asked to do, but it wasn’t simply a function of their skills or capacity. The marketing team had a real propensity for suggesting that everything was a top priority, that it didn’t matter what else might be needed. Priorities seemed to shuffle all the time, there had been a number of urgent things that just had to be done that turned out not so urgent after all, and through all of this, the perception was that the development team couldn’t do their job. In my experience, this is more the norm than the exception.

What is needed in a situation like this is recognition that the effort and risk associated with adding a feature to the system is just as real and important as the relative value of having this feature available. Two very different, real and important perspectives that must be balanced together to allow the business to move ahead with the best combination of value for effort. If you let the marketing side call the shots you are likely to build very expensive features when simpler ones will do, or take on too much risk when you don’t need to. Let the tech side call the shots, and you might get easy-to-build features or cool technology, but will likely miss the value expectations.

A simple spreadsheet is the weapon of choice for managing this duel, and it is easy to implement and maintain (the original idea comes from Karl Wiegers, the essence of it is explained here). Start off by listing all of the things that are discretionary, but someone wants in the system. Remember, just because someone says they want it, does not make it a mandatory feature – the list of mandatory features is likely a lot shorter than you think, and unless it is something driven by a business rule such as external legislation, I would tend to put it in the discretionary list to start. Don’t get caught in the trap of wanting everything your competitor has, or you will never get a chance to build something unique. All the potential features, major architectural elements, anything else that might fit at that magnitude, whether it is requested by a user proxy or by an internal stakeholder such as an architect.

You will find there is a reasonable level to balance this list. At too high a level, you will miss out on some opportunities to partially implement major features, so your flexibility diminishes. At too low a level, you will have an administration nightmare on your hands, and you also run the risk that you could prioritize out something that is needed by a feature that you want to build. I find that if you work around the level of use cases or scenarios within these use cases, you will do well.

Take this list, and let each side fill in their own scores for the features. Each one is scored on a 1-9 relative scale, there’s no need to be too precise (don’t mess with larger or smaller numbers, fractional numbers or anything like that). One of the things I particularly like about this system is that if one side tries to game the numbers by ranking all of the features very high or very low, they essentially lose their voice in the overall results. Nothing like a game where the rules force you to play fairly.

The customer representative rates each feature on two dimensions: the relative benefit of the feature being included in the system, and the relative penalty if the feature is not implemented. A feature will have a high relative benefit if it is seen as a differentiating feature (if you ask Tiger Woods or Kelly Ripa, OnStar automotive navigation systems would fall into this category), and would have a high penalty if the feature is expected in that class of products (for those same cars, cup holders or a good sound system or tinted glass would be expected). Interestingly, something that starts out having a high benefit in a class of products will eventually migrate to having a high penalty if absent over time, as the competition all picks up on the good features – just try to buy a new car without onboard navigation in ten years.

The technology team team then ranks the same features from their perspective: the relative cost of implementing the feature, and the relative risk involved in doing so. While the cost appears straightforward, be careful to consider the full lifecycle costs for implementing the feature, rather than just the code and test effort. For the risk, this can include the uncertainty in the cost (remember that no estimate should be a single-point number), and the risk to the architecture caused by injecting this feature at this time. A big feature, added to the system late in the game, can be a significant risk. This is just one example of how any of these numbers can evolve over time.

With all the numbers input, let the spreadsheet to what it does best. Sum the benefit and penalty for each feature, divide by the sum of the cost and risk. Sort the spreadsheet by this resulting column, and you have converted different perspectives into important information. Near the top of the list (sorting highest to lowest) will be a collection of features with an attractive balance of high value and low cost: where you want to be with your work. Near the bottom will be those things you want to stay away from: costly or risky items that don’t add a lot of value. In the middle will be those features where it is appropriate to have the rich discussions and debate.

I’ve seen clients take this basic form and extend it in many directions. Once you are comfortable with the basics, you may want to weigh one or more of the columns differently (you may want to bias the list to be more strongly differentiating, or less sensitive to risk, for example). One team added columns with absolute costs and uncertainties, so the spreadsheet could give them their first cut at effort automatically. Another team found they had a couple of major clients that drove their prioritization, so they augmented the basic formula to include weightings for these clients. Whatever you do, make the adjustments and agree to them in advance. Debate the model first to remove the politics, then deal with the results afterwards.

This list can be large, but that is ok. Once past the initial hurdle of populating the list the first time, things get a lot easier. This is something best considered as an ongoing management tool rather than a one-shot exercise, so plan to keep it around. Indeed, this prioritization list serves as a great backdrop for ongoing change management, and has been used as a more refined backlog for teams using a scrum-like approach for developing their products.

Ah, the eternal hardware/software debate. Almost all of the books on requirements have ‘Software’ in the title, or are filled with examples from software applications, so there is a natural tendency for hardware (and firmware and drivers) developers to think that the content doesn’t apply.

Almost by definition, though, when we are talking about requirements we are talking about the essential, externally discernable behavior of a system, something that should be captured and expressed independently of the implementation to artificially constrain our options. We should be able to completely describe the needs of the stakeholders even before we decide whether to implement the solution in hardware, software, firmware, some combination thereof, or purchase the solution from an outside vendor (whether off the shelf or custom development). Capturing the requirements of the system helps us understand which direction we should take the solution, though in most shops the thinking is reversed – we already have the solution in mind, and it severely biases our thinking.

With that in mind, there is still much that can apply in these books and courses, even if the title suggests software. Use cases might diminish in value for hardware unless there is a rich sequence of steps involved in the device interacting with external systems (though there are some that would disagree), but other elements will definitely be relevant. An overall vision and scope for the project remains essential to set the bounding box and priorities for the project.

Some of the analysis techniques will be more relevant than others – state transition analysis will usually be important, often more important in embedded systems or drivers. Quality requirements will be essential (though, as with software, are often poorly expressed) and business rules (such as electrical standards) and interface requirements will need to be considered.

In hardware, the tradition seems to be that instead of detailed functional requirements, the equivalent information is captured in design documents, but I see that only as a terminology issue – call it what you will. Hardware developers are also far more comfortable with breadboards and other prototypes than their software peers, even if the practice provides great insight in either domain.

One of the key points is that on the hardware side, there is a stronger appreciation for the need to capture the requirements as effectively as possible, as the cost of errors is more readily apparent – throwing away a production run is much more tangible. But still, there has been pushback against the training.

Particularly in the software world, we are inundated by absolutes. Someone will argue that one platform is always the best, someone else will unequivocally state that outsourcing never works. Always this, never that: a barrage of absolute positions that take on the fervor of religious debate. I have learned that the only right answer really is "it depends", in every situation save one. There is only one absolute: that everything else depends on a clear, common understanding of the context of the situation.

All the other absolutes are hogwash. Decisions about which methodology to use, which language to program in, or even what the right ratio of testers to developers is, are often made as absolutes. While this might work some of the time, there will also be times where this will burn you.

These absolute decisions are made for all the wrong reasons. The whole industry is headed in that direction, so we had better jump on the bandwagon as well (think offshore outsourcing 5 years ago, think Agile today). It may be the belief that if it worked on the previous project, it will work for the next one. Perhaps it is the only approach that the team knows, so there isn’t even an awareness that a decision is being made. One of the most damaging reasons can be that it is the only approach that the consultant cares to pitch – and every prospect seems to need that solution, if they are inclined to pay for it (think the same two examples and timeframes as before...).

The only absolute about software development that I have seen is that you absolutely need to understand the context of the situation if you are at all interested in successful decisions. Unfortunately, most teams fail to take the time to understand this context before making these decisions, and they pay the price. They may have already promised the impossible, or they may be trapped in the flawed perception that if the developers aren’t busy slinging code, the project is not yet progressing. It is far too easy to be busy, but headed in entirely the wrong direction.

I have been working with a company recently as they head down the path of developing a product for the first time, after years as a service-based group. We are engaged in analysis at this point, and have already stepped back from a debate about the relative importance of a list of features to a higher level discussion of user classes, use cases, and corresponding dialog maps and screen shots. While we have managed to channel discussions more clearly in a fruitful direction, there have been signs that we need to step back yet again. There are at least 3 different user classes or external systems that have been uncovered recently that nobody had thought of yet (how many more are there?). It was recently unearthed that the scope of the first release was seen as one thing to some of the people, something entirely different to others (what is in scope for the first round, and what to the subsequent releases look like?).

While we were moving more effectively than before, we were still stumbling, and all of those stumbles had to do with a missing piece of the system context. Everyone involved in making a decision has to have a shared understanding of that context. Getting to that point is a nontrivial exercise, but it is far better to develop that common understanding in a focused effort than it is to discover elements of this context throughout the project lifecycle, often when the cost of this discovery is significantly higher (not that it is ever measured, of course). Even for the largest of projects, a good understanding of the context would be enough information to barely fill a dozen pages, but the fan-out of ramifications is enormous. This context has several components.

There is the product/business context: identifying the specific boundaries of the system (the things I am developing or manipulating are internal, the things or people I interact with are external), the need we are trying to fill, and what makes us different from what already exists out there. Identifying what success for the business looks like at the end of the day – holding our feet to the fire, in terms of expectations.

There is the project/resources context (given that, particularly for large or complex products, it will take several projects to develop everything we want): knowing the constraints I need to work within, the gaps in my resources that I will need to work around, the risks that I will face, preferably by mitigating them rather than being broadsided by them.

It comes as a surprise to many people when I suggest that the requirements analyst (or whoever has the role of ensuring the requirements are captured, regardless of their title) is not the author of the requirements. I don’t come by that declaration lightly – I have been the designated analyst and tried to give birth to that tome myself, and from all my travels in a wide range of teams, the default assumption is usually that the analyst writes the spec. The result in most cases is very similar to that birthing hut analogy: the analyst hides away and struggles dearly with the task of capturing the requirements (and there is rarely any epidural or nitrous oxide to take the edge off), while the rest of the team waits expectantly outside. They are cooling their heels, or working on something that may end up being irrelevant in the end. When the spec comes out, nobody dares note that it is an ugly baby: they will make do with what they have, and the project will struggle along. Half of all the defects throughout the project will be the result of missing or otherwise flawed requirements, but few will make that connection.

To suggest the analyst is the author of the spec is like saying the project manager writes the code on a project. It might work for the smallest of projects or simple little prototypes, but certainly does not scale. Thinking in these terms is just as bad for everyone else on the team as it is for the analyst: their expectation is that they stay on the outside, and only receive the finished product. I once had a corporate lawyer ask me how she could possibly have any interest or role in the requirements stage of a project. I asked what she did, she replied that she was in charge of ensuring the company was up to date with tax issues worldwide. I asked how she got involved with projects, and she noted that there is usually a frantic call in the eleventh hour of a project, and she has to rush in to help clean up the mess. It didn’t take much discussion to convince her there might be a better way.

The analyst’s job is to coordinate the entire team, to drive a deep collaboration, out of which the team has arrived at a common understanding of what needs to be built. At most, the analyst may at times act as a scribe, capturing the elements of the discussions that are held, the different forms of analysis and elicitation that are components of that common understanding. First and foremost, though, the analyst brings an understanding of the structure of a well crafted set of requirements, and deploys the range of tools as a mentor, bringing expertise from all stakeholder communities together as required for each piece of the system.

She’ll bring the business representatives together with the product champions to understand why this product is being built at all, to put some structure around the boundaries of the project, to hold their collective feet to the fire and quantify what success means to all parties. Out of this context, she will pull together representatives from all external stakeholder communities and gather information from as wide a range of sources as makes sense. More than just asking what they want, she’ll drive discussions down the path of identifying the goals they want to achieve, but will also look at competitive products, defect reports, and will also solicit an understanding of what is possible or even feasible with given technology.

She will then make sure that everyone has an appropriate perspective into the needs of the system, using whatever means is appropriate. She will ensure that the essential data relationships are understood, that the user interface has been clarified, that the objects with states are well understood. She’ll make sure that the tax lawyers or the people managing SOX compliance get their say at this early stage. She’ll look at the team structure and ensure that all this information is expressed in enough detail to ensure that the product can be implemented with minimal risk.

I was recently working with a client that develops and deploys large-scale, complex solutions. Their product consists of custom and COTS hardware, embedded software, networks, and server-side installations.

At one point, we entered into a discussion that included the age-old requirements vs. design concern, the idea of system decomposition, and the nature of responsibility between the analyst and the architect for their systems. Each of these ideas were a source of debate within their organization, and with no final decision, the waters were sometimes murky.

Through discussion with the team, it gradually became clear that this complex mass of topic could indeed be sorted out. The core of the matter is the area of system decomposition, and once we get a handle on how the organization needs to manage that, everything else falls into place.

For the systems they build (and for any large-scale, complex product), it makes sense to consider first the requirements of the system as a whole. The end user is far less concerned that the product contains this widget or that doodad, they are concerned primarily that the system meets their overall needs. We need to perform that analysis to ensure we have identified and prioritized their needs, and captured the features we will be providing as embodied in our product. We do this analysis as if our system is a black box, without regard to the internal makeup of the system (unless there are specific constraints that may drive some elements).

Once we are clear on what we need to build, we get to the point where much of the confusion arises. To this point, we have solely been in the domain of analysis, and generally it is the analyst that drives the show. From here, though we need to start to break apart that black box, and devise an architecture for our system that will meet our end user’s needs. We layout the division of responsibility between hardware and software, identify firmware components and network architecture. We are performing a system design, and the architect is generally driving the show here.

The key thing to note here, though, is that in performing a system design, we are in turn specifying the subsystem requirements at a high level. Each subsystem that we identify in this design activity inherits a collection of the system-level requirements that we originally analyzed and specified. At times, several subsystems will need to share and collaborate to support these system requirements, at times a subsystem will be able to manage a higher level requirement on its own. We are simultaneously designing the system and analyzing the requirements of the subsystems. It is not an either-or, it is both in conjunction, depending on your perspective (am I architecting the system, or am I analyzing the subsystem?).

For those subsystems that we need to develop internally, this will kick off a sub-project that starts with a fleshing-out of the requirements for that component, be it hardware or software. There is potential, if the system is comprehensive enough, that this subdivision needs to be performed recursively until we get to a point where we can bite off a chunk and say “I can go build this”. For some subsystems, the allocated requirements will become purchasing criteria for COTS products, or selection criteria if we have a standard library of reusable components to draw from.

Who is responsible for this at any point in time? While it is simpler in theory to try to suggest that the analyst starts the ball rolling, then hands off to the architect, this is not a reasonable reflection of reality. It is much more a coordinated effort throughout, with each participant taking responsibility for one portion of the effort. Initially, the analyst will drive things to ensure that the system as a whole is identified to meet the end-user’s needs, but this is done in conjunction with the architect, who will have better insight into concerns such as feasibility, existing components that can be leveraged, and so on. Indeed, to do the job right, the analyst will rely on representatives from all stakeholder communities to ensure all their needs are met.

When we get to the point of decomposing the system into its components, the architect will take the lead to ensure that the components will all work together effectively to support the system needs, but the analyst will be right there to ensure that each component is reasonably specified. It is not simply a matter of passing requirements down to their respective subsystems here. As the system is subdivided, we are actually creating new interfaces, and deriving interface requirements between these components that the analyst must capture.

As an example, if the system needs to shut down when it gets too hot, this gets allocated down to a hardware sensor somewhere in the system and the software that needs to monitor the temperature and perform a controlled shutdown when necessary. On top of that, though, we discover new requirements that are not part of the system analysis. That software needs to know what register to check to read the temperature, it needs to know how to convert that reading to an actual temperature for comparison, it needs to talk to other software components or other systems on the network to tell them to shut down or provide status, and it needs to be able to read the register and not be blocked by other processes along the way. All requirements, derived at this subsystem level.

From there, we looked at different techniques for narrowing the list of attributes and prioritizing the remaining ones, to make our task easier and to ensure that we sufficiently specify the requirements for the areas that are most important to us.

Last week, we took this list and performed a critical translation: from the set of terms that gives us reasonable coverage for quality, to a different set of terms that can be expressed in a quantified form.

Finally, we look at the characteristics of good requirements, provide some tips on how these apply to the approach to quality requirements we are expressing here, and tie the whole process together.

Karl Wiegers suggests 10 characteristics that you should consider when developing your requirements: complete, consistent, correct, feasible, modifiable, necessary, prioritized, traceable, unambiguous, and verifiable. While you will never have a perfect specification, I have found that using this list while reviewing requirements can dramatically improve the results.

To some degree, we deal with several characteristics simply by following the process outlined in this series. Our initial range of quality attributes and disciplined breakdown supports completeness, and the series of steps we take give us a natural traceability that we can retain for future review. If we retain the rationale as part of this process, we have improved the modifiability of the requirements: we will be able to refer back to understand why we made the choices we did, and have better confidence in our modifications as a result.

Prioritization is addressed, but it makes sense to further prioritize your quality requirements in the context of all the requirements to ensure you are focusing on the most important areas of your system. Similarly, the approach supports consistency and necessity by deriving criteria from the attributes, but again these should be considered in the broader context as well.

The final four characteristics (correctness, feasibility, unambiguity, verifiability) all relate to the specific statements we generate from the criteria we have selected. Having translated to measurable criteria helps us for verifiability, but these four characteristics are closely related to one another. There are a number of points to consider when expressing the requirements:

• Ensure the statements are broken down so that each is uniquely testable.
• Carefully express the conditions surrounding the requirements, and be sure the statements are written in terms that can be controlled in a test environment.
• Express valid ranges and boundary conditions where necessary.
• Specify both expected behavior as well as how the system should behave in exceptional conditions (indeed, this is a key part of expressing quality requirements).
• Avoid ambiguous terms (such as several, if possible…) or expressing requirements that simply are not feasible (such as instantaneous response).

There are other points to consider beyond these, but you get the idea. Once we have followed the first few steps of this approach, we are left with a set of terms that allow us to express the quality of our system in a manner that is identical to the way we express the functionality.

It is important to follow all of the steps in this series - skip any one and you open yourself up for all kinds of grief:

• If you fail to start with a broad taxonomy for overall software quality, there is potential for missing an entire area of quality that may be critical. Often, neglected areas of quality are of interest to internal stakeholders: there are no quality requirements to ensure that the system is developed to be testable or maintainable, for example. Left to chance, systems are rarely built to satisfy these critical areas.
• If you fail to cull the irrelevant attributes and prioritize the remainder, you are making the entire effort of developing quality requirements more difficult than it has to be. Given a finite amount of time to develop requirements, you will be less likely to adequately cover the areas that are most important to you, diminishing the value of the effort overall.
• If you fail to make the translation from the broad quality attributes to a set of criteria, you will have a tough time expressing the quality of your system in terms that can be objectively verifiable. In addition, failure to make this translation as a conscious part of the overall process will make it more difficult for you to identify criteria that support several of your key attributes at the same time.
• If you fail to specify these criteria with due consideration to the characteristics of good requirements, you will have gone through this effort and still not developed a solid spec. Indeed, while the first few steps of this process are the equivalent of using reasonable analysis models to derive the functional requirements, this last step of ensuring the statements adhere to these characteristics is common across all requirements.

Much of the effort for identifying quality attributes for a project can be done in a single collaborative session with appropriate stakeholder representatives. Certainly, the narrowing of the broad list and prioritization of those remaining attributes is achievable in an hour’s moderated session. From there, an initial translation may also take place in this group session, or different representatives can be tasked with actions to clarify specific criteria that will support the important quality needs for this project. At this point, simply follow the review and refinement process you use for your traditional functional requirements (you do have one, don’t you?).

Unfortunately, the terms or attributes we have used to this point are not easily framed in a quantifiable fashion. To specify "the system shall be user friendly" brings us no closer to testable requirements, and a reasonable approach to getting to more appropriate terms is missing from many of today’s prominent sources of information on requirements.

This critical step is actually one that has been around for decades, and is more often found in books related to software quality or quality engineering than to books focused on software requirements. What we need to do here is perform a translation, from the landscape of quality that has ensured we are properly covered, to a corresponding set of criteria that will allow us to specify our desired in appropriate, testable terms.

There are many of these testable criteria that we can choose from. They come from a variety of sources, and it is a safe be that you have identified many of them as specific measures for projects in the past. They can range from specified GUI standards and response times that can support usability, Mean Time to Between Failures and Mean Time to Repair that address availability, and many others.

A couple of key points here. Generally, you will need to identify more than one of these criteria to give you coverage of any one aspect of quality (otherwise, we would simply specify our quality requirements in the original terms we had previously identified). Secondly, most of the criteria we can use will at least partially satisfy more than one of these aspects of quality at the same time (specified error handling mechanisms can support robustness, usability, interoperability and safety, for example). Clearly, if a criterion at least partially satisfies two or more of the attributes we deem as important, it is an attractive one to use.

To aid in the identification of appropriate criteria for your project, I have provided a mapping between the attributes we have been discussing over the past few weeks and a set of criteria that has grown as more sources are identified. This is captured in a spreadsheet that maps these relationships, and supports all the other steps identified in this series. It can be found here, and is free to download and use on your projects. This is by no means closure on the list of possible criteria: grow the table based on your experience, and it can be a strong resource for you and the rest of the organization for future projects.

This table has grown from a number of different sources. When reviewing current requirements-based books for quality requirements, we get a few examples of terms that are used for specific quality attributes (such as response times for usability), but nowhere near a list that covers all criteria, and no discussion of the need to perform the orthogonal mapping identified here. Indeed, in one popular text, the author simply laments "the fuzzy notion of usability". Instead, we need to dive into Quality based references, such as Software Quality Assurance by Daniel Galin or Software Quality Engineering by Michael S. Deutsch and Ronald R. Willis, for a more complete set.

There are some very interesting candidate criteria in this table, many of which help underline the notion that we should consider quality for the system, rather than simply for the software component of that system. Imagine, for example, a system requirement to support maintainability and reusability that specifies key documents that are maintained and readily available in a specified form and location, for a specified time. While not testable by running code, this is a strong requirement for a system that will be fielded and maintained for decades.

All this is a long-winded way of saying that once we have identified the important attributes of quality for our project (as discussed in the previous two weeks), we need to then identify a set of measurable criteria that will give us reasonable coverage of these attributes. These will provide the terms we use to identify our measurable quality requirements, and as with the functional requirements, we need to continue to specify these elements until we have reached a point where we have adequately addressed the quality needs of our system.

As noted last week, the landscape of quality attributes I tend to start with comes from Karl Wiegers (based on Software Requirements, 2nd Edition, and expanded based on his experience), and consists of the following: reliability, usability, integrity, efficiency, interoperability, robustness, safety, availability, flexibility, maintainability, testability, portability, reusability, and installability. The last 5 of these are of interest to the developer, the first 9 to the user. Safety and installability were added based on experience.

For any specific project, what is important across this landscape of quality will vary tremendously. Embedded systems may have critical efficiency requirements given limitations of space and power, but usability considerations may be completely irrelevant. Conversely, while handheld applications may see similar efficiency requirements to those of traditional embedded systems, usability will be an important consideration for adoption and differentiation from the competition.

An effective way of prioritizing the range of quality attributes is to tackle the problem in two stages. This makes the process efficient, but still protects you from missing any important elements (as may occur if you start out with an abbreviated list).

For these prioritization steps, it is critical that all stakeholder communities are involved in the discussion. A common mistake in requirements analysis is to decide on requirements by acting for a stakeholder that you don’t adequately represent, and it is very rare for an analyst to be able to authoritatively make decisions for the broad range of categories we are considering here.

The first stage is to consider each of the attributes in turn, and to determine if there is any current knowledge that would make that attribute clearly in or clearly out of scope. Not all of the attributes can be simultaneously optimized, and indeed, some of the areas of quality are at odds with one another (for example, a system that is build to accommodate high efficiency will inherently be less maintainable). Generally, what you will find is that at least half of all the attributes are quite clearly in or out, but the others will require some discussion.

It is these discussions, with appropriate stakeholder representation, that is important. While it may be appealing to skip this step by using a shorter list (based on the products you generally build), this results in a very small time savings but increases the risk of overlooking critical requirements.

As training examples of potentially surprising requirements, we often discuss a Cafeteria Ordering System, an online system that will allow employees to order food and have it delivered, to save time. While the criteria of safety may seem irrelevant to most that participate in the discussion, allergies can be a cause for concern, and drive requirements such as the need to post ingredients for all meals. Similarly, interoperability would appear to be easily handled through straightforward deployment on existing servers, but the growth of PDAs could generate requirements to handle a wide range of different devices. No attribute should be considered clearly out before the discussion takes place.

Once we have weeded out the attributes that are clearly not a concern, we can take the prioritization a step further. An approach that I have found effective is to build a matrix of the remaining attributes, and to perform a comparison of each pair in turn. For each pair, the group selects one as being more important than the other. This will result in an ordered ranking, and is easiest managed with a spreadsheet that can tally the totals as you go. Note that this approach is effective anytime you wish to rank a collection into an ordered list – I’ve used it to help select which movie we would see on a Friday evening, (which may explain my wife’s panic when I open Excel...)

Had we not performed the first culling of the list, this would become quite a tedious exercise. By chopping the list in half (which is typical), we reduce the number of comparisons by a factor of four. As with the first approach, you will find here that many comparisons are quite straightforward, and some will generate significant discussion. Again, we cannot know this in advance, or without reasonable representation.

When we are done this, we have narrowed the large list we started with, using two approaches, and are left with a subset of quality attributes that are ranked in overall importance for our specific project. We have done nothing to actually produce testable requirements at this point, but we are now ready to do so, and our narrowed focus will ensure that we build the appropriate requirements. This culling and prioritization takes little time in a focused session, and as with many problems that appear difficult, we are breaking it down into a series of steps, each of which is straightforward on its own.

Software quality attributes, often called the non-functional requirements, are likely the most neglected category of overall project scope on software projects. This is due to a perfect storm of influences: we naturally think of requirements as the functional capabilities of our system, the discipline of extracting and refining these quality attributes is a difficult process requiring the collaboration of all the project stakeholders, and there is little guidance in the popular literature on how to break down this difficult problem.

When we discuss the scope of a product, we almost invariably discuss in terms of capabilities, features, or how this system needs to do the same things as that system that we are replacing. Whether this is done as a list of features on the back of a dinner napkin, a more detailed collection of use case, or the elusive detailed set of functional requirements, this is the traditional way of looking at things. This perspective is critical, to be sure (though I would suggest that use cases are relied on for more situations than they should be), but focusing solely on this view is a key reason that many software projects end up disappointing the client in the end.

Understanding the functionality of the system allows us to discern between a piece of tax software and the latest first-person shooter game, but we need to go beyond this to understand many of the distinctions between one tax application and another. The two systems will have roughly the same features (based on the current taxation business rules), but very often, one package will be far more attractive to the end user. This difference can and should be expressed in advance, so that we can actually build these distinctions proactively, rather than hoping for the best or trying to retrofit them in after we discover their absence in our first attempts to integrate the system.

These characteristics cover a broad range, which is one of the first challenges to overcome. Usability, performance and security are common examples of capabilities that are difficult to express as functionality, and there are many more areas to consider. What we need is a taxonomy that we can expect to cover the breadth of quality issues we may run into.

There are a number of taxonomies available. In Managing Software Requirements by Dean Leffingwell and Don Widrig, the authors suggest usability, reliability, performance and supportability (then include an ‘other’ catch-all category to pick up the pieces). Indeed, as published in Roger Pressman’s Software Engineering: A Practitioner’s Approach, some groups at HP have settled on an acronym, FURPS, that captures their perspective: functionality, usability, reliability, performance and supportability. Note that in recent years, FURPS has evolved into FLURPS+ for some teams, adding localization and another catch-all category.

I particularly like the idea of functionality, that which most of us consider to be the breadth of requirements, is merely one element in a broad landscape of considerations for overall product quality. This sentiment echoes the taxonomy that I was first exposed to for overall quality, from the Rome Air Development Center (RADC), when I was involved in avionics-based embedded systems development. Their overall taxonomy was thirteen items, correctness being one of them.

This number of elements seems to be roughly the amount required to gain reasonable coverage for overall quality. McCall, Richards and Walters suggest 11 different elements, Karl Wiegers in Software Requirements (second edition) suggested 12, RADC had 13, and based on experience, Wiegers has added 2 more to his list, and currently uses a total of 14. A comparison of these lists shows that there are many terms that are common to all of them, an indication that we are coming to a consensus.

These broad taxonomies can be broken into several schemes. Wiegers breaks the list into items that are of interest to the user, and items that are of interest to the developer. McCall’s list is broken down into the product’s operational characteristics, its ability to undergo change, and its adaptability to new environments. While this really doesn’t help in specifying software requirements for your project, it helps us understand the rationale for the breakdown, and to appreciate that no single stakeholder can adequately speak to all factors of quality for a product.

Given the similarity between these lists, it could almost be a matter of drawing straws to determine which one to use for your projects. My preference is to go with the current list of 14 from Wiegers, but to lean on an appreciation of the contributions of others that led to this point. I would expect that more people in the industry are familiar with Wiegers’ work than with the other sources (and besides, I train with his material!).

Pick a taxonomy, but be sure to start with one of these broad taxonomies! While it can be appealing to start with a shorter list, such as recommended by Leffingwell or used at HP, there is danger that you will miss a critical area for your project. As we will see, it is quite straightforward to start with a broad list and then remove those elements that clearly do not apply, but to start with a short list and expect to recall other areas as required is problematic. Think about it: there is a good chance that some of the techniques you apply for developing products today have been borrowed from successes on other projects. This propagation makes sense of the technique starts from a general starting point, but a narrow list that worked on one project will likely have gaps for your project. If anything, start with a broad list and feel free to expand it into other areas, based on experience, as Wiegers has done.

Generally, when we talk about testing, this is seen as the dynamic execution of the software or system and peering into the code, rather than any other form of validating or verifying the system. Not static analysis of code, not peer reviews or inspections.

This trend of greater emphasis on testing is not good for the industry.

Don’t get me wrong, being effective at testing software products is not a bad thing. Indeed, given the way most companies develop software and the increasing trend towards agile approaches, we had better be damned good at finding all the bugs in our products. The problem here is not testing, it is the emphasis on exhaustive testing over more efficient analysis techniques earlier in the lifecycle.

Let’s pull out some stats from the industry. Numerous studies show that half or more of all defects found on software projects have their root in a requirements issue. One study at TRW indicated 54% of all errors were found after unit testing, and 83% of these were requirements and design based errors. Think critically about the defects that have been logged for your products, I would expect you to see a similar trend.

Layer on top of that a study with the US Navy that indicated that incorrect facts accounted for 49% of requirements errors, and an additional 31% were errors of omissions. Having worked on defense projects as well as a wide range of projects in other sectors, I’d say that it is safe to assume that those 'errors of omission' could easily double in most shops.

One more data point that has been reflected in numerous studies: a requirements-based flaw can cost over 100x to fix after the system has been deployed, compared to the cost of fixing the problem closer to when it was injected into the system. Problem is, many shops first validate their systems only when the code is running.

Alright, enough with the data. What are we getting at here?

The point is this. Throw all these data points together, and it is pretty clear that we spend way too much time cleaning up the spills that never should have happened in the first place.

More and more, I am seeing that organizations are leaning on testing to actually figure out what their requirements should have been. In the recent past, the following points came out in discussion with various companies:

• One shop identified numerous defects for non-typical issues around data that should have been considered as alternatives and exceptions in their use-case analysis.
• In the same place, a tax lawyer indicated that she gets called into projects when a problem has been discovered, rather than identifying the relevant tax-related business rules up front and developing the system to accommodate them accordingly.
• Another shop found a particularly nasty bug where one data element was implemented in slightly different ways in different locations throughout the system. A data dictionary might have come in handy here.
• Then there is the company that only realized when integrating the whole system together that the system took minutes for what should have been a very fast response time. And the company that arrived at the integration stage to find they don’t know how to test what they had built in the first place. Oops, maybe an analysis of quality attributes would have set reasonable expectations for the design.

I could go on with other examples for quite a while, and I know that you would have stories to contribute as well. All of these issues were discovered through running the code, and all issues were fixed in the code. Until these points were raised in the context of requirements training, the leap had not even taken place that all of these were actually requirements-based defects in the first place.

It sadly reminds me of an old Calvin and Hobbes cartoon, where Calvin asks his dad how they determine the load limit on bridges. His dad replies that "they drive heavier and heavier trucks across the bridge, and when the bridge breaks, they weigh the truck, and that’s the limit!" Absurd, but that’s effectively the approach in far too many software shops today.

It is easy to see why testing is leaned on so heavily in the industry. Even the most novice person on the team can be seen as productive by poking at applications and quickly find issues with the system (hell, I’ve had bosses that took pride in being able to break applications...at least they achieved something!). Testing has traditionally been the point where we get the bugs out of the system, and it is the teams and individuals that magically pull a bug-ridden application to a state where it can be shipped that are often seen as heroes. In some ways, test-driven development takes this to the extreme: tests are crafted in advance of the code.

Isn’t that what those requirements and design stages we rush through are supposed to accomplish?

While I respect that good testing is a mature discipline and a skilled tester can wring out even the most elusive defects, the point remains that everyone is better off if the defects weren’t injected into the system to begin with. Nothing against skilled testers, as there is a huge niche to be filled in the industry, and they are making a good living.

The industry as a whole has far greater opportunities available in getting better at analysis, not in more testing or even more effective testing. We need to figure out our equivalent to those bridge load limits in advance, and build systems that meet these specs.

Problem is, right out of school it is a lot tougher to develop the requirements for a system than to break it. Even after a lot of time in an industry, most have a really tough time unambiguously stating what they intend to build. It is indeed a tough thing to do, an intractable problem that is still solvable.

Good, effective analysis is a matter of selecting which techniques to use that will give us the most valuable perspectives of what we are trying to build. Use cases alone are insufficient, as are 3x5 index cards. Having a customer embedded with the team gives us timely opinions and answers, but does nothing to make us more effective at understanding whether we have appropriately analyzed the problem.

Pick the right techniques, and each view peels another layer off the problem, and no step is difficult on its own. The challenge is in picking those appropriate techniques, and I have seen very few groups that consciously choose how they will attack the analysis challenge for each project. Almost all defined approaches lean towards specific analysis techniques or deferral of these important decisions downstream, rather than ensuring the team is equipped with a wide range of techniques required to do effective analysis, and the judgment required to select the right ones for the problem at hand.

Come to think of it, this is a technique that effective testers do well, except that they do so at a time when the project has already fallen off the rails. The skills and attitudes that have already been developed in effective testers are the same required for effective analysts, except that the deeper need is in an underappreciated phase of building our products. Make the shift to deciding what you need to build rather than discovering it after the fact, and I would guarantee that you will need far fewer test resources (or use them more efficiently earlier in the project), you will reduce risk and uncertainty on your projects, and you will ship better products faster.

We in the industry have put new lipstick on that same old pig, and the marketing has been very effective. What was originally merely an approach to understand how the end user interacts with the system to achieve their goals has recently become far more popular.

Granted, Use cases are a powerful technique indeed, if the situation warrants. The approach allows us to work with the user community to understand the breadth of actions and responses built into our systems, working in the language of the user domain. It serves to enlighten the technical community about the domain. If there is a great deal of interaction, even with non-human systems, Use Cases are quite valuable as an analysis tool.

But there are many systems where Use Case analysis provides no value, and many more systems where solely Use Case analysis is incomplete. Back end systems, algorithm-intensive applications, anywhere there is no rich give-and-take interaction with users or external systems.

Even for a system that benefits from the Use Case approach, these models are best complemented with other models. Data flow analysis to capture the internal transformational processes and logical data stores, data analysis in the form of entity relationships, data schemas, and data dictionary entries. The user interaction can be further analyzed in the form of GUI mockups, and dialog maps. There are wide range of potential approaches, both from the structured world and the more recent object-oriented world. While some would prefer to stick with only one of these two categories, I think that any analysis approach that a) is commonly understood by those involved and b) provides insight into some aspect of the system, is a good approach to use.

That intermediate step between business level requirements and detailed functional requirements should be more generically called Analysis Modeling, with one of many potentially valuable approaches being Use Cases.

To start off the analysis of any system, after we understand the vision and scope of the product, we should lay out the analysis landscape (call it the architecture of our analysis, but that’s overloading an already overused term). We set out our plan of attack for the types of analysis we intend to use for the system, and where to deploy our resources. Use cases here, perhaps complemented by GUI mockups and Data flow diagrams. These things here beg to be modeled with state transition diagrams, while that area over there will benefit from some process flow modeling. Here are the key data elements we want to nail in our data dictionary...

We then have a good understanding of the analysis work in front of us, allowing us to resource it properly and to know when we are done. As always, we prioritize to be sure we nail the difficult pieces first.

Our goal is to pick the appropriate analysis steps so that we never encounter an “and then a miracle happens” leap from the information at hand to the derivative work product. Each fuzzy transition we have, while potentially providing the adrenalin we can be addicted to in software development, introduces significant risk to the project by overloading our decisions.

The best written use cases that follow suggested guidelines will not cover any of the user interface issues, the quality attributes, the necessary logical transformational processes or data stores for our applications. Algorithms are not represented. A leap from use cases to design or even code bundles all of these issues and more together for someone to deal with in the wrong context. We will get guesses, rework, and disappointment with the results.

The goal is to have precisely the right number of steps. None that fail to add clarity in some respect, and enough to make the process predictable and repeatable. There is generally no way to identify these in advance, it is entirely dependent on the nature of the product being built and the knowledge and experience of the team involved. While at the highest level we can say that we need to know what we want to build before we decide how we are going to build it, before we build it, this is insufficient as reasonable guidance for the team.

On many large projects, or projects that are developed under the guidance of an exhaustively detailed methodology, there can be a great deal of waste in dogmatically applying steps that add no intrinsic value. I’ve been involved in the development of large systems where the decomposition of the system into hardware and software components (a critical aspect if done properly) was essentially a cut-and-paste exercise of moving system-level requirements into their appropriate subsystem level specs.

What was missing in this exercise was the required analysis and clarification of the interfaces between the subsystems needed in order to support the system’s ability to achieve its goals. We had literally performed functional decomposition, which the textbooks will tell you is a critical stage of software development. What we should have seen this as is more appropriately functional decomposition and alignment, ensuring that the pieces we broke the system into held together. The glue was the missing value-add, and the key part of this activity.

For example, let’s say that we have a system that needs to shut down if the temperature gets too high. We have a system-level requirement (which needs to be more precisely specified, of course), that could break down into requirements for the hardware to have a temperature sensor, and the software being able to provide a controlled shutdown feature. This is functional decomposition to be sure, but there is a little something missing.

We need to understand here the mechanism by which the sensed temperature is transferred to the software. Is there a polling that takes place periodically (and what frequency is this), is there a hardware-based trigger that sends a signal to the software? Are there conditions where the system may be too busy to poll at the appropriate frequency, potentially preventing a reasonable shutdown? What are the valid ranges and parameters associated with the messages or signals, and are these subject to change at design or run-time? Does the information need to be cached for audit purposes, does the shutdown event need to be logged?

Perhaps more than a little something missing, and I’m sure this is not an exhaustive list of questions. This is all requirements-level work, too.

This glue is the value added information that is the critical component of this stage of analysis, and often falls way short. Without this, the coordination of the subsystems to achieve the overall goal is left to be discovered later. This discovery will be done, but in the context of individual decisions or integration-stage discoveries. Often incorrectly made or found far later in the project through trial and error, where the cost is significantly higher.

In most projects, this higher cost will not be measured or well understood.

We spend most of our time in the implementation stage, inefficiently making what should be requirements decisions. This failure to provide value-add often happens at all stages of development, and we need to take care to add this appropriate understanding at each stage.