Hardly a Silver Bullet
We’ve been collecting data on a number of teams over the years, and now have responses from close to 800 participants. The questions cover demographics, practices and performance, and are designed to understand what is going on, not to pitch a particular approach or practice. I thought it would be interesting to compare the results from those that claim they are using agile approaches against those using other industry frameworks.
Note first that the data does not come from a preponderance of teams that have embraced agile, or from those that are necessarily hung up on any other approach. This is a cross section of over 80 groups that I have worked with in the past 6 years, and cover a very wide range. This is not strictly a survey of the converted.
Secondly, there were no questions that try to correlate any aspect of performance directly with a particular approach, questions that can give a bias to the results in a certain direction, such as “How much has your project improved since you started approach x?”. We ask about the approach, then we ask about the performance separately. Just for fun, we have included the results from those that stated they used an ad-hoc approach.
Thirdly, this is just raw data: we have made no refinements or adjustments. I’ll give what I think are reasonable conclusions afterwards. Here we go:
The results, from my experience, are not surprising.
Working with a wide range of teams over the years, I have seen success and failure for any stated approach. Additionally, we have found that for most organizations that would suggest they are using a particular approach for developing their product, things are a little different once we dig deeper. It is a very rare organization indeed that has a clearly identified approach, that everyone involved understands and buys into, and most importantly, follows that approach particularly when the pressure mounts on a project.
As the chart shows, while those that suggest they are working in an agile environment appear to have fewer issues with late delivery and product quality, in the other dimensions the results are statistically indistinguishable from other frameworks, or even an ad-hoc approach.
That’s right, folks: these numbers say that in some dimensions, agile approaches are statistically indistinguishable from an ad-hoc approach.
Respondents could indicate they have more than one issue, and while those that stated an ad-hoc approach averaged over 2.3 issues each, agile and other frameworks were virtually indistinguishable, with 1.55 and 1.61 issues each, respectively.
From my perspective, I am not particularly overwhelmed by the numbers that come from this data set. It is interesting that the results that come from the a community that has a vested interest in showing the superiority of their approach tell a different story.
While I believe there is great value to be had in following an agile approach, I also believe that there is great danger in overstating the case for agile, or suggesting that anything that is not agile is going to be inferior. Experience tells me differently, and the numbers here bear that out. - JB
Comments
3 Responses to “Hardly a Silver Bullet”
Feel free to leave a comment...
Clarrus Proudly Sponsors VanQ
http://www.leankanbanconference.com/presentations/Sutton_Let_Lean_be_Lean_and_Agile_be_Agile.pdf
Hi Jim,
You’ll be interested in the charts in the early pages of this presentation.
…I would wonder how the conclusion that “Lean and agile are by far the two most effective software approaches”. The data on the previous slides seems to align with the data I presented here, that agile is not the silver bullet that some would suggest.
Mind you, I think that is the case primarily because of weak implementations, rather than there being anything dodgy in the principles of agility…
And without any background on how the numbers were obtained for Lean, I’m quite dubious of those results.
[...] few months ago, I posted some data that suggests that when we compare performance against different approaches to developing software, [...]