Agile Succeeds Three Times More Often Than Waterfall

Agile projects are successful three times more often than non-agile projects, according to the 2011 CHAOS report from the Standish Group. The report goes so far as to say, "The agile process is the universal remedy for software development project failure. Software applications developed through the agile process have three times the success rate of the traditional waterfall method and a much lower percentage of time and cost overruns." (page 25) The Standish Group defines project success as on time, on budget, and with all planned features. They do not report how many projects are in their database but say that the results are from projects conducted from 2002 through 2010. The following graph shows the specific results reported. 2011 CHAOS Report Data


101 Ways to Get Inspired About Agile

101 Ways to Get Inspired About Agile

Get this free PDF of inspiring agile quotes when you sign up for Mike’s weekly tips email.

Get weekly tips from Mike Cohn

64

Posted:

Mike Cohn

About the Author

Mike Cohn specializes in helping companies adopt and improve their use of agile processes and techniques to build extremely high-performance teams. He is the author of User Stories Applied for Agile Software Development, Agile Estimating and Planning, and Succeeding with Agile as well as the Better User Stories video course. Mike is a founding member of the Agile Alliance and Scrum Alliance and can be reached at [email protected]. If you want to succeed with agile, you can also have Mike email you a short tip each week.

64 Comments:

Steve Duitsman said…

Do they give their definition of “Failure?”  Late?  Over-Budget?  Completely missed what the customer wanted?

I really like that the “challenged” number didn’t drop as much as the success rate increased between waterfall and agile.  Shows that agile methodologies aren’t a silver bullet, but they do help most project/teams.

Mark Levison said…

Mike - like you I’m heartened to hear to the Standish numbers. However they leave me with some concern. Since we don’t get very solid information on how they collect the information, on what they call Agile, etc. its hard to know how to interpret the results.

Cheers
Mark

Dan Creswell said…

I guess that leaves me with more questions than answers…

“The Standish Group defines project success as on time, on budget, and with all planned features. “

I’ve always had the impression that successful projects tradeoff budget, time and features to achieve something satisfactory. The definition proposed above seems to be more in the holy grail category and assumes perfect estimation and staff availability amongst other things which seems at odds with the underlying assumptions of agile thinking.

“The agile process is the universal remedy for software development project failure. Software applications developed through the agile process have three times the success rate of the traditional waterfall method and a much lower percentage of time and cost overruns.”

Agile of course, is not a process, right? Universal remedy? Sounds like the dreaded silver bullet syndrome to me.

I guess ultimately the problem is these are soundbites and without a greater understanding of what the Standish group defines as agile, planned features etc any attempt at a conclusive interpretation is unwise.

Mike Cohn said…

Hi Mark—
Indeed, there has been a lot of academic angst over the Standish numbers for years. Much of that is because they don’t make all their data public, as an academic would. I suspect their definition of what makes an agile project is a bit different from mine. What I’d take from this is that something they called “agile” outperformed something they called “waterfall” by a wide margin. Improving the definitions of those would probably not change the direction of the finding (i.e., it wouldn’t switch things to waterfall outperforms agile). I remain skeptical of some of their exact findings and I wouldn’t define success as they do. For this post, I’m merely trying to report the information.

Mike Cohn said…

Hi Dan—
Yes, the fact that they call it “the agile process” points to a fundamental understanding problem.

As for on time, on budget, with all planned features. I’ve never delivered such a project :(  My issue is with “all planned features.” If a six-month project, for example, delivers all planned features that means our brains shut off. It means that at no time during those six months did we ever come up with a good new idea and decide to do it instead of an originally planned idea.

Mike Cohn said…

Hi Steve—
No they don’t really define those further. I suspect they go with an approach that we all know what “late” means. My definition would have been “got the value you expected from the project,” but I understand why they go with something simpler. To be honest, I’m surprised that agile would do well with a definition of success that includes “all planned features” since scope management is such a fundamental agile thing.

Laurent Bossavit said…

Hi Mike,

As much as anyone else I would love to see reliable research showing that “Agile” - for some well-defined operationalization of the term - tends to result in better project outcomes.

But frankly I think we’re all better off ignoring the CHAOS report.

It does not provide a sound basis for thinking that Agile works better, and to be too quick to give legitimacy to bad numbers is a bad idea which (from both a scientific and marketing point of view) will backfire when (not if) better grounded methods come along and the CHAOS figures come to be viewed as anecdotal.

Christopher Riesbeck said…

I often start with the definition of project success as “on time, within budget, as planned,” and show how a project can violate one or more and be successful, while another project can meet all three criteria and leave no one satisfied. My own definition is “client is happy, team is happy, our company made money.”

I’m not sure even “got the value you [the client] expected” is right. Expectations are often too optimistic. But if the development process is transparently designed to maximize client value, a client should feel they got their money’s worth.

Mike Cohn said…

Hi Laurent—
I’m with you. I think we already have more compelling evidence. Michael Mah published some good results on time-to-market and productivity a few years ago that I liked the best. But, the CHAOS Manifesto will get talked about a lot and a lot of people have wondered how agile would compare to their well-known dismal statistics on project success.

Mike Cohn said…

Hi Elisabeth—
That’s great that you have similar findings. I will have to watch that webinar to see what they were. I’m very familiar, of course, with Larry Putnam Sr.‘s work and with the QSM database. Doug Putnam reported some very interesting data on productivity related to team size that I referenced in Succeeding with Agile.

Mike Cohn said…

Hi Chris—
You’re right, “getting the value you expected,” isn’t right. I knew that at the time but was rushed. The right way to assess success is “you got an appropriate amount of value” or something more objective like that.

As for “client is happy, team is happy…” that isn’t even it. Sometimes we start a project and have to tell the team, “you won’t be happy on this one. The schedule is really compressed” or something like that—no budget for new tools so make do with what we have. I always promised teams I would never do that twice in a row. (I rarely did it at all.) I like how Rob Thomsett created six sliders that can be used to define success in advance on a given project. We put them here:
http://www.mountaingoatsoftware.com/tools/project-success
for people to use.

Christopher Riesbeck said…

I’d put a project that satisfied the client and made money for the business but left the team really unhappy under “challenged” in Standish terms, not “successful.” It’s not something you’d want to repeat, is it?

Glen B Alleman said…

Please realize that the Chaos report is completely bogus statistically and violates nearly every sampling and analysis principle of a High School stats class.

Mike Cohn said…

Hi Glen—
There are a lot of criticisms about it and I’ve voiced many myself. However, how do you *know* it is bogus and violates good statistical principles? Since they won’t publish the data or even things like exactly what are the questions they’ve asked people, it would seem hard to *know* rather than just suspect it’s bad.

And to reiterate here, I’m the last person who would defend them, I just reported their claim here with the hope this type of discussion would ensue.

Mike Cohn said…

Hi Christopher—
Well, a project that left the team unsatisfied is certainly not one I’d want to repeat very often.

Glen B Alleman said…

Mike,

Because it is false statistics. IEEE and ACM articles and papers on the false statistics. But for a quick survey.

(1) No sample space normalization. What is the space of “all” possible projects?
(2) No error bands - no single point estimate can be true
(3) No units of measure definition - what does it mean to be “over budget.” What was the Management Reserve? e.g you might be $10,000 over budget on a $100,000 project, or you could be $100,000 over on a $1B program. What are the ranges of GREEN, YELLOW, RED. Same for schedule.
(4) No definition of “working.” What is acceptable performance?

If you can’t find the journal articles on their approach, I have them.

Bogus = statistically invalid

Mike Cohn said…

OK, fair enough. I’ve already got quite a collection of articles on them, including some by Magne Jorgensen who is my favorite researcher. My understanding of their attitude toward many of the criticisms you raise are that the responses are in the executive (respondent’s) opinion. I can somewhat buy that. If sometime after a project is done the boss remembers it as being late then there’s a problem. Perhaps it wasn’t late or perhaps it was a decade-long project that was a year late, but if your boss recalls you being late, you’ve got a problem.

One criticism I read of their research was that they surveyed people asking about their “failed projects” and then reported that a high percentage of failed projects failed. Uh….

Glen B Alleman said…

So, now regarding Agile versus Waterfall.

(1) Did those Waterfall projects follow the ACTUAL waterfall process defined in say DoD 5000.02 or Royce’s 1972 TRW paper? Or did they just do bad project management and call it Waterfall. Standish does not define what they mean by Waterfall.

(2) Was the domain appropriate for Waterfall? Was it implemented poorly, correctly?

(3) Were the agile projects appropriate for agile methods? In other words were there projects where agile was not appropriate, applied and failed?

(4) What’s the control group? Multiple projects in the same domain and context, with actual assessment of parallel processes - this would be the Software Engineering Process Engineering Group at Lockheed Martin, where Agile and DO-178, and DOD 5000.02 all run concurrently?

(5) The notion of “design of experiment” is missing from the report, so there is no way to assess the validating the even the data gathering and assessment process.

This approach would get you a D in any science or stats class in HS.

Mike Cohn said…

Glen—
These are all good points but you put me in a position I don’t like: that of defending their “research” but here goes: The point of measuring something is to reduce uncertainty. I can measure and reduce uncertainty, make better decisions within my business and still get that D in a stats class. I’m not arguing for doing shoddy research but I am arguing that holding out until a perfect experiment can be done is not always appropriate.

Glen B Alleman said…

Mike,

Yes. The sampling process for a credible survey needs to know the underlying distributions of projects BEFORE conducting the survey. This is the H0 test approach. But you’ve got the essence - when you call and ask about “failed” projects, you’re going to get what you asked for.

An example of a better approach is the KPMG Global Project Reports. Another example of statistical “gaps” is Ambler’s charts http://bit.ly/zdhSg2.

The core problem is to test the hypothesis of what correlation a SW Dev method has on the success factors. Not that it doesn’t, but what is that statistical correlation. E.g. How much does Good and Stable requirements impact success? The Caper Jones series of material speaks to those correlation factors.

robert seres said…

My POV is that the research is good enough for continuing (or even starting) dialogue about waterfall processes and their success rate (at my organization vs. an Agile mindset and how that could impact our overall execution capabilities. 
I never thought that Agile would replace waterfall in 100% of the cases…so really, does it make a difference is Agile was 28% or 20% or 15% more successful than waterfall or that I’m continuing to challenge my stakeholders, management, team, etc. on CI…

Glen B Alleman said…

Mike,

All of this is moot, except of course if your spending someone elses money. then that decision to adopt one process or another, move in one direction or another becomes a budget expenditure issue. If it’s a sovereigns, money, or shareholder money, it’s no longer a decision based on bogus statistics.

Ken Clyne said…

It has been a very popular pastime to debunk CHAOS reports.  I am not going to defend them - I’ll leave that to the Standish group.  However my observation over the years is that they do align well with what we all intrinsically know.  For example, I don’t think anyone disputes that there has been an incredible amount of feature waste in some of the products we build and likewise I have no doubt whatsoever in my mind that an agile approach is an order of magnitude better than a waterfall process.  What is Agile?  For that matter, what is Waterfall?  Forsooth what is success? I’m fairly sure that in this and any other similar survey the question is asked briefly and in the mind of the respondent for that brief moment when they formulate their answer it is very clear.  Let’s not let mix precision with accuracy.

Joel Bancroft-Connors said…

Mike,

I think Glen tangentially raises an important issue, that of transparency. It’s a major fundamental of agile we all believe it. Standish is anything but transparent.

This same issue came up recently, when Peter Saddington (Agilescout) wrote an article about Agileexams.com. The issue wasn’t the quality of the product Agileexams .com provided. The issue was the total lack of any information on the process, the company and so on.

I’m a proponent of agile and think it has a lot going for it. I just have problems getting behind a report that isn’t clear on how it got its data and still seems very mired in old school thinking of success. Does this report do more harm than good because it will lead to more bad agile implementations as companies hope for a “magic cure”?

Mike Cohn said…

Hi Joel—
I think you’re absolutely right about transparency. That is why so many people have complained about the Standish Group’s reports over the years.

Hi Ken & Robert—
Ken’s comment about not mixing up precision with accuracy is very true. And, Robert, you give a good example of this by saying it doesn’t matter if agile is 28% or 20% or 15% better. And I’m with you that something like this can be a good thing to use to challenge the thinking of some in our companies.

Hi Glen—
I just read the KPMG survey and it seems to suffer from some of the same issues you cited with regard to the CHAOS one. In particular, they, too, report very precise values (“42% of business organizations independently verify business case assumptions”) rather than tell me ranges around those. But the rest of the report is sufficiently professional that I read that and think, “OK, there’s really a range here (and some assumption of ‘how much verification’) but I trust them and they just took some shorthand in telling me on ‘42%’.”

Badri N Srinivasan said…

Hi Mike,

The Standish Report - Chaos Manifesto has been oft quoted in many slides presented by organizations over the years focusing on providing additional proof of their point of view through additional business data and this makes it all the more important for them to not withhold data behind how the conclusions were arrived.

As the data points are varied and vast, it makes it extremely difficult to arrive at meaningful conclusions without bias from a set of projects and there should be a disclaimer along with the report so that the conclusions are interpreted correctly in the right context.

This may lead to more misconceptions about the conclusions which are not having the full background data and basis being available and when more concrete data becomes available with basis from other reputed agencies, it may create actual chaos as to who is right in their conclusions….

However, it is widely agreed that agile does better than waterfall in various specific contexts; the only point of contention being the factor of improvement and this report can queer the pitch as to which findings are actually correct and which we could quote as additional proof supporting a specific point of view, especially as the basis for the conclusions are not available !.....

Ken Clyne said…

Badri hits on a pivotal point for me “organizations ... providing additional proof”.  You should not be relying on data points like this to change the way you do product development.  You change because there is economic pressure to deliver earlier and to improve quality.  Crafting a business case is very expensive and you will undermine yourself by putting too much stock in potentially flawed data.  Much better and less expensive would be to create a safe-fail experiment and start an agile team,  very soon you will know whether it is the right thing to do or not and you will have much better and more relevant data.

Mike Cohn said…

Hi Ken—
I think you’re right in many ways. I am constantly telling organizations, “Look, we know agile works. There are too many companies having success with it to deny that. But what we don’t know is how it works *here*...” And then I proceed to encourage a pilot or just starting so we can learn how to make agile succeed in their company.

Richard Janssen said…

Universal remedy…? Get real… little historic sense both in a world sense and in IT sense. I’ve seen DSM, Waterfall, Object Oriented, DSDM, RAD, IAD, and I guess some more… and guess what: All of them, without exception were pitched to us by their advocates or ‘believers’ at the time.
I have never seen any solid scientific research of real productivity (in an economic sense) or ‘effectiveness’ of these different software development ‘models’... claiming ‘universal remedy’ is nothing short of either blatant arrogance or sheer ignorance… Sic.

Kenneth P. Katz said…

“The agile process is the universal remedy for software development project failure.” Is that really true? Can agile remedy ill-conceived projects? Can agile remedy externally imposed schedule and resource constraints that are unrealistic and inconsistent with the project goals and scope? Can agile remedy multi-tasked project team members? Can agile remedy poor business sponsorship and disengaged product ownership?

That statement smacks of hucksterism. I happen to think that agile is very good, but the idea that is some sort of universal solution to problems is going to get agile put in the same category as Six Sigma, Total Quality, diversity, etc. - the previous silver bullets which were going to magically solve all problems and didn’t, thus discrediting things which were actually useful in some limited way.

Mike Cohn said…

Hi Richard—
I completely agree on how over-the-top “universal remedy” was. Universal is just crazy and “remedy” sounds like snake oil to me.

You should look at Michael Mah’s research on productivity and time to market for agile vs. non-agile in the large QSM database. That’s research I’d agree with.

Mike Cohn said…

Hi Kenneth—
I’m with you on “universal remedy” as well. Richard just commented along the same lines.

Badri N Srinivasan said…

Hi Ken,

I fully agree with you.

However, what happens in the case of service organizations as compared to product organizations is that if they need to pitch for a new client, they need to show data that highlights their process (whatever that is being followed) as meeting the customer expectations and requirements. In such a scenario, they tend to latch on to the data provided by third party agencies which portray their process being followed as being better and short shrift is given to the basis on which the conclusions and data are arrived. This leads to the state of trying to keep up with the expectations (conclusions) of these reports by the organizations….

It is another matter that agile genuinely improves the way you do product development in specific contexts irrespective of the way the data and conclusions are arrived by the agency. The only point here is that this may actually create more chaos rather than validate key findings since we cannot really decide how the data and conclusions have been worked out by the agency. The chaos will increase when more agencies publish different conclusions based on different sets of data. In such cases, it helps if the conclusions and data are validated appropriately with the basis for these data points and conclusions….

Magne Jorgensen said…

Looks like I’m a little late for this debate and sorry for the length of this comment ...

I find it interesting that the Standish Group manage to get so much attention to their results given their history of flawed research, leading to absurd results, such as:
- Chaos 1994: “Projects do on average cost 2.89 times more than estimated”. (No other studies are even close to these results. Most likely asking for “failure studies”.)
- Chaos 2002 (or 2000): “45% of the features are never used”. (NEVER used!! What research method would lead to confident statements about that?
See more detailed comments on these two results at simula.no/publications/Simula.simula.902/simula_pdf_file

This year’s research (agile X times more successful) seems, to me, to be of the same, low quality. A few example of low quality - based on the very limited information available of the research method - are that they seem to fail to:
... know what they measure, e.g., to make it clear what separates “agile” and “waterfall” (were there no other approaches?).
... separate correlation and cause-effect. If the types of projects compared are quite different, e.g., in size, then the cause may ot be agile, but perhaps the larger size of the project.
... use meaningful comparison measures. Would any agile developers accept to connect success with delivery of “all the specified functionality”” (no learning)? If an agile project delivers all the functionality initially specified, this would in essence mean that one of the main arguments for using agile is not there. Alternatively, it means that for waterfall “all specified functionality” means all the INITIALLY specified functionality, while for agile it means the functionality specified at a much later stage. Knowing that waterfall-based projects also update the requirements, the latter would mean that the comparison is quite unfair.

I was one of those who found the CHAOS report very useful in the early days of my research, since the reported chaos motivated the need for more research on effort estimation. After several years of using it I found a couple of issues that were strange and starting reading it with a more critical mind and found that the study was very, very poorly designed and not trustworthy at all. In hindsight, I have wondered many times why I believed in their reports for so many years and why, before our evaluation of it NOBODY had spotted (or at least reported) the quite obvious flaws of the reports (such as no category for effort underrun). People, including researchers, seem to be very good as seeing and accepting what they like to be true. The amount of time we spend on finding flaws of a study is highly correlated to how much we disagree in its outcome. This is what they in psychology term “confirmation bias” and seems to be a basic instinct in people.

Glen B Alleman said…

Mike,

Correct regarding Mah’s approach. SLIM is a calibrated tool for IT projects.

Now look at the SEI annual effectiveness measures, the Galorath Seer measures of performance improvement of embedded software in space craft, or the ACE tools (Tecalote) for Space and Missile Systems (extreme software dependencies). All have calibrated, verified, and statistically credible databases - IN THEIR DOMAIN. All calibrated to the domain

The notion of universal is nonsense - as you point out. But this leaves a huge gap for those not yet up on that maturity curve of understanding, who are looking for the simple solution.

What is happening in the Federal IT and embedded systems space is we’re inverting the approach. We start with the immutable principles of success in that domain. THEN adding the agile processes that make those immutable principles more effective, efficient, and just “better for everyone.” When you start with the agile methods, then go looking from problems to solve, you end up where many agile advocates end - by quoting bogus statisitcs and making uncalibrated statement - “we know it work,” without the second part “in the domains where we know it works.”

But that of course is bad process, because the domains where we know it works may or may not be representative of other instances in the same domain. I have hands on in the medical claims processing world. Dramatic differences between two firms doing the same processing. Same for flight avionics. Two firms doing nearly identical software development. One with wonderful Scrum processes, the other struggling with those same Scrum processes for many reasons having nothing to do with Scrum.

The problem we encounter in SWDev have little to do with the process and a lot to do with the people implementing those processes. Much of the nations anti-ballistic missile defense system was built using a spiral method and FORTRAN on time, on budget, and working to this day in Bangor Maine.

Would you apply that method to a rapid development insurance claim processes system - ah that’d be a NO.

Mike Cohn said…

Hi Glen—
I’ll look for those additional reports and see what I can find. I’m in complete agreement on the need to match process to project needs. Thanks.

Mike Cohn said…

Hi Magne—
It’s fantastic to have your comments here. I mentioned some of your papers on the Standish reports in earlier comments. I read somewhere that they got such high failure rates because their survey asked managers to “complete this survey about failed projects.” And then they got data showing that lots of failed projects fail.

I’d heard that 45% of features never used were from one internal-use (not commercial software) project. I’m sure they did find one project like that. What has always irked me is how often that number is quoted (or given as “64% of features are rarely or never used”) and presented as a truism for all software projects, not just one particularly miserable internal use project. Hmm, I wrote an Excel spreadsheet one time and think I only used half the columns in it. That’s even more than 50% not used. 😊

Your point about “no learning” is very valid. And it’s why I said above that I don’t know that I’ve ever led a “successful” project per their definition. We always find some new idea more desirable than a planned idea and throw the old idea out.

I think the new CHAOS Manifesto (they seem to have changed the name from Report) serves a couple of purposes: It starts discussion like this, which is always great. And, second, it will get some people to try agile who will believe the report is 100% valid. Even if we can agree the report isn’t valid, it’s gotten a lot of publicity over the years and anything that gets more people to try agile is a good thing in my opinion, even if they start for shaky reasons.

I think Standish has gotten so much attention because they’ve made outlandish claims. They can’t back them up, of course, but when someone makes an outlandish claim, they seem to get attention. I notice that on others’ blogs. I’ll see an intentional incendiary post and it gets a huge number of comments compared to that blog’s regular comment volume.

Thanks for taking the time to comment. I’d love to hear about your latest research sometime. The last time I looked at the simula.no site, I couldn’t find the usual collection of papers and got blocked by a page asking for a password.

Magne Jorgensen said…

Hi Mike

All the reports at Simula should be possible to download without password. You can find my latest articles at: http://simula.no/people/magnej/bibliography  (please tell me if still still does not work). Perhaps the following (in progress) about relative estimation is of interest: http://simula.no/publications/Simula.simula.814/simula_pdf_file

By the way, my story with the Standish group is that I first emailed them and asked them about how they had done their studies, e.g., simple questions like “How did you measure estimation error?” since even this basic information was not provided. No answer other than that this was business secrets they could not give away (Meaning that they wanted us to believe them that they have found that the overrun is large, but will not tell us what they mean by it). Then I tried to buy the quite expensive report, but got no response and I had to buy it through another person. That was when I discovered the text (somewhat hidden): “We then called and mailed a number of confidential surveys to a random sample of top IT executives, asking them to share failure stories.”

If you are right about the source of the “45% is never used”, that research result (which is very much quoted) is based on even worse research than I thought!  As long as we don’t require them to tell how they do their studies and they continue with their poor research method, I guess we will have to deal with yet another outrageous finding by Standing pretty soon. My hope is that we do start to ask them about their research method (sampling of projects, how measures are defined, how they asked, who they asked, what is the meaning of “45% of features is never used”...) when they present their results.

Mike Cohn said…

Hi Magne—

The “64% rarely or never used” quote is from XP 2002 in Sardinia. I wasn’t there. In their first book, Mary and Tom Poppendieck reference those numbers and then add this footnote:

8. When Jim Johnson, chairman of the Standish Group reported these ratios at XP 2002 in Sardinia, they came from a limited study. Since then we have asked almost every group we address if these numbers match their experience, particularly if they are developing custom software. The response has always confirmed that these numbers, if not precisely correct, are in the right ballpark.(page 24)

So those numbers are definitely from a “limited study” and refer only to internal-use software, not commercial products. I’ve printed out the “Relative Estimation of Software Development Effort: It Matters With What and How You Compare” paper you linked above and will read it on my next plane trip.

I’m also very interested in To Read Two Pages, I Need 5 Minutes, but Give Me 5 Minutes and I Will Read Four: How to Change Productivity Estimates by Inverting the Question but when I try to download that I only get text about the paper, not the actual paper. Is it available?

Magne Jorgensen said…

It would be very interesting to see how they (Standish, Poppendieck) measure this (i.e., that a large proportions of the features are never or seldom used). I can see very many reasons why these numbers are misleading at a system level, e.g.,:
- The people they ask will not have the information necessary to know this. For example, if each individual person use only 30% of the functionality, this does of course not mean that not only 30% of the functionality of the system is being used. To measure this properly is hardly a matter of asking people through a questionnaire or on courses, unless they are very well-informed about the totality of usage.
- The functionality have been used earlier or will be used in the future
- The rarely used functionality is essential in a only a few contexts, but nevertheless of high value

I think we need, as a community, to be better at asking what such statements means. Does the statement “X% of the features are never used” for example means that X% of the features has never been used by anyone? In that case, I would be very surprised that a high X is the average for the software community.

Mike Cohn said…

Hi Magne—

The Poppendiecks were definitely just reporting anecdotal evidence. I think that’s fine since they were clear about it.

A perhaps more credible report on the percentage of features used was given at the 2000 International Software Engineering Research Network (ISERN) Workshop and is referenced by Basili and Boehm (certainly two credible individuals!). It was reported at that conference that individuals working alone used 12-16% of Microsoft Word and PowerPoint. Ten-person workgroups were found to use 26-29% of features in those products.

Clifford Shelley said…

Hooray for Glen A. for pointing out the limitations of so many published stats. It may not be Standish’s fault though. The information content of data attenuates fast as it strays from its context, is orphaned from it’s operational definitions and is mis - represented - give me absolute numbers, not percentages and bar charts, not pie charts. (And as the info content disappears the propaganda opportunities seem to increase.)

Sabari said…

Hi all,
Its really a breath taking stats displayed above on success rate of projects. 42 vs 14. But i just feel that for successful implementation of Agile software projects we need to have a really mature client/customer who knows actually what he wants. A major portion of software project failure is due to immature client who most of the time is not able to define the scope. So if we want to prove agile is better than waterfall we need to identify one project which failed due to traditional waterfall approach and now start it again but this time with Agile methodologies.

Mike Cohn said…

Hi Sabari—
I don’t think that a project which fails with waterfall and succeeds when restarted with agile will truly prove anything but enough such projects and things start to get interesting.

For an example of doing exactly this see the User Stories Applied book and the example of a biotech company I describe in there.

Peter Hawkins said…

Hi,
There has been some compelling studies on the bogus nature of the standish report outcomes.  I have little faith in any of these reports basically because they are hauled out to support whatever agenda is being pushed.  I have been doing Agile way befor the term was coined and done properly it has every chance of success because it is a set of good SDLC principles applied to certain types of engagement. The problem is that doing a half-arsed job of agile does not work and plenty of places do this.

Mike Cohn said…

Hi Peter-
I agree on all fronts, especially regarding the credibility of their reports. See my comment above recommending the papers on it from Magne Jorgensen.

Brian Wernham said…

I would appreciate some feedback on the idea below and how it relates to Mike’s point - do Agile projects simply manage risk better than Waterfall?

Could we try a new approach to risk management of all [rpojects perhaps?  All projects should be assumed to be in ‘Red’ risk status until they start to deliver.  More detail on my thinking is given here:

http://brianwernham.wordpress.com/2012/12/29/the-dod-caught-with-its-pants-down-a-revolution-in-risk-assessment-needed

Thoughts?

Brian

Axel Vanhooren said…

Hello Mike,

Honnestly, I don’t see what conclusion one can draw from this.

IT projects can’t be compared with other types of projects because it’s not only about building a solution, but also about acquiring insight in the problem area. so there is no established, detailed picture of the required solution at the project start.

As it has already been said, a project’s success (measured by delivering accordingly to the iron triangle?) is different from the product’s success (benefits delivered by the solution).

A methodology is not a recipe. We may never follow as prescribed. We must adapt it to suit the project’s specific situation. A methodology doesn’t relief us from having knowledge and understanding of our domain and of what we are doing and doesn’t replace common sense. So, is the methodology repsonsible?  I don’t think so.

Waterfall projects suits to some types of projects and Agile to others, usually smaller ones, lesser critical, more local scope, more oriented to end-user, ...

Although, I can add some other thoughts, but I will end with this one. Within a waterfall project, plans are made from the very beginning. These plans cover the whole project duration, which means many months. Normally, one would expect they should be reviewed and adapted as the project progresses. Sometimes this happens, sometimes not. But postponing the delivery date can then already be seen as a failure (not delivering on time). If a same project would be executed on Agile, the backlog and planning is continuously adapted, nearly until the last day .. actually (last iteration). So the planning is not about months. It is sliced and in the end it concerns only a few weeks. If one can adapt/define the scope (selected item from backlog for the last iteration) and delivery time two weeks before the end of the project, it is difficult of not delivering on time and accordinlgy to scope and budget.

The only thing I can conclude from the Chaos report is that IT projects rarely deliver accordingly to plan. What are the causes ? Is it proper to the nature of IT projects? Can we do something about it? Maybe or maybe not. Maybe we can reduce the “failures” but accept a certain rate. In the end, projects ARE risky by nature. These are then questions to further reflect on.

Best regards,

Axel.

Mike Cohn said…

Hi Axel—

While I can largely agree with everything you wrote, I don’t agree that agile is usually used on smaller, less critical projects. That hasn’t been my experience at all.

Don Reifer said…

We just completed a study on software productivity, cost and quality performance of agile versus traditional software development methods in ten applications domains using data supplied by 60 firms on 800 completed projects.  Besides performance, the report highlights trends observed by agile adopters over a ten year period and summarizes both the “hard” and “soft” benefits and the weaknesses observed by study participants.  We have looked at metrics, methods and a host of other issues and have identified facts versus fiction relative to agile hype especially for those moving into adoption and trying to quantify benefits.  Needless to say, the data paints a different picture than the literature and your three times better comment.  For example, while productivity and cost are impacted positively by about 10 to 20% (trended over a ten year period and compared against domain specific benchmarks), agile quality does not measure up to that generated on traditional projects.  As another example, the big three issues in agile according to the data are scaling, agile maintenance and contracting/subcontracting for agile. 

We have data to substantiate our conclusions now.  The study will be released next month.  Undoubtedly, it will help the community understand where it needs to pay attention.

mikewcohn said…

Thanks for the information, Don. I look forward to reading your study. I’m a big fan of your work so it should be interesting reading.

emily matalobos said…

Don, was the study released yet?  If so, where do I find it?

mikewcohn said…

Emily—

In case Don is monitoring the comments, I’ll mention that his research is published at http://www.reifer.com/reports…. The agile report is the bottom one on the page (as of now). You need to email him at the address on that page to purchase the report.

mr.bkry said…

Hi all
first thanks for the article Mike…..
second i want some help here .....i want to know if there’s any surveys of “agile practices using in industry or any company”  if there’s any surveys or paper’s about this topic or related topic i will be glad and thankfull if any one send it to me at .(JavaScript must be enabled to view this email address)

mr.bkry said…

Thanks Mike for your help i’m really glad for your instant help ...

mikewcohn said…

You’re welcome.

Jorge Hernan Abad Londoño said…

Mike, In The Chaos Manifesto 2013 -  http://www.versionone.com/asse…, the results are different. look the page 29.
Agile - Succeful 46%
Agile - Failed 6%
Agile - Challenged 48%
Waterfall - Succeful 49%
Waterfall - Failed 8%
Waterfall - Challenged 43%
Is it true?
What do you think?
Regards from Colombia
—-
PD /PS: Could you promote our Latin American event or send us a small greeting?-Agiles 2014 - http://agiles2014.agiles.org/e…

mikewcohn said…

Hi Jorge—
I am very skeptical of the Chaos reports. I find them an interesting curiosity but I wouldn’t base any decisions on them.
I completely agree with the comments on Magne Jorgenson: “Our main conclusion is that we should doubt the validity of the 189% average cost overrun reported by the Standish Group in 1994 until such time as the Standish Group disclose how they measure cost overrun and how they conduct their research.” From http://www.umsl.edu/~sauterv/a…
I’ll send out a tweet about Agiles 2014 this week.

Rachana said…

Hi;
I am a graduate student at NIU, I am conducting a survey on topic ” Transition from waterfall to agile”. Can u please help me by providing the response. Here is the link to my survey https://www.surveymonkey.com/s…. It will only take 5 mins of ur time.. Thanks a lot in advance

mikewcohn said…

I filled it out but am not sure how many others will since it really doesn’t have anything to do with this blog post.
Mike

Konstantin Pogrebnoy said…

After 4 years.. http://reifer.com/agile-quanti…
“Agile has become the mainstream approach for developing software in the USA.”
“Software quality increases when agile methods are used by as much as 20 percent as measured by defect densities during the first year of operations.”