A few factoids about the success or lack thereof of cloud-related projects have been making thee rounds of late. This is an edited version of my response to a recent query from a journalist about this topic.
A lot of large IT projects do fail. For example, a 2012 McKinsey study found that "on average, large IT projects run 45 percent over budget and 7 percent over time, while delivering 56 percent less value than predicted." And 17 percent went so bad that the very existence of the company was threatened. What's a typical number? That's hard to say because it depends on so many factors--perhaps most notably size and length of the project. Big ERP projects are the poster children for IT project failure with failure rates of at least 25 percent commonly cited.
Here's my take on two specific numbers that I’ve seen cited recently. I’m inclined to interpret them rather differently from each other although they do have aspects in common.
The first number comes from Capgemini Consulting: "Only 27% of the executives we surveyed described big data initiatives as ‘successful.’" Although the details are much different--not least by the dominance of open source technologies within current big data storage (including from Red Hat) and analysis solutions--it feels as if we're in somewhat the same place as we were amidst all the data warehousing hype in the mid- to late-90s. There's this feeling that with all the data out there, we must be able to do *something* with it even if we don't know the right questions to ask or the right models to apply.
So I chalk up a lot of the failures happening in the big data project space to projects not having a clear goal and a clear path to that goal. If you look at Gartner studies, for example, the #1 and #2 big data challenges are "Determining how to get value from big data" and "defining our strategy." Many organizations are undertaking big data projects mostly because it's something they think they ought to be doing even if they don't know how or why. Of course they're not going to succeed!
What should they do? Well, don't do that. Look at the success stories that do exist and ask if you could do something similar. But be careful. Many of those stories are something between Photoshopped reality and myth.
Technologies are shifting too--for example, the emergence of in-memory processing with Apache Spark. Software-defined storage (both file and object) is also maturing rapidly and coming into its own. So it is challenging to pick the right technologies and apply them to a specific problem. We've seen far too much "The answer is Hadoop. What's the problem?" But the bigger problem is not having sensible and actionable objectives.
On the cloud side, Tom Bittman's informal poll and some of his associated Gartner research about private clouds highlight quite a few reasons (perhaps most of all organizational ones) for problems that cloud projects encounter. But it's worthwhile noting that that his quote was "95% of the 140 respondents (who had private clouds in place) said something was wrong with their private cloud.” It's the rare IT project (or indeed any type of project of any consequence) that doesn't have at least some problems. So I'm less inclined to focus on the 95% and rather on the steps that can be taken to reduce the number of problems: Set objectives, get the right organization in place, engage with the right partners, take an iterative approach and don't try to do everything at once.
At Red Hat, we've been doing a lot of work with clients in both the Infrastructure-as-a-Service (Red Hat Enterprise Linux OpenStack Platform) and Platform-as-a-Service (OpenShift by Red Hat) spaces. Part of our involvement is delivering supportable enterprise product subscriptions of course. But it's also working with customers to implement a proof of concept, often with a small consulting team. It doesn't need to be a mega-project (and it's usually better if it isn't) but some initial guidance by consultants who have done this before can go a long way toward getting projects headed in the right direction and having fewer problems as a result. We’ve also partnered with other vendors including Dell (Dell Red Hat Cloud Solution, Powered by Red Hat Enterprise Linux OpenStack Platform) and Cisco to create integrated infrastructure solutions.
Pretty much no project of any size is going to go without a hitch. But understanding your objectives going in and working with the right partners can make a big difference.