End-of-Year Surveys Show Hadoop Growing, Still Complex, Though

As the year draws to a close, several research reports and surveys are quantifying just how big an open source success story Hadoop has become. According to a new best practices survey from TDWI there is a big increase in how many enterprises plan to have Hadoop clusters in production. By Q1 of next year, 60% of survey respondents said they will be in production, up from 16% when the report was published earlier in 2015. We covered the complete results of the survey here.

Meanwhile, the 2015 edition of the Wikibon Big Data survey shows companies making steady progress in not only adoption of Hadoop, but related tools and front-ends. 

"Steady progress relative to last survey 18 months ago shows pilots and proofs of concept continue conversion into production applications, with 41% reporting one production deployment, up 10%," writes Wikibon analyst George Gilbert.

His post adds:

– Like all major enterprise technology adoption lifecycles, customers’ ability to deploy software gated by ability to absorb and assimilate complex software.

– Gating factors to higher growth with on-premises Hadoop, specifically, are high administrative overhead and specialized skills requirement.

 It's no surprise that Hadoop's complexity and the specialized skills needed to run it are popping up. Gartner, Inc.'s 2015 Hadoop Adoption Study, involving 284 Gartner Research Circle members, found that only 125 respondents who completed the whole survey had already invested in Hadoop or had plans to do so within the next two years. The study found that there are difficulties in implementing Hadoop, including hardship in finding skilled Hadoop professionals.

Notably, though, the Wikibon survey found this: "Hadoop-as-a-Service and cloud native services from AWS, Azure, Google are easier to 'consume' because of less demanding administrative and skills overhead."

 In other words, within cloud implementations that already incorporate Hadoop, organizations are finding it easier to leverage.

The full Wikibon report is found here.