Datadog Announces New Hadoop Monitoring Solution

There are many more enterprises running Hadoop at scale now, and for a lot of them, monitoring has become important. Toward that end, there are new front ends and dashboards that make monitoring easier. Datadog, which has a SaaS-based monitoring platform for cloud applications, has announced support for Hadoop with a focus on monitoring.

Datadog’s solution offers dashboards, full stack visibility and correlation, targeted alerts, and collaborative tools and integrations. Integrations with HDFS, MapReduce, YARN, and Spark can be turned on immediately, so that DevOps teams can monitor more easily.

“Hadoop is an incredibly successful tool, making complex distributed data processing possible,” said Amit Agarwal, chief product officer at Datadog. “That said, as with most distributed systems running on many machines, when things go wrong it can be quite difficult to pinpoint exactly what happened or why. This is especially true in a team setting where everyone is running simultaneous, siloed investigations. This is why we integrated Hadoop with the Datadog platform. Engineering and operations teams now have the ability to turn data produced by Hadoop into actionable insight.” 

Datadog's dashboard lets users see hundreds of Hadoop metrics alongside their hosts' system-level metrics, correlating what is happening within a Hadoop cluster with what is happening throughout their stack. Users can also set alerts for  when critical jobs don't finish on time, and more.

Monitoring Hadoop technologies offers users a variety of benefits, according to the company:

HDFS – enables users to monitor the number of data nodes and blocks, disk space remaining on each host and cluster, as well as namenode load and lock queue length.

MapReduce – includes metrics for map and reduce jobs pending, succeeded and failed, bytes read by job or in total, as well as input/output records.

YARN – gives visibility to nodes, applications, cluster cores and cluster memory.

Spark – ensures users can see driver and executor, RDDs, tasks, job stages and more.

Since launching in 2010, Datadog reports that its solution has been adopted by thousands of enterprises, including Twilio, Airbnb, Netflix, EA, Spotify, Warner Bros. Games and AdRoll.

Among other Hadoop monitoring solutions, Pepperdata's is worth a look. In our interview with the company's co-founder Chad Carson, he noted that there is a shortage of skilled Hadoop-focused workers: "A lot of the focus has been about a shortage of data scientists who are skilled in Hadoop, but we also see a real shortage of skilled Hadoop operators. More and more people are learning how to deploy and operate Hadoop, but industry's adoption of Hadoop is growing ​at an ​even faster​ rate​."

Expect more monitoring solutions to proliferate, as more enterprises leverage Hadoop.