Databricks Weaves Deep Learning into Cloud-Based Spark Platform

Databricks, a company founded by the creators of the popular open-source Big Data processing engine Apache Spark, is a firm that we've been paying close attention to here at OStatic. We're fans of the company's online courses on Spark, and we recently caught up with Kavitha Mariappan, who is Vice President of Marketing at the company, for a guest post on open source tools and data science.

Now, Databricks has announced the addition of deep learning support to its cloud-based Apache Spark platform. The company says this enhancement adds GPU support and integrates popular deep learning libraries to the Databricks' big data platform, extending its capabilities to enable the rapid development of deep learning models. "Data scientists looking to combine deep learning with big data -- whether it's recognizing handwriting, translating speech between languages, or distinguishing between malignant and benign tumors -- can now utilize Databricks for every stage of their workflow, from data wrangling to model tuning," the company reports, adding "Databricks is the first to integrate these diverse workloads in a fast, secure, and easy-to-use Apache Spark platform in the cloud." 

 The 2016 Spark Survey found that machine learning usage in production saw a 38 percent increase since 2015, making it one of Spark's key growth areas. Many leaders in machine learning, such as Yahoo, are choosing Spark for deep learning to achieve groundbreaking results with big data.

In March 2016, Databricks created and open sourced TensorFrames, a software library that enables the popular deep learning framework, TensorFlow to run on Spark. "The enhancements announced now simplify deep learning on Spark by adding out-of-the-box support for using TensorFrames with GPUs -- specialized hardware that can perform an impressive amount of deep learning-specific computations in parallel," says Databricks. With Databricks, data teams can easily conduct deep learning on highly optimized hardware with a few clicks or API calls.

"We are proud to enable organizations to achieve better results in their mission-critical applications and are always looking ahead at the latest technologies -- such as deep learning -- to provide the Spark community with the most flexible, approachable big data toolset," said Ali Ghodsi, CEO and Cofounder at Databricks.

Databricks has posted to its blog if you want to learn more: http://dbricks.co/db-deep-learning