Mesosphere Leverages Apache Technologies for Big Data, IoT Stack

In a move that could make a big difference for everyone from data scientists to IT administrators, Mesosphere, creators of the Datacenter Operating System (DCOS) for managing cloud and datacenter resources, today announced Mesosphere Infinity, billed as "the first real-time enterprise-ready open source offering that lets companies capture the enormous new business opportunities created by the rise of ubiquitous data, including fraud and anomaly detection, real-time personalization, and Internet of Things (IoT) applications, such as connected consumer devices and supply chain optimization."

The new offering was co-developed by Cisco, and is powered by DCOS, which in turn is built around Apache Mesos and a stack built around Apache Spark, Kafka, Cassandra and Akka.

Mesosphere views these open-source technologies as standards for developers and data analytics folks working on real-time data pipelines for Big Data and IoT applications. As is true with DCOS, the new offering will coordinate the benefits of many tools in one place. 

Mesosphere sees Infinity as capturing the convergence of three massive business trends -- Internet of Things (IoT), real-time data-driven customer interaction solutions, and real-time business insight platforms. "These trends are driving data volumes, velocity and variety at unprecedented and exploding levels that threaten traditional datacenters with a literal data deluge," says the company. Gartner estimates that Big Data will be a $263 billion market by 2020.

“Building data pipelines can be extremely complex and time consuming, affordable only for the largest companies with significant technology budgets and deep engineering teams,” said Florian Leibert, CEO and co-founder of Mesosphere. “Collaborating closely with our partners like Cisco and Confluent, Mesosphere Infinity makes it possible for companies of all sizes and levels of engineering sophistication to adopt these new technologies. The product removes operational obstacles and makes this combination of technologies accessible to the broad enterprise computing market.”

Cisco is a collaborator with Mesosphere on the development of Infinity and will be one of the first distributors to enterprise customers. As part of its development contributions, Cisco has optimized Cisco Intercloud Services for Mesosphere Infinity and already has a number of customers running the stack in production and already has a number of customers running the stack in production. Intel has validated the stack as being suitable at scale on Intel hardware.

“Our collaboration with Mesosphere will accelerate developers' ability to create innovative cloud services for the Intercloud, the globally connected network of clouds Cisco is building with our partners, which will offer a new breed of cloud services to help organizations capture the multitude of opportunities created by the Internet of Everything (IoE),” said Ken Owens, chief technology officer for Cisco Intercloud Services.

Mesosphere has also partnered with several individuals and organizations. The partners include Apache Kafka contributor Elodina, Typesafe - whose founders created Akka and the Scala programming language, in which Spark, Akka and Kafka are written - and Confluent, founded by the creators of Kafka.

Confluent is partnering with Mesosphere to adapt their Confluent Platform and offer it as a fully integrated component of Mesosphere Infinity. The Confluent Platform includes Kafka but also expands upon that technology to add new stream processing capabilities that will enable analytical operations on real-time data streams.

“We see Mesosphere infinity as the perfect way to make real-time data processing a turnkey operation for the internet of things,” said Jay Kreps CEO and Co-Founder of Confluent, and one of the original creators of Apache Kafka. “Apache Kafka and the Confluent Platform help provide a scalable data backbone for real-time processing which is critical for organizations addressing the scalability challenges of building applications and services for that process real-time data streams. Mesosphere DCOS and Infinity are an excellent way for organizations to jump start their usage of this popular platform.”

As part of a broad partnership, Intel and Mesosphere have been building hardware and software integrations to improve the performance of Apache Mesos, the foundation of Mesosphere’s products, on Intel Architectures.

“We see Mesosphere as a key technology partner in the modern datacenter,” said Jonathan Donaldson, VP & GM, Software Defined Infrastructure at Intel Corporation. “Intel has deployed the Mesosphere DCOS in our Software Defined Infrastructure lab, and we are collaborating with Mesosphere on advanced scheduling and oversubscription technology. The goal is to bring better utilization and performance to datacenters that utilize Intel and Mesosphere DCOS.”