Since the last Build Newsletter, we keep thinking we might be observing the fastest moving markets in the history of the world. In this month’s Build Newsletter, we cover a big question posed by Gartner on the growth of the Apache Hadoop® market; machine learning development with Apache™ Geode and Spark; open source and cloud programming; mobile and the Internet of Things container news (out the wazoo); and finally, DevOps culture shifts. It is 100+ links of news-round-up info to keep you current on these space(s). Yep, that’s 100.
If you have feedback on what is good and or not-so-good about our newsletters, please leave comments below!!
The Big Data Market: Questioning the Growth of Hadoop
Right around the publish time of our last Build Newsletter, Gartner released the 2015 Hadoop Adoption Study which suggested rather anemic adoption in the enterprise for a variety of reasons. In media and many industry circles, this perspective caused a stir, perhaps a backlash.
Alternative viewpoints abounded. The WSJ’s CIO Journal, author and research Thomas Davenport, who wrote Information Ecology almost 20 years ago and Competing on Analytics in 2007, made a key point about the Gartner announcement, “I thought the fact that 26% of companies in Gartner’s survey sample already had Hadoop projects underway, and 46% intended to have them in place within a couple of years, was a pretty fast transformation,” and goes on to talk about the shift to the new data architecture in more detail. Wikibon and TechRepublic also saw a perspective similar to Davenport.
In a part one and two clarifying tweet, Gartner then agreed, saying, “Hadoop has demonstrated enormous value to many users,” and also tweeted, from Hadoop Summit, several cases of Hadoop’s, like Pivotal’s investor and partner, GE, with their Predix platform. Gartner went on to provide a more in-depth part one and two of “Perspectives on Hadoop,” but only cited three Hadoop vendors. Interesting. The new Register publication, The Platform, recently wrote about how advancing the skills needed to widely deploy Hadoop means offering SQL on Hadoop and cites many examples of SQL running on Hadoop—including Pivotal, Hortonworks, Cloudera, MapR, IBM, Couchbase, and in academic arenas. Oh, Apache™ Drill™ came out with 1.0 for SQL on Hadoop.
At Pivotal, we do not see the momentum or possibilities for big data slowing. In fact, we are working hard to ensure that it becomes easier and faster to mine results as proven by the news that we just acquired another order of magnitude for query processing speeds to our existing 1000x improvements. If you want more about Pivotal’s perspective, read more from our president and COO, Bill Cook, who recently spoke to Computer Business Review about where Pivotal sees the market. We also answered the 3 biggest questions before initiating a Hadoop project.
Machine Learning Development, Google I/O, Apache Geode, and Apache Spark
Is machine learning development becoming mainstream?
About two months ago, Google’s Eric Schmidt said, “The core thing Google is working on is basically machine learning.” Well, move over Google Self-Driving Car! Just a few days ago, Orange Silicon Valley announced quite the extensive report on the machine learning topic, starting off with the fact that machine learning-run driverless trucks are already in operation in Australia!
Authors at Forbes Tech believe the development of machine learning systems and algorithms are becoming more mainstream in a recent part two of an article. In another example, Datanami uses the now overused “XYZ is eating the world” to apply to machine learning but covers some great information. AirBnB open sourced a machine learning package called Aerosolve, and LinkedIn open sourced its Pinot real-time analytics. Machine learning is becoming as smart as a toddler when it comes to recognizing images and using words to describe those images and things “it” has never seen before. As well, many Wall Street funds are already using machine learning technology for automated research and trading.
Apache Spark™ is considered an important environment for developing machine learning capabilities, and the newly released Spark 1.4.0 is the first to package SparkR.
At Google I/O, machine learning was front and center. Leveraging the machine learning advancements, Google highlighted Google Now on Tap to extend Google Now along with a big push around optimizing Android. The Verge covered the latter in The 12 Most Important Announcements from Google I/O 2015—these included making app-to-web movement smoother, simplifying app permissions, Chrome Custom Tabs make Chrome an in-app browser, better battery life, USB-C support, Android Wear improvements, new photo apps, and improvements to Chrome and Maps. Google also released Polymer, a JavaScript library for creating reusable and interoperable HTML widgets that follows W3C specifications. Will this be as popular with web developers? We’ll have to wait and see. It certainly claims to be faster in Chrome and Safari.
If you didn’t catch our director of technical marketing, Fred Melo, explaining Spring XD for real-time streaming and machine learning, he also highlights how in-memory data grid technology like Pivotal GemFire, also known as the open source project Apache Geode (incubating), fits within an overall, high scale architecture and related use cases. He will also be speaking at the inaugural In-Memory Computing Summit 2015 on the topic. As well, Pivotal’s Roman Shaposhnik will cover the directions of GemFire and Geode within the Apache Software Foundation (ASF) at the event, explaining that Geode can do for RAM what HDFS has done for disk.Of course, Pivotal is a sponsor at OSCON—so be sure to come talk to our experts about this stuff!
Don’t forget, you can download a Geode stock prediction demo or social network demo with Chaos Monkey on GitHub. Geode is also available for download from the Docker registry.
On a last note, we also released 20 Examples of ROI and Results with Big Data and are making many technical, online classes FREE! That’s free training on Pivotal products folks. Come n’ get it!
Open Source Software: Surveys, Swift, A/B DB Tests
A brand new survey on open source explained the “how and why” of open source adoption, explaining that 78% of companies run on open source (doubling since 2010) and 64% participate in open source projects. Has open source won yet? Even Walmart loves open source.
According to Wired, the biggest round of applause at Apple’s Worldwide Developers Conference keynote came with the company announced that it will open source the next version of Swift because “developers have demonstrated a growing preference for open source tools and platforms.”
Alongside Android advancements, take a moment to celebrate 20 years of Java!
Interestingly, Microsoft is bringing Node.js to Windows Windows IoT Core Platform.
Have you ever wanted to A/B test a database? For those re-architecting their data platforms, this article on A/B testing data platforms with Apache Kafka™ is quite compelling. Perhaps we can reduce the number of opinions and make these decisions based on facts.
In the world of Spring and Java, Spring for Apache Hadoop® was released as version 2.2 GA and support for Cloudera CDH 5.3.3, Hortonworks HDP 2.2.24, and Pivotal HD 3.0. Our own Stuart Williams presented, Reactive Application Design for High Volume, Multi-Dimensional Temporal Data Series, released on InfoQ and covering billions of events per day. We are also pleased to announce Spring Data GemFire’s support for Apache Geode. A noteworthy survey on Spring 4 and Java 8 was posted. An Apache Ambari™ plugin for Spring XD was released and supports provisioning for Hortonworks HDP 2.2 and Pivotal HD 3.0. A webinar replay on Reactive Data Pipelines with Spring XD and Kafka is available for replay and you can check out the slides. Matt Stine explained the challenges of microservices and distributed computing, also explaining key components of Spring Cloud. We also talked microservices at QCon New York. The rockstar Spring engineering team is always busy—there is so much more we didn’t cover.
Our Pivotal Cloud Foundry team has also been busy. We got a chance to talk to the Cloud Foundry Foundation CEO, Sam Ramji, and we reported on the CF Summit. Our team also explained how to migrate apps to Pivotal Cloud Foundry.
Mobile and the Internet of Things
At Cloud Foundry Summit, GE Software’s VP and CTO gave a great talk about PaaS and IoT.
In other related news, Tag Heuer, Google, and Intel announced a collaboration on smart watches. Harvard researchers find new ways to implant electronics in our brain. The U.S. women’s cycling team has been using analytics to better compete—extreme Internet of Humans, optimizing athletes.
Toyota entered an agreement to explore collaboration with subsidiary Livio—evolving the in-car app market with the SmartDeviceLink standard. Car fan developers might also want to check out developer.ford.com.
Andreesen Horowitz Partner, Benedict Evans, gives an insightful view of mobile, challenging much of the current thinking. New market research shows 79% of companies have to integrate mobile with up to 15 different legacy systems. Android now offers non-connected mobile apps for iOS and Android.
And, the latest Pivotal Perspectives Podcast covers how Pivotal Labs and Pivotal Cloud Foundry are making push notifications for new devices like the Apple Watch really easy.
Containers Mega-month of News
The container wars are heating up. If Docker thought it was the only game in town, they were wrong.
Forbes provides a nice, top-of-the-trees view of how Docker, or any container, impacts enterprise apps. Docker Hub images were found to have a significant number of high priority security vulnerabilities, and they were quick to respond with a tool for checking security configurations and also announced a certification and partner program. Rancher, a container management startup landed $10M—now there is yet another start-up container player. Canonical claimed that it’s LXD container-based hypervisor rocks Linux’s KVM and also a Snappy Ubuntu Core-powered Smart Fridge with GE. CoreOS explained their state of the union at CoreOS Fest, announced availability in the OpenStack App Marketplace, and is at virtually every event on the planet. Microsoft showed a preview of Nano Server and Drawbridge-based security enhancements. Intel is also entering the container arena with Clear Linux. Forbes Tech also provided a high-level, credible comparison of several container OSes.
Extending this container update to the world of PaaS, Microsoft advanced their support for Cloud Foundry to Azure.
DevOps Culture Shifts and Transformations
There is a solid perspective from CIO.com on why CIOs and CEOs need to prioritize DevOps.
If all the updates above weren’t enough, we would be remiss if we didn’t share some interesting perspectives on the DevOps culture shift from several sources. The WSJ CIO Journal shares two great stories in this space—how Netflix manages security in the Age of DevOps with Security Monkey and AT&T is shifting to a DevOps culture with a massive training effort. We also released Chaos Monkey’s cousin, Chaos Lemur. Computer Weekly did a case study on Etsy’s DevOps strategy. GM shared its IT culture shift journey with a $1 billion investment with Hadoop and MPP technologies.
Lastly, on the Pivotal side, we discussed change navigating on the Project Sputnik at Dell and shared how Lockheed Martin is taking on agile.
Editor’s Note: ©2015 Pivotal Software, Inc. All rights reserved. Pivotal, Pivotal Cloud Foundry, and Pivotal GemFire are trademarks and/or registered trademarks of Pivotal Software, Inc. in the United States and/or other countries. Apache, Apache Hadoop, Hadoop, Apache Ambari, Apache Kafka, Apache Geode (incubating) and Apache Spark are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
About the Author
