Build Newsletter: OSCON, IoT, Cloud-Native, & Data—August 2015

August 17, 2015 Gregory Chase


sfeatured-buildThis month, the big news for app developers and architects spans across open source, digital transformation, big data, in-memory data platforms, machine learning, data science, programming language popularity, developer salaries, the world of Cloud-Native, and security issues of the Internet of Things.. There is so much good stuff in here, starting off with a roundup of OSCON 2015—the predominant open source conference. Upcoming events are listed too!

Open Source: OSCON Roundup

Last month, OSCON 2015 took on it’s annual role as the predominant open source pulpit—we were excited to be a Gold Sponsor. O’Reilly has already pumped out 75 videos from the event, and there was plenty of anticipation for Kubernetes 1.0.0, which Wired also covered.

On to recaps and notables. Ben Kepes mentioned one of the more unexpected announcements, that Hitachi’s Unified Compute Platform will support Google Kubernetes. The talented team at The New Stack organized a pre-event dialogue and published it as an article, podcast, and on YouTube. The article talked about things like the advantages of Go on Docker, but it really focused the fact that microservices talks, in one year, went from zero to 30 talks—actually the only talk in 2014 is one that Pivotal did. The New Stack team also did a Live at OSCON recording. As well, published a large number of live reports and speaker interviews.

From a Pivotal point of view, here is where we spoke and what we covered at OSCON, a peek at one of the most unbelievable slides we presented, and our top 10 quotes from the event.

Digital Transformation, Strategy, Agile, and DevOps

Focusing on IT leadership struggling to demand change, Steve Dennning, Director of the Scrum Alliance, penned a great, in-depth piece on Forbes about the corporate need for innovation, the success of agile models like Scrum, and the issues these methods have inside corporations.

The Altimeter group worked with Capgemini to research business investments in innovation centers—the paper starts off by explaining the importance of innovation in business and says, “Fifty-two percent of the Fortune 500 have merged, been acquired, or gone bankrupt since 2000.” Now, we wouldn’t say this if we haven’t heard it from many of our customers. But, if you haven’t experienced a Pivotal Labs environment yet, then it is really hard to get a sense of a truly innovative environment. This is why just named us to The 10 Web Dev Companies You Want to Work With list.

With Agile and DevOps, there have been a ton of great perspectives published in the last month. One of the keynotes from Agile2015 promoted adaptability for Agile, making it OK to tweak for fit in specific environments. One expert recently covered “Agile as a Process vs. a Concept,” and another penned “Why Agile is Fragile,” covering the importance of quality and planning not just speed and iteration. We ran across three other really good pieces of advice—what character traits to avoid when recruiting for DevOps, how CIOs can build a new operating model, and enabling agile for distributed teams.

Lastly, Matt Asay recaps the 2015 State of DevOps Report and highlights the most interesting findings. As well, we are excited to have a new leader for our digital transformation practice and welcome Siobhan McFeeney, who is considered one of the most influential women in the Bay Area.

Big Data: Data Platforms and Data Lakes

At the end of the day, only one V of the “Big Data Vs” matters. That V is value. Datamation published an in-depth view of the current Hadoop landscape and ecosystem, pointing out that SQL and Java are two pillars that make adoption much easier. This is why Pivotal Greenplum and HAWQ are two key keys to our growth.

As an example, a recent SQL for big data article was recently published—to stop over $100M per year in electricity theft, BC Hydro is feeding data from two million smart meters into Pivotal Greenplum and using SAS predictive analytics to understand where energy losses are happening. In this case, EMC provided the data lake storage.

On to other topics. Sometimes, industry veterans offer very worthwhile things to think about. This article covers the concept of “data swamps” and reinforces the importance of schemas, metadata, profiling, cleaning, refining, enriching, and validating data. If data is the new oil and analytics is the combustion engine, dirty data must be addressed.

In Memory Data Platforms: Geode and Apache Spark™

First off, we are excited to announce the winner of our Apache Geode Ambitious Apps at Amazing Scale Hackathon. Taking first place, Dynamic Geode Warping built a pattern matching capability for real-time scenarios—a quite impressive, yet simple demo and GUI.

In the Apache Spark™ world, KDnuggets wrote up an excellent interview with the creator of Spark. There is also a recent article by InfoWorld’s Serdar Yegulalp that points out the recent investment in Spark by companies like IBM, Microsoft, and Huawei. Adrian Bridgwater explains Huawei’s “Project Astro,” combining the best bits of Spark, SQL, and Hbase. Microsoft recently added Apache Spark support for Azure, and IBM has been investing heavily in Spark. Lastly, the team from theCUBE noted some important points about Apache Spark as they concluded the Hadoop Summit 2015—importantly, that Spark isn’t as enterprise-ready as Apache Hadoop® and that the Open Data Platform is going to serve a critical function.

From our perspective, Apache Spark is one components within our Big Data Suite, and it has long been on the road map for Spring XD as well. This article on InfoQ explains how Spring integrates with Spark, including the use of Spring Batch and Spring Integration. There is also an Introduction to Spark for the Spring Developer presentation. Looking at the Spring XD documentation, you can also see support for Pivotal GemFire, JDBC, JMS, Kafka, Redis, MongoDB, RabbitMQ, Spark streaming and much more. Here was Databricks’ view of our supportfrom last year and the Apache Geode (Pivotal GemFire) proposal gives a high level comparison of Geode and Spark.

One key difference between Spark and Apache Geode (Pivotal GemFire)—one of them has been performing Wall Street’s trades for a long time and is enterprise-ready.

Advanced Analytics: Machine Learning and Data Science

Machine learning (ML) interest continues to grow. If you aren’t familiar, here is an excellent overview of a modern, real-time ML system based Spring XD, Apache Spark, Apache Geode, Apache Hadoop, and Pivotal HAWQ.

Machine learning is solving all types of problems that have historically been hard. There are evolutionary and disruptive examples in advertising, finance, manufacturing, media production, supply chain, retail, energy, IT security, news, and every other industry. Of particular note for software folks, some say machine learning will revolutionize the software industry itself. Check out MADlib, our open source machine learning project or read about some advances of machine learning into deep analytics, as published by our data science team.

Development: JavaScript, Cool Stuff, Language Growth, and Programmer Salaries

Within programming circles, the single biggest news is—ECMAScript 2016. One important element of the new release—modules are now first-class citizens.

Commercially, Netflix announced a major new innovation regarding JavaScript as well. They have dropped Java from their rendering pipeline, reduced the amount of HTML transmissions, and can render the same output on a browser or in a Node.js server. It’s kind of a big deal, and we are eager to see how “whatever they do” impacts the work between their open source projects, Spring, and Cloud Foundry.

In the “hey, that’s cool” category, there is a new programming language to manage heterogeneous robot swarms. Also, some new research found that nearly 8 out of 10 “IoT cloud development tasks” have to do with big data and analytics. Out of what was left, the next two were middleware and server-side development. Lastly, there is some news out of London that software exists to transplant and re-use code automatically, much like an organ donor.

Back to practical matters. With language popularity and salaries, there is a new report on the most used languages at hackathons. The IEEE Spectrum also took 12 metrics from 10 data sources to identify the most popular of 48 programming languages. The top 10, in order, were Java, C, C++, Python, C#, R, PHP, JavaScript, Ruby, and Matlab. They also have an app to slice, dice, and adjust weighting. Udacity took this data and hacked their own infographic together, laying out application types, salaries, geographies, and popularity over time. Tiobe’s monthly index shares a similar perspective—that Java is on top and C is second, then C++, C#, Python, Objective C, PHP, VB.NET, JavaScript, and Perl, also in order. There is also a great graph from their source data that shows the ebbs and flows over time. Sitepoint painted another view of the average salary for job vacancy advertisements and then showed the top 10 average technology salaries by language—the top five paid programmers are Erlang, Clojure, Haskell, Lua, and Lisp. So, specialists must get paid more but don’t have as many job options.

Cloud-Native Foundations, Platforms, Containers, and Companies

The race to adopt Cloud-Native architectures is gaining speed. At Pivotal, we have been working on this for a while largely through our efforts with Cloud Foundry, but also regularly posting blogs and even writing a book. As a signal more companies are working together to make this easier for companies, the Linux Foundation announced the Cloud-Native Computing Foundation at, and twenty two companies were part of the announcement, including AT&T, Cloud Foundry Foundation, Docker, eBay, Goldman Sachs, Google, IBM, Joyent, Twitter, VMware, Weaveworks, and many more. Importantly, Google is taking Kubernetes to the new foundation. As Brandon Butler explains, the group plans to create a stable, operable, well-integrated group of projects for cloud-native applications—apps that are container—packaged, dynamically scheduled, and micro-services oriented. It will also work together with the Open Container Initiative (OCI), another Linux Foundation project.

Speaking of the OCI, John Waters gave a an update on the group since its initiation a month ago. He goes on to explain how the number of members doubled in one month and interviewed several people from Docker, who made the first, big upfront investment. Waters also wrote a separate article, interviewing Pivotal’s James Watters, and explaining the intersection between Spring Cloud, containers, microservices, and Cloud Foundry.

From Pivotal’s perspective, one of the most exciting things we’ve seen in the past month is the groundswell around structured versus unstructured cloud-native application platforms—Wikibon’s Brian Gracely points out some key differences and explains how very-early-adopter web-innovators, like NetFlix and Pinterest, built their own app platforms that were unstructured. He then explains how structured PaaS offers a lot more tooling to support and address enterprise needs. He does cite one of our talks at Cloud Foundry Summit, but there is a also a good set of questions to help people make a decision between the two.

As covered by InfoWeek’s Charles Babcock, GE Software is moving Predix to the cloud. Later this year, GE is moving internal systems to the Predix Cloud and will then make it available to customers is early 2016. GE says this business had $4B in revenue in 2014 and will have $6B in 2015—that means GE is a pretty big software company these days, a revenue stream all created since 2011.

Two key articles were also written about Pivotal Cloud Foundry telco partners—CenturyLink and Telstra. We are partnering with the biggest telcos in the world to innovate and delivery innovation with cloud-native platforms.

Hackers Kill a Car on the Highway: Things that Connect to the Internet

The Internet of Things is a hot new area with massive expansion throughout industries of all types, and, as expected, has its fair share of growing pains. To help guide companies through some of the challenges that arise, our CTO EMEA, Chris Mills, penned an article on IoT called Future Proofing your Business with the Internet of Things.

Of course, we are already getting some stories of some failures in IoT. Wired featured some remote hackers who killed a Jeep on the highway. There are also some scary stories about hacking chemical plants or power plants. We expect that security will be a continued area of concern for some time, and will be sure to continue to cover developments here.

Upcoming Pivotal Events

Editor’s Note: ©2015 Pivotal Software, Inc. All rights reserved. Pivotal, Greenplum Database, GemFire and HAWQ are trademarks and/or registered trademarks of Pivotal Software, Inc. in the United States and/or other countries. Apache, Apache Hadoop, Hadoop, Apache Lucene, Apache Cassandra, Apache Geode and Apache Spark are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.


About the Author

Greg Chase is an enterprise software business leader more than 20 years experience in business development, marketing, sales, and engineering with software companies. Most recently Greg has been focused on building the community and ecosystem around Pivotal Greenplum and Pivotal Cloud Foundry as part of the Global Ecosystem Team at Pivotal. His goal is to to help create powerful solutions for Pivotal’s customers, and drive business for Pivotal’s partners. Greg is also a wine maker, dog lover, community volunteer, and social entrepreneur.

Immutable Infrastructure, Windows, and BOSH
Immutable Infrastructure, Windows, and BOSH

System administrators are rethinking their approach to manage large scale infrastructure. Previously, they ...

Two Subtle Influencers in Product Design
Two Subtle Influencers in Product Design

A discussion of a few factors that can influence users and affect product design.

SpringOne 2021

Register Now