EMC’s CIO Vic Bhagat was recently named to CIO Magazine’s CIO 100 List for his work in leveraging Big Data and business analytics initiatives to provide the company’s business units with advanced, on-demand analytics capabilities that drive agility and value for EMC. He sat down with Michael Cucchi, Pivotal’s Sr. Director of Product Marketing, to discuss their challenges and accomplishments.
MC: Vic, thank you so much for taking the time to join us today, and congratulations on being nominated to the CIO 100 List. We’d love to get your perspective on the challenges EMC faced with data, how those challenges were solved with Pivotal technology, and what you have planned for the future.
Let’s start with telling us a bit about what was going on at EMC, and the factors that drove you to overcome and achieve the outcome for the business that drove you to find a new approach these challenges.
VB: EMC has a lot of data that historically was not fully harnessed. We sought to better control large amounts of unstructured data generated by machine, for example from our storage systems or disk drive, or the data is generated by our structured data like our databases that keep track of our part numbers like serial numbers. We wanted to put all this data together and leverage analytics to drive our business, but factoring the unstructured data into the equation was extremely difficult.
We started looking at how to create a ubiquitous data lake capability where we could ingest a large amount of data, and then put the intelligence on top of it—not just to show what the data is but to actually start driving some analytics and behavioral changes. For instance, things that you want to do with predictive maintenance: Monitoring a disk drive’s behavior, taking all that raw data into it, marrying it with product support trending data for that specific part number and serial number and into which cabinet it fits, and then predicting and even fixing the drive before it fails.
MC: Who was driving you towards that end? How did you end up being driven to capture the data and do the new analysis capabilities, and now even delivering this analytics as a service? What created that catalyst?
VB: Mike, it’s a great question. First and foremost, because of the data volumes, we were facing challenges delivering to the service level agreements. We have tons of information from our devices in the field—100 to 150 TB of information, with hundreds of thousands of rows coming in every second from machine sensors on installations across the world. Our customer quality team would really struggle with the data in a traditional data warehouse and they could only run it weekly or monthly because they couldn’t handle anything more frequently.
This was compounded as we tried to augment the information in our data warehouse with the unstructured data from our field. We wanted a single system with access to all the financials, inventory records, and transactional data, and complement it with unstructured information to drive intelligence. It forced us to look at a new way of serving our data to make it accessible from a ubiquitous data lake.
This opened up so many more possibilities for us. Our internal clients were asking, “How does this disk drive behave? Why is it failing?” At first we just looked at the historical data, but realized we were only reacting when the drive actually fails. Our goal was to find out if there is a way we could actually tell our Field Service reps that the drive is about to fail: this could be offered as an additional capability that EMC could provide to our customers in future.
MC: Are you including any public data in there, like weather data, regional data, maps of historical weather patterns, or anything like that?
VB: The whole reason to go to a ubiquitous data lake is to allow us to do that. If I have a ubiquitous data lake, I can just go into the Internet, take all that data regardless of format and parse it. Our marketing team already does this by capturing data from LinkedIn and Twitter feeds to do some market intelligence analytics. The beauty of the services is that it’s so easy for us to allow people to ingest external data sources and combine with internal data sources to do that.
That said, analytics is more than just visualizing the data. For example, before you buy a house, what do you do? You’re going to look for a mortgage calculator; you enter your variables and determine what you can afford. You can then change a variable to drive a different decision because now you’re saying I can afford a little bit more payment. In this example, the analytics are modifying certain variables to get a different outcome. We want to bring this capability to our business users to enable them to play with the data and change the variables to drive different outcomes and different behaviors.
MC: So it’s like giving instant access to the people that need to be tweaking their models and running different scenarios.
VB: It allows them to change the variables and enables them to say I can now take the predictability of this drive from X number of hours to X plus Y number of hours if I were to change these variables. We have done a lot of work to become more proactive and predictive. Another example of this is in our Sales organization where we are looking at our customer’s propensity to buy, and how we can become more valuable by offering them the right tools. Leveraging our data scientists, sales built models around likelihood of maintenance contract renewals. This is now deployed to business and has resulted in more renewal uplift for EMC.
MC: You talk about analytics the way I do actually, which is that there’s historical analytics or business intelligence, business analytics that’s focused on asking questions about past data. It sounds like you have swallowed the pill in terms of predictive and you’re on your way towards or now achieving some predictive analytics. You early on in the conversation talked about prescriptive or actually taking the models and pushing them into either a business process or a product change or an application for doing drive maintenance differently, right. Where do you find yourself–have you declared victory, or where are you trying to get to on that curve?
VB: I think we are trying to get to where you talk about very predictive capability and how do we transition from predictive to prescriptive to a more ground-breaking enablement for our products. I believe we are at the nascent stage and are just starting to harness Pivotal’s capabilities. The goal is to make that a part of my core development no matter what I do, whether it’s a mobile app, a business intelligence app or an ERP app. If that becomes my core capability and I have everything linked together, we can have end-to-end visibility into a transaction through the entire enterprise. For example, this would provide more predictability into the success of our new product introduction processes.
MC: Man, I would love that. I’m in product marketing. Are you kidding me? If I had that kind of data, I would own the world. People would listen to me.
VB: Of course. However, there’s a lot of heavy lifting that needs to be put into place. We’re on the first step of that process–evolving how IT is organized from a traditional IT organization to a more contemporary and elastic consumption model.
MC: Alright, just two more questions. First of all, for a similar business going through the same transition, what were the early things—the gotcha’s—that if you had thought about earlier would have ramped the project faster, or made your life easier and saved some of your weekends?
VB: Well, we started this as a complementary service. We were not able to solve some of these tough problems and were having time to value problems. We had unstructured data we couldn’t ingest and could not be resolved in traditional databases like Oracle. And, we needed to continue evolving our traditional database and data warehousing skills to support big data and analytics services.
One of the biggest challenges in the industry is availability of skills to deal with this new platform of architecture. We have data science capability and data ingestion experts, but it’s still something we’re bridging.
The other one is about the concurrency. We are building a brand new infrastructure with hundreds of nodes of scale out architecture. We did not foresee the amount of success with 19 analytics projects already in the enterprise nor the appetite and demand for even more help throughout the company.
While it’s great being popular, it creates scalability challenges. We have put on some bandages, but we will need to permanently fix some of these challenges. This is one area where the collaboration with Pivotal has been very helpful. We were trying to figure out how do we design the next solution and also replace our traditional global data warehouse built on Oracle to have one massive data architecture where we can scale to do analytics as well as enterprise reporting in a heterogeneous pool.
MC: Final question: what are the next challenges ahead of you that you’ll be tackling in the next six months?
VB: The main challenge is redefining our model of IT. One of the things that we have done now in IT is evolve our traditional roles like a CTO, or DBA, or application administrators. All of the roles have now been changed into services roles like cloud services, data services, security services, app services. The IT leaders are now responsible for providing anything that’s related to their service area in a software development life cycle not a two-year life cycle.
We no longer say, can you give me a couple of DBA’s to provide a logical design and then build the physical database supporting it? It has to be consumed as a service. That’s the challenge that lies ahead of us—converting that model of IT from those silos of the past to a much more service-oriented architecture. We then ask how I am going to leverage all the data services? What data do I need? How is my code going to be scanned for vulnerabilities? Do I have naked SQL’s? Do I have embedded passwords? How do I consume that as a service? That’s how we have started to think about IT to really provide this as a service.
MC: There, we have come full circle. We started the whole conversation in data management and data powering business information, analytics driving potentially new applications but where you just finished is really with the statement that the only way to achieve that is with a platform as the service and that’s the finish of a journey path that we have identified for our customers as well—that you need to first be able to collect all these different data sites, then, you need the right analytic tool that your people know how to use to analyze it, and finally that you need to learn agile development so you can take those insights and build applications rapidly.
The thing you just summarized perfectly is those application developers need instant access to services and they need to be abstracted from the complexity and so is the IT operations team really. We can’t afford to rack and stack hardware at the request of the BU anymore. We have to be ready to provision instantly.
Anyway, Vic, I just want to thank you. Congratulations on this prestigious distinction by CIO magazine and thank you so much for taking the time to share your story with us. Looking forward to continued collaboration in the future.
Learn more about CIO Top 100:
- CIO 100 Winners Turn Analytics Into Money-Making New Products
- The 2014 CIO 100 Winners: Showcasing the Transformative Power of IT-Business Innovation
Learn more about Pivotal Big Data Suite
About the Author
