Pivotal People—Andrew Clay Shafer, DevOps Polymath, Joins Pivotal

July 21, 2014 Stacey Schneider

featured-andrew-clay-schaferAt OSCON, in 2010, OpenStack was announced. Andrew Clay Shafer was there for the announcement, with a keen interest in building and operating open source cloud infrastructure. James Watters also attended OSCON in 2010 with an interest in open source and cloud services. A chance meeting turned into a late night conversation about the challenges and opportunities of managing all the different layers from the metal at the bottom, up through the systems, to applications and services. This turned out to be the first of many such conversations.

This year at OSCON, we are pleased to announce Andrew as a welcome addition to the Pivotal team. A ‘start up person’ who spent most of the last decade working on venture funded projects, Andrew made seminal contributions to the devops movement while helping IT organizations improve with better tools and processes. He has been frequently asked to present his little ideas on devops and cloud management related topics. Shafer is an O’Reilly published author contributing to the book ‘Web Operations’, and has also been a guest on ‘radar’. In addition to his work in technology, Andrew also concerns himself with people, and how they interact with that technology and each other. He’s been a core organizer of devopsdays since 2010, and this year is on the program committee for culture and operations at the Velocity Conference and the devops track co-chair at Agile 2014. James Watters asked him to help with the Platform and CF Summit, helping to convince Andrew he could be part of something special at Pivotal.

In this Q&A, Shafer shares his journey before joining Pivotal and how Pivotal aligns with his vision of people, process and tools working together.

1. Tell us about you growing up. Where you lived, siblings, school, activities, interests, etc.

I grew up in Plain City, Utah, the oldest of six children. School is a terrible place to get an education, but I eventually spent a year at Reed College before finishing a degree in Mathematics at Weber State and then a masters in Computational Engineering and Science at the University of Utah. I’ve done some interesting things, but we’ll save those stories for another day.

2. Tell us about your work background and how you came to Pivotal?

My first job in technology was doing technical support for America Online. I was going to school on a debate scholarship while studying math, but I knew enough about computers to get some extra spending money on the weekends. I could tell you what speed a modem was negotiating just by listening.

After the University of Utah, I started working at a venture funded startup in Salt Lake City working on a platform for managing applications and identity on the BlackDog at a funded startup named Realm Systems. After Realm wound down, I was torn between following David Flynn from there to Fusion-IO, which didn’t have any funding yet, or another group of the Realm Systems diaspora to Infopia, which had just raised 5 million. At Infopia, I went from being a developer building data analytics on the ecommerce platform, to being responsible for all the databases, to managing QA and technical support, to a team responsible for redesigning the organizations approach to sales, product management and technology. That was a crazy ride and I’m leaving out the details, but I definitely learned and grew in the process.

During this period I also became a regular at the Salt Lake Agile Roundtable, studying and discussing Agile methods with Alistair Cockburn and company. This whole time I’d been having conversations with my roommate from Reed about getting involved in a project he had been working on. That project was eventually called Puppet and I left Infopia to work with Luke Kanies on changing the way people manage systems. That started another adventure and my foray learning even more about systems and the people who run them. I was in the right place at the right time to help connect some dots between innovations in infrastructure and Agile software development.

At some point, Luke and I clearly wanted different thing for Puppet and I went on to the next chapter. Randy Bias pulled me into Cloudscaling and I ended up as VP of Engineering building and operating some of the largest OpenStack and CloudStack deployments in the world at the time. There was a short tour of duty at Rackspace and some consulting along the way but I basically spent the better part of a decade helping all manner of startups and enterprises adopting innovations in tooling and process.

I’d been watching the Cloud Foundry project evolve since the early days at VMware. I’d first met James Watters at OSCON when I was at Cloudscaling. James knew I’d been thinking about the evolution of technology but also just as much about open source, communities and how people adopt technology so he asked me to get involved the Platform event last Fall. I got to meet Rob Mee and see the momentum building around Cloud Foundry. Through the years I’d run into Pivotal Labs people along the way and I was almost jealous of the pure approach they were able to take with Agile engineering practices. I’d also been a big fan of Elisabeth Hendrickson’s perspectives and enthusiasm, so I definitely noticed when she decided to work for Pivotal. Then James asked me to help again at the CF Summit and arranged for me to meet Scott Yara. When I discussed Pivotal with Scott, the opportunity to combine the data capabilities and the Agile software capabilities together with the trajectory Cloud Foundry is taking aligned too well with my vision of where I believe things are going to say no. So, here I am.

Andrew Clay Shafer

3. What is your vision of where things are going?

This is probably better explained interactively at a whiteboard, but here goes. Technology transitions through phases. There are lots of interesting models that capture aspects of this but the predictability of commodification Simon Wardley preaches and the cycles of value migration suggested by Adrian Slywotzky come to mind.

Part of me doesn’t like the ambiguity in the marketing jargon, but for lack of a better word cloud computing represents a transformation in the way computational resources will be consumed across all industries. Everyone says ‘as a service’, but only a few organizations are really managing services yet. This is as much about mindset as technology. I won’t pretend I know how Google or Amazon manage their infrastructure but I’ve been fortunate enough to have had many interesting conversations with people who are actually doing that. If cloud computing means anything, then cloud is applying all the lesson that were learned building the big web and making the results generally consumable.

The infrastructure layers are starting to be widely adopted, but in the transition people are still focused on machines because that’s what they know. We’ve been building abstractions around configuring and orchestrating machines, and that’s powerful, but the end goal isn’t machines, the end goal is the services. Services don’t exist on one machine, so building and managing them has to account for the fact these are distributed systems.

There is also an ever quickening pace of delivery that requires predictably consistent deployments of existing and new services based on feedback cycles. This is a long journey and the social engineering is at least as difficult as the technology, but I see Cloud Foundry as an empowering technology that allows operations to declaratively manage distributed services as a top level abstraction while ensuring consistent application of policies and self service to the frontline developers. There are other approaches emerging around these same themes, and other organizations rallying to productize Cloud Foundry but with the additional capabilities Pivotal provides, we have a unique opportunity to enable our customers to take full advantage of a complete solution. That might seem like impenetrable jargon, but I assure you the words have meaning. Find me with a white board sometime and I’ll makes sure it makes sense.

4. What is your role at Pivotal? What are you most focused on?

My title is Senior Director of Technology and I work for the Head of Products, Scott Yara. My role is to do everything I can to support the Pivotal mission but then I think that should be everyone’s role. Circumstances have me working remote from Pittsburgh for now and I feel humble and grateful to be given a fortuitous opportunity to be part of this undertaking. For the moment, Scott asked me to focus on fostering a vibrant technical community around Cloud Foundry and look for ways to help align Pivotal’s products strategically with the opportunities in a market being transformed by cloud computing, open source, agile, devops and data. I know the most about Cloud Foundry right now, and then probably Pivotal Labs, but I’m looking forward to understanding the data products and our other communities to help build up synergy.

5. What are you most excited about being able to do in your new role at Pivotal?

I love to learn. Pivotal represents the confluence of different technologies and methods that I already have experience with in isolation, but what will be possible with seamless integrations between a fabric of data and applications? I’m excited to help us mature our internal devops capabilities and bring that attitude into products that are a pleasure to deploy and manage. I’m also excited to help demonstrate the transformative capabilities of Cloud Foundry’s elastic runtime and BOSH can already enable to the operations community.

Pivotal is about ten times larger than most of the organizations I’ve been part of before now. I’m excited to see what can be accomplished with the resources Pivotal has combined with the urgency and intensity of startup culture. We have all the pieces we need to build something special together. That is exciting.

6. What do you like to do in your personal time when you aren’t living and breathing Pivotal?

When you have 3 kids and a wife in a medical residency, you don’t have a ton of personal time. I like to read academic papers and texts on various topics. Last year I read books and papers about ‘organizational learning’. One thing I have enjoyed in the last few years are the quality courses available on various topics online. The last one I completed was the ‘Intro to Dynamical Systems and Chaos’ at the Complexity Explorer by the Sante Fe Institute. That’s how I tend to fill time, if there is a break in the action. The next one might not be for a while. I aspire to being more active and my wife likes to run, which means I have been known to lumber through an occasional 5k or half marathon.

7. Is there anything else you would like to add?

Just what an amazing time this is to be building infrastructure and applications. These inflections points have disproportionate impact on how everything else happens in the future. I’m not sure where everything ends up, but I’m passionate about being part of building tomorrow. If you want to help build the future, Pivotal is hiring.


About the Author


Semantic, Yet Flexible: BOSH Release Versioning
Semantic, Yet Flexible: BOSH Release Versioning

In this post, Pivotal Engineer, Karl Isenberg, explains how BOSH now supports semantically versioned releas...

10 steps to a successful project handoff
10 steps to a successful project handoff

At Pivotal, we rotate developers, designers, and product managers during a project. One of the reasons we d...

SpringOne 2021

Register Now