From sleepy storage admin jobs to MongoDB, there’s no end of jobs operations people can be doing now-a-days. Fresh off many years being an operations person, Bridget Kromhout (@bridgetkromhout), now at Pivotal, talks with me in this episode about DevOps and operations. We discuss the opportunities operations people have in a Cloud-Native world, moving to and from management, organization change management, being “promoted” to management, and, of course, USENET.
PLAY EPISODE
SHOW NOTES
- Subscribe to the feed
- Feedback: podcast@pivotal.io
- Links referred to in the show:
TRANSCRIPT
Coté:
I was watching one of your old talks, and you made a reference to a, what was it? A Solaris Storage admin job in the suburbs. I was wondering did you just make that up or did you actually know a nice storage, Solaris admin in the suburbs who was happily going along reallocating space?
Bridget Kromhout:
Hi Coté. No, I actually got a recruiter contacting me in the last couple of years for a Solaris storage admin job out in some far-flung suburb that I’m pretty sure I would have to buy a car, and then drive the car and then, these are just things that are not going to happen.
Coté:
Yeah, yeah, you can’t use your, what is it, your “transit oh my god” business to get out there, you have to buy a car, it’s terrible. This is one of the major criteria people in the tech world seem to have, is the transport to and from my job. Is it long, does it require a car, it’s very systems-type thinking that I think they get into.
Bridget Kromhout:
Honestly I prefer a job that requires me to go to the airport from time to time rather than one that requires me to go to a specific place where I use my computer.
Coté:
Right. Yeah, yeah I didn’t realize it until I was looking but you have a lot of talks. You’ve been out and about, how did that start up?
Bridget Kromhout:
Well, let’s see, I was the one woman ops team at an AdTech startup called 8thBridge in Minneapolis, where I did go and ride my bike the 7 blocks to the office to go into the office every day. My boss at the time, Jamie Thingelstad, who was involved with this local unconference thing said, “You should go talk about our MongoDB there” and I thought “I could talk about Mongo DB but it’s a very dev-focused conference so I’m going to bring one of my dev coworkers, Mike Hobbs to talk about the dev side.”
Coté:
Sort of like, your conference shield, hold them back.
Bridget Kromhout:
Right, I was like “There’s not even on Ops track at this conference.” I felt a little weird, but it was actually a great experience, I got to talk about how MongoDB is set up as a service, which I think everybody pretty much agrees with now. It was, I realize that even if you don’t have all the answers or you’re not an expert TM with something, it’s still valuable to share your experiences with other people.
Coté:
Right.
Bridget Kromhout:
Just keep doing that.
Coté:
As a little parenthetical, what is the deal with MongoDB? It’s sort of like, I haven’t used all these newfangled tools that the kids are up to nowadays, but it seems like everyone loathes MongoDB and yet it’s really popular. Or is it sort of like just making fun of people wearing the wrong type of pants or something? Like it’s sort of like an elitist thing to do, or it is actually like a loathsome thing? What’s up with it?
Bridget Kromhout:
MongoDB is actually great, if you want to prototype something as a developer and you just want to write some JSON and just store some documents. The interface is super simple, you don’t have to write any SQL, it is great for that. Unfortunately, it made a lot of marketing claims as to being, “webscale” but, to hilarious results, but also tended to have data losing bugs with every release–
Coté:
That’s sort of not what you want with a database, one might argue, the core feature.
Bridget Kromhout:
Right, so it got a reputation for being a pain for people who were trying to operate it. I opened a lot of bugs with them when they were still called 10gen, about things like the way their connection pool lazily checked connections such that if you changed your master route, you would expect that all of the open connections would not be valid anymore, but no, you got to handle that on the application side or every single one of the ones open in the connection pool will be tried and fail. It’s just like, what?
There were a lot of things like that, that made it difficult from an operational perspective, but if you’re just trying it out and don’t actually expect to operate it in production or any kind of scale, I’m sure it’s perfectly fine. It’s probably–
Coté:
Yeah, yeah. I, my SQL aside, I always wonder if the 2000s would’ve been different if Oracle just lowered their prices.
Bridget Kromhout:
If Oracle just stopped trying to sue everyone?
Coté:
Yeah. I mean if the close source alternative to all this stuff was, people more gleefully used it, for whatever reasons they may not like it, but it is a, databases are a strange space because it seems like there’s so many different ones of them nowadays. They’re all highly-specialized to a task and it’ll be interesting if things consolidate down just a few general purpose ones, but it doesn’t really seem like that’s the way stuff’s going.
Bridget Kromhout:
There is definitely a lot of room for data storage for unstructured data, right? At 8thBridge, that same, couple of jobs ago now, I also ran an HBase cluster and that was specifically because we had a firehose of data coming out of Facebook back when the Open Graph API would let you harvest not only the data from people who had offed your app but also some of their friends likes, so we had a great deal of data coming in from Facebook and it was not necessarily structured in a way that we could predict, so having schemaless data being dropped into a wide column data store is pretty handy, especially if you plan on mostly just running MapReduce jobs on it.
I think that there’s a lot of room for a variety of different kinds of data stores and I don’t want to malign MongoDB, document stores have their place, I just think a lot of times you end up realizing that there is structure to your data, so that’s sort of, that sort of thing where, I think I’ve made jokes before about how you don’t make your important technology decisions by running a Markov bot against the front page of Hacker News, you don’t necessarily want to choose a data storage just because it’s the hot trendy new data store that all the kids these days are using. Got to look at your problem space first and see what you’re trying to accomplish first before choosing a data store.
Coté:
Yeah. That brings up something I was interested to hear your take on, because I encounter this quite a lot, when we go out in our Pivotal world and tell people they need to do all this crazy cloud stuff and think about DevOps and things like that. Given your background as an operations person, I am not an operations person, I used to be the bad developer who would choose easy things and then have other people suffer for it, that was my form of entertainment I guess, but a lot of operations people, I go over and talk about DevOps and cloud with them, and basically the question they ask is like, “So where’s the part in your chart where I get fired?” The other way of asking that is, in a more genuine way, or helpful way is they’re curious what an operation person does beyond the obvious of working with the developers.
The reason this popped up in my mind just now is it seems like, if you have lot’s of different types of databases, well someone’s got to figure how to run all those weird things. There’s almost, It’s almost plays into the old Jevon’s Paradox or Jevon, or however you say it, of, well now we have an unlimited amount of types of middleware in services we can run, so someone needs to work on that, which is not going to be the developers.
Bridget Kromhout:
Nice. How many things with V on the front of them do you want to run today?
Coté:
That’s right, but the more general question is, as, I mean you’re well traveled in the scene where people would ask this, but how do you, what do you tell Ops people who are freaking out?
Bridget Kromhout:
Right, I would tell them that I totally am a sysadmin of the old school, as they possibly are. I poured one out on the curb for SunOS 413 okay? I was sad about Solaris, because come on, really? We’re going to go system 5? Even if we’ve been doing this for a while, there’s still always new stuff to learn. Yes of course, if you would like to just use Smitty or whatever, the same way as you’ve done for the last 20 years, it’s possible you’re going to have a bad time, but as long as you’re interested in learning new things–
Coté:
Or you could work in the suburbs, apparently.
Bridget Kromhout:
Well, I mean, there’s a very long tail of adoption. There’s probably always going to be places that you can do the thing you did 15 years ago, but if you are interested in learning new things, there is an infinite amount of new things to learn, and there’s, I like to say that if there’s, if it’s possible to automate a way, some portion of your job, that was the boring part. You should definitely automate that part so that you can do all of the more interesting scaling problems, or all of the more interesting solutions you can provide that can bring actual business value. The part where you just ran some script again and again, probably not the most interesting part of your job anyway.
Coté:
Yeah, I always think of that, for as terrible as it ended up, that wonderful part of Lost where they just had to push a button all the time. Seems like something that needed to be automated. I mean, I guess it did have great value, if I’m not pressing the button, something terrible will happen, but it turns out not. What are some examples of fun, interesting things to do instead of pushing the button that you’ve come across?
Bridget Kromhout:
Sure, exactly. For example, if you would like to migrate some of your workload to “The Cloud”, which by the way, I’ll say that the cloud is kind of BS because the cloud is just someone else’s computer. In some cases, perhaps that computer is in US East 1, in Virginia, and you’re renting time on it. It’s not this mystical magical thing, it’s just a series of computers somewhere, and if you’re going to migrate some of your workload to say, AWS, for example, there’s a lot of things you can learn about how to scale, how to perhaps autoscale up and down, how to connect their VPC’s with internal VPN’s you may already have. How you can have better data locality, so that you’re not constantly incurring the cost and also time sync of moving data around, depending on where you’re processing it. That kind of comes in when you’re talking about running massive jobs on some amount of data. Well, can you localize the data to some sort of place where you’re actually running those jobs?
There’s a lot of ways that you can add value that aren’t necessarily repetitive action. I would discourage people from wanting to pursue repetitive action if they’re interested in computers, because I think computers are really good at performing repetitive actions, the same, many, many times in a row. Humans less so. I think we should stick to our strengths, which is like creative problem solving.
Coté:
Yeah, it’s almost like the, I don’t know, tell me if you think this is fair, but it seems like in the Ops world you can divide up into at least two categories of, if we take out security and network admin and other kind of, not niches and small, but specialized tasks, and there’s basically, here’s off the shelf software and stuff, and also desktops, and someone has to make sure that stuff operates correctly. Then there’s another category with stuff of, here’s uniquely combined together IT that we have caused to come into existence and we have to constantly make sure that it’s operating properly.
It’s almost like a unique application or a system that we’ve done in-house and it might have our own software we’ve written in it or whatever. That’s where probably the bulk of the interesting work is, in maintaining your custom stuff, versus “time to add more hard drives to the exchange server,” like keeping your ERP system happy and things like that.
Bridget Kromhout:
I’ve actually had an opportunity to talk to some individual at some large enterprises where they are worried about things like their ERP system but also how people operate, interoperate with it. I think that there’s probably interesting problems in all of the realms, you don’t have to be a website, public facing, Open API, whatever, in order to have interesting problems to solve. I do think that perhaps running an exchange server yourself in-house these days is possibly better not done. I would say that you’re probably going to add more value to your business by just going and getting one of the numerous cloud solutions that’s available for that. Having people actually help, I don’t know, make the bring-your-own-device solutions work better for your enterprise, or something.
Coté:
Yeah. Exactly, there’s almost, even in that area of, not your customized software, there’s deciding what all that stuff is going to look like and then making sure it’s wired up properly. Making sure that if I’m working from home remotely, if my kids watch Netflix, it doesn’t ruin the day’s work. That would be an interesting problem to solve, for me.
Bridget Kromhout:
Is your Netflix profile something that is also affecting your work? Or can you not set aside the kid profile?
Coté:
Oh, yeah, it’s just a matter of sucking up the, it’s got this annoying, sucking up the bandwidth, it has this annoying thing of, my profile is now stuck in kid’s profile, and it won’t move over, and I don’t know what’s wrong, it’s misclassified me. It’s terrible.
Bridget Kromhout:
Well it kind of sounds like, for your internal routing in your house, that you need some sort of QOS.
Coté:
Sure. I don’t know what any of that is, I’m busy filling up a Mongo database with a bunch of unstructured data. I have no idea about them.
This does raise another small cultural point. Now, since you’ve been working here, I’ve been learning a lot about your buddy Joe there, I know he wears red wing boots, I got to look those up, but do you guys use different, do you pay for different services on your Netflix and Spotify and things like that or do you use one account?
Bridget Kromhout:
That’s a funny question, I think that’s probably one of those philosophical questions in any relationship, and the answer is, we don’t actually share any accounts. However, the person with the strongest opinions for whatever it is, is the only one who usually bothers to have an account. For example, I don’t have a Netflix account at all, I never have. I have very few opinions about what we actually watch movie-wise, but I have a lot of veto power, so he has a very long Netflix queue and he watches a lot of it when I’m out of town.
Coté:
But then isn’t he stressed out that you’re going to ruin the robot? Things you pick, if he doesn’t like it, it’s going to feed into the robot and next thing you know, he’s going to be recommended the wrong things.
Bridget Kromhout:
I’m not sure that he puts that much stock in the recommendation engines inside Netflix versus like, I think he reads the Onion AV club and talks to friends and what have you.
Coté:
Yeah. See, this is a crisis that’s looming. You have all these individuals and you’re going to be customizing stuff to the individuals but then you have families and if they intermix their stuff together, it’s going to pull families apart. They’re going to realize how different they are. We’ve got to obscure and hide this.
Bridget Kromhout:
Or they can just learn to accept some of what each other likes. It took probably 12, 13 years into our relationship for me to actually start watching the Green Bay Packers, I just wasn’t a fan of American rules football at all because I grew up as a hockey fan, which by the way, as a Texan, even though you’re not from Dallas, you have some explaining to do about how Dallas stole our hockey team when I was a kid. That was unacceptable.
Coté:
I don’t know, I’m sure money was involved.
Bridget Kromhout:
But yeah, I eventually started, I don’t know, having sort of a Stockholm syndrome appreciation for American rules football and now I’m actually a Packer fan. We both have Packer stock, he got one when, in the original offering, and I’ve got one from my birthday a couple years ago, we got them framed together. Some people put like a ketubah frame on the wall, we have our Packer stock framed together.
Coté:
Unifying. I guess your counseling is right, just, this is an opportunity for you to get closer with your family by, you’ve got your finely tuned music on Spotify and every now and then, some crazy 90s punk stuff is going to pop up in your list, and you’re like, “Where did that come from?”
Bridget Kromhout:
Are you telling me you have a problem with Fugazi?
Coté:
Well, I like that one where they’re not talking. That’s a good album.
Bridget Kromhout:
Oh my goodness.
Coté:
Anyhow, that stuff’s popular up in the Midwest, as I remember, so I should watch myself.
But, it’s changing topics drastically, speaking of SysAdmin-ing. I was, you do that Arrested Development podcast, which is good stuff, and I was listening to one of them recently, it was the one, I think it was episode 40 with Charity and that other guy, she’s pretty un-escapable because she giggles all the time. It’s a very, very known voice there.
Bridget Kromhout:
Charity is great.
Coté:
Yeah, yeah.
Bridget Kromhout:
We had MCR and Patrick from Etsy on that episode as well.
Coté:
Yeah, there you go, and you guys started talking a lot about the usual sort of thing of, what’s the part that I started keying into?
The thing that I thought was really interesting was the point you guys were making about when, everyone needs to understand that when you go to management, that’s not a promotion.
Bridget Kromhout:
Oh, right.
Coté:
It’s a career change.
Bridget Kromhout:
We had Lindsay Holmwood on the podcast recently, with Courtney Nash, actually, talking about cognitive neuroscience. He wrote a really good blog post that, if we have shownotes for this, we should put it in the shownotes, about how management’s not a promotion, it’s a different job.
Coté:
Exactly.
Bridget Kromhout:
I have actually done that different job, I did management for, from 2005 to 2012, before I decided, stuff that I want to be an individual contributor again.
Coté:
Yeah, and there was a good discussion of that kind of divide, which I think the, my lack of being able to talk about it quickly is the whole point, it’s like an odd topic of, for many people, management kind of blows, right? Yet it’s, counter intuitive is the wrong word, but it’s almost like, not the common sense of how you would go about thinking about your job, like your goal is to get into management because you’re excelling. It’s that, you get the call from your parent and they’re like “Are you taking care of yourself, how’s your job doing? Have you been promoted to management yet?” They’re checking in on you.
It struck me, in listening to that, that a lot of what we talk about in the DevOps space is like, trying to emphasize that the simplistic, common sense thing is what you should do, instead of what’s normal to do. Normally, as we’re joking about earlier, the developers write some stuff and put it in MongoDB and hand it off to Ops people to take care of, so that doesn’t work out. A lot of stuff that we talked about in the DevOps world over and over again is, “Hey look! That doesn’t work, so we should stop doing it.”
Bridget Kromhout:
Possibly we should talk to each other about our needs before we choose a data store. I mean, in that particular case, I blame no one because I walked into a MongoDB already in progress, and made the best of it, but, and that is the sort of conversation, that, say I were in Ops and working someplace where people wanted to put MongoDB in production, I would say, “Hey, let’s talk about this.”
Coté:
Yeah, indeed. I used to, and I still do, I always get a little upset when it’s “You should just do the smart thing.” But given the various, you have an interesting job history of been at basically, not basically, a big ISP, and then a university, very normal, large, institutionalized places and then several startups where they’re doing things, and I wonder, I don’t know, I mean what’s the deal with people getting stuck in a rut of not just following the common sense? How do you, especially since you’ve managed, how do you move them over to doing something that’s obviously better? Which, the broader question is, how do you do change management, cultural change management of moving stuff in? That seems to be the part that everyone always bulldozes up against, is people just keep doing the same bad stuff over and over again and we can’t convince them to do otherwise.
Bridget Kromhout:
I think in many cases, and I’m not going to say I was great at this this when I was in management, but I think in many cases, people are motivated by local optimizations, as Andrew maybe would put it. They’re thinking about what they’re day-to-day is going to be like. Or they’re thinking about whether or not they get the thing that they want if the thing that they want is, the exciting, trendy new framework that they’ve tried at home, and was fun, or whatever.
They don’t necessarily feel motivated to see either what’s better for the organization, or what everyone can agree on. Trying to get people to talk amongst themselves, honestly, about what they need, what they want, what benefits they think this particular technology choice will bring, I think those are the hard conversations.
A friend of mine tweeted just the other day that, who was it, something along the lines of, “What was this thing that they told us about how you don’t need communication skills to write software? Because software is mostly communication.”
Coté:
Yeah, yeah, That’s a good example, I remember seeing that and I remembered, did you ever read that book Microserfs? There’s an episode in it when one of the programmers goes off into his room and has a flat food diet, because the only food they can give them are the things that can be slid under the door.
Bridget Kromhout:
Oh gosh.
Coté:
The flat food diet mentality doesn’t really work out too well in that case, but it is–
Bridget Kromhout:
To your point, I’ve worked in some of these really large organizations and you end up really just optimizing for your tiny unit, however small you define that unit as being. When I worked at US West, and then we got bought by Quest, they laid off a bunch of managers, including mine, I reported to a voice on a speakerphone in Denver, I never met him. That was not what I would consider to be the sort of experience that really motivates humans. I’m actually going to London next week, or going to London tonight, speaking next week at a conference where I’m going to be talking about distributed systems in teams, and I think a big part of working successfully on a distributed team is feeling that human connection with the people you’re actually working with, even if you don’t see them everyday.
Coté:
Right. I think that’s definitely the case. It is, you’ve been at Pivotal, how long now?
Bridget Kromhout:
A month.
Coté:
A month, right? Yeah, and both of us work remotely, so it’s, I haven’t worked remotely at such a large company in a long time. It’s interesting, seeing the different remote working patterns play out and not play out in a company like this.
I think you’re right, that is, that’s a good way of really, drastically boiling it down, especially with remote working, you need to make sure that you’re not just a disembodied head somewhere, because it’s easy for each employee to just slip back into, “Well, I’ll do nothing helpful today,” essentially.
Bridget Kromhout:
Right. Then in terms of also, whether you’re remote or in person, isolation from the larger picture, I think Pivotal does a pretty good job, from what I’ve seen so far, in spreading information between teams, making sure that people on one team might understand some of the goals, some of the needs of other teams. If you work someplace where that’s not so much the case, where there’s a lot of entrenched battle lines and fiefdoms, and people trying to hang on to their little area of control, that, in a large organization is, that can be endemic. It’s a huge problem, right? Because that’s the siloing that we all talk about. Farm metaphors aside, it’s basically, if you’re only optimizing for local rationality, and you don’t even look beyond that, you’re going to have a bad time as an organization.
Coté:
Then no doubt, you’ve put together this talk already, and rehearsed it countless times, right? You probably know it forward and backwards.
Bridget Kromhout:
If by put together you mean I have some slides and keynote, sure.
Coté:
Sure that, if you were to highlight what you think some of the major things to get wrong that are fixable with remote working in distributed teams are. What would be those things? What are the important things to get right?
Bridget Kromhout:
I really, I think that it mostly boils down to communication, because if people feel like their management or their coworkers are distant from them, whether in terms of interest, in terms of connectivity, whether or not they’re seeing them across the cubicle wall is almost immaterial. The important part is engagement, the important part is actually connecting with one another. Get that right, and it doesn’t necessarily matter where the people are, since we’re knowledge workers, we spend most of our days inside our own heads anyway. Being explicit, like yourself, being explicit about what you’re doing, and then also having everyone around you actually, from time to time, engage with what you’re talking about, what you’re saying. I think chat is great for that, obviously, but any collaborative work tools can help that but people have to actually put in the effort to try to use them.
Coté:
Yeah, that’s one thing I’ve noticed. To a certain degree, at Pivotal but especially at other companies, is the idea of an information radiator in cyberspace is apparently very hard to get people to understand and do. Which is to say exactly what you’re saying, it’s somewhat easy if you’re in person, to trust that information will disseminate, just word-of-mouth and mouth-to-mouth, that started to sound a little weird there.
Bridget Kromhout:
Hopefully we’re not resuscitating anyone, but osmosis, right?
Coté:
Exactly, you have one of those funny little mouth jobs that the lifeguards all have nowadays. Keep it nice and sanitary. Pass your information from brain to brain.
It does seem like, I’m always shocked, I still am, looking in the white-collar world, how little information sharing there is, because usually in the development world, at least in most places there’s at least the information sharing of your version control system. There’s probably a ton more than that, right? But in general walks of life, as you were saying, being explicit about things and sharing information, it’s oddly hard to get people to do. Which is, I don’t really understand why.
Bridget Kromhout:
It depends on how they’re incented right? If information hiding can advantage you or if being the one with the scoop and getting in on the ground floor first or whatever, and cut some kind of competitive scenario with other teams, or with people outside your organization. If that would, if security through obscurity is actually providing you some sort of perverse incentive, you’re going to do it. People do what they’re incented to do and they do what they’re measured on.
Coté:
Would you say that’s like a optimistic or a cynical view on life?
Bridget Kromhout:
I mean, maybe I’m being cynical in saying that, if you incent people to hide information and to undercut each other, that’s what they’ll do, because humanity is, we’re all flawed human beings, but I mean I would say the solution to that is not just to tell people to be better but to incent people differently. I guess that’s where management comes into it.
Coté:
Yeah, I ask the question because it’s easy to think that it’s a cynical view of things, right? That people–
Bridget Kromhout:
I think it’s realistic.
Coté: Yeah, but it strikes me, for all the exciting business lore and white collar stuff that I read all the time, it’s very little spoken of, of like, you need to set up incentives correctly. This is a good example of one of those common sense things that we talk about in the DevOps world, that people seem to just not, it just doesn’t really fit into their world view, and it’s sort of like, if you set up a system where people are incented to do something, like incented not to share information, then they won’t share information. I don’t know what to tell you.
Bridget Kromhout:
Well, I mean I can give an example of something that I did wrong at the university, which is, I tried very hard to keep our open tickets low. That did not necessarily translate to people being incented to make sure that those who sent in requests were happy with the result, it was more of a, try to make them go away as quickly as possible. Perhaps a band-aid solution.
Coté:
That’s an example I’ve stolen from, I think it was a year ago, I saw a ThoughtWorks presentation. Or no, it was at the beginning of this year, and they were presenting at a local cloud group, and they were complaining about traditional IT. They were saying, well, the ticket people were paid on how often, or how many tickets they would close, so if you didn’t fill out your ticket perfectly, they would close it, right? So part of what, we lost several months just learning how to write, learning the camp if you will, the ticket camp, like how to properly write the tickets.
It’s another example of studying the, if you’re in a management position, or something like that, studying the system and figuring out what’s operating correctly and not. That’s a huge part of what, many of the conversations we have in the Pivotal world start and revolve around is once you get this stack in here, now you need to change how you operate quite a bit. I find myself, the reason I’m going on and on about this is, talking about those issues pretty much all the time, because apparently they’re hard.
Bridget Kromhout:
Yeah, who would’ve thought? I mean, I also think that there’s been a lot of talk lately about more structured solutions, or how opinionated would you like your software to be or your platform to be, and that sort of thing. I think that, that might be the wrong way to look at it because, ideally, I mean many of us were builders, right? We like to tinker, we like to play with things, a whole bunch of software Legos and Lincoln Logs that we get to glue together, look like fun to a lot of people like us. At the same time, letting people decide what’s going to best for their business based on what looks the most fun to assemble, is maybe not necessarily the most productive conversation to have. It’s like going in and saying, “Hey, what are you actually trying to accomplish” and starting from there, drives a better tool selection process than saying, “What looks the most fun? Let’s see if we can find a project that’ll use that.” Because I’ve totally done that, but is it necessarily the most constructive use of time? I don’t know.
Coté:
Yeah. We even have a common way of putting it, is having the right tool for the job, but there’s even a rule before that, which is, “Do you need a tool?”
Bridget Kromhout:
Or have you defined your problem space?
Coté:
Exactly. You go hang out at a lot of DevOps Days and things like that, and you helped run the one up in Minneapolis, or the Twin cities I guess, I’m never sure what I’m supposed to call it.
Bridget Kromhout:
We’re technically, I mean our DevOps Days takes place in Minneapolis. The Twin cities are, in fact, they’re in the area.
Coté:
There you go. Do you know why the C on their baseball hat is like the, as you say, what you call it, “American rules football”, like why it’s the same C that’s up in Chicago there? How did that happen?
Bridget Kromhout:
I have no idea. Don’t ask me how the Sports ball people come up with logos. I mean, I assume they did that before, it probably happened before they started naming all the stadiums after various banks, but I have no idea.
Coté:
I mean, apparently you own shares in the Packers, so you’re committed now. You’re on the bacon side of things. We’ll have to look this up, maybe someone listening knows.
Anyways, so when it gets to this idea of looking at the job and the value, how would you say in the DevOps community, we are at, I don’t know, getting product managers involved? I don’t know how to say it but every now and then, like Damon Edwards always comes in and gives this talk about value stream mapping and things like that, but do you think the maturity of the, maturity is the wrong word, because it’s always cyclical, but do you think at the moment, in the the DevOps Days world, we talk about product definition enough?
Bridget Kromhout:
I think that that’s, I hope that that’s where things are moving, I mean there’s definitely been a lot of talk about tools, and I think tools are one of those visible ways that somebody can sell to their boss why they need to take two days off of work to go hang out with a bunch of their fellow nerds. Oh, I’m going to learn about X, Y, and Z.
Coté:
Just like training, essentially.
Bridget Kromhout:
Right, but I think the larger value is when you start stepping away from a specific tool and seeing how people interact with the tools, how the tools can help us drive value. I have seen more and more talks being proposed and being accepted for DevOps Days that, like some of the talks you’ve been giving, that talk about the larger businesses’ needs, Andrew Clay Shafer was talking about this at DevOps Days Pittsburgh back in 2014. I think we have started going in that direction, I think there’s still an awful lot of emphasis on tools, and then people kind of wonder, “Why is there all this squishy culture stuff?” I want to learn about, fill in my favorite tool here.
Coté:
Yeah, that’s true. I mean, there is, I don’t know what the percentage breakout is, but it kind of whacks back and forth over the years. I think it was the goat can podcast, where there was a bit of a diatribe about how we’re deep in a DevOps culture conversation cycle now, which a couple years ago we kind of escaped. Now that there’s more and more people who are interested in DevOps, we have swung back to talking about that, sort of like the introductory stuff of DevOps which is encouraging people to think culturally, which to be fair, is a lot of what we’ve been talking about, right? You need to start switching around your process in doing things, which is yet more softening of, it’s actually okay to talk about culture stuff. We shouldn’t get sick of that.
Bridget Kromhout:
I don’t think we should get sick of that, just because, if humanity going to continue to have these issues. Also especially when you’re going with a, just from conference design, if you’re going for a single track conference and it’s not super, hyper-focused like say, Monitorama, then when you have a really tool-specific talk at a DevOps Days, you do risk having people tune out. I think it’s pretty important that even your technology-specific talks have some takeaways in them for the wider audience who may not be using that specific tool.
Coté:
Right, that gets to another topic I’m going to talk to you about. You not only go speak at a lot of conferences but you help organize several of them, as I was talking about earlier, and in the space of infrastructure software, Cloud-y stuff, DevOps, whatever, in your mind, what’s the ideal setup for a conference? Let’s say it’s a general purpose one, not a specialized one, like here’s a conference dedicated to just monitoring. How are you starting to see and like balancing conferences out? Is it all talks, or all not talks, or what’s your ideal format?
Bridget Kromhout:
Open space, as we say at DevOps Days, definitely has a lot of value, but if that doesn’t fit into your specific kind of conference, like say if there’s no open space at Velocity, on the other hand, there are birds of a feather sessions, there are topical lunch tables, but more importantly there’s enough time for hallway track. I like single track conferences, I’m also fine with multi-track conferences, but the conferences that just have talk after talk after talk after talk, are not only bad for people with small bladders, but are also sort of bad for letting people digest the information, talk to the speaker, talk to other attendees, even have time to get from one room to the next.
I would say if you’re choosing topics, theming things in a room is okay, spreading them out to make sure people do have time for the serendipitous encounters in the hallway is also a strategy, but making sure that there is enough time in between talks is, just like passing time in the hallway in high school, is actually important, or people are going to have not, I’m not going to say a bad time, but they’re going to have a better time if they have an opportunity for those serendipitous connections.
Coté:
Right. Then so, if you were to give out advice for conference goers, what are some things you would throw out to them?
Bridget Kromhout:
Let’s see, I would tell them, don’t be afraid to sit in the first couple of rows at a talk, because speakers don’t actually enjoy talking to empty chairs, they could probably do that at home, they probably own chairs. They’re not going to hate on you, probably not call on you, there’s usually not, out of the first three rows will get wet sort of sign, we’re not at Seaworld–
Coté:
Or a Gallagher event.
Bridget Kromhout:
Right, it’s usually fine, yeah you’re not usually at an improv show, where if you sit up front you risk getting pulled onto stage, so it’s probably fine to sit up where you can actually engage and interact with the speaker. If you’re worried that you might have to bail, just sit on one of the sides. Then if you get paged or whatever, because #Opslife, you can sneak out.
Then I would also say, at lunch, go sit at an empty table. This is advice from Tom Duffield actually, one of my DevOps Days Minneapolis and DevOps Minneapolis meetup co-organizers. He gave a really good Ignite about conference tips for introverts, where he says, “Sit at a table by yourself at lunch. There’s a limited amount of tables, people will have to sit with you, and they will have to talk to you.” At the very least, they’ll say, “Hey, can I sit here?”
Coté:
Oh, that’s good. I was just going to ask that, because I feel like over the years, I have mastered the art of sitting at a full table and not making eye contact.
Bridget Kromhout:
If you go into the full table already, you can get away with that.
Coté:
Yeah, because you can sort of hide in plain sight, as it were, right?
Bridget Kromhout:
They probably have a conversation started and you can just kind of look at your food or look at your phone, but if you would like to interact with the other people at the conference, which I would say is often a goal, because we can all watch conference videos for free on Youtube. If we didn’t want to interact with any other humans, there would be very little point, going to the conference.
Coté:
Indeed.
Bridget Kromhout:
If you want to go to the conference and interact with other humans, I would say, feel free to actually go up and say something to the speakers, and feel free to actually talk to the person sitting next to you in the audience before things kick off. You might end up finding out that they’re actually, because they’re sitting in the same talk as you, there’s a very good chance that they’re also trying to learn similar things, or that they have similar problems, or maybe they work someplace super cool that you would like to work, or maybe they have exactly the skill set that you’re trying to hire for. I mean if you don’t talk to them, you’ll never find out. You know you already have something in common with them because you’re both sitting there at that talk.
Coté:
Then back when you were at the universities, I was reading through which stuff you were up to today. You listed USENET as something that you monitored and set off. What was that like?
Bridget Kromhout:
That was actually an ISP bill, that’s what the universities would say.
Coté:
Even better, that seems like it would be a lot of work. What was involved in that Web USENET stuff?
Bridget Kromhout:
Honestly, the part that was the most work for running news servers in the early 2000s was the fact that we had this ITIL-like process, where to make any changes to production we were supposed to fill out this 42 page form and then wait 2 weeks for some people in the Change Control Review Board to decide whether or not we could do the thing. When somebody would complain about missing parts for their binaries, we can draw a veil over exactly what it is was they were trying to download, but probably pictures of cats, yes that is what the internet is for.
Coté:
Or maybe cars.
Bridget Kromhout:
When people would complain, the easiest solution was just to add some more news peers. It was a completely non-invasive, simple change to production. We did it all the time routinely. Every single time, I think the statute of limitations has expired on me admitting this, I would just do it, and then I would file the Emergency Change Control Process, which was a much shorter form, because I was not going to wait two weeks for some people who had no idea what Usenet was to tell me it was okay for me to do a simple thing that we needed, and ideally the sooner the better, so that the missing parts could trickle in. That’s an example of how heavy-handed IT processes can end up making even the simplest tasks unbearably painful and then also incent employees to lie.
Coté:
Yeah, well it also highlights the example of it’s, to steal a metaphor from a famous book of that time, you’ve got to smell the cheese often to figure out if it’s gone bad. In the sense of, at some point, it was a good idea, or there was a reason to set up this 2 week process, and instead of revisiting that and figuring out if it’s still a good idea to have, but if it’s satisfying the needs that you have, you don’t revisit it. It’s yet another example of, well that seems like common sense, but it’s almost the, it’s hard to go make up why something like that would happen, but often it seems like people just kind of slip into autopilot and they don’t really look at their job as continual gardening of IT processes they have in place. Instead it’s just like, fortification of unchangingness.
Bridget Kromhout:
But what I really strongly believe, is that there is no such thing as standing still, right? Because if you choose to take no action, that’s still a choice, and the world’s going to keep changing around you, whether it’s just the environment that your servers are operating in, or bit rot or the way your users interact with it, there’s always going to be change, so trying to enact some sort of change freeze is just using your ability to react to the change, it’s not going to prevent change from happening.
Coté:
Indeed. Yeah, like you said, you’re not going to be able to check your binaries. That’s going to be terrible. Yeah it’s great, I found that because I, Usenet is one of those things I forget about often. Then every now and then I’m reminded of it and I’m just like, “What a crazy system that was.” I mean, not really crazy but how different the whole attitude and architecture and everything of Usenet was versus the way things are nowadays. Did you ever have to set up Gopher servers?
Bridget Kromhout:
Despite the fact that I worked at the University of Minnesota, I never had to admin a Gopher server. I think Mark McCahill actually left the University of Minnesota, went to some other university where I think he had plans to bring Gopher back, I have no idea whatever came of that.
Coté:
Yeah, that one just lost out. Got creamed like everything else, by the old web.
Bridget Kromhout:
Hey you know, Waze and Archery and Veronica were something, but we at least got to something that works okay, most of the time. I mean, if you think of the original RFC’s that were, if you look at, say RFC 822, you think about email started out, well obviously they had to add on a lot of stuff like SPF and DCAM and all sorts of things later, to try to deal with the abuse of the system. It’s like that constant iterating, informed by conditions, is a super important part of how the internet works.
Coté:
You’ve been here a month, as you’ve been saying. I watched, I forget where it was but it was a pretty short presentation you gave, where you were in a very nicely wood-paneled room and you’re like, “I’m going to go work in marketing” and the whole audience kind of erupts in laughter, along with myself.
Bridget Kromhout:
Was that my “Adventure, Excitement, a Jedi Craves Not These Things”
Coté:
Yeah, there you go. I hadn’t thought of that but ironically, it’s sort of ironic. What made you want to sign up for the adventure and excitement of marketing? What attracted you to this lot in life? Because tell me if I’m wrong, but this is the first non-practitioner job that you’ve had, essentially?
Bridget Kromhout:
Yeah. Yeah it is. I was perfectly happy working in Ops at DramaFever, and then my boss Tim Gross was leaving, and when important life changes like your boss leaving happened, you reevaluate what you want to be doing.
Coté:
Smell the cheese often.
Bridget Kromhout:
My reevaluation had me talk to Andrew Clay Shafer. He’s a very convincing sort of man. I had talked to a few other people, and it was just seriously about Ops jobs. Then I talked to Andrew and he made this very compelling argument over the course of a 90 minute phone call as to how I could still play with tech and also talk to people about and educate people about and continue to learn about tech and it would be fun, so I figured I would give that a shot. I hadn’t really thought to myself, “I want to work at Pivotal” or “I want to work in marketing” it’s more, Andrew convinced me that this could actually be really interesting, and so far it is.
Coté:
What would you call it? As always, ask a question then talk a lot, but it seems like there’s this taboo nowadays of just calling it, well we used to, which is evangelism, which I don’t know if that’s good or bad, and so it’s always a question I have of, as the job you described there, right? How do you call that thing? What is that?
Bridget Kromhout:
I really don’t know, I mean the problem with the term evangelism is that sort of sounds like you’re just out there trying to promote your thing. That’s not really what this conversation is about, right? I’m interested in learning a lot about all facets of the ecosystem, and trying all sorts of different stuff, and maybe collaborating, cooperating with people who are competitors, or frenemies, or whatever you want to call them, I don’t have to be in a, strictly speaking, I am only going to talk about how great our thing is. That’s actually not my role, so I think maybe that’s why evangelism kind of sounds like you’re doing that, and this is more just general technologist, I guess?
Coté:
Yeah, it’s, I don’t know enough about the security word but I always think it’s kind of funny that people will describe themselves as a security researcher, which usually seems to mean, “I hack into people’s systems and tell them about it.” That is a researcher. To some extent, that’s an interesting term for what you’re talking about, what many of us on the team here do is, we’re sort of, I always explain what I do as, I try to understand what problems are happening in the industry, and explain how people are fixing them. Sometimes I’ll say something corny like storytelling or something like that because that just sounds quaint, but to some extent, it is sort of like researching, you’re still looking into technologies, not even still, you’re figuring out how these technologies work and fit together and solves people’s problems. Instead of publishing it in a classic research type of way, you’re just disseminating that information, which seems like it would be helpful.
Bridget Kromhout:
Right, so whether it’s like blog posts, conference talks, podcasts, whatever, contributing to the the general store of information and knowledge out there is super valuable. I did a lot more case study type talks as a direct practitioner, and I think that moving forward, my talks look like they’re going to be a little bit more of a synthesis, still some actual practical stuff, also observing some possibilities, looking at some larger patterns.
Coté:
Researching.
Bridget Kromhout:
Perhaps.
Coté:
Yeah, security researcher. Always a fun category. To that end, before we wrap up here, what do you have planned out there? Speaking of traveling out there into the world, as you humorously wrote somewhere, “Not all of the tech conferences are in the Twin City area, yet.” As you’re going about, what are things you have in the books that people can look up to you and maybe make eye contact and talk to you at?
Bridget Kromhout:
Let’s see, I’ll be at Operability.IO in London next week, I will also be at Reinvent, I’m not speaking, I’ll be at the Pivotal booth talking to people about AWS and Pivotal stuff. Velocity New York, I am speaking, at that one I’m actually, one last hurrah of talking specifically about a DramaFever case study, because that talk I submitted some months ago with a now-former co-worker Pete Shannon. Velocity EU, my talk is actually going to be a little bit more of a general platform sort of talk, not just the DramaFever docker in production story, then let’s see, I think I’m going to panel at FutureStack, I’m giving another, the same talk from London about distributed systems and teams at ReCon, and probably some other stuff that I’m not remembering at this very moment. We did submit, Casey West from our team and I, did submit a joint talk to Cloud Foundry Summit in both Berlin and Shanghai so I guess it’s possible, that people could expect to chat with me at one of those.
Coté:
Whoa, whoa, what was the topic of the talk?
Bridget Kromhout:
That one is about platforms that enable a continuous delivery, and the tooling and also organizational choices and operability of your platform that can allow for a CI/CD sort of workflow.
Coté:
Right. Making sure your stuff works.
Bridget Kromhout:
Ideally, our stuff would work.
Coté:
That’s right.
Bridget Kromhout:
That is something that almost everyone wants, I would imagine.
Coté:
We’ve already mentioned the Arrested DevOps, is that weekly? I lose track in my podcast, listening, I just listen to whatever’s there, but how often does that come out?
Bridget Kromhout:
Sure. Arrested DevOps comes out, usually twice a month is what we aim for–
Coté:
Oh, that’s relaxing.
Bridget Kromhout:
Mm-hmm (affirmative). We’re not quite as ambitious as Software Defined taught can be from time to time.
Coté:
Yeah, yeah. Well that, we slack on there all the time.
Bridget Kromhout:
Well we also try to schedule guests so it probably makes the scheduling a little bit more spaced out.
Coté:
Oh scheduling guests, that’s the worst. See that, that is a problem that needs to be solved. Scheduling. I think that’s beyond all of our capabilities. We’re down here at the Morlock levels but someone way up there needs to figure out scheduling. I don’t know what the deal is.
Bridget Kromhout:
Yeah, calendaring, not a solved problem. Teleportation, also not a solved problem, all this large array of various insundried conferences on multiple continents is something that I think is going to require me to spend a lot more time on airplanes than I really prefer.
Coté:
Oh yeah.
Bridget Kromhout:
I got me some of that TSA global entry, and pre-check business, but it’s still a giant airport-shaped hassle.
Coté:
Yes, yes. Next thing you know, you’ll be installing the Starbucks app.
Bridget Kromhout:
Oh, I don’t even want to think about what kind of life choices would lead me to the Starbucks app.
Coté:
You just start preparing yourself. Then when it happens, it’ll be completely normal, it’s fine.
Bridget Kromhout:
I already have the set of life choices that makes it cheaper to have the monthly go-go inflight wifi subscriptions.
Coté:
All right. There you go, we’ll talk about that next time when you’ve had some more travel, even more travel under your belt, we’ll have travel tips. I’m sure there’s a laundry list, travel tips for former practitioners.
Bridget Kromhout:
I think that’s a good bet, is there being a great deal of information that can be gotten there. Certainly from you. Possibly for me at some point.
Coté:
Where, you have some main hub of information on the interwebs there, right? What’s your main place people can go to?
Bridget Kromhout:
Probably the easiest way to keep up with me is to either follow me on Twitter, Bridget Kromhout on Twitter or BridgetKromhout.com, I tend to link an update to any talks I’m doing, any podcasts that I’ve been on, possibly this one, I think will be out on the public internet at some point. If they send me email it will go to the bottom of the pile and it will take me a non-zero amount of time to reply to it. That’s my warning, that’s my warning on email. There’s a lot of people who like to email me requests for my time, and that is probably not the optimal way to get my attention.
Coté:
Yeah, yeah I have some emails sitting in my inbox from my financial planner of some forms to fill out. It’s just like, that’s never going to happen. I don’t know how to fix that problem but filling out the forms, it’s too big of a batch, it’s a good metaphor, and I need to have a small, if you ask me 2 questions everyday over the series of like 2 weeks, it’s probably more likely than if he sends me a big spreadsheet app to fill out.
Bridget Kromhout:
Yeah, we had the, I actually just signed the contract for the venue for DevOps Days Minneapolis, 2016, which is going to be July 20th and 21st in Minneapolis in 2016, mark your calendars now, do not schedule your conferences on top of it.
Coté:
That’s right.
Bridget Kromhout:
I entreat you. We just signed that contract but I think the thing that took me the longest was just, there is no change control or change tracking. There’s no versioning of any sort when they’re sending you hotel contracts, so just going through and reading it again and again and again.
Coté:
That stuff is a nightmare. Which, then, yeah. This is then another problem we’ll put on this problem to solve, introducing DIFF to the rest of the world and building that into the system.
Bridget Kromhout:
Getting people who send you banquet equipment orders to use version control, would be a thing.
Coté:
Well, on that note, as always, this has been Pivotal Conversations. You can find the shownotes that we’ve referenced, that don’t exist yet but they will, once, if you’re listening to this, at Pivotal.IO/podcast with an s or without an s, depending on how you’re feeling about it.
With that, we’ll see everyone next time. Bye bye.
About the Author
