Once IT has decided to change to a new way, how do you manage that change and keep up the energy?
We discuss a recent experience Casey West (@caseywest), newly of Pivotal, had in transforming to a small batch mentality and some of the challenges—and rewards!—his team faced.
PLAY EPISODE
SHOW NOTES
- Subscribe to the feed
- Feedback: podcast@pivotal.io
TRANSCRIPT
Coté:
You started working at Pivotal just a little while ago, right? I think it’s like, is it three weeks now?
Casey:
Yeah, today would be day 12, so it’s about two and a half weeks.
Coté:
Day 12. I’m like five or six months in. You figured out what the remote worker should do when they’re working for one of these wacky bunch-of-free-things-to-eat-in-the office companies. You go buy yourself some doughnuts on the first day. See this is a trick I did not apply.
Casey:
That’s right. Well, yeah, I think getting doughnuts on the first day is definitely good. The other bit about it was having a co-working space here in Pittsburgh helped me out quite a lot. Having a place to go even on the first day was kind of nice.
Coté:
Now are you’re one of these people who when you’re doing your remote working you make yourself actually shower and wear a clean shirt every day? Do you have that kind of structured routine?
Casey:
I do. I have the mentality that I could always be on a video call and I should be at least somewhat professional in that environment.
Coté:
There you go, at least somewhat professional.
Casey:
Right, well at least in presentation. Whatever get said that’s up for debate about whether it’s professional.
Coté:
Then more so than even just being let’s see a node in a network of people remote working. You’ve actually had the experience of managing multiple teams or doing leadership if you will remotely, which seems like a whole other level of having to be somewhat professional. How did you come into the position where you were managing people remotely?
Casey:
I should actually make a clear distinction between the concept of managing people and being a technical leader. They don’t always go hand in hand. On paper I certainly wasn’t an official manager, but I did have three teams that I led and certainly they did the things I asked them to do, which were largely born out of the things they already knew to do, just giving them the autonomy and the ability to push them forward.
But the way that I ended up in that position was taking a role at a company at the time where I was a remote individual contributor, a senior engineer. They were trying to rebuild their user interface web application. Struggling to do so taking a couple of years a lot of folks who were working on the project had never done anything like a single page front end application before, especially in the browser, especially built on top of a REST API. I came in and helped them push that project forward so that it could be shipped.
Unfortunately, by the time we had reached that point there was enough technical debt that it was difficult to maintain and difficult to advance. So it didn’t really live up to the promises they hoped it would under those circumstances and so we decided to venture into a new world where we would build web applications from the ground up, specifically designed to be easy to maintain and easy to deploy, easy to change overtime, respond to the market.
At the time I was the UI architect at that company and moved forward from there to make a plan to rebuild again the user interface. It was certainly costly but the right thing to do for the company at the time. The way that that brought itself into leading a few teams was that before we could rebuild the user interface and ship and deploy quickly we had to have some culture built around the idea of agility and moving more quickly, and this company was not quite there yet.
You have to lay the foundation before you can build a house, and the foundation that we had to lay was more of a DevOps culture. Instead of having a conversation about whether or not we were using the right programming language or the right frameworks, we had a conversation about our processes and culture and methodology of delivering software, delivering functionality to the customer, and figuring out how to reduce the overall time it takes and the overall effort to get something from the developer to the customer.
If we had ventured out to rebuild the UI alone without that then we certainly would have still struggled to deliver functionality to the customer, because we did not have a culture of quick iteration or moving fast.
That was a long answer but what ended up happening was I had to build out a DevOps team in order to get us over the hump of going from really no automation in a process that was, I dare say, a little more waterfall. From there, once we had that in place, we could build a UI team to build web applications on top of that supporting structure. Then just before I joined Pivotal we had spun up another team to do different web application but along the same lines.
Coté:
You know I have this bad habit of entertaining myself by reading a lot into stories and what people say. With that disclaimer said it sounds like there’s an interesting multiple layers of getting some software out the door that you were just going over. One of them is like well obviously you need to write the code that has the UI. That was what you had in the first case, like we wrote a bunch of code that resulted in the UI that was working, and then the notion of technical debt of course is we made some choices or compromises that meant in the future we were not going to have as much flexibility as possible. It became slower in the future for us to like, I don’t know, add new features or change things around. The end resulting product while the code was working was not sort of like this flexible fungible thing that we needed.
In that case it seems like, I like the second part of your story where whether it was explicitly done or just sort of as these things often do, sort of accidentally turns out in a good way to do it this way, it’s sort of like well what we really need to do, what we need to do in addition to building code properly is we also need to create the practice of how we go about doing code, all the process and the culture, we need to be more mindful of the organization that’s building this thing and how it operates and what the product that comes out, everything that isn’t actually like the code and the tools and everything.
Then in that sense, and again, I’m reading a lot into some of the stuff you’ve been writing about, I guess it’s recently, it’s recent to me because I just read all of it, about technical leadership, but it’s like part of being a technical leader in addition to the mentoring and other kind of things that you talk about is you’re sort of architecting and programming the process and the practice and the organization or at least part of the people who are doing that so that you ensure you can get to release two and three. Because there’s a certain amount of, and I won’t say or anymore, but in the craft of programming we don’t spend a tremendous amount of time on organization as much as we do on code, if you will.
Casey:
I think that’s right. It’s really critical though. Any piece of software does not exist in isolation and the craft of building it is something that happens over time. Building a product is a journey. In order to go along this journey you have to have the right resources and the right mindset. You have to be in it for the long haul, ideally. Not always but that should be an explicit decision.
I think that Fred Brooks often talks about the socio-technical team or the socio-technical product that what we’re building is a social thing as well as technology and they go hand in hand. You need to curate and be careful about the architectural decisions and the software design decisions that you make in your code. But you need to be just as careful with the decisions that you make about your organization and your process and your culture. If they are both moving forward and evolving together then I think you have a healthy ecosystem where healthy products will be your end result. If your culture isn’t healthy though then I don’t think that your software is very likely to be healthy either.
Coté:
Right, right. I mean if your technical factory, I always hate the metaphors of factories and machines or whatever so that’s great. I guess you could use an organism instead. That sounds like the type of rhetoric that people like. But whether you want your machine of factory or an organism if you’ve got a bunch of corroded bits inside things are not going to work out well essentially. It’s sort of caring and feeding of how all those bits fit together and making sure they’re healthy.
Casey:
Yeah, imagine if you were going to run a marathon but you did no training. You were not prepared. Your body wasn’t prepared. You didn’t eat the right things. Your muscles are not ready. You’re going to have a bad time.
Coté:
It’s funny you say that. That reminds me. There’s a podcast Speaking in Tech, and one of the hosts on there every year does like the doughnut marathon or something where I think they eat like a dozen doughnuts and then see how long they can run.
Casey:
Oh my goodness.
Coté:
They do it for charity or something of course, but anyhow.
Casey:
Hurting yourself for charity. At least it was an explicit decision, right?
Coté:
That’s right, and it’s for a good cause. The purest reason to eat doughnuts. I’m curious in the kind of narrative arc there, like there’s a couple of…I would imagine, and tell me if I’m wrong and insert other moments, but there’s a couple of typical moments that I certainly have gotten involved in discussions with like they’re all fond of the category of so how do I change, or like that’s nice for all those fancy people who are successful and can do things, but I’m a normal organization. How do I get through the change?
The first one you sort of, I guess you could call it realizing you have a problem and that you’re going to need to fix it, and that fixing it requires what seems to be downtime. It’s like we ended up with this piece of software and it doesn’t quite do everything that we wanted, so now we need to get together and it’s going to take us some time to get things back into order, like we can’t just simply add a new feature to this. We need to re-architect how we do things and we need to initiate pretty large change.
One, that’s a broad generalization, but when you got to that point and when you’ve gotten to that point historically like what does someone in a leadership position end up doing? Like how do you win that if you will, so you can move to the next phase which is actually doing things?
Casey:
That’s a great question. I think the trick really is as you said to make sure that everyone agrees that we have a problem. There’s a book that I love by Gerry Weinberg which is “The Secrets of Consulting.” In it he says very clearly that you should not accept a consulting agreement where you see a problem and the client doesn’t see the problem. So you can’t put it—force it, I guess is another way to say that. The organization has to agree that they have a problem. Certainly the evolution of the user interface software that I was describing before, everything came to a head when we released a version of the customer facing application that everyone knew was essentially dead on arrival and so we had to sit down and have a hard talk about that.
You said that, I’ll use a different word, but you said that in order to get the opportunity to actually address that problem you need to have some slack or some leeway in the process. One of the biggest fears about getting to a place where you need to really overhaul your culture is second system syndrome which can happen in culture and in software. You don’t want to try to rebuild a new thing with all the bells and whistles that you always wanted and then you end up holding your company hostage because you’re not actually shipping.
One of the big challenges as a technical leader is to figure out how to avoid a situation where your organization feels as though they’re held hostage that they’re not getting any value. What’s challenging about that is that if you’ve gotten yourself into this position in the first place you’ve done it by not paying down the technical debt, and we should dive into technical debt in a little bit and discuss it more broadly.
But if you’re not essentially re-factoring either your code or your architecture or as we’ve been discussing your culture and process is around delivering value, then they atrophy and you get to a place where you can’t ignore that anymore and you have to make big change.
When you find yourself in a position where you need to make big change the best thing that you can do is advocate for it. I feel that I’ve been most successful in essentially never letting up. When you have a big problem and everyone can see it, but you’re not given the opportunity to address it, you have to constantly make the point. It can make some folks feel uncomfortable that you’re just pushing and pushing and pushing, but the truth is that it is the presenting problem.
The presenting problem for us was certainly not just that we had delivered a piece of software that we could not maintain, we could not get to those releases two and three as you said before. But the reason that we couldn’t get to those releases two and three while there was a technical aspect to it, the root cause was a culture that was built around moving slowly and releasing infrequently and introducing a huge amount of risk in the process. We needed to reduce that risk over time.
One of the tricks in not letting up isn’t just to hound and hound and hound without providing any alternatives, but to actually come up with some ways of making it better. Certainly there is a lot to be gleaned in recent history from the DevOps culture, the movement that DevOps as a term or an umbrella term seems to encapsulate about being lean and lightweight and iterating on everything from process to software.
But there’s another trick that I think is also helpful which is finding the pain that others are experiencing and figuring out how to articulate that their pain will be resolved if we make a big change.
Coté:
It’s sort of like… You see that “Wolf of Wall Street” movie? It’s sort of like the sales 101 course where they’re sitting at a table and one guy is like, “Hey, can you write something down for me,” and the guy says, “I don’t have a pen.” He’s like, “I have a pen,” and he just creatively made for him the need to acquire a pen.
Casey:
I haven’t seen that movie. However this is an economic principle. Unfortunately I actually forget the name of it, there is a term for this, but there is a principle of not determining what is in it for you as the sales person and certainly advocating for changes to sell it. We are all sales people in that respect. It’s not to articulate what makes it better for you, but the better value is to articulate what makes it better for others.
For example, in my prior experience, the one that we’ve been talking about, there was a single production data center and there was an interest in spinning up another one. Our IT team did not have a culture of automation at the time. That was a large project. It was going to be very costly. We knew that by having to deploy to multiple environments we would increase the cost of release because it was a fairly manual process.
The idea of automation in your release process to get somewhere close to continuous delivery or certainly further down the continuum of the continuous delivery, that would ease our burden as engineers because we can get smaller releases out the door faster. That’s good for us. But in order to get anywhere close to that we also had to automate and codify our infrastructure and our production operational configuration. There were many machines out there and we needed to understand what types of machines there were and how to configure them and how to repeatedly configure them, and so we had a lot of work to do from the ground up.
Folks who are used to a Pivotal process or Pivotal software, including things like Cloud Foundry, this will seem like a lot of stuff that you don’t have to deal with on a day to day, but there are many organizations out there that are nowhere near the level of automation or sophistication that a platform provides. That was the case that I was finding myself in. In order to sell this to IT who is nervous about a lot of things coming out development because development has a practice in breaking things in production, especially when they’re brittle to begin with, I was able to show them how automation can reduce the cost and the effort to spinning up another production data center.
Doing so helps convince them that this might be a decent idea and then we got buy in from IT. We already had buy in from development and that helped out a great deal and using the pain of change or at least the pain of agreeing to change.
Coté:
Yeah, I mean to brutally simplify it it’s like the stone soup method of creation, right? Like I’m hungry, I want a bowl of soup. How can I convince these other people to help me do it? I have to find a problem that they have or an interest and then get them on-board as well. I think that’s a fantastic example in the sense of at least in my experience a lot of what developer and technical driven people, developer people do is they take an argument for, I don’t know what the technical term is, but it’s kind of like the platonic way of arguing something, which is here is the truth of the matter, and once you’re exposed to the truth of the matter or the logos you’ll be compelled to comply to the consequences of it.
As an anti-version of what you were saying, or the bizarre-world, it’s like we need to speed up our least cycles which means we need to automate the production environment so therefore you must go automate the production environment, which it really address anything you were going over. It’s like again to put it in… One could interpret it as cynical or just like real world, like you haven’t really expressed what it’s in for the ops people. There’s really no benefit except you telling them what to do.
Casey:
Exactly. I think I take the role of being an architect pretty seriously. I think a software architect has a very significant role to play in any organization and I think the technical track of leadership as opposed to the human resources track is really critical.
Architects often get a bad rep as being ivory tower, which means they sit on high in their tower far away from the individual contributors on the ground who are doing all of the hard work and they pontificate and create these beautiful conceptually beautiful designs. Then it gets passed down from on high and everyone is supposed to think it’s beautiful and just start doing it. Everyone laughs at the ivory tower architect because they’ve missed a critical component. I think this is the component that is the social part of a socio-technical system is that you have to figure out how to get from where you are to that beautiful concept and you have to provide a plan. We talked about the journey, and it is a journey.
One of the most valuable things you can do if you aspire to be a technical leader is to always understand how to get from where you are to where you want to be. It’s not just for you to understand but to actually articulate and share that vision. That really is the social aspect of it. We talked about sales a little bit and I think it’s very much that.
Coté:
Speaking of things being like a journey that was the other thing I was curious to ask about is, it seems like we have this dichotomy notion of day one and day two problems around Pivotal. It seems like in the, I don’t know, in figuring out we’ll just sloppily apply the idea and doing your architecture or technical leadership we just describe like the day one problem which is like now you have permission to do go stuff and resources and stuff like that. The day two problem is like so it’s three months into that, and how do you make sure people are still engaged and don’t get bored or kind of cycle out?
Because it’s very common to see like here’s our world saving mission we’re going to all go on and then you never hear from it again. Like an initiative gets in motion. It’s almost like there’s another type of craft which is making sure that ongoing you’re like topping off the energy level and keeping things running and going, like what the day to day is. I’m curious to hear how you managed that process, because that seems equally a wicked problem to solve.
Casey:
Sure, yeah, I would say that, I might suggest that a lot of what we have been describing comes before day one. I like to think of day one as the first release and not necessarily the announcement. But once you first get everyone fired up with the prospect of change and the possibility that good change can actually happen, then you actually have to deliver.
There’s frankly a lot of hard work in that. In order to bring about a culture of automation we first had to change our software delivery cycle. I said before we had a waterfall process some people like to call it scrummerfall. We used all of the Scrum and Agile words and then we mapped them to our very waterfall process. In order to change that process in this case just as an example we brought in different tools. We were running on SVN. That was fairly updated for the time. We brought a GitHub enterprise server and then an open source model of software development with multiple repos and pull requests and forks. That was a complete change in the behavior of work.
To get people down this path we had a huge amount of work to do to coordinate the development, QA, release, IT, technical writers, product management, even marketing that were messaging out to customers about changes that were coming, because we wanted to go from releasing every three weeks to releasing every week, and in order to do that we wanted to overhaul the entire process.
The huge amount of work there, again, this gets right back into the social and cultural aspect was coordinating all the teams, but doing all of the heavy lifting. We had a small skunk works team, just a couple of engineers that were doing a huge amount of writing and conversation around the changes that were coming for everyone and convincing individuals from every development team to participate with us as a trial run and they became our champions.
We were able to seed it out carefully and it came from the ground up. All of us pushing the change, we’re practitioners, software developers that otherwise would’ve been in the trenches building the product. We were trying to jump ahead on the skunk works project that would get us down the path to a tighter iteration and release cycle.
Coté:
It sounds like part of what you’re saying is in order to move to a three week release cycle if I heard that right and also get-
Casey:
From three weeks to one week actually.
Coté:
So move down to a week release cycle and you’re also introducing the new practices and tools that are required for that. You said something, the theme interesting is there is a lot of tool coding and maybe process coding that needs to happen or changes in engineering. The part of what you guys did to make that work out was you almost, you said you had a separate your skunk works people who were doing that work. It’s almost like you took the pain of doing all of that meta work if you will, that engineering off the table for other people, so that I guess they could focus on something else. I mean does that…
It’s almost if I were to force it. It’s sort of like this is going to be a hard process and I can take like three or five people and make part of it not so hard because they’ll do all that work. You as the overall organization can benefit from all of this tooling that they’ve done, and we don’t have to suffer through all that tooling and it makes it easier to swallow the pill of change.
Casey:
I think that’s a great way to rephrase it. We certainly did the heavy lifting of the tools I think, both technically and socially. Also I guess evangelism might be the best way to say it of explaining how this will be better, not just to the engineering organization but to the wider product development team which includes a lot more disciplines, many more disciplines, and even further than that the folks that are dependent on what we deliver.
There were a lot of conversation that had to happen ahead of time to make sure that everyone’s concerns were addressed. There were concerns about releasing faster meant the possibility of introducing and not finding security issues quicker, which we had to talk through and make a plan for. There was messaging out to customers about release notes and we had to make plans for how we were going to deal with that because we were potentially going to release three times as fast so that may mean additional work for other teams.
There’s an idea that engineers don’t like talking to people, which I don’t think is actually true because I’ve been to conferences. But nevertheless certainly when we’re at work we like to be heads down in the code. I did, in fact, take a handful of engineers and we did all of the hard work of preemptively making sure the rest of the organization was on board with the change and understood why it would bring us to a better place. So by the time we rolled it out to the wider engineering organization everyone understood the implications and it was a matter of a release day essentially.
That was just getting to the first step. That was our day one. It took about three months in that organization, but that was a huge amount of change in three months. I don’t think that it would’ve been possible if we tried to iteratively change the entire development organization without doing any of that ground work.
Coté:
So then after that initial sort of like the first ship if you will, I don’t know, in month four, five and six, how was the overall organization perceiving it? Like where there interesting sort of epiphanies or insights that they had? Or did everyone just convert over to like, well this is better, let’s keep doing it? Like how did you see the realization that… Well hopefully it was a better way. How did you see the realization that this is a better way of operating overall kind of roll out onto the organization?
Casey:
I think what we found was being able to release three times as fast meant that each release was smaller. One of the big wins that we saw was the reduction in the risk in terms of the amount of change that was happening in each release. As the amount of change grows so does the risk of the change being problematic, and we were able to keep the release sizes relatively small.
What happened there was that IT and the release team in particular and to some degree the QA organization understood that our releases were likely to be successful and they could have greater confidence in them and it reduced the amount of effort that they had to employ on each release. We went from a verification cycle that took a couple of days. Actually scratch that. For every three weeks it was one week of verification by QA and release team and we went from that to about a day. That over the course of three weeks was a two days savings.
Now as an aside ideally you’ve got automation and in development tests that are thorough enough to give you a high degree of confidence in your releases and you don’t have to have a release level of verification unless it’s fully automated and largely those are integration and infrastructure type of tests. But again many organizations are nowhere near ready to do that, including the one that we’re describing now.
That was something that certainly helped the team that was worried about actually doing the releases and shipping the code. The developers realized that they had less stress to rush code to completion. If we have a three week release cycle and you miss a deadline to get some code in for the release then you may be waiting another three weeks on top of that which is a six week development and release cycle.
The pain of that is if you are that late then you may miss customer deliverables. The expectation was that it would get in and that someone’s waiting another three weeks and that creates problems for you. By going to one week it was a much shorter cycle where if your code wasn’t ready then you could wait just one week and it would go out the next week. That was certainly better for us and better for our customers.
The benefit to this in terms of the quality was astronomical. By reducing that stress we found that folks were rushing less to get code in the door at the last minute, and that meant that they were talking a little bit more time to concentrate on the quality. Now I will say that this wasn’t universal. It takes some time to change behavior, but a number of teams were able to understand this and make pretty big advances in the quality of the work that we were pushing out because they felt that they could take the time to make it right.
Over the course of three to six months after that we found that our quality increased, which meant that our confidence in our releases increased and the amount of human labor associated with the deploy went down. A deploy under that scenario was still an event. Certainly you don’t want your deploy to be an event. You want it to be uninteresting and to happen frequently almost as if people wouldn’t even notice, but again we weren’t quite there. But it was much less of an event anyway.
Coté:
Yeah, I mean it strikes me that doing software releases is a lot like the delightful annoyance of taking your kid to school. Every morning where you’re like, “You know, yesterday you had to put your shoes on. Today it’s going to be the same thing.” Yet like, it’s almost like when you get to that state of not being an annoying event finally your kid learns I’m going to have to put my shoes on. One day, I don’t know, your kids are a bit older than mine, maybe you can tell me, but it seems like the transition to school smooths out at some point, just if you’re a little more mindful of it being a regular occurrence.
Casey:
It does. The beauty of software is that we can get further and further automated and further way from a process like that being an event. I’m thinking about The Jetsons and wishing that I could just put my children inside of a machine on one side and on the other side they pop out ready for school.
Coté:
I guess that’s the advantage that we have software land, we can automate things.
Casey:
I think that’s a good analogy. It’s always an event and there’s complaining and someone throws an exception. That’s kindergarten. By the time middle school rolls around, I’ve got my second child is headed into middle school this coming year. They will be ready to go at the appropriate time. And if they’re not actually the consequences are on them. It’s nice to not worry about it as much. But yeah, it’s still not The Jetsons.
We’re at a place where you certainly can have your release process be fully automated but it takes a number of evolutions to get to that place. Organizations always move much more slowly than you want them to, so you can’t just have the epiphany that it would be greater. It would better to have a more automated process without putting in the effort to get there. Most organizations are not willing to be held hostage in order to get to that place like we talked about before. You have to find a path to follow to get to that place. But it is kind of like growing up and getting ready in the morning. Certainly, I’m still struggling myself.
Coté:
That’s right. As you get older new problems are introduced that have nothing to do with getting shoes on. Your sort of meatware debt starts to pile up that you have to address. I’m interested in delving into the, I don’t know what to call it, the benefits of small batches. At this point for like how much I read about it’s not necessarily counter-intuitive to me, but I think it’s still kind of counter-intuitive that doing less results in more. I mean rhetorically that’s a paradoxical statement.
You highlighted a few things that explain in software development at least why a smaller batch results in doing… I shouldn’t say more but doing better. One of them was that… There’s a couple of things you said. One was we don’t do rush jobs so much. Whenever you’re doing a rush job there’s a higher chance that an error will occur because you’re cutting corners and process and things like that I guess. Also, maybe you’re just stressed out and things like that.
That kind of makes sense that if there is more predictability in the ability to get software out the door faster or get software out the door, you’re kind of like well, like I think the way you basically charted it out is if I’m an engineer or a product person and a customer wants a feature at a minimum they’re going to have to wait probably four weeks to get it, like it takes three weeks development cycle plus a week of QA and then sometime to release it.
I have this 30 day window to get something out, so that basically means I need to cram, I need to do a lot of coding to make sure it gets out, or it’s going to take two months for them to get that feature. Like there’s this compounding of the time window. That can be a stressful situation. It’s almost like if you can chunk the feature down into a small enough part, or parts of the feature, I’m overanalyzing this a little too much, but you almost can be more relaxed because you know that well we can always get to it next week. Even all of that is a little, it’s slightly counter-intuitive but we’ll just file that away as you do less rush work as a consequence of small batches.
But there’s this other part that I still haven’t conquered in my mind which I was stumbling through there, which is well we also end up doing less work like each release period in theory. There’s another sort of leap of epiphany or faith or something proven where it’s like we’re going actually going to end up doing more work because we have small batches and so you shouldn’t worry about us doing less work.
I don’t know. I’m curious on your reflections on this because to round out my statement question it seems like part of the beneficial tricks that having small batches does is that you don’t take on too much work, like you don’t have unrealistic expectations about how much work is going to be coming out in each release. Versus if you were on like a month or even worse a 12 month cycle you sort of have this gigantic like I don’t know Word doc of all these features and it’s almost impossible to know if you can actually do that much work in a year. Versus thinking in a one week interval, I don’t know, you’re a lot more realistic in the estimating that you do, which is always a third-rail topic with developers.
But I don’t know it’s this other area that I’m always foggy on that since you’ve gone through the process I’m curious how you navigate the question of like, so if we’re only doing batches in a week that means we’re going to get less done, right?
Casey:
Right, this is a conversation that I had to have very often. There is a lot to unpack here. When you’re shipping software, especially in software as a service or hosted environment you have the ability of course to modify that environment at will which is different than shrinkwrap software. That’s certainly an advantage to us. I don’t think that we always take advantage of that opportunity.
One of the issues that has to be dealt with a lot is the concept of what a release means. As a developer we know it to be a modification of the software through some means which is very abstract. I found that in particular the conversation around so we’re doing less every time we deploy, we’re deploying less value to the customer, is that okay, that often came from product management which of course they’re not day to day in the code necessarily. What they’re primarily focused on is making sure that we’re delivering things that they can get customers excited about. So every time they hear that you’re doing a deployment or release the idea is what’s in it for the customer, what are we getting that is business value that we can promote.
As you start to reduce the cycle time or reduce the time between releases it does reduce the amount of change in each release necessarily, it’s a byproduct that’s hard to avoid. What you get to is that sometimes this release does not have anything that you can champion out to customers or the broader organization and there’s a concern there from product management that you’re not delivering value then what are you doing.
The conversation that I try to have around that is to make a clear distinction between the concept of a release and a feature release. A feature release is when you open up a new ability or a fix or something like that that is customer facing, it’s obvious, people know about it, it’s something that you could put in marketing material. That should happen every now and again. Certainly we like to release functionality to customers and they like to get it in theory. So you want to do that frequently, but that should not be the focus of every release, and to change you have to change the minds of the folks who understand releases to be feature releases that sometimes a release is just a modification of the software that makes it better.
This is a very lower case a agile concept of making small changes, measuring them, and then assessing where you are and making another change and moving forward. You want to be able to reduce the overall size of those changes so that you can measure better in isolation. This brings some fairly solid software design principles into your release process. Certainly along the idea of so you’re doing less every time you release the thing that I would suggest is to change the conversation from feature releases to software modification to make things iteratively better with the occasional feature release. That’s certainly one aspect.
On the subject of keeping releases small because it reduces change and uncertainty there are two things that you always want to reduce when you’re making a modification to a system. Risk is one of them. Sorry, you want to reduce risk but you want to increase confidence. There are two ways to increase risk. One of them is time. The longer of a wait between the completion of a piece of work that could be released and its actual release, the higher degree of risk you introduce. That isn’t necessarily risking the system as much as it is risk within your team and your organization. The tighter the time from completion to delivery, the more likely it is that the programmers that were involved in that work will remember anything about it in the event that they need to make a modification.
We are often as programmers working on so many things and moving from problem to problem that it’s easy to let information leak out of our brains and often that’s ideal in order to keep up. If you increase that time, if you have three to four weeks from the time something was completed to the time it gets released and if when you release it there’s a customer found defect that needs to be addressed, there is a greater likelihood that the programmer won’t remember enough about the system to be able to quickly fix it, and it will be more costly to fix it overtime. You want to tighten that cycle in order to reduce the cost of responding to customer found defects because they will inevitably happen. That’s one way.
Then the other way that we’ve already been talking about is batch size. If the size of a change, an incremental change in production is growing, the larger it gets the higher degree of risk that you have. You can imagine plotting this out. You can see a rise in batch size overtime is also an increase in risk. Then when you release there’s a cliff where your risk goes back down to zero because of course if you were to … then you don’t have any modifications to make in production so you don’t have any risks of modifications breaking things. Then you start again. It rises until you release and then it falls off a cliff.
You want that rise to be as small as possible. The way that you do that is to reduce the amount of change in each release and also the amount of time between releases. If you can keep all of that low then your rise is small and your rise in risk is small and your confidence level in each release can go up. We like to talk as software developers about unit testing and isolated testing being a pretty decent way of testing individual components. I would say the same for releases that we want to make individual changes, put them in production and test them in isolation.
Coté:
To brutally boil it down you want to test if it works. Not only if it functionally works but if the thing you’re releasing is what the customer actually wants. It could technically work. It could just be they don’t want it. The longer you wait to verify is this what the customer wants, which as a requirement it must work, you’re basically, there’s another type of debt you’re building up which is sort of like unknown debt. You don’t have feedback about it if you’re doing the right thing or not.
I mean I remember one summary of, I don’t know, I think it’s in Lean Enterprise somewhere. There’s a summary of some how to measure anything book which is a delightful little tale about ROI and it’s sort of like well the ultimate killer of ROI is that you built the wrong thing. It’s good to test that out as frequently as possible.
I think I’ve broken-in to highlight this because I think it’s important to highlight that, and tell me if you think there’s other things or if you would modify this, but it seems like as you’re describing those peaks of risk that you’re building up overtime one of the main causes of that risk is just the unknown if you’re doing the right thing essentially. You haven’t tested it out to see if it’s actually going to work in the sense of it’s what someone wants and it’s actually going to work as well. Of course the peak pulls off because once you release you get all this input about if it works or not. Hopefully the peak goes all the way down so to speak, but the more frequently you can verify if you’re doing the right the less failure you might encounter in the future.
Casey:
That’s a good way to boil it down. Yeah, absolutely. The ultimate goal here is to increase the effectiveness of your development process. Releasing your software gives you a huge amount of incoming information to understand whether or not you are effective. Sometimes folks are afraid of that so they hold off, and by waiting you compile more and more change into a release, and then you get muddy information, you don’t have clear associations necessarily between the inputs that you’re getting about whether a change was effective from your customers and being able to map that to specific development initiatives.
Coté:
That’s a good point, you can’t really… If you have 50 changes in a release it’s hard to figure out the causation of failure across those 50 things versus if you have like three or one essentially.
Casey:
That’s exactly right. Again, if you’ve disassociated the concept of releasing from feature releases to understand that those are different concepts then you should be able to make many small changes and it is a series of small experiments. Ideally you’re getting your input as fast as possible in order to inform your next actions.
Coté:
Before we wrap up, here, it strikes me that after talking through all of this stuff one way of framing a lot of all this soft process stuff, all this talk about sometimes, which means a slightly different thing but there’s exciting Venn diagram overlap of all those buzzwords. But is that when you’re doing software the process is as important as the product, so a lot of people will spend much of their time focused on the product and the features. Like you were saying, what features are we releasing that are valuable to customers? And if there’s not a tangible feature that’s valuable, if there’s not a product gained, then nothing worthwhile has happened.
In fact, to put it another way, part of the product is the process that creator did. In the same way that you minutely manage the product and think about what goes in there and spend a lot of time on it, it’s also, it’s probably equally. It’s equally important to focus on the process as an extension of the product and the thing that creates it. If you have a crappy process you’ll probably end up with a crappy product. There’s a lot of things that you might do that have no customer facing… Not value. It’s not a direct connection or something. It’s not visible. That’s the word I’m looking for.
Casey:
That’s it, it’s not visible.
Coté:
As you perfect the process it will ensure that you have a better product essentially. It seems like intuitively as a developer with the developer mindset that kind of makes sense, but getting that idea across to people is probably a challenge that’s worth going up against, that we’re engineering a process here that results in good products essentially.
Casey:
That’s right, it is a challenge. There’s a concept of Conway’s Law which it describes exactly what you just suggested which is the organization that created software will be mimicked in the design of that software. So if your organization is not responding to customer feedback, is not agile, lower case a, iterating quickly, measuring and making course corrections all along the way, then your software will reflect that.
You do need a culture and a series of processes that can support what you’re trying to accomplish in your technical architecture, your software design. I would agree strongly that it is worth taking on those challenges in order to build better software. It is difficult to build better software if you’re not building with better processes and better tools and a better culture for it, so you do have to take great care.
The bottom line I would say is all businesses want to increase their margins. You can either bring in more money or reduce the cost of doing business. A lot of the things that we described today, especially over the long haul while they are an initial investment they lower the cost of doing business. That is a benefit for everyone because cost is not just dollars but it’s also time and energy. If you can reduce the overall cost of doing business then you can spend that money and that time investing in your product, investing in your people. I think that makes for a healthy environment and certainly better software.
Coté:
Well that makes sense. Well this has been exciting. It’s a nice conversation. We’ll have to come back again and as we were flagging some point talk about like technical debt and as they say deep dive into some other topics. But yeah, I think it was pretty informative and entertaining. You’re out there on the Internet. If people wanted to check out your stuff where should they go?
Casey:
Sure. I write from time-to-time at caseywest.com. I enjoy the Twitter so you can also catch me @caseywest on Twitter.
Coté:
Well great. As always this has been Pivotal Conversations. If you want to check out the show notes or subscribe to the podcast you can just go to pivotal.io/podcast or search around for it. I’m sure if you search for pivotal and conversations and Cloud Foundry and all those great things you can find it and listen to it. It’s always good to get feedback about what you the listeners think and are interested in. You can email us at podcast@pivotal.io and with that we’ll see everyone next time.
About the Author
