Progressive software creators have realised that they do not always get their designs right the first time around. And that there may be multiple “possible” ways to do something, but one “better” way in terms of outcomes.
When it comes to user interaction with applications and websites, there is a constant flow of learning, experimentation and change that should be fundamental to the processes of any modern digital organisation.
The process of A/B testing enables multiple version of an application to be tested, with real users, at the same time and in real-time. The results of different interactions can then be compared, and the application tuned.
Examples of this abound in the areas of marketing, online game in-app purchases and the like. If you are not measuring how well customers are interacting with you—on what are you basing your decisions?
Pivotal Cloud Foundry makes it super-easy to deploy, maintain and update the A and B versions of your application and make changes in real-time. In this week’s episode Simon dives into how this is done, and the commands to use.
PLAY EPISODE
RESOURCES:
- Subscribe to this feed
- Find more episodes of All Things Pivotal podcast
- Feedback: podcast@pivotal.io
- Links Referred to in the Show:
Transcript
Speaker 1:
Welcome to the All Thing Pivotal Podcast podcast. The podcast at the intersection of agile, cloud, and big data. Stay tuned for regular updates, technical deep dives, architecture discussions and interviews. Please share your feedback with us by emailing podcast@pivotal.io.
Simon Elisha:
Hello everyone and welcome back to the All Things Pivotal podcast, fantastic to have you back. My name is Simon Elisha, CTO and director of field engineering here in Melbourne, Australia. In this weeks episode, we’re going to be talking a bit about A/B testing, and this is from a listener suggestion and thank you for giving us these suggestions. We really do like to get the feedback from the listening audience. This one is from Josh, and Josh wanted us to talk a little bit more about A/B testing and he was thinking about the previous episode we had a little while back, we were talking about blue-green deployments, which is where you can swap out a version of an application for a newer version without affecting uptime or user etc., which is a neat capability of Pivotal Cloud Foundry.
He was wondering how this could apply to the world of A/B testing, and I thought excellent topic—let’s cover it. So let’s think firstly about what A/B testing is, and then we’ll talk about how we can achieve it. A/B testing is a concept of having two versions of the same thing being exposed to your customer base at the same time and comparing the results of each of them. So why would we use this?
Well, we may use this in a marketing campaign, so a very common example is, we send fifty percent of our recipients of email formatted a particular way with a particular offer and the other fifty percent get a different email, formatted a different way with a different offer, and we see which one is responded to more readily. From an IT perspective or an application perspective a very common pattern is to locate buttons, and options, and widgets in certain places on the page in different version, and to compare which is more usable or which one has a better turn over, etc.
Another really common one that I see a lot in is online gaming, particularly as online games where in-game purchases are made, is the presentation of different versions of packages or different combinations of things you can buy, to see which one is purchased more frequently, etc. Many organizations have tried to absorb and adopt A/B testing into their life cycle because it gives them real feedback, and as we often talk about there’s nothing better than that feedback loop from your user base into how they actually use your application, your website, your system, etc. So to make this effective your application must log activity—it must be tracking what’s going on in the application. Which button was pressed, or which path was taken, or which option was chosen. The other challenge when doing A/B testing is you need to be able to host version A and version B simultaneously and at the same time. There are a few challenges that come into play with this.
Often platforms or infrastructures are made for one copy, and only one copy, of the application running at one time. That would not be very helpful, also you need to have a data model that can cope with this, so it’s important to consider that all fields are supported in both versions of the application. There are a whole lot of techniques around that, we can talk about another time, but just be aware to make sure that happens. This is another reason for having primarily stateless type applications at either point. The other trick is you need enough capacity to run both versions of the applications at the same time. This is where being on an elastic platform leveraging infrastructure as a service comes in to play. The other thing you need to do is be able to transparently deploy these versions of your application, version A and Version B, into production at the same time. Rerouting traffic to both of them transparently you also want the ability to take one or the other out of service depending on what’s going on or maybe replacing one of them. Just because we have A and B doesn’t mean we can’t have C, D, E and F, but A and B are typically enough for most people.
How do we achieve these in Pivotal Cloud Foundry? Well its deceptively easy and very, very, powerful. The first thing we do is we create our route, so this is the location or the domain which the particular application will run, and we just do the CF create route command with the various parameters that specify the domain and the hostname, etc., that we want to use. At the moment that route, or route, is not mapped to any particular location, then what we do is, we push the A and B version of our applications as separate push commands, so the CF push command, but we use the no route option, and that specifies that when we push the application it’s not bound to any particular end point at that time. It’s available, it’s there, but it’s not visible on the network. Once both version A and B of the application are up and running, we then do a CF map route command, to map the running version of version A into our existing route that we created at the start, and we also do the same command for version B against that same route.
What will then happen, automatically without you having to do anything, is that Pivotal Cloud Foundry will be mapping transactions to both of those versions for you automatically and it’ll switch between the two of them. If you had a very simple application, you just were hitting refresh on your browser you’d be reloading, you’d be seeing tracking going between the two.
So, this makes it very easy to load balance and separate between the two versions of the applications. Now, obviously, you need to think about things like state and other things to make sure that works, but for most cases and if you engineer your application correctly, you can make significant changes in the versions without affecting the backend stuff, without affecting the user experience it’s the nuance that you’re trying to track. So again, the command sets we were talking about there, CF create route, CF push, CF map route. Three simple steps to a lot of answers.
Now what do you do with those answers, so again, it is very important to capture the behavioural data of the user and send it back to some location. Now you may send it back to a consolidated location or just be tailing to some log files. You can pull those log files off digital Cloud Foundry very easily and send it to any one of a number of third party products or you can be sending it into a Hadoop distribution , you could be sending it to a data lake and running analytics on there. You need to be creating repositories of all of that data analyzing it on an ongoing basis and making real time decisions.
The ultimate situation to get into is where by you create a dashboard that’s interpreting that information so it’s interpreting clicks per minute or purchases per hour. Some meaningful metric and that’s updating real time, maybe even in a screening or development environment, and you’re looking at it and the product management is looking at it and saying, hey this is the behavior we’re seeing over time, now based on that let’s make a change. I apply that change in code, run it through the test procedure, and then again CF push into the environment, CF map route and remap that application into the mix, and start using that new capability straight away.
Again, really simple to talk about and simple to explain, but super powerful in what this can do in your own environment and for your own business. A/B testing is one of the secret weapons of the internet giants, in terms of understanding their customers, providing a better customer experience, and honing how that works, and it has tremendous commercial value. In terms of driving increased revenue, better customer satisfaction, and better experiences overall.
I’ve hoped you enjoyed that and that’s been useful. A few links in the show notes to some of the commands and some of the little areas you need to worry about, but very straight forward but very powerful.
Thanks again Josh for your suggestion, again if their are other suggestions that people have, I would love to do podcast on topics that you enjoy. You can reach us at podcast@pivotal.io at anytime, and until then.
Thanks for listening to the all things pivotal podcast if you enjoyed it please share it with others we love hearing your feedback so please send any comments or suggestions to podcast@pivotal.io.
About the Author
