Why executives need to embrace experimentation

June 15, 2014 Addison Huddy

Funding New Products: Why Executives Need To Embrace Experimentation

There is no question that the game has changed, and it is going to keep changing faster and faster. Making software (and recently hardware too) is easier and faster than ever before. A motivated team, using modern software methodologies and a network of services, has the power and ability to compete with any large enterprise.

Today, the only competitive advantage is the ability to learn faster. More specifically, learning what your users want, learning how to build the product, and learning how to go-to-market as quickly as possible.

So what is holding large enterprises back? A willingness to experiment.

The traditional approach to funding new product development in a large enterprise:

An ambitious middle level executive & team have an idea for a new product. They look around the ecosystem, talk to their sales and other cross-functional teams, and determine that the next blockbuster product will be ABC.

The idea for ABC begins to circulate. Hallway conversations, lunches, and formal meetings solidify the idea to the point that the team says, “we need a budget to get this product into market!”

In most large companies, budgeting for new product development happens during an annual or quarterly funding cycle. When the cyclical budgeting process starts, the ambitious, middle-level executives present their best cases to C-level executives to increase budget for a new product.

To prepare for the budgeting process, the product team talks to cross-functional stakeholders and, more often than not, via PowerPoint and excel, begin to draw out a 2-5 year implementation plan based on estimates from marketing, engineering, sales, etc. Growth models are generated to make hurdle rates and thousands (yes thousands) of assumptions are made about what the product is and how it is going to work.

The months of work that go into generating ABC’s business case are slotted into one of the many budgeting meetings that the C-level executives attend. The mid-level executives are often asked questions that are impossible to answer without taking a leap of faith and putting their job on the line. * How did you get your 5-year growth rate for ABC? * Are people actually going to like ABC? * How did you determine your milestone dates? * How much money is it going to take to launch ABC off the ground? * What is the process for making ABC? * Are our customers going to pay $X for this product?

Despite all these questions, what happens in a successful budgeting meeting? The mid-level executive’s meticulously prepared PowerPoint, supporting excel models, logical go-to-market strategy, and beautiful leave-behind impress the C-suite. As a result, the product gets its $millions of funding. However, its already doomed: loaded with assumptions, unmanageable uncertainty, and unrealistic or vanity metrics.

The months and quarters go by, check-in PowerPoints are given to key stakeholders, and product development and sales are well underway. At this point, you name the roadblocks (technical resources, legal issues, marketing plan schedules, sales channel coordination is slowing), are jeopardizing the KPIs set by management. Let’s not focus on all these roadblocks, let’s focus on the biggest roadblock of all: learning that potential customers actually want something different.

Too often, a team refuses to pivot after learning something about their users. Why?

  1. “Too much money has been spent now. We can’t go ask for more.”
  2. “Don’t rock the boat.”
  3. “Our KPIs have nothing to do with the customers actually using the product. Let’s just get it out the door.”

If the product manages to get out the door, everybody celebrates. Then no one uses it, and well, let’s not talk about what happens next.

The leaner approach – It is not about how fast you can build, it is how fast you can build the right thing.

The game has changed: not only do you need to be building the right thing, but also you need to build it fast. How do you do this? Experimentation!

Senior executives have an opportunity to make their organization more agile by approaching the funding process differently. Rather than funding new product initiatives through a cyclical budgeting process, senior executives should approach funding a new initiative based on what the company is going to learn with early investment.

Instead of funding a product idea, fund an experiment that will help validate the many assumptions that traditional product funding methods make.

Rather than a middle manager being asked to project out a complete product timeline, roadmap, and funding cycle with projected revenues and costs, senior managers can ask these middle managers to present them with a product hypothesis and a series of well-designed product experiments to try and validate, or just as good, disprove the product hypothesis. Senior executives should be open to funding lots of this little product experiments that will lead everyone down the right product direction.

These experiments cost a fraction of what a failed product will cost, and compiled together are a powerful tool to use to make the big product bets.

What does a good product experiment look like?

Hypothetical Example:

Hypothesis: Our customers have a willingness to pay for ABC that will allow them to XYZ.

Experiment: Using a tool like InVision, build a prototype of ABC. Schedule interviews with potential customers. Put an Amazon gift card worth $40 on the table. Start by saying, “thanks for coming in, here is your Amazon gift card.” Have the potential customer take possession of the gift card.

Next, start talking about the product and where it is headed, and that today “you are going to a sneak preview of ABC.” Using the InVision prototype, have the customer walk through ABC. Ask him or her to complete workflows and talk about what they are seeing. At the end, present the potential customer with a signup page. Ask them if they would like to sign up for an early trial. See what they do. Do they start signing up?

If so, present them with a payment page. Remember that Amazon gift card? Say, “If you want to be one of the first 50 users of ABC, then I am going to need that Amazon gift card back”. Record his or her response. Ask them why they did or did not pay for ABC.

Work to synthesize the information into groups and calculate how many users completed the entire flow from signup through payment using the gift card.

Result: What would all this synthesized user testing look like?

  • 25 users tested
  • 15 completed the sign up process (60% conversion)
  • 5 exchanged the Amazon gift card for early access to the product (20% conversion)
  • 8 out of the 10 users who signed up, but refused to pay, said they didn’t because the price was too high.

Next steps: benchmark the results against other tests, lower the price of the product in the next test, and iterate.

After this experiment, the findings can be summarized and presented to senior management. Even though product ABC didn’t make a huge splash, it did perform better than earlier product ideas and iterations. On the next iteration, it might perform even better, hopefully well enough to make a case for a bigger bet.

Experimentation is a path to enable larger organizations to operate leaner. Making grand, poorly validated product bets with traditional business planning methodologies ends, more often then not, in disaster. Senior level executives have the opportunity to embrace experimentation, enable their teams to learn more quickly, and make the big product bets with less uncertainty.

About the Author


Our Agile Team Grows Stronger
Our Agile Team Grows Stronger

The Luxr Team Joins Pivotal Labs Pivotal Labs has a mission to change the way the world builds software, an...

Field Report: The 3 Reasons Why the #CFSummit Rocked
Field Report: The 3 Reasons Why the #CFSummit Rocked

The first Cloud Foundry Summit fueled by the support of the Cloud Foundry Foundation wrapped Wednesday—and ...


Subscribe to our Newsletter

Thank you!
Error - something went wrong!