Why Lean Startup Experiments are Hard to Design

Why Lean Startup Experiments are Hard to Design.

by Grace Ng

Four years ago, 50 bright-eyed wanna-be entrepreneurs sat in a room overlooking a deserted cobblestone sidewalk in New York’s Financial District. It was a Saturday afternoon and their minds were brimming with crazy ideas for new business ventures. They came to this business workshop expecting to build out their ideas. As they entertained all the possibilities, the workshop mentors started shouting, “Get out of the building!”

We wanted to force these wanna-be innovators to get out onto the street, interview strangers and validate demand. Did their big ideas solve for real customer problems? This was the first Lean Startup Machine workshop, and our goal was to teach entrepreneurs Lean Startup thinking in just three days. Eric Ries told us this wasn’t possible.

Validate Need

Today, we’ve taught the process to over 50,000 entrepreneurs around the world, entrepreneurs who have gone on to launch products and services that thrive on rapid cycles of experimentation. They’ve gone to implement Lean Startup thinking at their jobs and launch new products that thrive on rapid cycles of experimentation.

For my co-founders and I at Javelin, our question has been the same from the beginning. How do we turn people’s deeply-ingrained execution mindsets to ones that constantly question assumptions and explore possibilities in environments of rapid change and uncertainty? What’s the most effective way to do this? It’s a complete paradigm shift, but one we think is necessary for building 21st century businesses and solving 21stcentury problems.

The tool that enables us to teach Lean Startup at scale is theExperiment Board. We’ve developed it through rounds of learning from the most successful entrepreneurs we’ve seen come out of Lean Startup Machine.

Effective Experiments are Hard to Design

Every experiment starts with a hypothesis. You want to test the validity of your assumptions. But few of us know how to formulate truly test-able hypotheses! Very few of us understand how to identify and test our really risky ideas and assumptions.

The Experiment Board is a tool we use to help others easily turn business, product, or feature ideas into experiments. We created it to eliminate (ok, minimize) the most common mistakes teams run into when conducting experiments for product and service validation.What the Experiment Board does is lay out the components of an effective experiment and help teams communicate and collaborate with their team members and stakeholders. Teams who use it gain alignment and make faster decisions as a result. Download it here.

Common Challenges in Product Validation Experiments

Getting Buy-In

“Why do we have to validate this? We know people want this feature. Let’s just build it!”

Sound familiar? If your workplace culture discourages experimentation, the mere suggestion that your company validate the need for new products or features will prompt looks of horror from individuals in leadership positions. For people who are used to operating on gut instincts, the need to fact check their intuition is a barrier to getting buy-in. Especially as someone who’s been on the job for years, or even promoted for their intuition, it’s uncomfortable to have years of experience suddenly challenged. The know-it-all trap is easy to fall into, and it takes self-awareness to acknowledge that times have changed. What worked yesterday, does not work today.

One way to get buy-in is to provide your team members with what they need, but go ahead and still run the experiment on the side, showing them the results later. Involve one or two team members in designing your validation experiment so you have as much alignment as possible on what you’re actually testing, but run the test!

Structuring Effective Hypotheses

Once bought-in, many teams find themselves stumped as to how to actually structure their experiments.

To design an effective experiment, start with a hypothesis, a prediction for what you think will happen. (This can be phrased as a customer-problem hypothesis or a problem-solution hypothesis). The customer-problem hypothesis tests if your intended customer, or user, experiences the problem you think they have. The problem-solution tests if your proposed solution addresses that problem with a quantifiable outcome. Ideally, start with the customer-problem hypothesis, so you’re not assuming anything by jumping into the solution too soon.

Start by asking two questions: What problem does this solution solve and who has this problem? Identify who the customer or user of your proposed solution is. Next, brainstorm potential problems your customer could have that calls for your proposed solution. Make sure you timebox each brainstorm session and have each team member individually write on sticky notes to contribute. (This ensures that everyone’s opinions are heard and enables faster decision-making). Once you’ve selected your customer and problem to test, you’ve formed your hypothesis.

To define a problem-solution hypothesis, simply state that your solution will solve the problem for that specific customer. You’ll need to quantify the outcome you expect to see that will prove the problem is solved. (We’ll go over how to set a quantified outcome in a bit).

The right side of the Experiment Board tracks the progress of your experiments, so you can stick your hypothesis into the respective boxes to indicate the start of the experiment and keep everyone aligned and what you’re testing. This exercise is a collaborative and simple way to get your team engaged in running experiments.

Identifying the Biggest Risks in the Idea

After forming your hypothesis, it’s time to identify your riskiest assumption. This is where most teams get tripped up. Assumptions are beliefs that are core to the viability of your hypothesis. Think of assumptions as the behavior, mentality, or action, that needs to be true in order for the hypothesis to hold true. Your riskiest assumption is the one that is both core to the product or service’s viability andmost unknown, meaning you have little data to prove it’s valid. When you’re testing a customer-problem hypothesis, your riskiest assumption is one that supports the belief that your customer has that problem. In a problem-solution hypothesis, the riskiest assumption is if it’s the right solution to solve the problem. It’s important to always test your riskiest assumption first.

On the Experiment Board, brainstorm all the possible assumptions you’ll want to test, then identify the riskiest assumption to test. Involve everyone on your team (and a few people who are totally unfamiliar with your project) in brainstorming assumptions. Why include “strangers” in your project? These people are less prone to what we call confirmation bias. If you’re intimately involved with the project, there’s no way around this.

Once you have a few assumptions, ask your team members to dot vote on post-its to select the assumption they think is the riskiest to test. Your experiment will test if the riskiest assumption is valid or invalid. If it’s valid, then your hypothesis holds true. If it’s invalid, you’ll need to change your strategy by changing one or both elements in your hypothesis. This is called a pivot. The goal is to get closer and closer to something that holds true. The Experiment Board helps you track pivots, which communicates your validated learning and progress to stakeholders over time.

Staying Accountable to the Results

Defining what success should look like is the most crucial step before conducting an experiment. Many teams want to just build and see what happens. The problem is, without setting a success criterion, all the positive results you see will look like validation. After spending six months on a project, what does one paying customer really tell you? Is it really worth your time to allocate more resources and keep building? Or is there not enough interest to spend more money on it? The results you see after running an experiment are subjective. You might think you have enough validation to put more budget behind it, and another person will disagree.

The minimum success criterion is the weakest outcome you expect to see in order to take on more risk and continue. Without defining success up front, teams will argue over whether an experiment was successful or not; it is impossible to interpret the results of the experiment without succumbing to some form of bias. Make your experiments measurable!

Running Quick, Iterative experiments

A common misconception is that the only way to test an idea is by building it so customers have something to interact with. But there are many, surprisingly counterintuitive ways to test customer demand without building anything. The most insightful, low-cost ways include conducting interviews, setting up a landing page test, or delivering the service manually (aka concierge). The technique you choose depends on the assumption you’re testing, the time you have, and how much certainty you have about the direction you’re going in. These quick, low-cost ways to test your assumption will reduce the time it takes to get customer feedback.

How We Validated the Need for the Experiment Board

Validate Need

Our hypothesis in designing the Experiment Board was that by increasing the speed at which teams can set up and conduct experiments, the sooner they would discover viable ideas for products and services people will use and buy.

Before we created the Experiment Board, we found that teams too often spun their wheels, argued indecisively, and wasted time struggling to understand what and how to test. When we tested the Experiment Board tool in our workshops, teams began running their first experiments within 10 minutes. Simple things like using stickies and timeboxing enabled communication, alignment, and faster decision-making. The flow of the board design gave teams a tangible sense that they were making progress.

Many teams often discover that their customers didn’t experience the problem badly enough to use or pay for a solution. They often abandon their ideas completely after three pivots in favor of an actual problem they find during their fourth round of experimentation. The most successful teams often pivoted multiple times until they found a key insight leading them to a bigger opportunity. This all takes place within 24 hours of designing and running experiments with actual people! Crazy, right? But it works.

Today, our tool enables us to scale Lean Startup education to 400 cities around the world so any entrepreneur or corporate team, however new they are to the scientific method, can start running experiments. I hope you can use the tool in your workplace to build products people want, in less time.

For more about the nuances in designing experiments watch my video tutorial hereIn my next post, I’ll talk about the unique challenges intrapreneurs, innovation teams, and change agents within large organizations face when running experiments and using Lean Startup techniques, and how to structure an environment that enables rapid experimentation and pivoting.

JOIN LEAN STARTUP MACHINE MANILA – July 4-6

Advertisements

10 things I’ve learnt about lean startup

We Love Lean – Lean startups, design, happiness and everything in between

This is a transcript of my recent talk at Lean startup Brighton.
In this article I’ll be highlighting the things I’ve learnt from applying lean startup techniques over the last couple of years and how that pans out in reality. In particular the challenges we face integrating lean within an agency environment, with a particular focus on the early stages of a lean startup – customer development.
First, a bit of background…

For those of you that don’t know I’m co-founder of Spook Studio. We’re a startup consultancy and design studio by the sea that helps startups get started. We’ve worked with many startups over the years and have learnt a bit along the way.

1-spook

We love working with startups as we get to see an idea come to life, plus also we get to work with visionary people that are looking to disrupt the status quo and do…

View original post 2,333 more words

Lean StartUp Machine Manila 2014

We need sign-ups to be able to confirm the DATES for Lean StartUp Machine here in MANILA. Sirfaraz Piracha, the Growth Manager, Europe And Asia Lean Startup Machine gave an update recently that we need to get people to re-assert their participation to  Lean Start Up Machine to Manila this 2014 so that we can peg the dates down. Check out more about Lean StartUp Machine below:

SIGN UP and help us get the dates down!

Lean StartUp Machine is a workshop that “kicks off with a series of product pitches that help attendees organize into teams. Everyone works on a NEW idea. Each team then develops its problem hypothesis, solution hypothesis and a series of assumptions which are core to the success of the business.”

“Next, each team creates an MVP, or “Minimum Viable Product.” The goal of the MVP test is to “Get Out Of the Building,” speak with real customers and to collect cash or non-cash currency which serves as validation.”

“The process promotes a problem-centric approach to business ideation. By continuously validating and invalidating assumptions, teams are forced to pivot their solution (and even problem) toward a true customer pain; rather than creating a solution and after-the-fact seeking customers with a problem.”

“The event culminates with each team pitching its new solutions and its experience using the process. In this competition the winning team is not selected based on who has the best idea, but instead on which team honors the process and gains the most insights through its pivots.”

Check out the Lean StartUp Machine experience:

Brian Watson – Analyst, Union Square Ventures

Melissa Perri – Project Manager, OpenSky

Source: Lean StartUp Machine

SIGN UP and help us get the dates down!

Lean Start Up Machine in Manila 2014

Sirfaraz Piracha, the Growth Manager, Europe And Asia Lean Startup Machine recently got in touch to encourage us here in Manila to get a move on it and secure those last mile sign-ups required to bring Lean Start Up Machine to Manila this 2014. Check out more about Lean StartUp Machine here.

UNLOCK MANILA NOW!

Lean StartUp Machine is a workshop that “kicks off with a series of product pitches that help attendees organize into teams. Everyone works on a NEW idea. Each team then develops its problem hypothesis, solution hypothesis and a series of assumptions which are core to the success of the business.”

“Next, each team creates an MVP, or “Minimum Viable Product.” The goal of the MVP test is to “Get Out Of the Building,” speak with real customers and to collect cash or non-cash currency which serves as validation.”

“The process promotes a problem-centric approach to business ideation. By continuously validating and invalidating assumptions, teams are forced to pivot their solution (and even problem) toward a true customer pain; rather than creating a solution and after-the-fact seeking customers with a problem.”

“The event culminates with each team pitching its new solutions and its experience using the process. In this competition the winning team is not selected based on who has the best idea, but instead on which team honors the process and gains the most insights through its pivots.”

Check out the Lean StartUp Machine experience:

Brian Watson – Analyst, Union Square Ventures

Melissa Perri – Project Manager, OpenSky

Source: Lean StartUp Machine

UNLOCK MANILA NOW!

POSTPONED! Sign up for the Lean Startup Workshops on Nov 13 and 14

NOTICE TO THE PUBLIC: On behalf of LSManila community and our partners, we sincerely hope that your family and friends in the Visayas region are safe and sound.  In light of the recent calamity, we feel its most appropriate that all resources be directed to ongoing disaster assistance endeavors.
As such the following events for the week have been cancelled:

  • LearningOpp: Lean StartUp in Action (Introduction), scheduled to occur on Wednesday, November 13, 2013 2:00 PM
  • LearningOpp: Pitching Your Idea, scheduled to occur on Wednesday, November 13, 2013 7:00 PM has been cancelled.
  • LearningOpp: Developing Customer Personas, scheduled to occur on Thursday, November 14, 2013 7:00 PM has been cancelled.

These events will be rescheduled if not before the year ends then early January. LSManila will be having a meet-up before the month ends (November) so stay tuned for the details of that or better yet, join the MeetUp Group here to be notified.

I highly encourage you to help out in any way you can – through cash and in-kind donations and by being generous with your time and energy in assisting relief operations.A list of relief operation centers can be found on Rappler. 
If you have further questions, you can email me.  – Guita T. Gopalan, LSManila MeetUp Organizer

November 13, 2013 (Wed)

Introduction to Lean Startup: http://leanstartupmanilasocent1.eventbrite.com/

lsiamanila4
lsiamanila3
November 13, 2013 7pm
Pitching
November 14, 2013 7pm
Customer Personas