Skip links

An introduction to experiment design: a two week sprint, an evidence-based decision to embrace or kill an idea

Imagine this. You order a new pair of shoes, but you aren’t at home when the courier delivers your package. The neighbors aren’t either (or the courier was in a hurry, and didn’t even rang their bell), so you need to pick up your package at the post office. Tough luck, because it’s already closed for the day. Tomorrow? Bummer, the post office is closed on Saturday. So you’ll need to go to that wedding party wearing your old, slightly worn off pair of shoes … and feel a little bit ashamed all night.

All too familiar for you? The ‘smart door’ might be a solution. You get a message on your smartphone when the courier rings the bell, you open the door from a distance, the couriers drops off the package and closes the door again. The delivery is being filmed, so no worries about safety or privacy. However, I can imagine it’s still a pretty big step for a lot of people to open the door for a complete stranger without even being around themselves.

We don’t do sentiments, we do numbers

Only way to figure it out: testing it. Find out exactly what part of the online shoppers would be prepared to install such a smart front door bell/lock. And if they are open to the idea, would they be tempted to purchase more products online when they can be sure they won’t end up with the neighbors or at the post office? To map and measure that willingness, we created a dedicated website and a survey for Bpost and Zalando.

That is what experiment design is all about. Running (small) tests allows companies to make evidence-based decisions. Experiment design validates assumptions. It’s a way to translate gut feeling into hard boiled facts and figures.

Don’t companies already test new products and services? Yes, they do. But they test them in a qualitative way, that’s open for interpretation. They set up surveys with open questions, they organize focus groups, they bring together user panels. Experiment design aims for quantitative results. Think traffic, think clicks, think downloads, think inscription. You don’t get sentiments, you get numbers.

Wanted: the best failed idea

Experiment design is an interesting way to discover things quickly. We run sprints, not marathons. Too many companies still make decisions based on last year marketing plans. But technology evolves with the light of speed, and consumer behavior is unpredictable. Ideally, the experiment runs no longer than two weeks. After the experiment is done, you evaluate the result, and you either embrace and implement the idea, or you relentlessly kill it. No regrets, the numbers have shown it won’t work.

The Indian giant Tata has a prize for the best failed idea. Each year, the company rewards an idea that terribly failed. Not to make fun of the employees behind the idea, but to convince everyone working at Tata from the importance of trying out new things. Even if they fail miserably.

What experiment design essentially does, is radically speeding up that process of trying. And failing. Or succeeding. Experiment design done right can save you a tremendous amount of time and money. By implementing great ideas. Or by NOT implementing bad ideas.

Canary in the mine

Experiment design has an impact on your organization that goes beyond just the experiment. It makes you question whether you have the innovative power to compete, it exposes the roadblocks that stand in the way of innovation, and it forces you to switch gears. It’s like the canary in the mine. had dozens, even hundreds of different versions of its website running, focused on all kind of different markets and target groups. When it takes you weeks to get the approval of your legal department for setting up even one landing page, you know you have a problem you need to solve. As soon as possible! 

The golden formula: if-then-because

In order to get the right answers, you need to ask the right questions. If you wish to validate certain assumptions, you need to make them measurable. You do that by translating them into hypotheses. An assumption is a claim without proof, a hypothesis is testable. If the assumption is nobody clicks on the ‘call to action’ buttons on your website, the hypothesis is that the number of clicks will increase by 10% if it doesn’t say something generic like “learn more”, but something really practical like “request a demo”.

A hypothesis always follows the same pattern: if-then-because. If I change the call to action, then more people will click, because it’s clear for them what they will get in return. To help you translate vague assumptions into measurable hypotheses, we wrote a blog post and developed the experiment card.

What are you waiting for, corporates?

There are a few tough misconceptions about experiment design.

No, it’s not just for startups. On the contrary, especially big companies benefit from experiment design. Startups are agile and flexible by definition, which gives them a huge competitive edge over corporates. For these corporates, experiment design is a way to bypass slow and sluggish internal procedures.

And no, experiment design is not the playground of the innovation team. Experiment design is suited for every team and every department, that’s the beauty of it.

Let’s say that an abnormally high percentage of applicants drop out between round one and round two of your application procedure. The HR department can easily run one or more experiments to find ways to raise the level of candidates pushing through. Will the tone of voice of your e-mails make a difference? Maybe candidates will stay on board when you give them a call instead of dropping them an unpersonal e-mail? Or when you change the time of their meeting? Only one way to find out.

Start experimenting! Need help with your experiment design? Reach out and let’s see how we can lend a hand.