10 Pitfalls of Corporate Experimentation and How to Solve Them

banner with skyscrapers in the background and yellow title "10 pitfalls of corporate experimentation"

Experimentation is engraved in our daily lives.

Whether it is something simple, such as trying out a new shampoo or going for a new hobby or job, experimentation triggers a freeing and daring sensation that can go both ways. Whatever the outcome, we adjust and move forwards towards or away from it.

Other than bigger risks and obstacles, experimentation within a business context isn’t too different from that.

Managers invest money, time and people into experimentation to pick holes in their assumptions, change their practices and get ahead of competitors.

Experimentation is a long-term, richly informative process where each trial defines the starting point for the next.

Taking the time to develop a comprehensive testing environment is one of the best ways to create a culture that encourages risk-taking and a company constantly booming with fresh ideas.

When you encourage your workforce to brainstorm new ideas and new ways of doing things, you’ll find that staying competitive and flexible is much easier in the constantly shifting business landscape.

We thought we could give you a hand and expose the obstacles of corporate experimentation and how to tackle these.

‘If you cannot fail, you cannot learn.”

Eric Ries, author of ‘The Lean Startup’

1. Working Alone

Working alone might be great for lone brilliant thinkers, but in an experimental context, working alone means having one single perspective on a problem and limiting the outcome and the brainstorming process to one single mind.

Instead: Build an experimentation team

Gather as many backgrounds as possible around one table.

Bringing people together from different genders, socioeconomic backgrounds and principles will bring versatility to the experimental plan. Diversity of thought fuels creativity and innovation.

Empower junior employees to plan and execute their own experiments. You’ll be getting fresh ideas while motivating new employees to be entrepreneurial.


In several studies, inherent diversity (gender and race) and acquired diversity (experience cultural background) lead to successful business results.
For example, a
2009 analysis of 506 companies found that companies with more racial or gender diversity had more sales revenue, more customers, and greater profits.

Workers pointing at papers


2.Complicating the process

Complicating the process of experimentation exposes your business to unnecessary risks and may end up compromising results. It is important to understand that simple experiment is. Any action that is taken should have a low risk and budget impact. It means it can be self-contained in case something goes wrong.

Risking a complex experiment means risking high failure probability in handling the issue.

For example, if you run a pricing test involving thousands of items across different stores, you will have a huge labor cost, and an intricate operation harder to coordinate. If you run the same repricing test but including fewer products, fewer stores and therefore less labor action you will come up with similar results and an easier to manage experiment.

Instead: Keep It Simple

Have you ever had that feeling of ‘That is so simple I could have come up with it myself?’. Feel sorry all you want, somebody beat you to it.

Keep a clear goal in mind and build an easy road towards it.

If you build a simple experiment you will find it easier to tackle arising problems. The impact of the experiment doesn’t need to be compromised because of it, it just makes it easier to address possible problems.

Look for experiments that are easy to execute using existing resources and staff because a simple risk is one that is self-contained and easily manageable.


Complexity is your enemy. Any fool can make something complicated. It is hard to make something simple.

Richard Branson

Companies such as Google, Amazon and Dunkin’ Donuts live by hassle-free working methods.

A simple brand concept is easier to digest by the consumers and therefore more trustworthy. The ‘Global Brand Simplicity Index’ shows how both customers and employees benefit from simplicity in innovation and customer service.

 3 Trying to Prove Too Much

Once an experiment is on-going, you’ll be faced with several challenges at the same time. This may cause you to seek solutions on all fronts, ending up in an overwhelming process of random fixups.

Instead: Focus on the big picture

Make it easy for yourself by focussing on one metric at a time.

There is a reason why enterprise organizations struggle to build Minimum Viable Products (MVP), as innovators try to prove multiple hypotheses within the same experiment and end up validating none.

Experimentation teams should focus on solving a single core problem and identify a single key metric for success. Narrowing your focus to one metric allows your team to maintain focus and manage progress more effectively.


Companies like Airbnb, Medium, Quora and Facebook use the ‘North Star Metric’.

The North Star Metric is the one metric that most accurately captures the unique value of a product or service. It drives all other metrics and is the guiding light you should focus on above everything else.

To paint a picture: Airbnb’s North Star Metric is the number nights booked per month. They’d rather have one member book ten trips in a year than five members booking just one.

Other notable examples: Medium’s total time spent reading, Quora’s total number of questions answered and Facebook’s the number daily active users.

4. Rejecting failure

Failure and innovation are two faces of the same coin.

If you are going to jump into experimentation, you’re going to fail… But that doesn’t mean you lost the battle. Knowing what doesn’t work gets you closer to finding out what does work. Avoiding failure doesn’t push your company forward, it’s holding it back.

Instead: celebrate it

To win at experimentation, you’ll have to turn things around: seek failure instead of avoiding it.

“Fail fast, fail often” is a mantra you should start living by.

Failing means you gain a true understanding of problems and needs. This puts you in a stronger position moving forward.

Not every idea will play out as intended and as long as it only affects a testing environment, that failure becomes teachable data.

Share the results and lessons learned from experiments across the company, then use those to inform new ones. The biggest breakthroughs are hidden within failed experiments.

The Tata Group in India celebrates failure by giving a prize to the most failed experiment. Every year the group selects the least successful idea in their award ceremony.

coffee being poured in a mug saying 'ugh'

5. Part-time experimentation

Experimentation is not something you can try on the side, and it certainly doesn’t happen overnight.

Much like any investment, it takes time before you can enjoy its fruits. Put differently: quick ROI doesn’t rhyme with experimentation. Big ROI does.
Your learnings will grow exponentially as you go to multiple rounds of experimentation. It takes time but the rewards are big.

Instead: Trust the process

Yes, a lot of experiments will fail and it might cost you valuable resources, but all it takes is a few successful ones to boost your company forward.

Always take the cost of opportunity into account when analyzing the success of your experiment.

Investing in innovation and experimentation means bringing fresh energy to your company. Every time you add something new to your business, it will increase your business potential energy.

However, generating ROI also doesn’t happen in a snap of a finger. It needs time and dedication.

Without experimentation, you won’t unlock the innovation needed to stay ahead of the curve. So yes, there will be bad days but you have to stick with it and trust the process.

Set aside a percentage of your marketing and product-development budget strictly for experimentation.

Most companies spend 0%–5% of their time and money on this when it should be closer to 10%.

Nike’s Lab was primarily developed to understand runners’ needs and match them by integrating that data into a new shoe. Today the research department is an advanced science lab where all new products come directly from.

6. Free falling on experimentation

Your experimentation team is set, trained and equipped with the right tools to start an experimentation program. But now what?

Although it can feel like it sometimes, experimenting shouldn’t be like skydiving into this exciting no-rule road. And yet, it often is.

Instead: Come up with a standard methodology.

Building a process for experimentation starts with studying the scientific method: a series of steps you’ll need to follow. This iterative process is the foundation of any experimentation culture.

The point of this process is to get everyone on the same page, to create structure without taking away creative freedom. And yes, experimentation should have little restraints but guidelines are necessary to avoid chaos and keep the team aligned.

Read up on the scientific process used in business experimentation on our article:

At the growth revolution, we implement a scientifically based methodology to our experiments.

1.Set Goals
Ask yourself what do you want to improve (be as specific and realistic as you can be about this), how are you going to measure it, who is going to work on it, how long will the experiment last?

2.Generate Ideas
Gather your team and develop brainstorm sessions. Try to produce as many ideas as possible during each session (even if they seem out of context or simply ridiculous, it is important to not censor any of the ideas). Question the ideas, develop them, reject some and select the ones that present the biggest potential.

3.Design Experiments
How do you turn ideas into experiments? You make an action plan.
It is important that this plan is a realistic one. Define step by step which actions are going to happen, who is going to execute them and define clear time-frames for this.
For example: If you are going to start posting informal content on your company’s LinkedIn page: Who is going to develop that content? Who is going to review it? When will that content be launched and how frequently this will be posted? How will you measure the results? What does a successful experiment mean?

Jump on it!

5.Study Data
Just as important as executing experiments is knowing how to analyze the outcome. As soon as you start your experiment you will get new information that you will have to put in perspective and analyze: Is the data conclusive? Why did this happen? Are there any outside factors that could have influenced the results? What conclusions can we take from the data collected? What can we improve?

6.Rinse and Repeat.
Experimentation is a never-ending path that only leads to even more growth. Keep on trying, keep on falling, keep on raising the bar and most importantly never be satisfied with the results.

It’s a trial-and-error process that allows us to get better every time and be ahead of the competition.

man jumping in the water

 7. Following the rules:

From the moment we are born, we are taught a certain code of rules: the do’s and don’ts of life to be able to fit in the society and succeed.
The same happens in the company culture: people follow a certain set of rules and norms that are part of the company’s ways of working

The problem: following these sets of rules is counterproductive when it comes to experimentation.

Instead: Empower curious minds

Most people have the assumption that all problems have a single right answer.

Schools and workplaces often develop this mentality by rejecting anything that falls outside of the line even though it might be a correct answer too.

Don’t expect your staff to change that mental model overnight, or on their own. It’s up to you to reinforce and foster the way they think.

It’s not the ability to come up with an original idea, but the ability to come up with lots of different answers to the same question.

Divergent thinking looks more like an insatiable curiosity than original ideas. It is an essential skill for innovation because it provides team members with the foundation to create great tests.

The goal is to gradually change a company’s culture from one that focuses on finding the right answer to one of exploring many possible answers.

During brainstorming, Steve Jobs used to invite his employees to join him for a walk instead of sitting around a table. This single habit was proven to boost creativity and improve divergent thinking. In a study for the American Journal of Human Biology, Yale researchers proved that non-cognitive movement while trying to come up with new ideas was relevant to creative thinking.

8. Micromanaging

It sounds scary to set your employees free into the experimental world. Although most C-level executives understand the added value of experimentation, they tend to stick to safe, calculated steps in moments of truth and end up oppressing the very experimentation culture they want to foster.

Instead, push laboratory mentality

Again: mindset mindset mindset.

The only way to encourage a team to be more innovative is to shift from a static, ‘best practices’ mentality to a lean, ‘laboratory’ mentality and to make each team member, not the manager, accountable for the results.

Managers should nurture a broad testing culture but stay on the sidelines when it is time for results for either good or bad.

The manager should focus on providing the right amount of support and keeping the team focused on the end goal.

Wells Fargo Bank ensured innovation inside their big corporative association by keeping a lean startup mentality throughout their experiments.

9. Thinking the most useful experiments are the most successful

I know it sounds counterproductive but some companies mistakenly believe that the only useful experiments are successful ones.

Without experimentation, managers generally base decisions on gut instinct. What is surprising is not just how bad those decisions typically are, but how good managers feel about them.

Instead: Create room for improvement

The goal is not to conduct perfect experiments but to learn and make better decisions than you are making right now.

Organizations that cultivate a culture of experimentation are often led by senior managers who have a clear understanding of the opportunities and set experimentation as a strategic goal of the firm.

Decisions based solely upon intuition should be censured, even if the hunch was subsequently proven correct.

Companies like Amazon, Netflix and P&G head the pack by favoring the 10.000 experiment method over the 10.000 hours of work one.

10 .Selecting Data

When running experiments you might encounter data that validates your hypothesis and pushes you forwards. Or, you can find data that isn’t conclusive. At first sight, it might seem that that kind of data brings nothing to the table, but you’re wrong.

Instead: Let the data speak.

Once you’ve analyzed the main results, ask yourself whether you’ve really discovered the underlying mechanism behind your results — the factor that is driving them. If you’re not sure, refine your experiment and repeat to learn more.

If you find yourself in a situation of chaos it might be because you’re not listening to what the data is telling you.

Look at it from a different angle: it might be that your experiment gave you new and important information that demands a rethinking of approach. If so, you should pivot and change direction.

Running experiments has no use when you neglect what the data is saying.

screen showing a series of coding text

Besides Airbnb running A/B product testing and analyzing the results through scientific methods, they also run dummy experiments. They find that any knowledge about how the system behaves is useful for interpreting results. Analyzing these incoherences has led them to discover a number of bugs and counterintuitive behaviors in their system.


All experiments should have one goal in mind: validate or invalidate an original propositionn and whether these experiments are small or large, natural or created.

Intuition is a natural important part of innovation but it needs data to back it up and testing before ideas get actually implemented. On the long-term, companies that truly embrace this data-driven approach will be more comfortable with running small-scale experiments to even low levels of management.
This will encourage out-of-the-box innovations that trigger a real transformation.

New innovative products, services, or innovation in business model will differentiate your company from the others on the market and that will unleash new potential to your company.

Related Posts

Comments (1)

Leave a comment