Get All Access for $5/mo

The Case Against A/B Testing at Early-Stage Startups A/B tests have become so ubiquitous that founders who don't frequently employ them have been subtly made to feel bad about themselves for having such a weak data-driven culture.

By Andrew Cohen Edited by Dan Bova

Opinions expressed by Entrepreneur contributors are their own.

Shutterstock

If you've been listening to the tech startup illuminati over the past few years, you'd be forgiven for thinking that the cult of A/B testing has become permanently ingrained in the "lean startup" mentality. Indeed, whenever there is a product question that cannot be answered by mere intuition, today's rock star startups seem to unquestionably advocate the so-called "split test" as the arbiter of all crowd wisdom.

A/B tests have become so ubiquitous that founders who don't frequently employ them have been subtly made to feel bad about themselves for having such a weak data-driven culture. This has gotten a little out of hand. Yes, it is true that A/B tests can provide tremendous insight into online user behavior, which can -- with a lot of data -- help you improve your conversion rates. But there is a problem.

Related: So, What's the 'X-Factor' in the A/B Testing Formula? (Infographic)

The problem at early-stage companies is that early optimization can distract management from much higher yield activities, while often not having a large enough sample size to yield meaningful results.

Performing split tests is simply a management-intensive task, no matter how cheap and efficient A/B testing plugins such as Optimizely have become. Someone needs to devote their time to determining what to test, setting up the test and verifying and implementing the results of the test. Even if this technically does not require very much "time," it requires management's mental bandwidth, which is the scarcest resource at any company (especially an early-stage startup).

Do you really want your co-founder to be distracted by yielding a 5 percent better conversion rate (if he or she is lucky), when you haven't yet proved that there is a burning need for your product or that you are able to scale your marketing effectively?

Related: The Biggest, and Most Avoidable, Mistake of New Entrepreneurs

Similarly, how can you be sure that those A/B test "results" are statistically significant in the first place? Young companies typically have low amounts of web traffic, which may not produce usable A/B test conclusions for several weeks or months, if at all. And the frequently changing nature of your web traffic can make it very hard to draw meaningful conclusions to begin with. If 80 percent of your traffic this month came from a single TechCrunch article or Facebook ad blitz, is that really representative of how your core user base will behave on that target web page in the future?

Early-stage startups should therefore be careful not to get too bogged down in perfecting their marketing copy or the color of their call-to-action buttons until they are confidently nailing all the fundamentals. Building a metrics-driven culture at your startup means ensuring that you are always focusing on the right key performance indicators and not optimizing them too early. Sometimes a quick focus group, hallway usability test or intuition can be sufficient for you to make product decisions that are "good enough" for your company's current size.

Have you had positive or negative experiences with A/B testing at your startup? Sound off in the comments.

Related: The Pros Agree These Are the 2 Best Tools for Conversion Optimization

Andrew Cohen

Founder & CEO, Brainscape; Instructor, TechStars and General Assembly

Andrew Cohen is the founder of Brainscape, a web and mobile education platform that helps people study more efficiently. Brainscape originally grew out of a personal project that Cohen created to help him improve his Spanish, while working in Panama for the World Bank. It later inspired him to seek a master's degree in instructional technology from Columbia University and transform his pet project into a fundable startup that can help people study any subject. Brainscape has since raised several million dollars from top venture capitalists.

Want to be an Entrepreneur Leadership Network contributor? Apply now to join.

Editor's Pick

Science & Technology

5 Automation Strategies Every Small Business Should Follow

It's time we make IT automation work for us: streamline processes, boost efficiency and drive growth with the right tools and strategy.

Business News

Former Steve Jobs Intern Says This Is How He Would Have Approached AI

The former intern is now the CEO of AI and data company DataStax.

Leadership

Visionaries or Vague Promises? Why Companies Fail Without Leaders Who See Beyond the Bottom Line

Visionary leaders turn bold ideas into lasting impact by building resilience, clarity and future-ready teams.

Marketing

5 Critical Mistakes to Avoid When Giving a Presentation

Are you tired of enduring dull presentations? Over the years, I have compiled a list of common presentation mistakes and how to avoid them. Here are my top five tips.

Side Hustle

'Hustling Every Day': These Friends Started a Side Hustle With $2,500 Each — It 'Snowballed' to Over $500,000 and Became a Multimillion-Dollar Brand

Paris Emily Nicholson and Saskia Teje Jenkins had a 2020 brainstorm session that led to a lucrative business.

Green Entrepreneur®

How Global Business Leaders Can Build a Sustainable Supply Chain

Businesses can build sustainable supply chains by leveraging technology to reduce environmental impact, optimize resources and track emissions while balancing operational efficiency and sustainability goals.