Kill That Project Dead! (Or at Least Rethink It.) Early experimentation separates viable products from ones that looked better on paper. Here are four signs that you should bail.
By Rudy Mutter •
Opinions expressed by Entrepreneur contributors are their own.
Product development is all about experimentation. Consider Twitter, which announced in late September that it would begin testing 280-character tweets.
Related: 10 Experiments to Test Your Startup Hypothesis
Twitter believes strongly in A/B testing, which is why the company is constantly prototyping, scrapping and revising new ideas. One experiment from 2014, involving a user's ability to add commentary when retweeting someone, almost didn't make it to the implementation stage. Today, it's one of the platform's most-used features.
Whether this latest experiment succeeds is not the point, however. The point is to prototype and test viable ideas, eliminating the ones that flounder, and throwing resources behind the ones that show promise. Twitter even has an "experiment shepherd" on staff who ensures that the team understands the experiment life cycle and follows set procedures.
Although most entrepreneurs recognize that experimentation is important, few recognize just how important. By embracing experimentation, founders can save money, boost sales and build stronger products. Here are the specifics of why experimentation is important.
Experimentation leads to innovation.
Experimenting earlier in the product's life cycle takes only a few weeks, compared to months when the product is mostly finished. It's a better, cheaper alternative than bringing to life every idea your team has.
Running experiments on every idea might sound expensive, but it's not nearly as expensive as building an entire product only to realize in the final stages that the product will fail. According to Harvard Business School professor Clayton Christensen, more than 30,000 consumer products go to market every year, and 80 percent fail. Early prototypes' failures cause a lot less stress than a full-blown product failure (not to mention the lost revenue).
Related: The 7 Steps of Effective Product Development
In advertising and monetization, A/B testing small UX changes helps businesses fine-tune their signup processes. Slow rollouts of changes to hardware and software also provide opportunities for experimentation. The key is to test as early as possible using a close approximation of the final product.
Of course, the purpose of an experiment is to determine whether an idea will work -- meaning many won't. Early experimentation separates viable ideas from ones that looked better on paper. But, to make experimentation valuable, businesses need to recognize the warning signs of a bad experiment and know when to bail or change course.
When to kill an experiment -- factors to help you decide
1. Deadlines and budgets are blown. Whether for digital or physical products, experiments are supposed to be temporary. Ideally, a product stakeholder will follow both budget and timeline and then determine whether the idea is worth pursuing further -- but the unknown element of experimentation can knock plans off track.
When working with the unknown starts inflating the budget or pushing the deadline far beyond initial projections, halt the project. Ask whether the potential outcome justifies the added investment. If not, give it up for good.
My company faced this situation when we were developing a mobile app for our client National Fitness Campaign. We wanted to experiment with Instagram's API to build an integration that would make our client's app more robust. But when we noticed both our deadline and budget cap looming without the accomplishments we'd wanted by then, it became apparent that we needed to change course. We ditched our original idea to find a smarter solution.
2. Data and analytics don't support the hypothesis. Set reporting parameters before beginning an experiment, and check the numbers regularly. If the results of the experiment are discouraging, don't ignore them. Recognize the need to adapt or cancel the project.
Twitter will rely on data to determine whether to keep the new 280-character limit. According to the blog post announcing the experiment, English users of the platform hit the current 140-character limit 9 percent of the time. Whether that group makes use of the extra space, along with many other factors, will determine the experiment's fate.
3. Scope creep derails the initial experiment. Scope creep refers to experiments that grow beyond their initial objectives (it can also refer to times when clients ask for so much extra work, the initial quote no longer covers the cost of labor).
According to the Standish Group's CHAOS Manifesto 2012, 43 percent of projects go over budget, beyond the deadline or beneath the promised results. When scope creep begins, projects can end up costing much more than their initial projections. As ROI shrinks and maintenance costs rise, consider how well the experiment is progressing. Some projects deserve extra investment; others should be cut.
4. Users are pushing back. User testing and interviews are critical components of successful experiments. Negative feedback merits a closer look.
Do users genuinely dislike the experiment, or are they simply afraid of change? Jakob Nielsen recommends using five-person test groups to get an accurate assessment. Fewer than five people could skew results, while more than five could become prohibitively expensive.
Related: When to Selectively Listen to Feedback and Ignore What Users Say
User satisfaction is key to any successful experiment. Customers 2020, a study from Walker, estimates that the customer experience will become more important than the price or product features by 2020. Listen to users, judge the merit of their responses and halt the project if they have too many shared complaints.
Related: The 3 Biggest Roadblocks in Product Development
Experimentation is the heart of innovation, but not every experiment will pan out. The solution isn't to stop experimenting, though; it's to recognize early which ideas are worth pursuing and which should be shut down.
