Why You Don't Want to Fail This Direct-Mail Test

To measure the success of your direct mail efforts, you've got to test your campaigns first. Follow these guidelines to get the most bang from your buck.

learn more about Robert W. Bly

By Robert W. Bly

ideabug | Getty Images

Opinions expressed by Entrepreneur contributors are their own.

The following excerpt is from Robert W. Bly's book The Direct Mail Revolution: How to Create Profitable Direct Mail Campaigns in a Digital World. Buy it now from Amazon | Barnes & Noble | Apple Books | IndieBound

In direct mail (DM), testing is the process of putting a letter or package in the mail, counting the replies, and coming to a conclusion based on the results.

Testing is a huge advantage that direct marketers have over branding and general advertisers: We first do a small test to determine whether our direct-mail package works. If it does, we can gradually expand the campaign. On the other hand, if the test bombs, we know early on that the package doesn't work. The test costs only a few hundred or a few thousand dollars, and it saves us many more thousands of dollars by not continuing to mail a DM package that consistently loses money.

Testing is one of the central ideas of direct-mail marketing: Test small, then roll out in larger quantities once the tests show you which is the winning package.

Related: The 5-Step Plan for Turning Prospects Into Customers

The 3 most important factors to test

What are the three most significant factors you can test -- the ones that can have the greatest influence on response?

Number one is the mailing list. There could be a half-dozen mailing lists suitable for your offer -- or even more. You can't assume you know which one is best based on your personal biases. The only way to know for certain which list will pull best with your package is through a test mailing.

The second most important factor to test is the price. This applies mainly to mail-order selling. For instance, let's say you've published a thousand-page market-research report on broadband internet. How much will people pay for it? $195? $495? $1,200? You simply don't know until you test. And frequently you'll be amazed at how many people place orders at prices you think are sky-high.

The third most important factor to test is the offer. Should you try for mail orders or leads? Should you offer a premium? If you do, will you get better response offering a gift item such as a digital watch or free information such as a booklet or special report? You won't know which works better unless you test.

A/B split tests

When two mailings or mailing factors are tested against each other, it's called an A/B split test, with one version labeled as test cell A and the second as test cell B. For instance, you might test letter A against letter B to see which pulls more orders. Or you might take letter A and mail it to two different lists, to see which list produces the better response. Or you might mail a control as test cell A against a new test package as test cell B.

A control is the current best-performing DM package. For instance, a marketer may be mailing thousands of the same direct-mail package month after month because it's profitable. But how do they know another package, with different graphics, size, colors, and copy, won't generate even better results? They can't, unless they test it. So they periodically commission a new direct-mail package or put one together in-house and then mail it against their control in an A/B split test.

There are two approaches to split testing, and each has its place. The first approach is to test two completely different DM packages and see which one is the winner. The winner beats the current control and becomes the new control. However, if the new test package is completely different from the control (different graphics, copy, price offer, guarantee, package format, etc.), you won't know which of these elements made the difference.

The second approach is to test multiple versions of the control where just a single element is different; for example, the envelope teaser, copy theme, size of envelope, price, premium, or first-class vs. third-class postage. By testing just one variable at a time, you can determine how each factor influences response.

Related: Looking for Leads? These 4 Direct Mail Offers Are Your Answer

Number of DM Pieces per Test Cell

Statistical analysis shows that you can get a valid test result with as few as 2,000 names per cell. The validity is determined not by the number of pieces mailed but rather the number of replies received per cell.

Experience and statistical analysis indicate that 14 responses per test cell give you a fairly reliable reading of each cell's performance. If your average response is 1 percent, then 1,400 names per test cell is an adequate size. However, because response is unpredictable, 2,000 per cell will give yourself some wiggle room.

Tracking Responses

When testing, you must be able to track response -- to identify that reply as coming from a specific mailing or from a recipient whose name was on a specific mailing list. There are several ways to do this. The simplest is to put a key code on the reply element. This code can be a series of numbers and letters in fine print tucked away in the corner of the reply card, or it can be worked into the address. Your list broker can handle this for you.

If you're affixing or imprinting the recipient's address on your reply card or order form, the mailing-list owner can add a key code to the order form. The same coding can be done for telephone responses. For catalogs, when you call to order, the customer service representative typically asks for a code printed in a blue or yellow box on the back cover near the recipient's address.

Related: How to Avoid Each of the 6 Most Common Reasons Digital Lead Generation Systems Fail


After a successful test, a winning direct-mail package is "rolled out," meaning it's mailed to more names on the profitable lists. But can the results of a small test mailing remain statistically valid regardless of how many additional names we mail to? No. The rule of thumb is that the total quantity you mail to should be no more than 10 times the number of names you tested. Therefore, if you got a 5 percent response in a test of 5,000 names, you can mail to as many as 50,000 additional names on the list and be confident that you'll get a similar response.

Robert W. Bly

Author, Copywriter and Marketing Consultant

Robert W. Bly is an independent copywriter and marketing consultant with more than 35 years of experience in B2B and direct response marketing. He has worked with over 100 clients including IBM, AT&T, Embraer Executive Jet, Intuit, Boardroom, Grumman and more. He is the author of 85 books, including The Marketing Plan Handbook (Entrepreneur Press 2015), and he currently writes regular columns for Target Marketing Magazine and The Direct Response Letter.

Related Topics

Editor's Pick

Everyone Wants to Get Close to Their Favorite Artist. Here's the Technology Making It a Reality — But Better.
The Highest-Paid, Highest-Profile People in Every Field Know This Communication Strategy
After Early Rejection From Publishers, This Author Self-Published Her Book and Sold More Than 500,000 Copies. Here's How She Did It.
Having Trouble Speaking Up in Meetings? Try This Strategy.
He Names Brands for Amazon, Meta and Forever 21, and Says This Is the Big Blank Space in the Naming Game
Business News

These Are the Most and Least Affordable Places to Retire in The U.S.

The Northeast and West Coast are the least affordable, while areas in the Mountain State region tend to be ideal for retirees on a budget.

Business Ideas

55 Small Business Ideas To Start Right Now

To start one of these home-based businesses, you don't need a lot of funding -- just energy, passion and the drive to succeed.

Business News

I Live on a Cruise Ship for Half of the Year. Look Inside My 336-Square-Foot Cabin with Wraparound Balcony.

I live on a cruise ship with my husband, who works on it, for six months out of the year. Life at "home" can be tight. Here's what it's really like living on a cruise ship.

Business Solutions

Master Coding for Less Than $2 a Course with This Jam-Packed Bundle

Make coding understandable with this beginner-friendly coding bundle, now just $19.99.

Starting a Business

Ask Marc | Free Business Advice Session with the Co-Founder of Netflix

Get free business advice during our next Ask Marc, live Q&A, on 3/28/23 at 3 p.m. EDT. You don't want to miss it—send in your questions now.

Health & Wellness

5 Essential Steps to Expand Your Vision and Start Living Your Dream Life

It's time to break free from your comfort zone and expand your vision. When you refuse to settle for a mediocre life, you can start building a life you love.