Get All Access for $5/mo

Why You Don't Want to Fail This Direct-Mail Test To measure the success of your direct mail efforts, you've got to test your campaigns first. Follow these guidelines to get the most bang from your buck.

By Robert W. Bly

Opinions expressed by Entrepreneur contributors are their own.

ideabug | Getty Images

The following excerpt is from Robert W. Bly's book The Direct Mail Revolution: How to Create Profitable Direct Mail Campaigns in a Digital World. Buy it now from Amazon | Barnes & Noble | Apple Books | IndieBound

In direct mail (DM), testing is the process of putting a letter or package in the mail, counting the replies, and coming to a conclusion based on the results.

Testing is a huge advantage that direct marketers have over branding and general advertisers: We first do a small test to determine whether our direct-mail package works. If it does, we can gradually expand the campaign. On the other hand, if the test bombs, we know early on that the package doesn't work. The test costs only a few hundred or a few thousand dollars, and it saves us many more thousands of dollars by not continuing to mail a DM package that consistently loses money.

Testing is one of the central ideas of direct-mail marketing: Test small, then roll out in larger quantities once the tests show you which is the winning package.

Related: The 5-Step Plan for Turning Prospects Into Customers

The 3 most important factors to test

What are the three most significant factors you can test -- the ones that can have the greatest influence on response?

Number one is the mailing list. There could be a half-dozen mailing lists suitable for your offer -- or even more. You can't assume you know which one is best based on your personal biases. The only way to know for certain which list will pull best with your package is through a test mailing.

The second most important factor to test is the price. This applies mainly to mail-order selling. For instance, let's say you've published a thousand-page market-research report on broadband internet. How much will people pay for it? $195? $495? $1,200? You simply don't know until you test. And frequently you'll be amazed at how many people place orders at prices you think are sky-high.

The third most important factor to test is the offer. Should you try for mail orders or leads? Should you offer a premium? If you do, will you get better response offering a gift item such as a digital watch or free information such as a booklet or special report? You won't know which works better unless you test.

A/B split tests

When two mailings or mailing factors are tested against each other, it's called an A/B split test, with one version labeled as test cell A and the second as test cell B. For instance, you might test letter A against letter B to see which pulls more orders. Or you might take letter A and mail it to two different lists, to see which list produces the better response. Or you might mail a control as test cell A against a new test package as test cell B.

A control is the current best-performing DM package. For instance, a marketer may be mailing thousands of the same direct-mail package month after month because it's profitable. But how do they know another package, with different graphics, size, colors, and copy, won't generate even better results? They can't, unless they test it. So they periodically commission a new direct-mail package or put one together in-house and then mail it against their control in an A/B split test.

There are two approaches to split testing, and each has its place. The first approach is to test two completely different DM packages and see which one is the winner. The winner beats the current control and becomes the new control. However, if the new test package is completely different from the control (different graphics, copy, price offer, guarantee, package format, etc.), you won't know which of these elements made the difference.

The second approach is to test multiple versions of the control where just a single element is different; for example, the envelope teaser, copy theme, size of envelope, price, premium, or first-class vs. third-class postage. By testing just one variable at a time, you can determine how each factor influences response.

Related: Looking for Leads? These 4 Direct Mail Offers Are Your Answer

Number of DM Pieces per Test Cell

Statistical analysis shows that you can get a valid test result with as few as 2,000 names per cell. The validity is determined not by the number of pieces mailed but rather the number of replies received per cell.

Experience and statistical analysis indicate that 14 responses per test cell give you a fairly reliable reading of each cell's performance. If your average response is 1 percent, then 1,400 names per test cell is an adequate size. However, because response is unpredictable, 2,000 per cell will give yourself some wiggle room.

Tracking Responses

When testing, you must be able to track response -- to identify that reply as coming from a specific mailing or from a recipient whose name was on a specific mailing list. There are several ways to do this. The simplest is to put a key code on the reply element. This code can be a series of numbers and letters in fine print tucked away in the corner of the reply card, or it can be worked into the address. Your list broker can handle this for you.

If you're affixing or imprinting the recipient's address on your reply card or order form, the mailing-list owner can add a key code to the order form. The same coding can be done for telephone responses. For catalogs, when you call to order, the customer service representative typically asks for a code printed in a blue or yellow box on the back cover near the recipient's address.

Related: How to Avoid Each of the 6 Most Common Reasons Digital Lead Generation Systems Fail


After a successful test, a winning direct-mail package is "rolled out," meaning it's mailed to more names on the profitable lists. But can the results of a small test mailing remain statistically valid regardless of how many additional names we mail to? No. The rule of thumb is that the total quantity you mail to should be no more than 10 times the number of names you tested. Therefore, if you got a 5 percent response in a test of 5,000 names, you can mail to as many as 50,000 additional names on the list and be confident that you'll get a similar response.

Robert W. Bly

Author, Copywriter and Marketing Consultant

Robert W. Bly is an independent copywriter and marketing consultant with more than 35 years of experience in B2B and direct response marketing. He has worked with over 100 clients including IBM, AT&T, Embraer Executive Jet, Intuit, Boardroom, Grumman and more. He is the author of 85 books, including The Marketing Plan Handbook (Entrepreneur Press 2015), and he currently writes regular columns for Target Marketing Magazine and The Direct Response Letter.

Want to be an Entrepreneur Leadership Network contributor? Apply now to join.

Editor's Pick

Business Solutions

Increase Productivity with This Microsoft 365 Subscription, Now $25 Off

It can make the entrepreneur life a lot easier.

Business News

Apple Pay Later Is Ending. Here's What's Taking Its Place.

The program was available for less than a year.


This Artist Answered a Businessman's 'Powerful' Question — Then His Work Became 'the Poster Child for Juneteenth': 'Your Network Really Becomes Your Net Worth'

Reginald Adams was the executive director of a Houston-based art museum for more than a decade before he decided to launch his own public art and design firm.


Harvard Business School Professor Says 65% of Startups Fail for One Reason. Here's How to Avoid It.

Team alignment isn't nice to have -- it's critical for running a successful business.

Business News

Here's What Companies Are Open and Closed on Juneteenth 2024

Since it became a holiday in 2021, Juneteenth has been recognized by some major corporations as a paid day off.

Growing a Business

I Hit $100 Million in Annual Revenue by Being More Transparent — Here Are the 3 Strategies That Helped Me Succeed

Three road-tested ways to be more transparent and build relationships that can transform your business — without leaving you feeling nightmarishly over-exposed.