Power Direct Marketing: The Book
How Big Is a Test?
Lets have a little dialogue about how big a test should be.
How large is largehow small is small? And what difference does it make?
Vin Jenkins, principal in his own direct response agency in Australia, has gone to great lengths to demonstrate that an honest and valid test of lists, an offer, a packageanything with direct mailcan be done with fewer than 2,000 units. If you do all the right things right and have the sophisticated math models to measure same, Im sure Vin is right. It can work.
With another view is Al Mercer of Canada. Al talks about a 10% rule. A rule of thumb which says that you always test 10% of a list to predict how it is most likely to perform in a rollout. No matter the size of the list universe, you always test 10%.
And then there is me. I more or less disagree with both these fine, experienced direct marketing pros. Why? Because I sold lists for a number of years and have bought hundreds of lists for scores of clients since.
Ive been involved with direct response programs that used mail, space, telephone, and allied collateral and support materials. My comfort level is different because of my experience. It is different from both Vin and Al for testing of lists, for testing offers of any type, for testing creativefor testing anything.
Why do list owners, managers, and brokers have minimum order numbers for lists? One reason is the volume required for computers to be cost efficient. But a more important reason is that unless you put enough numbers into the marketplace, you will not have a measurable response to analyze.
It is not uncommon for lists to be available in a minimum 5,000 or 10,000 quantity. The list owner/broker wants the list to work just as much as you do. Without a sufficient test you may never be sure. Even if you hit it lucky and get a high response off a small number mailed, there is no assurance youll experience the same results with the rollout.
Personally, Im not comfortable with less than 2,500 names in any one test cell. Nothing less than 2,500 in an offer testa test of price, a premium, a limited time offer, anything. And I really prefer 5,000 names on all list tests. As a minimum.
The key in testing, and then being able to measure and analyze what happens, is not how many prospects hear, see, or read about youit is how many respond! Response is the key, the trigger. Not the number out the door, but the number that come back.
When you test direct mail packages, layout and design, different copy approaches, various package sizes, offers of any and all types, what you are seeking is a winning package. And, as the title of this chapter suggests, "winning isnt everything, but who cares about all that other stuff!"
Television and space are a little different because your audience reach is different. Here youre probably talking to thousands, maybe millions, with a single effort. And even though the media of print and broadcast are certainly different from direct mail and each other, you will get a "feel" of the success quickly. Maybe with one insertion of the ad or a short flight of broadcast youll learn sufficient from your analysis to decide direction.
Whats best for you? Beats me! Broken record, but "I dont know" and "it depends" are still the correct answers. What you test and the size of the test, even if you test, very likely will be influenced, if not decided, by the size of your marketplace.
If youre a consumer marketer, you probably have tens of thousands to millions of prospects. If youre business-to-business, you may have only a handful. You decide whats best for you. You decide.
by ROCKINGHAM*JUTKINS*marketing, all rights reserved.