Primer on Statistical Sampling Measures

In many direct marketing channels [direct mail, telemarketing, email marketing], we control exactly who is offered a product/service. Because we control the targeting of the offer, this means we can scientifically test different offers/communications against samples to see if we can find a better way to target our markets. While many marketers - particularly those new to direct marketing - are often intimidated by statistics and the concepts required to build statistically reliable samples, there are a number of free tools you can use on the internet to assist you with the statistical part...provided you understand how to use them. That is the mission of this blog entry.

There are two key uses of statistics in developing direct marketing samples. The first is to build the correct sample sizes to accurately read the market. This means having samples which allow you to compare your control sample with a test sample to determine if the test is a winner. This calculation is done before you create your marketing plan.

The second application is after your campaign is completed. When done, you might see test panels which have exceeded the control in the critical measures you find important. In this case, you might have a "winning" test package which can now become you new roll-out and control. The key question is "while the test looks like a winner, can I be assured it will perform that way once I roll-out the test?" What you DON'T want is to accept a test panel and then - in the next campaign - roll it out to the majority of your market with disasterous results.

To address these two statistical sampling questions, here are the things you need to know and sites you can visit to find sample size calculators:

Determining Sample Sizes


If you create a sample size that is too low, you run the risk to accepting a test as a winner and then findings it doesn't work when rolle-out to larger populations. If you create a sample size that is too large, you are wasting money [most of the time]. So creating the appropriate sample size is important...and relatively easy. Here is what you need to do:

1. Determine the expected response rate for your campaign

Take a look at past campaigns and determine the average response rate you will receive from the product your are selling and the channel you are using. If you are new to the channel and/or the product, ask your list brokers, lettershops, or processors to help you determine the "average" response rate you will get from the channel and the product. If you are unsure, estimate low.

2. Determine the Acceptable Error for the channel / product
This sounds like a diffult step but it - too - is relatively easy. Take a look at your past campaigns again and see the "above and below average" levels your see for the same channel / product. For example, if the average response rate is 1% with a high level of 1.2% and a low of .8 percent, we would say our average response rate is 1% plus or minus .2%. The .2% is the acceptable error. If you don't know it, estimate it low. Be sure to not be too skewed by seasonality. What you want to determine is how accurately can you predict the response rate around an acceptable level of error.

3. Establish your Confidence Level
This is relatively easy. How much error can you live with? If you are a direct marketer, not much. Generally, all statistical sampling calculations are done with a 95%, 98% or 99% confidence interval. This means if we see a winner, we want to be right in reading this result 99% of the time. When in doubt, use 99%.

Once you have these calculations, then you need to determine your sampling size. You can do this by entering the numbers into a sample size calculator. Where can you find them? Here are two free calculators which work well:

Here is one from Torque.com
Here is another one from Open Up to Mail

For example, if you had an average response rate of 1% with an acceptable error at .2% and a confidence level of 99%, the sample sizes for our control and test panels would need to be at or larger than 16,474. To make life easier, we should probably test 16,500 or 17,000 in our test and control panels. Going higher NEVER gets you into trouble. LOWER...BIG TROUBLE!

Both calculators work well. If you can't get these to work or need more info, contact me at info@msinetwork.com


Identifying the Winners

You now have a test campaign which might be a winner. It has a stronger response rate compared to the control panel...but your not sure. How can you determine if it truely is a winner?

The answer to this is another statistical calculation and - like the last one - there are calculators which can make your work simpler.

When you sample, you are accepting some level of error around an "average" performance rate. What does this mean? Remember the bell curve?

If you have a control package with a response rate of 1%, if you mailed the same size population over and over again, the response rate would not always be 1% but many would be slightly above 1% and others slightly below 1%. Some would be significantly different [outlyers] but most would cluster around 1%. When we set a confidence interval of 99%, this means we want to consider all of the possibilities around an average response rate of 1% except for the extreme .5% at the high and low extremes of the bell curve. This means we consider 99 out of 100 campaigns.

Why is this important? If we have a control with a response rate of 1% and we get a test "winner" at 1.2%, we need to know if the test will replicate if we accept it as our new roll-out pacakge. In other words, the 1.2% might be a reliable winner or it might be the "high side" of a bell curve which will likely not happen again. How do we determine if it is a reliable winner? With a simple statistical test.

For this test, you - again - need to know 3 things...all of which you can get from your campaign results. They are:

1. What is the size of your samples?
This is what you calculated in the last step. Let's use the example sample size of 16,474.

2. What is the response rate?
Here we have two. Our control is 1% and our test is 1.2%

3. What is the confidence level?
Again, let's be consistent and use 99% ... just like our sample size calculation.

If you use a result range calcuator, you get the following results:

Control 1% plus or minus .2%
This means a high of 1.2% and a low of .8%

Test 1.2% plus of minus .21%
This means a high of 1.41% and a low of .99%

What does this mean? If you look at the high and low ranges, you will see that the low end of the bell curve of the test result [.99%] overlaps significantly with the high end of the control [1.2%]. This means there is considerable risk in accepting the test as the new control.

What do we ideally want? We want results where the high of the control does not touch the low of the test. This means the test is clearly better and we can roll it out with confidence.

To calculate these types of measures, you can use a results range calculator like that offered by Torque. To see one, click here

I hope this gives you some insights into using basic statistical measures. Once again, if you are having problems, please email me at
info@msinetwork.com or respond to this blog.

While this explaination will not be copmlete for a statistician or a sophisticated direct marketer, it will get you started in developing statistically reliable tests. Good luck and let me know if you have questions.