Darren Herman

Owyang and Caddell are testing Facebook’s SocialAds system by purchasing $20 worth of ads and calling it a day. Both of these guys are immensely respectable and I’ve been fortunate to have some great recent conversations with Jeremiah. Right when Jeremiah published a post about his SocialAds test, I submitted a comment on his blog:

1. Darren Herman November 7th, 2007 8:28 am

   Jeremiah, is $20 going to give you a statistically valid sample to extrapolate enough data from to see whether or not Facebook will work?

Jeremiah quick responded with the following:

1. jeremiah_owyang November 7th, 2007 9:37 am

   Darren

   Nope it’s not, it’s just at test, a trial. I talk to many clients who are using social media, I’ll also be able to hear from them.

   My goal is to understand how the system works, break it, make mistakes, and make sure my clients don’t have to.

I’m concerned about the following: If Owyang and Caddell both have negative experiences with SocialAds over a $20 campaign, I’m sure that they will post (since that’s why they started the topic) and many secondary sources will pick it up and talk about how SocialAds do not work, based on their data. Gotta love the power and influence of the Internet.

Let me please go on the record and say that these $20 “tests” are “trials” and they do not have any statistical relevance. The sample size is too small to make any overarching statements whether or not Facebook’s SocialAds work and I believe a larger test with some 3rd party auditing & measurement may be required.

Market research is an entire industry in itself and wasn’t invented overnight. There is a lot of technical innovation that is forcing rapid change to the industry, but the underlying methodologies are relevant. Calculating sample size is important for any research and here is a posting on how to do it.

If you don’t feel like reading the entire posting, here are some simple details:

For a meanThe required formula is: s = (z / e)2

Where:
s = the sample size
z = a number relating to the degree of confidence you wish to have in the result.? 95% confidence* is most frequently used and accepted. The value of ‘z’ should be 2.58 for 99% confidence, 1.96 for 95% confidence, 1.64 for 90% confidence and 1.28 for 80% confidence.
e = the error you are prepared to accept, measured as a proportion of the standard deviation (accuracy)
For example, imagine we are estimating mean income, and wish to know what sample size to aim for in order that we can be 95% confident in the result. Assuming that we are prepared to accept an error of 10% of the population standard deviation (previous research might have shown the standard deviation of income to be ?8000 and we might be prepared to accept an error of ?800 (10%)), we would do the following calculation:
s = (1.96 / 0.1)2
Therefore s = 384.16
In other words, 385 people would need to be sampled to meet our criterion.
*Because we interviewed a sample and not the whole population (if we had done this we could be 100% confident in our results), we have to be prepared to be less confident and because we based our sample size calculation on the 95% confidence level, we can be confident that amongst the whole population there is a 95% chance that the mean is inside our acceptable error limit.? There is of course a 5% chance that the measure is outside this limit. If we wanted to be more confident, we would base our sample size calculation on a 99% confidence level and if we were prepared to accept a lower level of confidence, we would base our calculation on the 90% confidence level.`

If we’re going to go testing the SocialAds system and other new and innovative systems and draw conclusions, lets do it as best as possible so that the world at large can benefit from statistically relevant data.