There’s a problem that I see often. Companies are looking for fast results, and are unwilling to see a strategy through to its conclusion. I see this in AdWords, when companies want to test new ads weekly, and want to turn off low performing keywords before there’s enough data to really get a picture of what’s happening. I also see this in split testing, when the desire to ‘call’ a result outweighs the rigor required for statistical validity.

I noticed yesterday that Google’s free split testing tool, Website Optimizer, suggests that each page variation have at least 100 conversions before a split test can be considered valid. So for an A-B-C test, you would need 300 conversions before declaring a winner. Even with a 10% conversion rate, you are talking about 3,000 impressions, which might take some sites months to gather.

When looking at AdWords I understand the need for faster (and perhaps more intuitive) decision making, but there is still a simple rule I use here. When looking at a keyword (or ad, etc) that I think might be a low performer, I ask the question, if the next two clicks on this keyword (or ad, or whatever) were both conversions, would I still consider this to be a poor performer? If the answer is yes, then I feel okay making the change, but if the answer is no, then it’s a clear indication that there’s simply not enough data for a good decision.

Inside AdWords, I also try to look at at least a month of data, preferably two months. I never look at a single day of data, there are just too many swings of results for that data to be useful.

So, c’mon web marketers, let’s develop the discipline to see these strategies through!

PS If you’re Amazon.com, feel free to ignore this advice. You’re probably running split tests in a matter of hours…