AB Testing: Too Little, Too Early?

AB Testing: Too Little, Too Early?: "Lately, everywhere you go analytics industry folks are talking about AB Testing. That's a good sign, since it means the industry is focusing on an overlooked leverage point in their web analytics investment. But as so often happens, achieving full buzzword compliance has become the goal rather than the means; what lies behind the words is often lost. In this case, 'AB testing' – the buzzword – has become a euphemism for plain old 'testing', which, like ordering liver on a first date, may be good for you, but is certainly not sexy. But throw some 'AB' in front of 'testing' and your dour liver is magically transformed into paté de foie gras. This is a bit disturbing, especially when you hear people sprinkling the 'AB' condiment to add flavor to anything from a focus group ('Hey, did you AB Test the response to the new company logo?') to the mundane ('Suzie's lamp is out, can you AB Test the light bulb?') to the painfully comical ("Honey, let's AB test the Lord of the Rings Director's Cut with the Wide-Screen edition!").

Mixed in there, perhaps lost among the cacophony of buzzword hype, are the ingredients to some real AB testing and with it a future vision of how to achieve its true objective.

What is AB Testing?

AB Testing is based on a simple principle that we're all familiar with: compare and contrast alternatives; based upon measurement, act accordingly. Let's say we want to determine whether Nolan Ryan is a better baseball player than Homer Simpson? How should we proceed? First, we might set a metric for what we mean by a "better" baseball player. We can measure evidence in concrete ways, noting the two subjects' different batting averages or RBIs or the like. What we're searching for is the right metric—a formula that would lead us to a correct decision. Such a formula is more precisely termed a "fitness function."

We might decide that considering indirect evidence will lead us to a better decision than comparing pure statistics. In that case, our fitness function may involve such things as the difference in salary paid for services or a comparison of the prices paid for our subjects' autographs on eBay.

AB Testing: Too Little, Too Early?

In virtually all such measures Nolan is the better candidate. If you were choosing a player for your team you'd certainly pick Nolan; you can be confident you've made the correct decision.
But let's think on that a moment: the reason you feel confidence in signing Nolan stems from your familiarity with the metric and fitness function that are implicitly applied when we speak of baseball. Your decision might be quite different if we want to pick an effective donut quality assurance taster. Suddenly, Homer Simpson is back in the running.


Source: John Quarto-vonTivadar,
Chief Technology Officer,
Future Now, Inc.

http://www.futurenowinc.com


Want more info on AB Testing get in touch at www.ju2.com