Statistics for PPC

There has been a couple of good posts recently about statistics and PPC advertising. This in an area that I have thought about a lot so, even though my thoughts on this are no where near complete (or perhaps even correct) I thought I'd share them with you. This post is a bit of a monster.

The first post that caught my eye was by Craig Danloff who wrote about Bad Stats and Good PPC on the Click Equations blog. His post was motivated by an article in the New Yorker called The Truth Wears Off in which the author describes several scientific results, the strength of which (in terms of statistical significance) appears to be reducing over time. What hope is there for a lowly PPC manager when professional scientists are unable to produce rigorous results?

The second article on PPCHero is titled Statistical Validity Is Leading You Astray and Jessica makes a similar point to Craig; that there are too many uncontrolled variables to make statistical testing the be all and end all of marketing decision making.

This month I will run more experiments than the average PhD does during their whole course

I think that both these authors make a very good point that I am not going to disagree with. Instead I want to put across my own, similar view point about why the statistical techniques of the science lab are not the best ones for our purpose:

1. We run split tests until we get a result. I have never heard anyone talk about their split testing procedure for when they get a null result; when there is no statistically significant difference between the variations. People just run the test until they get a result. This is WRONG! Imagine flipping a fair coin and calculating the chance it was not fair after every flip (this is similar to someone testing the significance of their ad text every day). If you flip your coin for long enough you will prove it is not fair, even though it is. I have made a coin toss simulator spreadsheet to illustrate this; it simulates 300 fair coin flips and also tells you if the test would be stopped at any point. I just ran it 10 times and got told the coin was unfair on 4 of them.
2. We run huge numbers of tests. This month I will run more experiments than the average PhD does during their whole course. Basically I am data dredging which might be good marketing, but it is not good science.
3. We are not trying to find an unchanging law of the universe. There is often no expectation that experiments should be repeatable (this is a very important part of the scientific method). My "Free delivery before xmas" adverts are not going to perform well in January no matter how many nines of significance I have when I test them.
4. We have no control group. Things are getting more scientific here with the introduction of AdWords Campaign Experiments and similar.

The closest we get to peer review is a flamewar

This is very different to how I imagine "good science" where experiments are clearly defined before they start (including what is to be tested and what counts as success) and other scientists will try to replicate or disprove the results.  Of course this doesn't happen all the time but the closest we get to peer review is a flamewar so I don't feel in a position to criticise.

I think there may be some hope for us in Bayesian Statisitcs.

Bayesian's take a different view on what probability is. I don't want to go into it too much here because it is complicated and I will get it wrong. I think the best way of explaining it is to say that frequentists (the opposing school of statistical thought) believe an experiment will help them tell what a particular value is whereas bayesians think that the results of an experiment can only influence what we believe about a value.

This all sounds a bit crazy, I'll try and simplify things by going back to the coin tossing example. A frequentist might say "the experiment shows the coin is not fair" but a bayesian would conclude "this experiment makes it more likely that the coin is not fair". Whether or not the bayesian will bet on the coin depends on what they believed about the coin before the experiment started.

The way that bayesian statistics can take into account prior beliefs is very useful for us. We can use it to get better results faster:

• "The conversion rate for this keyword is likely to be similar to the conversion rate for the rest of the ad group because the ad group is tightly grouped": We can make better and faster bid decisions
• "Last year this message worked really well so it is likely to work well again this year": Faster ad testing in the run up to seasonal events
• "The CTR for the new ad is not going to be about 12%": Faster ad testing

The prior beliefs thing is also one of the main reasons why bayesian statistics is not more widely used:

• It feels a bit dishonest to introduce the subjectivity of my prior beliefs into what I want to be objective decision making.
• Two people can reach different conclusions from the same data if their prior beliefs are different enough.
• It is often hard to quantify prior beliefs into the mathematical form necessary to use them in a calculation.

The folk over at Less Wrong have a lot to say on the subject of bayesian inference (and thinking/decision making in general). Parts of that blog are well worth a read.

The Bayesian analysis Journal has a pretty cool paper about predicting hitting performance in baseball which is not too far away from a model of CTR or conversion rate. I will put a more detailed analysis of this paper up on here some other time.

I'm meant to finish with a conclusion here, but I don't really have one yet. I have a long way to go before this nut is cracked (if I ever manage it). A good note to finish on might be to point out that whatever method you use to make your decision, decision making under uncertainty leads to disappointment (JSTOR link, also avaiable on Google Scholar).

Comments

• 

  Richard Fergie

  Posted 1 year, 3 months ago.

  How do I update the Google Docs spreadsheet?

• 

  PPC Analytica

  Posted 1 year, 3 months ago.

  Hi Richard,
  
  Looks like the Google docs spreadsheet is borked and not working the way I expected. You should be able to copy the formula into Excel if you want to see the results for yourself

Name

E-mail

Message

Send