Margin of ErrorPosted: November 19, 2008
Few weeks ago, I was at a friendly dinner party, discussing the upcoming elections, and specifically the results of recent voter surveys. One of the participants in the discussion said “I never pay attention to the error margins, since they apply to both candidates”.
I think he meant that if a specific poll said that 52% of the sampled voters preferred Obama and 46% preferred McCain, and the poll has a margin of error of 3%, then perhaps the “real” numbers are 55% for Obama and 49% for McCain, or maybe 49% Obama and 43% McCain, but it doesn’t really matter since the difference between them is constant.
This is of course, very false. For three important reasons:
- The margin of error is 3%, which means that the result of 49% for Obama and 49% for McCain cannot be ruled out. It is possible that Obama has no lead at all. It is important to understand that the 0% difference between the candidates is just at likely as the 6% difference the poll result actually show. There is no statistical way to differentiate detween these scenarios and both are just as real.
- 3% margin of error actually means that there are 95% chance that the “real” result is within 3% of the reported result. (Where “real” means what theresults would be if the entire adult population had been polled with complete accuracy). Remember that around election times, many polls are published. 5% of them have a bigger error than they report. How big? We have no idea.
- The reported margin of error is correct assuming that the sampling was perfect. Which means that no one refused to answer questions, no one lied, the questions were not worded or ordered in a way that caused bias, the selection of the sample was not biased, etc, etc. All these factors are likely to cause errors much larger than the theoretic sampling error, and what’s worse – we have no idea how big they can be and in which direction.
If you are really interested in the subject and not afraid of some mathematical notation, Terence Tao has a much deeper analysis of the subject.