What is the “Margin of Error”?

This posting is a very good explanation of margin of error and should be mandatory reading for everyone.  It was originally posted in a blog called, “Math, Eloquently.”

What Is the “Margin of Error”?

Newspaper and television reporting on election polls often mentions the “margin of error” of a specific poll.  In fact, if the poll doesn’t state a margin of error, it’s not considered very reliable, and it probably won’t be reported.  Statements like the following are common:  “In this recent poll, candidate A is preferred by 44% of those polled, while candidate B is preferred by 42%.  But since the margin of error was ±3%, this is a statistical dead heat.”  Most mathematicians would consider that poor reporting, and most people don’t really know what that “margin of error” means.  So let me explain.

I will not go into deep statistical detail here, but I hope to give you enough insight to interpret polls a bit better.  So let’s consider an example situation:  you live in the city of Electopolis, which (conveniently) has a voting population of exactly 1,000,000 people.  If you could read everyone’s mind about next month’s election, you’d know that 490,000 people (or 49%) plan to vote for candidate A, 450,000 people (45%) for candidate B, and 60,000 (6%) haven’t made up their minds yet.

But, of course, you can’t read everyone’s mind.  Yet you’d like to have some idea of who might win this very important election.  So you decide to call a bunch of people at home and ask them who they’re planning on voting for (that is, you’re going to take a poll).  After calling 10 people, 6 tell you they’re going to vote for candidate A, 3 for B, and 1 hasn’t decided yet.  Wow!  Candidate A has a 30% lead in your poll!  She will clearly win, right?  Are you pretty confident about that prediction?  If I also called 10 random people, are you pretty sure I’d get the same result?

When pressed, you’re not so sure, are you?  It does seem possible that the 10 people you happened to call might not accurately represent all 1,000,000 people in your city.  In fact, while it seems unlikely, it does seem at least a little bit possible that if you did your little calling-10-people experiment often enough, then on some of those experiments, the results would be completely unrepresentative of the population.  For example, it turns out that if you did the experiment 100,000 times, then about 34 of those times, all 10 people would happen to ones planning to vote for candidate B!  That’s a pretty small chance, of course.  But how do you know this isn’t one of those times, and your results are just completely useless?

Well, for most people, the solution is obvious:  call more people.  I think almost everyone has the right intuition here;  that is, if you call 100 (or 1000) people, the chances are much better that your sample results will closely match the actual percentages from the population of the city.  In fact, if you do the calculations, you’ll find that it takes fewer calls than you might think to be reasonably sure your poll percentage matches your population percentage somewhat closely.  1000 is plenty, it turns out, and if you ask another 1000, it won’t improve this accuracy much.

So here’s the first (very rough) answer to the question posed in the title:  The margin of error measures the effect of various sample sizes on the likelihood of a match between your poll results and the true percentages in the population.  This is a crucial point:  there are many, many factors that influence the error in a poll, and I’ll discuss some of these below, but the reported “margin of error” measures only the effects of sample size.  In other words, if Polly Pollster’s margin of error is smaller than Quincy Questioner’s, that can only mean that Polly asked more people than Quincy.  This emphatically does not mean that Quincy’s poll is less accurate—Quincy might have better polling techniques that reduce other possible sources of error.  The only way to know is to look at their past poll results.

Okay, so the margin of error measures the effect of sample size on the accuracy of a poll.  Let’s make that more precise;  what does the number reported as that margin of error mean, exactly?  Well, let’s continue our example.  It turns out (you can look it up here) that if you have a sample of 1000 people, your margin of error is going be very close to 3%.  Let’s use the poll results from the first paragraph (44% for A, 42% for B, 14% undecided, margin of error is ±3%) in our Electopolis example.  We now know that the 3% comes from having interviewed about 1000 people.  But what does the 3% mean?  Since the difference between the candidates is only 2 percentage points in the poll, does that mean we have to remain clueless about who’s ahead?  The answer is no.  It is still more likely that candidate A is winning.

Here’s why.  Choosing 1000 people randomly might lead you astray if you happen to randomly pick a non-representative sample whose percentages don’t match up with the population’s percentages.  But surely that’s not the most likely result.  The most likely result is that the sample matches the population.  If you did the calling-1000-people experiment over and over and over, the results would cluster around the true percentages, except that every now and then, chance dictates that one or more of the numbers will be off—usually only a little, occasionally a lot.  How far off?  And how often would that happen?  Aha!  That is what the margin of error measures.

Before continuing, you have to know that in all of these polls, there’s one number that they haven’t told you.  That number is usually (often enough that you can assume it) 95%.  It’s called the 95% confidence level.  The idea of a confidence level should make some sense to you since we’ve been talking about how confident we are that our poll numbers reflect the actual numbers (I should really write “actual”  numbers using quotes here—more on that in the first note on technical details below).  The 3% margin of error means that the number reported in the poll will fall within a 3% stones-throw of the true percentage (answering ‘how far off?’) 95% of the time (answering ‘how often?’).

So that brings us to a much more refined answer to the question posed by the title.  In our example, 49% of the population plans to vote for candidate A.  The ±3% margin of error means that if you do the calling-1000-people experiment 100 times, then in about 95 of those times, we’d expect a score between 46% and 52% (that’s 3 percentage points on either side of 49%) for candidate A.  And on those occasions when we do happen to get a score outside of that range, probability dictates that most of those scores will not fall very far outside that range.  Every once in a while, though, crazy things will happen, and we will indeed get a score quite far away from the true percentage.

Let’s look at the poll results again.  The 44% score for candidate A is apparently a little bit of a fluke—it falls outside (though not too far outside) the ±3% range.  And by chance, candidate B’s 42% score is also low, though (barely) within the margin of error.  But when you hear the results of the poll, your reaction shouldn’t be “the margin of error makes this too close to call”.  Rather, your reaction should be “This does seem to be a pretty close election, but the most likely state of the world is that candidate A is leading by a little bit”.

Technical note #1:  the “actual” numbers?

True statisticians might have cringed when I discussed comparing the poll results to the actual numbers.  In their view, there’s simply no way to know the actual numbers without asking every single person.  Even the election result won’t reveal the actual numbers at the time of the poll, since the undecided people will decide, and since some people will change their minds.  So to a statistician, the best available estimate of the actual percentages are the percentages from the poll.  Thinking about this in the way described above will help your understanding of poll results immensely, but technically, 44% for candidate A with a ±3% margin of error (at a 95% confidence level) really means that the pollster expects that 95% of identical polls (using a different, but equally-sized random sample) would put candidate A’s score within 3 percentage points of 44%, not 49% (because there’s no way to discover that 49% number).  To the pollster, then, the margin of error is a measure, derived from the sample size, of how closely they would expect the results of similar polling experiments to match theirs.

Technical note #2:  small numbers.

In our example, 6% of the population of Electopolis was undecided.  If you think carefully about the chances of getting that 6% wrong in a 1000-person poll, you’ll notice that it’s harder to miss the 6% mark by a lot—say, 10% too high—than it would be to miss a 49% mark by 10%.  There simply aren’t that many undecided people to disproportionately choose in your random choice.  So at smaller reported percentages in the poll, the margin of error decreases.  In this table, for instance, look at the right-most column, which represents margins of error for 1000-person samples.  The 3% we’ve been working with shows up at the bottom, for a reported 50% score in the poll.  But in the middle of that column, we can see that if the poll reports only 15%, the margin of error is just over 2%.  So the 14% we found in our example poll for the undecided voters is actually extremely unlikely, given the actual 6% value.  95% of the time, that score should be between 4% and 8%—2 percentage points on either side of 6%, because the margin of error near 15% is ±2%, not ±3%.  The reasoning in the example poll is that it was very unlikely that A’s score would be 5% off, and it was somewhat unlikely that B’s score would be 3% off, and it was even more unlikely that they would both turn out low (if you missed some A-voters in your random sampling, it should be more likely that you found B-voters instead of undecided voters simply because there’s more of the B-voters).

Technical note #3:  other sources of error.

This should probably not appear in a technical note because it’s so important.  But putting it in the text above would have distracted from the main point.  We have seen that the margin of error measures the amount of possible discrepancy, due to sample size, between a poll’s resulting figures and the presumed actual population figures.  Crucially, then, the margin of error does not take other possible polling errors into account.  There are many such possible errors, all discussed elsewhere on the web more extensively than I can here.  But any list of reasons that it’s so hard to conduct an accurate poll includes:

1.  How can you be sure your sample is really random?

Do you call people during the day?  Then you’ll only reach people who don’t have an out-of-the-house day job, such as night-shift workers, stay-at-home parents, the unemployed, etc.  It seems highly plausible that this group of people would have different voting patterns than people who do have the out-of-the-house day jobs.  Do you even use the telephone at all?  The only listed numbers are land lines, not the cell phones that some people (especially young people who move a lot) use as their only phones, and you therefore risk missing a large demographic.  This is a huge topic worthy of dissertations.

2.  Is your question phrased properly?  Or is it leading?

“Do you support redrawing the City’s boundaries beyond their historical ones?” could sound a lot different from “Do you support increasing the City’s tax base by annexing Edgeville?”

3.  Are people telling the truth?

4.  Are the people who refuse to answer the survey question likely to hold similar positions?

Accurate polling is tough, and pollsters themselves will be the first to tell you that.  All sorts of factors can skew a poll away from divulging the true sentiment of the population, and only one of those factors is the sample size.  However, that is the only factor whose effect is completely measurable, and that measure is exactly what we mean by the “margin of error”.

Advertisements

Tags: , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: