Erring on the margin of error.

1. Introduction

As most teachers of probability and statistics know, one of the most difficult concepts to convey to students is that of sampling error. Yet with the proliferation of the reporting of the results of public-opinion polls in the news media, students and the general public alike are exposed to this concept almost on a daily basis. In fact, when the authors recently accessed the Dow-Jones Interactive News Library (1) and typed in the words "public opinion poll," we registered nearly 60,000 "hits" in major newspapers and newswires for the period 1990 to the present. Often accompanying the discussion of the poll results is a statement describing the accuracy of the poll's estimates, which ordinarily reads something like, "The margin of error is 3 percentage points with a 95% level of confidence." (2) Because to many readers the meaning of this statement is fuzzy, the article sometimes attempts to clarify what the margin of error indicates about the poll's accuracy. However, from our experience, the attempted explanation is often completely in error--sometimes outrageously so.

In this article, we first briefly explain the correct way of interpreting the margin of error, which currently seems to be the fashionable way to explain in the media what statisticians routinely call sampling error. The margin of error concept is also gradually making its way into the professional economics literature (see Borenstein and Rose 1994; Brezis 1995) and into business/economics statistics textbooks as well (e.g., Anderson, Sweeney, and Williams 2002; Bowerman and O'Connell 2003). Next, we present some typical misinterpretations of the margin of error drawn from the news media that will prove interesting to both teachers and students and useful as classroom examples. Finally, we suggest some ways of using the media misinterpretations to teach and reinforce the correct interpretation of the margin of error (as well as of confidence intervals).

2. Interpretations and Misinterpretations

Suppose that an opinion poll taken of 1000 people (assume the sample has been selected randomly) finds that 60% of those sampled believe that economists are extremely boring. Suppose also that the margin of error (with 95% confidence) is reported to be 3 percentage points. The correct interpretation of this margin of error is that if repeated samples of size 1000 were to be taken, approximately (3) 95% of the time the sample proportions (p) would lie within 3 percentage points (0.03) of the true population proportion (p)--the proportion of all people who believe that economists are extremely boring. In other words,

Prob([absolute value of p - p] [less than or equal to] 0.03) = 0.95.

Conversely, only 5% of the time would the sample proportions be more than 0.03 away from the population proportion. But what kinds of interpretations are often given for the margin of error by the news media? Here are several examples:

* From a survey on the incidence of AIDS among clerics, an article described the poll's 3.5 percentage-point margin of error to mean that "if the same poll were conducted 100 times, 95 percent of those times the results would be no more than 3.5 percentage points higher or lower than the results of this poll" (Thomas 2000, p. A18) (italics added).

* In a telephone survey of Ohio adults on how trustworthy they believed real estate agents to be, the Columbus Dispatch reported that the survey had a "2 to 3 point margin of error and a 95 percent confidence level, meaning the results have a 95 percent probability of being true no matter the size of the polling group" (Columbus Dispatch 1998, p. 8J).

* A telephone poll of 1014 Texas adults on the death penalty reported a margin of error of 3 percentage points, which was interpreted as "each response can vary that much in either direction" (Hoppe 1998, p. 39A).

* In a Newsday report attempting to explain what a "1.8 rating-point plus-or-minus margin of error" meant for a Nielsen rating of news shows, the following explanation was given: "That means, among the many possible permutations of those figures, ... that last week's ratings could have been what they were said to be the week before, or that the week before's [sic] could have been what last week's ratings were said to be." (Kubasik 1990, p. 9). (We've read this explanation a number of times and are still not sure what it means.)

* In a Jakarta Post article (misinterpretations of the margin of error are not restricted to U.S. newspapers) concerning the results of a survey of 1000 people's opinions about human rights violations, the survey's claim of a 3.1% margin of error with a "reliability level of 95 percent" was interpreted in the following way: "This means that if a similar survey is conducted again ... there is a 95 percent chance that it will be of the same result--with only a 3.1 chance of error" (Jakarta Post 1998, n.p.).

* In a poll taken during the 2000 presidential race showing that Al Gore had a 6-point lead over George Bush, a margin of error of plus or minus 4 percentage points was said to mean that "there is a 95 percent probability the results of the poll are within 4 percentage points higher or lower than the finding" (Associated Press Newswires 2000).

* In a consumer survey regarding customer service issues that were the subject of labor negotiations, a telephone survey of 2400 customers of US WEST reported a margin of error of "plus or minus 2.0 percentage points." The interpretation given was that "if the survey had been conducted 20 times using the same approach, the results would be the same 19 out of 20 times" (PR Newswire 1998).

* In a poll of 729 voters in Arlington, Texas, regarding their positions concerning a proposed sales tax increase, the 3.7 percentage point margin of error was explained in this way: "If the poll were conducted in the same manner 100 times, in 95 of those instances the reported results would be within 3.7 percentage points of the results that would be obtained by interviewing every registered voter in Arlington" (Phillips 1991). (This interpretation errs by implying that in exactly 95 cases out of 100, the reported results would be within 3.7 percentage points of the population results.)

3. Some "Authoritative" Source Explanations

Interestingly, further search also revealed a number of authoritative sources that attempt to explain the margin of error but that also do so incorrectly.

* On a Public Agenda Online Web site, a posting entitled "A Guide to Sample Size and Margin of Error" has the following explanation for a "3 percent margin of error": "That means that if you asked a question from this poll 100 times, 95 of those times the results would be within 3 percentage points of the original answer" (italics added). The explanation continues: "For example, if 50 percent of a sample of 1,000 randomly selected Americans said they favor recycling laws, in 95 cases out of 100, 50 percent of the entire population in the U.S. would also have given the same response had they been asked, give or take 3 percentage points" (Public Agenda Online).

* In a Washington Post article (which ironically criticizes Marilyn vos Savant for a slip-up regarding the margin of error in her "Ask Marilyn" column), Richard Morin (1998, p. C5) quotes a former standards chair of the American Association for Public Opinion Research as the source for his explanation of a 3 percentage point margin of error: "What that plus or minus 3 percentage points means is that if the same survey were conducted under the same conditions 100 times, about 95 of the resulting proportions should be within 3 percentage points of the one that you now have" (italics added).

* The most surprising discovery came when we checked the home page of the Gallup Organization, which contains a link to a posting entitled "How Polls Are Conducted: Your Frequently Asked Questions Answered." (4) The posting provided the following explanation for a margin of error of plus or minus 3 percentage points:

Thus, if we find in a given poll that President Clinton's rating is

50%, the margin of error indicates that the true rating is very

likely to be between 53% and 47%.... To be more specific, the laws

of probability say that if we were to conduct the same survey 100

times, asking people in each survey to rate the job Bill Clinton is

doing as president, in 95 out of those 100 polls, we would find his

rating to be between 47% and 53% (Gallup Organization).

In short, the Gallup explanation--indeed, all three authoritative-source explanations above--commit the same error. (5) Instead of saying that in 95 out of 100 polls we would find the president's ratings to be within 3 percentage points of the population rating, the interpretation states that in 95 out of 100 polls we would find his ratings to be within 3 percentage points of the rating found in this sample. (6)

4. Some Classroom Applications

We strongly believe that examples drawn from the news media can greatly enhance and facilitate the effective teaching of statistics (see Becker 1998). We suggest below several ways that we have done this with margin-of-error misinterpretations.