Plug In the Numbers
In the wake of September's terrorist attacks, news outlets from the New York Daily News to the Los Angeles Times saw a trend. Americans were cocooning--we were pampering ourselves at home rather than going out.
Hold that, some of these same organizations reported weeks later--we were wrong: There is no trend.
Sure there is. The trend is for media--newspapers, TV, the Internet and (ahem) magazines--to report as many trends as possible. Usually the media support their claims with polling data for credibility. The numbers, however, also get them in trouble.
"Numbers for a lot of people, and [for] journalists in particular, have a mystical quality," says G. Evan Witt, president of polling firm Princeton Survey Research Associates Inc. in Washington, DC. "If it's true, it's precise."
Ah, that magic word: If. As 19th century British Prime Minister Benjamin Disraeli put it: "There are three kinds of lies: lies, damned lies and statistics."
Unfortunately, statistics frequently trip journalists up, says Michael Fumento, author of Science Under Siege (William Morrow & Co.) and frequent critic of the way the media handle numbers. "One mistake they make is projecting a trend from too few points," he says. Malicious misrepresentation or the inevitable clash of English majors with numbers? "It can be intentional or unconscious."
No wonder the public already discounts journalists, according to Lawrence T. McGill, a researcher at Princeton University and co-author of the influential 1995 Media Studies Journal article "The Citizen as Media Critic." "Typically, half the country sees the news as [poor to fair]," he says.
To which you may be saying "Duh!" But before you get smug about scribblers eating crow, ask yourself, What's an entrepreneur to do? You need some means of discerning which trends suggest real opportunities for your business and which are shoddily reported filler. You can't make decisions in a vacuum. Media critics and supporters suggest the following when you read about a trend:
- Gauge the source. "The best way to decide whether to believe the poll number is to look at where it came from," says Witt. He says prominent magazines such as Newsweek typically provide good filters.
Media critic and Slashdot.com columnist Jon Katz finds most social trends and analyses laughable, but respects data in business publications more than studies cited in the mainstream press. "The [business] audience is more demanding," he says.
- Investigate the facts. Read news items closely for information about how the poll was conducted. Harrison Rainie, director of the Washington, DC-based Pew Internet and American Life Project, suggests looking at two criteria: Who responded to the survey, and how representative was the sample? Web site surveys fail the smell test on both counts. Respondents have chosen to answer the survey, making them self-selecting (perhaps they have an unusual interest in the topic). They're also unrepresentative of the general population--in part because not everyone has access to computers.
Rainie also considers conditional questions a red flag. For example, beware the survey claiming people would be willing to pay X for a particular service they don't already have. "People often don't know about things they've never done," he says.
- Bear in mind when the survey was conducted. Individual events alter behavior, but rarely over the long haul. "Polls are a snapshot in time," says Witt. "Values and behavior don't change quickly."
- Dig deeper. Can't find much information about the survey in the news story? Go to the Web site of whatever group produced the survey. Frequently, groups will post their press release, which includes details the journalist left out.
"If you see just two poll numbers in a blurb, don't base a decision on it," says Witt. "At least spend the effort [to find out about a poll] proportional to how much you're going to rely on the info."
- When you find the source, ask the questions good journalists ask. Read "20 Questions Journalists Should Ask about Poll Results," written by former journalist Witt; it's available from the National Council on Public Polls (www.publicagenda.org).
- Beware the hockey stick. Among chary Silicon Valley types, market projections with a sudden uptick at the end are known as hockey sticks--because that's what they look like. "You'll see a fairly well-defined trend line, then suddenly there's a huge jump," says Fumento. "At the point of the jump it says 'estimated.' They're making it up." You'll do better basing business decisions on recent facts than future predictions.
- Don't be too skeptical. "The phenomenon of being shocked by public opinion results always happens when it comes to surveys," says McGill. So don't discount a trend just because it doesn't fit with everything else you know. If everything else about the survey checks out, believe it. It may be telling you vital information that could save your business. And that, after all, is why you read the news.
- ew Internet and American Life Project
(202) 296-0091, ext. 463, www.pewinternet.org
- Princeton Survey Research Associates Inc.
(202) 293-4710, ext. 8, email@example.com
For reprints and licensing questions, click here.