I read this letter in the Edmonton Journal the other day, and wanted to share a bit of “plain language” statistics with folks since I deal with this type of work day in and day out. I’m really passionate about the field of market research and how it can shed light on so many issues, so I thought I’d take a moment and use my blog as an outlet.
Downtown should reflect Edmontonians, not image Katz wants
Publication: Edmonton Journal
Date: Wed Jan 19 2011
Page: A12
Section: Opinion
Byline: Joan Swain
Source: Edmonton Journal
A phone poll of 800 people is taken as indicating overwhelming support for the arena facility, while a written petition of 78,000 Edmontonians protesting the closure of the City Centre Airport was ignored by council.
There was a fourth question that was not offered on that poll which, no doubt, would have changed the results considerably.
That would be: “Daryl Katz wants a new arena, and wants to keep all the revenue it generates for himself, so should he build and pay for it on his own?”
Asking 800 people loaded questions is not representative of the thoughts of many citizens and should not be taken as such.
Joan Swain, Edmonton
I’ve heard this argument a few times in the media (the whole legitimacy of the phone poll vs. petition) and I’d like to shed a little light on the issue behind this.
A random sample of 800 Edmontonians has a margin of error of +/- 3.46 percentage points, 19 times out of 20. But what does that mean, exactly?
To put it simply, it means that if you were to replicate the study again – that is, ask the same questions of another, DIFFERENT 800 people in Edmonton, you would get roughly the same findings – either 3.46% less or 3.46% more than the result you received the first time.
To illustrate: the telephone survey found that 59% of Edmontonians support building an arena. If we asked another randomly selected 800 citizens the same question, we would expect to find that between 55.5% and 62.5% also agree. Thus, we can very confidently say that “the majority” of Edmontonians (more than half) support building an arena.
The airport petition, however, does not stand to the same level of scrutiny. Although it appears that many more people feel a certain way about the issue (in this case, the closure of the City Centre Airport), we don’t know whether these 78,000 folks (about 10% of the population) consist of EVERYONE who may feel this way or not. The “sample” was not random. Because people could seek out the petition and sign it if they were passionate about the issue, this 10% of the population could actually consist of every single person out there who believes the airport should stay open – which means that potentially, only 10% of Edmontonians are opposed to it closing. In other words, 90% may actually want to see it closed. Without a random sample, we don’t know how many people this 78,000 actually represents.
I also would like to address the comment about “loaded questions” – a loaded question is a question that uses words that can be interpreted in many ways (either positively or negatively). Having not seen the actual questions on the survey, I can’t comment on this; but I would like to note that the author’s proposed question: “Daryl Katz wants a new arena, and wants to keep all the revenue it generates for himself, so should he build and pay for it on his own” is actually a double-barreled (I’d even argue quadruple-barreled) question. It explores several issues lumped into one (wanting an arena, keeping the revenue for the arena, building an arena and paying for an arena).
Hopefully this helps everyone understand a little bit of basic research methodology and why organizations lean heavily on using random samples to help inform decision-making.