Gun Rights versus Gun Control: On the Need to Understand Sampling Error in Reporting Statistics

Based on a recent survey by the Pew Research Center, we can be 95% confident that the actual gap in American’s attitudes toward protecting gun rights versus controlling gun ownership is as small as 0% and as large as 12%.

Depending on where you stand on the issue, things might not be as bad as was reported (no gap between gun rights and gun control!) or they could be much better than what was reported (12% more favor gun rights than gun control!).

These important caveats got lost amid all of the controversy over this Pew Research Center survey question about American’s attitudes toward protecting the right of Americans to own guns versus controlling gun ownership. (I wrote about this controversy here and here.)

Pew Research reported that in 2014, the scales tipped in favor of protecting gun rights (52% of respondents) over controlling gun ownership (46% of respondents). That result, as well as the trend in responses over the past 20 years, is displayed graphically below.

Gun rights supporters rejoiced and gun control proponents lamented – the latter even going so far as to petition Pew Research to change the wording of this question. But as Peter Berger said, “the first wisdom of sociology is this—things are not always what they seem.”

We are bombarded so constantly with statistics like these that we often forget (or for some, do not know in the first place) that the results of surveys based on samples are estimates (sample statistics) of the true underlying values in the population being observed (population parameters). Ideally, a sample statistic will match the population parameter, but in the real world that is basically impossible 100% of the time.

Having some idea of how big the gap is between a sample statistic and population parameter is vital to interpreting results like those reported by Pew Research. Unfortunately, in this particular graphic, what is not reported is the survey’s margin of (sampling) error.

However, if you go to the section of the Pew Research site that gives further information about the survey,  you will see that for the total sample size of 1,507 respondents, the margin of error is +/-2.9% — basically 3%.

So, looking at the responses above, what we really know is that the proportion of the population in favor of protecting gun rights could be as high as 55% and as low as 49%, and the proportion favoring controlling gun ownership could be as high as 49% and as low as 43%. (Assuming no other biases or errors in measurement or methodology – about which more in another post.)

So, the actual gap in American’s attitudes on this question ranges from 0% to 12%. Things could be either not as bad as gun control/safety advocates think OR much worse! And things could be either worse than gun rights proponents think OR much better!

Moreover, we are only 95% confident that the sample statistic is +/-3% of the actual population parameter. That means there is a 5% chance that the statistic is off from the actual underlying population parameter by more than 3%. In other words, in 95 out of 100 samples of the same size that are drawn, the survey estimate will be +/-3% of the population parameter. 5 times out of 100, it will deviate more than that.

All is not lost, however. There are more advanced statistical methods we can use to determine the likelihood that a difference we see in survey statistics is “real.” But to just say X% of people believe this and Y% of people believe that based on a survey is very imprecise indeed.

The Pew Research Question “about the survey” section also concludes, “In addition to sampling error, one should bear in mind that question wording and practical difficulties in conducting surveys can introduce error or bias into the findings of opinion polls.” I will take up this issue in a future post.

ADDITIONAL NOTE ON THE SAMPLING DISTRIBUTION OF THE MEAN AND THE CENTRAL LIMIT THEOREM:

Sampling error is based on the central limit theory (nice overview from the Khan Academy), which holds that the sampling distribution of any statistic will be normal or close to normal if the sample size is sufficiently large – even if the values in the population are not normally distributed. A normal distribution is sometimes called a “bell curve.” This is fairly well represented in the following graphic from the Web Center for Social Research Methods:

Source: http://www.socialresearchmethods.net/kb/sampstat.php