“If you want to measure change, don’t change the measure.” – Attributed to sociologist Otis Dudley Duncan
I was surprised recently to read that the Greenwich Council Against Gun Violence has started a petition on the website Change.org to pressure the Pew Research Center to ask a new question about gun rights and gun control. The full text of the petition appears below. To date, it has only received 541 or the 1,000 signatures necessary to be delivered to Carroll Doherty, the Director of Political Research at Pew Research, despite being promoted by the Campaign to Stop Gun Violence and others.
The heart of the matter for the petitioners is their belief that “the premise of the question [that Pew currently asks] is flawed. It presents a false choice between regulating firearms and protecting Second Amendment rights.”
The question Pew has asked in 25 surveys since 1993 is: “What do you think is more important –to protect the right of Americans to own guns, OR to control gun ownership?”
As this graph highlights, the 2014 results showed that more respondents said it was more important to protect gun rights than to control gun ownership. That there were no complaints about this question for all the years it showed more support for gun control than gun rights – that trend peaking in 1999, the year of Columbine, when more than twice as many favored controlling gun ownership (66% to 29% for gun rights) – suggests that it is not the question that most concerns the gun control movement, but the answers.
To be sure, there are bad ways of asking survey questions. I won’t review them here, but you are likely to see them next time you get a newsletter from your political representative that includes a “constituent survey.”
Among proponents of gun control who were obviously much aggrieved by the results of this survey, much has been made of the fact that Pew’s Doherty told Mother Jones, “Is it a perfect question? Probably not.”
This statement was somehow turned into an admission by Pew that the question is “flawed.” For example, ironically, by “Media Matters for America” – a group “dedicated to comprehensively monitoring, analyzing, and correcting conservative misinformation in the U.S. media.” Apparently they do so – in this case, at least – by spreading misinformation. Media Matters took Doherty’s comment and wrote about it under the headline, “Pew Admits Flaw in Poll Being Used to Attack Stronger Gun Control.” As you can see in the screen cap below, Media Matters writes that Pew “has acknowledged that the question was flawed.”
Huh? Far from it! Having been involved in scientific survey research since before I started graduate school over 20 years ago, I can assert with confidence that saying a survey question is “not perfect” is far from saying it is “flawed.”
Indeed, included at the very end of the Mother Jones story, but scarcely mentioned by those supporting gun control, is this:
“Despite the limitations of the particular language of the question, Doherty says, ‘tracking it over two decades—asking the same question with identical wording—tells you a great deal about changing public sentiment on this issue.’”
Absolutely correct. In fact, around the same time I heard about the Change.org petition, I read a blog post by one of the most respected historical sociologists of American society and culture, Claude Fischer (Natalie Cohen Professor of Sociology at my alma mater, UC-Berkeley). In his entry on “Surveying Change,” Fischer explains that to study change in attitudes over time, you have to have comparable survey samples and “the questions asked should be the same.” He quotes another distinguished sociologist who specialized in survey research, Otis Dudley Duncan (who has actually weighed in on guns previously, by the way ):
“If you want to measure change, don’t change the measure.”
There are many ways to ask about people’s views on guns, on gun rights, on gun control, on gun violence, etc. The Pew Research Center’s question is not perfect. No single survey question in and of itself is perfect. But its imperfection does not make it flawed or useless. To the contrary, the fact that it has been asked repeatedly over time makes it extremely useful in tracking changes in Americans’ attitudes toward gun rights and gun control over time.
FULL TEXT OF CHANGE.ORG PETITION
The Pew Research Center is “one of the least biased, most reliable polling organizations in the country” according to slate.com. Because of its authority, Pew can shape the debate on issues of importance. So when a Pew survey asks a question in a biased manner, it matters. That’s the case with Pew’s question about gun laws, and why we’re asking them to add a new, more balanced question.
To better track public opinion on the gun issue ask a new question: “Which do you think is more important: make it harder for criminals, domestic abusers and the severely mentally ill to get guns or protect the right to own guns with minimal restrictions?”
In December, Pew reported that for the first time in 20 years, Americans believe it is more important to protect gun rights than it is to control gun ownership. The report received widespread media coverage because it stands in stark contrast to the crisis of gun violence that claims 30,000 lives each year. But the premise of the question is flawed. It presents a false choice between regulating firearms and protecting Second Amendment rights.
Pew’s director of political research, Carroll Doherty, admitted as much when he told Mother Jones “Is it a perfect question? Probably not.” Experts on gun policy are critical of the question. Daniel Webster, director of the Johns Hopkins Center for Gun Policy and Research told Media Matters “I could not think of a worse way to ask questions about public opinions about gun policies.”
The Pew survey question (“which is more important: protect gun rights or control gun ownership?”) does gun violence prevention a disservice by suggesting there is less support for common-sense gun laws than there actually is. That’s why we’re petitioning the Pew Research Center to add a new question that accurately measures public opinion on the importance of strengthening gun safety laws versus protecting easy access to firearms.
Pew’s question creates a false perception of gun safety advocates, who are not trying to “control gun ownership” but rather want reasonable regulations that keep firearms out of the hands of dangerous people. We also believe in balancing public safety against individual rights by restricting highly lethal weaponry such as large capacity magazines and firearms designed for military use.
Asked in a way that appropriately frames the debate, Americans support gun safety laws in far higher proportions than the Pew survey suggests. In a survey conducted for Everytown for Gun Safety, 63 percent of voters believe it is more important to make it harder for dangerous or severely mentally ill people to get guns than it is to protect the right to own guns.
Politicians follow public opinion – we need to make sure they’re hearing the right message on gun violence prevention!