And People Wonder Why I Don’t Trust Surveys
DUBLIN – When anti-porn activists talk about the pervasiveness of online porn, one of the statistics they frequently cite (typically without attribution) is a claim about the average age at which people are first exposed to porn.
Depending on who is making the claim, this number may be age 11, eight, the quite specific 6.6 years old or (my personal favorite) an average expressed as a range, like “8-11 years old.”
What can’t be argued plausibly is all the numbers listed above are correct at once. The question then becomes where and how did the person making any given claim come up with his/her number?
The answer, almost invariably, is the number comes from some form of survey. Some are surveys conducted as scientifically as a survey can be, all things considered, while others are online polls open to all comers who stumble across them, with no safeguards to prevent the same individual from submitting several responses.
While anti-porn activists are never going to be at a loss to find survey data that backs up their most alarming claims about the consumption of pornography, those of us who are less apt to think society is headed down the toilet with porn as the proximate cause of the spiraling occasionally can find surveys supporting our less apocalyptic viewpoint, as well.
For example, a new survey conducted by the Irish polling firm Red C suggests that within Ireland, at least, the average age of first exposure to porn is considerably older than the Porn Panic crowd asserts.
“The average age that people claim to have first consumed porn is 17 years old,” reported NewsTalk.com, which commissioned the survey, “but one in three say their first experience was before turning 16.”
So, does this survey mean the Irish are first exposed to porn much later in life than those who took surveys elsewhere?
The only honest answer to this rhetorical question is very unsatisfying: Who knows?
If you think I’m being uncharitable concerning the reliability of survey results, I’m sure my skepticism can be countered by some of the people most responsible for the prominence of polling in modern politics. I mean, we’d all bet our bottom dollars someone like Humphrey Taylor, the chairman emeritus for the Harris Poll, would wax poetic about the terrific accuracy and unquestionable reliability of survey results, right?
Well, here’s something Taylor wrote almost 20 years ago about polling, the media and the reliability of survey data (with emphasis added):
“On almost every occasion when we release a new survey, someone in the media will ask, ‘What is the margin of error for this survey?’” Taylor wrote. “There is only one honest and accurate answer to this question — which I sometimes use to the great confusion of my audience — and that is, ‘The possible margin of error is infinite.’”
An infinite margin of error? How can this be?
As Taylor explained, a survey or poll often tells us more about those who wrote the questions than those who answered them.
“In the real world, ‘random sampling error’ — or the likelihood that a pure probability sample would produce replies within a certain band of percentages only because of the sample size — is one of the least of our measurement problems,” Taylor observed. “The main problems of survey measurement, or more accurately mismeasurement, include: the sample design (for telephone surveys, how the numbers were selected and how the individuals are selected within the household); the non-availability problem (are people who are available different on the variables we are measuring than the people who are not available?); the refusal problem (is the refusal rate different on the particular variable we are measuring?); question wording; question order; deliberate, or unconscious, lying or false reporting by respondents; inappropriate or inadequate weighting of the data.”
This paragraph explains quite efficiently why surveys aren’t to be trusted, or not without rigorously examining the methodology used, at any rate. It also explains why surveys conducted by agenda-driven people and organizations come up with numbers so radically different from those collected by their ideological opposites.
This isn’t to say the surveys referenced by anti-porn groups are “wrong” and those contradicting them are “right.” It is to say, however, the surveys cited by both groups are likely to be as useful, accurate and reliable a measure of public opinion as having a psychic read your palms to determine how long you’re going to live.
If inclined to trust the outcome of surveys and polls at face value, great; knock yourself out. All I ask is for people in the media to stop calling these stupid, hopelessly biased, push-poll surveys “pornography studies” or “science.”
This seems like a modest request, but it’s one I don’t expect to be honored, ever.
Don’t worry, this isn’t mere pessimism on my part; it’s a scientific conclusion. I arrived at it through an extensive, carefully-considered survey of every middle-aged white man currently sitting in my chair.