GARBAGE IN, GARBAGE OUT
INTERVIEWER FALSIFICATION IN SURVEYS
In computerese, it’s “Garbage In, Garbage Out” which, of course, means that the integrity of data collection is equally as vital as the methodology employed. Hence, without safeguards, checks or accountability during the crucial, often-overlooked stage of the actual interviews (the data collection)--“science” is bound to become “art.”
And the reason why (particularly in surveys undertaken in the Philippines) is that the anonymity of the respondent and “the confidentiality of the data supplied,” must be respected, being “the prime concern to all reputable survey organizations” in their code of ethics; thus, how can the public confirm the “raw data” interviewers collect from respondents whose identities must not be divulged and whose survey responses must remain confidential?
So, it’s not just after all the highly technical aspects of, say, the “randomness of the sampling base and the framing of the questions” (already the subject of endless debate by professionals and the academe) that are to be regarded as critical; for there remains a question that no doubt is more significant to Filipinos--the “integrity of the surveyors,” or the interviewers who go out in the field to record and collect the “raw data” (the “Garbage In”) to be encoded.
In short, since the respondents are to remain anonymous, is there a “paper trail” or audio recording (even hi-tech laptop-collected inputs) to confirm if these interviewers faithfully wrote down the “face-to-face” response (assuming interviews were really conducted)?
If these “raw data” are conveniently unavailable for public scrutiny, are Filipinos putting too much trust in “the integrity of the surveyors”? In fact, to pollsters, it would seem that, aside from respondent anonymity, the debates on the methodology are clearly a welcome diversion to what ought to be a livelier, more pertinent talking point--“Interviewer Integrity.”
The title of the recent Pulse Asia survey speaks for itself: “Trust Ratings.” The survey indicates that Filipinos do not even trust 100% the Chief Justice, NAMFREL and COMELEC. If so, why would they suddenly--inexplicably--trust 100% the interviewers that pollsters hire?
To wash their hands with, pollsters have already coined a term--“INTERVIEWER FALSIFICATION,” defined by the American Association for Public Opinion Research (AAPOR) in “Interviewer Falsification in Survey Research: Current Best Methods for Prevention, Detection and Repair of its Effects” (2003) (http://www.aapor.org/) as “the intentional departure from the designed interviewer guidelines or instructions unreported by the interviewer, which could result in the contamination of data. ‘Intentional’ means that the interviewer is aware that the action deviates from the guidelines and instructions.”
“Falsification” includes, among others: “Fabricating all or part of an interview--the recording of data that are not provided by a designated survey respondent and reporting them as answers of that respondent.” (NOTE: “Curbstoning” is another similar polling industry term, defined as “falsifying information about a household.”)
Under the heading of “Prevalence of Falsification” in the same AAPOR paper cited above: “Interviewer falsification has long been recognized in survey research, both in the published literature and in the professional practices that have been developed over the years to prevent and detect it …”
So, if Interviewer Falsification “has long been recognized in survey research” to be widespread, why have Filipinos treated survey results as “gospel truth”? Is this not a classic case of the “blind faith” of idiots “taken for a ride”?
If there are “flying voters,” mind you, there can also be “flying respondents.” If voters (or judges, too?) can be bought, can interviewers be the exception? Or are they what Madison (The Federalist No. 51) was looking for and was never able to find--“angels”?
Who, by the way, were these nameless interviewers Pulse Asia hired to conduct the recent survey? The Report does not mention even a line about them. What if some (if not, all) of these interviewers (acting alone, perhaps, and without the knowledge of Pulse Asia, but conniving or conspiring with others) intentionally distorted the response and entered the opposite of (or the contrary input to) what they heard?
The danger of “trust misplaced” here is that “If information is power, then political information is political power”--Claude Emery “Public Opinion Polling in Canada” (1994), (http://www.parl.gc.ca/information/library/PRBpubs/bp371-e.htm).
Survey results of public opinion polls are, no doubt, generally viewed as “accurate and scientific.” Hence, pollsters--although “non-partisan” and “unelected”--wield enormous “political power”; survey results, in fact, can have a significant influence not only on policy-making (survey on hunger) but on the conduct of election campaigns as well (survey on ranking).
To Emery, however, “public opinion surveys are blunt instruments of prediction and are susceptible to many forms of error,” since “polls can present an opportunity for deliberate misrepresentation or connivance by those who publish survey results.”
“One should not be blind,” he adds, “to the possibility that interviewers occasionally ‘make up their interviews or to substitute easy to contact individuals for members of the sample group they were supposed to reach but found it difficult to do so,’” and concludes with the caveat that, “pollsters have become a dangerous new breed of political advisor.”
To underscore: “polls can present an opportunity for deliberate misrepresentation or connivance” and “interviewers occasionally ‘make up their interviews.” In the Philippines, that “opportunity” to occasionally “make up their interviews” is certainly greater (or to be expected) than in Emery’s Canada.
Polls, to repeat, greatly influence public opinion. Coup plotters, in fact, never tire of repeating survey results to justify their adventurism; hence, the manner in which pollsters operate at ALL stages of a survey should be “open and transparent” to the “public” they claim to serve.
The World Association of Public Opinion Research (WAPOR) Code of Professional Ethics and Practices provides under Sec. 19, Rule II (http://www.unl.edu/WAPOR/ethics.html) (NOTE: SWS is a WAPOR member):
“19. Every complete report on a survey should contain an adequate explanation of the following relevant points …
“(k) (where the nature and the research demands it) the characteristics of those employed as interviewers and coders and the methods of their training and supervision;
“(l) a copy of the interview schedule or questionnaire and instructions.”
In the case of the Philippines, the main concern should be focused on the reliability or “the characteristics of those employed as interviewers,” for the “Garbage In” the interviewers record, collect and encode are the critical “inputs” which are to be made the basis for the survey results later.
To allow the public to evaluate the credibility of the results, pollsters and the media, in particular, that circularize survey results (to avoid being dragged into what may turn out later to be a massive fraudulent scheme of disinformation), should be required to submit, simultaneously, together with the survey results for public scrutiny: (1) the confirmatory data (“paper trail” and audio recording, even laptop inputs) that interviewers record and gather during the survey (without naming or jeopardizing the privacy of the respondents); and (2) the more important biodata or “characteristics of those employed as interviewers” WAPOR requires “a survey should contain an adequate explanation” about.
The argument that Pulse Asia and SWS certainly do not want “to lose their business” by “deliberately skewing the results” and taking “the science out of their surveys” skirts the issue altogether and does not resolve the gnawing question of interviewer accountability or credibility.
Fact is, the management or incorporators of these firms do not conduct the surveys themselves; they employ (or hire and pay) interviewers to gather the “raw data” for them. And it is this particular “outsourcing” aspect of their “business” that the public needs (or demands) to know more about and to be assured and convinced that data reported is the “face-to-face” answer of the “designated survey respondent,” and have not been made up or falsified by the interviewer.
Pollsters must bear the responsibility of preventing and detecting intentional interviewer falsification without having to be required or compelled to do so, although little attention, at least publicly by the media, has been given to the extent (or even the prevalence) of falsified data collection. (see Johnson, Parker, and Clements, “Prevention and Detection of Data Falsification in Survey Research,” (2001) at http://www.srl.uic.edu/Publist/Srvrsch/2001/01v32n3.PDF)
Note it well, nonetheless, that pollsters forecasting the wrong set of winners in Philippine elections, even by substantial margins, have nothing to worry about; they can always blame the “error” later on massive “cheating” and “vote-buying,” even on “Garci.” Anyway, by then, pollsters may have already achieved their primary “business” goal--“Conditioning.”
Meanwhile, absent the confirmatory data on survey response of individual respondents and the information on the interviewers that WAPOR recommends, Pulse Asia and SWS survey results should be regarded--for the sake of the Republic--as “GARBAGE OUT”!
And the “public” that Pulse Asia in the Preface of its Report wants to be informed concerning “Filipino perceptions, opinions, sentiments and attitudes relating to current developments here and abroad” should likewise be made aware of this festering sore, oozing from the reality and prevalence of “INTERVIEWER FALSIFICATION” during survey research soon.
The AAPOR paper cited warns: “Survey researchers have an obligation to truth in data collection … If interview data do not reflect the answers or characteristics of respondent, but rather are the invention of the interviewer, data integrity is directly affected.” Thus, unless convincingly shown that contaminated data have not been encoded, pollsters and, particularly, the media should refrain from disseminating tainted survey results that may well decide the fate of this Nation.
