Caveat on So Called “Scientific Methodology” on Surveys

Surveys are not only used during elections. Surveys are used daily by most marketing and advertising departments of large corporations and conglomerates. Branding is based on preferences by customers or brands are created according to the needs of the market.

In this 2016 election season, there are surveys that tout their “methodologies”. These may sound authoritative but once dissected they cannot stand up to close scrutiny.

This observation and insight by a Statistics graduate of the University of the Philippines deserves careful perusal and analysis.

Days are numbered for the much awaited 2016 National and Local Election. Surveys are here and there saying that this are the result of the surveys they made, etc. And so, after delivering the results to the people, many were just contented with the result while there are some who reacts and disagree and point out their reasons.

This is why David Yap, a statistician, and graduate from the University of the Philippines School of Statistics, became viral after stating his points about the 'Bilang Pilipino SWS Mobile Survey'.

As per this guy,

I am a statistician. I am a graduate of the University of the Philippines School of Statistics. I do not approve of the methodology of the Bilang Pilipino SWS Mobile Survey. More importantly, I detest their use of Statistical methodologies to lend credence to their obviously flawed methodology. It obfuscates its lack of credibility by invoking Statistical methodologies that are not easily understood by many Filipinos. In my opinion, it is a blatant and deplorable misuse of the Science of Statistics.

(Hear more from him by reading his entire Facebook post below about the issue)

Few hours after he posted his message on his Facebook wall, it went viral and as of the moment, it now reached 1000 shares and still counting.

This is what David Yap posted on his Facebook wall about the 'Bilang Pilipino SWS Mobile Survey'. Kindly read.
I am a statistician. I am a graduate of the University of the Philippines School of Statistics. I do not approve of the methodology of the Bilang Pilipino SWS Mobile Survey. More importantly, I detest their use of Statistical methodologies to lend credence to their obviously flawed methodology. It obfuscates its lack of credibility by invoking Statistical methodologies that are not easily understood by many Filipinos. In my opinion, it is a blatant and deplorable misuse of the Science of Statistics.

The Bilang Pilipino SWS Mobile Survey claims that it has a margin of error of +/- 3%. The implication here is that the results of the survey are off by at most three percentage points. People are led to believe that the survey has accurately estimated the lower bound and upper bound of the vote share of each candidate. Given their latest results this would suggest that Sen. Poe has a range of 31% to 37% and Mayor Duterte has a range of 28% to 34%. Here is an explanation as to why I find it absurd.

Students of Statistics know that the margin of error in sampling is often computed as the reciprocal of the square root of the sample size. The Bilang Pilipino SWS Mobile Survey had a grand total of 1,200 possible respondents. SWS has not divulged the number of actual respondents of their latest survey so let us use the sample size in the March 22 survey - 806. If you take the reciprocal of the square root of 806 you'll get 3.52% (Right of the bat SWS rounded their margin of error down - but let us not quibble over relatively trivial details such as this). Notice that the margin of error does not use the size of the population as a variable (It does not matter if the voting population is 1 million of 50 million). It may come as a surprise to many of you that this isn't the crux of the issue.

The crux of the issue is that the validity of the aforementioned margin of error depends on the sampling methodology. The margin of error will only hold if the survey is conducted using an unbiased methodology. Put simply, the margin of error will only hold if the survey results are generated from an unbiased sample. What exactly is an unbiased sample? An unbiased sample is one wherein every element of the population has an equal chance of being selected. I repeat: An unbiased sample is one wherein every element of the population has an equal chance of being selected.

Here's the rub: in order for the margin of error to be credible, the underlying sample has to represent the entire Philippines. How could it represent the entire Philippines when millions upon millions of validated voters were not given a chance to be selected?

Herein lies the test as to whether or not the SWS Bilang Pilipino Mobile Survey should be believed. Herein is a simple question that would tell us whether or not we should lend credence to the results being promoted by this survey: Was every validated Filipino voter given an equal chance of being selected?

According to the SWS Bilang Pilipino primer the survey uses a 'nationally representative sample of 1,200 validated voters'. The question here becomes: Did SWS draw its 1,200-strong sample from the entire roster of validated voters? After all, each and every validated voter has to be given an equal chance of selection. There is no indication, at all,that SWS sampled from the entire voter registry (As a segue ask yourselves: would COMELEC have given SWS, a private entity, free reign over the ENTIRE voter registry? Hmm.).

This suggests that SWS did NOT give all validated voters a chance to be selected. If this is indeed the case then the SWS Bilang Pilipino survey should not be trumpeted as a credible survey with a very small margin of error - because the sample was BIASED.

It gets better (worse), SWS generates its samples according to strata - or groupings. For the SWS Bilang Pilipino Mobile Survey it used four strata - NCR, Balance of Luzon, Visayas, and Mindanao. There is nothing wrong with using strata and employing stratified random sampling. There is, however, something very wrong about giving these strata equal weights. For its Bilang Pilipino sample, SWS gave NCR, Balance of Luzon, Visayas, and Mindanao 300 samples each. This distribution means that NCR, Balance of Luzon, Visayas, and Mindanao are all given equal weights. The problem here is that the distribution of the Filipino population - and consequently the Filipino voting population is not spread equally across the four strata! Here are approximate values: Balance of Luzon has 44%-45%, Visayas has 20%, Mindanao has 23%-24%, and NCR has the remainder. Putting all of these together we see that Balance of Luzon is severely under-weighted and under-represented and NCR is severely over-weighted and over-represented. BIAS.

The choice of the level of stratification is also dubious given that geography plays a key role in determining the outcome of national elections. National politicians often have home provinces and regions that would lend them tremendous amounts of support during elections. Choosing our island groups as the stratification level ignores the highly nuanced dynamics of local Philippine politics. You do not need to be a political scientist of statistician to realize that this is a poor way of distributing your sample. It is, for example, distinctly possible to generate a Balance of Luzon sample without Ilocos Norte. It is distinctly possible to generate a Balance of Luzon sample without Camarines Norte/Sur. It is distinctly possible to generate a Visayas sample without Capiz. It is distinctly possible to generate a Mindanao sample without Davao del Sur. The choice of stratification assumes that voting preferences within each island group are largely homogeneous. It assumes that there is little to no heterogeneity within each island group.

But wait, there's more!

The SWS Bilang Pilipino Mobile survey is an opt-in survey. As I indicated earlier, the actual sample could be smaller than the indicated sample. The people who were given mobile phones to respond to text prompts could choose NOT to answer the text prompts. Given the nature of the survey they cannot re-sample to fill-up the desired or indicated sample size! This is obviously another source of bias!

It is also of note that respondents can only respond from 7:00 am to 12:00 noon. If you fail to respond during that window you are excluded from the actual sample. If you forgot to charge the phone, if you could not get a signal, if you were busy at work, if you were sick, it doesn't matter - you are excluded. Bias, bias, bias!

The survey serves no discernible function. It provides no discernible benefit. One can even make a compelling argument that the SWS Bilang Pilipino Mobile survey is designed to condition the minds of voters and establish trends that would legitimize or sanitize fraudulent election outcomes. I urge you: Do not lend this survey any ounce of credibility.

Voters should not choose their candidates on the basis of who is 'winnable' and who is 'not winnable'. Your choices should not be about who can or cannot win. Elections should be about visions, platforms, and track records. Do NOT reduce the elections into a popularity contest wherein the winners are not chosen based on their capacity to govern - but rather by their perceived popularity. Do NOT cheapen our democracy.

Do yourself and the Philippine democracy a favor: Do not let this survey affect your choices. Think for yourself.
Source: http://www.allthingspinoy.com/2016/04/up-graduate-statistician-shares-insights-about-bilang-pilipino-sws-mobile-survey.html