Some people get flustered with the New York Times because they print a story about a political candidate that is blatantly fabricated. Others say it is not the newspaper it used to be. Some people think it is the greatest newspaper in the world. As a social psychologist, I got my jolt from the New York Times when I took a look at the recent article titled “Latest Election Polls 2016.”
By way of background, I have a Ph.D. in social psychology from the sociology department at Indiana University. I was a pre-doc and a post-doctoral fellow in a NIMH-funded training program in research measurement. In other words, I spent about eight years of my life studying quantitative and qualitative social research methods.
So, I am going to tell you what is required to take a great election poll, and then I am going to share with you what the New York Times instructs in a section of that article called How Different Polls Work. Then, you can decide for yourself what is going on in the world of election polls.
One of the greatest innovations of science has been the development of statistical research methodology. In the social sciences, it led to the ability to take a survey of a small sample of a population, poll people’s opinions, and yield results delivered with statistical precision. The famous Gallop Polls, Nielsen Polls, and Nielsen Ratings are probably the best known examples of this amazing phenomenon. The whole trick comes down to the statistical sample.
For example, if you want to survey national opinions about the upcoming Presidential election, you could never talk to the entire universe of subjects implicated in the election. Nearly 127 million people voted in the 2012 Presidential election. We would have to recognize that there may be at least 127 million people that might vote this year, but we could never talk to them all in a survey. Here is where the brilliance of the statistical sample comes into the picture.
Theoretically, it is possible to take a survey of only a sample, or a minuscule proportion, of the electorate and know how everybody would vote today if they could. This would be possible if you could know that this very small statistical sample is, in fact, a true reflection of the entire universe of voters. If you had 100% certainty that your sample was true, then you could poll 20 people and know how the whole nation felt. In reality, 127 million people is too many people to poll, and a perfect sample of 20 people is impossible to believe in – so what to do?
Statisticians have formulas for determining sample size. Researchers always want to know the minimum acceptable sample size. They even have large tables allowing one to quickly determine appropriate sample sizes based on known universes of n numbers of people. But, that is not a silver bullet to be used in all research. Imagine you have one study of how people’s self-esteem relates to educational attainment. Imagine you have another study of how people feel about race relations in America in relation to victims of police shootings. Even if you could assume that the universe of subjects was similar, say it was 127 million, which is arguably a good estimate of active adults in the U.S., you still could not use the same minimum acceptable sample size for both studies.
Why? Because the universe of subjects giving you self-esteem self-assessments are going to do the same thing all people do when asked to give self-assessments of their self-esteem. They are going to cluster around a rating of 4 out of a possible 5. For example, in a typical Likert-type self-esteem scale you might have 5 items on the scale as follows:
1 - extremely low SE 2 – low SE 3 – okay SE 4 – high SE 5- extremely high SE
And almost everybody is going to mark off 4 to say they have high self-esteem. Many will also mark off 3 for okay self-esteem. The mean will tend to be between 3 and 4 and tend to be closer to 4. The survey responses will cluster around that mean every time because no one wants to project that they may be suicidal in a self-esteem survey and no one wants to project free-floating narcissism in such a survey either. More than anything in the world, people want to project normalcy. It does not matter how bad off or how well off people are. Most people, most of the time, just want to appear to be captains of normalcy in the eyes of others.
However, the survey on race relations – the single most volatile topic in America - is going to have results swinging wildly. There are so many people with so many views on racism that we could expect these results to vary widely, and thus show a large standard deviation. Therefore, you will need a larger minimum sample size to gauge the results accurately. Great statisticians get paid 5 figures to work for weeks determining the appropriate sample size for a big research project, and then get flown in for a day of seminar presentations to hash it out with the survey administrators. While this applies to a large research project, the election pollsters have the luxury of knowing the minimum acceptable sample size, and not having to determine it for every poll. Although the samples of often 500 or less people used for a Presidential election poll would be questioned by some people, that is barely the beginning of the project.
The other problem is that once you have determined an acceptable minimum sample size, then you have to determine how to find that many people. Let’s say you have determined that a minimum acceptable sample size of 500 is worth polling. This is actually miraculous if you contemplate that 127 million voters being represented by 500 people, only make up only 4 10 thousandths of 1 percent, 0004%, of the universe of voters. Once you know that you need 500 subjects for the survey, then you are faced with finding 500 good people. They have to be randomly selected, or else you could just get the people anywhere and it would not matter. You could get all your sample subjects from different party election offices; 200 from Trump campaign offices, 200 from Clinton campaign offices, and 100 from the other party offices. But, that would be like stacking the deck.
You want to get a truly random sample of adults from the population, but you also want to make sure that you do not end up with 80% Democrats and 20% Republicans. So you want it to be as random and representative as possible. There is no hard logic about it at this point. Somewhere along the way you surrender to common sense, along with a dedication to rigid methodology, for the sake of accuracy. How could you make sure you have a random and representative sample? Nielsen discovered they could sample people’s TV sets and do so with great precision. But, these Presidential elections come in four-year cycles. It is one thing to track the TV signals from a sample of TV sets every day, all day long. But, it is entirely another thing to ask people to give you their subjective opinion about something. You cannot poll the same people continually or you will wear out the sample. Sometimes, you can set up a panel of people, like the Los Angeles Times Presidential Poll, which is using a panel of voters to poll their changing preferences throughout 2016. Some people criticize this method, while others think it is more stable and reliable than the random, representative samples that other pollsters are required to be developing for each nationwide election poll they execute.
In the next installment of this two-part posting, we will continue looking at the slippery slope faced by researchers who take nationwide election polls. For example, we will analyse what the New York Times means when it explains this in How Different Polls Work.
"Pollsters have several methods to choose from when conducting a poll. Regardless of method, it's hard to get a representative sample of the population to answer survey questions, so most polls weight their response data to match the expected composition of the electorate."