Banner
    Are Americans Polarized Or Is Polling More Scientific?
    By Hank Campbell | November 7th 2012 03:32 PM | 14 comments | Print | E-mail | Track Comments
    About Hank

    I'm the founder of Science 2.0®.

    A wise man once said Darwin had the greatest idea anyone ever had. Others may prefer Newton or Archimedes...

    View Hank's Profile
    Something happened last night that you don't see very often - almost all of the polls were right. Nearly everyone predicted the state electoral results correctly and that, my friends, is truly rare. 

    It's easy to get most of them correct. I used to live in Pennsylvania, for example, which was once pretty mainstream Democrat but then became among the last of a more rare "Blue Dog  Democrat" state - Democrats, but not the kooky social authoritarian ban-happy New York and California kind. It was understandable because union people in Pittsburgh don't have much in common with the residents of 'Frisco except a D next to their name.  Yet that changed over time too as both parties adopted Big Tent mentalities and candidates had to match.  Senator Joe Lieberman was victim to it. In his Senate re-election campaign he had to run as an Independent because he could not win a Democratic primary. He was the wrong kind of Democrat for Democrats six short years after he was their Vice-Presidential nominee. Sen. Dick Lugar fell victim to that same thinking among Republicans this year.

    Pennsylvania, like California, is not in contention any more. We don't start electoral votes at 0 in 2012, we start with 190 on each side. There were supposedly 9 "battleground states" this year, which means 5 if you are not a mainstream media company that wants to keep people watching their program.  As of this morning, Florida refused to declare a winner but 7 of the 8 other supposed battleground states went for President Obama, which means it was only a battleground in the eyes of the media. In reality, the polls had already shown how people were going to vote, right?

    Yep, we are that polarized. Averaging polls is all that scientific, and I was so unconvinced that this was a battle I wasn't even going to bother watching Tuesday night, but Razib Khan at Discover got me interested because he 'gamified' the election for me - by betting that the polls would be really accurate - not get the correct answer, but be accurate.  We had to settle on one test of accuracy and he picked Nate Silver, who writes the 538 blog at the New York Times, because Silver averages models from lots of polls.  Like I wrote in my article on the topic, 'if you look around the Poker table and can't spot the sucker, you are the sucker'.

    Well, I was the sucker this year, Razib had his 50 bucks by 8 PM and I had conceded well before that. He didn't need Silver, he could have used Real Clear Politics or just about anyone else and won this bet. The only people predicting a Romney win were hopeful Republicans so that was not the issue but I could at least predict a loss for poll averaging. Statistical averaging is not science, outside of psychology. Right?

    Sure, though it is academic because I am out $50.  But while science media beatifies Nate Silver of the New York Times, we have to ask why, since he is not the only one who was right, he is getting more attention than people who were more right.  Since Razib and I picked November 4th as our bet, Silver got 49 out of 50.  Still impressive but everyone else who averaged got that also.

     Heck, even people betting in Europe got 49 of 50 correct, they got the same one wrong Silver did:



    What is intriguing, and it also answers why Silver is getting such a response, is how many people are so violent about the election - not the results, no one with a clue bet against Obama, but about Nate Silver himself.  They are you-insulted-my-sports-team levels of irrational because I bet against the accuracy of his model, a guy they never met who writes for a $2 billion company as opposed to independent, non-corporate-owned science media site Science 2.0, which he is regression toward the mean guaranteed to have never read or heard about.

    Even though he is no more right than the polls, and less right than other statisticians, people are saying he is super scientific.   It looks a lot like motivated reasoning to anyone impartial.

    If we want super scientific, why not extol Prof. Drew Linzer of Emory University?  His votamatic nailed the electoral votes a week ago, when Nate Silver was off by a whopping 33 electoral votes. He's a political scientist.  Or  Prof. Sam Wang in the department of molecular biology at Princeton, who was predicting 99% for Obama and 319 electoral votes when Silver was still only at 75%. Those were predictions.  Waiting until the last possible moment and predicting exactly what the polls show is not scientific wizardry, if we had bet on Tuesday instead I certainly would not have crashed my $50 plane into the aircraft carrier of reality.  

    And that is all that happened, the polls were right, not the model. Yet people on Twitter are comparing Nate Silver to Chuck Norris as if he is, well, the Chuck Norris of statistics.  Meanwhile, the poor schleps at Real Clear Politics got the same result and earn no respect at all.

    Americans are polarized and that has nothing to do with science.  Married people, old people, religious people and white people voted exactly as polls expected. Single people, minorities, young people and atheists voted exactly as expected.

    No one's votes are in play, any more than California and Texas are up for grabs to a good candidate unless the right letter is associated with their name. That should be the concern but instead we now have villagers with pitchforks erecting cults of personality around people at the New York Times for being less right than academics.  But more right than Fox News.

    It turns out that the reason so many people on my Twitterfeed are genuflecting over him now is because right wing pundits had been critical of him. In other words, it has nothing to do with rationality, it is the same political gamesmanship as November 5th, another tool in the ongoing culture war. It isn't science. 

    If averaging polls were science, we wouldn't need to have everyone vote. In physics modeling you can assume symmetry in some parts, you can use boundary conditions and you can use parameters and you will still get the correct result.  That is science.  In polling, you can't do that - not yet anyway.  But maybe it is coming.

    In 1936, Arthur Nielsen used sampling to create a model that used a limited number of set-top devices to determine what all Americans were watching with the same accuracy as polling has today. And that was when there were only a thousand TV sets on the whole planet. It determined what we all saw and how fortunes were made and how advertisings spent money. It was a republic of media.

    With modern technology, maybe we could create a model where 1,000 representative people determine elections.  That would be science. Averaging a bunch of polls is not, it is just a sign that people are becoming more entrenched and so polls are more accurate - and that means we are less likely to have elections that are meaningful.

    Comments

    Could you explain the science part a bit? As a layman, Bayesian searches seem a bit like voodoo, yet Dr. John Craven found the lost H-Bomb as well as USS Scorpion based on what would appear to be an average of polls - polls with unknown error bars, etc.

    I completely understand that Silver's fame (or notoriety, depending upon the side one chooses) is almost entirely a partisan issue and thus, not science. I'm more interested in why you thought Silver's technique would not be as accurate as it was. I'm not that interested in what other firms did with their educated guesses; I'm interested in Silver because of your bet and because Silver claims a mathematical basis for his results, thus science.

    Hank
    Silver went into polling because he was an Obama supporter when all the polls showed Hillary Clinton ahead and he knew (rightly) that it was only because of her name recognition.  The polls were as accurate as ever but converging on the wrong answer and so he averaged the polls but then factored in how much 'lean' they have, Republican or Democrat in the case of this election. 

    So early on he was very statistically biased.  The highest chance Romney he ever had of winning was 41%, in June - yet know one was trumpeting his accuracy outside NYT readers who were, as I said, engaging in some motivated reasoning because it was silly to think Romney had no chance of winning at all with 5 months to go.  Clearly Silver was not going by just averaging polls, his 'lean' was biasing the results. Over time, Bayes inference will get better parameters which means his model will be more accurate.  Each baseball season we do a baseball playoffs simulation update using Bayes and by the last game of the World Series it is really accurate.  But what it can't do is predict a winner accurately before the season starts.  The more teams fall the better it answers because the model converges with more accurate parameters.

    So it goes with politics.  In June, we know each side had about 190 electoral votes, Montana and Texas were not voting for Obama and California and New York were not voting for Romney, we didn't even need polls for that. So there were some parameters.  Then there were gradients of left and right in the remaining states but those parameters changed over time because polls showed X was happening with a confidence interval.  +/- 4% or whatever with the 4% being 'we do not know what we are doing and the results are crazy'.

    The closer to election time the less likely people are sill undecided so the more accurate polls would be  - if polls are reaching the likely voters in the key groups; married, religious, older and white on one side, single, atheist, younger and minority on the other.  I was counting on the fact that polls were oversampling someone - older people still have landline phones, for example, while younger people do not, so older people might be oversampled. It didn't matter who was being oversampled, it just had to happen.

    Well, it didn't. Americans were absolutely as polarized and immovable as people said they were. Silver's prediction changed by 33 electoral votes in one week so the one I am impressed by is the political scientist who got the electoral votes right in October using the same averaging but his own 'lean' of those polls.
    I can't speak for the rest of Twitterers, but for me, contributing to the #natesilverfacts was as much a joke on Nate Silver as it was on Republican pundits. I also agree with you that people exaggerate the value of a model by looking at its accuracy on the night of the election.

    I was counting on the fact that polls were oversampling someone Thank you. I think I understand your bet better now. You felt that there was enough systematic error in the polling process that the results would necessarily be skewed. Whereas random errors tend to cancel out.

    The reason people are lionizing Silver more than the others is that he bore the brunt of the criticism from the "centrist" political pundits and pro-Romney crowd. The reason he got most of the criticism is that he has been the most prominent of the aggregators due to his position at the NYT and the reputation his web site earned in 2008.

    In the end this was a battle of two world views. On the one had there was the data-based world view, in which people like Silver and Wang agreed that probabilities could be determined for future events based on polling and other data. The other side was the gossip-based view that relied on narratives and memes and for inside campaign sources to tell them who was really winning. When people like Joe Scarborough and David Brooks were publically disparaging Nate Silver it was because he threatened their world view. As we know now the Romney campaign had been "unskewing" their internal polls by overweighting them with assumptions of a high pro-Romney turnout and actually believed they'd win. So Scarborough and Brooks, who are used to getting their "whose really winning" info from off-the-record campaign insiders concluded that the race was a toss-up since both sides truly thought they were winning. When Silver and others published their analysis the Scarborough/Brooks crowd condemned them as biased, when in fact the bias was within the Romney campaign.

    There are lessons to be learned from this, but I'm not sure that the people who need to learn them will do so. Groupthink is not a new concept. Nor is the idea that if there are two strongly held viewpoints one of the two may be flat out wrong. The correct answer is not always found by averaging the two viewpoints: If a creationist says that the earth is 6000 years old and a geologist says it is 5 billion years old the correct answer is not 2.5 billion years - one of these two people is utterly wrong. This is true for all kinds of topics, from species evolution to election forecasting to climate change to macroeconomics. If a group of people falls into groupthink mode on one topic they are likely doing so on many other topics as well.

    Some of this is unfair and unclear, so let's clean it up a bit.

    First there is "pure" poll averaging and there is hybrid averaging/modeling. Silver's method and Linzer's methods are hybrids that combine predictive models using factors like economics and GDP, factors other than pure poll averages. Very far from the election, these other factors dominate, but as time approaches election day, poll averages dominate. On election day, the results is 100% poll average. These "models" are held out by their creators as attempts to PREDICT the outcome of the election. If you thought either of these models are pure poll averages you are mistaken.

    Simon Wang and Hugh Jackman (HuffPo Pollster) are computing "pure" poll averages. If anyone is using a pure poll average computed say on October 1st to PREDICT the outcome on November 6, then they are mis-using the poll averages. I follow the language of Sam Wang, Hugh Jackman, and Mark Blumenthal very closely and they do not overstate or mis-state what their poll averages mean.

    There is a difference between computing a poll average and predicting an outcome. There is no particular reason that a poll average computed on Oct 1, should accurately predict an actual outcome on November 6. What a poll average computed on Oct 1 tells us, is how people say they would vote if the election were held on Oct 1. So let's create some notation to make our lives easier. Suppose PAVE(t(x),t(e)) means a poll average computed on date t(x) for an election to be held on t(e).

    When a journalist or a reporter or a news organization reports the state of the race on t(x) , the reporter should be reporting PAVE(t(x), t(e)) rather cherry picking a single poll. That is the benefit of poll averaging. This is not a trivial use of poll averages. Reporters and news organizations who cherry pick outliers on t(x) to write a, "the race is tightening" story, or "Romney has momentum" story are actually fabricating nonsens, which in some ways, however innocent, is just as bad making up facts.

    Poll averages are absolutely and unequivocally the best and only way for professional journalists to accurately and succinctly report the state of the race using polls at any point in time, and accurately discuss whether or not the state of the race has or is changing. (Obviously more resourceful news organizations like the NYTIMES, WAPOST, are going to commission their own polls and "cherry-pick" their own poll, but the vast majority of local, state, and cable newspapers, web sites, cable channels, radio and tv stations do not have these kinds of resources.)

    Pundits have editorial license to do whatever they want. Pundits who are partisan assets of campaigns also enjoy this license, but reporters do not. So it would a great benefit to the public if "hard" journalism adopted poll averaging as the preferred method for reporting polls or other horse race stories, to help the public distinguish between true voter signal and pundit noise. It would also be great if news organizations adopted a methodology based on pure poll averaging when creating and showing their red/blue electoral votes maps.

    None of us should expect news organizations to abandon the perverse incentives they have for understating what they know about a race to make it seem more close and exciting than it is, however this creates a real market niche where cable channels in combination with web blogs can either take that business away from them or force them to admit more than they want to admit. Nate SIlver didn't become notorious for overstating the closeness of the race, he became notorious for stating the non-closeness with force and precision, so there is some economic incentive for those who report the race more accurately than those that don't.

    For that reason we should thank Nate Silver, the popular guy, whose employer, the New York TImes, by hosting his blog, also sets the example for how a news organization can talk out of both sides of its collective organizational mouth. The Times officially overstated the closeness while its blog accurately stated the race.

    Now lets talk about PAVE(t(x),t(e)) as t(x) --> t(e) when the time of the poll average is very near election day. Do election day poll averages accurately predict outcomes on election day?

    On or near election day, I would venture to guess, but do not know, that the poll average becomes more and more accurate as a predictor. Hugh Jackman wrote a nice article about how he converts the polling average into a prediction, and Nate Silver has done good empirical research showing how many times and when election day state polling averages have picked the wrong winner (rarely.) If your point suggests that polling averages shouldn't be predictive late in the race, I disagree. If your point is that polling averages should be predictive early in the race, I could agree but you should say it more clearly.

    If your point is that its a no-brainer to accurately call the race on election day, might I point you to all those who got it wrong, including "serious" people on both "sides"? Accurately calling the race on the day of or the day before is a recent development and there are still many who are struggling with it.

    As far as oversampling is concerned. Mark Blumenthal articles in the final week of the race suggested we were getting 30+ new state polls for swing states each day. Is your point that you think that 30 separately polling agencies would each systematically over sample the same group? I don't understand the point.

    And, if the point of all the attitude is to castigate most people for confusing poll averaging as a method of predicting outcomes rather than simply reporting, I suggest you are part of the problem, because you placed a bet on a hybrid system and mis-stated it as a poll averaging system.

    If you want people to understand that pure poll aggregations only coincidentally have predicting power, well in advance of election date, you should say it more clearly.

    The fundamental furor over Nate Silver happened as you say, because Nate and others, correctly used both pure averaging systems and hybrid systems to debunk the campaign-originated media narrative about RO-MENTUM, thus becoming the object of hatred by the Right. And many on the left probably flocked to his site for comfort.

    I agree that there are many nuances to poll aggregation and much confusion over whether poll aggregation should be an accurate predictor of final outcomes, but this article doesn't help clarify or explain any of the issues and why they are important.

    Hank
    There is a difference between computing a poll average and predicting an outcome. There is no particular reason that a poll average computed on Oct 1, should accurately predict an actual outcome on November 6. What a poll average computed on Oct 1 tells us, is how people say they would vote if the election were held on Oct 1. So let's create some notation to make our lives easier. Suppose PAVE(t(x),t(e)) means a poll average computed on date t(x) for an election to be held on t(e).
    I certainly agree here but what about November 4th?  October 26th?  On the former Silver had the same state predictions (I am leaving out the probability parts and sticking to just the outcome) as everyone, including people who know nothing in Europe and were just betting, and on the latter he was off by 38 electoral votes.  Can we really say on November 4th all those people were undecided but Silver somehow calculated them on November 6th? He said just the opposite on his article of November 3rd, that undecided and media were not factors by then.

    My gripe is somewhat biased because (a) I lost 50 bucks and (b) Americans were incredibly disappointing in their ability to be predictable sheep catering to party lines but - and I may be the only one seeing this due to selection bias - the over-the-top glorifying of the guy Karl Rove does not like as being 'accurate' while ignoring people who were actually more accurate but were not criticized by Republicans is just unfair.  If Linzer got the electoral votes right a week before everyone else and Wang's Brier Score is better than anyone, it should get some attention.
    Not sure I fully understand the point about what changed in Silver's calculations between Nov 4 and Nov 6, but If the real gripe is that Silver gets more attention then he deserves, or rather , that jackman, wang, and linzer deserve more attention and may be more accurate. I agree.

    To your other point about oversampling. Had you actually read more of Wang, Linzer and Jackman, (rather than Silver) then you would have seen some of their discussion about the unlikely possibility of systemic bias across distinct polling services, and you probably would have been less likely to bet the $50! Karma's always fair in that sense! :)

    Hank
    So statistically I would have been less likely to lose money on the accuracy of polls if I had consulted a wider sample of people using the same polls???

    I am inclined to disagree but that is how I lost 50 bucks in the first place - so you win. :)
    As others, I'm not quite sure what you're complaining about here. Is it a problem that Americans accurately reported their electoral preferences to polls, and consequently the polls turned out to be accurate? Or that human populations are samplable, and that polls of them follow statistical rules? Or maybe that after a year of campaigning (or arguably more), the preferences didn't change very much in the last few days?

    You also say "If averaging polls were science, we wouldn't need to have everyone vote." and, in point of fact, it's pretty clear that we don't need to have everyone vote. Fairly basic poll averaging by Sam Wang have accurately predicted election outcomes three elections running. I'm not sure if you're claiming that means of samples from a population somehow don't trend to the mean of the population as the sample size increases simply because we're dealing with humans, or what. I think it's pretty evident that running decent-sized fair polls on the election day would serve just as well as actual voting. So I suppose this would make poll averaging "science" according to whatever definition you're using here.

    For the record, I read Sam Wang (and came here because he linked to this article), and have not done more than glance at Nate Silver's blog.

    Hank
    I suppose my issue is subjective.  People have beatified Silver and said the age of science in predicting elections has arrived, but they seem to be doing it because he writes for the NYT and got attacked by Republicans.  Hardly science.  Wang and Linzer got far less attention and virtually none from the 'scientists who have never done any math' contingent glorifying Silver.  I have nothing against the guy, and nothing against averaging polls, but he is not a witch or a guru, people were simply as predictable as polls said for the first time pretty much...ever.
    It's not really that "people were as predictable as the polls said"; it's that the amount of polling done increased substantially, so starting from 2004 there were enough polls to accurately predict the result (and in 2008 and 2012 the number of polls/total sample increased even further). It's a change in sample size, not a change in people. After writing that initial comment, I read two of your other articles, and it feels like you believe there is something inherently unpredictable about people that should make polls inaccurate even with large sample sizes and good designs, but I'm just not aware of any evidence to suggest this is the case.

    Agreed about too much focus on Silver as an individual, though.

    Hank
    Mid-term polls are not accurate, though, so can it be that polling is really better?   Will we get 435 accurate results in 2014? We should if the polls are well designed. I think instead people have become hardliners demographically which made polling look more accurate and that will change once Republicans get more nuanced - if they come up with an immigration plan, for example.
    I think the problem with midterms is two-fold. First, they are a smaller event, and as such fewer polls are conducted. And second, and perhaps more importantly, it is much more difficult to predict House outcomes based on polls, since there are so many seats that most congressional districts are not polled very often. For example, Sam Wang only got the general shape of the House right this year (he predicted a change of D+2-22, actual change D+7), since working with a combination of district level polls and national polls still left a lot of uncertainty. So, it's not just midterm House races that are hard to predict. Conversely, for the Senate races, Sam got 10/10 for the close races (although he probably got slightly lucky - it would not have been surprising according to his own probabilities for him to be off by one or two), since they actually match the dense state-wide polling, and I would expect for the Senate to be only somewhat less predictable in the midterms, as the number of polls decreases.

    I'm also not convinced that people becoming more "hardliners demographically" would affect the accuracy of polls one way or the other. It is true that if more people make up their minds earlier, polls from earlier in the election cycle are likely to be more predictive of the outcome, however I think that polls late in the elections should be equally predictive either way. After all, even if the electorate has no preferences at the start of the election cycle, surely after numerous debates, countless ads, and discussing the election with friends for a while, they will still have made up their minds and be able to report as much to pollsters. For polls to not be very predictive, a large part of the electorate would have to make up their minds only once they enter the voting booth (or on the last day, in any event), and I think that's in equal parts unrealistic and undesirable.