Banner
    How Accurate Are Those Political Polls?
    By Hank Campbell | November 5th 2012 05:00 AM | 81 comments | Print | E-mail | Track Comments
    About Hank

    I'm the founder of Science 2.0® and co-author of "Science Left Behind".

    A wise man once said Darwin had the greatest idea anyone...

    View Hank's Profile
    Nate Silver, a sports statistician, made waves when he accurately projected the results of the presidential election in 2008.  So a few days go when he predicted that President Obama was 75% likely to win when polls only showed the candidates were about even, it set atwitter those people who think statistical models of polls are meaningful.  They were vindicated, the age of scientific projections had arrived, in a baseball cap.

    They thought that because they don't understands polls or statistical models.

    After I wrote on Twitter that 'polls are bad models', Razib Khan of the GNXP column at Discover, wrote back: "want to bet money that polls are bad models?"



    Well, yeah, I kinda do.  I'll bet my house right now.  The problem is, no one will try and take my money when accuracy really matters. The minute you try and get a statistician to make a prediction, they waffle and want a range. Silver was saying 'Obama will win' but he was really only saying he believes there is a 75% chance Obama will win. How can he be so confident but not be really confident?  If his model is good, why did he not declare Obama the winner and stick 100% on it and go on vacation?

    Everyone knows that this far into an election cycle 47% of voters are decided - it was politically incorrect for candidate Mitt Romney to say that 47% of people were not changing their minds no matter what he does, but not statistically incorrect. Elections are often won or lost by people staying home, not flip-flopping from one party to another.  

    But Razib is a statistics guy.  He likes to play with data.  If you say Democrats are pro-science, he will show you how the numbers only show that when they are framed a certain way. Why would he make such a crazy bet then?  Was he just being contrarian, is he a Silver fanboy or did he know something I don't know?  

    As I explained to someone shortly after Silver's claim and what it really meant and why he would bet Joe Scarborough, if you look around the poker table and you can't spot the sucker, you are the sucker.  Am I the sucker?

     Silver is predicting Obama will get 50.6% of the popular vote and 305 electoral votes. Razib agrees.  He is not betting on Obama, he is betting on beating me about my opinion of Silver's model. That's what betting is.  Anyone who bets on their team is goofy, you bet to beat the spread and the spread is how much disdain I have for a numerical model based on polls.

    So I had to weight that properly.  After all, weighting is everything.  And he did too, that is what makes it fun.  Silver correctly nailed 49 out of 50 state results in 2008 and was within 0.9% of the popular vote and 21 electoral votes.   He did it by creating probabilities using his weighting of the accuracy of polls. Tricky stuff.

    Razib's contention had been polls are accurate, not Silver, so I thought about that and suggested we use the Rasmussen Poll rather than the Poll of the United Astrology Conference showing Obama at 100% likelihood to win that I have been saving since May for just such an occasion. But he wasn't taking that bait, he wanted to stick with Silver because Silver averages lots of polls and that improved the methodology, he must believe.

    Yet that actually made me more confident, not less.  The 'why' of being more confident is because of errors.  In full-wave analysis of physical models, like electromagnetics, introducing more potential errors means you can get an answer that is 99.999% accurate and still completely wrong.

    It's called convergence.  Converging on the wrong answer is quite easy if the data is not good. If you run a model with the wrong data, you might have total confidence in results of your chip package, for example, but then you spend a million dollars taping something out and it fails and you are out of a job.  There is a reason the phrase "it's academic" is part of the lexicon, meaning 'it doesn't really matter', because you can be wrong in academia. In the world of corporate modeling, and not the statistical woo of the social sciences, results matter.  

     I submit a quote from the important science work, "Ghostbusters":



    In politics, like in academia, if the model is wrong, you get to write articles about why it was wrong. Silver knows about the risks of bad data, he worked too.  He will caution people that prediction is not prophecy, so he is putting a stake in the ground but he still has an out - if he is wrong, it is because the state polls are statistically biased. He said that and then raised his confidence to 83.7 percent. Still not 100% and I have mentioned that a few times and there is a reason I do so.  He won't say 100% because even he doesn't believe it.

    "If the Giants lead the Redskins 24-21 in the fourth quarter, it's a close game that either team could win. But it's also not a "toss-up": The Giants are favored. It's the same principle here: Obama is ahead in the polling averages in states like Ohio that would suffice for him to win the Electoral College. Hence, he's the favorite," Silver told POLITICO.

    But when he was in sports, Silver kept a cool head.  He did not go into sports because he disliked what bookies predicted about his baseball team.   But he went into polling because he was an Obama supporter and did not like that media reports during the primaries kept saying polls showed Hillary Clinton ahead when he knew it was only because fewer people had heard of Obama. And today he does not disclose his 'weighting' of polls in his magic sauce, he simply declares that a poll is 'leaning' Democrat or Republican and how it changed from last time. No problem with that, he is not writing for a journal, his mystique while he claims to be transparent is part of the brand.

    What does all that mean? It means if you want to make money on a Nate Silver prediction, you can't bet on Obama. And I don't care about that anyway. I am not betting for or against a candidate, I am betting against Silver. And I am betting against him because the voting demographic has changed from even four years ago. A whole lot more people use only cell phones, for example, and that skews the respondents in polls.  Polls could be wildly oversampling Democrats, as some conservative groups claim, but they could just as easily be oversampling Republicans.  It doesn't matter because Silver can't account for oversampling until after the fact. No matter which way the polls are wrong, I win. So methodology is a problem and I mentioned voter turnout before so let's go back to that.

    Here is what we know about actual results because, as I said earlier, a huge chunk of elections comes down to people who show up.  There are rarities, like Bill Clinton winning with 43% of the vote in 1992 while Ronald Reagan trounced Walter Mondale in 1984 with 58.8% of the popular vote, but for the most part elections stay within a narrow range. President George W. Bush easily beat John Kerry in 2004 but only got 50.7% of the vote.  Obama got 52.9%, which was clearly a mandate.  Yet the numbers are telling the real story. McCain only got 59,597,520 votes while Obama got 69,297,997 out of 230,872,030 eligible voters, versus 203,483,455 in 2004, a 27 million vote difference. Even with all those extra available voters, McCain got 3,000,000 fewer votes in 2008 than Bush did in 2004. What happened? People were excited about Obama and went to the voting booth while people inclined to vote for McCain stayed home and thus never voted.

    Knowing that George W. Bush got 50.7% of the vote makes Silver's 50.6% prediction a very bad basis for betting because only 10 presidents have ever gotten under 50% of the vote. At first Razib asked for 0.5 +/-  on that 50.6%, a real sucker bet.  Both ends of that range are in the sweet spot for the majority of elections.  That forced me to consider one other possibility, along with the chance he is being contrarian, being a Silver fanboy or he knows something I don't know.  I had to consider he just needs $50 and was asking everyone to bet until he found someone crazy enough to give him even money on Silver getting Obama in the range of 50.1% to 51.1%.

    Regardless, I want to prove statistical averaging of polls is useless, and such a large range  does nothing at all to validate the accuracy of numerical models based on polls, which is where he said I was wrong and was so confident he wanted to put up money. So I wasn't interested in that.

    But Silver was right about 49 out of 50 states in 2008.  If his predictive model is accurate today, it will be where he was also strongest in 2008.  And if I am going to be right in my belief that polls are terrible predictors, it has to be beating him where he is at his best.  

    So Razib and I agreed on a bet and I was so confident I even gave him one extra state. Silver was only wrong on Indiana in 2008 and our bet is whether or not Silver will be accurate on 48 state results this election. Easy if his model is accurate. He really should be 50 out of 50 because most states are not really in play.  We know how California is voting, we know how New York and Texas are voting, etc.


    307 electoral votes to 231.  If these colors do not match the real election 3 times, I win.  Credit: New York Times November 4, 2012

    Why do I still feel so confident knowing Silver feels confident too? He is a wildly popular stats guy for the New York Times and I am a nobody, I should be nervous.  But unlike Silver's baseball performance prediction system, an election is a one-off event. Statistical corrections happen in baseball, people can hit .750 on the first day of the season but it has never happened over the long haul and never will. Elections, on the other hand, happen once every four years. Anyone making a model from poll numbers and expecting it to be accurate would have to be sure of the poll numbers. Sampling is more rigorous than ever but it is only reaching a less diverse population than ever. They can sample all they want but the error is still the same as it was 40 years ago while turnout is much less predictable.


    If he had weighted polls as accurately as physicists or engineers make models he would not have been off by 21 electoral votes and 1,100,000 million people in 2008.  Waiting until this close to the election to bet does not help him, though people think it does.  As any of you Bayes disciples know, the farther we go in an election, just like the farther we go in baseball playoffs, a model will converge on a better answer because it will self-correct. But the election has not started so time beforehand makes little difference.  During the 'silly season' we are in right now polls are less meaningful, not more, yet he still takes them as seriously as he did a month ago, he just weighs their 'lean' and how much they have changed a little differently. He thinks voting is linear, or at least nonlinear by being linear in tiny enough steps he can see when it is happening.

    That is why I remain confident I will have money to burn on Wednesday.  Being off by 21 electoral votes and 1% of the popular vote will get you hired by the New York Times, but in physics, and certainly the corporate world, being off by over 1,000,000 will get you fired.

    Who will win the election? I don't know, it is crazy to bet against Obama's get-out-the-vote machine but that is how fortunes in polling are made. I just know I will be $50 richer.

    Comments

    Why not just join Intrade? Currently you can bet $3.33 and if he wins you get $10. Great odds. Put your money where your mouth is!

    Hank
    If who wins, Silver or Razib?  I did put my money up, $50.  $50 is putting my money where my mouth is a lot more than $3 would.
    So you're not saying you think Obama will lose -- or even that the polls are inaccurate. You're saying you think that *Silver* is wrong in averaging polls... right?

    Hank
    You could help me be a better communicator by telling me how I can be more clear than writing "I am not betting for or against a candidate, I am betting against Silver."
    I noticed that line. I thought it was meant to clarify what your actual BET was, not the motivation for the bet.

    If your motivation was that you thought, for example, that Romney would win, then the bet against Silver is pretty darn good -- but the line you wrote would still be valid because you were not betting against Obama but against Silver.

    The problem I had was that your arguments were based on very precise concepts. It was clear you wanted to word things carefully -- so I interpreted your line very literally.

    Hank
    I also wrote articles on how to bet on the Higgs boson existing and on winning at roulette. I wasn't against the Higgs boson when I noted some mass ranges were a terrible bet and one was good. So it goes with Silver.  Betting against his popular vote was bad but betting against his state predictions is good. It isn't perfect, or Razib would never have taken the bet, but it favors me.
    Unless he really has a magic crystal ball or equivalent, he can't be right all the time. So I agree that the bet favors you.

    Why not just join Intrade

    intrade makes it hard for americans to bet (probably because of american laws). you can't use a credit card. need to send a check or a wire transfer. the latter costs a shit load of money, and i'm not sure about how quick a check could get there. once you have money in your account, then you can trade. but you need to get it in there in the first place.

    (i know this because i wanted to put $500 into intrade, but i wasn't willing to spend $50 to get it there from bank fees)

    Any idea of how likely you are to win under the assumption 538's probabilities would be exactly correct?

    Hank
    If he is correct I lose.  If he is even close I lose. That is the whole point. I am so confident that, despite the pretense of statisticians, the method of averaging polls is inaccurate not only was I willing to bet he is wrong (about the results, not the winner of the election itself - you'd have to be crazy to bet against Obama's finely-tuned get-out-the-vote machine) I am basically 'giving points' and letting him be even extra wrong.  

    Some gamification and 50 bucks is the only way to make election night interesting.  
    If he's 100% correct, there's a 45% chance Florida flips and 30% for Virginia and Colorado. Even just those three would give you a roughly 4% chance of winning the bet. I suspect you'd get some 10-20% of 3+ states flipping if they're independent of each other.

    Obviously it's a sucker's bet if Nate is even close, but you'd still have some chance of winning. It would be interesting to see what his own opinion on your bet would be.

    Hank
    He's got partisan people going after him for being so confident so he is not going to notice me. He bet Joe Scarborough $1000 (and then $2000) that Obama would win after Joe said the model was flawed - but those are two different things.  As every middle school math student has learned, you can get the right answer with the wrong method so predicting Obama is easy, there are only two candidates, he is the incumbent and has an unreal get-out-the-vote organization, maybe better than anyone in history. That doesn't Silver's model of polls right.

    But if his model is as good as he says it is, he should get 50 out of 50 states right.  I am betting he gets 47 or less, even though he got 49 in 2008.  So I am wildly overconfident and putting money on it, which I think he would have to respect even if he did not agree with my lack of confidence in his polls model.

    Basically, if a whole bunch of polls are inaccurate, averaging them to get a 'wisdom of crowds' effect will not work unless he can correctly gauge how to weight over 500 different results.  If he does, he has my profound respect and Razib has my 50 bucks.
    Thor Russell
    "But if his model is as good as he says it is, he should get 50 out of 50 states right."The comment thread where I disagree with this seems to have disappeared?
    Thor Russell
    Hank
    You're arguing minutiae and it is distracting to the broad audience I would rather reach so I deleted the thread. Obviously you can bet your own $50 on whether or not Silver will get 48 out of 50 states correct using his model, Razib put his contact information in a comment below. Then you can write as much as you want about how Silver's model is correct and you lost money believing in it.
    Michael Martinez
    I don't see Romney winning.  I have no science to back that up but my subconscious has crunched all the signals it has received and in my "inner heart" I just don't see Romney winning.


    One of the problems I have with all these polls is that Presidential elections are not decided by the popular vote but rather by the Electoral vote.  Most of the polls don't take that into consideration.  Another problem I have with polls is that when I have been asked to participate I have usually been very annoyed by the process.  Sometimes I answer the questions and sometimes I don't.  I just cannot believe I am the only person who resents the interruptions.


    I did enjoy being a Nielsen family once and an Arbitron family twice, but I don't consider those to be political polls.  They're entertainment diaries.  Maybe we need political diaries -- but then, Nielsen has rarely ever supported any of the handful of TV programs I have fallen in love with.  That is probably why I don't watch much television.


    My heart just cannot take the strain of all this disappointment.

    Hank
    I don't see Romney winning. 
    You'd have to be crazy to bet on that happening. I am not betting on or predicting a winner, I am betting that the statistical darling of the New York Times is too wrong to be meaningful in predicting a winner, other than what everyone can see and just adding on a few percentage points.

    That aside, I have no issue with the electoral college.  Living in California, it is easy to see the pitfalls of tyranny of the majority and what it does for diversity and tolerance.  Without an electoral college, 5 states would decide the president for all of us.
    Michael Martinez
    I may be crazy but I'm not betting on anything in this election.  I gave my disposable money to the Red Cross.  :)

    However, given how solid the red is on every 2012 electoral map, I think it's coming down to about 5 states deciding the election -- that has always struck me as ironic.


    Human application of tools acts something like a centrifuge: everything is stirred up until divisiveness sets in and then we become stratified, leaving a "cream of the crop".  Of course, one man's cream is another man's sediment so I suppose that's really a terrible metaphor.
    My 8 year-old predicted 2008 when the media campaigned for Obama over Hillary.
    This year while the media campaign continues for Obama, the enthusiasm is gone.

    My kid says Romney wins, now we'll see who looks silly after the election.

    Hank
    We'll see. One thing that isn't being considered is the impact of media on all this.  While TV networks would never call an election before polls close, a blogger at the New York Times, and therefore a whole bunch of other people, called it 5 months before the election.

    So it could be a self-fulfilling prophecy.  That is why the outcome is a bad bet. Does the NYT have the ability to swing an election to the president?  No idea, and there is no way to prove it but I bet their business development people will be happy to let people believe it.

    Your child is with 49% of America either way, so that is good, but don't let him put money on Romney unless the odds are 4:1 and up.
    You've made a number of logical and factual mistakes in your article. These mistakes don't vindicate Silver but they do tend to undermine your arguments:

    You state that 'everyone knows that 47 percent of voters have made up their minds at this point' and that Romney was politically incorrect but not statistically incorrect in stating this. This statement is false. Romney stated that 47 percent of voters had made up their minds to vote against him. If your statement were true then only Obama supporters have made up their minds and NOBODY has made up their minds to vote for Romney.

    You state that a poll's proximity to the date of the election has no bearing on its usefulness in predicting the outcome of the election. You then go on to state that the opposite is actually true: polls closer to the election are less accurate. Unless you believe that polls in general are completely useless and wildly inaccurate.. both of your assertions are false. The reason a poll's proximity to the election date matters is because unpredictable things happen in the course of a campaign that cause a large number of people to make up their minds one way or the other. Obama's performance in the first debate is one example. Romney's embarrassing statements regarding roughly half of Americans being dependent and 'victims' is another. Time proximity simply limits the number of opinion-shifting events that can occur.

    You state with some hubris that being off by 21 electoral votes and 1.1 million popular votes is good enough for the New York Times but would get any physicist or anyone employed in the 'corporate world' fired. Both statements reveal an ignorance of basic statistics and are symptomatic of black and white thinking:

    Raw numbers by themselves have no value. In terms of accuracy, percentages and context are the only things that matter. 1.1 million would be a large error if the total vote count was 10 million. 1.1. million is not a large error when the total vote count is 200 million (0.5%). His prediction of the electoral count was less accurate (missed by 3.9% but based on mis-predicting a single close state: effectively 1 state out of 50 or 2%.

    Keep in mind this is all in the context of predicting a single event (the election outcome) based on what a very large number of free agent participants decide to do on election day.

    The comparison between tolerances for inaccuracy in the field of physics compared to political predictions is based on a fallacy. Unlike physics, most professional fields deal with a set of highly unpredictable free agents. In economics, statistics, meteorology, investing, and many other fields .. success is based simply on the ability to predict outcomes more accurately than others in the same field. In some cases, being right just 51% of the time is enough to make a sterling career. Or to use your baseball analogy, a batter that is *right* even 40% of the time would be considered a superstar.

    You and I don't pretend to know who will win the election and neither, by the way, does Nate Silver. That's why he gives probabilities and percentages instead of stating the outcome as a certainty. However, knowing that something has an 80% possibility of occurring is very very useful - whether you want to believe it's true or not.

    Hank
    You state with some hubris that being off by 21 electoral votes and 1.1 million popular votes is good enough for the New York Times but would get any physicist or anyone employed in the 'corporate world' fired. Both statements reveal an ignorance of basic statistics and are symptomatic of black and white thinking
    Everything else you wrote is well-spoken and subjective so I am fine with it (and thanks for your constructive comment) but this part is not correct; I am not ignorant of statistics and I am certainly not ignorant of modeling.  Is my thinking black and white in this article?  Of course it is, we are making a bet on a result, there is no range of probability on whether or not I will win the money, I will or I will not win.  Making that error and claiming it is nuance is why statistics is not scientific.
    Hank: "it was politically incorrect for candidate Mitt Romney to say that 47% of people were not changing their minds no matter what he does, but not statistically incorrect."
    -----------------------
    [off topic, but..]

    No, what was "politically incorrect" about Romney's 47% comment was were he called them [the 47% of Americans who wont be voting for him] victims whom are unable to take personal responsibility for lives. I like the spin, though. His campaign tried that, too. Guess I know how you vote. ;)

    Hank
    That's fair, I got no dog in that fight so when I saw it he seemed to me to be saying that because some people were getting something they weren't going to vote for someone who was not going to give them stuff and that wasn't going to change.  Whether he Forrest Gump'ed into it or not his number is about right.  The times when we have lower than 47% for one side are when motivation to vote is low.  Really, about 6% of the population determines the president.

    The point of this is to deflate this fetish with science-ing up guesses based on polls, not to endorse Obama or Romney. I am in California, so if you want to think I am one of the 4 people in this state who votes Republican, go for it.
    responsibility for lives = responsibility for THEIR lives. Sorry.

    "Is my thinking black and white in this article? Of course it is, we are making a bet on a result, there is no range of probability on whether or not I will win the money, I will or I will not win. Making that error and claiming it is nuance is why statistics is not scientific."

    Although the election is an 'either/or' proposition, at this point it is not a 50/50 proposition. That is your logical error. Most events are 'either/or' but very few are 50/50. For example, Romney will either win the electoral votes in California or he will not. That is an either/or with only two possible outcomes. If you want to wager even money on Romney winning California, I'll bet you all day long and happily take all of your cash.

    You deride statistical analysis for not providing 100% predictability/reliability as 'not scientific' but.. that is hardly the point. Estimations that not 100% reliable are very useful, in fact essential, in everyday life. When you disregard any data that does not correlate with predicted outcomes with 100% accuracy - you are engaging in black and white thinking. Essentially you are equating something that is not 100% reliable with something that is 0% reliable.

    MikeCrow
    We'll have to see if the turnout is closer to 2008 or 2010/2004.
    I think most of the polls predict based on the 2008 counts. If the actual turnout is more like 2004/2010 you'll win your bet in a big way.
    Never is a long time.
    Hank
    It should be higher than 2008 just because we have more people, what may make the difference is which side is motivated.  Mid-term elections are always low.
    Hank
    Like all statistical models, new information can change things dramatically.  Glendon Mellow, the artist behind the outstanding Flying Trilobyte nom de plume, reminded the world that Halo 4 is set to release tomorrow and, he notes, XBox 360 fans are not leaving their couches.

    The rush is on by pollsters to find out whether PS3 fans are Republicans or Democrats because they are apparently going to be picking a President.  Everyone knows Wii fans are too young to vote.

    Maybe Xbox people in Jersey will still vote, since they can do email due to the hurricane.  But look for a lot of Master Chief write-ins.
    JohnK.
    This is a clever bet by you.  I predict 5-6 states that are within +/- 0.5%.  When you are talking margins like that it is easy to be "wrong" 3 times.  The variation in the polling data is way too large.

    Enjoy the easy $50 you are going to have on Wednesday.  :-)
    Hank
    I was lucky enough to bet a biologist and not an engineer.
    john kehr, want to bet? seriously. contact me at contactgnxp-at-gmail-dot-com

    JohnK.
    Just so everyone knows, I agreed to take the same bet has Hank.  That Silver is off by 3 states.  :-)
    or, contact me via
    https://twitter.com/razibkhan
    https://www.facebook.com/razibkhan99

    please! if you want to make a bet! time's a wasting.

    Thor Russell
    I don't know about it being a clever bet, more like a pointless bet. As I have tried to point out to Hank, but got deleted for the effort, even if the model is extremely accurate in estimating probabilities, it does not mean it will get 2 or less states wrong. I have done the numbers, and if the probability estimates are correct, then there is still a 20% chance of Hank winning. 
    With 5-6 states toss-up like you think, there is a 61/71% chance of Hank winning. If you square the model error estimates, e.g. a 90% prediction made by the model is in reality a 81%, chance then Hank has a 50% chance of winning the bet. So if the model is dead right, Hank wins 20% of the time. If you are right he wins about 65% of the time, and if the model is reasonably wrong and the error is squared, then he wins 50% of the time. Doesn't sound like that smart a bet or easy money. Even if Hank is mostly right, he still has a close to even chance of losing and even if he is completely wrong he wins 20% of the time.

    So whos side would I take on the bet? Well I wouldn't, neither side has that much of an advantage even if their logic/maths/modelling is mostly correct. However it is easy to explain the behavior and flawed logic presented:

    http://lesswrong.com/lw/gw/politics_is_the_mindkiller/


    Thor Russell
    I'd still take Razib's side of the bet. What you don't have the info to account for correctly is that the state probabilities aren't independent, which should lower the odds of states going in separate directions. The model accounts for poll to election differences based on historical data so I don't think there can be significant factors missing.

    50 states right is very unlikely, but 48 or 49 is almost certain imo.

    Hank
    That's what makes it a bet.  Some assertions are silly, namely that no one can get 50 out of 50 correct, even with a perfect model.  Clearly a monkey could get 50 out of 50 correct, just by guessing and so could a model that accurately weights polls results.  

    A perfect model could predict the election based on one response.

    I am not saying Silver's model is perfect but I am not saying he is as incompetent as Thor says he is, either.  And, I keep noting, an election if a one-off event, there are not 100 elections to have the results level out.  Because his model does not account for the errors in polling, it simply takes them at face value and uses a subjective adjustment (though he is a smart guy, his adjustment is a lot more accurate than the average person) it can only be right by luck, in a one-off event. It isn't predictive any more than the polls are, with too large a range to be what anyone in science would consider useful (fine for psychologists and political scientists, though).

    But that is also why I could lose $50.  He could get lucky and get 50 out of 50.
    Thor Russell
    I never said he was incompetent, don't know how you thought that. I am agnostic about him, and the bet most likely won't tell you whether he is competent or not. My points were always about statistical errors and logic. 
    Thor Russell
    Using Silver's own percentages, he would probably predict himself to have <5% chance of getting all 50 right.

    As it looks now he may have Virginia wrong. Florida too, if you use the older version of the map you have in the post above.

    Hank
    Sure, that is the bet.  Obviously a Bayes guy will get the right answer eventually, it is in advance that the model is either good or bad.  The election was never in doubt, virtually everyone called it, but getting the electoral votes and states right is the hard part.
    Yeah, of course the bet ha to be on the prediction as it was when you made the bet.

    lol, Silver is going to be considered some kind of magician after this. It looks like his last prediction did get all 50 right! :)

    Hank
    Everyone got 49 out of 50 and all of the big names who use averaging got 50 out 50.  And, like I said, one statistician who does not have the audience of the NYT got all the electoral votes right - a week ago.  Because he averages polls, he got his results because all the polls said so, not because his model is better.  The ones who did not get 50 out of 50 did not do a Tuesday prediction so they still have Florida as Romney - just like Silver did.

    It is sort of funny that Silver is being adored by seemingly rational people.  It's like the determinism of the 1830s all over again - we are slaves to polling results and don't even have to vote, according to my (admittedly left) Twitter feed, Nate Silver can pick our president.
    ROTFLMAO. Right on as we hippies used to say. Polls are about intentions and are applicable to those polled. Fairly easy to see if those polled resemble those who COULD vote, hard to say what if any connection that might have to the universe of those who DO vote.

    More fundamentally isn't the accuracy of polls or averages thereof, it's about how the intentions of other people are or are not relevant to the reader. I don't care what other people do; no exceptions whatsoever. That's THEIR decision imho. So why would I care about their intentions? When I want to DETERMINE as best I can what they might do, I look first to what I feel like doing and feel good at; then at who would be easiest to manipulate that way. I could care less what their plans are although I will seek info on their DESIRES.

    Hank
    Yes, that is why I note the change in 'cell phone only' people since 2008.  The polling agencies are increasingly reaching a narrower demographic, which means it may not be representative.

    Now, these polling agencies have smart people, they are thinking about this too, but they are only able to calibrate after the election - in physics we got to calibrate against measured data so we knew better how to avoid converging with high accuracy on the wrong answer with high accuracy. So the 2012 election will lead to changes for 2016 but things may be different again then.  Rasmussen, for example, did well in 2008 but has been so far off the mean this year most media companies do not use them. They look like outliers.

    We had one party not all that motivated in 2008 versus one that was - but the latter may be less so now. Polls don't account for that. And Republicans may have closed the Obama 'get out the vote' gap this year, polls don't account for that either.  If Republicans have galvanized people we know are going to vote - gun people and religious people and old people - while Democrats have not galvanized people who could vote, like hispanics, it really changes the way the model works.

    If I am right, then what I said that inspired the bet - "polls are bad models" - is clearly true.

    That is also why I spent time in the article talking about convergence.  If a good model has bad data, it is going to give you a really, really accurate but wrong answer. I think Silver has complete confidence in a model that has converged with high accuracy on a wrong answer in at least 3 states.   That is why I have no idea who will win the election but I am sure I will make $50.
    Hank,

    I also wonder if the lottery principle didn't come into play in 2008 WRT Silver's prediction. There are a large number of people who try to predict the result of a US presidential election. Any of them who are even marginally self-aware are not going to get the vast majority of the state predictions wrong. California, Texas, Illinois, New York, etc. are states that anyone with ANY little bit of knowledge of US politics would have no problem predicting. Thus, it's really not about predicting the winner of 50 states, but more like 8 or so swing states that really decide things.

    Now, let's assume for the sake of argument that NOBODY really knows what will happen in those swing states. That is, all the people predicting the election results are just flipping coins to predict those 8 states. Then the probability of a given predictor being right about at least 7 of these 8 states is about 3.5%. That's certainly not likely, but also not impossible. However, Nate Silver was not the only person attempting to predict the 2008 election. Assume that there were 20 people doing such predictions. Then the probability that NONE of these predictors would get either 7 of 8 or 8 of 8 states right is 48.9%. Therefore, it is more likely that SOMEBODY would actually get 49 out of 50 states right just by flipping coins than it is that nobody would, assuming that there were at least 20 people trying to do such predictions.

    Of course, for an event as consequential as a US presidential election, I would find it difficult to believe that there weren't at least 20 people trying to make predictions about the results. Therefore, it was likely that SOMEONE would get 49 of 50 states right. It is distinctly possible that Nate Silver was not really any better at making such predictions; he might just have been that SOMEBODY who got it right. If his model holds up as well this time around, I would be inclined to change that assessment, but predicting one election right doesn't prove that he's more knowledgeable than anyone else.

    Hank
    Yes. As I noted elsewhere, RealClearPolitics got 48 out of 50 correct in 2008 but, assuming 5 states actually in play in 2008 rather than your 7-8 (Obama was far more overwhelming then) getting 3 of the 5 was only 60% - slightly better than flipping a coin.  That means Silver was 80%, in an easy election, but he was also a political amateur doing it on his own, not a company, so that added some sizzle.

    This one is not so easy so it is a better test of the model.  If his model is really good, as I have said, in a one-off event he should get 50 out of 50.  So I am betting his averaging plus subjective weight is not accurate, so inaccurate he will only be 40% right, but he has an easy out no matter what; he can just claim the state polls were statistically biased and that ruined his model. Statistical prognostication is apparently good work, if you can get it.  Creating a model that works 80% of the time in the engineering world would lead to unemployment.
    Hank,

    Granted that there were really only 5 competitive states in 2008, that just makes my point all the more relevant. Consider that there are only 32 possible outcomes with 5 competitive states. The truth is that there were MANY amateurs who made such predictions, so it's even more likely that someone would get it right, or nearly so. We remember Silver's prediction because he was right; we forget the rest. It is possible that Silver was just the one who got lucky last time and that his model really isn't any better than any other prognosticator's model. Obviously, if he can repeat that performance, that would give evidence that his model is a good one.

    Hank
    Right, and that is the bet, really.  Razib has confidence in the science of polling and in Silver specifically.  I do not, I think he got lucky.  Rasmussen got a great scorecard for consistency and accuracy in 2008 but this year no network uses them because they are so far from the mean. However, if they are so far off and right it is a much different story for them.

    While every election someone gets credit for being accurate even campaign insiders don't rely on them.  The joke is that 'polls are accurate, except in the elections of 1992, 1994, 1996, 1998, 2000, 2002, 2004, 2008 and 2010. All those years had someone accurate, just not the same someone.
    I really enjoyed reading this article, Hank.

    Thank you! I learned something= convergence

    PS...sort of a tangent .....I voted Yes on 37. The last Pepperdine poll showed Yes on 37 to be losing, yet, in spite of every newspaper quoting it, the Pepperdine website doesn't list the methodology and the margin of error for the poll.

    http://www.aapor.org/Poll_andamp_Survey_FAQ/5511.htm
    I trust that you were correct when you predicted it will win 53-47? :-)

    Hank
    I think my projection is accurate. It's unfortunate, but the only thing up for a vote in California is how much higher our taxes will be and how anti-science we will be.  But it's not the end of the world.  The lawsuits will keep Prop 37 from going into effect until another referendum repeals it.

    It just costs us a lot of money.
    Looks like Silver nailed all 50 states this year. Still just lucky?

    Hank
    3 are undecided but I am certain to lose the bet - how do you conclude he nailed all 50 yet?  Who has called all 50 states? 

    It isn't just Silver, Intrade is just as accurate and that uses no polling at all.  CNN and NBC nailed them all so far too.  It may be that polling is really good this year.  Or America is so polarized there are no tossups any more.
    CNN has called all but Florida, and it's unlikely that Romney will be able to catch Obama there. Intrade had Obama at 33% in Florida as of this morning. I don't know what you are referring to on CNN or NBC. I don't think they made any projections ahead of time.

    Hank
    Well, if Florida goes Obama then he is still not 50 out of 50, he had FL for Romney. He's doing a Bayes analysis, that is why we had to pick a spot for his projection.  If he runs his model right now, it will get everything right because that is what a Bayes model does.  

    However, there is no way I am not out 50 bucks.  But a whole lot of people will have 49 out of 50 this year, including Intrade, which is just people betting, no numerical model. The only people proclaiming him a science of polling wizard are people who only read the New York Times.
    Hank
    Emory University's Drew Linzer and his Votamatic is the big winner. He got all 50 states correct and predicted 332 electoral votes when Silver was predicting 307. 
    You still don't get it. Nate showed 332 as the most likely outcome with over 20% likelihood. 307 was the central tendency. Nate nailed it.

    Hank
    Everyone nailed it. Razib could have used anyone and beat me.  Almost everyone got 49 out of 50 and Florida is still up in the air, according to Florida  - and Silver changed his Florida prediction the day of the election.   That is not predictive. Almost everyone else also had Florida going for Romney just like he did, but didn't do a new model the day of the election.

    The test was to see if averaging polls worked and Silver was he test case because, let's face it, science media is leftwing and we all like the New York Times.  To be predictive, you have to predict. The graph in the article is from 2 days ago and it showed he got the same number correct as 2008, except this year everyone outside Fox News writers got that result. If we go back a week Silver's results are even farther off.

    However, Drew Linzer of Emory University is the one who really nailed it. He predicted 332 votes a week ago.  But he is an academic who does not write for the Democratic paper of record, so he is not being hailed as a 'wizard'.

    You seem to be saying we don't need to hold an election at all, we could just poll a few thousand people and average them and pick a president.  I think that would be real science.  If we can do that for a few thousand, eventually we can do it using one person, which used to be a science-fiction concept.

    Gerhard Adam
    If we can do that for a few thousand, eventually we can do it using one person, which used to be a science-fiction concept.
    Hmmm ... not really.  I believe such one person systems referred to the individual as King or Emperor.
    Mundus vult decipi
    Hank
    I am not confusing science with policy. If, as claimed, a statistical wizard can predict a winner with small samples of people he can do it with one, it just needs a better model.  That is not to say we should do so.
    JohnK.
    :-D

    I got a paypal bill from Razib at 1am.  It has been paid in full. 
    Hank
    I paid mine last night while he and I discussed live on Twitter.  A strange election.  Everyone but me (and you) got this right. When even Europeans betting get 49 of 50, there is no election any more.

    I don't think the polls are any better, I think Americans have become more entrenched.
    "Everyone nailed it. Razib could have used anyone and beat me."

    Sorry but you're hedging. You didn't call everyone out. You called Nate Silver out. This wasn't about who called it the earliest. This is about Nate and how well Nate's predictive model worked.

    Hank
    No, he used Nate Silver, I sent him Rasmussen as my test case (you didn't read my article), because I knew how far off they were from the mean and he said he would be able to successfully normalize it.  But it was too much effort, he said.

    Anyway, Silver did not do anything special since, as I note in another comment, everyone who used averaging and did a prediction Tuesday morning got 50.  The ones who got 49 didn't do a new model and, as you see in the bet, Silver only got 49 too if you go back those 2 days. 
    Too funny. Apparently crow doesn't even taste good at $50 a plate.

    Be a mench and just own up to your mistake instead of also minimizing your opponents success by saying 'oh anybody but I could have done it.. and in fact they did!'. Unfortunately for many of the right-wing pundits, and we now clearly have to count you among that group, the polls proved very accurate - and all the more so with proximity to the actual election date. Silver used those very accurate polls and made the right calls. You (plus Dick Morris and all the rest on the right claiming in accurate and biased polling) got it spectacularly wrong.

    Hank
    I paid the 50 bucks before the voting even closed in California.  I took my shot that people were not as polarized as polls claimed and lost.  

    I didn't say polling was biased against Republicans, not one time and not ever so I don't get your rant about Dick Morris but, again, people who have not read this article are injecting their own bias.  As I said, Europeans betting on the election got the same accuracy Silver got so his model was nothing special - they simple believed the polls and I did not.  I do agree the polls were accurate, but he did not poll anyone, he just averaged someone else's work.  Everyone who averaged and did a prediction Tuesday morning also got 50 of 50.

    Re-reading your article again - you are correct about my bias comment. I read between the lines based on your singling out of a writer from the New York Times (Silver) even though 'everyone' made the same call, your preference for Rasmussen polls, and your comments after the article regarding leftwing media.

    In the article itself you never explicitly stated, and apparently did not intend to imply, a leftwing bias in the polls. Maybe you thought the polls were randomly skewed or even skewed in Romney's favor (?) I was wrong about that one. I lumped you in with Dick Morris who boldly predicted just days before the election that the polls were all biased, Romney would actually win in a landslide, and that he would even carry Pennsylvania (!). I admit it.

    See how easy it is? I made a mistake and owned up to it. =P

    Don't hedge. Just say flat out that Silver was right, his underlying data was right, and he did a good job. Readers will respect you more if you don't minimize and hedge every failure.

    Hank
    Well, Drew Linzer was right, he predicted the electoral votes correct when Silver was still off by 33. Sam Wang was correct, he had 99% when Silver had 75%.  As I wrote in my look at why people are gravitating to Silver (and you exemplify it), it isn't because he is accurate, it is because right-wing media were attacking him.

    Those of us who do not watch right- (or left-)wing media had no way to know that, we just look at polls and averaging as a shoddy way to make a prediction.  My belief (incorrect) was that Americans were not entrenched and therefore the polls were incorrect and the election would have surprises.  I did not care if polls were oversampling Republicans or Democrats more, as long as they were oversampling someone more. And they weren't.  On Tuesday he went by what the polls said and they were correct, as was he. He did no polling himself so he can't be 'right', he just averaged.  Someone else did the work creating the polls.

    But he wasn't exceptional, Americans were unexceptional, to such a degree that we could have taken 1000 people composed of demographically accurate representations of married people, single people, minorities, religious people, athiests, the young and the old and had just them vote and they would have gotten this same result.   No one is 'in play' in 2012.
    Very interesting stuff, Hank. I read the article that Razib posted yesterday (before I came here) and I was delighted that you had posted about this as well. I enjoy trying to understand the ways that the minds of smart people work. The reasons why each of you took your position and the logic you each used is instructive.

    Hank
    I am contrarian by nature and it cost me $50.  The next time we have everyone saying how polarized and predictable the electorate is, I am keeping my mouth shut.  :)
    Contrarian or not, I don't think you could buy the exposure you got for only $50.

    Hank
    I got no exposure that I know of. I can make a statistically significant claim I sent more readers to the NYT than they sent to us.
    You said: " As I wrote in my look at why people are gravitating to Silver (and you exemplify it), it isn't because he is accurate, it is because right-wing media were attacking him."

    You've made some faulty assumptions about me (and I think about a lot of other people) just as I made a faulty assumption about you: Until I saw your comment, I really had no idea that right-wing media *was* attacking Silver. In fact, yours is the first and only anti-Silver piece I've read to date.

    Silver got my attention because of his recent book 'the signal and the noise', his appearances on entertainment (not news) talk shows to promote the book, and more recently because of his highly publicized NYT blog statements assigning very high probability (80%-90%) to Obama winning when I would've guessed - just based on how close all of the national and battleground polls were - that it was still closer to a 50/50 race. And that is ultimately why Nate Silver's work has value. at least to me: it is informative and entertaining because he states and publicizes predications that are definitely NOT in line with what most people are saying. The 'everything you think you know is actually wrong' genre is getting a little played out but.. I still enjoy it and I find it informative. That's why Nate Silver is getting attention, at least from me.

    As far as American voters being 'in play' that is fundamentally and demonstrably incorrect based on the way the polls (which you now accept as accurate) shifted following key events in the timeline. If the election were held shortly after the first debate, Romney might've won. After three debates, Sandy, and the Romney's tragically erroneous commercials in Ohio (about Jeep), the polls were trending back to Obama.

    There are many other ways Romney could've won but most would involve pushing back harder against some of the more radical statements by the tea party and pro-life elements of his own party. The GOP only needs to move more to the center and large groups of voters are back in play.

    You said: "Americans are unexceptional"

    How dare you sir! Where is your giant flag pin? Why do you hate America?!? ;P

    Okay. I'm done playing. If you want any more posts you'll have to put me on the payroll.

    Hank
    Silver got my attention because of his recent book 'the signal and the noise', his appearances on entertainment (not news) talk shows to promote the book, and more recently because of his highly publicized NYT blog statements assigning very high probability (80%-90%) to Obama winning
    It may be why people are annoyed. If they saw him on TV he must be accurate. Unfortunately, that is not true, as Drs. Oz, Chopra and the other Four Horsemen of the Alternative show us every day with their alternative medicine and scare journalism claims.

    So what we don't know is that Silver's model works any better than anyone else - I was barely even looking and I found two statisticians more accurate and making their predictions well before he did, they just do not have the benefit of a $2 billion media company behind them.  What we do know is that getting a blogging gig at the NYT will sell a lot of books and thus get you on TV and thus seem more credible.
    You seem to make a number of incorrect statements -- for example, that the election had not started when you wrote. But early voting and exit polling do give indications, even with bare assumptions that Democrats vote for Democrats, etc. The bias in polls, for example, landlines vs. cell phones and age distributions, etc. are well known and corrected for. You can argue about how well that is done, but your piece appears to display considerable ignorance of what is actually done -- and you only needed to listen to a few hours of jabber on CNN to learn about these things.

    Hank
    Saying these things are 'corrected' for...well, that was the bet, wasn't it?  There was no way to correct for that with what anyone in science would consider accuracy and just because the answers were still right does not mean the method was -  'you only needed to listen to a few hours of jabber' in math class to learn that.  What polls did get right is  that people are more divided than ever.  There were not 9 battleground states, there was one. Deifying polling and statistics is not warranted but a look at what is going on in the electorate is. This is not the America of even the 1990s, it is the America of the 1960s.

    Since the election was all settled science and deterministic in advance, as you discovered watching CNN, how many people do we actually need to bother with voting?  Could we just do a representative election, since the result was the same as the polls? If we could hold an election for 12,000 people who have landlines and correct for that it would be a lot cheaper.
    Hank
    We need to keep in mind the issue is whether or not averaging polls can accurately make a prediction.  Here is a screenshot from the day of the bet:



    As you can see, 2 days before the election he was at 307 electoral votes and the week before that he was at 295.  In order him to be a wizard, as opposed to Drew Linzer who nailed 332 the week before the election (but unfortunately for him he does not write for the New York Times, apparently all that is needed in the 'appeal to authority' world of Twitter) we have to assume that:

    (1) A whole bunch of people changed their minds on the day of the election, versus 2 days earlier, or

    (2) Polling was not very good 2 days and a week before the election but was somehow accurate the day before.

    Which of those two makes sense?  The entire east coast was voting Democrat last June so the hurricane did not change anything. No one cared about climate change during the debates or the conventions so the hurricane's being linked to climate change by science bloggers did not sway any votes by undecideds. Silver himself said the voting was not changing by then, so why did his prediction? "But it’s now the weekend before the election. The vast majority of voters are locked into their choices. In some states, in fact, a fair number of them have already voted. (Perhaps about 20 percent of the vote nationwide has been cast, and the tally may be as high as two-thirds of the vote in some states like Nevada.)"

    If polls are only accurate the day of the election, isn't the NYT an unregistered PAC, since they were calling the election for 5 months before the election and using waffling terms like 'polls' and 'social media trends' as justification?

    I didn't want to bet on electoral votes, as I wrote in the article, because that would be too easy, and he was even more off this year than he was in 2008, so why are people contending he 'nailed' it? 
    People are contending that he 'nailed' it because he called 50 out of 50 states. He missed on one senate race. That's what I'm hearing, anyway.

    Hank
    He got 49 of 50 right 2 days before the election (that is the result I bet on) but so did everyone - as I noted in other articles, other people got 10 out of 10 close Senate races (he got 8) and all of the electoral votes well before him so, really, the reason he is getting so much attention is because he writes for the New York Times and that made him a lightning rod for criticism by the political opposition of NYT readers.  He didn't herald in some new Age of Statistical Reason by averaging polls, they just like declaring they are smarter than Republicans and find confirmation bias to strengthen it.

    If the 'secret sauce' models are good, they will get all of the Congressional races too, but they don't.
    He got 50 out of 50 right of the actual election, which I believe is what people who are saying he 'nailed' it are looking at. I don't know how many people knew of your bet, but the number that knew about Nate Silver's results of the actual election is probably higher. If there were more, and more frequent, polls of Congressional races, the aggregate would be better at predicting those races, too.

    Hank
    They don't poll less but perhaps people answer the phone less.  But 'who owns a landline phone' was supposedly well-controlled for in polls for the last 8 years so if the model is accurate it should get smaller races correct also.  A good model is a good model.

    Everyone who did a model election day got 50 of 50, just averaging polls.  As I mentioned, Europeans who know little about polls got 49 of 50 on a betting website the day before.  So, yes, his attention is due to writing in the NYT, not because his model was better. Anyone who got Florida right won a coin toss, statisticians all agree, so that isn't a superior method.
    "Who will win the election? I don't know, it is crazy to bet against Obama's get-out-the-vote machine but that is how fortunes in polling are made. I just know I will be $50 richer."

    But you're not. You're $50 poorer. Were you wrong, or not?

    Hank
    I was wrong, obviously, I lost 50 bucks betting against polls in an election that was no contest. Every poll was right. every statistician was right.  The only people who did not get 50 out of 50 did not do a prediction the day of the election and they were still 49.

    The question is, does it make all the models right?  Everyone who claimed their model was better than the other people tied.  Hard to say to say if any models are right, we just know the polls were right.  But if the models and predictions are as accurate as they are being claimed, we should see 435 Congressional elections and 33 or so Senate races nailed in 2014.