Guest Post: Chris Hill, Who Is Your Savior ?
    By Tommaso Dorigo | November 10th 2012 09:09 AM | 15 comments | Print | E-mail | Track Comments
    About Tommaso

    I am an experimental particle physicist working with the CMS experiment at CERN. In my spare time I play chess, abuse the piano, and aim my dobson...

    View Tommaso's Profile
    I read the text below in the Facebook page of a colleague and friend, Christopher Hill. The text was meant as a facebook rant - sort of - but it raises important points. I liked what he wrote and I asked him if I could repost it here to the benefit of a larger audience. He graciously agreed. NB: the title of the post is mine.

    Now that the election and Sandy is (sort of) over, I want to post a rant that I didn't think was appropriate before now.

    First, Nate Silver. I have heard him called everything from a genius to a wizard to a witch in recent days. He is none of the above. He is a guy who understands mathematics (in particular statistics) enough to be able to use it to predict elections.

    Of course, he also needs to do research, and understand political science, etc. But the thing that distinguishes him from the hundreds of other people that understand those things but yet can't predict anything reliably is his ability to really *use* more than just basic math.

    This when viewed by the innumerate masses, including the punditry, can obviously only be explained by supernatural means. Well, you know who else uses math that most people can't comprehend to predict things even more complicated than elections? The scientists at NOAA (and elsewhere, particularly in Europe) who warned everyone that a terrible storm was brewing that could be catastrophic for the east coast.

    The reason this storm wasn't (even more of ) a tragedy was this warning that enabled people to evacuate or otherwise prepare. This was not wizardry either, just science based on mathematics.

    After the election, I have heard many people say they will pray for a better future since we are in such dire straits. Fine, it is a free country. But remember this - people prayed for thousands of years when hurricanes surprised them and wrought havoc on their communities to no avail. The thing that actually saved lives this time around was mathematically based science (which just happens to be the same reason I can type this on facebook instead of writing it on vellum like a monk would have done in the dark ages).

    To bring this full circle, I've seen many headlines which have said Nate Silver was the real victor of this election. I hope by this they mean mathematics, and science based reasoning. If so, then maybe more people will understand the value of these things (instead of deriding them or their conclusions for political expediency), study them deeply enough to so that they are no longer magical, and work hard to use them to extract real truths about the world (as opposed to vacuous truisms derived from pre-conceived dogma) and have the courage to confront these facts and solve the problems facing our country and the world.

    If so, then our future is indeed bright. If not, well then I pray for all of us.


    Some of the Silver hype is because Republicans criticized him for predicting a victory for the president in the New York Times. Other statisticians were more accurate but they do not write for the New York Times and so he has been beatified as some sort of science guru by Democrats - but he didn't do any science at all.   He averaged polls that companies took and gave each poll a 'lean' based on his instinct.  But his instinct was no better than the other people who just averaged polls without any lean.

    Polling is not science, any more than Bayes inference and statistics is science. People who should know better have gotten awfully reductionist about polls. If averaging polls is science-based reasoning and accurate, why do we need elections?  Just poll 12,000 representative people and declare a winner.
    In fact there's a short story by Isaac Asimov where they decide on elections based on a very well-chosen single voter. Worth reading!

    You are very unfair to Nate Silver when you say that he just gave each poll a lean "based on his instinct". What he did do, among other things, was to look at how different polls have faired in previous elections and then adjusted their weight and bias according to that. In that way he was able to make better predictions. Using empirical data to improve a model that helps us make better predictions sounds very much like science to me.

    If that were true, his lean would not change for polls every time a new one came out.  He factored in how much lean to give each from week to week, which is pretty subjective.  He is transparent about everything except how he determines lean - because it is subjective, as I said.

    Again, I have nothing against the guy, but a large part of his soothsayer reputation is because he writes for a large newspaper, the media has ignored the actual statisticians who were more correct and did it all sooner.
    I knew Chris Hill when we were undergraduates at MIT far too many years ago. I do believe he's the most brilliant person I ever personally knew, and one of the very few who did *exactly* what he originally set out to do in life.

    In any event, he is quite correct: We need to base public policy on facts and whatever light reason, science and mathematics can shed on those facts, not on political or religious dogma (which in the Republican Party have almost become one in the same). With one of the two major parties openly pandering to ignorance, America is more anti-intellectual than at any point in my lifetime. It's a worrisome trend, to put it mildly, and while it is worst on the political right, it is by no means limited to the right as you have plenty of people on the left who do such things as not vaccinating their children and refusing to think rationally about nuclear power.

    The scientists at NOAA (and elsewhere, particularly in Europe) who warned everyone that a terrible storm was brewing that could be catastrophic for the east coast.

    So you're saying that NOAA is full of witches and they're responsible for Sandy? I knew that! Burn them all!

    On a lighter note about statistics/probability see:

    @Anon: I agree about Republicans (those backwards foaming-at-the-mouth morons).

    I always improve my self-esteem and confidence whenever I realize how much better I am than those silly peasants with their 'belief' and 'religion'. Then I go to internet blogs written by like-minded people and we all relate to each other how much brighter we all are than everyone else. We feel very good about ourselves. Sometimes I make sarcastic or satirical comments on Facebook that belittle dumb Americans. That is also very nice for me to do. It makes me feel very good.

    Fred Phillips
    Hank, dude, Chris is on the side of science-oriented government. Why trash him? Even if he took the wrong route to get to the right view?
    I have no idea who Chris is, but I am sure I never mentioned him, much less trashed him. Regarding Silver, he did what every person in math class has done; gotten the right answer using the wrong method.  It isn't a big deal, but he should not be put on a pedestal because he writes for a newspaper, while other statisticians did better work.
    Can you please recommend some statisticians and where they put their results?
    I'm especially interested in the ones you mention previously that published stuff before Nate and are more accurate.

    I read Nate's stuff, but if there are better sources please let us know so we can all benefit.

    Thank you.

    The one who really nailed it was Emory's Drew Linzer.  Sam Wang of Princeton also.  What both of those academics lacked is writing for the $2 billion New York Times conglomerate. 
    Can you substantiate your claim that Linzer and Wang had better results than Silver? Also, Silver has been doing political predictions since before 2008, and he got attention before he was on NYT. And you're absolutely wrong about the house effect calculation - although Silver does not publish his algorithm he specifically states that it is a calculated quantity, not a hunch.

    You aren't willing to go to their websites and look at their results so I am not sure how much help I can be.  Silver was a baseball statistician until the Democrat primaries of 2008 and he started doing his own averaging because he didn't like that Obama showed poorly compared to Clinton in polls.  That is well known.  His 'calculated quantity' is a lean, just like I said - he determines what lean each poll has, there is no objective measure.  That makes it a hunch.
    Fred Phillips
    WASHINGTON (The Borowitz Report)—Opting for a bold “big tent” strategy to rebuild the party, Reince Priebus, the chairman of the Republican National Committee, told reporters today, “We need to welcome people who believe in different things than we do, like math and science.”