Yesterday I gave a lecture at the 3rd International Conference on New Frontiers in Physics, which is going on in kolympari (Crete). I spoke critically about the five-sigma criterion that is nowadays the accepted standard in particle physics and astrophysics for discovery claims.
My slides, as usual, are quite heavily written, which is a nuisance if you are sitting at the conference trying to follow my speech, but it becomes an asset if you are reading them by yourself post-mortem. You can find them here (pdf) and here (ppt) .
(Below is the cover of my lecture slides)
Here I cannot go into the details of the lecture, which lasted 45 minutes and went into quite a bit of detail of the history of the criterion to set the stage for a critical assessment of the few pros and the many cons of having a fixed, and totally arbitrary, threshold for the many searches for standard and non-standard signals, be them indicative of new physics or just confirmations of the standard model. Maybe it is more interesting if I mention here a funny exchange that occurred after my talk, when Eef van Beveren gave his own lecture on the existence of a superlight scalar at 38 MeV (among other things).
During my lecture I had spent some time in "educating" the audience on how easy it is for a random mass histogram to contain an apparently striking upward fluctuation. I used a Monte Carlo simulation of sets of 100 histograms generated from a flat background model, showing how the most significant "peak" in one of the histograms of each set of 100 may easily correspond to a four-standard-deviation significance effect. For example,
For instance, look at the "peak" in the histogram below. Would you be excited by seeing it when you expected a flat distribution ? The trick is that if you did not specify in advance the abscissa at which you expected a fluctuation, this may happen anywhere in the spectrum, boosting the significance of the local fluctuation.
So after my talk Eef van Beveren took the stage, and he quoted my own talk a few times as he went over the evidence for his 38 MeV particle. I was then reminded of a discussion we had in the comments thread of the article I wrote two years ago on his studies. (You can find the article at this link, and you are invited to check the lively discussion which ensued in the comments section.)
By reading the thread, I was reminded that two years ago I had offered a $1000 bet on the reality of that particle. As Eef appears still convinced that there is a particle there, I could point out in a comment after his talk that perhaps, in addition to the "five sigma" criterion for deciding whether a new signal can be trusted or not, one has an additional piece of evidence, more of Bayesian nature: you offer a wager, and if nobody takes it that is strong evidence that the claim is not believable!
In the case of the 38 MeV diphoton resonance, I think the fact that nobody took my bet can be taken as a mild evidence that we should just move on... On the other hand I must say I sympathise with Eef and his attempts at getting experimental collaborations to consider the potential effect and prove or disprove it with their data. I think it's a shame that ATLAS and CMS, who have the data to do that, have not taken the time to do it.