[By the way: if you were coming here to learn the solution of my riddle about the mysterious plot I posted here yesterday, be patient - I will publish an answer tomorrow on that issue.]
As a clear example of the title of this article, take the issue about the upper limits on parameter space points of Supersymmetric theories that the ATLAS and CMS collaboration has produced for the last round of winter conferences. In a series of posts last month I covered the experimental results, and I repeatedly pointed out (albeit rather quietly, without making a big fuss of it) that the method that ATLAS had started to use to set upper limit on signal cross sections -called "PCL", for "power-constrained upper limits"- was, hum, let's say, questionable. Check the content as well as the comments thread of this post, for instance.
Twenty five years ago, the method on which ATLAS PCL is based was abandoned, in favor of techniques that did not suffer of the same shortcomings. The PCL method uses the old frequentist Neyman construction with a CL(s+b) measure, modified by an arbitrary cutoff of unlikely downward fluctuations.
The details of the ATLAS method, as well as the ensuing issue, would be long and too complex to debate, but the bottomline is the following: there are three "main" methods for setting upper limits on signal intensities that are mentioned in the Review of Particle Properties (the statistics section of which is as of late handled by Glen Cowan, who is an ATLAS collaborator). The methods are: a bayesian technique employing a flat prior on the signal strength; a frequentist technique by Feldman and Cousins; and the CLs method, which was used by LEP II searches for the Higgs but has some shortcomings of its own. There was an understanding that the experiments would always quote results employing at least one of those three methods, but this hope was disattended by ATLAS in a couple of recent papers.
By the way, I feel I should say that of all three methods, despite feeling I am a Bayesian deep inside, I prefer the Feldman-Cousins one, which has quite a few nice properties and guarantees coverage, while avoiding some pitfalls of CLs. But let us not discuss my own likings here, but rather the happenings of the recent months.
In one of the posts I argued that the method that ATLAS was using to set upper limits was too aggressive, and yielded too strict upper limits. Using that method made ATLAS look good, but it put the experiment on a league of its own, making its results not comparable to other experiments. A really awkward situation, given that for years physicists had tried to foster a situation where experiments could easily combine their findings. Awkward is an euphemism here.
Somebody in one of the threads argued otherwise, arguing that the ATLAS limits were minimally dependent on the choice of the method they used, claiming that all of the difference with the CMS reach were due to a much better analysis technique, and blaming me between the lines for being an incompetent. Of course a mixture of those claims could be a good representation of reality... And okay, maybe the above were just my impressions. Anyway, I kept my opinion and walked on.
Now, ATLAS has finally come clean with a comparison plot which shows their results on SUSY searches which compares their results using the new method with the ones they would get using the more standard CLs. You can see it below (the document is available here).
As you can see, the expected CLs and PCL limits are quite far apart (compare the observed limits, which is the most meaningful thing to do here: these are respectively shown by the dash-dotted black and full red curves). It might seem like a detail, but we are talking about 100 GeV differences in the lower limit allowed for squark masses (for gluino masses of 600 GeV, which are shown by the horizontal dash-dotted black line at the lower center of the figure) -that's large factors in production cross section! It's just as if ATLAS, by employing a rather aggressive method for setting upper limits, were boasting a two-times higher sensitivity on SUSY particles. Anomalous. Silly. Pick your favourite complaint.
In the future, ATLAS will probably always quote CLs in their upper limits, as a comparison. At least, I hope so... It will make things more clear. Not that I personally mind too much: I believe there are no SUSY particles to find anyway!