Banner
    Taming The Look-Elsewhere Effect
    By Tommaso Dorigo | September 1st 2011 06:15 AM | 18 comments | Print | E-mail | Track Comments
    About Tommaso

    I am an experimental particle physicist working with the CMS experiment at CERN. In my spare time I play chess, abuse the piano, and aim my dobson...

    View Tommaso's Profile
    Note: I have discussed today's topic in one of my best articles here some time ago, and I also gave even more technical insight in another piece. I decided to revisit the topic once more under the stimulus of a online HEP magazine, which is going to feature a text of mine soon. They do not care if I use the same text here too, so you get to read it here first.

    A dangerous beast is hiding in today's searches for new physics -or even for "old" physics, such as the Higgs boson- at the Large Hadron Collider. It is called "Look-Elsewhere Effect", LEE for insiders. What is it, and why should you care ?

    Imagine you look for a heavy particle decaying to a pair of hadronic jets: a commonplace test case in high-energy physics. You have your background model, which predicts the observable shape of the dijet mass distribution, and you know what kind of a bump in that shape a new particle signal would produce. So you search for such a bump in the data, but -not knowing where it might appear- you search everywhere.

    You have worked all day, and the night is nearing; you prepare yourself a Martini and spin your analysis program. To your amazement, the program finds a significant bump at some particular mass value: is it a real signal?

    To claim it is a new signal, the effect must reach or exceed the "five-sigma" significance level, five standard deviations away from the expectation: that's a rather silly but well-established rule. But if yours gives only 3.5 or four sigma, are you allowed to get excited and wake up your boss, or should you sit back and sip your Martini, with a "I know better" grin on your face ?

    I claim the latter is a better option. You have fallen prey of the LEE: you looked in many places for a possible signal, and found a significant effect somewhere; this happens more often than it would if you had stated beforehand where the signal would be, because of the "probability boost" of looking in many places. A good rule of thumb is the following: if your signal has a width W, and if you examined a spectrum spanning a mass range from M1 to M2, then the "boost factor" due to the LEE is (M2-M1)/W. This may easily amount to a factor of 10 or 100, depending on the details of your
    search. An effect occurring by chance once in ten thousand cases in a given place of your spectrum may actually be just a unexciting one-in-a-hundred fluctuation!

    In fact, the "5-sigma" rule I mentioned above was conceived with exactly this particular effect in mind. Five sigma is a really, really rare occurrence (three in ten millions), and even including the LEE, plus considering non-Gaussian tails in measurement systematics (the other worry that kept the significance bar high in finding a working point for an "observation" claim), it is still something to take quite seriously.

    Nowadays the annoying persistence of the Standard Model has brought us to seek compromises. We cannot grow old waiting for five-sigma signals, so we content ourselves with publishing 3-sigma ones; yet our scientific integrity demands us to account for the LEE. This is actually less easy to do than just multiplying a probability by (M2-M1)/W as in the example above: in complex searches such as that for the Higgs boson, which combine bump hunts in many channels, this is actually quite a headache.

    A recent paper by Eilam Gross and Ofer Vitells (Eur. Phys. J. C70:525-530,2010) has clarified some of the technical issues. The searches for the Higgs boson by ATLAS and CMS nowadays size up the LEE by studying the probability of the background-only hypothesis as a function of the Higgs mass: the more the observed p-value distribution wiggles up and down as the signal mass hypothesis change, the stronger is the "trials factor", i.e. the required Look-Elsewhere-Effect correction. Another important thing to keep in mind is that the trials factor grows linearly with the observed significance (see figure below), a fact which had been overlooked in the past.

    All the above is stuff for experts, for sure. But outsiders have better be aware that a three-sigma effect should not be blindly dubbed "evidence" for something new in the data. As travelers to a foreign country whose tax habits are unknown, you better ask before you buy, "LEE included or not" ?


    Above: trials factor due to the LEE in a idealized bump search as a function of the observed significance Z_fix of a signal (blue curve). The growth with significance of the trials factor matches at high Z_fix the result of an analytical approximation (red dashed curve);  the black curve shows the upper bound of the trials factor. For more details see Eur.Phys.J.C70:525-530,2010.

    Comments

    Can these:

    http://vixra.files.wordpress.com/2011/08/wwslideeps.jpg

    have different LEE "boost factors"? Assuming the usual Brazil band LHC combined plots do not take LEE into account, would differing boost factors in different channels have an impact?

    dorigo
    Hi JJ,

    the answer to your question is most certainly no. The trials factor in Higgs searches depends almost entirely on the mass resolution, which does not change in a Cut&Count vs MVA analysis. The residual possible effect (the reason why I wrote "almost") is that different analyses (which get combined in those limits) may have a different relative weight in the combinations, and this could in principle remain.

    In general, however, I need to point out that there is no look elsewhere effect in upper limits. It only applies to the small p-values dealt with in computing significances.

    Best,
    T.
    Thanks for the explanation.

    Dear Tommaso,

    Can you give an idea what is the width of the Higgs signal expected versus the mass range examined in a "typical" search at the LHC (if there is a "typical" search), and how the hypothetical probability of background-only can be tested, or is this asking too much (since I don't know what I am talking about)?

    dorigo
    Hi David,

    the natural width of the Higgs boson is a steep function of the particle's mass:


    That said, this is not what we reconstruct in typical analyses. In fact, the experimental resolution dominates for M(H)<250-300 GeV, depending on the final state sought. For H->gamma gamma searches in the 100-150 GeV range, the typical observable width is in the few GeV ballpark; for H->tau tau searches it is O(10) GeV in the same range. For H->WW decays the mass resolution is much poorer, but for H-ZZ it is of the order of 10 GeV in a wide range of Higgs masses. You can check these details in a ATLAS web page with mass histograms of the various searches.

    Cheers,
    T.
    The Stand-Up Physicist
    Most excellent to hear that we who watch on the sidelines should just sip Martinis on our beach chairs. For those of use rooting for a strikeout, a similar line of logic would hold, no? Waiting for a five sigma "no" over the entire >100GeV range is a simple game to play, glass in hand. I prefer Johnny Walker Black to a martini if anyone is buying.
    couldn't one get rid of look else effect by just dividing the sample in two? i.e one has 1 fb-1 collected during a year and a bump in it, then after dividing into separate samples colected in different times one can naively assume to have bump/sqrt(2) in both samples. if that is the case it is not LEE caused, is it? or one needs to take into account the possibility of having LEE accidentaly in the same place (in both smaples)?

    dorigo
    Excellent question, which draws a non trivial answer.

    First of all: the significance sharing of the two subsamples depends on the statistics we are looking at. If we are talking of a 1000-event signal in a 101k-event set, then yes, this 3-sigma signal should split into two 500/50.5k samples of 2.2-sigma each, and there's not much uncertainty in this expectation since sqrt(500)/500 is less than 5%. But if you are discussing a 12-event signal in a sample of 28 events, with 16 expected background (again a 3-sigmaish effect, if B=16+-4 is the bgr expectation), then the two subsets you get by dividing in two the data may contain really widely different signal fractions.

    In any case, there is an additional problem in the procedure. Once you observe a bump somewhere, the bump will likely (see above how much) appear in the two subsamples too. These two sub-sample significances will still be "local" significances, i.e. will not tell you how likely it was that you observed a bump anywhere in the spectrum. They still require a look-elsewhere correction.

    A different problem instead arises when the proposed procedure is modified: suppose you see a 3-sigma bump in a 1/fb data, and then say, fine - I'll forget about these initial 3-sigma and concentrate on the next sample of data, where I now can happily look exactly where the first set told me to look. This latter search is not affected by the LEE, is it ? Well, yes, it still is. That is because of the fact I mentioned in the post above, that the LEE trials factor grows with the significance of the bump. When you look in the second dataset you will not fix the mass of the signal to exactly the value found by the first search, because that will have a small uncertainty. Instead, you will allow it to float slightly around the input value. The returned significance -and signal strength- on the new data will be still biased high, because the fitter will be increasingly capable to take advantage of the small degree of freedom, as the signal strength and true significances grow. This is a subtle effect, and it would require a post of its own. In fact, it is tightly connected with an observation I made ten years ago, and wrote about it in a CDF note. I wrote on the effect in a series of posts a while ago, see here for a starter.

    Cheers,
    T.
    xkcd.com/882/

    Best explanation ever. (Not to say that yours is not good Tommaso).

    I mean http://xkcd.com/882/ of course.

    Good one! ;)

    That's awesome. I don't have the technical expertise necessary to add insight into "five sigma" stuff, but I have been forced into the position of being an amateur scientist by all the "erratic data points" I was observing in my own work (mathematical theology). I say I am a careful experimenter; if you're saying the same thing only differently - slainte!

    Tommaso, do you happen to know why it is called the "look-elsewhere effect" and not the "random fluctuation effect"? Whenever I hear the term, I always think that people who hear it for the first time will think "I _did_ look elsewhere, and I still found this bump!" Wouldn't "random fluctuation effect" describe it better?

    dorigo
    Well, Anon, maybe. But the term is kind of hinting at what the experimenter is doing, so in a way it is a description of the whole procedure that produces the effect. I don-t know about "random fluctuation effect", it is too vague. It is all random flukes anyway ;-)

    Cheers,
    T.
    Hi Tommaso,

    ... the trials factor grows linearly with the observed significance...

    I didn't quite understand this. Does that mean that if I observe a 5 sigma significance, I must correct more for the LEE than if I observe a 2 sigma significance?

    Do you expect more publications in hep which include an LEE calculation?
    After compensating for LEE would you say that 3 sigma is enough to credibly establish a new particle?

    dorigo
    Hi Martin,

    yes, the reason for the trials factor growing with the significance is that a higher significance bump has a larger number of "effective regions" in the spectrum where it may appear. It is nicely explained and derived in the paper, which you can find in the arxiv as well (1005.1891v3).

    I expect that subtracting the LEE from the local significance will become a rule when claiming a signal. I also expect that CMS and ATLAS will converge to a less restrictive threshold for "observation" than five sigma, for fully-corrected significances, if systematics appear well under control.

    Cheers,
    T.
    Hi Tommaso,

    As discovering Higgs is probably one of LHCs most important targets, do you think that LEE will be used for that?
    Does LEE have to be used differently for estimating sigma if you want to rule out Higgs existance. I wonder whether there might be an anti-LEE if you have to rule out a particle's existence everywhere, i.e. an ROE (Rule out everywhere) effect.

    dorigo
    Hi Martin,

    the LEE will certainly be used (it is being used) for estimating Z-values (i.e. corrected significances) even in the case of the Higgs boson. However, for exclusion, there is  no LEE, since each mass point you consider is a different theory you are testing, and a different claim you make. We know, and it is built in the procedure, that 5% of those claims might be wrong, even in the absence of a Higgs boson.

    Cheers,
    T.