Note: I have discussed today's topic in one of my best articles here some time ago, and I also gave even more technical insight in another piece. I decided to revisit the topic once more under the stimulus of a online HEP magazine, which is going to feature a text of mine soon. They do not care if I use the same text here too, so you get to read it here first.
A dangerous beast is hiding in today's searches for new physics -or even for "old" physics, such as the Higgs boson- at the Large Hadron Collider. It is called "Look-Elsewhere Effect", LEE for insiders. What is it, and why should you care ?
Imagine you look for a heavy particle decaying to a pair of hadronic jets: a commonplace test case in high-energy physics. You have your background model, which predicts the observable shape of the dijet mass distribution, and you know what kind of a bump in that shape a new particle signal would produce. So you search for such a bump in the data, but -not knowing where it might appear- you search everywhere.
You have worked all day, and the night is nearing; you prepare yourself a Martini and spin your analysis program. To your amazement, the program finds a significant bump at some particular mass value: is it a real signal?
To claim it is a new signal, the effect must reach or exceed the "five-sigma" significance level, five standard deviations away from the expectation: that's a rather silly but well-established rule. But if yours gives only 3.5 or four sigma, are you allowed to get excited and wake up your boss, or should you sit back and sip your Martini, with a "I know better" grin on your face ?
I claim the latter is a better option. You have fallen prey of the LEE: you looked in many places for a possible signal, and found a significant effect somewhere; this happens more often than it would if you had stated beforehand where the signal would be, because of the "probability boost" of looking in many places. A good rule of thumb is the following: if your signal has a width W, and if you examined a spectrum spanning a mass range from M1 to M2, then the "boost factor" due to the LEE is (M2-M1)/W. This may easily amount to a factor of 10 or 100, depending on the details of your
search. An effect occurring by chance once in ten thousand cases in a given place of your spectrum may actually be just a unexciting one-in-a-hundred fluctuation!
In fact, the "5-sigma" rule I mentioned above was conceived with exactly this particular effect in mind. Five sigma is a really, really rare occurrence (three in ten millions), and even including the LEE, plus considering non-Gaussian tails in measurement systematics (the other worry that kept the significance bar high in finding a working point for an "observation" claim), it is still something to take quite seriously.
Nowadays the annoying persistence of the Standard Model has brought us to seek compromises. We cannot grow old waiting for five-sigma signals, so we content ourselves with publishing 3-sigma ones; yet our scientific integrity demands us to account for the LEE. This is actually less easy to do than just multiplying a probability by (M2-M1)/W as in the example above: in complex searches such as that for the Higgs boson, which combine bump hunts in many channels, this is actually quite a headache.
A recent paper by Eilam Gross and Ofer Vitells (Eur. Phys. J. C70:525-530,2010) has clarified some of the technical issues. The searches for the Higgs boson by ATLAS and CMS nowadays size up the LEE by studying the probability of the background-only hypothesis as a function of the Higgs mass: the more the observed p-value distribution wiggles up and down as the signal mass hypothesis change, the stronger is the "trials factor", i.e. the required Look-Elsewhere-Effect correction. Another important thing to keep in mind is that the trials factor grows linearly with the observed significance (see figure below), a fact which had been overlooked in the past.
All the above is stuff for experts, for sure. But outsiders have better be aware that a three-sigma effect should not be blindly dubbed "evidence" for something new in the data. As travelers to a foreign country whose tax habits are unknown, you better ask before you buy, "LEE included or not" ?
Above: trials factor due to the LEE in a idealized bump search as a function of the observed significance Z_fix of a signal (blue curve). The growth with significance of the trials factor matches at high Z_fix the result of an analytical approximation (red dashed curve); the black curve shows the upper bound of the trials factor. For more details see Eur.Phys.J.C70:525-530,2010.
- PHYSICAL SCIENCES
- EARTH SCIENCES
- LIFE SCIENCES
- SOCIAL SCIENCES
Subscribe to the newsletter
Stay in touch with the scientific world!
Know Science And Want To Write?
- Kudos To "The Independent" Newspaper For Debunking Nibiru "Blood Moon" Hoax
- Climate Change Has Less Impact On Drought Than Previously Expected
- Dogs Understand Both Vocabulary And Intonation Of Human Speech
- Your Microbiome Did Not Cause Your Weight Problem
- Fewer Cardiovascular Drugs Being Studied In Clinical Trials
- State Of Academic Freedom 2016
- USDA Microbiologist Warns Bacteria In Vaping Products May Be A Health Concern
- "Apparently planet 9 exists. Not Nobiru but another dwarf planet. This planet could destroy a couple..."
- "Thank you. You know, I am of the firm belief that if a blog has a title that includes Science in..."
- "Firstly, I want to inform everyone apart from Kaylee that the following doesn't apply to the topic..."
- "Hello again Robert, Once again, thank you for your informative reply. It's wonderful to still find..."
- "Hi Margaret, thanks, glad you like my replay, and any views are welcome here :). It's only trolling..."
- <a href="/news/2016/08/29/how-herpes-drugs-work-9926">How Do Herpes Drugs Work? <i class="fa fa-angle-double-right"></i></a>
- <a href="/news/2016/08/30/school-meals-eat-tu-principals-9938">School meals - Eat tu Principals? <i class="fa fa-angle-double-right"></i></a>
- <a href="/annual-reports">Annual Reports <i class="fa fa-angle-double-right"></i></a>
- <a href="/news/2016/08/30/some-sugars-help-babies-thrive-9934">Some Sugars Help Babies Thrive <i class="fa fa-angle-double-right"></i></a>
- <a href="/news/2016/08/30/concussion-recovery-time-cut-half-when-athletes-immediately-sit-9933">Concussion Recovery Time Cut in Half When Athletes Immediately Sit <i class="fa fa-angle-double-right"></i></a>
- <a href="/news/2016/08/29/icymi-acshs-most-popular-articles-summer-9931">ICYMI: ACSH's Most Popular Articles of the Summer <i class="fa fa-angle-double-right"></i></a>
- Progress in refining the genetic causes of schizophrenia
- Researchers discover machines can learn by simply observing
- Technique could assess historic changes to Antarctic sea ice and glaciers
- Factors associated with improvement in survival following heart attack
- System may help treat rare genetic disorder, reduce severe side effects