Banner
    Supernatural Coincidences And The Look-Elsewhere Effect
    By Tommaso Dorigo | October 16th 2009 12:54 PM | 23 comments | Print | E-mail | Track Comments
    About Tommaso

    I am an experimental particle physicist working with the CMS experiment at CERN. In my spare time I play chess, abuse the piano, and aim my dobson...

    View Tommaso's Profile
    Have you ever caught yourself wondering, upon observing a seemingly utterly unlikely coincidence, whether there was anything supernatural at work that made it happen ? I would guess that all of us, even the most rational thinkers, have caressed that thought for a minute, at least once. A few typical examples can be given of situations which one apparently fails to ascribe to natural causes:

    1) Grandma dreams of her deceased husband spelling a sequence of numbers, and the following day she sees the same sequence coming out at Lotto. Was grandpa trying to let her win a large sum ? It will be quite hard to convince her otherwise.

    2) You wish a horrible death to a guy who fails to yield at an intersection, and a few miles down the road you drive by his wrecked car in flames. Supernatural powers ?

    3) A nurse claims that the moon causes a unusually large number of births: every time there is a really busy night at the delivery room, she watches out of the window and sees a full moon. An astral influence ?

    4) You visit a remote foreign country for the first time, and bump into your old friend Seymour, whom you had not seen in decades. Destiny ? Whatever it is, this freaks you out, n'est ce pas ?

    The logical explanations

    Granted that complete chance is always a possible answer, there are usually logical explanations to those unlikely occurrences, and they can be ascribed to different effects:

    1) Grandma listened those numbers spelt on the radio in the morning while she was distracted, and involuntarily merged that memory with that of a dream of her husband, which she had the night before. Seeing the same numbers in the newspaper in the afternoon startles her: the effect is involuntarily made up by her declining mental faculties.

    2) The guy you cursed on the road was totally drunk, and his reckless driving was the cause of both your curse and his accident. Among all the drivers that elicit your cursing, the fraction of drunk drivers is very high, and the latter have a much higher chance of causing accidents: this explains the apparent mysterious powers of your death wish.

    3) The nurse watches out of the window every time there is a unusually large number of births, and she more easily records in her mind the "positive reinforcements" of her theory. When there indeed is a close-to-full moon she will get a kick out of it and she will remember that night much better than all those when she looked out and saw no moon or just a slice of it. After a couple of years of practice, nothing will shake her confidence that the moon influences the date of delivery, although her "evidence" is entirely based on biasing herself into picking the successes in a random sequence of trials.

    The three cases above highlight different mechanisms through which a "unbelievable coincidence" may arise in our everyday life: the coincidence may be a fake -intentional or not- as the one due to grandma's poor memory; it may be a hidden correlation between two apparently random variables, one of which "selects" the other, as in the case of the drunk driver; or it may be due to our selective memory, as it happens to the nurse in the example above. Fake, Correlation, and Bias are the three tags of the effects at work, in the cases we have considered this far.

    The Look-Elsewhere Effect

    The fourth case, however, deserves a different discussion. Upon meeting Seymour, you cannot invoke any of the above effects to mitigate the significance of your observation: you are damn sure it was him, since you even ended up sharing a beer and talking about ye good ole times -no fake; you have no reason to believe he followed you, nor are you in that remote bit of land for the same reason that he is (as might happen if you have a common profession and you are there for the same reason) -no correlation; and this is a single event, since you either met an old friend or not in your trip: no selection bias. Or not so ?

    In fact, a selection bias is indeed present, but as is often the case with similar problems, in order to see it you must observe the system from outside. You visited many unfamiliar places in your trip on that occasion; and in general, you travel a lot. Moreover, you would have been just as surprised to meet a famous person, like a well-known actor, or a politician, as you were to find Seymour. So indeed, the question "How likely it may be that upon visiting this remote land for the first time in my life, I meet a good, old friend ?" is very, very ill-posed. You had even forgotten of Seymour, and now you talk as if he is a very important person in your life!

    The right question to ask would instead be "How likely it is that, in the last twenty years of my travel, I ended up meeting a person I knew, either personally or from newspapers or TV, in a remote place far from where we both live or work?", and this must be a one-in-ten sort of probability, not a one-in-a-million like the previous question would suggest! By loosening the net, and by throwing it several times, your chance to catch even just one fish increases enormously.

    The mechanism at work -the negligence of the universe of possibilities that could have yielded a similarly cataloged event- is quite common, and quite relevant for scientific investigations at least as much as it is to model the chance of a cheerful rendez-vous. It is called "look-elsewhere effect". One way of thinking at this effect is that it arises when one focuses on characteristics of the observed phenomenon which are inessential.

    And in particle physics...

    The look-elsewhere effect is very common in physics searches for uncommon events. It is present in all searches for new particles, for instance. A new particle may be discovered by reconstructing its mass from the measured energy and flight direction of all its decay products. Take as an example the mass spectrum shown on the right, which is constructed by taking events containing two opposite-charge muons recorded by the CDF detector at the Tevatron proton-antiproton collider, in its 1994-1995 run. You measure the muons, compute their total mass, and draw its distribution (the black points with error bars). Then you ask yourself the question: "Do I see a bell-shaped bump (a Gaussian) in this smoothly falling distribution ?". (The curve actually shows two hypothetical bumps, but leave this detail alone now.)

    Indeed, a bump seems to be there, at a mass of about 7.2 GeV. If one tries to fit the bump and the smooth background with a suitable functional form, one obtains the continuous line, and by comparing the parameters of that signal-plus-background fit to the parameters of an alternative background-only fit one can compute the "significance" of such a signal of a hypothetical 7.2 GeV particle, using a quite well-known statistical formula (but one which even large experiments sometimes manage to use improperly, obtaining incorrect results). However, the answer one gets from that statistical formula does not account for the fact that the bump may have appeared anywhere in the spectrum we examined. Look-elsewhere warning! Whether due to a Gaussian-shape fluctuation of background processes or to a few genuine decays of a so-far-unknown exotic particle, the bump would have been taken just as seriously if it had happened at 7.0, or 8.0 GeV, or anywhere else in the examined spectrum.

    Physicists use the p-value of an observation to classify its significance. A p-value is the computed probability of the effect. In the case of a unknown-mass particle signal, the p-value resulting from the statistical test must be multiplied by the number of places in the spectrum where it might have been spotted (this is a rule of thumb, but it gets close to the right answer, which requires more accurate studies to be obtained). This factor equates to roughly the width of the mass interval considered in the histogram, divided by the width of the particle signal (we imagine that the signal must have a width compatible to the experimental mass resolution in order to be taken seriously, in this particular case; otherwise, some accounting for a unknown width must also be made). In the case at hand, the probability of the Gaussian bump gets multiplied by at least a factor 20, and an effect having an apparent significance of almost four standard deviations is therefore correctly interpreted as one corresponding to little more than two standard deviations.

    A similar case is discussed in my recent paper on the Omega_b observation (see here for a post about it). There, the CDF and D0 collaboration both failed to fully account for the effect. Or to be precise, CDF did it partly: since they claimed they accounted for the look-elsewhere effect within the mass region where they expected to observe the Omega_b signal (which, after all, was a well-predicted state, whose mass had to lie in the quoted interval); D0, on the other hand, ended up fitting a signal outside of that interval, but in their publication they neglected the tenfold increase in probability of a background fluctuation due to the look-elsewhere effect.

    D0 did come clean, after my paper appeared on the arxiv. A few days ago they revised their "frequently asked questions" web page, where they admit that the significance gets decreased (according to them just by a factor of 7, though) if one accounts for the look-elsewhere effect, which I had estimated in my paper. Unfortunately, they still have not answered my other question on the way they use the p-value formula. I guess it does not matter much in the end: 5-sigma, 4.6-sigma... It does not make much difference: the Omega_b exists, it has been discovered by D0, and the mass has been measured well by CDF one year afterwards. End of the story, for me.

    Comments

    Some of this is explained well by the 'Law of Truly Large Numbers' see wiki:http://en.wikipedia.org/wiki/Law_of_Truly_Large_Numbers
    and also 'Littlewood's Law'.

    Tommaso,

    I have a few examples of this. Several years ago my wife and I were sightseeing in Paris. We'd heard that in a certain park there was a scale model of the Statue of Liberty, but had trouble finding it. Speaking very little French we spotted another American couple who were kind enough to give us directions. Two days later in London while visiting Trafalgar Square we ran into the same couple! Good time for a snappy remark: "So, as I was saying the other day in Paris..." I have always regarded this as an astronomical coincidence, since they were the only people on the entire continent of Europe I was acquainted with. But as you point out, the probability is enhanced when one considers the number of other ways the event could have occurred. I would have been equally surprised to run into them at the Parliament building, Buckingham Palace, or No 10 Downing Street!

    Secondly - in the book Mindswap by Robert Sheckley, Marvin is trying to find his girlfriend Cathy. He has no idea where she might be. Concluding that all locations are equally likely, he decides to wait for her 1000 miles out in the desert, on another planet. Pretty soon a spot appears on the horizon. As it gets nearer, a figure appears, and sure enough it is his girlfriend. Although this may seem at first glance to be a highly unlikely event, Sheckley argues that when you consider the vast number of places they might have met, it becomes almost inevitable.

    Thirdly, in physics. You have two lab teams, A and B, searching for the Higgs. Team A follows the conventional wisdom and "knows" it will lie somewhere between 115 GeV and 180 GeV. Team B feels no such restriction and is willing to believe it might be anywhere between say 50 GeV and 600 GeV. They are both given the same data set. However team A only analyzes it within the smaller energy range. Subsequently they both find the same bump at 120 GeV. It sounds to me like you're claiming by the look-elsewhere principle that team B must assign a lower p-value to it, since such a bump might equally well have been seen anywhere within in the larger range.

    I believe what we're talking about is conditional probabilities, as they depend crucially on the a priori assumption of what range is to be considered, and most importantly whether all points within that range are in fact equally likely.

    dorigo
    Yes Bill, conditional probabilities is the name of the thing. I did not want to get technical in the post, for once :-) Conditioning is in fact a frequently used technique by statisticians to get rid of unnecessary parts of the universe of possibilities. In principle, however, if probabilities can be computed correctly, one does not need to condition. One may just as well compute the probability without constraints, but the denominator may be harder to estimate.
    Cheers,
    T.
    Hi Tommaso,

    I have an example too. When I was 15 I visited Saint Petersburg. One day I with my parents and sister was in Petergof - in the outskirts of the city, the summer residense of emperors. On my way back to the city, in a train, I saw a girl. She impressed me a lot, but some reason disallowed me to speak to her (I was teenager and I travelled with my parents - of course I told them nothing). Then we went to the hotel. That night I even saw her in a dream - she really impressed me. Next day I with my parents was walking in the city. We visited a fortress there. I bought pankakes to have a dinner and the others desided to visit a place, which I'd already visited. When they came back, they said me, that they've meat that girl (they didn't speak to her, just noticed. I didn't tell them I'm looking for her)! It shocked me very strong.

    Why I've told it? Because it'll be really surprise if that girl become physicist too and if she read this blog. We can meat each other in an infinite number of places, but total configurations number - when we are distant - is squared infinite - infinte times larger than the first infinity. No predictable coincidence. Only if one occurs, you'll remember it, depends on how surprised it is.

    dorigo
    Ah, women, women, eternal gods... I wish you the best of luck quantense. May all the desires of men and women merge in a global act of love! Wars would be no more.
    T.
    Posterior probabilities are always amusing. For instance, at this current moment a taxi went by my street at 3:48 AM with a tag number 34A1 (im making this up) and honked his horn. What are the odds that a taxi with exactly that 4 digit tag identification would cross my street at exactly that time, and honking his horn to boot? A simple calculation gives a pretty large number (and you can make it arbitrarily large by adding in more detail to the event). But the point is, it definitely happened.

    Dear Tommaso,

    A nice post, however naive in many aspects. Try for instance to give a reasonable explaination to this fact:
    "Some 5 years ago I dreamt with my grandmother, she was suffering from Alzheimer and living in a hospital (at that time she was just barely self-aware). But in the dream she told me quite clearly that she would pass away in 9 days, when she would be in the company of my grandfather (dead since the 70's)."... Clearly, I woke up frightened and checked my watch... I took notes: 00:27h.
    Well, unfortunatelly hers "prediction" happened: in 9 days +- 30 minutes (with incredible 99.7685185% accuracy).
    Cheers,

    Rafael.

    dorigo
    That is very easy rafael. There are at least two convincing explanations that are logical and rational. The first: out of about 5000 readers of my post, one will be able to claim that, in a 40 year long life (on average), something very, very unlikely happened to him or her. It is in fact a quite clear example of the look elsewhere effect. One reader will be very likely to have something this weird to report. How weird? Take 1000 minutes, 10000 days, 5000 readers: a one in five billion chance should be the rule, on average! Now, your grandma might expect to live how long more? Say 1000 days, or 100,000 time units each 15' long. So dreaming of the date and time of her death to such accuracy is only a one in 100000 chance. This is very, very common! So much so, in fact, that I invite other readers to come up with an experience which is at least ten thousand times less probable than yours. Ah, and- about the second explanation. You might have made that up. Science, I will never get tired to teach, must be based on independently verifiable facts, let alone repeatable. Best regards, and don't bother dreaming of me ;-) cheers, T.
    Dear Tommaso,

    I am sorry, but science just pretends to be fully based on repeatable experiments... to see this under extreme situations:
    (1) surely one can not repeat "the big bang (BB)", or
    (2) observe "the origin of life on earth" again.
    However, both that facts are not "less scientific" than EM thermal echos detected by WMAP (suggesting that BB hypothesis is "real" up to high degree, but not unique!) or the observation of DNA-based life on earth (indicating a proper chemical way for terrestrial origins/evolution of life).
    After all, any "scientific theory" is just made up by human minds... they are always (more or less) correct during some temporal-lapse (in a restricted region of spacetime). Thus, science is not objective in the sense it can not exist by itself, i.e. without scientists' desing.
    Sometimes, when searching for reasonable explanations people have to give up current paradigms and "look-elsewhere" ... notwithstanding, this "elsewhere" may imply using concepts that currently live outside usual scientific mainstream!
    Cheers,

    Rafael.

    PS: about that dream, well, if we use the probabilities you computed, we should consider tinny chances as (at least) P=1:10**10 == P(1:10**5)&P(1:10**5)... since it was also really "observed" (or dreamed) simultaneously by another family member, living 200km away from me!

    dorigo
    Rafael, Rafael... You give me no choice then: you are just lying. Maybe to us, or maybe to yourself too. But it's ok, not everybody likes reason. Many prefer to live under the delusion of the supernatural. Your pick. As to non repeatability, you should realize the difference between theory and experiment. But you do not seem to love science nearly enough to stimulate me to explain that. Cheers, T.
    Tommaso, Tommaso... I quite well understand the differences, as you may do. However, I am not an experimentalist and I must agree with Mr. Einstein when he said: "Whether you can observe a thing or not depends on the theory which you use. It is the theory which decides what can be observed." By the way: don't you assume a theory (SM) a priori to prune new signals from backgounds (by Monte Carlo simulations) in your detectors?
    I remember to have seen you also calling N.A.-Hamed a "lier" in your former blog, when refering to leptonic jets (and you had to apologize after that!)... so, just please, take a look on this edge research that (a serious institution) is pursuing in USA: http://noosphere.princeton.edu/
    Maybe you can use your bold experimental ability there and help that team with another "Look-Elsewhere" Effect... (is it just another background? is it pure coincidence? or, is Princeton lying?)
    Regards,

    Rafael

    dorigo
    Hi,
    sure, I did not believe in Nima's claims at first, and then, after talking to him in person, led me to reconsider the matter. Do you want to have me talk with your relative ? That might help, but I still do not understand whether you are just arguing or if you really believe that you received a prophecy during your sleep.
    Cheers,
    T.
    2) The guy was driving poorly, not surprising he got into an accident up the road.
    3) Women's biological rhythms are tied to the lunar cycle, is it really surprising that some biological functions like going into labour could have some correlation with that?

    dorigo
    Hi, we are discussing how a belief in the supernatural arises, not the examples and their ground. In any case, the birth rate does not show a lunar cycle, period. Cheers, T.
    Tommaso,
    back to the implications of the Look-Elsewhere effect in particle physics, I have a question: assume that the theory predicts the existence of an elusive particle, its possible mass being within, say, a 100 GeV interval (fictional situation, of course!).
    Now, if an experiment is run, at energies capable to explore, say, the lower 20 GeV of this mass range, could you clarify if the experimenters need to take into account the Look-Elsewhere effect for a breadth of 20 GeV or for the whole 100 GeV of not-excluded mass values?
    Because I would say the latter, but I also fail to see how this could be done exactly, with the current policies in sharing data across experiments that I am aware of.
    Thanks.

    dorigo
    Hi,
    the fact that I can only look at 20 out of x GeV of interval has nothing to do with the look elsewhere effect: the only important thing is that I am looking at a certain interval within which I have no clue where the signal might be. If my mass resolution is 1 GeV, there are 20 spots where a fluctuation might occur, so my fit probability must be increased by a factor 20 to account for that.
    The above, of course, is a back-of-the-envelope estimate. One should run pseudoexperiments to test how likely it is to see a so-and-so signal anywhere in the 20 GeV interval in the absence of any signal.
    Cheers,
    T.
    My point was, if you are looking in a 20 GeV rang with 1 GeV resolution, but the theoretical permitted range for the mass of the "elusive particle" is 100 GeV, my back-of-the envelope factor should be 100, not 20.
    As for pseudoexperiments, they are obviously a better way to handle this effect, but they require I think a close-to-perfect knowledge of the background, which is probably the case with SM electroweak background (I don't know if the same can be said about baryon background).

    dorigo
    No, the LEE factor is 20, because what matters is whether the observed data can produce a fluctuation or not. The data cannot produce a fluctuation in a place where we do not look, so the factor is 20, not 100.
    As for PE's, you can model your background. Estimates of fluctuation probability usually depend very weakly on the actual shape of the background.

    Cheers,
    T.
    I see. I still wonder how one should compute the LE factor if a number of different experiments cover, say, a 100 GeV interval by "windows" of, say, 20 GeV each; wouldn't using for each experiment the "20 GeV LE factor" underestimate the likelihood of a fluctuation? It's a bit like travelling each year to a different destination, and finally meeting an old, lost pal...

    dorigo
    Good point. It is in fact one we debate upon, but the trend is to allow each experiment a new virginity. The effect you are talking about is in any case a small one, since there are not thousands of experiments, but just a handful. And the coveted "five-sigma" significance is very tough to reach: in fact it has been designed to be so high to automatically account for the typical "look elsewhere factor" implicit in searches.

    Cheers,
    T.
    We don't get many phone calls, maybe three a day. Let's assume they arrive within eight hours. So three calls in 480 minutes - 160 min between calls.

    When I visit the toilet, I find that it's quite likely that the phone will ring during the two minutes I'm in there. It would seem that there's a 1 in 80 chance of that happening in a given day, or about once every two to three months.

    But for me, it's closer to once a week. And it's usually my wife. She just knows.

    Then there's the urgency detector with which all electronics are equipped. For example, I'm in the process of applying for a mortgage. We have a meeting, and the loan officer follows up with an email. It has an important attachment which I have to print, sign, scan, and email back.

    I bring up the Mac, which has been working perfectly for months. It fails to boot its OS. It is hanging at a particular point - probably a corrupted sector on the HD. No amount of subtle digging suffices; I have to reinstall Mac OS X 10.4.6, upgrade to 10.4.11, apply latest patches, apply latest patches again, locate and install the scanner driver, which isn't in the OS, locate and install the printer driver, ditto. Yup, tripped the priceless urgency detector, again.

    Makes one belive in solipcism.

    dorigo
    Hello Tiffany,
    really. Surprising for a man of faith to separate fact and fiction like that.
    Cheers,
    T.
    A whimsical note by Gamov from "Thirty Years That Shook Physics" (1965), p.63f:

    Pauli started his scientific career very early and, at the age of twenty-one, wrote a book on the Theory of Relativity which (in the revised edition) still represents one of the best books on the subject. He is famous in physics on three counts:

    1) The Pauli Principle, which he preferred to call The Exclusion Principle.
    2) The Pauli Neutrino, which he conceived of in the early twenties and which for three decades escaped experimental detection
    3) The Pauli Effect, a mysterious phenomenon which is not, and probably never will, be understood on a purely materialistic basis.

    It is well known that theoretical physicists cannot handle experimental equipment; it breaks whenever they touch it. Pauli was such a good theoretical physicist that something usually broke in the lab whenever he merely stepped across the threshold. A mysterious event that did not seem at first to be connected with Pauli's presence once occurred in Professor J. Franck's laborartory in Göttingen. Early one afternoon, without apparent cause, a complicated apparatus for the study of atomic phenomena collapsed. Franck wrote humorously about this to Pauli at his Zürich address and, after some delay, received an answer in an envelope with a Danish stamp. Pauli wrote that he had gone to visit Bohr and at the time of the mishap in Franck's laboratory his train was stopped for a few minutes at the Göttingen railroad station. You may believe this anecdote or not, but there are many other observations concerning the reality of the Pauli Effect!