Banner
    And Another Three-Sigma Evidence Of New Physics Goes...
    By Tommaso Dorigo | December 9th 2011 08:51 AM | 14 comments | Print | E-mail | Track Comments
    About Tommaso

    I am an experimental particle physicist working with the CMS experiment at CERN and the CDF experiment at Fermilab. In my spare time I play chess...

    View Tommaso's Profile
    It is so annoyingly sweet to be right, when being right means that one's job is not going to become more exciting in the near future... Today CDF published their analysis of CP violation in the Bs sector, where a very exciting three-sigma deviation from the Standard Model predictions had been a bit prematurely and uncautiously claimed by a group of phenomenologists in 2008.

    The group had analyzed the results of CDF and the results of DZERO on the phase of B_s mixing, fitting them together with other information using some assumptions on experimental details they did not have access to (if I recall correctly, DZERO had not made available their full likelihood, so the combination was on shaky ground).

    To be fair, many in the CDF and DZERO collaborations were excited by the experimental results, too: the effect observed would be in line with several NP models. I mean to say I do not want to cast the blame on happy combinations of results: it just happens that three-sigma results evaporate, once one analyzes more data. And this occurs especially often when the three sigma are put together by collecting a one sigma result here, a one and a half sigma one there, etcetera.

    In fact, three years and a half after the excitement, the analysis of an over three times larger dataset allows CDF to bring back the departure from SM predictions to just 1 sigma, i.e. no effect. No new physics. Nothing of the sort. To be clear, a one sigma effect is something that happens to you 32% of the time...

    The results are contained in this long paper, which I have no time nor energy to review for you. I can only dig out of it the money plot, which shows the angle beta_s of Bs mixing as a function of the width difference Delta-Gamma_s between the neutral B_s mass eigenstates: the SM predictions are shown as a black point with error bar, the green band shows the expected dependence of the two parameters when CP violation effects are present induced by the mixing of the B_s mesons, and the blue contour shows the one-sigma region of the experimental result on the two parameters.

    If you want more detail on the status in 2008, above I was referring to a discussion I made in my blog in March 2008 of the experimental status of measurements in CDF and DZERO and of the theoretical interpretations which had previously appeared in a preprint (0803.0659). Self-quoting myself from there:

    I cannot help thinking that it would be really remarkable if among all the beautiful measurements that CDF is making these days, one were unable to squeeze out a 3-sigma discrepancy with standard model expectations. Sure, it is a discrepancy that fits well with some model-independent new physics scenario. But is that enough to get hyper ? Here is what the authors have to say in their conclusions:

    “With the procedure we followed to combine the available data, we obtain evidence for NP at more than . [...] We are eager to see updated measurements using larger data sets from both the Tevatron experiments in order to strenghten the present evidence, waiting for the advent of LHCb for a high-precision measurement of the NP phase”.

    What can I say. Good luck. I would be so damn happy if you were right… But I bet you are just being optimistic.


    So my bet would be a winner, as of today. No new physics, and all is in good order in the B_s sector according to CDF.

    Comments

    The three-sigma had already gone after LHCb presented their analysis this summer.

    I think you forgot to mention LHCb's result which is more precise than this one anyway?

    Interesting to see that CDF have also shot down the s-wave component that D0 claimed in their recent result. LHCb saw nothing like it either. I'm wondering how D0 allowed such a claim to be made...

    dorigo
    You are  both right. The LHCb result simply escaped me in the orgy of new stuff that appeared this summer... But I think it is interesting to see the same experiment progressing in the understanding of a physical quantity with the amassing of new data.
    I think my LHCb colleagues will pardon me if I am too lazy to update the article above... Your comments suffice to alert the interested reader.

    Cheers,
    T.
    As a mere biologist, I'm confused (BTW a constant state)

    I see many physics experiments with 3-sigma significance then found to be non-significant. As 3-sigmas equate to a 1/1000 probability. Does this mean there are many thousands of these experiments taking place and I only notice those that regress to the mean?

    dorigo
    Hi Acleron,

    your observation is correct. The answer is twofold. On one hand,  high-energy physics is hard, and it is easy to underestimate your statistical uncertainties or to just overlook the fact that one of your systematics may have long non-Gaussian tails. On the other, as you guessed we do test many, many things in our analysis of experimental data, and only report "interesting" finds. So three-sigma effects are a hundred times more probable in physics preprints than they are from random picking of Gaussian distributions...

    Cheers,
    T.
    vongehr
    No - it simply means that in those large collaborations there is so much going on that a proper error estimation is impossible and only a bare statistics-only version remains. This means, which is openly admitted by HEP physicists, that in HEP physics a 3 sigma means nothing. In my lab (and in yours probably too), if I come up with a three sigma result, you can bet your first born on it (distributions of real stuff do not have Gauss tails that reach out to infinity).

    This makes the sigma stuff, whose whole point originally was to be cross disciplinary, not a reliable indicator anymore. You cannot anymore take two accuracies from two different fields, for example say you want to design your own experiment using a nanoparticle with a size distribution from my lab and something else from another field and want to know whether your new experiment can possibly resolve an issue at all, i.e. whether the in-build error allows the desired resolution. A proper combined error is hardly available, simply because there is no error propagation calculus that has variables depend on scientific field.

    HEP scientists do not see a problem here. Their point of view is basically: We are the creme de la creme, our sigmas rule the world, everybody else please relearn statistics until they see how awesome we are. ;-)
    Tommaso, I recall the figure in your post from a CDF conf. paper in February 2011, concretely http://arxiv.org/abs/1102.0436 .

    dorigo
    Hi Francis,

    yes the result is not "new", but the paper is, and I wanted to mention it today... Besides, I had left this matter unattended for a while here.

    Cheers,
    T.
    Ok, thanks to both of you I think I get the picture.

    Hi Tommaso
    Concerning your response to Acleron about why so many 3-sigma results go missing in action: there's a third, and more fundamental reason. Contrary to popular (and indeed, HEP practitioner) belief, sigmas cannot be interpreted as strength of evidence against the null hypothesis of a fluke result. They are calculated on the basis of the null hypothesis, and so _by_definition_ cannot give a measure of the probability of the null hypothesis being true.

    To estimate the latter, Bayes's Theorem shows we need a measure of the prior probability of the null hypothesis being true; the sigma value then contributes weight of evidence against it.

    This doubtless all sounds like boring statistical gorp, but it is crucially important in assessing experimental results. To answer Acleron's initial question, one reason (perhaps the principal reason) that three sigma results go missing so often is because this level of evidential weight isn't strong enough to compensate for the (low) prior probability _against_ the null hypothesis.

    All this "boring technical crap" is of direct relevance to issues much closer to Acleron's heart: the failures to replicate affecting so many life sciences claims (the subject of a recent issue in _Science_).

    dorigo
    Hi Robert,

    I agree with most of what you wrote, of course (you can see part of your first paragraph replicated in my other post on the glossary for Higgs press release, where I estimate at 40% the HEP pratictioners who are likely to fall in the trap of probability inversion), but I have a different slant on this:
    ", one reason (perhaps the principal reason) that three sigma results go missing so often is because this level of evidential weight isn't strong enough to compensate for the (low) prior probability _against_ the null hypothesis".

    I agree, but Acleron's is a valid question, without calling in alternate hypotheses at all: if we fished at random from a pool of results in HEP, we'd find many more than 0.3% of 3-sigma deviations from theory predictions. Is this a proof that we preferentially publish those ? Well, perhaps, but the other thing one unavoidably must consider is that systematical uncertainties are often hard to evaluate and non-gaussian...

    As for the comment being statistical gorp, if you knew me better you'd have avoided that comment - I work in the CMS statistics committee, so that gorp is my daily bread.

    Cheers,
    T.
    Hi Tommaso - very much appreciated your views on this vexed issue. And apologies for assuming you were a physicist who finds statistical analysis boring beyond words (though you'd admit I think that there IS a high prior probability that any given physicist is in that camp...). Best wishes R

    LHCb measures phi_s in Bs->J/psi f0(980): http://arxiv.org/abs/1112.3056

    Result: -0.44 +/- 0.44 +/- 0.02 rad, again consistent with the standard model.

    Add a comment

    The content of this field is kept private and will not be shown publicly.
    • Allowed HTML tags: <sup> <sub> <a> <em> <strong> <center> <cite><TH><ul> <ol> <li> <dl> <dt> <dd> <img> <br> <p> <blockquote> <strike> <object> <param> <embed> <del> <pre> <b> <i> <table> <tbody> <div> <tr> <td> <h1> <h2> <h3> <h4> <h5> <h6> <hr> <iframe><u><font>
    • Web page addresses and e-mail addresses turn into links automatically.
    CAPTCHA
    If you register, you will never be bothered to prove you are human again. And you get a real editor toolbar to use instead of this HTML thing that wards off spam bots.