The Golden Rules Of Error Analysis
    By Sascha Vongehr | September 2nd 2010 10:42 PM | Print | E-mail | Track Comments
    About Sascha

    Dr. Sascha Vongehr [风洒沙] studied phil/math/chem/phys in Germany, obtained a BSc in theoretical physics (electro-mag) & MSc (stringtheory)...

    View Sascha's Profile
    Presenting measurement values together with some educated guess on accuracy or precision is scientific standard. It is very important to any good scientist, so much so that it is basically my religion. Delta (Δ) is my god! Those statisticians who serve sigma (σ) and teach the primacy of it do not understand that delta is the larger one after all (sorry - the pun here is strictly for the geeks among you).

    One would think that any undergraduate in any scientific field knows the basics, but what I find even among principle investigators and postdocs in the exact sciences like physics is disheartening. The following is pretty close to what I in some form or another encounter in different fields in different countries, something that happens all over the world all the time:

    (Every following quote is the gist of a long email – so this did waste a whole lot of time!)

    A: Hey, this reviewer is a pain and wants to know standard errors and whether there is a red shift in the data or not.

    I: Well, why don’t you just state the errors – you surely know the errors right?

    A: Well, eh, see the value is 5.38 and the shift is 0.18, but I am afraid that maybe the shift is too small and not so accurate and blah blah blah

    I: Just present the errors; they will show whether the shift is significant or not.

    A: Blah blah cry whine blah

    I: What are the errors, the accuracy, the resolution of your spectroscope for example? Have you taken many values and calculated an average and a standard deviation? First the errors, THEN we discuss the implications and whether there are any at all.

    A: Blah blah blah, maybe the shift is too small

    I: GIMME THE F'N ERROR! THEN (read “afterwards”) we discuss the shift! Grrrrr!

    A: Blah whine blah

    I: Look dummy, 5.38 seems to have a certain number of significant digits. You have a scientific academic degree, are almost an established scientist, why would you ever bother anybody or yourself with insignificant figures, right (?). 5.385 rounds to 5.39 instead. 5.374 rounds to 5.37, not 5.38. Hence, if 5.38 is the value you believe in, and you seem to do, then the accuracy automatically implied is about ±0.005, the shift 0.18 is much larger than that. Good luck with the third reviewer.

    Straw-man: “It is now standard to give 2 digits of the error and to write the value out to the same length, like 5.382 ± 0.013”

    Sock-puppet: “Straw-man, shut up!”

    A: Ok, I rewrote it, but still I think the shift maybe goes into the other direction. The Origin software has sigma being 0.009.

    I: Dude listen up! Origin or Excel do not know your instrument. How is it that you have no idea about the accuracy of the stuff you use in your own lab? And how is it you have no clue about the very basics of the propagation of errors?

    A: Blah blah red shift blah cry blue shift tears …

    Finally the guy has me so far as to actually open his darn spreadsheet. Turns out that all values in there have without exception an 8 in the end: 5.38, 13.38, -1.08, and so on! Does it need Einstein to immediately see that the resolution is obviously worse than 0.01, perhaps much worse? The guy is one of the smartest I met in a while, top in his research group, yet he does not get this !?!

    It went on, but to make a long story short, there was no shift. Of course, one can with many measurements and proper statistics improve on the resolution of the apparatus, but in this case, it got worse with every bit of information he divulged.

    What is the moral here? Again – this is just one story of many similar ones that I have personally come across. NEVER trust people to state only significant figures. INSIST on the plus/minus notation. It forces people to think one tiny step further.

    I explained such many times. I realize now that my efforts back-fired. If people don’t get it, they don’t get it, but they will get one aspect: They need to pretend having mastered it all to publish papers. So here is the lesson I effectively taught:

    The Three Golden Rules of Error Analysis:

    Equality (or confirmation) Rule: If you measured 5.37 but the literature clearly states 5.39 and you have reason to assume that the reviewer is V.I. Prof, round the values to 5.4 or just write “(5.37 ± 0.03) confirms the pioneering results of V. I. Prof et al (Journal of High Impact Factor 2010)”.

    Inequality Rule: If you measured 5.37 but the literature clearly states 5.39, and your paper needs some more interesting content to be up to the desired journal, you can argue about an interesting shift of 0.02 somewhere, but then the stated error must be smaller than ± 0.02. Write “(5.37 ± 0.01) indicates an interesting and potentially tunable red shift due to size effects encouraging much further research into our novel method”.

    General Rule: In general, searching hard for different references is always better then doing any maths. Trust me – you will find a reference that somehow supports the needed value.