Banner
    I Authored 700 Papers. Did I Read Them All ? No.
    By Tommaso Dorigo | May 29th 2012 03:38 AM | 20 comments | Print | E-mail | Track Comments
    About Tommaso

    I am an experimental particle physicist working with the CMS experiment at CERN. In my spare time I play chess, abuse the piano, and aim my dobson...

    View Tommaso's Profile
    Measuring the value and the impact of a scientist on her field of research using as data her scientific papers, the number of citations these papers got, and the prestige of the scientific journals where these were published is no easy matter.

    Grading Researchers: The H-Index

    There is a large body of literature on how best to account for all these factors together: the discipline is called "scientometrics". Of course, the goal is to summarize the productivity of a scholar in a single number; possibly one with at most double digits, since decision-makers who hire or fund are usually incapable of handling more complex data. One notable attempt is the Hirsch Index, proposed in 2005 by a physicist, Jorge Hirsch.

    The H-index combines together the number of your publications with the number of citations of each: its value N is, in fact, equal to the number N of your publications which have at least N citations. Suppose you have 100 publications and you order them by the number of citations each got: if the top 15 of them got 15 or more citations, and the others got fewer than that, your H-index is 15. So if you write a lot but nobody reads you, your H-index is zero. Kind of smart, but maybe a bit too succinct. Good for a CEO, as I said.

    Coming Clean

    For some reason I never really got excited by the whole business. Perhaps this is due to the fact that in experimental particle physics, my field of research, you just need to be accepted as a member of a large collaboration and you are done: papers will be published with your name on them even if you cannot spell their title. I have been in this business for almost twenty years, and my paper count is well over 700. My H-Index is probably in the whereabouts of 60 or so, which would be stratospheric for almost any other field of research, but is not uncommon in high-energy physicists of my age.

    You might be asking yourself whether I consider those papers as my own. Did I write them ? No, I only wrote or helped writing a small fraction of that large number. Did I at least review them ? No, I only reviewed maybe two hundred of them (and believe me, that is a large fraction for the field's standards!). Hell, did I read them all at least ? No, I cannot even say I read all of them; perhaps I read a third. If this coming clean with my true contribution to papers I signed shocks you, please consider: I did not ask to sign those papers -it is automatic !

    Indeed, in a world full of honest people, one could invent a way to correct for the overinflated output of particle physicists: a "opt-in" policy could be enforced by large collaborations, whereby anybody who wishes to be listed as author of a paper should declare it. By default, the author list of a paper should be empty (let's say it would start containing only the name of the paper editor); then, people would add their names, maybe with a line justifying their request.

    The above mechanism would be quite nice, but unfortunately it would hardly work: the few honest scientists who stick to the idea of only signing papers to which they feel they gave a contribution would quickly end up at the bottom of the list of any scientometric index, being surpassed by the less upright ones.

    The Italian way

    So we are stuck with huge lists of papers we don't really own. The funny thing in Italy is that, at least until some time ago (but I believe the practice is still on) if you participated in a selection for an academic position you were obliged to include in your application an envelope with a faithful copy of all your publications.

    No kidding: putting together the material and sending it cost you a week of work and at least a few tens of euros. And if you cannot imagine the mess at the receiving end, I can describe it: rooms filled with boxes containing scientific papers -usually the same ones in many boxes, since applicants were often members of the same collaborations! And those boxes, needless to say, were not even opened by the examiners.

    The revenge, however, was during the oral phase of the exam: you could be asked to describe the details of any one of your publications. I know stories of people being caught totally oblivious of having presented a publication. Dramatic stuff.

    I was led to think back to all the above when I got aware of a "disturbance" among italian researchers that a new evaluation method for italian research institutes is causing. The method (called "VQR") is actually a good one: every researcher belonging to the institute is asked to insert in a database a list of papers he or she acknowledges of having authored. The system, however, needs to sort out the papers such that every researcher claims a different set -a subset of those they authored. Then the whole scientific output of the institute can be evaluated with that data.

    The disturbances I was mentioning above are caused by the fact that the system's choice of papers that each individual has contributed constitutes an official record, and in principle these data could one day be used to judge the value of individual researchers. This would be silly and dangerous, since a researcher would be judged on the basis of a random process of attribution of a subset of authored papers. Given the sorry situation of research careers in Italy, due to the chronic lack of funding, you well imagine that 50-years-old scientists still at the bottom of the career ladder are growing anxious: they have published countless papers, and now there's an official record that certifies they recognize six of those (randomly picked) as their main contribution to the science.

    I am confident that the problem will soon be solved... My attitude is that if I ever grow tired of the slowness of my career in Italy, I will move some place else. Almost anywhere else in the world than in my sorry little country I would be guaranteed to get a better salary and a higher recognition for my status.

    Comments

    vongehr
    Modern science: A statistician (necessary of course, but so is the cleaning lady) who does not know the papers, gets his name on somewhere among hundred others, and it counts as a full paper in his record, while I have written almost every single paper myself, even if my name only appears third down the row, always adding scientific creativity, and it counts less because surely it won't be cited as much in comparison.

    All these systems are nonsense anyway. As long as we have so much effective censoring, every paper that is critical should immediately count five points for all the hassle and time the author had to fight to get it through the establishment bouncers. My newest paper took four solid years of fight, almost ten journals that rejected/lost it/asked for revision to afterwards refuse it/ ... , and even months hung in moderation on the archive (a fight with them and miss categorization being mandatory it seems). Of course, nobody established will cite it.
    dorigo
    Dear Sascha, I am a physicist, not a statistician. I happen to have some notions of Statistics too, so I was put in charge of the Statistics Committee of my experiment, but that does not allow you to falsify the facts.

    My contribution to the papers I signed in collaboration with 3000+ (in CMS) or 600+ (in CDF) other colleagues is certainly no less than that of my colleagues, on average. I did not invent the system, nor did I take advantage of it - all selections I passed had other candidates with just as many papers they did not write to their credit. Your branch of physics is not HEP so things there are different.

    Also, if you write papers that have other names before your own you should blame yourself or your colleagues/bosses for being treated like a slave, not me.

    Finally, if your paper took so long to get through, but eventually did, good for you - see my other article of today. Unless it is on neutrinos faster than light that decelerate in an attosecond, in which case you lack my support.

    Cheers,
    T.
    Tommaso,

    in some fields, and especially in mathematics, authors are listed in alphabetical order of surname. V is rather far down the list, so being 3rd author very often on a statistics paper does not necessarily reflect the fact said author is not the main researcher. I do not know the exact field of Sascha's work or the practices therein, so I cannot comment on the exact conditions for him.

    dorigo
    Fine with me - also in HEP names are listed alphabetically. If Sascha got his name far down in the list it means that either all positions are equivalent, so he should not be bitching about it, or that he was treated unjustly, which might be a cause for his invariably sounding disgruntled but certainly cannot explain it alone.
    Cheers,
    T.
    vongehr
    if your paper took so long to get through, but eventually did, good for you - see my other article of today. Unless it is on neutrinos faster than light that decelerate
    Would take you one second to check whether it is on FTL, but then you do not even read your own papers, sorry, papers that happen to have your name on it. Anyway - you seem to not grasp the vital:

    1) Criticism is supposedly published in the same journal that the original work was published! "but eventually did" means usually a journal that nobody of the intended audience will read. I.e. effective censoring!

    2) The time arguing with journals/reviewers rewriting waiting and so on, if I had it instead for the usual writing of papers on fluke results with silly explanations as is standard in my field, I could have literally added five more papers that will be cited during those four years. Do you understand the impact of such (four years with four non-cited papers compared to having another 20 already cited papers) onto the job opportunities for a young scientist, dear Tommaso? Do you understand the phrase "effective theory" and do you understand the similarly meant "effective silencing"? No?
    LHC experiments are a case by themselves. The collaborations are gigantic and the flow of papers too.
    Anyway, when you compete for a position, similar people will compete with you, so among them the H-index
    could at least indicate the lenght of the experience in the field. Moreover, candidates are interviewed and in most
    of the cases the interviewers know them in advance "by fame", so there are not too many problems.

    General comment: LHC experiments have to be done. They are the forefront of particle physics and the results
    are extremely exciting...BUT.....there is a small drawback.
    In my opinion a whole generation of young hep students/phds will be destroyed by this process. In those collaborations
    a young guy do not learn much and runs just tons of already written software in a company-like environment.
    This is very bad for science-loving persons. I always advise people that it is much better to work first in a small
    experiment and really learn experimental physics and THEN move to, say, an LHC experiment.

    For you Tommaso it is surely exciting, and it is also for me, but we have years of experience and we know how
    really an experiment works. It is not the case for young people.
    In this respect, ATLAS and CMS are a complete disaster. I feel sorry for all those students struggling with
    tens of meetings a day, huge softwares hard to understand and fuzzy beyond SM models to learn and thow away
    in few months.

    To be clear: we have to do those experiments, they are super-important, but very bad for a young scientist that
    should learn physics first. The LHC environment is too bussiness/company/managenment oriented....

    dorigo
    Dear Anon,

    I totally agree with you on your observations, but I reach a somewhat different
    conclusion, maybe more optimistic...

    I think it's like these students had "bought futures" to work in the LHC without having
    a suitable training in HEP and making the proper experiences first. Most of them, though,
    will have a chance of pay back by studying those basics and fill those holes
    in later years. The LHC experiments are quite likely to last two decades, so the students of today will be the professors of tomorrow, still working on those LHC experiments. They will be less knowledgeable, probably, than their fathers; but maybe they will not need that knowledge
    after all !

    I am reminded of the lower quality of instruction in general in this internet age with respect to twenty, thirty years ago: students learn less, cut and paste from the web, and find shortcuts to avoid sitting through a text for long. By observing the phenomenon, I come to think that they have evolved to adapt to the new environment. They are not dumber, at all: they are smarter than us in a way, despite knowing less.

    Cheers,
    T.
    Good that you are optimist about that, but still the lack of patience and staying power in front of problems is not
    good in general. It might work for having an army of quick and smart "slaves" but who is going to solve
    real problems? There is a general lack of thinking, in myu opinion. And this conclusion is drawn after looking
    after several students, which rush to the web for overcoming difficulties and problems. When the solution is
    not available in 5 minutes of googling, they come to you crying. You know the answer just because you have
    indeed studied more...
    The way science is done in the LHC experiments is horrible and many students are harmed from that.
    Still, these experiments are VERY important and there is no other way to do them..fortunately I escaped in time and
    I can enjoy their beautiful results without participating in extracting them! It is sad to observe this army of students
    crunching data all the time and participating to 100 meetings/day 'round the clock without understanding anything
    but being excited to belong to this over-advertised adventure.
    Moreover, there is not special "smartness" in the LHC experiments. It is all standard stuff, build on the solid
    experience of previous detectors. Other particle experiments are much smarter in detectors, techniques and
    even accelerator technology....
    Again: I acknowledge the absolutely high scientific value of the LHC results that indeed are very exciting.

    I am reminded of the lower quality of instruction in general in this internet age with respect to twenty, thirty years ago: students learn less, cut and paste from the web, and find shortcuts to avoid sitting through a text for long. By observing the phenomenon, I come to think that they have evolved to adapt to the new environment. They are not dumber, at all: they are smarter than us in a way, despite knowing less.
    This is a common perception, but I'm not sure that the first part is entirely true. So much depends on expectations. Just as much can be learned because so much basic information is easily accessible through the internet if the young are educated to recognize better sources. If there is a tendency to read less off the screen, it's not as if all the books have disappeared from the library or bookstores.

    The cut and pasting can easily be dissuaded because it's so easy to catch, and there are stil a lot of young readers out there.

    In the same way that terrorist and cult groups have switched from pamphlets and street corners to the internet, this age is simply offering different tools to critical thinkers. And they do not know less; they just have newer hard drives.
    Hank
    I've echoed this sentiment.  When I was a young guy starting out in business side of the physics business (not academia, obviously) I would meet older engineers and physicists who did not want to hire anyone who could not use a slide rule.  They did not trust calculator people, their brains were mushy.  Then time went on and I met guys who did not want to hire anyone who had not built their own ham radio, or something equivalent.  Mushy brains again.

    Now some think the Internet is making young people weak - but people who do not memorize as well as others are not really more intelligent.  I think most older people regard the way they learned as important.  But as the world of physics gets bigger - we are talking about billions and billions of collisions - people who can understand how to find good stuff in a world of almost unlimited possibilities (like the Internet) have a lot of value.
    C'mon this is different. I'm the guy writing the messages above.
    I'm under 40 but still with more than 10 yrs experience in hep. I'm using the computer every day
    since I was 10 yrs old so I cannot be confronted with people loving the slide ruler.
    I'm talking about the difference between people able to sit down and think..maybe some original
    solution to problems, to people used to get their food within 5 minutes from google.

    Second, try to participate to an LHC experiment and then come here and tell me where is science.
    Maybe people at "high level" like Tommaso see it, but it is a disaster for the young guys.

    The problem is different here wrt a clash between generations (pc vs ruler).

    If all of this is positive or negative i cannot tell definitely: physics is a broad subject and not everybody
    fortunately works in ATLAS or CMS, which are anyway great and valuable experiments.

    The Stand-Up Physicist
    The LHC generates the largest amount of data of any experiment done in the name of science.  Perhaps it is fitting that requires the longest list of authors.  Big, bright teams are necessary to digest super computer centers worth of data.  The open question for me is why should the simplest stuff require such effort?
    "Bright" is not necessary. You just need "big".

    Is there a problem of "you cite me, I'll cite you" cronyism in research these days?

    Our social groups following many mania, like in our politics. Just like their beliefs Only reliable groups with a an innovative technology making best valuable products for our society. Thus citations treads also varies.

    dorigo
    Well, in HEP there's not that particular problem, as far as I know. I see a related problem though: some theorists and phenomenologists who interact actively with the experimental groups get cited much more frequently. But maybe this is a good thing - it gives an incentive to theorists to get out of their little office full of paperwork and engage in active exchanges with people doing measurements.

    Cheers,
    T.
    In my University, if a student claims to be an author of a paper that they, in fact, didn't write, they stand a good chance of being expelled. Yet faculty members at the same University commonly allow their names to be listed as authors of papers that they didn't even read. There is something very wrong about this.

    "The above mechanism would be quite nice, but unfortunately it would hardly work: the few honest scientists who stick to the idea of only signing papers to which they feel they gave a contribution would quickly end up at the bottom of the list of any scientometric index, being surpassed by the less upright ones."

    I strongly disagree.
    If an individual claims authorship of a paper for no good reason, this will remain in the records forever and people who know the facts will comment negatively about that.
    What is the point to "cheat", if that is going to make you look ridiculous to your colleagues?
    You think about people who cheat for getting a promotion, but just think about that:
    1) people when applying for a job are *already* requested to list what they *really* did and what was their impact in the collaboration
    2) with the opt-in system, if you cheat on your contribution you will be listed as author forever; and therefore, forever you are at risk that your next job application will be reviewed by somebody who know the truth;
    3) instead, with the current system of automatic authorship you can boast differently according to your knowledge of the selection board (if you know that member A did work on a certain analysis, you will not be so foolish to claim a leadership role on the corresponding paper, but you will be able to do so at the next occasion.)

    By the way in my CV I list only the papers I really feel author of, and provide a link to the full list on spires (just in case the reviewers are curious, which I doubt). I am not the only one, of course.
    I find this the only non-ridiculous way of presenting the list of publications for a member of a large collaboration, and I encourage all readers who belong to large collaborations to follow my example. Then it's up to you to decide the criteria for calling a paper "your paper" (which can also be to have just participated to the internal review, or having cared for the good functioning of a piece of detector which is particularly crucial for the precision of that particular result). In essence, this is equivalent to the "opt-in" mechanism (but it would be better to have the opt-in mechanism rather than obliging people to do this manual selection of their publication list.)

    dorigo
    Hi Andrea,

    personally I also have been doing exactly what you say - making two lists of papers, that is, with comments on the list where I contributed directly. I think this was even requested explicitly in a few of the selections I took part in, already almost a decade ago. But we are discussing the Italian system here, while I was more general in the post. I only have partial information on what happens in other countries when selecting scientists, so I discussed scientometric indices instead.
    I remain of the opinion that the opt-in method would hardly work, because different individuals have different feelings as to what is their contribution to a specific work.

    Cheers,
    T.
    > because different individuals have different feelings as to what is their contribution to a specific work.

    But this objection applies also to the solution of maintaining a "reduced" list of papers...

    Anyway, I think any sensible person agrees that the Italian way (both the old one and the new one that you describe here) is quite wrong, especially for exp-HEP (although the new one may make sense outside of exp-HEP, as you point out).
    When I applied for jobs in Italy I typically spent ~70 euros per application, because of the amount of paper to be sent from abroad (10 most representative papers + CV + all the pointless paperwork, all in triple copy). The most stupid requirement was to send everything in triple copy! Which I always found unjust: they certainly have xerox machines in the secretariats and it would be much cheaper for them than for the candidates. Not to mention that it would be just logical, in this modern world, to send the documentation by mail as PDF attachments, or upload the files on some secured website, as is the norm in other civilized countries.
    (Although I must say that in the same country it may depend on the organization: in France, CNRS demands to upload on a secured server, while universities typically demand the idiotic three copies as in Italy.)