Banner
    Two Ideas For Publishers
    By Tommaso Dorigo | July 1st 2010 05:58 AM | 25 comments | Print | E-mail | Track Comments
    About Tommaso

    I am an experimental particle physicist working with the CMS experiment at CERN. In my spare time I play chess, abuse the piano, and aim my dobson...

    View Tommaso's Profile
    I have just finished four slides (well, five, if you count the cover) which I will show at ESOF 2010, the "EuroScience Open Forum" which is starting in Torino tomorrow. At ESOF, Sense About Science has organized two interesting sessions. One of them is about Peer Review, and it will discuss the results of a recent survey that SAS conducted on the subject with the help of Elsevier.

    The abstract of the session, during which Tracey Brown will moderate a discussion among Nature's Philip Campbell, Elsevier's Adrian Mulligan, and myself, is the following:

    What is the future of peer review? What does it do for science and what does the scientific community want it to do? Should it detect fraud and misconduct? Does it illuminate good ideas or shut them down? Does it help journalists report the status and quality of research? Why do some researchers do their bit and others make excuses? And why are all these questions important not just to journal editors, but to policy makers and the public? In September 2009 Sense About Science in association with Elsevier are publishing the latest results from worldwide survey of 100,000 scientists’ preoccupations and preconceptions as both authors and reviewers of scientific papers. The survey will explore whether researchers attitudes to peer review are changing and whether there is a gap between their perception of peer review and the reality of what it can do. These insights will provide the baseline for discussions on how the system needs to evolve to cope with challenges it faces such as the expansion of the international research community, the issues of fraud, the development of open access and the role peer review plays in science policy and public debates about the quality of science. In this session a panel will respond to these latest results and discuss what the future for peer review is and what the international community can do to address the challenges facing peer review.
    Before the discussion, each of us will have six minutes to introduce a few of the topics, on which later we will take questions from the audience. Below I attach the slides, with minimal commentary. Of course, if you have a comment or a question, be sure to leave it on the thread below, and I will try to represent it during the panel discussion.


    The one above is my cover slide, which uses a nice evening view of Turin as a backdrop.


    This is slide 1. I need to qualify my presence in the panel by explaining that my specific expertise is in high-energy physics, while on other topics I am just as dumb as the next guy. In HEP the issue of peer review is a bit less critical than in other fields; for one thing, we do not claim to cure diseases! Further, experimental HEP is a self-reviewing system, so for us peer review is a nice but almost useless appendage.


    Crackpots! There are those who see them everywhere and feel threatened, and then there is me, who finds them innocuous, potentially brilliant, and in any case not as troublesome as some deranged expert. This again is relevant to fundamental physics only.

    The motivation of referees is possibly the main focus of the survey, as it is a growing concern in an ever-expanding publication world. I have a recipe which would go a long way in the direction of guaranteeing publishers an offer of free reviewing. I thought hard and could not find a single drawback to my idea. Please comment.

    And the last slide discusses a topic that is dear to all science bloggers. How to go from a scientific publication to the public ? How do we bridge the gap ? Publishers should help -they have the means, the dough, and the motivation to do so. I will press Nature and Elsevier on this issue. At the bottom of the page I put forth another revolutionary, a bit flippant idea.... Again, comments welcome!!!





    Comments

    The background of the slides make the text unreadable.

    One suggestion for you: why don't you get rid of the background picture in your slides? It doesn't add anything, just the opposite.

    Then, how do you prove there is no fraud in experimental HEP? In any case, the point is not simply whether the papers
    submitted were published but also whether they benefited from the review process.

    dorigo
    The text is not unreadable, it is less readable. But only in this small snapshot-like version. In the powerpoint presentation the backdrop does not disturb. I like, these are my slides, your comments are appreciated but no, thanks.

    Fraud is by and large associated with one or very few authors- I would be surprised to find cases in literature where the work of large groups (not one author and a dozen relatives who co-signed to increase the author list) were explicitly discovered as fraudolent. It is like the old saying about secrets being such if only one person knows them.

    In experimental HEP groups are larger. Of course, a single author can still manipulate the data and try to fool a hundred co-authors. But if this happens, the idea that an external reviewer catches it from outside is ridiculous to me.

    Cheers,
    T.

    You could choose another photo for the background. Like a photo of an ethanol plant engineer, LOL.

    Well, Tommaso, that was quite a shock. Glad I didn't have a cup of hot liquid in my hand. Thanks for the mention.

    Hey, in the interests of full disclosure I should admit that I'm interviewing for a job as a digital engineer on Thursday. If I get the job, I'll have lots of money for travel to conferences, but little time for physics.

    Hank
    Crackpots! There are those who see them everywhere and feel threatened, and then there is me, who finds them innocuous, potentially brilliant, and in any case not as troublesome as some deranged expert. This again is relevant to fundamental physics only.
    This is an issue we talk about monthly here, in regards to openness versus quality, so it isn't just in HEP or even physics.   There are well known (though not well qualified) bloggers on other sites who I think do a lot more damage to science blogging than someone who invents a mathematical speculation of time travel but we have to think about the overall reputation, including younger scientists who want jobs and don't want a prospective employer showing up here and seeing goofiness on the front page.

    It can be abused and Carl not even getting a chance because of a lack of pedigree is an example, but overall the system works, especially in the experimental field because the confidence level has to be quite high before data gets anywhere.    No one is claiming results with a 65% confidence interval - if they could, the Higgs has to have been discovered already and it is a question of who eventually gets credit.  It is even on t-shirts!


    Want more no-nonsense, independent science? Buy Science Left Behind
    I'll take a crackpot over a deranged expert any day!
    Fred Phillips
    The slides are wordy, but make the case very nicely.

    Peer review is not just to sort good ideas from bad ones, but also to make sure the ideas are new and original. Not that an enormous HEP team could or would steal an idea, but the plagiarism cases highlighted on these SB pages show that a problem exists in other fields. Peer-review can help catch it before publication. There are so many journals these days, that certainly the more (and more diverse) people who examine a manuscript before publication, the lower the probability that a plagiarized piece will slip through.

    I suspect, tho can only argue from anecdote, that crackpottism is more dangerous in social science than in HEP, as it is in medicine.
    Slide 1: I have the impression that this applies only to big collaborations. What about HEP experiments with a few dozens of people involved or even less?
    Slide 4, last paragraph: I think that enforcement is not suitable in this case, if someone wants to promote their work (and has time to do so!) they should be encouraged and supported, but why push it?

    (Funny how the first two comments were in big-collaboration-style ;)

    dorigo
    Hi Eleni,

    HEP experiments with few collaborators are rare, and they contribute minimally to the throughput of papers. But I agree, I should probably qualify a bit my statement.

    Why push the popularization of science. Good question. I guess if the existence of this blog does not provide you with enough of an answer, I don't know what I can bring p.x.

    Cheers,
    T.
    "Why push the popularization of science."

    Hey, going from the "accompaniment of each scientific publication by an online summary devoid of formulas" to "popularization of science" takes some number of steps, especially if it is mandatory.

    I don't know if I should reply here or on the comment below or above :) but I want to make clear what I didn't manage to previously! By saying that it takes a number of steps, I complained about putting words in my mouth (as if I opposed the popularization of science), not that this practice wouldn't help outreach.
    Actually I find the idea very interesting but I'm quite skeptical about the mandatory character of the suggestion. Also, I wonder if anyone has tried this already on an institutional, or at least group, level.

    dorigo
    Hi Eleni,

    the closest example I know is that of Fermilab, where the authors in experiments that produce results for publication are invited to write for Ferminews, where they need to explain the result and its relevance to outsiders.

    My suggestion is not for the world. It is for a publisher to pick up. Imagine that Elsevier wants to launch a blog of "PRL results explained". They may start making the policy that effective August 1st, 2010, any PRL submission must be complemented by a summary for non-experts. If it takes on, other magazines might pick it up...

    Cheers,
    T.
    dorigo
    Well. If there was a way by means of which a scientific publication could be appreciated for its content by unqualified but willing readers without any further mediation, we would have bridged at least part of the communication gap that plagues science. If it is on the shoulders of authors, it will be done (they are the most interested party in the process - they want to publish).

    Cheers,
    T.
    But who are "peers"? If you discover something nobody else has discovered which is way out, you don't truly have any peers who are capable of checking your paper. E.g., if you send a paper to Classical and Quantum Gravity on an alternative to string theory, they send it to dogmatic string theorists to act as "peer" reviewers, who send back a rejection report saying the idea is useless for the progress of string theory! The problem is finding a genuine, unprejudiced, "peer" reviewer if you are working on a new idea which hasn't as yet attracted any other interest.

    Your position of publishing ideas written up by large numbers of collaborators misses this point. If you have a large group of authors, even if they get rejected, they have enough weight in numbers to say, start up their own specialist journal with its own peer review advisory panel, and get on the citation index. In reality, science requires groupthink support for new ideas. It's unlikely that a radical idea can be completely implemented, explored, and popularized in the face of ignorant hostility and sneering rejections from mainstream "peer" reviewers who are basically the doormen to the trade union "closed shop" or maybe "old boys club" (depending on the analogy you prefer) of the Elitist Scientific Research Corp.

    Peer review is great for the incremental progress of everyday science, but it is not so good where the new idea contradicts the entire ongoing mainstream research program, especially if that program is being defended as being "the only game in town". ;-)

    Just a word about the "deranged minds" with the "theory of everything".

    Here's how the "theory of everything" gets developed:

    1. You try to publish a single idea with a prediction for, say, quantum gravity.
    2. You get rejected because your paper does not say how your idea fits into the Standard Model or general relativity.
    3. You work on this problem and find a solution. Now you have a "theory of everything".
    4. Tommaso then says you must be deranged.

    ;-)

    I like what you are saying, but a few points.

    The preliminary findings of the survey are at http://www.senseaboutscience.org.uk/index.php/site/project/395

    The survey was conducted by a UK organisation "sense about science" but was sponsored by Elsevier. The survey itself is quite enlightening but the interpretation presented to media is biased. For example they say that "Researchers want to improve, not replace peer review" This conclusion is based on the finding that "84% believe that without peer review there would be no control in scientific communication". Of course these two statements are not at all the same thing. The survey did not ask if people think that this type of "control in scientific communication" is necessary. This is just assumed to draw the conclusion wanted. Your presentation casts doubt on it.

    Slide 16 of the preliminary findings shows some big differences between what people think peer review should do and what it actually achieves. These things are not highlighted in the media summary except by saying that people want peer review improved.

    One thing they say is "Reviewers want anonymity: 58% would be less likely to review if their signed report was published. 76% favour the double blind system where just the editor knows who the reviewers are." Sadly this seems to contradict your idea that publishing the reviewers name will encourage more to review. I wish it were otherwise but it seems like something else is needed. See slide 8 of review for full stats if you have not already.

    It is a pity that the key results of the survey were not broken down by subject area. I think there would be big differences.

    dorigo
    Dear Nige,

    there are ways nowadays to let your ideas be known by the scientific community without the need to get published in a peer-reviewed journal. The arxiv is an example. It is perfectible, and it has some screening which is probably not careful enough to let interesting but "way out" contributions pass, but it is a possibility for new ideas to emerge.

    I do not think that for a scientific advancement, being published in a peer-reviewed journal is the absolute must. If new Einsteins arise, they will get the attention they deserve.

    Cheers,
    T.
    Thanks for the helpful idea to submit to arXiv.

    I did that in December 2002 when only my email at the University of Gloucestershire could be used to set up an account. I uploaded a paper. It appeared online with an arXiv number. Thirty seconds later when I refreshed my browser, it was replaced with somebody else's paper. Then editor of Classical and Quantum Gravity sent it to string theorists for "peer" review, then sent me the report without the names of the "peer" reviewers. The one good thing about refusing to give the names of the string theorists who wrote that report is that you are forced to suspect the whole lot of them as being your secret enemies ;-)

    Amateur Astronomer
    Tommaso, You opened some important questions about peer review, but there might never be a perfect answer to how it should be done. I missed the earlier discussion, preoccupied with writing one of my books. This website does a good job making the attempt at openness. Writers in the fringe areas have to look around for a category that will accept an article, but there usually is one. It's a publishing business with an editorial policy. If a great many readers want to see a topic that is a bit far from center, the editors can find some way to present it. If there aren't many readers for a topic, even if it is technically sound and it advances the science, then it is really doubtful if a publishing business can afford the space to present it. In your context there is competition for publication space among groups of specialists. High energy physics doesn't have many fringe area writers compared to other types of physics. For one reason, the science is fairly well represented by main stream research groups. It uses data that comes from only a hand full of large research machines. There isn't much chance for an outsider to find something that was missed, or even understand the things that were discovered. In my experience, peer review was applied mostly to closely held information about dangerous things that can never be published. One item that I did get published was a report from a government sub committee in The Hague. The report roughly translated PREVENTING DISASTERS FROM PHOSGENE, failed peer review, not for technical reasons, but because a majority of the reviewers were reluctant to put the information in public domain. One agency director went against the group and published it in his own papers. The report remained in print for 30 years, during which there were no disasters of that type. Unfortunately there was a disaster of a different type. Another report from a different organization in a different country, roughly translated PREVENTING DISASTERS FROM ISOCYANATES was also reviewed and rejected for non technical reasones. There was no agency director to go against the group. Ten years later the recommendations were included in a remedial document that was eventually published. The remedial document in marked up form 903R85001 can be found in a simple search in EPA. http://www.epa.gov/nscep/ Hot link address is too long to put on a web page. If a reader finds it, the first page should have 903R85001 stamped near the top. The item I contributed to was in number 6 of 12 on page (ii) of the index. Other factories in other countries did not have the same type disaster, because they applied the remedies in advance. When you mention peer review and difficulty with publication, it gets rather serious when health and safety is involved. In a lifetime of technical achievements the peer reviews tended to be rather hard to pass, especially when experimental data was not over whelming. The greatest achievements were accomplished under highly successful managers who were able to get a full scale practical demonstration before the report was submitted for peer review. In those cases the reviewers listened politely, said nothing, and left quietly when the presentation was finished. A working machine with a wealth of technical data tends to pass peer review rather well.
    Tommaso,

    You may find interesting looking into this new journal: Papers in Physics www.papersinphysics.org
    I am one of the editors.

    Authors can choose to have an open review in Papers in Physics. If the papers does not present serious technical errors (which leads to a direct rejection) the discussion between reviewer and author is not anonymous. Moreover, the author decides if publication must proceed in the knowledge that a Commentary from the reviewer will be published (with her/his name on it). If the reviewer has nothing interesting to write in a Commentary, her/his name is nevertheless put on the paper. A reviewer Commentary is a independent piece with its own DOI. Authors and reviewers seem to benefit a lot from the process.

    dorigo
    Hi Luis,

    I will have a look. Interesting concept! If you had mentioned it before, I would have probably discussed the model at the round table at ESOF.

    Cheers,
    T.
    "unqualified but willing readers" There is no way I can explain my research to such people. Dear Tommaso, the average 30-year old nowaday has heard the words neutrino, quark, maybe even hadron (also DNA, cells, virus). In other fields of science, older concepts haven't yet made it to the general public, since the school programs stop at 1850. Take for instance differential geometry, and ask yourself who has heard the words manifold, vector bundle, Riemannian metric, curvature tensor.
    It's difficult to talk when you cannot use any words, and yet this is the situation of most mathematicians (which also explains how come there are so few of us among popular science blogs).

    Amateur Astronomer
    Reply to Anonymous Mathematician. Blog writing isn't directed toward the average 30 year old. Readers of science only make up about 20% of the population. School doesn't stop at 1850. The text books might stop there, but the teachers fill in some missing parts. Willing readers can become qualified if the writers are competent and knowledgeable about the reading public. There is a gap in knowledge of mathematics that is difficult to bridge. The disconnection is not in differential geometry. The gap is in multi linear analysis. That's where most readers get lost. Some time ago I recommended a rather old text book by G. A. Hawkins on multi linear analysis that is helpful in bridging the gap. http://www.amazon.com/Multilinear-Analysis-Students-Engineering-Science/... Albert Einstein was criticized for using mathematics that few people had studied, without defining his terms or telling his readers where to find the mathematics. Tolman provided the remedy in his book on Relativity Thermodynamics and Cosmology. It is a rather old and slightly obsolete example of an engineer explaining science to scientists, defining the terms and developing the mathematics. http://www.amazon.com/Relativity-Thermodynamics-Cosmology-Richard-Tolman... Roger Penrose extended the explanations into a more modern view of science in his monumental book, THE ROAD TO REALITY. It differs from most writing in science by defining the terms, developing the mathematics, and refereeing to the explanation when the concept is used later. http://www.randomhouse.com/catalog/display.pperl?isbn=9780679454434 If writers of mathematics feel that the readers are not up to speed, then it is fairly simple to insert a link to the missing explanation. The words manifold, vector bundle, Riemannian metric, curvature tensor can be found rather easily in a search of the internet. The missing part is acquired quickly when a link tells where the fundamentals have been defined and developed. Isn’t it better to offer a remedy than to complain about the problem?
    Bonny Bonobo alias Brat
    I couldn't agree more Jerry, thanks for this post.
    Make love not war