Sometimes the Refereeing Process Actually Works :)

We had a paper just accepted to PRE, and I wanted to show it to someone who I thought may not have seen it.

Me: "Yes, we need to upload the revised version to the arXiv reasonably soon. We changed it substantially (we had a very thorough and helpful referee), so I didn't want to only point you to the old version."

Response: "I hesitated to tell you this in the first message, but I was the referee--thanks for the compliment, all the more meaningful since you didn't know!!"

As frustrating as this can all be (and it is very often indeed), sometimes the process really does work, and sometimes you end up accidentally complimenting somebody a lot instead of accidentally having egg on your face. ;)

Almost 11 Years in Review

11 years passed between this paper's initial submission and its final acceptance.

no word needed... @YourPaperSucks pic.twitter.com/WW3zue9kmO

— Sarah Lambart (@Sarah_Lambart) December 8, 2017

For the story, see this Quora answer

(Tip of the cap to Joel Miller.)

"Mistakes Reviewers Make"

Here is a useful article about mistakes reviewers make and things to think about when one writes a referee report on a paper. It's good for both junior and senior people, and reminders even for senior people (myself very much included) is definitely a good thing. (It's views on how much one should stress the details versus the big picture of a paper do, however, appear to be different from mine. I didn't look at that part carefully but focused on the headline points rather than on the details therein --- possibly ironically.)

(Tip of the cap to Vinko Zlatić and Bruno Gonçalves.)

Brutally Honest Referee Comments

Here are some occasionally amusing and often harsh referee comments from submissions to scientific journals. I have been both on the giving and receiving end of such things, such as the following gem that one of my papers was subjected to earlier this year.

One of my PNAS referees wrote the following in the last line to summarize his/her review: "The results here represented a substantial step backward in terms of value and sophistication from the many published analyses that have already been published in Korean and/or English."

(The referee never provided any example paper or author, and we were blown off by PNAS when we requested to have a name or reference as a starting point. We looked very thoroughly both before and after the submission to PNAS, and we could never find any such paper. At the end, though, our story has a happy ending: our paper is now happily residing in PRX, which is PNAS-level but for physics specifically.)

(Tip of the cap to Joshua Bodyfelt.)

Update (12/14/14): Here are a few more harsh comments from referees. A couple of these are amusing, but overall they aren't as good as the ones above.

Quote of the Day: Referee Request Edition

Well, I haven't used this line in my decline letters (at least not recently) because maybe it isn't the kindest thing in the world to write, but there are certainly times when I want to end my e-mails declining a request like this: Please also make it clearer in the future what method of communication you would like from me for declining review requests.

To give some context, I consider it exceptionally obnoxious when somebody asks me to referee something on a short timescale and assumes that I will accept the reviewing assignment. It's a very poor assumption, and in fact it increases by a rather substantial margin the chance that I will decline the request.

It would be good, by the way, if every such request automatically included the procedure to decline and not just the procedure to submit a referee report.

P.S. Get off my lawn!

Those Funny Referees...

Sometimes, referees say the darnedest things. Some of these comments are really damned funny, though it's bad as an author being on the receiving end of such snarkiness. I've had my moments as well, but I do try to minimize the number of such things I do in the referee report (unless it's a good-natured comment). I do let the snarky comments fly elsewhere, of course!

One must be very careful, however, with the proverbial "third referee".

(Tip of the cap to Mariano Beguerisse Díaz.)

Adversarial Reviewing

If you are (or were) an academic, you know you've experienced adversarial reviews. (The link points to a paper that shows how to prosperously referee papers adversarially.)

I'd like to dedicate this entry to the adversarial referees that have been a part of my life. (Among others, I'm looking at you, PLoS Computational Biology referee who (incorrectly!) complained about the fact that we used a method that didn't work on weighted networks on a paper in which we only study unweighted networks. (Sadly, the Editors actually considered this report to be legitimate even when presented with a published mathematical proof to the contrary and a pointer to a very prominent line in which we state rather clearly that all of the networks under study are unweighted. Se la vie.)

(Tip of the cap to Aaron Clauset.)

Not more than two physical forces in nature

The October 2008 issue of Physics Today had an awesome article, in which physicist Lawrence Krauss discussed a hoax article that he submitted to Physical Review Letters claiming that he had found a new, overlooked force of nature. In contrast to what we typically see nowadays, the editors of PRL showed that they had a sense of humor and fired right back, responding with several fake referee reports (one for each force of nature). Both the hoax article and the referee reports were reprinted in Physics Today.

Here is the abstract for the Physics Today article:

When I was a young assistant professor at Yale University in 1986, a great deal of interest was aroused by a paper that had appeared in Physical Review Letters (E. Fischbach et al., Phys. Rev. Lett. 56, 3, 1986). The paper argued that a reanalysis of data from the famous Eötvös experiment provided evidence for a force that violated the equivalence principle—the equivalence between inertial and gravitational mass. While many people were skeptical of that result, I reacted with surprise to the notion that the paper had survived the refereeing process, which at the time had very strict self-imposed requirements of general interest, importance, and validity.

I loved the following line from the Editor's response (that contained the referee reports): "Although all advise publication (after some revision) the Editors, in their usual arbitrary and capricious manner, do not come to this conclusion."

I approve!

Quote of the Day (another one)

After an unintentional bit of "awesomeness" in a paper I refereed today, I think I might have to replace the previous quote of the day. This whole situation started because of an unfortunate substitute for the word "data". Here is the quote:

"For many cases a localized date breaks up into a finite collection of solitary waves."

Indeed! I couldn't have said it better myself!

How to write a sentence that is impossible to parse

I am currently refereeing a paper that has one of the most unparsable sentences I've ever seen. (The paper itself is pretty good, by the way.) This sentence includes a whopping total of ten prepositional phrases, including seven in a row.

Let's count them, shall we: "This is consistent (1) with the relative smoothness (2) of the return times, which are proportional (3) to the averages (4) of a function (5) of a flow (6) over the trajectory arclength (7) between successive crossings (8) through the Poincaré surface (9) of section and (10) to this arclength itself."

Say what?

(Well, in truth, some of the phrases aren't actually doing any harm, but others are making this sentence really hard to read.)


Students: If any of you are reading this, please don't ever write a sentence like that.

Report from the refereeing trenches

I am currently refereeing the revised version of a paper I had previously refereed. (Actually, I have been doing a number of these lately, but I have a specific one in mind.) One of my comments on the original version of the paper was that there were severe grammatical problems (and typos) that needed to be fixed. In their response to this point in the cover letter, the authors wrote:

"We have checked spelling and did our best to improve the grammatic."

Dude!

Best sentence in a research article ever

OK, so I'm exaggerating, but the sentence is "awesome." (I can't disclose the identity of the paper because I'm reviewing it. It won't be published any time soon.)

Here is the sentence:

"We will use a useful method [8,9,11,33] to construct the general solution of Eq. (8)."

(To be fair, the authors do go on to give a couple sentences about the method even though they don't elaborate sufficiently. The sentence just sounds awesome when taken out of context. The real problems with the paper lie elsewhere.)