Sometimes the Refereeing Process Actually Works :)

We had a paper just accepted to PRE, and I wanted to show it to someone who I thought may not have seen it.

Me: "Yes, we need to upload the revised version to the arXiv reasonably soon. We changed it substantially (we had a very thorough and helpful referee), so I didn't want to only point you to the old version."

Response: "I hesitated to tell you this in the first message, but I was the referee--thanks for the compliment, all the more meaningful since you didn't know!!"

As frustrating as this can all be (and it is very often indeed), sometimes the process really does work, and sometimes you end up accidentally complimenting somebody a lot instead of accidentally having egg on your face. ;)

Referee Reports for the Watts–Strogatz 'Small World' Paper

This is a great document!

Steve Strogatz posted his referee reports (with annotations for their revision plans) about his genre-defining 1998 ‘small world’ paper with Duncan Watts.

It just goes to show that misguided, skeptical reports (see Referee 2 in the upper right) happen to the best of us.

Referee 2 was confused about the implications of the work to systems other than those studied (and possibly even to those applications)…

(The source is Steve's tweet. I posted the picture separately to orient it correctly.)

Almost 11 Years in Review

11 years passed between this paper's initial submission and its final acceptance.

no word needed... @YourPaperSucks pic.twitter.com/WW3zue9kmO

— Sarah Lambart (@Sarah_Lambart) December 8, 2017

For the story, see this Quora answer

(Tip of the cap to Joel Miller.)

"Mistakes Reviewers Make"

Here is a useful article about mistakes reviewers make and things to think about when one writes a referee report on a paper. It's good for both junior and senior people, and reminders even for senior people (myself very much included) is definitely a good thing. (It's views on how much one should stress the details versus the big picture of a paper do, however, appear to be different from mine. I didn't look at that part carefully but focused on the headline points rather than on the details therein --- possibly ironically.)

(Tip of the cap to Vinko Zlatić and Bruno Gonçalves.)

Brutally Honest Referee Comments

Here are some occasionally amusing and often harsh referee comments from submissions to scientific journals. I have been both on the giving and receiving end of such things, such as the following gem that one of my papers was subjected to earlier this year.

One of my PNAS referees wrote the following in the last line to summarize his/her review: "The results here represented a substantial step backward in terms of value and sophistication from the many published analyses that have already been published in Korean and/or English."

(The referee never provided any example paper or author, and we were blown off by PNAS when we requested to have a name or reference as a starting point. We looked very thoroughly both before and after the submission to PNAS, and we could never find any such paper. At the end, though, our story has a happy ending: our paper is now happily residing in PRX, which is PNAS-level but for physics specifically.)

(Tip of the cap to Joshua Bodyfelt.)

Update (12/14/14): Here are a few more harsh comments from referees. A couple of these are amusing, but overall they aren't as good as the ones above.

What The Reviewers Say

Reviewer #2 concludes in his/her final sentence: "The results here represented a substantial step backward in terms of value and sophistication from the many published analyses that have already been published in ..."

Ouch. Yes, not only does this referee not like our paper, but we have devalued the work that already exists. (To give some context, let me point out that reviews of Tom Lehrer's work have included comments such as "Mr. Lehrer's muse is not fettered by such inhibiting factors as taste.", "More desperate than amusing", "He seldom has any point to make except obvious ones", and "Plays the piano acceptably".) We will survive to improve the paper and submit it to another journal.

Well, at least there are also comments that we can use to make our paper better when we submit it somewhere else.

Those Funny Referees...

Sometimes, referees say the darnedest things. Some of these comments are really damned funny, though it's bad as an author being on the receiving end of such snarkiness. I've had my moments as well, but I do try to minimize the number of such things I do in the referee report (unless it's a good-natured comment). I do let the snarky comments fly elsewhere, of course!

One must be very careful, however, with the proverbial "third referee".

(Tip of the cap to Mariano Beguerisse Díaz.)

Scientific Peer Review, ca. 1945

Miguel included this link in his comment on a previous post, and I think that this video about scientific peer review (circa 1945) is hilarious, so I wanted to give it the post it deserves. Very nice!

Adversarial Reviewing

If you are (or were) an academic, you know you've experienced adversarial reviews. (The link points to a paper that shows how to prosperously referee papers adversarially.)

I'd like to dedicate this entry to the adversarial referees that have been a part of my life. (Among others, I'm looking at you, PLoS Computational Biology referee who (incorrectly!) complained about the fact that we used a method that didn't work on weighted networks on a paper in which we only study unweighted networks. (Sadly, the Editors actually considered this report to be legitimate even when presented with a published mathematical proof to the contrary and a pointer to a very prominent line in which we state rather clearly that all of the networks under study are unweighted. Se la vie.)

(Tip of the cap to Aaron Clauset.)

Quote of the Day

Today's quote comes from an anonymous referee who wrote the tie-breaking referee report that caused a paper of mine to be accepted for publication:

While this manuscript is not a perfect or complete solution to the problem of community detection in networks...

You know, it's a good thing that such a perfect or complete solution of the problem of community detection (some formulations of which have been proven to NP complete --- that is, with an appropriate definition of community based on optimizing a particular class of quantities) isn't what's expected for publication. If so, I'd have only one answer: Good luck with that.

Anyway, the referee was perfectly fair and is completely correct with the implied comment that our paper doesn't blow the problem wide open but rather just makes a specific set of interesting contributions, but I was nevetheless somewhat amused by the use of that phrasing. Perhaps that comment was simply for the benefit of the editor who works in another research area? Presumably so, but I was amused nonetheless, and I wanted to share my amusement.