Community feedback

Last week we published the editorial policies of Quantum and opened the call for editors. In the first few days, we already received over twenty applications (and keep receiving more), from postdocs to full professors! Thank you for believing in Quantum and joining this enterprise.

We would like to thank everyone who has sent us feedback both privately and through lively discussions on social media. It is great to see that the community is passionate about improving scientific publishing, and that Quantum is seen as an opportunity to do it.

 

A platform for public discussion

We would like to stress that the editorial policies of Quantum are not, and will never be, final. One of the core ideas behind Quantum is precisely to be continually open to community feedback and improve the policies accordingly. The first step was to prepare a good initial proposal, and now we are ready to open it to public discussion.

For this we must provide a suitable platform for discussion. You may comment on the editorial policies and other aspects of Quantum on the subreddit r/quantumjournal. Everyone can see the ongoing discussions; to comment you need to create an account (which takes a minute).

So far there have been two main topics of discussion, which we would like to respond to.

 

Does Quantum care about the expected number of citations of papers?

No! It was suggested that our guidelines for referees implicitly asked them to judge the likely impact of a paper. We would like to be very clear about this: impact (as in projected number of citations in the first two years) will never be part of the acceptance criteria. Like many of you, we see the obsession with impact factors as one of the main problems of the current scientific publishing model.

So how could we have conveyed the antithetical message? We went over the guidelines for referees again, and found these two rating questions:

(1) “The paper makes a significant conceptual contribution to scholarship.”

(2) “The paper makes a significant technical contribution to scholarship.”

(Strongly disagree / Disagree / Neutral / Agree / Strongly agree)

The way they were phrased, these questions were vague and partially redundant, as later on we ask more specific open-response questions:

(3) “Comment on the technical correctness of the paper, and the significance of the technical contribution to scholarship (including experimental methods and/or mathematical tools).”

(4) “Comment on the conceptual contribution of the paper. Does it open new questions? Does it uncover limitations of previous approaches? Does it introduce a new perspective on the topic?”

We did not anticipate how widely different people’s associations with the term significant would be. For us, significance is not the same as expected impact in terms of number of citations. For example, a paper can also be “significant” and above threshold if it is:

  • A technical result that is only relevant for a small community – like the solution to a long-standing problem that is however purely of academic interest.
  • Negative results such as the report of a failed experiment, as long as it has a good analysis of what caused the failure, so that others can avoid it.
  • A paper finding a simpler or conceptually new way to prove a result or to understand a certain observation.
  • A paper that makes a widespread intuition rigorous, or that proves it wrong.
  • A good review of a topic.

The above list is certainly not comprehensive, but it helps convey the message. Significant papers interest the experts in a field (not necessarily the wider audience, which can easily be impressed by flashy rhetorics). They are the papers that you would recommend to students.

Following community feedback and consultation with the steering board, we removed the word “significance” from the referee response form, and revised the guidelines as follows:

  • The two rating questions (1) and (2) have been replaced with the more concrete formulation:

“I would recommend the paper to a colleague or student working on the subject.”

(Strongly disagree / Disagree / Neutral / Agree / Strongly agree)

  • We changed the open-response question about technical aspects of the paper (3) so that it emphasizes reproducibility:

“Comment on the technical contribution of the paper, focusing on correctness and reproducibility. Are the proofs clear and easy to reproduce? Is the experimental or numerical data (or software) openly available? Are the experimental methods and/or mathematical tools appropriate? Do the technical results represent a significant step forward in this problem?”

There is an ongoing discussion about whether open data should be a strict requirement for publication Quantum on the subreddit. If you have expertise in this field, we would love to hear from you.

  • We changed the open-response question about the conceptual contribution of the paper (4) to make it more specific:

“Comment on the conceptual contribution of the paper. Does it open new questions? Does it uncover limitations of previous approaches? Does it introduce a new perspective on the topic? Does it solve a long-standing problem? Does it make a widespread intuition rigorous, or prove it wrong? Does it provide a useful analysis of a failed experiment or fruitless technique? Is it a good review of a subject?”

 

Further community feedback led us to a second, broader discussion…

 

Should Quantum be selective?

The question was whether Quantum should:

  1. Provide a curating service, by accepting only papers above a certain quality threshold (as judged by referees and editors), or
  2. Make no judgements on perceived “quality” or significance” of papers, and accept all submissions that are technically correct and well presented.

We considered this question deeply before deciding on the current editorial policies, as there are strong arguments towards each of the options. At the time we reached the conclusion that Quantum can provide a better service to the community in the short and medium term by going for the first option and providing a quality filter for papers. Before we present our arguments in an upcoming blog post, we would like to hear from the community and welcome discussion in the subreddit.

Share

1 thought on “Community feedback

  1. Pingback: Should Quantum be selective? – Quantum

Comments are closed.