Author feedback

Posted on by

Two months go, we asked authors for feedback on their experience with Quantum. Without further delay, here are the raw results. We will write a follow-up with our conclusions in the near future.

 

Methods, demographics and bias

We asked for feedback via a post on Quantum blog, which was shared on Twitter and Facebook through Quantum’s accounts. We also sent an email to all corresponding authors of published papers, directing them to the post.

We received 56 responses, out of 222 papers that have reached a final decision at Quantum (acceptance or rejection). We heard back mostly from authors whose papers had been accepted for publication:

 

The actual acceptance rate for Quantum is closer to 50%, so this sample has a strong acceptance bias. Why didn’t we write to corresponding authors of rejected papers? Simply, we don’t collect email addresses during the peer-review process – those are stored by the online platform used for review, Scholastica. When authors send us their final version for publication, though, this goes outside Scholastica and we can automate the collection of those emails. We could still have manually gone through each rejected manuscript and sent a message through Scholastica to the authors. This takes a couple of hours of manual work, and we will consider it for the next round of feedback in half a year. For now, let us take this data as a measure of how well we are doing with papers that end up getting accepted.

 

General feedback

We asked authors why they submitted to Quantum:

The bottom two (free-form) responses fit into ‘faith in Quantum’s mission’.

The peer-review process lasted less than 4 months for 75% of the papers.

Published authors are generally happy:

The different aspects we asked about are, in order, Quality of reviews, Communication with the journal, Fairness of the decision, Duration of the process, Publication process, once accepted, and Publication fees.

Clearly, duration of the process is the main aspect that needs improving, from the perspective of authors. We have already put in place mechanisms to avoid unnecessary delays, and to improve communication with authors about the status of their manuscripts.  So far this seems to be working well, and in half a year we will ask authors for feedback on whether this has improved their experience.

Given the acceptance bias in responses, some of the metrics are of limited use: it is hardly surprising that published authors were satisfied with the fairness of the decision.

Most authors claim that they are very likely to submit to Quantum again, and only 2 out of 56 are unlikely to do it.

 

Open-ended questions

We asked authors what we are doing well, and where we can improve. Here are their answers. These may have been edited to protect the writers’ privacy when necessary.

What are we doing well?

  • creating a journal that is respected by the community
  • I like that the journal is run by the community for the community. No hidden agendas, no money-making scheme. No bullshit acceptance/rejection quotas. It’s all about the science.
  • Your job
  • Keep up the high quality standards!
  • Mission, process, quality, speed
  • The reviews were, for the one paper I have submitted, of very high quality, compared to those I have seen in other journals.
  • The overall process is very transparent and compared to other journals I have dealt with, also fairly quick.
  • Providing an ethical outfit for peer review
  • I really like the idea and the effort that you are putting in the project
  • The website interface is very nice, and Quantum does a great job of publicizing the published papers.
  • Almost everywhere I would say. Keep getting the nice quality published and you will surely be among the greats.
  • The reports I got were always quite good and I think fair. I do get the impression that the community pays attention to what happens and is published here and I think this is encouraging. I can see how having a community-run journal in which people get more involved means there is more of interaction and more of a conversation between researchers at discussing ideas and results.
  • The review process was transparent (the only downside from my opinion was one of our referees gave no comments and we were hoping for more feedback). I feel that many people read the paper because it was published in Quantum. I also like the format of the published article. The process from acceptance to publication was very fast. But my strongest positive point was that the decision to accept or not was independent of the “flashiness” of the result and purely on the quality of the paper.
  • I really trust quantum mission!
  • Quantum’s mission, and the reception of that mission within the community, is great. It feels good to publish somewhere that is open source and outside of the “mainstream” publishing environment. Thank you for providing this option! With regards to my experience publishing in Quantum, I found it to be a very smooth process. In my case, if I remember correctly, one of my reviewers took longer than the expected time, but I queried my editor about this, and they responded quickly, presumably reminded the reviewer, and the review came in shortly after.
    Reviews have been extremely high quality.
  • Communication
  • Clarity, efficiency, rapidity, the tables filled by the referees, low fees, non-profit.
  • Offering a high quality journal, organised by and for the community.
  • Great mission!
  • I like that you require DOI references and require the arXiv version to be up to date.
  • The mission of quantum is admirable and the journal seems to have been implemented with some success. In all of the above fields Quantum has in my experience performed very well.
  • Referee reports have a nice and useful format.
  • Communication with the team was good. Publish process clearly explained.
  • The journal genuinely strives for high quality physics papers without too much ‘political’ influence in the decision. In all what i have seen it seems to achieve this objective.
  • Editors and refs. know the field, and take time to read a paper.
  • Communication was very efficient.
  • Open access, community driven, good quality – all nice things 🙂
  • Mostly everything.
  • Submission and publication are much easier than elsewhere due to arxiv linkingand the provided template.
  • Low fees give the possibility to everyone to publish their excellent science.
  • I wish your mission was shared by every community around the universe
  • I was happy with the experience in general.
  • Good Latex template, open accounting.
  • Facility in the submission process, clear mission, good editors.
  • I totally support Quantum mission. I particularly like the fact that the journal is community driven, and the attempt to judge manuscripts beyond concepts as “impact”, etc.

Where can Quantum improve?

  • make the acceptance/rejection decision more predictable; if it is based on subjective criteria, decide quickly; don’t reject based on subjective “importance” criteria after a lengthy reviewing process.
  • I think there is always room for improvement as the journal matures, the peer-review and editorial system seems to be moving towards a system that is very author-friendly, so no complaints there, but maybe at some point it would be nice to see some more editorials, invited opinion pieces, issue highlights, comment papers, and similar things, to give the journal a bit of a broader scope than ‘just’ standard research papers.
  • Opening referee discussions
  • If Quantum is to succeed, the referees need to be pushed to give reviews quickly, in a month or two time. I was going to submit a third paper here, but because I have to apply for faculty/postdoc soon, I decided to try another journal.
  • All fine. Minor: add submitted/corrected/accepted dates per publication.
  • Indeed, the Scholastica interface can be improved. For instance, there is no option for sharing the review details with the co-authors.
  • Open reviews.. maybe?
  • Scholastica is quite terrible under many aspects.
  • “all our editors are full-time researchers who volunteer their work for Quantum” this is actually the biggest issue. The initial referee reports were split, and the editor informed us that they were attempting to contact a third referee. They were unsuccessful in doing so. Some months later I ran into the individual who was contacted to be the third referee, and they indicated to me that they had responded that they could review the paper, but that the editor never replied to their response. In the end, no third referee was contacted, and the additional time and effort required to respond to referee misunderstandings lengthened the review process. While I understand the appeal of having editors who are high profile scientists, they are also those with the least amount of free time. It might be worthwhile to consider having a set of junior volunteer editors who are earlier career scientists.
  • The refereeing can maybe be done a bit quicker. Also, I think that it would help a lot to encourage referees to be a bit more critical in the review process.
  • I guess it will be good when the point is reached at which Quantum has an impact factor or has a better established place within the journal space. As a young researcher I am quite keen to publish here but I would not want to put all the eggs in one basket, if that makes sense. As a reviewer I have had some minor problems with the scholastica software, but reading the above it seems that you are on top of that.
  • It was extremely annoying to change the tex file from revtex to the quantum journal format, especially regarding references I had to enlist outside help of other people who were discouraged from submitting to Quantum based on this struggle. The information online was not very helpful or clear, I was reduced to looking at the source information of a different accepted paper’s arXiv posting to see how they hacked the references into the correct form.
  • Could you consider publishing Jupiter notebooks (in a publishable form?)
  • Clearer criteria for acceptance
  • 1) The style file does not work, at least not without some modifications when I try it use it with some fairly standard latex. I just decided not to use it. 2)Looking up DOI numbers is really annoying.
  • Unfortunately, journal metrics seems to govern the publication landscape and dictate author’s willingness to submit. Perhaps quantum could try to engage more with more mature/established researchers who can afford to be less concerned with journal metrics (unlike early career researchers such as myself).
  • The web platform. Co-authors should be also able to see the status of the paper, for instance.
  • I think you are addressing our only issue – the delayed decision
  • I would find a goal of < 12 weeks per referee round adequate. But i am well aware that often it is not the Editors who are delaying the procedure.
  • It would have been nice if there is a automatic way to get an official invoice for the publication fee.
  • The submission platform is not super intuitive to use.
  • Sending a manuscript to six referees and using the first three reports would produce faster feedback to authors.
  • You might allow authors to suggest a larger number of referees.
  • In the time for a single revision round. In our case our two rounds took each 3 months which is far longer than the average on other APS or IOP journals.
  • If you can figure out how to make it free to publish, that would be great. I don’t mind the cost — it’s a very small amount on a research grant — but it’s a bit of a bother, and it would eliminate semblance of a vanity press if it were free.
  • Our review process had a huge delay and lack of communication, because of the unresponsiveness of both referees and (in particular) of the editor in charge. It was not clear to us if/how referees and editors received any reminder about this, and in this respect the Scholastica website interface is definitely not helping the authors. Eventually we managed to get some prompt communication from a member of the Executive Board. This definitely speeded up the process. I had already pointed out these problems to the editors, and I am happy that some changes have been incorporated in the new guidelines made public by Quantum

Further comments on Quantum

  • happy to help if I can.
  • Good job Quantum! Keep it up!
  • The quality of the journal stands or falls by the quality of the referee reports. Quantum should think about how it can continue to ensure high quality refereeing by knowledgeable people, maybe give referee tokens or ‘best referee’ nominations. Good refereeing is an issue at many journals. I also believe that Quantum risks being too much of a ‘theorist’s journal’, giving preference to nice-looking theoretical results and analysis with limited applicability in practice, over results that actually push the field ahead.
  • I hope you get into SCI soon so that I can submit more manuscripts without risking retaliation from my institution.
  • My experience with Quantum has been no different from other journals, on average neither better or worse. Take that as you will.
  • Keep up the good work. Don’t hesitate to let us know how we can help.
  • My only negative experience came after acceptance when I needed to fix the paper’s formatting.
  • ❤️
  • please keep going!
  • Keep up the nice work 🙂
  • The problem of getting experimental papers into Quantum is a big one. Maybe they’ll start submitting once the impact factor is up, maybe you need to reach out. I don’t know.
  • Concerning delinquent reviewers (of which I saw no evidence of with my submissions, but just wanted to share my own experience as an editor), this is a problem for every journal that I suspect will never go away. My partial and perhaps obvious solution is to always (or whenever possible) ask for one more review than needed for a decision. When one review is late, I give them a week or two to produce a review, making it clear that a decision is imminent with or without their input. Doing this every time helps to reduce the likelihood of a long delay.
  • I really like the machine learning journal \href{https://distill.pub/}{Distill}. Hopefully someday one can create a similar journal for physics. (Maybe start with interactive plots)
  • I am convinced the ‘Quantum initiative’ is becoming more and more important (in particular also in view of ‘Plan S’), although the response in the community at the moment does not appear to be overly enthusiastic. I hope the response will change sooner or later.

And an extra question…

We added the following question at the end of the questionnaire:

Bonus question: have you suffered from burnout in your academic career?
This is unrelated to Quantum, optional and confidential. We will process it independently of all the answers above. We added this question in response to several occurrences of burnout among people involved with Quantum, at all levels (from the executive board to editors, referees and authors), which suggest that this may happen much more commonly than we hear about. If you answer, it will help us get an idea of how prevalent burnout is in the quantum community. With those statistics we could start a conversation with institutions on how to support researchers, and a conversation within the community on how to deal with it. By asking the question in the context of an unrelated survey, we hope to avoid survivor bias, in which only those who have experienced burnout would answer a questionnaire about it.

We obtained 48 responses (out of 56 questionnaire respondents). Of those 48:

  • 15 (31%) said “no, never”;
  • 13 (27%) said “maybe, not sure if it counts”;
  • 6 (13%) said “yes, once”, or a variation thereof;
  • 14 (29%) said “yes, several times”.

This is a small, limited sample, where ~40% of researchers self-reported experiencing academic burnout.