We were amazed by the detailed and lively discussion on various aspects of Quantum happening over at /r/quantumjournal. We would like to thank all participants for their time and over a hundred thoughtful comments! While this public discussion had the most visibility, we would also like to acknowledge all the private feedback that we received (in particular from applicants to the editorial call), as well as the myriad of offline lunch discussions taking place in research groups in various places.
Here we would like to go over and respond to some of the main arguments in the discussion about how selective Quantum should aim to be. The main question was whether Quantum should:
- Provide a curating service, by accepting only papers above a certain quality threshold (as judged by referees and editors), or
- Make no judgments on perceived “quality” or “significance” of papers, and accept all submissions that are technically correct and well presented.
Let us first quickly review three subjects: the role of journals, global constraints, and goals for Quantum.
The role of journals
Arguably, scientific publishing provides the following main services:
- Dissemination of research results,
- Peer-review for correctness,
- Curating according to topic and different quality and significance criteria.
In our field, the first point has been rendered redundant by the arXiv. The second is universally accepted as essential (although it can never be perfectly achieved). It is in the third point that different journals can distinguish themselves with different scopes and policies. Ideally, this “accreditation” service would be useful both for readers and for authors (see below).
Current issues and constraints
While Quantum advocates for changes in how government agencies and hiring committees evaluate candidates and proposals, it is aware of the current constraints imposed on researchers.
Indexing and bibliometrics are important factors in many decisions made, with variations across countries and institutions. Fellowships, permanent positions, grants and bonuses are conditioned on the number of publications in high impact journals. As much as many of the affected researchers agree that these are not good metrics for quality, they are not in a position to fight them and are effectively forced to try to get their works published in high impact journals.
There are problems with how most high impact journals function. With some exceptions, they are expensive (for authors or readers), encourage authors to oversell their results, and force articles into unnatural constraints, at the cost of clarity and reproducibility. Furthermore, papers are often handled by editors who are not experts actively working on the topic of a manuscript and cannot always judge the relative value of different referee reports, which may vary wildly in quality. Finally, copy-editing is often overzealous, at little added value for the reader and disproportionate effort for authors and final cost.
Goals for Quantum
Naturally, Quantum will not solve all the issues discussed above, but it presents an opportunity to demonstrate the viability of better models of science publishing and evaluation, and, hopefully, to lay the ground for a wider community discussion. For Quantum to have a lasting impact, it must strive for ambitious but realistic goals. Specifically, it aims at the following.
First two years:
- Engage the community and show that scientists can organize a reliable curating and quality control service at a fraction of the cost of traditional, profit-driven models of publishing.
- Become an attractive journal for researchers who support the cause and can take the risk of publishing in a new journal.
- Become known as a journal where papers are clearly written, and where the peer review process works towards improving papers.
After two years:
- Become an attractive journal for researchers in every stage of their career.
- Use the platform of Quantum to carefully experiment with, for example, dual-consent open review, payed review, or other ways of evaluating scientific research.
- Support other academic communities in launching similar initiatives.
- Promote a dialog among the scientific community, funding agencies, governmental bodies and universities about bibliometrics and how to improve hiring and funding policies.
The point of “becoming an attractive journal for researchers in every stage of their career” is crucial: Quantum should become an attractive publishing venue for the whole community of researchers in the field, including those who support Quantum’s vision but who, due to external constraints, cannot afford to take the risk of not publishing in a high-impact journal. Arguably this can be achieved in one of two ways: either the whole incentive system is overturned, or Quantum must become comparable to high-impact journals for the purposes of career advancement. In the short and medium term, only one of these options seems realistic. The decision to create Quantum as a selective journal is to a good extent based on the belief that it will give Quantum a much higher chance of making a difference.
With these goals and constraints in mind, let us now analyze the arguments presented.
Isn’t selecting for quality superfluous given the other criteria?
If all the other criteria are strictly enforced, then selecting, or even trying to estimate the quality of a work would be unnecessary. Quality would come for free, because authors required to honestly spell out the shortcomings of their work would not want to publish low quality works in Quantum in the first place.
We partially agree with this argument, and indeed, we very much hope that all the criteria for acceptance that Quantum defines in the editorial guidelines will help to ensure a high quality among the accepted papers.
However, there are works that are correct and honest in their self-assessment, but are not scientifically useful because of redundancy or poor motivation. An example would be a work presenting numerical simulations confirming a result that had already been proven analytically. The authors could rightfully claim that their work is on an interesting subject and that their results agree with theory, but the scientific value of the work would be negligible. Quantum would not want to publish such works.
Also, Quantum has realistic expectations towards authors. Honesty is a fuzzy concept. In the above example it is questionable whether, to be honest, the authors would really have to say in a prominent place that their results contain no information that wasn’t available before. In fact, they might not even be aware of the general rigorous proof.
The number of potential submissions below the quality bar is largely a symptom of the wider problems mentioned in the “Current issues and constraints” section. If those problems begin to subside, so that there is little incentive to submit to a journal unless authors believe their paper is suitable for it, the attention that editors and referees are asked to pay to quality and significance can be gradually reduced.
But quality is subjective!
Since quality and expected impact are subjective criteria, Quantum should not select on either of the two. Instead, Quantum should publish everything that is correct, as this is an objective criterion.
First we would like to point out that just because perceived impact and quality share a feature (they are to some extent subjective) they should not necessarily be treated equally. We wrote before about what we mean by quality or significance, and how that differs from estimated impact.
Coming back to the core of the problem, both judging quality and judging correctness are very hard tasks, and judgments of correctness are to some extent also subjective. Naturally, referees and editors will spend much less time reviewing a paper than the authors spend producing the results and writing them up. For experimental works, they will not be able to gain the same deep knowledge about the inner workings of a complicated experiment as the authors who spend years building the machine. It is unrealistic to expect that a referee will always be able to, for example, find a subtle misunderstanding by the authors of a result they cited from another paper, or to identify a particular faulty cable as the reason for an unexpected bump in a plot of experimental data.
So, for both assessments of quality and correctness we will have to put some trust in the referees, editors and authors. Instead of giving up on evaluating quality because it is “subjective”, Quantum wants to put procedures and policies in place that discourage very subjective judgements and that make it easier for referees to produce useful reports. This is why the referee report form of Quantum is so detailed.
Less demand for non-selective journals
One point that was not very prominent in the discussion, but which was an important factor during our initial decision process for proposing Quantum as a selective journal, is that there is less demand for another non-selective journal.
There are plenty of both traditional and open-access journals that have very low bars for acceptance apart from correctness, with varying quality of peer review. It is in the realm of selective journals that more alternatives are needed and where Quantum can distinguish itself by promoting a different incentive structure and editorial policies.
One could launch two journals, e.g. “Quantum” and “Quantum Proceedings”, such that Quantum Proceedings would judge only correctness, clarity and reproducibility of submissions, whereas Quantum could be more selective. It would be up to the authors to choose where to submit. Perhaps correct papers submitted to Quantum that did not meet the selection criteria could be redirected to Quantum Proceedings.
This is an interesting proposal and we will keep it in mind. At the moment we want to focus our limited time and energy into creating one journal. Introducing a second journal as described is one of the experiments that we can try later on. The demand for such a journal can be discussed again with the community after the first year of operation of Quantum, for example.
Competition with other journals
If Quantum tries to be selective it is trying to beat traditional publishers at their own game and this is bound to fail.
Aiming for high quality standards does not mean that Quantum is trying to beat traditional publishers at their game. Quantum will still have different acceptance criteria than other journals. Quantum will never ask referees or editors to judge the impact of a paper. Referees are guided through specific questions to judge the quality of the paper, and the editor should be an expert in the subject. Quantum will publish negative results as well as technical results of interest to small communities, and focus on clarity, correctness and reproducibility.
Quantum will not have broad interest as a strict requirement, like the Physical Review Letters, nor the emphasis on cross-community research of Physical Review X. There will be many papers that fit better into some of the established journals, and Quantum is aiming for peaceful coexistence and healthy competition with them.
In addition, Quantum is by nature very different from commercial publishers: Quantum does not want to sell a product and cannot make profits. Quantum’s ruling body is a steering board composed only of active researchers. It is set up as a non-profit and designed to serve the community.
Assessing significance a posteriori
It is hard to judge the relevance and scientific impact of a paper at the date of publication. Quantum could instead regularly reward the best papers of the year a posteriori.
Indeed. After the first couple of years of operation, Quantum can introduce awards for the best papers. There could also be community recognition awards and editorial highlights when papers published in Quantum find new applications. Quantum could also publish the access data of published papers. These options will be discussed with the community and the steering board in due time. We are also looking into more sophisticated community feedback features, and we will provide more information about this soon.
Our overall take from the arguments presented here and in the discussion is that the editorial policies suit the current needs of most of the community. We acknowledge that every implementation of a tool towards a cause is bound to have drawbacks and limitations, determined partly by the nature of the wider system. With Quantum we hope to start making a difference where it is needed the most: among selective journals. We hope to lead by example, by having fairer selection criteria, encouraging honest, clear, correct and reproducible papers, and keeping a peer review system that works to improve them. We are aware of the risk of conflating “quality” with “perceived impact”, and we count on the community of editors, referees and authors to keep Quantum in check. Finally, we are willing to discuss several of the suggestions presented (open review, retroactive awards, a second journal) after the initial effort of launching Quantum.
In an upcoming post we will address the question of to what extent Quantum should require open data. In the meantime, there is one day left in the call for editors (deadline 15th of September)!
1 thought on “Should Quantum be selective?”
Pingback: Links for November 2016 – foreXiv
Comments are closed.