Peer Review on Crossref

Get involved with Peer Review Week 2020 and register your peer reviews with Crossref

Amanda Bartell — Mon, 21 Sep 2020 00:00:00 +0000

Just when you thought 2020 couldn’t go any faster, it’s Peer Review week again! Peer Review is such an important part of the research process and highlighting the role it plays is key to retaining and reinforcing trust in the publishing process.

As the Peer Review Week team states:

“Maintaining trust in the peer review decision-making process is paramount if we are to solve the world’s most pressing problems. This includes ensuring that the peer review process is transparent (easily discoverable, accessible, and understandable by anyone writing, reviewing, or reading peer-reviewed content) and that everyone involved in the process receives the training and education needed to play their part in making it reliable and trustworthy.”

A key way that publishers can make peer reviews easily discoverable and accessible is by registering them with Crossref - creating a persistent identifier for each review, linking them to the relevant article, and providing rich metadata to show what part this item played in the evolution of the content. It also gives a way to acknowledge the incredible work done by academics in this area.

For Peer Review week last year, Rosa and Rachael from Crossref created this short video to explain more.

Fast forward to 2020 and over 75k peer reviews have now been registered with us by a range of members including Wiley, Peer J, eLife, Stichting SciPost, Emerald, IOP Publishing, Publons, The Royal Society and Copernicus. We encourage all members to register peer reviews with us - and you can keep up to date with everyone who is using this API query. (We recommend installing a JSON viewer for your browser to view these results if you haven’t done so already).

Register peer reviews and contribute to the Research Nexus

At Crossref, we talk a lot about the research nexus, and it’s a theme that you’re going to hear a lot more about from us in the coming months and years.

The published article no longer has the supremacy it once did, and other outputs - and inputs - have increasing importance. Linked data and protocols are key for reproducibility, peer reviews increase trust and show the evolution of knowledge, and other research objects help increase the discoverability of content. Registering these objects and stating the relationships between them support the research nexus.

Peer reviews in particular are key to demonstrating that the scholarly record is not fixed - it’s a living entity that moves and changes over time. Registering peer reviews formally integrates these objects into the scholarly record and makes sure the links between the reviews and the article both exist and persist over time. It allows analysis or research on peer reviews and highlights richer discussions than those provided by the article alone, showing how discussion and conversation help to evolve knowledge. In particular, post-publication reviews highlight how the article is no longer the endpoint - after publication, research is further validated (or not!) and new ideas emerge and build on each other. You can see a real-life example of this from F1000 in a blog post written by Jennifer Lin a few years ago.

As we’ve said before:

Article metadata + peer review metadata = a fuller picture of the evolution of knowledge

Registering peer reviews also provides publishing transparency and reviewer accountability, and enables contributors to get credit for their work. If peer review metadata includes ORCID IDs, our ORCID auto-update service means that we can automatically update the author’s ORCID record (with their permission), while our forthcoming schema update will take this even further, making CRediT roles available in our schema.

How to register peer reviews with Crossref

You need to be a member of Crossref in order to register your peer reviews with us and you can currently register peer reviews by sending us your XML files. Unfortunately, you can’t currently register peer reviews using our helper tools like the OJS plugin, Metadata Manager, or the web deposit form.

You can find out more about registering peer reviews on our website - we even have a range of markup examples.

We know that there’s a range of outputs from the peer review process, and our schema allows you to identify many of them, including referee reports, decision letters, and author responses. You can include outputs from the initial submission only, or cover all subsequent rounds of revisions, giving a really clear picture of the evolution of the article. Members can even register content for discussions after the article was published, such as post-publication reviews.

Get involved with Peer Review Week 2020

We’re looking forward to seeing the debate sparked by Peer Review Week and hearing from our members about this important area. You can get involved by checking out the Peer Review Week 2020 website or following @PeerRevWeek and the hashtags #PeerRevWk20 #trustinpeerreview on Twitter.

We’re excited to see what examples of the evolution of knowledge will be discoverable in registered and linked peer reviews this time next year!

Peer review publications

Jennifer Lin — Sun, 12 Aug 2018 00:00:00 +0000

Peer review publications—not peer-reviewed publications, but peer reviews as publications

Our newest dedicated record type—peer review—has received a warm welcome from our members since rollout last November. We are pleased to formally integrate them into the scholarly record, giving the scholars who participated credit for their work, ensuring readers and systems dependably get from the reviews to the article (and vice versa), and making sure that links to these works persist over time.

Many of our members make the peer review history available to researchers in different ways. Their extra effort to post review materials alongside the article will now go further once they are registered with us and linked to the journal article. They spoke of publishing peer reviews as a standard part of their publishing operation. The scholarly contributions of their editors and referees are validated, stewarded, and published in the manner of the articles: as per general practice. To fully realize this, they are ensuring that these publications are discoverable, citable, and part of the formal scholarly record—for all the thousands of systems which draw on Crossref metadata.

Article metadata + peer review metadata = a fuller picture of the evolution of knowledge

The growing collection

As of August 12, 2018 three publishers have registered 12446 peer reviews in the dedicated resource type schema we rolled out last November. PeerJ (10.7287) with 12015 at time of writing and Stichting SciPost (10.21468) with 297 works. ScienceOpen (10.14293) has registered 126 reviews of papers on their post-publication platform.

The peer review metadata collected is partly similar, though otherwise unique to other content. In the former, general metadata that we accept for the articles, as well as the reviews, include an ORCID iD to identify the reviewer, editor, and/or author 0; license 0. This metadata is quite distinct from the article and is important to collect, not only as a discrete publication in its own right, but also to provide richer context for the actual results shared in the associated article. They are authored by different people than the paper’s contributors (author response/rebuttal excepting). They need not have the same license.

Currently, none of this data has been registered. (From the publishers we’ve talked to, this is largely due to factors related to limitations in their technology systems.) And like other record types, we link up scholarly materials in the metadata and fill in the research nexus graph through relations.

There’s no better way to understand peer review metadata than to look at real examples from our members:

PeerJ review (https://doi-org.pluma.sjfc.edu/10.7287/peerj.2707v0.1/reviews/1) and its metadata (https://api-crossref-org.pluma.sjfc.edu/works/10.7287/peerj.2707v0.1/reviews/1)
ScienceOpen review (https://doi-org.pluma.sjfc.edu/10.14293/s2199-1006.1.sor-uncat.a5995373.v1.rhrmgu) and its metadata (https://api-crossref-org.pluma.sjfc.edu/works/10.14293/s2199-1006.1.sor-uncat.a5995373.v1.rhrmgu)
SciPost review (https://doi-org.pluma.sjfc.edu/10.21468/scipost.report.10) and its metadata (https://api-crossref-org.pluma.sjfc.edu/works/10.21468/scipost.report.10)

Review-specific metadata is also critical to capturing the shape of the scholarly discussion. These include:

Review date (required)
Scholarly work reviewed (required)
Recommendation
Revision stage
Review round
Contributor name

PeerJ, SciPost, and ScienceOpen have registered this whole set where applicable (review round not applicable to post-publication reviews), with the exception of the recommendation.

Scholarly contributions captured in time

Published peer reviews uniquely highlight the nature of research ideas evolving over time, spotlighting the nature of this as a collective effort involving multiple individuals. The more metadata, the bolder the story. We have created a set of reference metadata (fictitious) to illustrate this phenomenon. Josiah Carberry submits a manuscript to the Journal of Psychoceramics, entitled “Dog: A Methodology for the Development of Simulated Annealing.” It undergoes two rounds of review with two referees each round. The article https://doi-org.pluma.sjfc.edu/10.5555/12345681 is published and registered on May 6, 2012 along with the history of peer review materials on the same day:

First submission

Referee report 1 - https://doi-org.pluma.sjfc.edu/10.5555/12345681.9879
Referee report 2 - https://doi-org.pluma.sjfc.edu/10.5555/12345681.9880
Editor decision - https://doi-org.pluma.sjfc.edu/10.5555/12345681.9881

Revision round 1

Author rebuttal - https://doi-org.pluma.sjfc.edu/10.5555/12345681.9882
Referee report 1 - https://doi-org.pluma.sjfc.edu/10.5555/12345681.9883
Referee report 2 - https://doi-org.pluma.sjfc.edu/10.5555/12345681.9884
Editor decision - https://doi-org.pluma.sjfc.edu/10.5555/12345681.9885

Published reviews can show peer feedback in progress; the progress of scholarly discussion unfolding, as expert ideas build upon each other. Many of us have traditionally located the article’s publication as the climactic event, but the story in fact doesn’t end there. Pre-publication becomes post-publication. Throughout this time, research is validated and sprouts into new ideas.

Peer review platform Publons is working on getting reviews authored on its platform registered with us. Doing so will mean that PeerJ article, “Transformative optimisation of agricultural land use to meet future food demands” by Lian Pin Koh, Thomas Koellner, and Jaboury Ghazoul https://doi-org.pluma.sjfc.edu/10.7717/peerj.188 with three scholarly discussions published over the course of peer review, would also be accompanied by a fourth that occurred after publication from Gene A. Bunin https://publons.com/publon/3374/, not yet registered.

Research begets research

In my investigation of review publications registered, two examples cropped up, highlighting the richness of the research process not only as it shows a set of research results evolve through scholarly discussion, but as it is then folded into new research outputs.

A PeerJ article “Software citation principles” https://doi-org.pluma.sjfc.edu/10.7717/peerj-cs.86 has had a very rich life: https://api-crossref-org.pluma.sjfc.edu/works/10.7717/peerj-cs.86. It was originally submitted as a preprint and underwent multiple iterations of improvement (https://doi-org.pluma.sjfc.edu/10.7287/peerj.preprints.2169, https://doi-org.pluma.sjfc.edu/10.7287/peerj.preprints.2169v1, https://doi-org.pluma.sjfc.edu/10.7287/peerj.preprints.2169v2, etc.). It then was subjected to peer review. And three referee reports are published alongside the final publication:

We glimpse a view of time unfolding here:

NB: in the review metadata, all the dates provided reference September 19, 2016 when they were published with the accompanying research article. To really make the metadata useful, we recommend providing the date the review was received, rather than published (for publishers who are publishing pre-publication review materials).

The reviews were then cited in three versions of the F1000Research article, “A multi-disciplinary perspective on emergent and future innovations in peer review” (https://doi-org.pluma.sjfc.edu/10.12688/f1000research.12037.1, https://doi-org.pluma.sjfc.edu/10.12688/f1000research.12037.2, and https://doi-org.pluma.sjfc.edu/10.12688/f1000research.12037.3). These three all link up on the Crossref metadata map. The visualization below is only an entrypoint into this picture of research dissemination and the spread of ideas.

András Láng served as a reviewer for a paper by Danilo Garcia and Fernando R. González Moraga published as “The Dark Cube: dark character profiles and OCEAN” (https://doi-org.pluma.sjfc.edu/10.7717/peerj.3845). As of the blog release date, this paper has been cited by two sources:

Source: https://doi-org.pluma.sjfc.edu/10.7717/peerj.3845, CC-BY 4.0

What this view of the paper does not reveal is that Láng’s review (https://doi-org.pluma.sjfc.edu/10.7287/peerj.3845v0.1/reviews/2) provided such insight to the original researchers that the first author (Garcia) incorporates the discussion in his subquent work. This evidence is documented in the citation list of that new publication, “Encyclopedia of Personality and Individual Differences” https://doi-org.pluma.sjfc.edu/10.1007/978-3-319-28099-8_2302-1. What a wonderful illustration of the ways in which peer reviews can operate like other publications, and how far is it from being unique. But up to now, we have not yet programmatically captured them in a formal way as we do now with these materials registered properly as a review.

The evolution of Crossref’s piece

In the same spirit of ever evolving knowledge, we also continue to update our schemas based upon community feedback. Are references important? Tell us! What new metadata on peer reviews are important to answer your questions or help you do what you need? Members, if you are interested in registering your peer review content with us, please get in touch.

Peer reviews are open for registering at Crossref

Jennifer Lin — Tue, 24 Oct 2017 00:00:00 +0000

About 13-20 billion researcher-hours were spent in 2015 doing peer reviews. What valuable work! Let’s get more mileage out of these labors and make these expert discussions citable, persistent, and linked up to the scholarly record. As we previously shared during Peer Review week, Crossref is lauintroducing support for a new record type to support the registration of peer reviews. We’re one step closer to changing that. Today, we are excited to announce that we’re open for deposits.

If you missed the first episode, here’s a recap:

Publishers have been registering reviews with us for a while (ex: Example 1, Example 2, and Example 3). But these have been shoehorned into other content: article, dataset, or component. So we are extending Crossref’s infrastructure to properly treat this special scholarly artifact. This includes a range of outputs made publicly available from the peer review history (referee reports, decision letters, author responses, community comments) across any and all review rounds. We welcome scholarly discussions of journal articles before or after publication (e.g. “post-publication reviews”).

We collect metadata that characterizes the peer review asset (for example: recommendation, type, license, contributor info, competing interests). We also collect metadata, which offers a view into the review process (e.g. pre/post-publication, revision round, review date).

This special set will support the discovery and investigation of peer reviews as it is linked up to the article discussed. It will also enable the following:

Enable tracking of the evolution of scholarly claims through the lineage of expert discussion
Support enrichment of scholarly discussion
Enable reviewer accountability
Credit reviewers and editors for their scholarly contribution
Support publisher transparency
Connect reviews to the full history of the published results
Provide data for analysis and research on peer review

Please come check out our documentation for more information.

As publishers are implementing this, we are finishing up the delivery of this metadata for machine and human access, across all the Crossref interfaces (REST API, OAI-PMH, Crossref Metadata Search) to enable discoverability across the research ecosystem. We are also working to make it possible for members to get Cited-by data for the peer reviews they register.

If you are interested in registering your peer review content with us, please get in touch.

Making peer reviews citable, discoverable, and creditable

Ginny Hendricks — Mon, 11 Sep 2017 00:00:00 +0000

A number of our members have asked if they can register their peer reviews with us. They believe that discussions around scholarly works should have DOIs and be citable to provide further context and provenance for researchers reading the article. To that end, we can announce some pertinent news as we enter Peer Review Week 2017 : Crossref infrastructure is soon to be extended to manage DOIs for peer reviews. Launching next month will be support for this new resource/record type, with schema specifically dedicated to the reviews and discussions of scholarly content.

Not disimilar to other registered resources (datasets, working papers, preprints, translations, etc.) publication peer reviews are important scholarly contributions in their own right and form a part of the scholarly record. In addition to the members who have been registering them, many more are looking to better handle these contributions and give recognition to this process which is so critical to maintaining scientific quality.

Here are a few examples of existing Crossref DOIs for peer reviews: https://doi-org.pluma.sjfc.edu/10.1016/j.engfracmech.2015.01.019 and https://doi-org.pluma.sjfc.edu/10.5194/wes-1-177-2016 and https://doi-org.pluma.sjfc.edu/10.14322/PUBLONS.R518142.

We are extending our infrastructure to support all members who make these scholarly discussions available to readers. To accommodate a wide range of publisher practices, this will include a range of outputs made publicly available from the peer review history, across any and all review rounds, including referee reports, decision letters, and author responses. Members will be able to include not only scholarly discussions of journal articles before but also after publication (e.g. “post-publication reviews”).

Central to this new feature of the Crossref Content Registration service is the special set of metadata dedicated to supporting the discovery and investigation of peer reviews as it is linked up to the article discussed. The peer review schema will provide a characterization of the peer review asset (for example: recommendation, type, license, contributor info, competing interests) as well as offer a view into the review process (e.g. pre/post-publication, revision round, review date).

Our custom support for peer reviews will ensure that:

Readers can see provenance and get context of a work
Links to this content persist over time
The metadata is useful
They are connected to the full history of the published results
Contributors are given credit for their work (we will ask for ORCID iDs)
The citation record is clear and up-to-date.

As with all the content registered with Crossref, we will make peer review metadata available for machine and human access, across multiple interfaces (e.g. REST API, OAI-PMH, Crossref Metadata Search) to enable discoverability across the research ecosystem. This metadata may also support enrichment of scholarly discussion, reviewer accountability, publishing transparency, analysis or research on peer reviews, and so on.

To reflect the nature of this special content, we will bundle the fees for peer review content fees into the cost of registering the article for members who publish the journal article and its peer reviews. No matter how many reviews are associated with a paper, there will be a fixed fee for the full set.

Peer review infrastructure will arrive at Crossref in one month, and we are excited to engage our members who want to assign DOIs to peer reviews or migrate previously registered review content to the new schema. A special thanks to the members so far who have given feedback and advice to develop the schema: BMC, The BMJ, Copernicus, eLife, PeerJ, and Publons.

Please contact our membership specialist if you’d like to know more.