REST API on Crossref

Retraction Watch retractions now in the Crossref API

Martyn Rittman — Wed, 29 Jan 2025 00:00:00 +0000

Retractions and corrections from Retraction Watch are now available in Crossref’s REST API. Back in September 2023, we announced the acquisition of the Retraction Watch database with an ongoing shared service. Since then, they have sent us regular updates, which are publicly available as a csv file. Our aim has always been to better integrate these retractions with our existing metadata, and today we’ve met that goal.

This is the first time we have supplemented our metadata with a third-party data source. Until now, our APIs have included metadata provided by Crossref members along with outputs from our internal enrichment workflows, such as matches found for bibliographic reference matching and funders. Third party metadata has been gathered in Event Data, but this has been stored and delivered separately.

Knowing when work has been retracted is critical for assessing the integrity of research, and this enhancement of the data will be a great benefit to the community.

Where does the data come from?

Retraction Watch carefully curates retractions, pulling them from several non-Crossref sources, including PubMed and publisher websites. Each entry is manually checked and annotated before being added to the database. The high level of curation and broad coverage is what made a partnership between Crossref and Retraction Watch attractive, and our shared goal of making changes to metadata more visible.

“Our goal with the Retraction Watch Database has always been for it to be as useful to as many people as possible, and available from as many sources as possible,” says Ivan Oransky, co-founder of Retraction Watch and executive director of The Center For Scientific Integrity, its parent nonprofit organisation. “Integration with Crossref’s REST API is a huge step in that direction.”

Where can I see the retractions?

If you use a service that collects Crossref metadata, you will start to see the Retraction Watch retractions as they are picked up. To access the data directly, you can find retractions from both Crossref members and Retraction Watch in our REST API, for example with the following request for all retractions:

https://api-crossref-org.pluma.sjfc.edu/v1/works?filter=update-type:retraction

Or for an individual record:

https://api-crossref-org.pluma.sjfc.edu/v1/works/10.1177/17588359231172420

In the results here you will see an update-to field:

"update-to": [
{
"updated": {
"date-parts": [
[2023,4,22]
],
"date-time": "2023-04-22T00:00:00Z",
"timestamp": 1682121600000
},
"DOI": "10.1177/1758835920922055",
"type": "retraction",
"source": "publisher",
"label": "Retraction"
},
{
"updated": {
"date-parts": [
[2023,4,22]
],
"date-time": "2023-04-22T00:00:00Z",
"timestamp": 1682121600000
},
"DOI": 10.1177/17588359231172420",
"type": "retraction",
"source": "retraction-watch",
"label": "Retraction",
"record-id": 44124
}
]

The source field states where the retraction came from. Currently, it can have two values: publisher or retraction-watch. Note that the same retraction may be included multiple times from different sources.

Retraction Watch retractions will remain available on Gitlab in csv format and be updated on working days. The record-id refers to the entry in the csv file with further details, such as the reason for retraction.

There is full documentation available for the Crossref REST API and if you are new to REST APIs, see our learning hub to get started which includes a tutorial about accessing retractions.

What can I do with the retractions?

Like the rest of our metadata, the retractions are freely available. If you use or operate a tool that ingests retractions, the new entries will start to be picked up immediately. The Retraction Watch database includes a larger number of retractions than the Crossref database, so you should see an increase in the total.

We have heard from organisations that would like to build new research integrity tools based on this data. We look forward to seeing the benefits brought by wider availability of the Retraction Watch retractions, and how they can provide better context to research outputs.

While Crossref metadata is freely available to reuse without a license, if you make use of the Retraction Watch retraction metadata in a published work, we kindly request that you provide a citation to the source.

If you have questions or comments, please head over to the section of our forum dedicated to integrity of the scholarly record.

Behind the scenes improvements to the REST API

Patrick Polischuk — Tue, 06 Jul 2021 00:00:00 +0000

UPDATE, 24 August 2021: All pools have been migrated to the new Elasticsearch-backed API, which already appears to be more stable and performant than the outgoing Solr API. Please report any issues via our Crossref issue repository in Gitlab.

UPDATE, 9 August 2021: The cutovers for the polite and Plus pools are delayed again. We’re still working to ensure acceptable performance and stability before serving responses from the new application and infrastructure. Each cutover is currently delayed by one more week–the polite pool is scheduled for 2021 August 17 and the Plus pool is scheduled for 2021 August 24.

UPDATE, 2 August 2021: The cutovers for the polite and Plus pools are delayed. We’ve been mirroring traffic to the new polite pool and want to ensure acceptable performance and stability before serving responses from the new application and infrastructure. Each cutover is currently delayed by one week–the polite pool is scheduled for 2021 August 10 and the Plus pool is scheduled for 2021 August 17.

UPDATE, 13 July 2021: The first stage of the cutover is complete, so requests to the public pool are now being served by the new REST API. We took a slightly different approach to performing the cutover, so the “Documentation” and “Temporary domain” sections below have been updated.

Our REST API is the primary interface for anybody to fetch the metadata of content registered with us, and we’ve been working hard on a more robust REST API service that’s about to go live.

The REST API is free to use and it gets around 300 million requests each month (we encourage users to adhere to our etiquette guidelines to keep things running smoothly). It is used for bibliometric studies, by platforms like Dimensions, by organisations like the National Library of Sweden, and to support countless other efforts.

We also offer enhanced access to our APIs and other services with Metadata Plus, and we recommend it for production services and others that benefit from guaranteed up-time, a higher rate limit, and priority support from our helpful staff.

For a while now, we’ve been working to migrate the REST API from Solr to Elasticsearch and from our datacenter to a cloud platform in order to address issues of scalability and extensibility.

We’re pleased to announce that we’ll be cutting over to the Elasticsearch-backed version of the REST API over the next few weeks, beginning July 13. This cutover will occur one pool at a time–the public pool will be migrated first, followed by the polite pool on August 3, and the Plus pool on August 10 (see ’etiquette’ link above if you’re unfamiliar with our different pools). Please note updates at the top of this post for changes to the original schedule.

We’ve thoroughly tested the functionality and performance of the new REST API, and we’d like to invite you to test it out before we move production traffic to the new service. Try out your favorite API queries at https://api-production-crossref-org.pluma.sjfc.edu/.

Feature parity, but note a few differences

One of our primary objectives was to maintain feature parity between the old and new services, avoiding any breaking changes that might cause problems for existing services integrating with the REST API. We implemented a regression test suite which has given us the confidence to make such a foundational change. During the course of this project, we found it necessary and a good opportunity to make a few modifications. In each case, we analyzed usage and aimed to avoid making any breaking changes. We hope these represent improvements to the behavior and consistency of the REST API.

The group-title filter uses exact matching. This filter previously worked but was undocumented and unsupported.
The directory filter is deprecated. This was meant to be an experimental, unsupported filter, and the data has not met the standard we require.
The affiliation facet returns counts of affiliation strings rather than counts of terms within affiliation fields (thus resolving this Github issue).
Cursors may be used to page through results from the /members, /funders, and /journals routes, in addition to /works.
While we suggest that everyone use cursors for pagination, we still support the offset functionality. We have introduced a limit of 80000 for offset values for the /members /funders and /journals routes
offset behavior is slightly changed, now applying to the sum of rows and offsets rather than just offsets.
The published field is now present in API responses.
The /licenses route returns paged results.
Sorting by submitted is no longer supported. This was never officially supported or documented.
The /quality route has been removed. This was an undocumented, experimental feature.
Funder name in /works metadata is the name provided by the publisher.
Empty relation fields correctly return an empty object.
Only ISBN and isbn-type for a record will be returned. ISBNs for associated volumes will be omitted.
The institution field is a list.
query uses different stop word defaults, though we expect querying to remain roughly the same.
API responses may feature slightly different scores, as they come from different backends.

Some technical notes on the cutover

Documentation

The above changes are documented in our new REST API documentation, which is now automatically generated via Swagger, resulting in more comprehensive coverage and more efficient feature development. During the cutover, the right documentation for you will depend on which pool you are using. The documentation for the new API can be found by visiting the API in a browser, or by navigating to https://api-crossref-org.pluma.sjfc.edu/help; and the docs for the old API remain here: https://github.com/CrossRef/rest-api-doc. The Github-hosted documentation will be deprecated once the cutover is complete.

This may not come as news, but bears repeating as we mentioned GitHub. We have moved our source code repositories from GitHub to GitLab, including all of our issue tracking.

Temporary domain

UPDATE: We ended up performing the public pool cutover via reverse proxies instead of redirects–please disregard the note about temporary domains below. The api.crossref.org domain will remain the domain regardless of which pool you’re using or where we are in the cutover process.

Please note that the api.production.crossref.org domain is a temporary domain we are using during this cutover period. Traffic will be redirected to the new service one pool at a time via a 307 http redirect. Once the cutover is complete, we will go back to using the api.crossref.org domain. Do not update any software, scripts, libraries, tools, etc. to use the temporary domain.

Differences in query results

Due to inherent differences in how Solr and Elasticsearch perform queries and rank results, you may see slightly different results when comparing the old and new services. If for whatever reason your workflow involves using multiple API pools (which we don’t recommend), you may see inconsistent results.

Cursor behavior

Cursors may break if your script is paging through results at the exact moment the cutover is performed, and you should retry your request once the release is complete. We will post the precise maintenance window to https://status-crossref-org.pluma.sjfc.edu/.

Filing issues

Feature requests and bug reports should be filed into the Crossref issue repository in Gitlab during this testing phase and once the new Elasticsearch-backed API is live in production.

Coming next

While we hope the benefits of improved stability and extensibility are as exciting to you as they are to us, “feature parity” may not be the most thrilling message for our API users. In truth, one of the more exciting aspects of completing this migration is the end of the code freeze we instituted at the start of this effort. Now, we can work on new feature development and a continuous stream of bug fixes. We also improved the automatic test coverage as part of the work, meaning we can deliver features with greater confidence.

The first new feature we’ll be delivering via the REST API will be support for the “grants” record type, allowing for the retrieval of metadata for grants that have been registered with us, now numbering over 20,000 from 8 different funder members. This work is well underway and will be released once we are confident that the new REST API is stable in production. From there, we’ll continue to select the highest priority issues from our REST API backlog.

As always, should you have any questions about our REST API, check out the metadata retrieval section of our website, start a discussion on our community forum, file a Gitlab issue as mentioned above, or you can contact us via support@crossref.org.