Research Integrity on Crossref

Why metadata matters for research integrity: a new joint guide from Crossref and DataCite

Madhura Amdekar — Thu, 23 Apr 2026 00:00:00 +0000

Preserving the integrity of the scholarly record is an important component of the overall endeavour to protect research integrity. Open scholarly infrastructure enables persistent recording of research objects and associated metadata, which provides an evidence trail for these objects for all in the research community. Crossref and DataCite – as providers of essential infrastructure for preservation of the scholarly record – we share our joint expertise in the new guide on “Why metadata matters for research integrity and how to contribute”.

Both our organisations enable our members to share metadata about the research outputs, resources, and activities that they produce and steward. That metadata about scholarly outputs provides important information about them, which can help evidence integrity. To highlight how and which elements in the Crossref and DataCite metadata schemas support this endeavour, we are excited to make our new guide, available at https://doi-org.pluma.sjfc.edu/10.5281/zenodo19695957.

This joint guide offers practical information for all stakeholders in the scholarly ecosystem about the different metadata elements supported by Crossref and DataCite that can help in assessing the integrity of the scholarly record. With growth in the type and complexity of research outputs over the years, there is also a growing need to be able to ascertain the trustworthiness of research outputs. Metadata directly supports this function. Creators and stewards of research outputs can provide metadata about the content that they produce, including information on who authored the work, who funded it, which other works it cites, whether it was updated after publication, how it relates to other items in the research ecosystem, and more. This guide lists the metadata elements that capture this information in the Crossref and DataCite metadata schemas and the important role played by each of them in assessing integrity and rigour. We hope that by knowing more about the applications of metadata for preserving the integrity of the scholarly record, prospective authors, researchers, publishers, repositories, integrators, and funders will be encouraged to contribute rich and accurate metadata when registering DOI records.

This information can also be leveraged by researchers and users of metadata who are looking to incorporate open metadata into their tools and analyses. All of the metadata in Crossref and DataCite can be accessed via open APIs and public data files. This guide contains details of what each of the metadata elements can be used for, helping you to identify the right data that you need for your analyses of interest, such as looking at citation patterns, network analysis, and other research integrity trends.

As you read through this guide, please share your feedback and any questions that you may have via the Crossref community forum.

Metadata for research integrity: a joint guide from Crossref and DataCite

Helena Cousijn — Wed, 01 Apr 2026 00:00:00 +0000

Why metadata matters for research integrity and jow to contribute

Some aspects of research integrity depend on accurate, complete, and connected metadata. This joint guide from Crossref and DataCite sets out the metadata elements most critical for assessing research integrity—and how all stakeholders can contribute to and benefit from a richer, more trustworthy scholarly record.

Strategists

Understand why metadata is infrastructure for research integrity. Why completeness, accuracy, and openness across the scholarly record matters for systemic trust in research.

Decision-makers

Know which metadata elements to require, check, and act on. A practical resource for publishers, funders, and institutions making metadata-related policy decisions.

Practitioners

Learn how to deposit, enrich, and query research integrity metadata. Step-by-step guidance using Crossref and DataCite services and open APIs.

What this guide covers

This guide walks through the key metadata elements that enable research integrity assessment, and explains how to contribute them via Crossref and DataCite:

Contributors and their roles—identifying who did what
Affiliations—linking researchers to institutions
Dates—submission, acceptance, publication, and update dates
Funder, funding, and grant information—transparency on who paid for the research
Versioning—tracking how a work has changed over time
Retractions, corrections, and updates—keeping the record accurate
Abstracts and descriptions—what the work is actually about
Clinical trials—registration and reporting
References—connecting works to what they cite
Peer reviews—when and how work was reviewed
Publisher and steward—accountability for the record
Record and resource types—what kind of object is this?
Relationships and related identifiers—linking datasets, preprints, articles, and more
Rights, licences, and access—how the work can be used

The guide also sets out a call to action for each stakeholder group: how to enrich your records, query existing metadata via APIs, and report inconsistencies.

Read the full guide

Loading…

Retraction Watch retractions now in the Crossref API

Martyn Rittman — Wed, 29 Jan 2025 00:00:00 +0000

Retractions and corrections from Retraction Watch are now available in Crossref’s REST API. Back in September 2023, we announced the acquisition of the Retraction Watch database with an ongoing shared service. Since then, they have sent us regular updates, which are publicly available as a csv file. Our aim has always been to better integrate these retractions with our existing metadata, and today we’ve met that goal.

This is the first time we have supplemented our metadata with a third-party data source. Until now, our APIs have included metadata provided by Crossref members along with outputs from our internal enrichment workflows, such as matches found for bibliographic reference matching and funders. Third party metadata has been gathered in Event Data, but this has been stored and delivered separately.

Knowing when work has been retracted is critical for assessing the integrity of research, and this enhancement of the data will be a great benefit to the community.

Where does the data come from?

Retraction Watch carefully curates retractions, pulling them from several non-Crossref sources, including PubMed and publisher websites. Each entry is manually checked and annotated before being added to the database. The high level of curation and broad coverage is what made a partnership between Crossref and Retraction Watch attractive, and our shared goal of making changes to metadata more visible.

“Our goal with the Retraction Watch Database has always been for it to be as useful to as many people as possible, and available from as many sources as possible,” says Ivan Oransky, co-founder of Retraction Watch and executive director of The Center For Scientific Integrity, its parent nonprofit organisation. “Integration with Crossref’s REST API is a huge step in that direction.”

Where can I see the retractions?

If you use a service that collects Crossref metadata, you will start to see the Retraction Watch retractions as they are picked up. To access the data directly, you can find retractions from both Crossref members and Retraction Watch in our REST API, for example with the following request for all retractions:

https://api-crossref-org.pluma.sjfc.edu/v1/works?filter=update-type:retraction

Or for an individual record:

https://api-crossref-org.pluma.sjfc.edu/v1/works/10.1177/17588359231172420

In the results here you will see an update-to field:

"update-to": [
{
"updated": {
"date-parts": [
[2023,4,22]
],
"date-time": "2023-04-22T00:00:00Z",
"timestamp": 1682121600000
},
"DOI": "10.1177/1758835920922055",
"type": "retraction",
"source": "publisher",
"label": "Retraction"
},
{
"updated": {
"date-parts": [
[2023,4,22]
],
"date-time": "2023-04-22T00:00:00Z",
"timestamp": 1682121600000
},
"DOI": 10.1177/17588359231172420",
"type": "retraction",
"source": "retraction-watch",
"label": "Retraction",
"record-id": 44124
}
]

The source field states where the retraction came from. Currently, it can have two values: publisher or retraction-watch. Note that the same retraction may be included multiple times from different sources.

Retraction Watch retractions will remain available on Gitlab in csv format and be updated on working days. The record-id refers to the entry in the csv file with further details, such as the reason for retraction.

There is full documentation available for the Crossref REST API and if you are new to REST APIs, see our learning hub to get started which includes a tutorial about accessing retractions.

What can I do with the retractions?

Like the rest of our metadata, the retractions are freely available. If you use or operate a tool that ingests retractions, the new entries will start to be picked up immediately. The Retraction Watch database includes a larger number of retractions than the Crossref database, so you should see an increase in the total.

We have heard from organisations that would like to build new research integrity tools based on this data. We look forward to seeing the benefits brought by wider availability of the Retraction Watch retractions, and how they can provide better context to research outputs.

While Crossref metadata is freely available to reuse without a license, if you make use of the Retraction Watch retraction metadata in a published work, we kindly request that you provide a citation to the source.

If you have questions or comments, please head over to the section of our forum dedicated to integrity of the scholarly record.

Research Integrity Roundtable 2024

Martyn Rittman — Fri, 15 Nov 2024 00:00:00 +0000

For the third year in a row, Crossref hosted a roundtable on research integrity prior to the Frankfurt book fair. This year the event looked at Crossmark, our tool to display retractions and other post-publication updates to readers.

Since the start of 2024, we have been carrying out a consultation on Crossmark, gathering feedback and input from a range of members. The roundtable discussion was a chance to check and refine some of the conclusions we’ve come to, and gather more suggestions on the way forward. As in previous years, we were able to include a range of organisations, which led to lively and interesting discussions. See below for the full participant list.

Crossmark feedback

We started by presenting Crossmark and a summary of the consultation process. There are a number of areas where we have learned more about how the community operates or found that Crossmark needs to adapt. These include:

Implementation: Our members have struggled to implement Crossmark and uptake is low. At the same time, in many organisations the workflows for handling retractions are not well-defined because they are rarely used, if ever. The responsibility for updating Crossref metadata can be unclear and this may be a factor in the low uptake.

Education: There are different levels of understanding about how to handle retractions. Some members are very defensive when asked about retractions, others state they will never make updates to published works. How can we have a constructive conversation where the value of communicating updates appropriately is recognised?

Community engagement: Given the different scales, locations, disciplines, and technologies used by our members, it looks like one size will not fit all when it comes to updates. How can we get continual, representative feedback on new tools and processes?

Metadata assertions: Crossmark allows the deposit of metadata using custom field names, however this metadata seems to have low usefulness and is not highly valued by the community. Should we continue to collect it? Can we make some of the most-used field names part of our standard schema?

Changing the Crossmark UI: Although we didn’t specifically ask about it during the consultation, the look of the Crossref logo often came up, and concern that it is not recognised and not well-used. Can we change the look and behaviour so that it has more impact?

NISO Recommendations

Patrick Hargitt represented the NISO group on Communication of Retractions, Removals, and Expressions of Concern (CREC). The group’s recommendations were published earlier this year and cover how retractions are communicated. CREC arose from an earlier project, IRSRS. A large part of the motivation is that retracted works continued to be cited, with citing authors apparently unaware of the retraction. Patrick presented the CREC recommendations, which cover:

Metadata receipt, display, and distribution,
Which metadata elements to communicate,
How to implement the recommendations,
Discussion of some special cases,
Key stakeholders and their responsibilities.

The two presentations prompted discussion, which was taken into the first of two workshops.

First workshop: Improving collection of retractions and Crossmark

The first workshop looked at proposed changes to Crossmark and how to encourage more members to deposit their retractions, corrections, and other post-publication updates. Several important themes emerged.

First, the question of whose responsibility it should be to provide metadata on retractions and similar updates. Crossref has a responsibility to work with the community to obtain high quality and complete metadata; publishers should take responsibility for handling issues of research integrity and reporting them to relevant downstream services, like Crossref; and platforms need to provide tools that allow easy reporting of retractions.

The value of Crossmark appearing in PDFs was reiterated. The fact that a PDF can be downloaded, and years later there is a way to tell whether it has been retracted or not is highly valued. There was also the suggestion that the Crossmark logo on web pages can indicate a change before it has been clicked. This is something that we have been considering at Crossref and it was useful to have the idea reinforced. Another suggestion was that a browser plugin would make a good complement to Crossmark.

Implementation issues with Crossmark were raised, including that it’s difficult to validate whether a specific implementation is complete. There are a number of different changes (to metadata deposit and content, and websites) that need to work together to have Crossmark fully functional. There were several questions and a discussion about Retraction Watch data. Some were about understanding its collection and validation. A number of participants are actively using the data and it was great to see the variety of applications.

Second workshop: Community use of retraction metadata

The second workshop focused on a broader set of downstream organisations that might want to make use of retraction metadata. We looked at stakeholders and their needs, and attempted to match them up with existing tools. Several gaps were identified as a result, which may provide opportunities for new services or collaborations to fill them.

We identified a number of tools available for publishers, editorial systems, metadata researchers, and readers. A good example is reference managers, many of which are now highlighting retracted works to authors. This can help to reduce the number of retracted works being cited. Publishing platforms are also providing support to editors, using tools that include retraction metadata.

Some of the stakeholders identified have limited tools for identifying retractions that are relevant to them. These include funders, archives and repositories, journalists, and institutions.

Often, there are pathways for retraction data to be communicated but they are not being sufficiently used. There needs to be a concerted effort to improve the quality of retraction metadata for tools to function better. For example, a second author on a paper might not know that a correction or retraction is planned for their article. If their email or ORCID isn’t included in the metadata, an alerting tool wouldn’t be able to let them know. A similar argument can be made for institutions or funders if they are not well-identified in the metadata.

The question of standardisation of metadata was raised. It seems too early to implement a full set of standards at the moment. CREC and similar initiatives have documented and accommodated for a range of practices while providing guidance and principles to work towards. More discussion is needed in the community to work out paths that could be applied across the broad spectrum of scholarly communication.

Conclusion

The event was very valuable in bringing up a range of topics related to retraction and communication of post-publication changes to scholarly works. We are grateful to all of the participants for their contributions and sharing their diverse experience and opinions with us.

Research integrity is an area of flux, with significant changes over the past few years. While there has been progress, there remain gaps in metadata and tools to communicate retractions. This is something that Crossref will continue to contribute to, and Crossmark clearly still has a role to play.

Some of the ideas and suggestions from the discussion can be implemented in the near future. Others need further development, and we will continue to engage the community. Reading this, there may be topics where you feel you have a role to play. We are keen to partner with other organisations in this space as we continue to improve the transparency and communication of metadata for post-publication updates.

Participants

Many thanks to the participants. Here is the full list of those that attended:

Name	Role	Organisation
Aaron Wood	Head, Product & Content Management	American Psychological Association
Adya Misra	Associate Director, Research Integrity	Sage
Bianca Kramer		Sesame Open Science
Constanze Schelhorn	Head of Indexing	MDPI
Guillaume Cabanac	Full Professor	University of Toulouse
Hong Zhou	Director of AI Product	Wiley
Jennifer Wright	Head of Publication Ethics and Research Integrity	Cambridge University Press
Johanssen Obanda	Community Engagement Manager	Crossref
Joris van Rossum	Program Director	STM Solutions
Kathryn Weber-Boer	Data & Analytics	Digital Science
Kornelia Korzec	Director of Community	Crossref
Kruna Vukmirovic	Publisher- Journals	The Institution of Engineering and Technology
Lena Stoll	Product Manager	Crossref
Leslie McIntosh	VP, Research Integrity	Digital Science
Liying Yang	Professor	CAS Library
Luis Montilla	Technical Community Manager	Crossref
Madhura Amdekar	Community Engagement Manager	Crossref
Martyn Rittman	Progam Lead	Crossref
Maryna Kovalyova	Member Experience Manager	Crossref
Mina Roussenova	Project Manager, Strategic Projects	Karger
Osnat Vilenchik	VP Content Operations	Ex Libris, part of Clarivate
Patrick Hargitt	Senior Director of Product Management	Atypon/Wiley
Paul Davis	Tech Support & R&D Analyst	Crossref
Sami Benchekroun	CEO	Morressier
Scott Delman	Director of Publications	Association of Computing Machinery (ACM)
Shilpi Mehra	Head, Research Integrity & Paperpal Preflight	Cactus Communications
Sichao Tong		Chinese Academy of Sciences, Library

Scholarly metadata as trust signals: Opportunities for journal editors

Madhura Amdekar — Mon, 04 Nov 2024 00:00:00 +0000

Scholarly metadata as trust signals: opportunities for journal editors

Published in Science Editor (Vol. 47, No. 4, December 2024), this article reframes scholarly metadata as a practical trust signal in the fight to protect research integrity — and sets out specific, actionable opportunities for journal editors to use Crossref metadata to detect, deter, and respond to integrity threats.

Strategists

Understand why metadata is integrity infrastructure. The presence and richness of metadata — not the DOI itself — is what carries trust signals across the scholarly record.

Decision-makers

Know which metadata to require and check. A practical framework for journal editors making policy decisions on contributor identifiers, affiliations, funding, peer review, and update notices.

Practitioners

See where to act in editorial workflows. Concrete opportunities to use Crossref metadata, Crossmark, and Cited-by to detect paper mills, citation cartels, and other integrity threats.

What this article covers

The integrity landscape — paper mills, citation cartels, fabricated peer reviews, fake papers, and AI-generated images as emerging threats to the scholarly record
Metadata as trust signal — why the DOI alone does not signal quality, and how the presence (or absence) of associated metadata acts as evidence of trustworthiness
Crossref’s role — how open, machine-readable metadata across member deposits enables downstream services and editors to assess research outputs at scale
ROR IDs — how organisation identifiers help connect problematic manuscripts to their institutional context
Crossmark — how update notices, retractions, and corrections strengthen the integrity of the scholarly record after publication
Cited-by — how citation context can surface citation patterns worth attention
A call to journal editors — concrete opportunities to enrich the metadata you deposit and to draw on the metadata others have deposited

Read the article

Read on Science Editor

Integrity of the Scholarly Record (ISR): what do research institutions think?

Madhura Amdekar — Thu, 09 May 2024 00:00:00 +0000

Earlier this year, we reported on the roundtable discussion event that we had organised in Frankfurt on the heels of the Frankfurt Book Fair 2023. This event was the second in the series of roundtable events that we are holding with our community to hear from you how we can all work together to preserve the integrity of the scholarly record - you can read more about insights from these events and about ISR in this series of blogs.

Research institutions are one of the most important stakeholders in the endeavour of research integrity, and any conversation around ISR is incomplete without the views of this key community. This fact was acknowledged at the second ISR roundtable event, and one of the main takeaways from the discussions was to make more focused efforts to hear the viewpoints of researchers and academics.

As the first step in this direction, we organised an online discussion on the integrity of the scholarly record, to which we invited: researchers and academics, research integrity experts based at academic institutions, Crossref members, as well as other organisations working on this topic such as COPE and Digital Science. The primary objective of this event was to hear from this community their perspectives on preserving and leveraging the integrity of the scholarly record and to identify opportunities for collaboration in this area. To ensure common ground, we also wanted to share information about Crossref metadata, the Research Nexus vision, and our position and role in the integrity of the scholarly record.

To facilitate this, the event started with an introduction by Kora Korzec, Head of Community Engagement and Communication at Crossref, to our mission and vision and the importance of capturing the relationships between the objects, people and places involved in research through the Research Nexus. Amanda Bartell, Head of Member Experience, was next and she spoke about the scholarly record and the role that Crossref plays in preserving the record’s integrity. In her presentation, Amanda emphasised that Crossref’s role is not to assess the quality of content deposited by the members but rather to provide infrastructure that enables the community to provide and use metadata about the scholarly content produced by members. It’s important not to put up barriers to entry, but to work with all publishers to encourage best practices.

Dominika Tkaczyk, Head of Strategic Initiatives, shared details of a few Crossref projects that focus on monitoring and improving metadata completeness, thereby supporting ISR. These projects include improving the Participation Reports, using metadata matching to discover new relationships (e.g., preprint published as work, work supported by funder, etc), and importing more retractions and other updates from the Retraction Watch database that was acquired and made openly available by Crossref. Dominika used these examples to highlight the ways in which open and complete metadata can help in uncovering large scale trends and systemic concerns. The final speaker was Amanda French, ROR Technical Community Manager, who introduced the audience to the Research Organization Registry, or ROR.

To accomplish the primary aim of the event, which was to hear the community’s viewpoints, the participants were divided into breakout groups for discussions and given three prompts to answer. The rest of the blog is a summary of what we heard from the participants.

1. Is Crossref’s role what you expected? What surprised you? What are we missing?

An overarching sentiment from the academics in the audience was that Crossref does so much more than is known to researchers! They were surprised by the range of activities underway at Crossref. At the same time, there were calls for Crossref to play a bigger role. Suggestions included playing a leadership role in deciding which metadata elements are a priority, providing guidance on the main metadata components important for signalling trust, playing a greater role in connecting various identifiers to ensure that relationships between different content types are preserved well, and to coordinate the efforts being taken by institutions, publishers and service providers around research integrity, by virtue of Crossref’s unique position in the community. There was a broad agreement that by providing the essential infrastructure, Crossref acts as the base upon which other actors in the scholarly community can build.

2. What metadata elements do you consider important for signalling trust?

Many participants spoke about the various ways in which author identity and affiliation are important as trust signals. Being able to identify when an author has changed institutions, or being able to make a distinction between authors who have the same name is important. Author affiliations that are authentic and verified would go a long way in establishing trust.

Multiple assertions, e.g. for affiliations, would be welcome. The use cases for this could be when research starts at one institution and is carried over to another, or when researchers affiliated with an institution may perform part of the research overseas. Some of the participants, who actively investigate research data, shared that abstracts are valuable because they can be used for large scale analyses related to research integrity.

Other metadata elements that came up during this discussion were data on peer review, ethics approval, patient and donor consent in medical research, editorial boards (especially of special issues), pre-registration, funding metadata, datasets and programming scripts.

3. What value do you see in the integrity and completeness of the scholarly record in the way you operate? How do you contribute to it? How can it support you to achieve your own goals?

Participants acknowledged that integrity of the metadata and the scholarly record is essential. Ensuring this integrity is a dynamic process, much akin to the concept of organised scepticism which is the notion that all scientific work should be trusted subject to its verification. Several ideas were shared on how to progress the integrity and completeness of the scholarly record. One recommendation was to use multiple metadata trust markers as that can make it harder for bad actors to game the system, but this may run the risk of making things complicated. Another suggestion was to make metadata part of the onboarding procedure- by gathering staff ORCID iDs during the onboarding process and sharing the institutional ROR ID with staff to promote its use, institutions can ensure that this information is routinely made available. The metadata deposited with Crossref should be integrated with downstream workflows to better facilitate the use of this rich metadata. An example of this is to integrate Crossmark with other research tools such as reference management software.

The participants acknowledged that this discussion underlined for them the fact that having identifiers in itself is not an indicator of quality and that the underlying metadata records and wider context is key to understanding trustworthiness of the content.

This event was a good first step towards engaging researchers and academics in the conversation about ISR. It connected folks working in different parts of the world who are united by their interest in research integrity. There was good engagement among all and commitment to continue these conversations in the future, with many participants planning to connect at the World Conference on Research Integrity in June (I’ll be attending as well, for anyone who wants to continue the conversation - along with my colleagues Fabienne and Evans).

At Crossref, we plan on continuing these conversations with all segments of the community to understand their needs and perceptions around metadata. The greater the awareness about the importance of metadata and its applications, including for research integrity, the richer the metadata that we are able to collect together. This will lead to building a comprehensive Research Nexus and emergence of more relationships therein. Please write in response to this post on our Community Forum if you have any thoughts on this as we’d love to hear from you.

List of participants


Manu Goyal	International Journal of Cancer
Panagiotis Kavouras	University of Oslo
Dorothy Bishop	University of Oxford
Zhesi (Phil) Shen	Centre of Scientometrics, National Science Library, Chinese Academy of Sciences
Wouter Vandevelde	KU Leuven
Leslie McIntosh	Digital Science
Elizabeth Noonan	University College Cork
Radek Gomola*	Masaryk University Press
	Queensland University of Technology
	London School of Hygiene & Tropical Medicine
	Vilnius Gediminas Technical University Library
	Committee on Publication Ethics (COPE)
Ginny Hendricks	Chif Program Officer, Crossref
Kornelia Korzec	Director of Community, Crossref
Amanda Bartell	Director of Membership, Crossref
Dominika Tkaczyk	Director of Data Science, Crossref
Amanda French	Technical Community Manager, Crossref
Madhura Amdekar	Community Engagement Manager, Crossref

*Note: name added 21-May-2024

ISR Roundtable 2023: The future of preserving the integrity of the scholarly record together

Madhura Amdekar — Tue, 06 Feb 2024 00:00:00 +0000

Metadata about research objects and the relationships between them form the basis of the scholarly record: rich metadata has the potential to provide a richer context for scholarly output, and in particular, can provide trust signals to indicate integrity. Information on who authored a research work, who funded it, which other research works it cites, and whether it was updated, can act as signals of trustworthiness. Crossref provides foundational infrastructure to connect and preserve these records, but the creation of these records is an ongoing and complex community effort. Crossref has always shown a deep commitment to preserving the integrity of the scholarly record in an open and scalable manner.

Given the increasing concerns in the community about matters of research integrity and integrity of the scholarly record (ISR), we at Crossref have been engaging with community members to understand what developments are needed. In 2022, we organised a roundtable discussion to talk about our role and the applicability of Crossref’s services in preserving and assessing the integrity of the scholarly record. We’ve acted on much of that feedback since, and so in October 2023, we organised a follow-up event, once more gathering representatives of publishers, research integrity experts, policy-makers, academic institutions, funders, and researchers (the full list of participants can be found in the appendix). This post aims to offer insight into the discussions at this event and the next steps. The objective of this event was to take the conversation forward by:

Sharing the progress made by Crossref on matters related to ISR since the last roundtable event.
Sharing information about how metadata contributes to the Research Nexus, and can act as trust markers for research outputs.
Apprising the community about the latest membership trends and examples of activities that we see, such as title transfer disputes, unregistered DOIs, requests for deleting records, and sneaked references .
Building upon the ideas discussed during the 2022 roundtable event to progress the conversation about issues related to ISR.
Learning from the participants about their experiences of pursuing research integrity initiatives.
Last but no less importantly, hearing from the participants their perspectives on strategies for preserving the integrity of the scholarly record, and opportunities for collaborating to leverage metadata to assess the integrity of the scholarly record.

The event was kicked off by Ed Pentz, who spoke to the participants about how integrity is key to Crossref’s mission, and Crossref’s vision of the Research Nexus. Next, Amanda Bartell, the Head of Member Experience at Crossref, shared the recent developments and trends in community behaviour. She expanded upon the actions taken by Crossref as part of its ISR program since the last roundtable event, which include:

Acquisition and opening of the Retraction Watch database, which makes it easier to access information on retractions and corrections.
Increased participation in the Global Equitable Membership (GEM) program, enabling a wider section of the community to provide and access trust signals.
Newer developments around metadata that act as trust signals: e.g. 120K grants or awards now have a Crossref DOI, and the planned transition of the Open Funder Registry into ROR.
Recruitment of a Community Manager to focus on working with publishers and editors, including on ISR (that’s me!), and recruitment of a Technical Community Manager to enable greater use of our APIs.

Amanda highlighted that all Crossref members should be using ROR IDs to provide affiliations for authors (along with ORCID iDs) in their Crossref metadata. She also shared some latest examples of community behaviours that we have seen, such as requests from authors to delete records of works that were published without their permission, title ownership disputes between publishers, and the recent instance of sneaked references.

Ivan Oransky, co-founder of Retraction Watch, and Lena Stoll, Product Manager at Crossref, were next, and they spoke about the future of the Retraction Watch database, and about the Crossmark service. After this, some of the other roundtable participants shared initiatives that they have undertaken that support ISR:

Jodi Schneider from the University of Illinois Urbana-Champaign spoke about NISO’s CREC Working Group that has created a Recommended Practice that should be followed by relevant stakeholders for communicating retracted research (Crossref’s Director of Product Rachael Lammey was the co-chair of that group).
Kihong Kim from the Korean Council of Science Editors shared information about the workshops that the Council has organised for researchers on publishing in journals.
Alberto Martín-Martín from Universidad de Granada presented his thoughts on how to reconcile the publishing system and the institutional view of tracking research outputs.
Bianca Kramer from Sesame Science spoke about her analysis of and the implications of sneaked references, duplicate references, and missing references for citation integrity.
Joris van Rossum from STM Solutions spoke about the STM Integrity Hub and the integrity tools that are being developed in collaboration with some publishers.

Some of the most valuable reflections stemmed from discussions in small groups on these three key questions:

What value do you see in the integrity and completeness of the scholarly record in the way you operate? How do you contribute to it? How can it support you to achieve your own goals?
Are you aware of Crossref services? What are the barriers to more uptake? What are the challenges and opportunities?
What information is essential and nice to have for you in the scholarly records to support trust signalling and ascertaining trustworthiness?

As groups shared their discussions, a few themes became apparent that I would like to elaborate on further.

What is “complete”?

Given the prompt to talk about the value of completeness of the scholarly record, an immediate reaction at most tables was: how much metadata qualifies as “complete” metadata? Can the scholarly record be considered complete if some publishers or journals do not use Crossref? What is the optimum level of metadata that should be deposited by members - should a minimum data standard be defined by disciplines, or should there be standard data requirements for all? The composition of metadata appears to change over time, too, as the processes change and our ability to record their facets increases. While there were spirited discussions about what constitutes a complete scholarly record, everyone agreed that “completeness” of metadata, as much as is possible, should be the aim. Unambiguous and consistent standards may help with this, for example, the Metadata 20/20 community creation of principles and best practices, and potentially also using a set of recognition standards and reproducibility badges.

Global participation is equally important for a truly “complete” scholarly record. In order to enable as many in the scholarly community as possible to participate in Crossref services and metadata, Crossref launched the Global Equitable Membership (GEM) program in 2023. Under this initiative, membership and content registration fees are waived off for members from the least economically advantaged countries. We are seeing first signs that this initiative meaningfully lowers the barriers to participation for organisations based in those countries, and allows the global community to contribute towards the building of a comprehensive research ecosystem.

At the end of the day, it is important to recognize that rich metadata is crucial because it can be used for all kinds of analysis, which in turn can drive decision-making. Even if some of the metadata components are sporadically missing, that could be acceptable, because every piece of data counts!

Corrections and Retractions

Similar to last year, retractions and corrections continued to be a topic of great interest in this year’s roundtable. This was not surprising given their relevance as trust indicators as well as the recent development with the acquisition of the Retraction Watch database by Crossref. Having heard from Ivan about the Retraction Watch taxonomy of reasons for retractions and the metadata included in the database, participants expressed the need to investigate this taxonomy as a community standard. While the Retraction Watch taxonomy is not widely known, we at Crossref are working to map the Crossmark taxonomy with the Retraction Watch taxonomy, which will enable complete integration of the Retraction Watch database with the Crossref database.

It would also be useful to add more information to retraction notices. Having more information about the reasons for retraction will not only destigmatize retractions, but certain additional information, such as submission dates for those outputs, might help with ethical investigations to determine whether manuscripts were being submitted to multiple publishers simultaneously.

On the topic of retractions, another aspect that came up in the room was about incentives for researchers to publish as much and as quickly as possible. If researchers indulge in unethical publishing practices due to this pressure to publish, that is hugely detrimental to the cause of research integrity and to the progress of scientific research in general. However, there is a distinction to be made between the integrity of the research and the integrity of the scholarly record - unethical research and publishing practices, including but not limited to data falsification, fabrication, and plagiarism, affect research integrity while integrity of the scholarly record is affected by unavailability of metadata, outdated metadata, incomplete metadata records, and incorrect metadata (e.g. as seen in the case of sneaked references).

There was a lot of discussion about Crossmark, a cross-platform service provided by Crossref that allows readers to discover whether an item has been updated, corrected, or retracted just by clicking a button that is standardised across publication platforms. While most participants acknowledged its importance, they also pointed out that its uptake has been limited and publishers do not use it as much, perhaps because it is difficult to implement and there’s a matter of providing more clarity about it to the readers. There were suggestions to add a notification system to Crossmark such that every time a published output is retracted, a notification goes out. This seemed of particular interest to funders, whose grievance was that they are usually the last to find out when research that they have funded is retracted. They would welcome notifications that would alert them to such events.

We already have plans to consult with the community more specifically about what changes they’d like to see to Crossmark that will enable them to implement it easily and use it more frequently. Take a look at this thread on our community forum and add your thoughts for our next steps on Crossmark.

The importance of education

There was an overwhelming sentiment that there was a need for collective arbitration of research integrity issues. However, everyone recognized that this is not a role for Crossref. We can act as a “trust broker” by bridging different metadata and identifiers that otherwise might not interact, creating a network of research outputs whose credibility can be verified by others. Many participants called for Crossref to increase its efforts in educating community members about the importance of metadata and how different pieces can be linked together to make meaningful connections.

Research practices vary between countries, and between institutions. Correspondingly, the metadata being provided by diverse Crossref members may also vary. There is an opportunity here for the global research community to work together to increase awareness about ethical standards, so that a lack of specific metadata or its variances (e.g. unusually formatted metadata, or non-standard metadata fields) may not be construed as “lower quality” metadata. Many felt that the greatest need for education about metadata is for the academic community – although individual researchers contribute a wealth of metadata associated with any published research output, they do not necessarily understand how metadata contributes to the completeness of the scholarly record. There is a further opportunity to talk to the academic community about how different metadata components link together to form a rich network, supporting visibility and confidence in their work. A greater awareness about these topics is likely to encourage researchers to provide more metadata and identifiers.

While most participants at the roundtable event agreed about the need for this conversation and the educational opportunities here, if Crossref were to lead these efforts, it would represent, in some eyes, a diversion from its mission. We do have several initiatives already to support our communities. As part of the Crossref Ambassadors program, volunteers from the international scholarly community who believe in Crossref’s mission liaise with our team to conduct training in their communities about using Crossref services and, generally, about the importance of metadata. In 2023, we also launched a new online public forum, the PLACE, in collaboration with the Committee on Publication Ethics (COPE), the Directory of Open Access Journals (DOAJ), and the Open Access Scholarly Publishers Association (OASPA). This forum is a place where new publishers can connect with these organisations and learn about best practices in scholarly publishing via discussion posts and by asking questions, as they get started. Another initiative that is designed to help new Crossref members is the “Managed Member Journey”: as members join and move through the various stages of membership, key information is shared with them during each of these stages in the form of triggered automated emails, web pages, and webinars.

While Crossref’s direct interactions with researchers are limited, we welcome the community’s recognition of the need to raise awareness about these matters. We have started engaging more closely with the reporters of metadata issues, in many cases investigators and ‘sleuths’ in the area of research integrity, and plan some closer collaborations with this group in 2024. We are open to supporting community efforts to inform other stakeholders about the importance and uses of metadata.

Incentives for the community

Another theme that was heard repeatedly was “incentives”: incentives for researchers to contribute to a “complete” scholarly record, incentives for publishers to improve metadata, and incentives for everyone to report on and register retractions.

As I mentioned before, a shared sentiment is that researchers may not be aware of the value of rich metadata. While more publications, increased citations, and greater grant funding are some examples of incentives that are part of the current academic settings, the right incentives probably do not exist for researchers to provide complete metadata. With the diverse set of participants present at this meeting, some groups also discussed how the current research assessment system can change to incorporate other metrics, perhaps those based on open science and open data.

What could be the incentives for publishers to improve the metadata collected and deposited by them? One suggestion was that clearly defined benefits of rich metadata can incentivise publishers. Being aware of what funders are mandating, can be another incentive. On the same note, funders will benefit from knowing what metadata is being provided by publishers. This metadata is available through our open API, and nine key checks on members’ activity are available through our public Participation Reports.

Retractions featured again in the discussion on the topic of incentives. As shared by Ivan, retractions are on the rise every year, with about 43k retractions currently in the Retraction Watch database. On the other hand, retractions registered in Crossmark at the time of the meeting numbered just 14k and have recently jumped up to 25k thanks to Hindawi/Wiley’s dedication to good open metadata. Besides the fact that the uptake of Crossmark by Crossref members is limited, another reason for the low number of retractions being registered is the associated stigma. Corrections and errata are usually conflated with retractions, and all these terms, which represent different kinds of updates that may happen to a published item, have a stigma associated with them in the academic community. There is a need to destigmatize retractions, and perhaps incentivize them by noting that these updates are essential to uphold the integrity of the scholarly record and to highlight the publishers that are showing leadership in addressing the issues openly through up-to-date Crossref metadata.

What metadata is nice to have in the scholarly record?

We asked everyone what information they think is essential as well as “nice to have” in the scholarly record to support trust signalling, and we heard a range of answers. Peer review information was recognized to be important. This would include data on who the peer reviewers were and standard peer review terminology that has been published by NISO. More generally, as much metadata as possible about the main actors of the peer review process was considered important - such as designating who the corresponding author is, and who the handling editor or the decision-making editor was.

As special issues led by guest editors in journals have been brought to the attention of late due to the uncovering of irregularities in some of them, one of the first suggestions in this context was more metadata about special issues. Participants thought that it would be useful to collect and distribute information on handling/guest editors of special issues, peer reviewers, as well as submission and acceptance dates. Recently, COPE has released guidance on “best practices for guest-edited collections” , highlighting that this topic looms at the forefront for the scholarly information industry.

Adding information on ethical approvals provided by institutional review boards would add more nuance to the research outputs. Metadata about clinical trials helps to add transparency to research in a field, where reproducibility is of primary importance. Conflicts of interest are another factor that could be a cause of concern if not reported accurately; these declarations were mentioned by the participants as important for signalling trust.

Recognizing that it is the relationships in the metadata that add context to research output, participants echoed that better interlinking between preprints and their published versions is required. To aid with all of this, it has been suggested that a complete list of all metadata that can be deposited with Crossref be made available in a simple format, so that members have more visibility about all the possibilities that exist for providing metadata.

Next steps

We asked all participants if the discussions prompted them to plan to take any actions in the near future. Several attendees reflected that the discussion encouraged them to go back and review the metadata that they are depositing with Crossref, and how they can make more use of the data openly available from Crossref. We also heard how some found training opportunities therein - discussion points from the event could be included in workshops for affiliated researchers, and in COPE guidance for members. As encouraged by members of the NISO’s CREC Working Group, some participants were looking to respond in the (then open) consultations of the draft Recommended Practice, NISO RP-45-202X, Communication of Retractions, Removals, and Expressions of Concern (CREC). One message resonated loud and clear: preserving the integrity of the scholarly record cannot be a lone endeavour and has to be a community effort. Attendees expressed their commitment to continue these conversations, with the next most opportune time being at the STM week. Everyone recognised that collaboration in this space is the need of the hour: facilitating information and data sharing across all the players in the ecosystem would be crucial to progressing this topic. As Bianca Kramer declared during her presentation, “I am committed to using only open data in my research, as access to data is important for the community to detect problems at scale”.

At our end, we are looking to act on suggestions that are specific to Crossref:

Consultation with the community about Crossmark

One of the first things that we are doing in early 2024 is to consult with our community about the developments needed in the Crossmark service. Our key aim with this exercise would be to understand how we can enable a more effective uptake of this service so that Crossref members can easily fulfil their obligation of keeping their records updated. We are keen to understand what we can do to help our members to send us metadata about updates to an output, and how we can help downstream services that use this data. Insights from this consultation will also help inform how the Retraction Watch data can be most effectively integrated into Crossmark and communicated to users. Please visit the discussion and add your thoughts here: https://community-crossref-org.pluma.sjfc.edu/t/communicating-post-publication-updates-inviting-feedback-on-the-next-steps-for-crossmark/.

Development of resources for using our API

As there is clearly no dearth of metadata components that the community thinks would be “nice to have” for signalling trust, it is equally important to equip users and downstream service providers to be able to access the rich metadata that is available with Crossref. This rich metadata opens up new avenues for the development of services and resources that can benefit the scholarly community. On account of this, we plan to prioritise development of resources for using Crossref APIs. These efforts would include making available workbooks with a variety of API use cases - ranging from how to use basic API queries, to how to use APIs for obtaining grant information or for obtaining citation data and so on, as well as retrieving corrections, retractions, and update information, especially when the Retraction Watch dataset merges in with the rest of the Crossref metadata.

Working group to facilitate community efforts for preserving ISR

We are looking to set up a working group that will facilitate the various stakeholders in the scholarly ecosystem to work together towards preserving the integrity of the scholarly record. One direction for the group could be to consider the role and impact of Crossref metadata in ISR. Another area of focus will be to enrich information about retractions, corrections, and expressions of concern. Raising industry-wide awareness about the current concerns in upholding the integrity of the scholarly record, and how comprehensive metadata can act as markers of trust about research output, would be another focal point.

Continued community outreach

We will continue our efforts to engage with the community on the very important issues surrounding ISR. We are particularly keen to redouble our efforts to include more funders and institutions in these conversations. Preserving the integrity of the scholarly record needs to be a truly inclusive effort and will benefit from diverse voices in the community. With that in mind, consulting with the community in Asia is next on our radar.

We look forward to working with the community further on this important topic - if you are keen to participate in these discussions and want to contribute towards preserving the integrity of the scholarly record, we would love to hear from you. Please write to us at feedback@crossref.org if you have any suggestions on this topic.

Appendix: Participant list

Name	Role	organisation
Ed Pentz	Executive Director	Crossref
Amanda Bartell	Head of Member Experience	Crossref
Madhura Amdekar	Community Engagement Manager	Crossref
Luis Montilla	Technical Community Manager	Crossref
Lena Stoll	Product Manager	Crossref
Kora Korzec	Head of Community Engagement and Communications	Crossref
Ivan Oransky	Co-Founder	Retraction Watch
Jennifer Wright	Research Integrity Manager	Cambridge University Press
Guntram Bauer	Director of Science Policy & Communications	Human Frontier Science Program
Wendy Patterson	Scientific Director	Beilstein-Institut
Sarah Jenkins	Director, Research Integrity & Publishing Ethics	Elsevier
Helene Stewart	Director, Editorial Relations Web of Science	Clarivate
Bianca Kramer	Advisor, Research Analyst, Facilitator	Sesame Open Science
Adya Misra	Research Integrity and Inclusion Manager	Sage
Andrew Joseph		Wits University Press
Theodora Bloom	Executive Editor	BMJ
Alberto Martín-Martín	Assistant Professor	Universidad de Granada
Aaron Wood	Head, Product & Content Management	American Psychological Association
Fred Atherden	Head of Production Operations	eLife
Kihong Kim		Korean Council of Science Editors
David Flanagan	Senior Director, Data Science	Wiley
Chiara Di Giambattista	Communications Director	OpenCitations
Scott Delman	Director of Publications	ACM
Chi Wai (Rick) Lee	General Manager	World Scientific Publishing Co (WSPC)
Leslie McIntosh	VP, Research Integrity	Digital Science
Adam Day	Director	Clear Skies
Damaris Critchlow	Project Manager	Karger
Tamara Welschot	Head of Research Integrity, Prevention	Springer Nature
Kathryn Dally	Research Integrity and Policy Lead	Research Services, University of Oxford
Masahiko Hayashi	Director, JSPS Bonn Office	Japan Society for the Promotion of Science
Simone Taylor	Chief, Publishing	American Psychiatric Association
Christna Chap	Head of Editorial Development	Karger Publishers
Coromoto Power Febres	Research Integrity Manager	Emerald Publishing
Carole Chapin	Project Manager	French Office for Research Integrity
Jodi Schneider	Associate Professor of Information Sciences	University of Illinois Urbana Champaign
Oliver Koepler	Head of Lab Linked Scientific Knowledge	TIB - Leibniz Information Centre for Science and Technology
Heather Staines		Delta Think
Eri Anno		JSPS Bonn office
Joris van Rossum		STM Solutions
Anita de Waard	VP Research Collaborations	Elsevier

News: Crossref and Retraction Watch

Ginny Hendricks — Tue, 12 Sep 2023 00:00:00 +0000

Crossref acquires Retraction Watch data and opens it for the scientific community

Agreement to combine and publicly distribute data about tens of thousands of retracted research papers, and grow the service together

12th September 2023 —– The Center for Scientific Integrity, the organisation behind the Retraction Watch blog and database, and Crossref, the global infrastructure underpinning research communications, both not-for-profits, announced today that the Retraction Watch database has been acquired by Crossref and made a public resource. An agreement between the two organisations will allow Retraction Watch to keep the data populated on an ongoing basis and always open, alongside publishers registering their retraction notices directly with Crossref.

Both organisations have a shared mission to make it easier to assess the trustworthiness of scholarly outputs. Retractions are an important part of science and scholarship regulating themselves and are a sign that academic publishing is doing its job. But there are more journals and papers than ever, so identifying and tracking retracted papers has become much harder for publishers and readers. That, in turn, makes it difficult for readers and authors to know whether they are reading or citing work that has been retracted. Combining efforts to create the largest single open-source database of retractions reduces duplication, making it more efficient, transparent, and accessible for all.

Product Director Rachael Lammey says, “Crossref is focused on documenting and clarifying the scholarly record in an open and scalable form. For a decade, our members have been recording corrections and retractions through our infrastructure, and incorporating the Crossmark button to alert readers. Collaborating with Retraction Watch augments publisher efforts by filling in critical gaps in our coverage, helps the downstream services that rely on high-quality, open data about retractions, and ultimately directly benefits the research community.”

The Center for Scientific Integrity and the Retraction Watch blog will remain separate from Crossref and will continue their journalistic work investigating retractions and related issues; the agreement with Crossref is confined to the database only and Crossref itself remains a neutral facilitator in efforts to assess the quality of scientific works. Both organisations consider publishers to be the primary stewards of the scholarly record and they are encouraged to continue to add retractions to their Crossref metadata as a priority.

“Retraction Watch has always worked to make our highly comprehensive and accurate retraction data available to as many people as possible. We are deeply grateful to the foundations, individuals, and members of the publishing services industry who have supported our efforts and laid the groundwork for this development,” said Ivan Oransky, executive director of the Center for Scientific Integrity and co-founder of Retraction Watch. “This agreement means that the Retraction Watch Database has sustainable funding to allow its work to continue and improve.”

Please join Crossref and Retraction Watch leadership, among other special guests, for a community call on 27th September at 1 p.m. UTC to discuss this new development in the pursuit of research integrity.

Supporting details

Crossref retractions number 14k, and the Retraction Watch database currently numbers 43k. There is some overlap, making a total of around 50k retractions.
The full dataset has been released through Crossref’s Labs API, initially as a .csv file to download directly: https://api-labs-crossref-org.pluma.sjfc.edu/data/retractionwatch?name@email.org (add your ‘mailto’). Edit: 2024-10-10: The full dataset is available in a git repository at https://gitlab.com/crossref/retraction-watch-data.
The Crossref Labs API also displays information about retractions in the /works/ route when metadata is available, such as https://api-labs-crossref-org.pluma.sjfc.edu/works/10.2147/CMAR.S324920?name@email.org (add your ‘mailto’). If you don’t have a .json viewer, please see below for screenshot.
Crossref is paying an initial acquisition fee of USD $175,000 and will pay Retraction Watch USD $120,000 each year, increasing by 5% each year.
The initial term of the contract is five years. ~~The full text of the contract will be made public in the coming fortnight.~~ EDIT 2023-09-26: Here is the signed agreement.
There will be a community call on 27th September at 1 p.m. UTC (your time zone here). Please register.
An open FAQ document is available to collect questions to be answered at the webinar.
This announcement will always be accessible via Crossref DOI https://doi-org.pluma.sjfc.edu/10.13003/c23rw1d9; please use this persistent link for sharing.

About Retraction Watch and The Center for Scientific Integrity

The Center for Scientific Integrity is a U.S. 501(c)3 non-profit whose mission is to promote transparency and integrity in science and scientific publishing, and to disseminate best practices and increase efficiency in science. In addition to maintaining and curating the Retraction Watch Database, the Center is the home of Retraction Watch, a blog founded in 2010 that reports on scholarly retractions and related issues in research integrity.

About Crossref

Crossref is a global community infrastructure that makes all kinds of research objects easy to find, assess, and reuse through a number of services critical to research communications, including an open metadata API that sees over 2 billion queries every month. Crossref’s >19,000 members come from 151 countries and are predominantly university-based. Their ~150 million DOI records contribute to the collective vision of a rich and reusable open network of relationships connecting research organisations, people, things, and actions; a scholarly record that the global community can build on forever, for the benefit of society.

Enquiries

For Retraction Watch/Center for Scientific Integrity: Ivan Oransky, ivan@retractionwatch.com
For Crossref: Ginny Hendricks, ginny@crossref.org

A screenshot of an example Labs API metadata record with a Retraction Watch-asserted retraction

ISR part four: Working together as a community to preserve the integrity of the scholarly record

Amanda Bartell — Wed, 26 Apr 2023 00:00:00 +0000

We’ve been spending some time speaking to the community about our role in research integrity, and particularly the integrity of the scholarly record. In this blog, we’ll be sharing what we’ve discovered, and what we’ve been up to in this area.

We’ve discussed in our previous posts in the “Integrity of the Scholarly Record (ISR)” series that the infrastructure Crossref builds and operates (together with our partners and integrators) captures and preserves the scholarly record, making it openly available for humans and machines through metadata and relationships about all research activity. This Research Nexus makes it easier and faster for everyone involved in research performance, management, and communications to understand information in context and make decisions about the trustworthiness of organisations and their published research outputs. Conversely, it can make it harder for parties to pass off information as trustworthy when the information doesn’t include that context.

The community needs open scholarly infrastructure that can adapt to the changes in scholarly research and communications, and we’ve been changing and adapting already by building on the concept of the scholarly record with our vision:

Like others, we envision a rich and reusable open network of relationships connecting research organisations, people, things, and actions; a scholarly record that the global community can build on forever, for the benefit of society.

We don’t assess the quality of the work that our members register, and we keep the barriers to membership deliberately low to ensure that we are capturing as much of the scholarly record as possible and encouraging best practice. We are careful to talk about Crossref’s specific role being with the Integrity of the Scholarly Record (ISR), and not the broader area of ‘research integrity’ (i.e. the integrity of the research process or content itself).

But there are many challenges and threats to research integrity and the integrity of the scholarly record, and there are tradeoffs with keeping the barriers to membership low. With that in mind, we have been dedicating more time to speaking with the community to explore what part we are and should in future play to help the community assess and improve trustworthiness in the scholarly record. We also want to work out where we can make use of our neutral, central role to convene different groups in scholarly communications to work together on these challenges.

A revealing afternoon in Frankfurt

Our starting point was a roundtable discussion in Frankfurt in October 2022. We organized it to coincide with the Frankfurt Book Fair, but the invited participants were from a wider spectrum than just publishers. The 40 invited participants represented editors, funders, research integrity professionals at publishers, representatives of ministries of science, and other partner organisations such as OASPA, COPE, STEM and DOAJ.

This half-day session enabled us to sense-check our thinking with the community and get input into whether our position is the best one for their needs.

Ed Pentz introduced the session by reminding participants that integrity is key to Crossref’s mission and is the basis of the shared Research Nexus vision. Amanda (that’s me) talked through our current membership processes, recent membership trends, and why wider participation is key and also the sort of questions the community comes to Crossref to solve (eg title ownership disputes). And finally, Ginny Hendricks talked through the specific services and metadata that Crossref has already developed to support the community as signals of trustworthiness, and introduced some new activities and ideas.

You can check out the slide deck and for more background, read our previous posts in the ISR series.

Participants then split into small groups representing a mix of communities, and we asked them to discuss three key questions:

Is Crossref’s role what you expected? What surprised you? What are we missing?
Are you aware of Crossref services? What are the barriers to more uptake? What are the challenges and opportunities?
What more could Crossref or its members do?

After discussion, each small group fed back to the room, and we followed up with a whole group discussion, before ending the day with a post-it note exercise for what Crossref should start doing, stop doing, and continue doing.

Here’s what we learned.

The importance of whole community involvement in research integrity and ISR

The need for all parts of the community to come together to solve the problems of research integrity came through loud and clear - there is no single group that can solve this problem on its own.

Publishers expressed frustration that responsibility for research integrity has been placed seemingly solely in their hands when institutions and funders can “unwittingly incentivise bad behaviour”. But it was clear that funders are just as concerned with research integrity issues, with many having made a dedicated trip for the roundtable. There were comments that bringing publishers and funders together around these issues was a rare but important opportunity, and there were calls for this to be an annual event. Both funders and publishers called for more involvement from and inclusion of research institutions in the discussion.

The group agreed that Crossref’s main focus should continue to be capturing and sharing the scholarly record, and that metadata and relationships are key for attribution, evidence, and provenance. One participant commented that “you can’t make open science work unless the metadata is complete” and that this would only happen with efforts throughout the community. Accurate and complete metadata needs to be:

pushed for by funders and institutions (through advocacy and policy)
provided by the authors and other contributors
collated, curated, and registered by the publishers and repositories
collected, matched, (sometimes cleansed), and distributed by Crossref.
(and we would add “prioritised by all who want to support open infrastructure over commercial alternatives”)

Interestingly, this echoes the ‘metadata personas’ output of the Metadata 20/20 initiative which defined roles in the community’s collective metadata effort:

Metadata Creators: providing descriptive information (metadata) about research and scholarly objects.
Metadata Curators: classifying, normalising, and standardising this descriptive information to increase its value as a resource.
Metadata Custodians: storing and maintaining this descriptive information and making it available for consumers.
Metadata Consumers: knowingly or unknowingly using the descriptive information to find, discover, connect, cite, and assess research objects.

Importance of whole-publisher involvement

A few participants, particularly those in editorial or integrity roles at publishing organisations, had not previously made the connection that metadata could be important signals of integrity. This highlighted a key problem - working with Crossref is seen by publishers as a technical/production workflow issue, and so knowledge of the benefits of metadata can be siloed within those teams. Crossref needs to reach out to editorial and research integrity teams to explain that good metadata isn’t just an end in itself and reinforce the impact it has on research integrity. This buy-in from across publisher organisations is vital.

We’re currently recruiting a Community Engagement Manager with editorial or research integrity experience to dedicate time to this area, to advocate for richer metadata within the editorial community, and progress this important conversation.

Agreement of the importance of metadata but an acknowledgment that this brings extra cost

Most participants agreed that rich metadata and relationships provide a core tool in establishing and protecting integrity. But they also acknowledged that collecting and registering more metadata often comes with an extra cost - whether that’s from system changes or just extra staff time. This is particularly true where publishers are working with third-party platforms and suppliers where there may be additional costs for adding fields and functionality to collect more metadata and register it with Crossref. Where knowledge of metadata is siloed in technical and production teams, and the wider benefits aren’t acknowledged, it can be hard to get internal buy-in for these extra costs and efforts.

The Frankfurt group also pointed out that the benefits of more comprehensive metadata (and what this means for ISR) are spread across the research ecosystem, but it is the publisher that usually bears the costs.

Need to define which metadata elements are trust signals and make it easier for the community to provide and access them

Through the course of the discussion, various elements were determined to be important to capture as “trust signals” and to identify relationships such as for retractions, conferences, reviewers, data, and when Crossref membership has been revoked for cause. We need to spend time identifying and prioritising these so that our members can do the same.

We need to make it easier for smaller, less technically-resourced members to provide this metadata, both through our tools and our documentation, as “doing this work can be very geeky and the documentation isn’t easy to understand as a layperson”.

There was also a discussion about where the metadata comes from - should community members be able to contribute metadata and assertions to other members’ records? If the provenance is captured then yes.

Once the metadata is captured, there remain challenges for users in where to start with the 145 million Crossref records. The groups asked Crossref to make it easier for community members to understand and use these records to make informed decisions, including by creating and sharing sample queries, libraries, and case studies.

We’re currently recruiting a Technical Community Manager to help improve the support we provide in this area to API users, service providers, and other metadata integrators .

The importance of retractions/corrections information

There was a lot of discussion about retractions and their importance as trust indicators. The group was surprised by how few retractions are currently registered with Crossref through Crossmark (12k). There was a lot of discussion around why Crossmark isn’t currently being adopted, and interest in taking this forward.

This needs to be a focus for Crossref, to encourage members to register retractions, corrections, and updates, and to make it easier for smaller publishers. There are new and emerging publishers who really want tools to help them demonstrate the legitimacy of their research, and an easy way for them to record corrections and retractions is key.

In their paper Towards a connected and dynamic scholarly record of updates, corrections, and retractions (September 17th, 2022), Ginny Hendricks, Rachael Lammey, and Martyn Rittman discuss how retraction information could be more effectively used - for example, letting a preprint reader know that the resulting article has been retracted, or letting the author of an article know the data that they’ve based their work on has been withdrawn.

Collecting the information is just the start - cascading retraction information throughout the research ecosystem is the main goal, and Crossref plays a central role here. As noted in the Information Quality Lab’s project Reducing the inadvertent spread of retracted science: Shaping a research and implementation agenda, “Many retracted papers are not marked as retracted on publisher and aggregator sites, and retracted articles may still be found in readers’ PDF libraries, including in reference management systems such as Zotero, EndNote, and Mendeley”.

It’s particularly important that this information is fed back to funders and institutions, and the group discussed having push notifications to these audiences for retractions. Some funders even employ staff members whose main purpose is to identify retractions.

It was pointed out that there may be good sources of retraction information (such as Retraction Watch) that Crossref could incorporate and match in our metadata.

Gaps in ‘ownership’, and Crossref’s role

The group discussed the many gaps in ownership for elements of research integrity, and some groups wondered if Crossref should actually change our approach and take on more responsibility for vetting content. However, after discussion, the group mostly agreed that this would mean a change of mission (and more staff) for Crossref and potentially limit global participation, thus making the metadata corpus less useful. Crossref should provide the widest possible metadata in an easy-to-consume format, and “other organisations can provide the verification layer”.

It was acknowledged that it would be easy for Crossref to get overwhelmed, so we ended the day by discussing not only what we should start doing, but also what we should stop doing. Unsurprisingly, there was a lot more to continue or start doing than stop doing!

However, the fact remains that there are gaps in ownership - for example, there is no central arbiter of who ‘owns’ a journal. Also, where do you go if you have a problem with a journal? Often the Committee on Publication Ethics (COPE) is seen as a solution, but they can’t solve this problem alone - it needs a coordinated effort from funders, institutions, publishers, and other partner organisations such as the Open Access Scholarly Publishing Association (OASPA), the Directory of Opena Access Journals (DOAJ), and like-minded organisations.

Many noted that Crossref is well-positioned to convene horizontal multi-stakeholder discussions to start to find solutions.

We also know that there are other industry initiatives aimed at supporting this work. The STM Association’s work on an Integrity Hub is gathering pace and aims to provide, among other things ‘a cloud-based environment for publishers to check submitted articles for research integrity issues’.

What happened next? Turns out, it really is all about relationships…

Since this meeting in Frankfurt last October, we’ve been focusing on relationships - thinking about how we capture them in our metadata, and working in partnership with other organisations to bolster our support for ISR.

The rest of this blog post highlights some of the activities underway:

Increasing participation in Crossref

In January 2023, we launched our new GEM Program, which offers relief from fees for members in the least economically-advantaged countries in the world. By opening up participation even further, we aim to extend the corpus of open metadata, giving opportunities for more connections, more context, and more relationships.

Supporting members in meeting best practices

ISR blog 2 explained more about how we help new members become “good Crossref citizens” with automated onboarding emails, extensive documentation, events and webinars, and help from our support team, Ambassadors, and other members in our Community Forum.

We’ve recently joined forces with COPE, DOAJ, and OASPA to create a new online public forum for organisations interested in adopting best practices in scholarly publishing. At the Publishers Learning And Community Exchange or The PLACE, new scholarly publishers can access information from multiple agencies in one place, ask questions of the experts, and join conversations with each other. Do take a look!

Being clearer on the impact of better metadata

As discussed earlier, better metadata can sometimes bring extra costs, and it’s helpful to understand the impact of this investment. We know from our ongoing outreach work that it’s difficult for our members to keep hearing that Crossref needs more and better metadata. They ask us for resources and increasingly want to see hard evidence of benefits to them. We recently showcased the journey of the American Society for Microbiology which went from ‘zero to hero’ in terms of metadata participation and completeness in Crossref. They describe their efforts to increase their registered metadata over the last few years, and note a significant increase in their average monthly successful DOI resolutions from ~390,000 in 2015 to an average of ~3.7 million in 2022. They found that “the more metadata we push out into the ecosystem, the more it appears to be used… Remembering that your publishing program benefits as much as everyone else’s when you deposit more metadata can help refine your short-term and long-term priorities.”

We know we sound like a broken record sometimes, but now other members can take it from ASM!

Encouraging better metadata and more relationships and identifying ’trust signals'

We’re trying to make it easier for members to accurately register key metadata fields, with the launch of our new grants registration form which will be extended to journals and other record types soon. This includes a ROR lookup - adding this unique identifier for research organisations gives even better context for the metadata.

We are also working to make it possible for anyone to contribute to metadata records, and have the provenance of these contributions clearly asserted.

Metadata adoption is still a key goal for our staff; indeed our new 2023-2025 strategic roadmap specifies…

“We want to be a sustainable source of complete, open, and global scholarly metadata and relationships. We are working towards this vision of a ‘Research Nexus’ by demonstrating the value of richer and connected open metadata, incentivising people to meet best practices, while making it easier to do so.”

… with item number one under projects ‘in focus’, being: “Adoption activities to focus on top metadata adoption priorities, which are:

We’re continuing to talk with the community to work out which metadata elements are most useful as trust signals, and we’re trying to prioritise some of the schema changes required to capture new elements. If you haven’t already, please respond to Patricia Feeney’s metadata priorities survey.

Thinking about retractions and corrections

We’ve been closely involved with the NISO CREC working group, and they should be making the initial draft recommendations public soon - watch this space!

Making it easier to view and compare metadata and expand the relationships

Our Participation Reports provide a visualisation of the metadata that’s available via our free REST API. There’s a separate Participation Report for each member, and it shows what percentage of that member’s content includes nine key metadata elements. It’s an important tool to help those in the community understand our metadata more easily.

We have been working on a new version of Participation Reports, allowing more comparison between members, and extra metadata elements to communicate trustworthiness, including whether each member has thought about the long-term preservation of their content, and whether it has been added to a repository. There is a test version to look at in our Labs sandbox. Do take a look and provide feedback.

We’ve also made public our list of members whose membership was revoked for contravention of the membership terms.

Continuing to work with funders

We’re continuing to work with funders through our growing funder membership, the Funder Advisory Group and other groups, including the Open Research Funders Group, the HRA, Altum, Europe PMC, and the ORCID Funder Interest Group. And we’re continuing to build the important relationships between funding and outputs (see Dominika Tkaczyk’s recent report) and engage with this key audience for research integrity.

Discussions with the community

We’ll be talking about ISR at our next community update on May 3rd - there are two versions of the meeting depending on your timezone - do sign up if you haven’t already. And if you’re attending the SSP conference in June, do come along to our panel “Working together to preserve the integrity of the scholarly record in a transparent and trustworthy way”.

ISR part three: Where does Crossref have the most impact on helping the community to assess the trustworthiness of the scholarly record?

Rachael Lammey — Mon, 17 Oct 2022 00:00:00 +0000

Ans: metadata and services are all underpinned by POSI.

Leading into a blog post with a question always makes my brain jump ahead to answer that question with the simplest answer possible. I was a nightmare English Literature student. ‘Was Macbeth purely a villain?’ ‘No’. *leaves exam*

Just like not giving one-word answers to exam questions, playing our role in the integrity of the scholarly record and helping our members enhance theirs takes thought, explanation, transparency, and work.

Some of the elements Amanda outlines in the previous posts in this series (Part 1, Part 2) really resonated from a product perspective:

We must be cautious that our best practices for demonstrating legitimacy and identifying deceptive behaviour do not raise already-high barriers for emerging publications or organisations that present themselves in ways that some may not recognize as professional standards. Disruption is different from deception. Crossref has an opportunity to think about how to identify deceptive actions and pair that with our efforts to bring more people on board and support their full participation in our ecosystem.

We don’t have the means or desire to be the arbiter of research quality (whatever that means). However, we operate neutrally, at the center of scholarly communications, and we can help develop a shared consensus or framework. Our metadata elements and tools can be positioned to signal or detect trustworthiness. An important distinction is that we can play a role in assessing legitimacy (activities of the actors) but not in quality (calibre of the content itself).

Crossref has lots of plans (and lots to do) to improve our role in ISR

Rather than a long list of things we want to do in terms of tools, services, and functionality, it feels more manageable to break this work into three key areas.

1. Collecting better information in better ways

We think many elements of the metadata our members record with us help expose important information about the research, e.g., authors, publication dates, and abstracts. We also help our members assess submissions for originality via our Similarity Check service, and the ongoing migration to iThenticate V2 aims to better support this aspect of the publication process.

Beyond this, as Amanda points out, ‘once members start registering their content, their metadata speaks about their practices’. Seeing who published a work along with the metadata they provide; validated ORCID IDs to identify the authors, reference lists and links to related research and data, and important updates to the work via Crossmark, all contribute to showing not just the ‘what’ but the ‘how’ so that the community can use that information to support their decision-making.

I always want to stress that this work is not just an ‘ask’ for our members. We are moving in the same direction as we improve the things we do to support organisations in registering their records with us, answering their questions, working with partner organisations like PKP, consulting with our community on pain points, and thinking about how we can better enhance and facilitate their work. We’ve been fortunate that our community has taken the time to engage in discussions with Turnitin on iThenticate improvements, do user testing sessions as we build simple user interfaces to record grants, lead calls and conversations on improving grant metadata and supporting the uptake of ROR and data citation, and provide thoughtful feedback on our recent preprint on CRE metadata. This all helps us to explain, structure, and prioritize our product work.

There are also some closely related R&D-led projects that are already informing our thinking:

A more responsive version of participation reports so that it’s easier for members to identify gaps in their metadata and compare against others.
Making it easier to get metadata back in a format where members can easily redeposit it.
Better matching to help us and our members augment the metadata they send us to add value to the work we all do.

We said in the previous blog posts that we’ll pose questions about what kinds of metadata give what kind of levels of trustworthiness, and have previously highlighted the following activities:

Reporting corrections and retractions through Crossmark metadata. We know that our members are collecting this information, but often it isn’t making it through metadata workflows to us. We’re part of the NISO CREC (Communication of Retractions, Removals, and Expressions of Concern) working group with many of our members and metadata users, as this feels like something critical to address.
Assessing originality using Similarity Check. On average, we’re seeing 320 new Similarity Check subscribers each year, with over 10 million checks being done each year by our members.
Establishing provenance and stakeholders through ORCID and ROR. At the time of writing, we have over 30,000 ROR IDs in Crossref, and this is growing steadily across different record types. ROR is keen to support adoption and so are we.
Acknowledging funding and other support through the use of the Open Funder Registry and registering grants metadata. This has improved in quality and completeness since we launched the Funder Registry in 2014 and with more comprehensive support for grants in more recent years. But we still have work to do, as this paper by Kramer and de Jonge points out: The availability and completeness of open funder metadata.
Citing data for transparency and reproducibility, including linking to related research data. Scholix, MDC and STM Research Data groups.
Demonstrating open peer review by registering peer review reports. Members have already recorded over 300,000 peer reviews with Crossref, opening up this information on their processes.

In your organisation, what weight do you give these? We know that some of our members register some of these things in more volume than others - is that due to their perceived value, technical limitations, or ‘we’re working on it, give us time?’ Do you think of them in the context of the integrity of the record or are we off the mark? Are there other things we haven’t mentioned in this blog that we could capture, report on and highlight?

2. Disseminating this information and supporting its downstream use

We want to make it as easy as possible for everyone to access and use the metadata our members register with us. Especially as some of the biggest metadata users are our members and, more selfishly, us! But there’s no point collecting metadata to support ISR if it’s unwieldy and difficult to access and use.

We’re working on a project, described in the mid-year community update by a number of my colleagues to break down internal metadata silos and model it in a more flexible way. This will lend itself to better information collection and exchange, and support of the Research Nexus by building a relationships API to let anyone see all of the relationships Crossref can see between a given work and well, anything else related to it (citations, links to preprints, links to data to name but a few).

Part of that work will involve supplementing the metadata our members register with high-quality, curated data from selected sources, making it clear where those assertions have come from.

We want our API to perform consistently and well, to contain all the metadata our members register, handle it appropriately, and be able to keep the information in it up-to-date.

Our API will underpin the reports we provide our members (among other things) so that we can provide simple interfaces for organisations to check how they’re doing along with more functional requests. Do their DOIs resolve? Are they submitting metadata updates when they publish a correction? How much will they be billed in a given quarter? We have a lot of internal reporting and need to build more, and if we want to use these, chances are many others do too, so we should open those up.

3. Trying to live up to POSI to underpin this work

When I see a new project, initiative, tool or service in the research ecosystem the first thing I want to do is find out about the organisation itself so that I can base some decisions on that. Lateral reading in action.

At Crossref, we want to show who we are beyond just our tools, services, and products and be transparent about our values. That’s why we have adopted the Principles of Open Scholarly Infrastructure or POSI for short. Now we need to meet these principles and we’re working towards that. POSI proposes three areas that an Open Infrastructure organisation like Crossref can address to garner the trust of the broader scholarly community: accountability (governance), funding (sustainability), and protection of community interests (insurance). POSI also proposes a set of concrete commitments that an organisation can make to build community trust in each area.

So POSI isn’t just opening code and metadata, it’s telling our community how we handle membership, governance, product development, technical and financial stability and security, holding our hands up when we’ve got something wrong, and actively looking to improve upon the things we do.

Are you still reading? If so, you’ve done better than many of my examiners, I’m sure. So stay with us as we work together to ensure we bring quality, transparency, and integrity to the work we all do.

The next part in this series will report back on the feedback and discussions and potentially propose some new or adjusted priorities. Join us at the Frankfurt bookfair this week (hall 4.2, booth M5) or comment on this post below.

ISR part two: How our membership approach helps to preserve the integrity of the scholarly record

Amanda Bartell — Mon, 10 Oct 2022 00:00:00 +0000

In part one of our series on the Integrity of the Scholarly Record (ISR), we talked about how the metadata that our members register with us helps to preserve the integrity of the record, and in particular how ’trust signals’ in the metadata, combined with relationships and context, can help the community assess the work.

In this second blog, we describe membership eligibility and what you can and cannot tell simply from the fact that an organisation is a Crossref member; why increasing participation and reducing barriers actually helps to enhance the integrity of the scholarly record; and how we handle the very small number of cases where there may be a question mark.

Who can become a Crossref member and do we check new applicants?

Membership is open to organisations that “produce professional and scholarly materials and content”, and this is deliberately defined broadly. We’re a global community of members with content in all disciplines, in many formats, with all kinds of business models - research institutions, publishers, government agencies, research funders, banks, museums and many more.

Essentially, if your content is likely to be cited in the research ecosystem and you consider it part of the evidence trail, then you’re eligible to join.

We ask organisations to complete an online application form and accept our member terms. On receipt of the application, we run a few very basic checks to ensure that:

The applicant can meet the membership criteria and seems to have the capacity to fulfill the obligations (and follow our code of conduct).
We are legally permitted to accept them as a member (for example, we can’t accept applications from some countries due to sanctions.
They haven’t previously been a member of Crossref whose membership was revoked.
They haven’t misrepresented themselves in the application (such as their location).
The applicant or an affiliate is not already a member of Crossref (so that we can advise they join under a single membership fee).

As long as the applicant can meet these requirements, and as long as they are able to pay any membership fees upfront for their first year of membership, they are able to become a Crossref member, get a DOI prefix, and start registering their metadata to share it with the global scholarly community.

We are aware that some organisations in some regions may not be able to join Crossref independently. There may be barriers for them - the cost of membership fees, the fact that we only accept payment in US dollars, language barriers or technical barriers.

To help increase participation globally, we work with sponsors in some regions. All sponsors facilitate membership for organisations who wish to participate in Crossref. They pay one central membership fee on behalf of all the members they work with, and they also pay content registration fees on behalf of their members. Many sponsors register content on behalf of their members, and even if they don’t, most provide local language and technical support. Sponsors are able to charge for their services, but it can be a very economical route for a member to join. In the last year, out of the 2,322 new members that we’ve welcomed, almost 58% joined via a sponsor.

We also waive registration fees for members in certain lower income countries who join via three of our sponsors, and we are planning to expand this program soon (pending board approval in November). [EDIT 2022-November-23: The new Global Equitable Membership (GEM) Program was approved and takes effect 1st January 2023]

The importance of keeping barriers to entry low

As you can see, the checks that we run on new applicants are fairly limited in scope. In the last year, we’ve welcomed 2,322 new members and we only declined 39 applications. And 34 of these declined applications were effectively from one organisation whose membership was revoked in 2019.

Even this minimal set of checks takes a lot of research and keeps our member support specialists very busy - thank you Sally Jennings and Robbykha Rosalien (as well as contractors Kim and Collin).

So why shouldn’t we run more extensive checks on new member applicants? Why don’t we check the quality of their content, or that they are following best practices? Why don’t we decline membership for organisations that can’t demonstrate editorial integrity or that aren’t meeting 100% of the membership obligations from the start?

Nevermind the additional capacity that more extensive checks on the over 200 applicants we receive per month would entail, it’s more fitting with our mission to:

enable equitable participation; and
focus on evidence:

Equitable participation

Inclusivity is very important to us - after all, one of our organisational truths (the guiding principles for everything we do) is “come one, come all”, and this is mirrored in the POSI principles that commit us to broad stakeholder representation. We know that for new organisations, it may take them a while to be able to completely fulfil the membership obligations. We support them with information to help them understand what being a participant in the Crossref community entails. These organisations would have less of a chance of developing better practices if we were to limit membership in Crossref to ‘proven’ candidates. Besides, it would introduce a race condition; if joining and sharing metadata through Crossref is widely considered best practice, new entrants need to join Crossref in order to show that they are adopting best practices.

Trust signals and the Research Nexus

Secondly, it’s not our role to make such a call; we don’t have the expertise to decide if an organisation would be considered “good” at what they are producing; there are other organisations guiding in this area, such as with the Principles of Transparency and Best Practice in Scholarly Publishing. Instead, we focus on the decision-making tools, metadata, and relationships that can help provide trust signals for the community.

Once members start registering their content, their activity and metadata speak about their practices – others in the community can process that metadata, combined with its wider context, and identify trust signals to make their own decisions. That metadata can only be shared in an open and machine-readable way if an organisation joins Crossref and starts registering their records and underpinning data with us.

To paint a more detailed picture of the scholarly record, our priority is to get more and varied organisations contributing to the research nexus, rather than putting up barriers and blockers until they are performing perfectly. If they aren’t acting in the best interests of the scholarly community, then having the metadata available to assess will quickly make that obvious and hopefully encourage changes - sunlight being the best disinfectant, as the saying goes.

As we said in the first ISR blog:

“Crossref itself doesn’t assess the quality of content or the integrity of the research process but rather enables those who produce scholarly outputs to provide metadata (effectively evidence) about how they ensure the quality of content and how the outputs fit into the scholarly record.”

In our next post in the series, we’ll talk more about the workflow and decision-making tools we have in place and are planning to develop. We’ll pose questions about what kinds of metadata give what kind of levels of trustworthiness.

Helping new members become “good Crossref citizens”

Once an applicant becomes a member, we help them to completely fulfil the membership terms - ensuring that, for example, they register and display DOIs, keep their metadata up to date, and implement reference linking properly.

We have a lot of documentation on our website, we run regular events and webinars, and we have a series of automated onboarding emails for new members to help them move through the key stages of the member journey from set up and onboarding to levelling up and using additional services like Crossmark and Similarity Check. Our staff are also on hand alongside Ambassadors and other members in our Community Forum. Speaking of POSI (and transparent operations) we receive around 3,000 emails per month with support requests so we are gradually moving support from closed 1:1 email to the more public and efficient community support forum.

We work with members who aren’t fulfilling the obligations to understand challenges and help explain what they need to do. This is currently reactive, but we have plans to automate checks on whether members are meeting the membership terms in future.

Outside of confirming that our members are behaving as “good Crossref citizens”, there aren’t many other areas where the membership team typically gets involved. Our mission is to help preserve the integrity of the scholarly record by making the metadata provided by our members openly available in a machine-readable format. We don’t investigate our members’ business practices or take a deep dive into their editorial processes (such as peer review), and there are many areas where we aren’t able to get involved. For example, we cannot arbitrate title ownership disputes.

It’s all about preserving the integrity of the scholarly record

We do sometimes revoke membership, but this is for limited reasons:

unpaid invoices;
legal sanctions or judgments against the member or its home country; or
contravention of the membership terms.

Membership revocation due to unpaid invoices

We spend a lot of time communicating with members who haven’t paid their invoices and ensuring they have the information they need to solve the problem. Revoking membership due to unpaid fees is an absolute last step for us, but financial sustainability means we can keep the organisation afloat and keep our infrastructure running.

Where members have unpaid fees, we eventually suspend their access to register new records and then ultimately revoke their membership if the fees remain unpaid. Once an organisation’s membership has been revoked, they would need to re-apply if they wanted to become a member again in the future. If accepted, the applicant would need to pay all outstanding invoices before re-joining.

In March 2022, we revoked membership for around 140 members due to unpaid invoices (out of a total of over 17,000 active members).

Membership revocation due to sanctions

Occasionally, we are informed of sanctions that we need to comply with, such as the recent case of Russia invading Ukraine where each Russian member needed to be checked for individual sanctions and some were revoked. Such revocations have to be voted on by the Executive Committee and then ratified by the board. Read more information on our sanctions process.

Membership revocation for cause

Very occasionally there may be evidence that a member is in contravention of the membership terms. This may include:

Misrepresentation in the original membership application
Fraudulent use of identifiers or metadata
Contravening the code of conduct
Any other basis set forth in our governing documents.

We always try to work together with the member to solve problems, and again, revoking membership is an absolute last step. The revocation has to be voted on by the Executive Committee and then ratified by the board.

Our first ever revocation for cause was in July 2019 for OMICS, after the board voted that the US Federal Trade Commission’s ruling against them amounted to a cause for revocation. There have been a handful of cases since. For example, most recently in September this year we revoked membership for a member who was registering DOIs for journals with the ISSNs of similarly-named publications.

There’s more information about our processes to revoke membership on our website.

More participation for the win

In conclusion, we believe that the more parties able to participate in Crossref and provide metadata and context for the research nexus, the more robust this makes the scholarly record.

But do you agree? Are these measures enough? What other information about our membership operations would help us be more transparent? As we said in our first blog, we need your help to establish whether our approach is still the right one, if we are missing anything and what else we might be able to do.

Here’s how you can help:

Join the discussion about the integrity of the scholarly record on our community forum.
Keep an eye out for future blog posts and meetings. We are having a small, in-person discussion prior to the Frankfurt Book Fair and will report on this in a future blog post.
Sign up to attend Crossref LIVE22 for updates on these topics and all things Crossref.
Join and support initiatives and organisations that we partner with or who use our metadata to look at ethical practices, for example, COPE, DOAJ, and OASPA, and review the Principles of Transparency in Scholarly Publishing, which these organisations worked on with WAME.

ISR part one: What is our role in preserving the integrity of the scholarly record?

Amanda Bartell — Thu, 22 Sep 2022 00:00:00 +0000

The integrity of the scholarly record is an essential aspect of research integrity. Every initiative and service that we have launched since our founding has been focused on documenting and clarifying the scholarly record in an open, machine-actionable and scalable form. All of this has been done to make it easier for the community to assess the trustworthiness of scholarly outputs. Now that the scholarly record itself has evolved beyond the published outputs at the end of the research process – to include both the elements of that process and its aftermath – preserving its integrity poses new challenges that we strive to meet… we are reaching out to the community to help inform these efforts.

Scholarly research, and therefore scholarly communications, are rapidly changing with the development of new approaches, technologies, and models. We need open scholarly infrastructure that can adapt to these changes and provide trust signals that enable assessment of the integrity of the research and reflect the ways that research is changing. Crossref has been changing and adapting by building on the concept of the scholarly record with our vision of the Research Nexus:

“a rich and reusable open network of relationships connecting research organisations, people, things, and actions; a scholarly record that the global community can build on forever, for the benefit of society”.

The foundation of the scholarly record and Research Nexus is metadata and relationships - the richer and more comprehensive the metadata and relationships in Crossref records, the more context there is for our members and for the whole scholarly research ecosystem. This will lead to a range of benefits from better discovery and saving researchers time to the assessment of research impact and research integrity. This is why Crossref is focused on enriching metadata to provide more and better trust signals while keeping barriers to membership and participation as low as possible to enable an inclusive scholarly record.

We want to engage with the community to emphasise this role, share our plans for the future, and get feedback to establish if we are heading in the right direction.

This blog explains our current position and will be followed by subsequent posts exploring all our services and plans in this area, as well as more details on our membership operations and policies.

What is “Integrity of the Scholarly Record” (ISR), and how does it feed into Research Integrity?

The US National Institutes of Health (NIH) defines research integrity as a set of values in scientific research: honesty; accuracy; efficiency; and objectivity. It’s concerned with the soundness of the process of science. As a subset of that, the outputs of the scholarly publishing process create a “scholarly record” which allows those in the community to find evidence and context to help confirm whether these values have been adhered to. The scholarly record is Crossref’s focus. This means that Crossref itself doesn’t assess the quality of content or the integrity of the research process but rather enables those who produce scholarly outputs to provide metadata (effectively evidence) about how they ensure the quality of content and how the outputs fit into the scholarly record (through reference links, ORCID iDs for authors, ROR IDs for affiliations, funding and licensing information, etc.).

Crossref members include any organisation that produces research objects and materials (publishers, societies, universities, funders, research institutions, scholars) so they can establish a persistent record—tied to a persistent and unique identifier—for these outputs and supply metadata about this content in an open, machine-readable way. Maintaining this record for the long term, and adding in an important layer of context, establishes the integrity of the scholarly record as well as ensuring it is something that can be used by the whole community to improve scholarly research for generations to come.

The scholarly record is about more than just published outputs - it’s also a network of inputs, relationships, and contexts

In the past, the Scholarly Record was seen as just the published outputs at the end of the research process - for example, journal articles or book chapters. But as the OCLC Research Group notes in their 2014 report on The Evolving Scholarly Record:

“The boundaries of the scholarly record are in flux, as they stretch to extend over an ever-expanding range of materials.”

OCLC describes how outputs at the “process” and “aftermath” stages of the research process are becoming increasingly important alongside the outputs at the traditional “outcomes” stage.

We like to take this even further. We think the evolving Scholarly Record is about more than just recording different types of works. As the above report notes “The scholarly record is evolving to have greater emphasis on collecting and curating context of scholarly inquiry […] One can imagine an article in quantitative biology published in a Wiley journal, the data for which resides in Dryad; the e-print in arXiv; and the conference poster in F1000. All of these materials may be considered part of the scholarly record, but no single institution will collect them all. Instead, access is achieved through a coordination of stewardship roles in which the scholarly record is decomposed into discrete, interrelated units that organisations specialize in collecting, preserving, and making available.”

It’s this interrelatedness that we think is important, and Crossref plays an important role in collecting, matching, and sharing those relationships. We now focus on this ‘nexus’ - so no longer primarily the different types of objects, but increasingly the interplay and relationships between them. The context, rather than the individual metadata elements, is what’s key.

Martin Eve explores this idea further in his blog What is the Scholarly Record, suggesting “the scholarly record is a decentralized network of evolving truth assertions” and “Whether a truth assertion is part of the scholarly record is determined by another set of distributed assertions and their power configurations (say, through institutional affiliation) of the individuals who make such assertions.”

Barbara Fister’s excellent talk about the importance of lateral reading as a way to understand information systems discusses how professional fact checkers “engaged in “lateral reading,” check other sources for context before spending time reading and analyzing a source.”

Fister highlights the “SIFT” approach from A Curriculum for Civic Online Reasoning, created by a group of educators at Stanford University for students to evaluate online content. And she argues that this approach is also useful for assessing scholarly materials noting

“The networked, social nature of scholarship is worth making explicit”.

Where does Crossref fit in? Where do we have the most impact and opportunity?

To address the question of our role in the integrity of the scholarly record, we need to understand several aspects that Crossref has to balance in this capacity, such as

We don’t have the means or desire to be the arbiter of research quality. However, we operate neutrally, at the centre of scholarly communications, and we can help develop a shared consensus or framework. Our metadata elements and tools can be positioned to signal or detect trustworthiness. An important distinction is that we can play a role in assessing legitimacy but not in assessing quality.
We must be cautious that our best practices for demonstrating legitimacy and handling less-than-legitimate behaviour do not raise already-high barriers for emerging publications or organisations that present in ways that some may not recognise as professional standards. Disruption is different from deception. In discussions with our board this point has come out strongly: that Crossref has an opportunity to think about how to help the community identify deceptive actions and pair that with our efforts to bring more people on board.
Addressing this issue may involve changes to our membership eligibility and processes, bylaws, policies, staff resources, and technical and metadata solutions; actually, a combination of all these aspects. Many of these are projects that are already planned and we have ideas for extending these.
We regularly review the process we use for evaluating when and why to revoke membership for reasons other than non-payment. The volume of cases that we believe justify membership revocation—while a tiny fraction of members—is growing and does take staff and legal resources to address.

Crossref and our members aleady help preserve the integrity of the scholarly record in significant ways

Almost all of our services in some way touch on enabling people to express and evaluate trustworthiness; our mission statement commits us to “making research objects easy to find, cite, link, assess, and reuse […] all to help put research in context.”

We have, of course, specific tools and services that augment this activity too. Many members are active in:

Reporting corrections and retractions through Crossmark metadata.
Assessing originality using Similarity Check.
Conveying their stewardship via the public participation report.
Establishing provenance and stakeholders through funding metadata, ORCID, and ROR.
Acknowledging funding through the use of the Open Funder Registry and registering grants metadata.
Citing data for transparency and reproducibility, including linking to related research data via Event Data.
Demonstrating open peer review by registering peer review reports.

As recently concluded in this Nature editorial calling for us to think beyond open references,

“Depositing all relevant metadata in Crossref should become the norm in scholarly publishing.”

For those members just starting out on their journey, there are some immediate specific things that all members are able to do. Check your participation report and start registering more metadata to add that contextual layer:

References
Abstracts
Corrections and retractions via Crossmark
License links
ORCID IDs for authors
ROR IDs for affiliations
Grant IDs for funding acknowledgements
Cite data (preferably using DataCite DOIs in reference lists)
Register all related objects such as versions and translations via relationships
Register grants with Crossref (funder members).

By enabling our members to register their research objects and create metadata records about them that are freely and openly shared with the scholarly community, we facilitate them in being able to communicate the context and trustworthiness of that object.

And within that metadata, they can create relationships not just between research objects and also between research stakeholders - the individuals, affiliations, funders, and other players involved. That’s why we work so closely with other parts of foundational scholarly infrastructure (ORCID, DataCite, ROR) and why we now have more than 30 funders registering grants with us. We want to help to capture, identify, and link together all these important elements and more to deliver context for the scholarly record.

We started this blog by talking about the changes that are taking place in the world of research and how the infrastructure needs to adapt and change. Although we have extensive plans in place to improve our contribution to ISR, we need your help to establish whether our role is still the right one, whether we are missing anything and what else we might be able to do.

Join the discussion about the integrity of the scholarly record, and the Research Nexus on our Community Forum.
Keep an eye out for future blog posts and meetings. We are having a small, in-person discussion prior to the Frankfurt Book Fair and will report on this in a future blog post.
Sign up to attend Crossref LIVE22 for updates on these topics and all things Crossref.
Join and support initiatives and organisations that we partner with or who use our metadata to look at ethical practices in publishing, for example, COPE, DOAJ, and OASPA, and review the Principles of Transparency in Scholarly Publishing, which these organisations worked on with WAME.

In the coming weeks, we will post more about our product and metadata plans and also about the specifics of membership operations and cases we see and how we’re currently addressing them.

Metadata and integrity: the unlikely bedfellows of scholarly research

Damian Pattinson — Thu, 14 Dec 2017 00:00:00 +0000

I was invited recently to present parliamentary evidence to the House of Commons Science and Technology Select Committee on the subject of Research Integrity. For those not familiar with the arcane workings of the British Parliamentary system, a Select Committee is essentially the place where governments, and government bodies, are held to account. So it was refreshing to be invited to a hearing that wasn’t about Brexit.

The interest of the British Parliament in the integrity of scientific research confirms just how far science’s ongoing “reproducibility crisis” has reached. The fact that a large proportion of the published literature cannot be reproduced is clearly problematic, and this call to action from MPs is very welcome. And why would the government not be interested? At stake is the process of how new knowledge is created, and how reliable that purported knowledge is.

The other issue driving this overview of research practices are the cases of deliberate fraud and wrongdoing that have recently created headlines (e.g., the STAP papers concerning the reprogramming of stem cells). While these cases are clearly dramatic outliers, they nevertheless serve to diminish public confidence in scholarly research and the findings that come out of this enterprise.

As with most inquiries, the question quickly boiled down to: who is to blame? As Bill Grant MP asked me directly, “Where does the responsibility lie?”

My answer was lifted from an article by Ginny Barbour and colleagues in F1000Research this November (https://doi-org.pluma.sjfc.edu/10.12688/f1000research.13060.1): publishers are responsible for the integrity of the published literature, while institutions and employers are ultimately responsible for the conduct of their staff. Misconduct entails intent, usually to deceive the reader into believing a conclusion that the researcher wishes them to believe. But journal editors can never know, and are not in a position to investigate, whether a researcher has deliberately falsified their data.

However, there are things that publishers can do to ensure high standards of integrity. Much of this involves making a study’s authors publish as much information about what they have done as possible - the more the reader can see of how data were generated, the more that reader can trust the findings communicated in the published article.

Article metadata directly supports this function. It provides structure and transparency to information pertaining to ethics and integrity. And because metadata is independent of the main article, it can be readable even if the article itself is locked behind a paywall.

Crossref already provides metadata that can demonstrate the integrity of published articles. The metadata collected on 91+ million scholarly works across publishers and disciplines is open and freely accessible to all. Bibliographic information, for example, allows readers to see who the authors of the article are, where they are from, and what else they have published. Similarly, funding data allows readers to identify potential conflicts of interest, for example if the funder has commercial or political affiliations. Even if the reader cannot see the conflict of interest statement (or if the journal has not provided one), they can use the funding statement to surface potential conflicts.

And if they wanted, publishers could provide additional metadata to add still more transparency to the research process. Ethical approval by institutional review boards, for example, could be captured, and any protocol numbers traced back to the original ethics committee approval. At present the process of ethical approval varies from country to country, and from institution to institution. Encouraging authors and journals to deposit information on the approval process would both demonstrate the high ethical standards the author is working to, and also improve the standards themselves, since institutions would have to encode their approval processes in a way that is understandable to others. This could pave the way to significantly higher international ethical standards, all through a simple addition to the indexed metadata underlying the scholarly literature.

One key recommendation that I and many others made to the Committee was, in short, “show your work”. As a researcher, that means showing your data. As a publisher, that means showing what checks you have done. In both cases, metadata can help.

A major issue that publishers and researchers can – and should – address is the provision of actual scientific data. Most papers, today, present only the end results of the authors’ (often quite extensive) analyses. The case for sharing data is an obvious one - many recent cases of misconduct could have been identified earlier, or even avoided altogether, if editors and readers had had access to underlying datasets.

With images, a requirement to submit raw images alongside the edited figures would dramatically reduce the cases of manipulation that are rife in the literature (studies suggest up to 20% of papers have some kind of inappropriate figure manipulation, with around 1 in 40 papers showing manipulation beyond that which can be expected to be a result of error). Similarly, providing the numbers that a paper’s analyses are based upon would allow readers to fully assess if datasets are distributed as would be expected through random sampling, and, if they choose, to determine if the data are sufficient to support the statistical inferences made in the paper. The Crossref schema – by providing unique identifiers to data citations - makes this link between data and paper possible. (See the recent blog post on the Research Nexus for more information.)

For publishers, showing your work also means being transparent to your readers about the editorial checks that a manuscript has undergone. Crossref has a tool that enables this editorial transparency: it’s called Crossmark. Crossmark allows readers to see the most up-to-date information about an article, even on downloaded PDFs. In most cases it is used to show whether the version of an article is most recent one, or whether any corrigenda or retractions have been subsequently added. But it can also be used to provide whatever information a publisher wishes to share about the paper. Some journals have experimented with using Crossmark to ‘thread’ publications together, for example, by linking all the outputs generated from a single clinical trial registration number (blog post here). But publishers could go further and display metadata pertaining to the editorial checks they have performed on a paper. So Crossmark could tell readers that the paper has been checked for plagiarism, or figure manipulation, or reporting standards such as CONSORT or ARRIVE guidelines. Here at Research Square we have been addressing this with a series of Badges that researchers can apply to their papers to demonstrate what checks have been performed.

Together, these implementations would provide value to the reader, who can see exactly what has been checked, and to the publisher, who can show how rigorous their editorial processes are. It would also serve to highlight the integrity of the authors who have passed all of these checks.

Research integrity is not something that can be easily measured but, unlike wit or charm, it is something that people generally know that they have.* This means that they just need to be transparent in their output to demonstrate this to the world. Metadata provides a simple way of doing this, so researchers and publishers should make sure they provide it as openly as they can.

*with apologies to Laurie Lee for the mangled quote