Crossmark on Crossref

Mission Accessible: building better user interfaces for everyone

Lena Stoll — Thu, 21 May 2026 00:00:00 +0000

Today is Global Accessibility Awareness Day, and accessibility has been on our minds lately. We’ve recently completed an internal audit of all our user interfaces, and have added a new accessibility page to our website, where you can find the accessibility documentation that we put together as part of the audit.

Why accessibility matters

Of course we want to keep the barriers to participation in Crossref as low as possible for users with various disabilities. But also, more accessible tools work better for everyone. A person’s access needs can change really quickly: even if you consider yourself to be relatively able-bodied, you are only one minor inconvenience away from at least a temporary disability. All it takes is some dazzling sunlight hitting your eye or your phone screen, or perhaps your dog going after a rabbit in an awkward direction while you are holding the lead (ask me how I know!) - and before you know it, you will be relying on accessibility features to navigate the digital and/or physical world for a while.

An accessible user interface is one that you can navigate and interact with by various methods, including a mouse or touchpad, keyboard, screen reader, voice control, and other assistive technologies. It can be used on various screen sizes and supports zooming in or out without losing any content or functionality. It has sufficient colour contrast, doesn’t flash fast-moving images at you, and has a clear structure that can be understood by both humans and machines.

Where we are today

It is worth mentioning that we didn’t only start thinking about accessibility when we started tackling the full audit of our user interfaces in March 2026. For example, Patrick Vale has previously written in this blog about a browser extension he has created to improve the accessibility of DOI links anywhere on the Internet. And we have known for a long time that there were accessibility gaps in many of our tools, but we didn’t have this centrally documented anywhere.

When we did begin testing all our interfaces for compliance with level AA of the Web Content Accessibility Guidelines (WCAG 2.2) as part of the audit, we knew that some of what we would find was not going to be pretty. In the 26+ years of working with and for the scholarly community, Crossref has built countless tools and reports to offer to members and users, many of which we still maintain today. These are often decades old and have been built in a way that makes it virtually impossible to make them more accessible without rebuilding them entirely. So we know that we will continue to have accessibility gaps for the foreseeable future, but at least now we have a better idea of the scale of the challenge.

It’s also not all doom and gloom: more recently created user interfaces, such our new Metadata Manager, performed much better in the audit than legacy alternatives such as the web deposit form. We found a similar trend when looking at our report interfaces. To illustrate this, compare what happens when running the axe DevTools extension for Google Chrome on a member’s participation report - this is a user interface that was completely re-implemented in 2025. Doing this brings up 26 issues:

Meanwhile, the browsable title list, which has completed a few more trips around the sun, has 254 issues listed:

Beyond WCAG

If you’ve read this far, I hope you are convinced that accessibility is more than just ticking boxes on a conformance report. But especially for a global community like ours, there are other, less technical barriers to participation that we have to consider. For example, language is a major accessibility factor: much of what we as Crossref staff write and say is in English. When we host community events, we enable captions, and we try to leave space for these captions at the bottom of our slides.

We have also started experimenting with simultaneous interpretation during our online events, such as our recent project showcase event for the 2026 metadata sprint in São Paulo. You can find recordings of this event in Spanish, Portuguese, and English on our YouTube channel to see the promising results of these efforts.

What we are working on next

We are currently addressing the accessibility issues identified in our audit of the Crossmark service. Many Crossref members have implemented the Crossmark button and pop-up on their own platforms and websites, so we thought this was a great place to start the remediation efforts following our audit.

We are also in the process of redesigning our main website, www.crossref.org, following an information architecture review completed in 2025. Making changes to the design and navigation of our website will be the perfect opportunity to make our content not just more discoverable and more understandable, but also more accessible.

Clearly there is even more to be done, so watch this space for more updates on our accessibility roadmap and improvements. And if you have first-hand experience of using Crossref services and interfaces with assistive technologies, or you have other input or feedback you’d like to share, leave a comment below or start a discussion in our community forum.

References

Vale, P. (2025). Enhancing DOI Accessibility for All Users. Crossref. https://doi-org.pluma.sjfc.edu/10.64000/pp4rw-mtv44
World Wide Web Consortium (2024). Web Content Accessibility Guidelines (WCAG) 2.2. Retrieved May 8, 2026, from https://www.w3.org/TR/WCAG22/
Montilla, L. & Mahomed, R. (2026). Voices from Crossref Metadata Sprint in São Paulo. Crossref. https://doi-org.pluma.sjfc.edu/10.64000/a5qzf-k1738
Stoll, L. & Korzec, K. (2025). Request for proposals: Crossref website information architecture review. Crossref. https://doi-org.pluma.sjfc.edu/10.64000/058mr-k3s56

Research Integrity Roundtable 2024

Martyn Rittman — Fri, 15 Nov 2024 00:00:00 +0000

For the third year in a row, Crossref hosted a roundtable on research integrity prior to the Frankfurt book fair. This year the event looked at Crossmark, our tool to display retractions and other post-publication updates to readers.

Since the start of 2024, we have been carrying out a consultation on Crossmark, gathering feedback and input from a range of members. The roundtable discussion was a chance to check and refine some of the conclusions we’ve come to, and gather more suggestions on the way forward. As in previous years, we were able to include a range of organisations, which led to lively and interesting discussions. See below for the full participant list.

Crossmark feedback

We started by presenting Crossmark and a summary of the consultation process. There are a number of areas where we have learned more about how the community operates or found that Crossmark needs to adapt. These include:

Implementation: Our members have struggled to implement Crossmark and uptake is low. At the same time, in many organisations the workflows for handling retractions are not well-defined because they are rarely used, if ever. The responsibility for updating Crossref metadata can be unclear and this may be a factor in the low uptake.

Education: There are different levels of understanding about how to handle retractions. Some members are very defensive when asked about retractions, others state they will never make updates to published works. How can we have a constructive conversation where the value of communicating updates appropriately is recognised?

Community engagement: Given the different scales, locations, disciplines, and technologies used by our members, it looks like one size will not fit all when it comes to updates. How can we get continual, representative feedback on new tools and processes?

Metadata assertions: Crossmark allows the deposit of metadata using custom field names, however this metadata seems to have low usefulness and is not highly valued by the community. Should we continue to collect it? Can we make some of the most-used field names part of our standard schema?

Changing the Crossmark UI: Although we didn’t specifically ask about it during the consultation, the look of the Crossref logo often came up, and concern that it is not recognised and not well-used. Can we change the look and behaviour so that it has more impact?

NISO Recommendations

Patrick Hargitt represented the NISO group on Communication of Retractions, Removals, and Expressions of Concern (CREC). The group’s recommendations were published earlier this year and cover how retractions are communicated. CREC arose from an earlier project, IRSRS. A large part of the motivation is that retracted works continued to be cited, with citing authors apparently unaware of the retraction. Patrick presented the CREC recommendations, which cover:

Metadata receipt, display, and distribution,
Which metadata elements to communicate,
How to implement the recommendations,
Discussion of some special cases,
Key stakeholders and their responsibilities.

The two presentations prompted discussion, which was taken into the first of two workshops.

First workshop: Improving collection of retractions and Crossmark

The first workshop looked at proposed changes to Crossmark and how to encourage more members to deposit their retractions, corrections, and other post-publication updates. Several important themes emerged.

First, the question of whose responsibility it should be to provide metadata on retractions and similar updates. Crossref has a responsibility to work with the community to obtain high quality and complete metadata; publishers should take responsibility for handling issues of research integrity and reporting them to relevant downstream services, like Crossref; and platforms need to provide tools that allow easy reporting of retractions.

The value of Crossmark appearing in PDFs was reiterated. The fact that a PDF can be downloaded, and years later there is a way to tell whether it has been retracted or not is highly valued. There was also the suggestion that the Crossmark logo on web pages can indicate a change before it has been clicked. This is something that we have been considering at Crossref and it was useful to have the idea reinforced. Another suggestion was that a browser plugin would make a good complement to Crossmark.

Implementation issues with Crossmark were raised, including that it’s difficult to validate whether a specific implementation is complete. There are a number of different changes (to metadata deposit and content, and websites) that need to work together to have Crossmark fully functional. There were several questions and a discussion about Retraction Watch data. Some were about understanding its collection and validation. A number of participants are actively using the data and it was great to see the variety of applications.

Second workshop: Community use of retraction metadata

The second workshop focused on a broader set of downstream organisations that might want to make use of retraction metadata. We looked at stakeholders and their needs, and attempted to match them up with existing tools. Several gaps were identified as a result, which may provide opportunities for new services or collaborations to fill them.

We identified a number of tools available for publishers, editorial systems, metadata researchers, and readers. A good example is reference managers, many of which are now highlighting retracted works to authors. This can help to reduce the number of retracted works being cited. Publishing platforms are also providing support to editors, using tools that include retraction metadata.

Some of the stakeholders identified have limited tools for identifying retractions that are relevant to them. These include funders, archives and repositories, journalists, and institutions.

Often, there are pathways for retraction data to be communicated but they are not being sufficiently used. There needs to be a concerted effort to improve the quality of retraction metadata for tools to function better. For example, a second author on a paper might not know that a correction or retraction is planned for their article. If their email or ORCID isn’t included in the metadata, an alerting tool wouldn’t be able to let them know. A similar argument can be made for institutions or funders if they are not well-identified in the metadata.

The question of standardisation of metadata was raised. It seems too early to implement a full set of standards at the moment. CREC and similar initiatives have documented and accommodated for a range of practices while providing guidance and principles to work towards. More discussion is needed in the community to work out paths that could be applied across the broad spectrum of scholarly communication.

Conclusion

The event was very valuable in bringing up a range of topics related to retraction and communication of post-publication changes to scholarly works. We are grateful to all of the participants for their contributions and sharing their diverse experience and opinions with us.

Research integrity is an area of flux, with significant changes over the past few years. While there has been progress, there remain gaps in metadata and tools to communicate retractions. This is something that Crossref will continue to contribute to, and Crossmark clearly still has a role to play.

Some of the ideas and suggestions from the discussion can be implemented in the near future. Others need further development, and we will continue to engage the community. Reading this, there may be topics where you feel you have a role to play. We are keen to partner with other organisations in this space as we continue to improve the transparency and communication of metadata for post-publication updates.

Participants

Many thanks to the participants. Here is the full list of those that attended:

Name	Role	Organisation
Aaron Wood	Head, Product & Content Management	American Psychological Association
Adya Misra	Associate Director, Research Integrity	Sage
Bianca Kramer		Sesame Open Science
Constanze Schelhorn	Head of Indexing	MDPI
Guillaume Cabanac	Full Professor	University of Toulouse
Hong Zhou	Director of AI Product	Wiley
Jennifer Wright	Head of Publication Ethics and Research Integrity	Cambridge University Press
Johanssen Obanda	Community Engagement Manager	Crossref
Joris van Rossum	Program Director	STM Solutions
Kathryn Weber-Boer	Data & Analytics	Digital Science
Kornelia Korzec	Director of Community	Crossref
Kruna Vukmirovic	Publisher- Journals	The Institution of Engineering and Technology
Lena Stoll	Product Manager	Crossref
Leslie McIntosh	VP, Research Integrity	Digital Science
Liying Yang	Professor	CAS Library
Luis Montilla	Technical Community Manager	Crossref
Madhura Amdekar	Community Engagement Manager	Crossref
Martyn Rittman	Progam Lead	Crossref
Maryna Kovalyova	Member Experience Manager	Crossref
Mina Roussenova	Project Manager, Strategic Projects	Karger
Osnat Vilenchik	VP Content Operations	Ex Libris, part of Clarivate
Patrick Hargitt	Senior Director of Product Management	Atypon/Wiley
Paul Davis	Tech Support & R&D Analyst	Crossref
Sami Benchekroun	CEO	Morressier
Scott Delman	Director of Publications	Association of Computing Machinery (ACM)
Shilpi Mehra	Head, Research Integrity & Paperpal Preflight	Cactus Communications
Sichao Tong		Chinese Academy of Sciences, Library

Crossmark community consultation: What did we learn?

Martyn Rittman — Tue, 02 Jul 2024 00:00:00 +0000

In the first half of this year we’ve been talking to our community about post-publication changes and Crossmark. When a piece of research is published it isn’t the end of the journey—it is read, reused, and sometimes modified. That’s why we run Crossmark, as a way to provide notifications of important changes to research made after publication. Readers can see if the research they are looking at has updates by clicking the Crossmark logo. They also see useful information about the editorial process, and links to things like funding and registered clinical trials. All of this contributes to what we call the integrity of the scholarly record.

Crossmark has been around a long time and the context around it is constantly changing. It last had a major update in 2016 and in 2020 we removed fees for its use.

The past few years have seen a more intense focus on research integrity among the scholarly communications community, leading to more retractions and calling out large-scale manipulation of editorial processes. At the same time, we haven’t seen an increase in the uptake of Crossmark, which is still used by only a minority of our members. We would like to know why the uptake is low and whether there is more we can do in this area. To dig into this, in the first part of 2024 we reached out to members of our community.

What did we do?

We wanted to learn about attitudes towards Crossmark and related aspects of research integrity. This was done in several ways:

Structured interviews with eight of our members.
Round tables at Crossref LIVE events in Bogota and Nairobi
Surveying a selection of our members, which led to 94 responses.

The topics we asked about were related to how post-publication updates are made and communicated, and which metadata demonstrates good practice.

We are extremely grateful to the members who contributed. They provided valuable feedback and have helped to shape the future of Crossmark and our approach to the integrity of the scholarly record.

What did we find?

Across the various groups there were a few common themes, which fell into several areas.

Communication of updates is highly valued, and seen as the most important role that Crossmark can play. Some of those we spoke to would like readers to see if there is an update as soon as a page opens, without having to open a popup. This could be done by having a logo that changes colour, shape, or size.
Conversely, not as much enthusiasm was shown for the metadata assertions. These are additional fields that can be displayed to readers in the Crossmark popup. There wasn’t a strong consensus on which commonly-made assertions are the most important for research integrity.
There is diversity in attitudes towards making updates to published works, what research integrity means, and approaches to workflows for updates. Even within a single organisation, a number of different workflows and multiple staff members might be called on to update published research. This makes things complex and means that it can be difficult to fit Crossmark in.
There are technical challenges to getting started with Crossmark. Those responsible for implementing Crossmark are often technical staff who struggle with the documentation we provide in English. There is also no plugin for OJS, a widely-used open source editorial software. It is more difficult to deposit Crossmark metadata for books than journal articles, and many article types don’t permit Crossmark metadata at all. On the other hand, those who successfully installed Crossmark found it easy to use and low-maintenance.

Overall, it seems that Crossmark still has an important role to play but there are changes and improvements we can make.

What’s next?

Here are the main areas we intend to follow up on in the coming months.

Implementation

We need to look at how to make implementation more straight-forward. Can we provide multilingual documentation, plugins, run workshops or webinars, or make changes to Crossmark to lower the barrier to entry?

Understanding workflows

Can we collaborate with our members and other organisations to reach a better understanding of how to update published works? Are there alternative workflows we need to support? Have we made it too difficult to understand and implement the options we currently have?

While updates are always likely to be rare, we want to help members understand the benefits of making them. We talked to some members who were proud of never having published a retraction or correction, which left us wondering whether they are missing legitimate opportunities to correct the scholarly record. We also know that for some members and many work types (preprints, for example), updates are made without a separate published notification. Can we better understand the role that the published updates play and communicate updates even if there isn’t a published notice?

Ongoing feedback

Clearly one size doesn’t fit all when it comes to implementing and communicating updates. We need to find ways of keeping in touch with the community to test new solutions with as broad a range of members as possible. We want to avoid catering to a minority and leaving others struggling to find ways to implement a solution.

Custom metadata?

Is there an ongoing need for metadata assertions? Many of the assertions currently made are possible as standard metadata and others could be included in our deposit schema. We want to consider removing the option to add assertions. This needs more feedback from the community, especially those who currently make use of assertions.

Redesign the UI

Crossmark doesn’t have the recognition with readers we would like. Is there a way we can redesign it to make it more associated with Crossref and accurate metadata? We intend to explore different designs, and test them with members and readers.

Encouraging even greater reporting of corrections and retractions

Kirsty Meddings — Mon, 30 Mar 2020 00:00:00 +0000

TL;DR: We no longer charge fees for members to participate in Crossmark, and we encourage all our members to register metadata about corrections and retractions - even if you can’t yet add the Crossmark button and pop-up box to your landing pages or PDFs.

–

Research doesn’t stand still; even after publication, articles can be updated with supplementary data or corrections. When research outputs are is changed in this way the publisher should report and link it, so that those accessing and citing the content know if it’s been updated, corrected or even retracted. This also emphasizes the member’s commitment to the ongoing stewardship of research outputs.

Many people find and store articles to read later, either as PDFs on their laptop or on one of any number of reference management systems - when they come back to read and cite these articles, possibly many months later, they want to know if the version they have is current or not.

Removing Crossmark fees

To encourage even wider adoption of Crossmark, and to promote best practice around better reporting of corrections and retractions, we will no longer be charging additional fees for our Crossmark service. This change applies to all Crossmark metadata registered from 1 January 2020. All members are now encouraged to add Crossmark metadata and add the Crossmark button and pop-up box to their publications - and you can do so as part of your regular content registration.

Richer metadata gives important context

We know that there are many more corrections and retractions that are not yet being registered, and to address this, we are now asking all of our members to start registering metadata for significant updates to your publications, even if you don’t implement the Crossmark button and pop-up box on your content. Remember, anyone can access the Crossmark metadata through our public REST API, and start using it straight away - even if you’re not ready to implement the Crossmark button.

Check out how to get started; if you only want to deposit metadata, follow steps one through four. If you also want to add the Crossmark button and pop-up box to your web pages/PDFs so that readers can easily see when content has changed, then also follow the rest of the steps.

Crossmark

We launched Crossmark in 2012 to raise awareness of these critical changes, by asking Crossref members to:

record such updates in your metadata, either as part of your regular Crossref metadata deposit, or deposited as stand-alone data for back-year records
help readers find out about the changes by placing a Crossmark button and pop-up box (which is consistent across all members making it recognizable to readers) on your landing pages and in PDFs

Members can also use Crossmark to register additional metadata about content, giving further context and background for the reader. These metadata appear in the “More Information” section of the Crossmark box. 7 million DOIs have some additional metadata, the most common being copyright statements, publication history, and peer review methods.

Anyone can access the Crossmark metadata through our public REST API, providing a myriad of opportunities for integration with other systems, and analysis of changes to the scholarly record.

Who has implemented Crossmark?

440 Crossref members have implemented Crossmark to date. 11.4 million DOIs have some Crossmark metadata.

	Total DOIs	DOIs with Crossmark metadata	%
Journal articles	80,862,460	10,155,340	12.56%
Book chapters	14,040,646	792,953	5.65%
Conference Papers	6,175,733	457,237	7.40%
Datasets	1,862,852	19,206	1.03%
Books	753,298	239	0.03%
Monographs	469,333	23	0.00%

Of those, about 130,000 contain an update:

You can see which members or journals have implemented Crossmark by viewing the relevant Crossref Participation Report.

Putting content in context

Kirsty Meddings — Mon, 13 May 2019 00:00:00 +0000

You can’t go far on this blog without reading about the importance of registering rich metadata. Over the past year we’ve been encouraging all of our members to review the metadata they are sending us and find out which gaps need filling by looking at their Participation Report.

The metadata elements that are tracked in Participation Reports are mostly beyond the standard bibliographic information that is used to identify a work. They are important because they provide context: they tell the reader how the research was funded, what license it’s published under, and more about its authors via links to their ORCID profiles. And while this metadata is all available through our APIs, we also display much of it to readers through our Crossmark service.

Crossmark is also about providing context. It is a button placed on content, which when clicked on brings up a pop-up box that tells the reader about significant updates such as corrections and retractions, together with other information about the publishing and editorial processes that have been applied to the content ahead of publication.

The Crossmark box can display information about authors, funders and licenses. In addition, our members can add “More information” and often do in the form of publication history, links to supporting materials, and peer review information. All of this supporting information helps the reader assess how well the content has been - and continues to be - curated by the publisher.

Who’s in?

250 Crossref members have signed up to use Crossmark (it’s an add-on service with its own fees). Though optional, some star pupils have even added Crossmark to their back-year content and as a result have Crossmark coverage on 99% of their content (kudos to PLOS, Rockefeller University Press and the societies represented by KAMJE, to name a few).

At the other extreme, some have applied Crossmark to less than 10% - these tend to be members with back-year records going back many decades, who are just implementing Crossmark for their more recent research outputs. Crossmark coverage is one of the things tracked in Participation Reports - pop over and take a look if you want to see what your organisation is doing.

So what additional metadata has been registered by members using Crossmark? (data snapshot from our REST API April 2019):

8,711,500 content items have some Crossmark metadata
104,650 updates to content have been registered. Of these
55,000 are corrections and 28,000 errata
16,000 are new versions or new editions
2,700 are retractions and 1,280 are withdrawals
4,830,510 content items have some custom metadata, which appears in the More Information section of the Crossmark box. The most common metadata provided here is publication history, followed by copyright statements, the peer review method used, and whether the item has been checked for originality using Similarity Check.

Some news on clicks and views

We’ve been collecting usage statistics more or less since the Crossmark service launched in 2012, but have lacked a suitable way to share them. This will change later this year! In preparation, I’ve been digging around in the data and uncovered some interesting things.

I was able to do a degree of comparison between Crossmark usage against overall article views using PLOS articles as they make their usage data openly available. I spot-checked fifteen articles and found that most of them had a monthly number of clicks on the Crossmark button in the low-twenties, regardless of the number of total page views the article had received.

The highly viewed paper above shows relatively very few clicks on the Crossmark button, whereas on the paper with fewer views, below, clicks on the button follow the overall pattern of usage.

It’s not unreasonable to suppose that a paper with very high usage has a higher proportion of lay readers visiting it, whereas a more niche paper is being visited by those with a research interest. This is encouraging, as it suggests researchers are interested in checking the status of the content and the additional “trust signals” that the Crossmark box can provide.

Web pages vs PDFs

We track the number of clicks on the Crossmark button in PDFs separately to those that come from web pages. (There are some that we can’t determine, usually because the link behind the button has been incorrectly formatted, but for most members these are minimal.)

I looked at the 30 members with most Crossmark coverage, and averaged the number of clicks over a six month period in 2018. For two thirds of these members, clicks on the Crossmark button on their web pages exceed those in their PDFs, but there are also definite outliers.

Some are easily explained: member #6 hasn’t put the Crossmark button in any of their PDFs, while member #21 has only put it in their PDFs. Member 10 has the button on its article landing pages hidden in a “more information” section that the reader has to click to expand.

That said, member #20 has the button displayed prominently next to the article title but gets 85% of Crossmark clicks from PDFs. There’s no obvious subject bias - four of the members above are physics publishers - two have many more PDF clicks, two have more HTML.

None of the findings above contain nearly enough data to draw any definitive conclusions, but I hope they pique your interest to find out more when we make Crossmark usage statistics available to all members later this year. In the meantime if you have any suggestions/questions, or would be interested in helping us when we come to testing the statistics interface, please let me know.

PIDs for conferences - your comments are welcome!

Aliaksandr Birukou — Thu, 19 Apr 2018 00:00:00 +0000

Aliaksandr Birukou is the Executive Editor for Computer Science at Springer Nature and is chair of the Group that has been working to establish a persistent identifier system and registry for scholarly conferences. Here Alex provides some background to the work and asks for input from the community:

Roughly one year ago, Crossref and DataCite started a working group on conference and project identifiers. With this blog post, we would like to share the specification of conference metadata and Crossmark for proceedings and are inviting the broader community to comment.

Why are conferences important?

One common misbelief is that most published research appears in journals. However, next to new ways of communication research results (blogs, presentations,…) and journals there are also other publication options, like books, very important in humanities, or conference proceedings, which are very important in computer science and a couple of related disciplines. Conference proceedings are collections of journal-like papers, often undergoing a more competitive peer review process than in journals. For instance, looking at original research in computer science in Scopus published in CS in 2012-2016, 63% of articles appeared in proceedings, while only 37% were published in journals. DBLP, one of the most important indexing services in CS, lists more than two million conference papers organized in ~5,400 conference series.

So, while it is true that CS has a significant share of conference proceedings, conferences are also relevant in many other disciplines which do not publish formal proceedings. For instance, inSPIRE contains ~23,000 conferences in high-energy physics, the American Society of Mechanical Engineers (ASME) publishes roughly 100 proceedings volumes annually.

Why do we need an open persistent ID for a conference or a conference series?

With publishers, learned societies, indexing services, libraries, conference management systems, research evaluation and funding agencies using conferences directly or indirectly in their daily work, a common vocabulary would simplify data processing, reporting and minimize errors. Right now, a publisher assigns a unique conference ID to the conference to be published, then an indexing service does it, then it is assigned in a library. Wouldn’t it be easier to do this at the very beginning of the process, when the conference planning starts, and keep the same identifier through the whole conference lifecycle?

The joint Crossref and DataCite group on conference and project identifiers has discussed this topic at half a dozen calls and various PID community meetings (PIDapalooza, FORCE conferences, AAHEP Information Provider Summit). The result of those discussions is a draft of the specification of conference metadata and Crossmark for proceedings.

The document first defines the concepts of a conference, conference series, joint and co-located conferences. It then introduces the information we want to store about those entities, e.g., the ID, name, acronym, other IDs, URL and the maintainer of the conference series, or the ID, conf series ID, number, dates, location, and URL for conferences. Such metadata can be submitted to Crossref and DataCite by conference organizers or publishers on their behalf and linked to the existing proceedings metadata, where appropriate. It can be then used for linking research outputs from a conference (beyond formal proceedings), recognizing reviewers via services such as ORCID and Publons, computing metrics of a conference series, conference disambiguation in indexing services and ratings (CORE, QUALIS, CCF), and so on.

The second part of the document introduces Crossmark for conference proceedings. Its goal is to structure and preserve the information about the peer review process of a conference as declared by the general or program chairs. Depending on how much information is available from the conference organizers, one can use the basic or extended versions of Crossmark.

In order to comment, please open the specification and leave comments using “comment” feature of Google Docs. The draft remains open for comments till the 31st of May 2018.

Next steps

After hearing from YOU, we will update the document to reflect the community comments. In parallel, we start a subgroup discussing the governance models, looking into whether we need a new membership category at Crossref, what fees should be covered, etc.

Crossmark 2.0 - grab the code and you’re ready to go!

Kirsty Meddings — Thu, 15 Sep 2016 00:00:00 +0000

On September 1st we completed the final stage of the Crossmark v2.0 release and sent an email to all participating publishers containing instructions for upgrading. The first phase of v2.0 happened when we changed the design and layout of the Crossmark box back in May of this year. That allowed us to better display the growing set of additional metadata that our members are depositing, and saw the introduction of the Linked Clinical Trials feature.

Now all publishers have the opportunity to complete the upgrade by simply replacing the Crossmark button and the piece of code that calls the box. The new button designs are, we think, a much better fit for most websites, and are designed to look more like a button than a flat logo. The new buttons are also available
as .eps files for placement in PDFs.

Crossmark box on a mobile phone

Most importantly, switching to 2.0 makes the Crossmark box responsive for better display on mobile devices.

Just two weeks after the code release a number of publishers have already upgraded and are running Crossmark 2.0 on their content. Congrats to the Pan African Medical Journal who were the first member to upgrade just a couple of days after the release. Of course we realise that many members will need time to schedule the upgrade, and while we are keen to see as many early adopters as possible, we will support version 1.5 of Crossmark through to the end of March 2017.

If your content is running Crossmark 2.0 we would love to see it. Drop us a line or put a link in the comments below.

Get ready for Crossmark 2.0!

Kirsty Meddings — Wed, 17 Aug 2016 00:00:00 +0000

TL;DR… In a few weeks, publishers can upgrade to the new and improved Crossmark 2.0 including a mobile-friendly pop-up box and new button. We will provide a new snippet of code for your landing pages, and we’ll support version v1.5 until March 2017.

We recently revealed a new look for the Crossmark box, bringing it up-to-date in design and offering extra space for more metadata. The new box pulls all of a publication’s Crossmark metadata into the same space, so readers no longer have to click between tabs. Linked Clinical Trials and author names (including ORCID iDs) now have their own sections alongside funding information and licenses. Feedback so far tells us that the new box is a vast improvement.

However, this was only phase one of the Crossmark makeover. We will soon complete the upgrade to display a fully responsive, mobile-friendly box. The Crossmark button has been given a facelift too, and we are excited to offer the first public preview today:

The new button brings the Crossmark icon up to date and is designed to be more “clickable” than the current button. It will be available in several different ratios and also in greyscale.

The first phase of the new design was rolled out in the existing Crossmark pop up window (Crossmark v1.5) without the need for changes within publisher systems. For the Crossmark v2.0 upgrade, publishers will need to update their landing pages with a new snippet of code, to ‘unlock’ the new button and functional enhancements.

Crossmark 2.0 will be available to adopt in a few weeks, and each publisher can decide when to switch over. We encourage members to upgrade sooner rather than later to get the benefits of the new box, but we also understand there are planned development schedules and the need for a testing period so we will continue to support Crossmark v1.5 until March 2017.

Many thanks to all of those who completed our surveys to help us shape the new button. And congratulations to Elizabeth Ramsey, a researcher from Trent University in Canada, who will be receiving a limited edition Crossref Moleskine notebook from the survey prize draw.

Our User Experience Designer, Rakesh Masih, will be blogging soon with details about the research and testing for this project, as well as more about our new approach to user experience at Crossref.

Linked Clinical Trials initiative gathers momentum

Kirsty Meddings — Tue, 21 Jun 2016 00:00:00 +0000

We now have linked clinical trials deposits coming in from five publishers: BioMedCentral, BMJ, Elsevier, National Institute for Health Research and PLOS. It’s still a relatively small pool of metadata - around 4000 DOIs with associated clinical trial numbers - but we’re delighted to see that “threads” of publications are already starting to form.

If you look at this article in The Lancet and click on the Crossmark button you will see that in the Clinical Trials section there are links to three other articles reporting on the same trial: two from the American Heart Journal and one from BMJ’s Heart. Readers can navigate between these four articles in three separate journals using the Crossmark functionality- a new set of links and routes for discovery have appeared.

In another example, three articles from PLOS ONE are threaded together around a trial for the treatment of Type 1 diabetes. And here another PLOS journal, Neglected Tropical Diseases links through to a PLOS ONE article about the same trial.

If you publish in the health sciences please do consider joining this exciting initiative so that we can expand these threads and build up the metadata. Read the tech specs here or drop me an email if you have questions.

Clinical trial data and articles linked for the first time

Daniel Shanahan — Tue, 17 May 2016 00:00:00 +0000

It’s here. After years of hard work and with a huge cast of characters involved, I am delighted to announce that you will now be able to instantly link to all published articles related to an individual clinical trial through the Crossmark dialogue box. Linked Clinical Trials are here!

In practice, this means that anyone reading an article will be able to pull a list of both clinical trials relating to that article and all other articles related to those clinical trials – be it the protocol, statistical analysis plan, results articles or others – all at the click of a button.

Linked Clinical Trials interface

Now I’m sure you’ll agree that this sounds nifty. It’s definitely a ‘nice-to-have’. But why was it worth all the effort? Well, simply put: “to move a mountain, you begin by carrying away the small stones”.

Science communication in its current form is an anachronism, or at the very least somewhat redundant.

You may have read about the ‘crisis in reproducibility’. Good science, at its heart, should be testable, falsifiable and reproducible, but an historical over-emphasis on results has led to a huge number of problems that seriously undermine the integrity of the scientific literature.

Issues such as publication bias, selective reporting of outcome and analyses, hypothesising after the results are known (HARKing) and p-hacking are widespread, and can seriously distort the literature base (unless anyone seriously considers Nicholas Cage to be causally related to people drowning in swimming pools).

This is, of course, nothing new. Calls for prospective registration of clinical trials date back to the 1980s and it is now becoming increasingly commonplace, recognising that the quality of research lies in the questions it asks and the methods it uses, not the results observed.

Uptake of trial registration year-on-year since 2000

Building on this, a number of journals and funders – starting with BioMed Central’s Trials over 10 years ago – have also pushed for the prospective publication of a study’s protocol and, more recently, statistical analysis plan. The idea that null and non-confirmatory results have value and should be published has also gained increasing support.

Over the last ten years, there has been a general trend towards increasing transparency. So what is the problem? Well, to borrow an analogy from Jeremy Grimshaw, co-Editor-in-Chief of Trials – we’ve gone from Miró to Pollock.

Although a results paper may reference a published study protocol, there is nothing to link that report to subsequent published articles; and no link from the protocol itself to the results article.

A single clinical trial can result in multiple publications: the study protocol and traditional results paper or papers, as well as commentaries, secondary analyses and, eventually, systematic reviews, among others, many published in different journals, years apart. This situation is further complicated by an ever-growing body of literature.

Researchers need access to all of these articles if they are to reliably evaluate bias or selective reporting in a research object, but – as any systematic reviewer can tell you – actually finding them all is like looking for a needle in a haystack. When you don’t know how many needles there are. With the haystack still growing.

That’s where we come in. The advent of trial registration means that there is a unique identifier associated with every clinical trial, at the study-level, rather than the article level. Building on this, the Linked Clinical Trials project set out to connect all articles relating to an individual trial together using its trial registration number (TRN).

By adapting the existing Crossmark standard, we have captured additional metadata about an article, namely the TRN and the trial registry, with this information then associated with the article’s DOI on publication. This means that you will be able to pull all articles related to an individual clinical trial from the Crossmark dialogue box on any relevant article.

This obviously has huge implications for the way science is reported and used. By quickly and easily linking to related published articles, it will enable editors, reviewers and researchers to evaluate any selective reporting in the study, and help to provide far greater context for the results.

As all the metadata will be open access (CC0), with no copyright, it will also be possible to access this article ‘thread’ through the Crossref Metadata Search, or independently through an application programming interface (API). This provides a platform for others to build on, with many already looking to take the next step, such as Ben Goldacre’s new Open Trials initiative.

However, in order for this to work, we must capture as many articles and trials as possible to create a truly comprehensive thread of publications. We currently have data from the NIHR Libraries, PLoS and, of course, BioMed Central, but need more publishers and journals to join us in depositing clinical trial metadata. After all, without metadata, this is all merely wishful thinking.

Let’s hope we’re the pebble that starts the landslide.

Linking clinical trials = enriched metadata and increased transparency

Kirsty Meddings — Mon, 18 Jan 2016 00:00:00 +0000

We will shortly be adding a new feature to Crossmark. In a section called “Clinical Trials” we will be using new metadata fields to link together all of the publications we know about that reference a particular clinical trial.

Most medical journals make clinical trial registration a prerequisite for publication. Trials should be registered with one of the fifteen WHO-approved public trial registries , or with clinicaltrials.gov which is run by the US National Library of Medicine. Once registered, a trial is assigned a clinical trial number (CTN) which is subsequently used to identify that trial in any publications that report on it.

Publications that result from any one trial are likely to be released in multiple journals from different publishers and at different times, for example secondary analyses coming some time after the publication of the initial results. Cross-publisher collaboration is paramount to linking all of these publications together so that researchers, funders, and regulatory agencies can understand the whole set of results from clinical trials. With this in mind, a group of medical publishers, led by BioMedCentral, approached Crossref to establish a working group, and here, they designed an approach to address this problem: “thread” all the various documents together surrounding a clinical trial.

Updated upstream

To implement threaded publications, publishers extract clinical trial numbers from papers, or ask authors to submit those numbers to them. Publishers add the CTNs to the Crossref DOI metadata via three new fields: clinical trial number, clinical trial registry where trial is registered, and trial stage (pre-results, results or post-results of the trial). Crossref has assigned unique IDs to each trial registry (much the same as we have done for funders in our Funder Registry and for the same reason - trial registry names and URIs can change over time and we need a persistent identifier). Using a combination of trial registry ID and clinical trial number, we can easily identify other content in the Crossref database that cites the same trial. Finally, Crossref displays the clinical trial metadata on the respective papers for all participating Crossmark publishers. Crossmark is a convenient place for readers to access the clinical trial information and is readily accessible directly from the journal article (online and PDF versions). And of course all of the data also goes into our open API so that anyone can make use of it.

The reporting of clinical trial results is notoriously inconsistent, something that the AllTrials initiative is also seeking to address. Publishers can help by collecting this information upstream and disseminating it using the existing Crossref infrastructure.

We ask all publishers to deposit the clinical trial data which is so critical to transparency in this area of research, and have already had the first data in from Crossref member the National Institute of Health Research. Once we launch the initial set of linked clinical trials, we will expand coverage of the threaded publications to include all content that reports on or references a clinical trial, from protocol to results to supporting data and systematic reviews.

Stay tuned and watch this space as threaded publications rolls out to journal articles across publishers!

A healthy infrastructure needs healthy funding data

Kirsty Meddings — Wed, 16 Dec 2015 00:00:00 +0000

We’ve been talking a lot about infrastructure here at Crossref, and how the metadata we gather and organize is the foundation for so many services - those we provide directly - and those services that use our APIs to access that metadata, such as Kudos and CHORUS, which in turn provide the wider world of researchers, administrators, and funders with tailored information and tools.

The initiative formerly known as FundRef

Together Crossref’s funding data (previously known as FundRef – we simplified the name) and the Open Funder Registry, our taxonomy of grant-giving organisations, comprise a hub for gathering and querying metadata related to the questions:

“Who funded this research?” and “Where has the research we funded been published?”

To support the funding data initiative, three key pieces of metadata are needed from publishers:

Funder ID
Funder Name
DOI

Unfortunately only around half of the 950,000 Crossref DOIs with funding data contain funder IDs, the unique funder identifiers from the Open Funder Registry that are needed to link up all of the data. So, only half of the data is useful. (And 950,000 DOIs is only a fraction of the 77 million DOIs in our database, but more on that later).

When we looked at the funding data that was coming in without funder IDs we were a little surprised. We had expected that most of these would be names that simply aren’t in the Open Funder Registry yet, and we thought there would be a certain amount of incorrect information that had been entered into the “funder_name” field.

Instead, what we found was that many of the names were correct, and the funder IDs were just missing.

Tidying the data

To help correct this, we decided to match incoming names to funder IDs where we could do so with the highest level of confidence. After much testing to minimize false positives, we switched this on at the end of August 2015.

Throughout September and October, we inserted funder IDs for about 25% of the names that have been deposited without IDs. For October, the real numbers were 68,000 funder names with no IDs deposited, and 18,000 funder IDs inserted by Crossref.

In the same period 42,000 funder IDs were deposited by publishers. With our matching on top of this, we are achieving a little over a 50% overall success rate of “good” funding data (funder names and funder IDs together).

We have been very careful to distinguish the funder IDs that we have added from those deposited by publishers - provenance of data is an extremely important part of what we do. All funder IDs are tagged as provided either by the publisher or Crossref. Every time we insert an ID into a deposit, the publisher is notified in the deposit report.

We have also now added these tags to our REST API so that publishers can query to find out exactly which DOIs we have amended*. The ideal scenario at this point is that the publisher checks that they are happy with the matching and then redeposits the funding data for those DOIs, over-writing the <span >doi-asserted-by: “crossref”</span> tag and claiming the metadata as their own.

Setting some limits

The second largest problem with funding data was incorrectly entered funder name – e.g. concatenation of several names or authors entering overly long or vague program names instead of the official funder name.

To help weed this out, we have made a couple of changes to the funding data deposit system:

Funder_name field can no longer contain a numerical string over 4 digits
Funder_name field can no longer contain a text string over 200 characters
Funder names that that do not adhere to these two rules will now cause the funding data section of the metadata deposit (not the whole deposit) to fail and return an error message.

Getting the growth we need

As of today, 198 publishers deposit funding data with Crossref. This amounts to about 3.5% of Crossref’s membership (although it’s a larger proportion of our total deposits). We need more publishers to deposit funding data so that funding data search can become a truly useful tool for the community. There’s no sign-up process or additional fee - read about how to get started, and take a look at our best practices for depositing funding data.

Finally, we ask you: how can we get more and better funder metadata in 2016?

This is not a rhetorical question. Please tweet your thoughts @CrossrefOrg or email your replies to info@crossref.org. You will receive something special via snail mail if you reply to us – just Crossref’s way of saying thank you.

^{*At the time of posting our database is re-indexing and the “asserted-by” tags are still filtering through to the API. Check back in a day or two for the full picture.}

Best Practices for Depositing Funding Data

Kirsty Meddings — Tue, 01 Sep 2015 00:00:00 +0000

Crossref’s funding data initiative (FundRef) encourages publishers to deposit information about the funding sources of authors’ research as acknowledged in their papers. The funding data comprises funder name and identifier, and grant number or numbers. Funding data can be deposited on its own or with the rest of the metadata for an item of content.

There are two ways that publishers can collect this funding information for any given piece of content: by asking authors to input the funder name(s) and award number(s) via their submission system, or extracting the funder names and award numbers from the acknowledgements in the paper.

The funding data is only useful if it is standardised, and so it is absolutely critical that funder names are deposited with their associated funder IDs from the Funder Registry.

For publishers considering or about to start collecting and depositing funding data, and for those already doing so, we have drawn up some guidelines that will help you to ensure good quality metadata.

If you are collecting funding information from authors via your submission system:

Provide very clear instructions for your authors. Your submission system should prompt the author towards the canonical name from Crossref’s Funder Registry as they type, or guide them through a pick-list. Make it clear to authors that they should choose funder names from this list and not copy and paste from their manuscript.
Work with your submission system vendor or adapt your in-house system to make it easy for authors to select from the Funder Registry, and more difficult to paste incorrect names or ignore the suggested names. Consider a warning message if an unknown name is entered, and offer a list of close matches.
Instruct authors to look for the name of the funding body rather than a specific program or project.

If you or one of your vendors is extracting funding information from papers:

Provide the same clear instructions to your vendor(s). Stress the importance of matching the funder names in the acknowledgements to the names in the Funder Registry.
Look for common text-extraction errors such as concatenated funder names, punctuation errors, and stop words such as “of/for” that are commonly used interchangeably, or the presence or absence of “the” at the start of a funder name.

For both workflows:

Add QA into your workflow. Many of the names sent to Crossref without IDs are very obviously funders that are in the Registry, and a check by editorial or production staff could correct misspellings or fill in blanks. Check that grant numbers have been separated and are not being deposited as one long string.
Be aware that funder names deposited without IDs are not valid funding data and will be hidden from Crossref’s search tools and APIs until such time as they are updated with a funder ID.
The funding data section of a deposit (but not the rest of the deposit) will be rejected by the Crossref deposit system if
- The funder_name field contains a numerical string longer than 4 digits
- The funder_id field contains a number that is not an ID from the Funder Registry
- The funder_name contains text that exceeds 200 characters
Consider only depositing data that has funder IDs and holding the rest to re-poll against the Funder Registry at a later date when more funder names have been added. The Funder Registry is updated at approximately two-monthly intervals. You can sign up to be alerted to updates here.
If there are funders that appear regularly in your particular subject or geographical area that are not in the Registry, send a list to funder.registry@crossref.org.

♫ Researchers just wanna have funds ♫

Geoffrey Bilder — Thu, 10 Apr 2014 00:00:00 +0000

photo credit

Summary

You can use a new Crossref API to query all sorts of interesting things about who funded the research behind the content Crossref members publish.

Background

Back in May 2013 we launched Crossref’s FundRef service. It can be summarized like this:

Crossref keeps and manages a canonical list of Funder Names (ephemeral) and associated identifiers (persistent).
We encourage our members (or anybody, really- the list is available under A CC-Zero license waiver) to use this list for collecting information on who funded the research behind the content that our members publish.
We then ask that our members deposit this data in their normal Crossref metadata deposits.

And that was cool.

But then people started asking us awkward questions. Questions like “what can I do with the funder data?” and “how do I query it?”.

Stoopit people.

Can’t you just let us bask for a few minutes in the sunny glow of actually conceiving of and launching a project within a year?

But seriously, funders, were interested to see how they could use the funder metadata being collected in Crossref. In particular, some funding agencies were interested in being able to measure Key Performance Indicators (“KPIs” to management wonks) related to recent mandates such as the February 22nd 2013 OSTP memo, Public Access to the Results of Federally Funded Research. Two groups also approached us, CHORUS and SHARE. Both are interested in exploring how to build reporting tools for funders, institutions and researchers and each brought us a gigantic hairball of use-cases they were hoping we would be able to meet.

Conveniently, we were in the process of creating a revised, modern Crossref API that is entirely buzzword-compliant, and so we set to work…

We thought people might be interested in seeing what you can do with the Crossref REST API in relation to funding information and the expectations that are increasingly being attached to them. CHORUS is already using the Crossref REST API heavily and we expect that SHARE will soon start making use of it as well. The feedback from both groups has been very useful, but we are looking for broader feedback as well. The API is still in development, so now is your chance to help us shape it.

Brief Examples

Please note, the following are APIs calls, although you can copy and paste the URIs into your browser, the data is returned in a machine readable representation called JSON. If you want the results to look a little more presentable, we advise you install the JSONView plugin:

Firefox Users: JSONView
Chrome Users: JSONView

Also note that publishers have only just started to deposit the metadata needed for these APIs to work, so the data is currently sparse. We know that many of our members are working feverishly to populate more of the needed metadata, but this requires updates to the their manuscript tracking systems, production systems and hosting systems. It takes time.

But for now you can paste the relevant URIs below into your browser and see the results that we do have. Expect these numbers to increase sharply over the next few months

To start with, you might want to know how many articles in Crossref have FundRef metadata:

https://api-crossref-org.pluma.sjfc.edu/v1/works?filter=has-funder:true&rows=0

You could then be interested in knowing how many works in Crossref use FundRef to credit the United States’ National Science Foundation (NSF) for funding their research? First you need to find out what the FundRef identifier is for the NSF:

https://api-crossref-org.pluma.sjfc.edu/v1/funders?query=NSF

You can see that there are several entries that match “NSF”, and that the one we are looking for has the identifier http://dx.doi.org.pluma.sjfc.edu/10.13039/100000001. Remember, funding agency names can change frequently, the ID provides a persistent link to the funder even if their name changes.

If you are curious, you can see the details for the NSF entry, including its location, parent and child organisations:

https://api-crossref-org.pluma.sjfc.edu/v1/funders/10.13039/100000001

Notice that the results also lists the work-count. This is the number of works in the Crossref metadata that list the US NSF as having funded the research.

So perhaps you would like to see the list of works. The following will list the first twenty:

https://api-crossref-org.pluma.sjfc.edu/v1/funders/10.13039/100000001/works

You can page through the results with the offset argument:

https://api-crossref-org.pluma.sjfc.edu/v1/funders/10.13039/100000001/works?offset=20
https://api-crossref-org.pluma.sjfc.edu/v1/funders/10.13039/100000001/works?offset=40
...

How many works that have listed the NSF as a funder have license information:

https://api-crossref-org.pluma.sjfc.edu/v1/funders/10.13039/100000001/works?filter=has-license:true&rows=0

Lets see the first batch that have license information:

https://api-crossref-org.pluma.sjfc.edu/v1/funders/10.13039/100000001/works?filter=has-license:true

Lets look at the metadata for one of the DOIs returned:

https://api-crossref-org.pluma.sjfc.edu/v1/works/10.1063/1.3593378

Interesting, the metadata shows an article published by AIP. It includes license information (CC-BY 3.0) as well as a link to the full text. If you follow the link to the full text, you can retrieve it:

http://link.aip.org.pluma.sjfc.edu/link/applab/v98/i21/p216101/pdf/CHORUS

Wow- A pretty short article. But you can see that it does credit the NSF and that the award number recorded in the text is the same as the award number recorded in the FundRef section of the Crossref metadata. Yay.

You can see in the brief examples above that there is a lot of other metadata you may want to query on and explore. It can include ORCIDS, information about archiving arrangements- even abstracts. It all depends on what the Crossref member has decided to provide.

You can get a simple overview of what a Crossref member has provided by looking at a member summary. Here is an example for Hindawi:

https://api-crossref-org.pluma.sjfc.edu/v1/members?query=hindawi

Note again that names are fickle, so the above query can also be accomplished using the member identifier like this:

https://api-crossref-org.pluma.sjfc.edu/v1/members/98

Groovy init?

If you want more pointers on where you can learn how to use the API, read on…

The Future of the Crossref REST API.

Our aim is for the Crossref REST API to go into production this Summer (2014). As with most of our newer APIs, there will be a free API for public use and a paid for API for professional use. The only difference between the two will be that the professional version will come with a service level agreement (SLA) covering uptime, response time and support. Naturally, this also means that the professional one will be on dedicated hosting equipment so that we can meet these SLAs, whereas the performance of the free version will be subject to the vicissitudes inherent in using a shared, constrained resource (i.e. the server and network it is running on).

Again, the basics of the API are in place. It should be fairly stable, but we do reserve the right to make changes to it over the next few months. Please send us feedback.

— The Weasel