2024 on Crossref

A progress update and a renewed commitment to community

Ginny Hendricks — Thu, 12 Dec 2024 00:00:00 +0000

Looking back over 2024, we wanted to reflect on where we are in meeting our goals, and report on the progress and plans that affect you - our community of 21,000 organisational members as well as the vast number of research initiatives and scientific bodies that rely on Crossref metadata.

In this post, we will give an update on our roadmap, including what is completed, underway, and up next, and a bit about what’s paused and why. We’ll describe how we have been making resourcing and prioritisation decisions, including a revised management structure, and introduce new cross-functional program groups to collectively take the work forward more effectively.

It’s important to acknowledge that Crossref has evolved significantly from just five years ago - our member count has more than doubled from 10,000 to 21,000 organisations since 2019 and they include all kinds of organisations such as funders, universities, government bodies, NGOs, and of course scholar- and library-led publishers. The smaller organisations now collectively contribute the majority of Crossref funding. We’ve gone from 100 million records to 160 million in five years, and our metadata is retrieved more than 2 billion times monthly, quadrupling what it was five years ago.

It’s within this context that we’ve spent quite a lot of time thinking about scalability, how we collect and process feedback and contributions from many organisations, how to automate our operations, and refining the plans for the next few years.

Our strategic agenda remains the same

A few times a year we update the strategy page where there is a quadrant of projects showing what’s completed, in progress, up next, and in planning/ideas - for each strategic theme. We also link from there to our live public roadmap which shows more specifics about individual projects, including projected timelines, and is updated more frequently.

If you’ve been watching the strategy page, checking in on the public roadmap or this blog, or joining webinars and annual meetings, you’ll know that we’ve had some longstanding plans to—among other things—reduce technical debt, rebuild our metadata management system, move to the cloud, modernise our schema, support multiple languages, and partner with multiple data sources to build the Research Nexus.

You’ve heard us talk about these initiatives a lot, but you’ve not seen particularly swift action.

Moving the work forward more effectively

Earlier this year, it became clear that our almost three-year project to build a new relationships API had not worked out. The project, dubbed ‘manifold’, was to initially deliver data citations, and eventually replace our central metadata system, but what was prototyped didn’t scale, even with a subset of our metadata. We weren’t confident enough about the project’s timeline or costs to justifiably continue investing further time and resources.

Meanwhile, we’d barely scratched the surface of our aim to pay down technical and operational debt, and we’d also been neglecting to keep the live system up to date with the numerous metadata changes that have been queued up, waiting to be implemented.

We knew the manifold project was ambitious – our system has grown in complexity over the years. We were trying to rebuild the car while driving it (our system needed to continue to operate and be maintained by our team) while trying to design a new approach to manage the many relationships between 160+ million database records. In the years we worked on this project, we learned a lot that will inform future plans for a large system redesign.

In March this year, we decided to pause the manifold project. We apologised to our community partners for not delivering the promised data<->literature matches they hoped to use. They were frustrated but thankfully understanding.

We then resolved to focus on backend infrastructural changes, conduct cross-training so that all of our staff would become familiar with current in-use systems instead of greenfield tech (for now), and start to make a dent in the backlog of bugs and long-promised schema updates in our mainstream services.

We’re happy to report some movement on these things and some milestones that have been achieved in these areas in recent months.

Fostering a happy and dedicated team

Any kind of work can only happen when our staff are in a good place, feeling supported and comfortable to question things, and well-equipped with information, purpose, and clear priorities. In June, when the whole staff met up in person, we had some really good conversations about culture, communication, and about sharing responsibilities. Some people ran birds-of-a-feather sessions to explore the issues that had been keeping them up at night, such as authentication/security, and rebuilding the Crossref System (CS), and the team also co-created a set of prioritisation drivers that are now in use within our roadmap and planning processes.

Taking on feedback from the all-staff meeting and then the July board meeting, we thought strategically about the organisational structure Crossref would need over the next few years to reflect the growth in scope and size, and fulfil its longer term goals. We have long had an ambitious agenda but realised we didn’t yet have the capacity to do it all. So we came to the conclusion that we needed an updated team and management structure to take us through the next phase of our development.

The structural changes were concluded at the end of November. They included:

Moving Technology under Operations, since Technology—though a vital enabler—still works in service to our mission and in support of our community, just like other operational things like board governance and finance.
Reframing product development as Programs and Services, and reducing our workstreams from five product portfolios to three programs. We formed cross-team steering groups around clearly articulated program areas (more on those below).
Broadening the leadership to include an Executive team and an extended Director team, and forming a Senior Management Team (SMT). These changes ensure that the collective responsibility for Crossref now rests on a wider group of experts who can back each other up and share the risk and the knowledge, rather than on just a few individuals.
We started recruiting for directors for two new leadership positions. We’ll welcome a new Director of Programs and Services and a new Director of Technology in the new year.
Evolving the strategic initiatives team into a data science team, integrating research & development functions throughout all teams and with the SMT taking collective responsibility for strategic initiatives.

Unfortunately, with the shift in approach for product development and by sharing responsibility for strategic initiatives and research among the wider team, we made the difficult decision that four positions would no longer work within the new structure.

A new approach: joined-up initiatives and cross-functional programs

Research has always been an important role for Crossref, but as this function had been annexed from our regular work, it became hard to coordinate strategic initiatives across the wider organisation. In recent years we inadvertently created more technical debt for ourselves, i.e., built multiple prototype tools without plans for adoption or moving them into production. Strategic initiatives, by their nature, need thorough research and high-level alignment, so we made such initiatives—things like Resourcing Crossref for Future Sustainability (RCFS) and improving the Integrity of the Scholarly record (ISR)—the responsibility of the whole senior management team.

Some useful research had been conducted, but we were never in a position to act on any of it. Particularly promising work has been in the field of metadata matching, and with the growth in the community reliance on our metadata, and attention on data quality rightly increasing, we decided to create a new data science team to be dedicated to this work, led by Dominika Tkaczyk.

We had also struggled with a traditional product management approach since all our tools and activities are interconnected, and we found we were trying to do too many things at once but not all of them very effectively. We also acknowledged that product management comes from the commercial e.g. retail world and therefore is designed to help companies sell/upsell, which is not our goal. So we looked to other approaches more suitable to mission-based nonprofits.

Introducing three programs

We have introduced cross-functional program management in order to work towards the following:

better cross-team alignment
shared responsibility
improve communication and learning
make more progress on the things members need.

Supporting the strategic theme of co-creation, a new program, facilitated by Program Lead Lena Stoll, now manages and oversees all activities around co-creation and community trends. A cross-team steering group just began meeting regularly and will be responsible for interfaces such as reports/dashboards, record registration interfaces, connections and collaborations such as Open Funder Registry, ROR, ORCID auto-update, as well as OJS and other partner integrations. This program also includes the Crossref website and any front-end things to support other programs. And it includes ISR (the integrity of the scholarly record) and our tools in this area such as Crossmark and retraction/correction tooling, and Similarity Check for text comparisons.

Supporting the strategic theme of complete and global metadata and relationships, a new program, facilitated by Program Lead Martyn Rittman, now manages and oversees all activities relating to contributing to the Research Nexus. Working particularly closely with the metadata team, led by Patricia Feeney, this program addresses how metadata is modelled, used, enriched, and extended. Work includes our APIs, incorporating external data sources like Retraction Watch and Event Data, building out metadata matching services with the new data science team, supporting the community of metadata users with API sprints and more modern options for retrieving metadata based on usage and need.

Supporting the strategic theme of open and sustainable operations and keeping to the POSI framework, a new program, facilitated by Program Lead Sara Bowman, now manages and oversees all activities relating to making our operations more open, transparent, and sustainable. This program focuses on supporting and strengthening the core functions our members rely on and enabling future growth. It includes metadata deposit and processing, most apps for e.g. managing titles, authentication, and architectural and infrastructural projects like moving from the data centre to the AWS cloud service. This program also includes modernising our operations in general, which is not just technology but also finance and human resources, so projects like membership process automation, fee modelling and financial analyses, and business system integrations.

The Programs will start to be reflected across our website and in our communications from next year.

What are Crossref’s new prioritisation drivers?

These are the drivers that our ~40 staff co-created in June that are guiding decisions about the priorities on our roadmap. New ideas will be evaluated in the following areas:

Encourage participation from new or under-represented communities
Respond to and lead trends in scholarly communications
Benefit the greatest number of members and users
Reflect on how the community works with each other and allow members to self-serve
Expand to support and connect relevant resource types and metadata fields
Make it easier to create and update metadata
Enhance metadata for completeness and accuracy
Make it easier to retrieve and use metadata
Automate repetitive/manual tasks
Address technical and operational debt
Maintain critical systems and operations and ensure their security
Control or reduce costs - to Crossref, our community, or the environment

We’re happy to report that the changes made this year have resulted in a productive last few months of the year. As reported in our annual meeting, here is the progress update.

What’s paused

A relationships API endpoint and, therefore, a specific data citation feed
Manifold, the three-year effort to modernise our tech stack
Most of the strategic initiatives prototypes that can’t yet be scaled, such as Labs API and Labs reports

What’s recently completed

We succeeded in moving the entire Crossref corpus to an open-source database, PostgreSQL
Fixed numerous REST API data quality issues and lots of troublesome bugs
Schema development - support for ROR as a Funder identifier is live and currently in testing
We automated some very manual membership and billing processes, saving hundreds of staff hours a year
Released a new form for journal article record registration, building on the grant registration form
Upgraded Participation Reports to include Affiliations and ROR IDs
Launched a new API Learning Hub

Since the rest of the community stops for no Crossref product roadmap issue, we also progressed a number of community and governance initiatives:

The Grant Linking System (GLS) reached 5 years with over 40 funders joining Crossref and registering over 130,000 grants and awards, including use of facilities and projects
Our research for Resourcing Crossref for Future Sustainability (RCFS) with the Membership & Fees Committee is going well, and we’ll have new fee proposals for review in 2025
The integrity of the Scholarly Record (ISR) conversations have deepened, and we’ve formed strong relationships with editorial experts and research integrity sleuths, who are getting up to speed on our metadata, and we’re working with some sleuthing consultants to change our processes to handle deceptive member behaviour such as paper mills, cloned journals, and citation manipulation. The new data science team plays a role here, along with membership and governance.

What’s currently in focus

In our efforts to do less but do it more effectively, we have two current priorities:

Get out of the physical data centre and into the cloud.
Develop Schema 5.4.

These two projects are underway, involving lots of communication and learning. Since we haven’t released any schema updates in many years, all our staff are learning for the first time how a metadata schema model is interpreted in a systemic way, learning about the structure of research objects, and honing the process as they go. We’ve high hopes we’ll be in a position to release continuous metadata schema versions and catch up on the backlog over the coming years.

What’s next

Continuous metadata development, with contributor roles up next
Retraction Watch data integrated into the REST API so users have a single source of retraction/correction data
Upgraded preprint matching and notifications
Modelling more equitable fees through the RCFS projects
Piloting a non-voting membership category

Once we’re fully in the cloud and in the groove of metadata updates, and with the support of newly-hired technology and program directors joining in the new year, we’ll turn our attention to rebuilding the central metadata system that we call the Crossref System, or “CS” and report more on this next year.

So that was our summary of 2024 and an indication of what’s coming in 2025 and beyond; sorry it’s so long, and thanks for reading this far! Next year we’ll get back to more regular updates as the strategic agenda and the programs progress.

A summary of our Annual Meeting

Rosa Morais Clark — Mon, 09 Dec 2024 00:00:00 +0000

The Crossref2024 annual meeting gathered our community for a packed agenda of updates, demos, and lively discussions on advancing our shared goals. The day was filled with insights and energy, from practical demos of Crossref’s latest API features to community reflections on the Research Nexus initiative and the Board elections.

Our Board elections are always the focal point of the Annual Meeting. We want to start reflecting on the day by congratulating our newly elected board members: Katharina Rieck from Austrian Science Fund (FWF), Lisa Schiff from California Digital Library, Aaron Wood from American Psychological Association, and Amanda Ward from Taylor and Francis, who will officially join (and re-join) in January 2025. Their diverse expertise and perspectives will undoubtedly bring fresh insights to Crossref’s ongoing mission.

The meeting started with a recap of our mission and priorities. Ed Pentz reiterated the Research Nexus vision of increasing transparency of the connections that make up the scholarly record and underpin the research ecosystem.

Crossref is dedicated to openness, community ownership, and a stable, accessible infrastructure that researchers, publishers, funders, and institutions can rely on for the long term. This is demonstrated by Crossref’s commitment to the the Principles of Open Scholarly Infrastructure (POSI), which constitute commitments to building a resilient and transparent infrastructure for research—sustainability, community governance, and openness. Ed emphasized how Crossref is aligning with these principles and collaborates with other adopters to reflect and continuously align these with the needs of the scholarly community, with a public consultation on proposed revisions to POSI forthcoming next year.

Ginny Hendricks highlighted key membership and metadata trends. She noted that as of 2024, half of Crossref members are based in Asia. This year, as always in recent years, we saw many new organisations from Indonesia, Turkey, India, and Brazil join us. Removing those fast-growing countries for the chart’s clarity, we can see that some of the next most active countries are Pakistan, Mexico, Spain, Bangladesh, and Ecuador, among others.

There are now ~163 million open metadata records with Crossref DOIs, and Ginny pointed out increases in the registration of preprints, peer-review reports, and grants. In terms of metadata elements, it’s good to see that more publishers recognize the importance of including abstracts and ROR IDs in their metadata records. Also, in line with the community’s concerns about integrity, our members have been enriching their records with direct assertions of retractions.

Then, Ginny went on to report on the progress towards our strategic goals:

Contribute to an environment where the community identifies and co-creates solutions for broad benefit
A sustainable source of complete, open, and global scholarly metadata and relationships
Manage Crossref openly and sustainably, modernizing and making transparent all operations so that we are accountable to the communities that govern us.
Foster a strong team because reliable infrastructure needs committed people who contribute to and realize the vision and thrive in doing it.

Demos

Lena Stoll and Patrick Vale’s session gave members a practical preview of our latest tools.

Patrick started by reflecting on the challenge of making our identifiers useful for people using screen readers (and other assistive technologies). He thanked all who responded to our past consultation on the topic and presented the Crossref DOI Accessibility Enhancer – the browser plug-in initially available for Firefox (and soon also for Chrome). He shared the Gitlab repo for anyone interested in trying it and invited feedback as we’re hoping to iterate on this.

Patrick then went on to talk about our openness to community contributions to Crossref tools, with an example of the recent contribution from CWTS Leiden to our Participation Reports. Thanks to their work, our members can now see the proportion of works they’ve registered that include affiliation information and ROR IDs, alongside the previously available key metadata such as references, abstracts, ORCID iDs, funding information, or Crossmark.

Finally, Lena demonstrated the latest extension of our record management tool that’s just been made available to make manual registration of metadata records for journal articles easier. The new form is flexible and driven by our metadata schema. Importantly for our members, it simplifies the workflow with input validations and automated ISSN matching, and it enables members to register author affiliations with an integrated ROR look-up. We hope this will support our smaller members, who are relying on our helper tools to register their content.

Throughout the session, members were encouraged to use these tools and explore new resources available through Crossref. We believe that by taking advantage of these resources, you can enhance your research and publishing experience, and contribute to the growth and development of the scholarly community.

The discussion about open scholarly infrastructure

The panel on open scholarly infrastructure brought together experts with a wide range of experience in the field. Moderated by Lucy Ofiesh, Crossref’s Chief Operating Officer, the discussion featured six invited speakers who shared their insights on the opportunities and challenges facing the scholarly ecosystem: Ed Pentz, Crossref; Sarah Lippincott, Dryad; Amélie Church, Sorbonne University; Joanna Ball, DOAJ; Ann Li, Airiti; and Richard Bruce Lamptey, Kwame Nkrumah University of Science and Technology.

The panel talked about what openness in scholarly infrastructure means, why it’s important, its sustainability, and how to tackle challenges and gaps across the ecosystem. They highlighted frameworks like the Principles of Open Scholarly Infrastructure (POSI), the Barcelona Declaration, and the FOREST Framework as key tools for guiding work on governance, sustainability, and equity. The discussion highlighted the need for more collaboration, inclusivity, and practical ways to ensure open infrastructure remains sustainable in the long run.

They also stressed how openness supports research integrity. How transparent systems allow researchers to question methods, verify findings, and preserve data. Amelie Church expanded on this point, underscoring the important role of open infrastructure in addressing challenges to integrity. She explained that such transparency enables the scholarly community to scrutinize research processes, ensuring the quality of outputs and their impact on society. Without openness, researchers face barriers to maintaining trust in their work, making open infrastructure necessary for research integrity and public confidence in science.

“By focusing on accessibility, transparency, and community engagement, open infrastructure can reshape academic and research ecosystems in transformative ways.” ~Richard Bruce Lamptey

Regarding sustainability, Sarah Lippincott stressed the importance of aligning funding models with community needs while addressing governance challenges. She pointed out that while initial funding can launch infrastructure, long-term sustainability requires consistent community investment and robust governance frameworks. This balance, she explained, is essential to ensure equity and transparency.

Collaboration was another important topic. Joanna Ball and Sarah Lippincott shared examples of how pooling expertise and resources—such as in the global support for ROR—can strengthen systems and make them more sustainable. These initiatives show the power of collective efforts in addressing technical and resource barriers. However, inclusivity remains an ongoing challenge.

The panel discussed the ways in which language barriers, resource limitations, and reliance on proprietary systems continue to exclude researchers from underrepresented regions. Ann Li highlighted how addressing these disparities is critical to ensuring the global accessibility of open infrastructure. By fostering inclusive practices, the scholarly community can mitigate biases and build tools that reflect a broader range of research contributions.

”My hope is that open infrastructure can have the resources that it needs to thrive, not just merely survive, and also that open infrastructure communities and organisations look to the value of frameworks that we’ve talked about today to help align themselves and improve their policies and practices, because there’s always room for growth, even in the best, most well-intentioned communities.” ~Sarah Lippincott, Dryad

The panel wrapped up the discussion by expressing optimism for the future of open scholarly infrastructure and emphasized the importance of continued investment, collaboration across organisations, and transparency in operations. The discussion reinforced the idea that open infrastructure provides a strong foundation for research that is equitable, sustainable, and accessible to all.

Updates from our Community

We enjoyed talks from our community about increasing their participation in the Research Nexus by adopting, using and enhancing metadata in different ways. Robbykha Rosalien hosted talks from the EuropePMC, Dutch Research Council, eLife, and CSIRO featured in Session I, and Amanda French hosted CLOCKSS, Sciety, and Redalyc in Session II.

Michael Parkin talked about preprints in Europe PMC. Europe PMC is a database for life science literature and a platform for content-based innovation. They started indexing preprints via Crossref REST API in 2018. Michael presented their work on discoverability of preprints in their database, including reflections on early challenges, as well as the latest efforts in surfacing available community reviews.

Hans de Jonge talked about the Dutch Research Council’s (NWO) dedication to open science, with policies ensuring that publications and data funded by NWO are openly available. They embrace open science principles for their own metadata and is a signatory of the Barcelona Declaration on Open Research Information. Hans focused on NWO’s recent introduction of Grant IDs through Crossref’s Grant Linking System (GLS). He shared their approach, the motivations behind introducing Grant IDs, and some challenges they faced.

Frederick Atherden explained how eLife, a nonprofit led by scientists, use Crossref’s Grant Linking System to include grant DOIs in their publication metadata. It allows authors to add grant DOIs during submission, and they developed a tool to match grant numbers with DOIs during the proofing process to improve accuracy. Their goal is to follow best practices for metadata, making content easier to find and link to.

Brietta Pike covered how CSIRO is working to improve metadata quality for its journals, making research more discoverable and trustworthy. CSIRO faced challenges like inconsistent XML tagging, outdated systems, and data loss. To address these, they formed a project team, created a clear XML stylesheet, and updated their workflows. Recent progress includes better funding data, clearer license information, and more complete affiliation tagging. These efforts aim to support a more transparent and accessible research environment.

Alicia Wise of CLOCKSS talked about recent collaborations seeking to safeguard our cultural and scholarly heritage over the long term. CLOCKSS, a community-run archive, is dedicated to preserving scholarly content to remain accessible and unchanged for future generations. True preservation requires securely storing content in trusted archives that are actively maintained. A group of librarians and publishers developed a guide to help publishers preserve content, they also established an archival standard for EPUB formats to ensure ebooks can be stored effectively, and launched a pilot project to track preserved books, helping libraries and scholars identify safely stored titles.

Mark Williams from Sciety talked about how Sciety uses Crossref metadata to create detailed preprint histories. By partnering with organisations and communities worldwide, Sciety platform gathers public reviews, highlights, and recommendations on preprinted research, helping researchers evaluate the quality and relevance of new studies. Through linking related preprints and journal articles, Sciety builds a connected view of each research work. Although challenges like inconsistent terminology and identifier gaps persist, these efforts enhance the visibility and credibility of preprints.

Arianna Becerril-García of AmeliCA/Redalyc shared insights on diamond open-access journals in Latin America. Redalyc is an open-access infrastructure that supports journals by providing free services like visibility and production tools. Redalyc has a role in sustaining Latin America’s unique approach to open-access publishing, where most journals are backed by academic institutions and public funds, allowing free access for both readers and authors. Arianna stressed the need to treat these journals as digital public goods and urged the communities they serve to help ensure their long-term sustainability. Despite limited resources and global under-recognition, these journals serve an international research audience, including authors from Europe, Africa, and Asia. Redalyc and other open infrastructures play a key role by offering tools that reduce production co-sts and improve discoverability, all without financial barriers. Noted was how this approach aligns with UNESCO’s open science framework, which promotes inclusivity and addresses long-standing inequalities in scholarly publishing.

Afternoon of more resources and updates from Crossref

After a mid-day break (in Europe), Luis Montilla kicked off the second session with a practical tutorial of Crossref’s REST API. Following his last year’s intro to the Crossref API, this time he offered a step-by-step guide to help attendees maximize the API’s capabilities for metadata retrieval with advice on:

Managing large data requests with pagination and iterations
Incorporating safety mechanisms - to avoid hitting rate limits, Luis recommended adding pauses between requests and sharing example scripts to streamline this.

For those interested in learning more, look at the new Crossref API Learning Hub— a new resource offering guides, scripts, and training materials to simplify complex queries. Please share questions about things you’re not sure about in our community forum, to help guide development of future demos.

Patricia Feeney followed with updates on metadata schema changes. She introduced our recent shift to integrate the Funder Registry with ROR, which allows members to use a single identifier system, simplifying data management by reducing redundancy. Patricia explained that, for now, the current identifiers remain valid, so members won’t need to make immediate changes. She also outlined planned support for version metadata, typed citations, and future plans to expand support for contributor role vocabularies, and invited community participation in a planned multilingual metadata working group.

Next, Kora Korzec offered an update on the progress in our research on Resourcing Crossref for Future Sustainability and opened up a discussion about the best ways of assessing our members’ size and ability to pay. In light of our ambition to streamline discounts, we also invited suggestions for discounts to support accessibility and fuller participation in the Research Nexus.

As part of the discussion, we’ve learned who was in attendance during the session:

We’ve heard a lot of support for our current GEM program. While it was clear from our poll that publishing revenue is not the most relevant measure of size or capacity for all those present – establishing a good alternative proved challenging. The idea of considering the size of the organisation as its largest entity has been discussed, and important points were raised about budgets in different types of distributed organisations (e.g., on the position of libraries within large universities).

The official Annual Meeting part commenced after the discussion, with a report on the State of Crossref from Lucy Ofiesh, and commenced with our Board election. Lucy highlighted some of the key accomplishments of the year so far, including:

Research for Resourcing Crossref for Future Sustainability (RCFS)
Integrity of the Scholarly Record (ISR)
Grant Linking System (GLS) reached 5 years
Automated some very manual membership processes
Released new form for journal article record registration
Upgraded Participation Reports to include Affiliations and ROR IDs
Launched a new API Learning Hub
Paused further development of a Relationships API
Migrated to a new open-source database
Schema development - ROR as Funder identifiers
REST API bug fixes and metadata consistency fixes.

Then she reflected on the membership growth––Crossref is now made up of 21,000 organisations from 160 countries. We reviewed our 2024 year-end financial forecast. As we’re bouncing back from COVID-19, our travel expenses have grown this year, and so have the fees for cloud services hosting. These are all as planned and happen in the context of healthy growth, including that from adoption and increased usage of paid services. We’re in a healthy financial position as membership revenue and usage fees, like content registration and Similarity Check document checking fees, continue to grow from the previous year.

Thank you to everyone who joined us for Crossref2024. This year’s meeting showcased our collective dedication to advancing open, accessible research infrastructure and underscored the power of collaboration in building a stronger scholarly community. As we reflect on the rich discussions and insights shared during the event, it’s clear our community is committed to advancing open and sustainable scholarly infrastructure.

Looking ahead, we’ll continue collaborating with members and partners to tackle challenges, expand accessibility, and foster collaboration. A key focus will be enhancing tools and metadata standards to serve the community better. Through innovative solutions and strategic initiatives like the Research Nexus, our collective efforts will make research more connected and accessible for all.

For anyone who couldn’t attend live, recordings are now available on our website. We’re excited to see how the ideas exchanged during this meeting spark progress across the scholarly ecosystem in the coming months.

2024 POSI audit

Lucy Ofiesh — Sat, 07 Dec 2024 00:00:00 +0000

Background

The Principles of Open Scholarly Infrastructure (POSI) provides a set of guidelines for operating open infrastructure in service to the scholarly community. It sets out 16 points to ensure that the infrastructure on which the scholarly and research communities rely is openly governed, sustainable, and replicable. Each POSI adopter regularly reviews progress, conducts periodic audits, and self-reports how they’re working towards each of the principles.

In 2020, Crossref’s board voted to adopt the Principles of Open Scholarly Infrastructure, and we completed our first self-audit. We published our next review in 2022.

The POSI adopters have continued to review the principles, reflecting on the effects of adopting them and providing a revision to the principles in late 2023. We use the revised principles for this latest review.

Key

We use a traffic light system to indicate where we believe we stand against each of the 16 principles. Now with up/down arrows to show any significant movement, and an ‘i’ where there is something of note with narrative.

red indicates we are not fulfilling the principle.
yellow indicates we are making progress towards meeting the principle.
green indicates we are fulfilling the principle.
or means this is a new change, where we’ve moved ‘up’ the traffic lights, in comparison to the previous audit. We would use the same if ‘down’ ever happens too.
or means that something has changed of note and in comparison to the previous audit.

GOVERNANCE

Coverage across the scholarly enterprise
Stakeholder governed
Non-discriminatory participation or membership
Transparent governance
Cannot lobby
Living will
Formal incentives to fulfil mission & wind-down

What’s changed with governance

Stakeholder governed

We’ve been yellow and we’re still yellow, but it has been improving. In the past, we’ve reported that we are working towards this but we’re not there yet because we didn’t have representation on the board from certain types of members, specifically research funders and research institutions. In the incoming 2025 board class, we have both. Six out of our 16 board seats are held by universities, university presses, or libraries. We also look forward to adding a new research funder, the Austrian Science Fund (FWF), to the board in January.

None of this, though, is hardcoded into the structure of the board. We extend an open call for board interest; any active member can apply for consideration. The Nominating Committee prepares a slate with a diverse range of candidates and organisations, and it is then up to the membership to elect board members.

With only 16 board seats and >21,000 members in 160 countries, being fully stakeholder-governed is challenging. Further, there are important contributors to the community that we all rely on who are not eligible for board seats because they are not members, as defined in our by-laws, such as sponsors, service providers, and metadata users.

We don’t consider this principle fulfilled, and that’s a good thing to keep note of; we must keep aspiring to have a broader, more comprehensive representation of our evolving community. The board continues to discuss stakeholder representation.

SUSTAINABILITY

Time-limited funds are used only for time-limited activities
Goal to generate surplus
Goal to create financial reserves
Mission-consistent revenue generation
Revenue based on services, not data

What’s changed with sustainability

Goal to create financial reserves

This was yellow and is now green. In 2023, we met our goal of maintaining a contingency fund of 12 months of operating costs. We also topped up this fund in 2024 to keep pace with our growing operating expenses. The revisions for POSI 1.1 actually removed the specificity of a 12-month timeline, allowing each adopting organisation to set its own goal; in Crossref’s case, 12 months remains appropriate.

INSURANCE

Open source
Open data (within constraints of privacy laws)
Available data (within constraints of privacy laws)
Patent non-assertion

What’s changed with insurance

Open source

This was yellow and still is, but we’re making improvements. In September of this year we migrated our database off of a closed-source solution and onto PostgreSQL. This has improved the performance of the system and is an important step towards paying down technical debt and moving the system fully into the cloud.

Patent non-assertion

This was yellow and is now green. We confirm that we do not hold any patents, and we have a published policy on it that is available for inspection and reuse by anyone in the community.

In summary

These are the main changes of note for our 2024 POSI update. The summary is that we’ve maintained all our greens, and of the four principles that were yellow last time, two have moved to green (financial reserves; patent non-assertion) and two have remained yellow but seen some progress of note (stakeholder governed; open source).

Please let us have any comments or questions; by commenting here it will add a public record of the discussion on our community forum. Here is an image to share, if needed.

We continue to learn from the POSI adopters group—now numbering 23 organisations—and the group will soon share a draft of POSI v2 for community comment. We look forward to the ongoing discussions with this group, and others, to keep improving and holding ourselves to account.

Summary of the environmental impact of Crossref

Ed Pentz — Thu, 05 Dec 2024 00:00:00 +0000

In June 2022, we wrote a blog post “Rethinking staff travel, meetings, and events” outlining our new approach to staff travel, meetings, and events with the goal of not going back to ‘normal’ after the pandemic. We took into account three key areas:

The environment and climate change
Inclusion
Work/life balance

We are aware that many of our members are also interested in minimizing their impacts on the environment, and we are overdue for an update on meeting our own commitments, so here goes our summary for the year 2023!

To be honest, the picture is mixed. On the positive side, we are traveling less and differently compared with 2019. Most of our events have been online, with some regional in-person ones, reducing our carbon footprint and increasing inclusivity with more people attending Crossref events. On the negative side, it hasn’t been easy to collect the data and figure out the best tools for calculating emissions, and we certainly haven’t captured all of our carbon emissions. Our approach has been to not let the perfect be the enemy of the good and we’ve focused on our largest source of carbon emissions - air travel.

Some of the positive things:

We have maintained our strategic approach to consider environmental, inclusion, and work/life balance issues when we plan travel and to make the most of in-person events by focusing on those that involve interaction, such as listening and learning from our members and users, deepening relationships, co-creating, and forming new alliances
Crossref Annual Meetings and community updates have been online and in different time zones.
Crossref board meetings have been reduced from three in-person meetings per year to one face-to-face and two online meetings per year.
We had an optional all-staff in-person meeting in June 2023 (and this year too).
For the in-person board and staff meetings, we have selected locations that minimize the overall amount of travel and maximize direct flights.
We have maintained our country focus for in-person local meetings supported by regional Ambassadors.
We met our goal of keeping total travel and meeting expenses below 60% of 2019 costs even though we have more staff and membership growth has continued. The amount of money spent is a rough proxy for our carbon impact.
We no longer have an office in Oxford and will not renew the lease on our Lynnfield, MA office, so we will have no physical offices by the end of 2024. This is not a large carbon emission reduction and is more a result of being a “distributed first” organisation with staff in 11 different countries.
We recorded data on staff travel (flights, trains, cars, hotels) for 2023 to use as a baseline for comparison with future years. In 2023 the carbon emissions from travel and meetings was about 105 tCO2e.
We used tools provided by Amazon Web Services (AWS) and Zoom to estimate the impact of these services. In 2023 this was 0.266 tCO2e for AWS and .1 tCO2e for Zoom.

Some challenges

Compiling data is difficult and time-consuming for a small organisation
There are many different calculators and metrics to use and it’s difficult to decide which to use and how much detail to go into
We haven’t yet estimated the carbon footprint of staff home working
We were able to calculate the emissions from AWS but not our data center
We didn’t estimate the emissions from our offices. We had a small office in Oxford until November 2023, and we have an office near Boston - we won’t be renewing the lease in 2025 so won’t have any offices.

Total travel and meetings spending

Year	Amount	Percentage of 2019
2019 actuals	$585,482	100%
2020 actuals	$91,700	16%
2021 actuals	$19,066	3%
2022 actuals	$74,416	13%
2023 actuals	$305,737	52%
2024 budget	$333,500	56%

We have recorded carbon emissions from travel at about 105 tCO2e, so we will compare 2023 with future years. Now that we have started collecting travel data, it will be easier—staff can do it as they travel throughout the year.

Our Executive Director, Ed Pentz, looked at his personal and work flights and the carbon emissions in 2019 were 18 tCO2e and in 2023 were 2.7 tCO2e so this is a big change in the right direction.

Hosting services

We use AWS for hosting our REST APIs, Crossref Metadata Search, the website, and Labs projects. Our main metadata registry is still in a data center, which is not included in this calculation. For 2023 Amazon reports Crossref’s carbon emissions were 0.216 tCO2e compared with 0.266 tCO2e in 2022. Crossref is planning to move out of the data center and fully to AWS by the end of 2024 so this will increase our AWS usage and therefore our emissions from related activities will increase. Compared to travel, the footprint from AWS is minimal.

Online meetings

As a distributed, remote-first organisation Crossref is a heavy Zoom user –– it’s essential for staff and for engaging with our community. However, Zoom doesn’t provide tools or estimates of the carbon impact of Zoom meetings. We used a tool provided by Utility Bidder, which makes a lot of estimates and assumptions. In 2023 Crossref had almost 800,000 meeting minutes. This translated into an average of 1.92 kg of CO2 emissions per week, or 100 kg per year.

Some studies have estimated that turning off video reduces the carbon footprint of meetings. However, this can be a false savings since video is often important for creating a connection and having a productive meeting, and a Zoom meeting with video is still much, much better than traveling, particularly if flying is involved.

Tools we used

In order to calculate emissions for flights and train journeys, we chose to use Carbon Calculator. We didn’t calculate emissions from hotel stays but looked at the Hotel Footprinting tools and may add hotels to calculations in the future.

Offsetting

We don’t offset our emissions from travel or other operations and don’t have plans to do this. Offsetting emissions is problematic in a number of different ways so we don’t feel confident in doing it.

We did tree-planting as a “thank you” for the time of respondents in our metadata survey. Intended as an alternative to more commercial types of incentives rather than off-setting for our emissions, this resulted in 921 trees planted for the Gewocha Forest, Ethiopia via Ecologi.

Wrapping up

Moving forward, we’ve learned a lot over the last couple of years. Collecting accurate data is challenging and time-consuming, especially for a small organisation. For us, this has been a new lens for viewing our activities, and it remains a true learning journey and we have made permanent changes. In 2024 and beyond we are going to continue to follow our travel, meetings, and events policies that we announced in 2022. We will continue to capture our air travel emissions, and in 2025 we will more accurately capture train journeys and hotel stays. We will also continue calculating our Zoom and AWS emissions as best as we can. What we’ve learnt in the process of capturing and calculating our 2023 emissions helped us set things up to enable more prompt reporting on these impacts in the future.

We expect that many of our members and our community at large assess their environmental impact or are embarking on similar projects, to understand and curb emissions. We’re keen to discuss this and learn together to reduce our environmental impact as an organisation.

Metadata beyond discoverability

Ginny Hendricks — Tue, 03 Dec 2024 00:00:00 +0000

Metadata is one of the most important tools needed to communicate with each other about science and scholarship. It tells the story of research that travels throughout systems and subjects and even to future generations. We have metadata for organising and describing content, metadata for provenance and ownership information, and metadata is increasingly used as signals of trust.

Following our panel discussion on the same subject at the ALPSP University Press Redux conference in May 2024, in this post we explore the idea that metadata, once considered important mostly for discoverability, is now a vital element used for evidence and the integrity of the scholarly record. We share our experiences and views on the metadata significance and workflows from the perspective of academic and university presses – thus we primarily concentrate on the context of books and journal articles.

The communication of knowledge is facilitated by tiny elements of metadata flitting around between thousands of systems telling minuscule parts of the story about a research work. And it isn’t just titles and authors and abstracts – what we think of as metadata has really evolved as more nuance is needed in the assessment and absorption of information. Who paid for this research and how much, how exactly did everyone contribute, what data was produced and is it available for me to reuse it, as well as, increasingly, things like post-publication comments, assertions from “readers like me”, who has reproduced this research or refuted these conclusions.

Different types of published works are described by different types of metadata – journal articles, book chapters, preprints, dissertations. And those metadata elements can be of varying importance for different users. In this article, we will talk about metadata from the perspectives of four personas highlighted by the Metadata 20/20:

Metadata Creators, who provide descriptive information (metadata) about research and scholarly outputs.
Metadata Curators, who classify, normalise and standardise this descriptive information to increase its value as a resource.
Metadata Custodians, who store and maintain this descriptive information and make it available for consumers.
Metadata Consumers, who knowingly or unknowingly use the descriptive information to find, discover, connect, and cite research and scholarly outputs.

Our approach delineates the metadata lifecycle, from authorship, through production, discovery and through continuous curation. Though some of the metadata is generated outside of that linear process, and much happens before the authorship step, we see it as a clear and useful breakdown of how metadata contributes to a new piece of content.

Authorship

The first stage in the metadata lifecycle, authorship, is just the beginning of a dynamic process with many collaborators. A formative piece of the puzzle, authorship involves the authors or contributors, the editorial team and/or the marketing team and this is when the shape of the project and its metadata takes form. During this stage, the book or journal’s metadata exists only between the originators and the publisher, allowing the most opportunity for creativity and enhancement. Once the metadata reaches the next checkpoint along the lifecycle and is sent out externally, it’s more difficult and riskier to make major changes to the key metadata elements. In scholarly monograph publishing especially, we have the advantage of longer production lead times during which to amend and manipulate metadata during this stage.

At this stage, authors may have ideas of titles, subtitles and descriptions and it is up to the editors and other team members at the publisher to think strategically about how this can be optimised. The marketing and sales teams may be thinking about how the abstracts, keywords, and classifications can be best optimised for the web, leading to increased sales. Discoverability and interoperability of metadata for a book or journal, especially the use of persistent identifiers, is beneficial both for the author – in that their book is easily discovered, used, and cited – and for the publisher – increased visibility, sales, and usage.

Current challenges at the authorship stage include changing goalposts for metadata standards and accessibility requirements, which also have knock-on effects in subsequent stages in the metadata lifecycle. One of the key challenges with these is that they require buy-in from multiple players to keep up with and amend, and publishers must think closely about how these changes may affect metadata workflows for books at different stages of publication.

Production

As a book or journal article comes into production, it’s time to update and release the metadata to retailers, libraries, data aggregators and distributors. The metadata should be updated and checked to make sure that it’s still a good reflection of the product or the content that it describes and complete enough to release, including a final cover image in the case of books. This is still very much a collaborative effort with multiple roles involved. Technical details, such as spine width, page extents, and weight, are added, capturing the final specification. The editorial team may update metadata entered into systems earlier in the process. For example reviewing the prices, updating subject classification codes or amending the chapter order. If any of the content is to be published open access, appropriate licensing and access metadata need to be included, so that users of the content are clear about what they can (and can’t!) do with it. Metadata that’s not yet captured upstream can be added or enhanced. For example, vendors already involved in the production process can verify that persistent identifiers (PIDs) are present and correct in funding metadata.

More and more metadata elements are being requested by supply chain partners. For example, new requirements being introduced to provide commodity codes, spine width, carton quantities, gratis copy value and country of manufacture. There may be differences in metadata depending on the methods of production. For example, country of manufacture will be supplied differently when using traditional print methods where the whole print run is carried out at a location, or where a title is manufactured print on demand and the location of printing is determined by the delivery address.

In an XML-first workflow, metadata can be captured with the content files to aid with discovery. This usually requires multiple systems, both internal and external. These systems need to be able to work together to ensure that only up-to-date metadata is used. Metadata will change throughout the production process, whether it’s the publication of an accepted manuscript through to the final version of record, or pre-order information to the published version, so updates need to feed out regularly.

The right metadata needs to go to the right recipient. Some is not useful or cannot be processed by certain recipients. For example, a printer, retailer, librarian or data aggregator each have their own needs and use cases and may receive and process metadata in different formats or require different fields.

Discovery

Discovery is the series of actions taken by an end user to retrieve and access relevant content they do not know about. Discovery can happen everywhere: Google (a search engine), a library catalog, a publisher platform, etc. However, Discovery is associated with using Discovery systems in the academic sector.

The technological landscape of libraries has developed in the last 15 years. Discovery systems are tools libraries subscribe to in order to allow their end users to have one search experience within their library holdings. It is paramount for librarians that library collections are used; hence, it is very important for them that the discovery system of their choice contains all the relevant metadata. Libraries expect their discovery service to include their content coverage as comprehensively as possible. Content items not represented or misrepresented in a discovery system create challenges to libraries in how they might otherwise ensure that these materials are discovered and accessed.

Libraries’ adoption and usage of discovery systems are surrounded by the belief that the great benefits of this technology are the one search box and the configuration flexibility, which are the most important benefits. Libraries invest a significant amount of money in discovery services. The increase in usage is the success indicator of this adoption and a positive return on investment.

The backbone of discovery systems is formed by three crucial elements: a user interface, a metadata index, and a link resolver or Knowledge Base. These elements, along with a back-end control panel for librarian configuration, are the key components that enable the discovery process.

The discovery index, a database storing descriptive data from various content providers, data sets, and content types, is a testament to the collaborative efforts of content providers and discovery systems vendors. Their work under the Discovery Metadata Sharing partnership agreements, which establish the format, scope, frequency, and support of the collaboration, is instrumental in meeting librarians’ expectations.

Format

The discovery metadata integration processes have settled down for most cases in these two metadata delivery workflows.

Metadata for the index of discovery: Discovery systems have traditionally made efforts to work with various metadata formats like MARC, proprietary templates, etc., but the preferred format is XML. This metadata could include all the bibliographic information data, including index terms and full text at the article and chapter level.

Metadata for link resolvers and Knowledge bases: Knowledge bases are tools that contain information about what is included in a product, packages, and/or databases. KBART is the preferred format in this area. It includes a set of basic bibliographic descriptions at the publication level and linking information for direct and OpenURL syntaxes.

Frequency

The delivery channels vary, and the frequency could vary daily to yearly, depending on the publication schedule.

Scope

Library collections include various content types, including archival materials, open access, and multimedia alongside the more traditional books and periodicals. Different content types will require different metadata elements to make a comprehensive discovery-friendly description, and the metadata elements will impact the formats in use.

Discovery services will receive this data and prioritise uploading. They will select and manipulate the required metadata elements according to their system requirements. These metadata tweaks and selections are not always communicated to the content providers and/or libraries. Ultimately, librarians decide which metadata will be visible on their discovery tool and the linking methods of their choice.

As described, Discovery is a complex area where the activities of its main stakeholders are interconnected. The success of the end users’ discovery journey from search to access depends on the successful integration, implementation, and maintenance of the discovery systems. This necessitates a combined effort from the three discovery stakeholders: content providers, discovery system providers, and libraries. Their collaborative work is not just crucial, but integral to supporting discovery and fulfilment in the most efficient manner possible. Your active involvement in this process is what makes it successful.

How do we ensure discoverability?

Electronic resources do not exist in isolation but are assessed and used depending on their level of integration in the discovery landscape where libraries and patrons are active. From a content provider’s perspective, discoverability is about the number and efficiency of entry points to our products created in third-party discovery products.

The level of discovery integration has a direct impact on sales and upsell opportunities. Products that are not discoverable are difficult to work with, and the opposite is true for products that are considered discoverable. Your role in ensuring discoverability directly influences the user experience and sales, making your work crucial and impactful. The term ‘Discoverability’ is critical in discovery library systems. It refers to the extent to which eResources are searchable in a discovery system, and it directly influences the ease with which users can find the information they need, thereby enhancing their overall experience. In practical terms, the degree of discoverability will be impacted by the quality of the metadata supplied, the transformations the metadata suffers in the integration process to discovery systems, and the configuration’s maintenance.

The general principles of metadata quality also apply in this area: accuracy, completeness, and timely delivery. Your attention to these principles is crucial to contributing to the effectiveness of the discovery process. Metadata enrichment practices like identifiers and standards are also applicable. Your meticulous attention to detail in maintaining metadata quality ensures the effectiveness of the discovery process.

Discovery as a mindset in the publishing process will increase discoverability, as it will be influenced by product designs (whether the content is linkable) and which metadata outputs are possible. For example, author-generated index terms will be more effective for meeting research search terms, and detailed article titles will probably be more discoverable than general titles. Finally, all the integration, descriptive metadata, configurations, etc., leave much room for errors. The flow is complex; on occasion, the products and content are more complicated to describe than tools can handle, and there are millions of holdings per library to manage. Constant maintenance and troubleshooting are crucial elements to maintaining and increasing discoverability.

Metadata beyond publication

In the lead-up to publication, finalising rich complete metadata can seem like establishing a fixed set of information. Post-publication, however, the metadata workflow should be dynamic, able to evolve to keep pace with new demands and opportunities. Think of metadata as a journey rather than a one-time destination, and look at ways to futureproof your metadata by actively adapting to some of the following types of change.

Changing Publisher Goals and Product Needs

Metadata should align with changing priorities for a publisher. Developing new formats, shifts in commissioning focus or building new distribution partnerships may require metadata updates. For instance, re-releasing content in audiobook form or digitising a backlist title warrants a metadata review to ensure current and prospective readers find accurate, relevant information.

Changing Technology and Metadata Standards

Advances in technology, from artificial intelligence to emerging metadata standards, offer enhanced possibilities for capturing and updating metadata. AI, for example, can help enrich metadata with more precise subject tagging, while new metadata formats may offer greater compatibility across platforms and discovery services. Staying current with these tools can help publishers manage metadata more efficiently and enhance discoverability.

Changing Societal Values

As society evolves, so do expectations for inclusive and socially responsible metadata. Utilising new categorisation codes, such as those for the United Nations Sustainable Development Goals, can align metadata with emerging social goals. Similarly, publishers may need to revisit keywords and category codes to reflect language changes, balancing the integrity of historic records with the need for current, appropriate terminology.

Changing Industry Priorities

Commitments to accessibility and sustainability have prompted developments in metadata. Increasingly, publishers need to be able to use metadata to build a record of sustainable production methods, such as paper sources, printing methods or ink types. New metadata fields for accessibility specifications will also support more inclusive reader experiences going forward. Metadata will play an increasingly vital role in meeting industry standards for accessibility, EUDR and EAA compliance, and environmental transparency.

Changing Customer and Librarian Expectations

Finally, as the metadata expectations of customers grow and the nature of roles and responsibilities in library and collection management professions develops, teamwork and making good use of available resources are essential. Publishers don’t have to tackle this alone. Working with organisations such as Crossref or Book Industry Communication (BIC), signing up to newsletters and webinars, and forming an in-house discovery group are all great ideas for sharing ideas and best practice, and ensuring your metadata workflow is adaptable and responsive. Be part of the conversation now rather than struggling to keep up down the line!

What are some challenges and opportunities with metadata?

JM: Metadata that establishes permanence is a real opportunity in a digital landscape where content can move or be taken down, links can rot, website certificates can expire. Persistent identifiers like ORCiDs for people and DOIs for content are key examples of metadata that establish enduring routes to, and provenance of, published digital content.

KM: Metadata creation, maintenance and change has long been seen as a manual process. AI tools offer a real opportunity for metadata creation and review, especially for keywords and classification codes, at a scale and speed that has the potential to transform metadata workflows. Especially for backlist transformation, AI could offer real opportunities in this area. A challenge we face for monograph metadata more specifically is that much of the scholarly metadata infrastructure is built around the journal article, and it can be difficult to fit longer form content into these systems of discovery.

MT: Metadata is crucial. Good metadata (complete, accurate, and timely) is the base for smooth integrations and easy discovery interactions with eResources. Bad metadata (inaccurate, incomplete, late) will be the main reason for undiscovered content. At this point, the eResources industry is still based on different versions of the same metadata, which is the leading cause of problems. It is probably time to start considering a unique record approach. This unique record, which will be complete and accurate, could be used by different systems for different purposes. I know there are many details to define here, but if you think about it, it is not impossible and could solve the many known issues.

How do you ensure the quality and completeness of your metadata? Do you have ways of auditing it?

SP: Validation of data is really important, so choosing or building a system that’s set up to do this is an important foundation. It’s straightforward to check for completeness of fields and I run daily checks on our book metadata to make sure there’s nothing missing in the files feeding out. Quality can be more challenging to monitor. Feedback from data recipients is key, and accreditation schemes such as the BIC Metadata Excellence Award are a great way to benchmark progress. Good training and clear documentation help to make sure that everyone involved in creating and updating metadata understands exactly what they need to do and the standards they need to meet.

KM: Earlier this year we completed a year-long data cleansing project as part of our move to a new title management database. This gave us the time to address gaps in backlist metadata as well as to identify any inconsistencies across records for the same book, and enrich key metadata fields like classification codes, keywords and PIDs. For frontlist titles, each person owns a number of fields to ensure they are complete before a book’s metadata is distributed – some of these have validation tools which will prevent a book’s metadata from being sent out unless it is complete.

MT: Strict and consistent internal processes are essential to ensure quality and completeness. Following the different standards and industry recommendations helps to keep the quality at high standards. Random manual checks and system-based checks help to ensure everything is good. We carry out projects where we work with specific aspects of the metadata. This building-blocks approach ensures the different data layers are as good as possible. As with any project, metadata projects should have specific goals, outcomes, resources, and documentation.

How do you know if (and how much) metadata helps achieve your goals?

JM: Take any available opportunities to find out what people think of your metadata – via library conferences, institutional customer feedback, and by working with the library team at our home institution, we’ve had some really useful and interesting conversations about MUP’s metadata and where we can improve it to make it as relevant as possible for different stakeholder needs.

MT: Customers and Discovery partners will inform us if something is incorrect. Usage data is also a good indicator of how healthy our metadata is. Following industry standards is another good reference point for assessing the metadata. Finally, the metadata is only good when we know what we want to use it for. So, always considering what we are trying to achieve helps us understand how effective the metadata is.

KM: As the others have noted here, and we represent a range of different types and sizes of publishers, measuring the direct impact of metadata is an ongoing challenge. We think about the different end users who might encounter our metadata further down the supply chain – retail customers searching on Amazon, librarians filtering results on purchasing platforms, researchers finding our books and journals through citations on popular online search engines – and consider what elements of our metadata might help reach those people in the right ways.

JM: Ideally, you’ll see an uplift in sales or usage for every metadata element that you add, review or expand, although it can be challenging to quantify and prove a direct correlation between richer metadata and higher revenue or discoverability, as there are will be other factors involved. For my Operations team, what is certain is that richer, more comprehensive metadata means fewer errors are thrown up by the distribution systems and feeds we use, which means colleagues save time and gain productivity by not having to resolve and rerun failed jobs, chase missing information from other teams, or manually send information to third parties. My job is also made easier because things like size and weight of every printed product are recorded in our bibliographic database as standard, easy to report on and analyse, which helps with forecasting costs for inventory storage or shipping. Metadata can be powerful.

Research Integrity Roundtable 2024

Martyn Rittman — Fri, 15 Nov 2024 00:00:00 +0000

For the third year in a row, Crossref hosted a roundtable on research integrity prior to the Frankfurt book fair. This year the event looked at Crossmark, our tool to display retractions and other post-publication updates to readers.

Since the start of 2024, we have been carrying out a consultation on Crossmark, gathering feedback and input from a range of members. The roundtable discussion was a chance to check and refine some of the conclusions we’ve come to, and gather more suggestions on the way forward. As in previous years, we were able to include a range of organisations, which led to lively and interesting discussions. See below for the full participant list.

Crossmark feedback

We started by presenting Crossmark and a summary of the consultation process. There are a number of areas where we have learned more about how the community operates or found that Crossmark needs to adapt. These include:

Implementation: Our members have struggled to implement Crossmark and uptake is low. At the same time, in many organisations the workflows for handling retractions are not well-defined because they are rarely used, if ever. The responsibility for updating Crossref metadata can be unclear and this may be a factor in the low uptake.

Education: There are different levels of understanding about how to handle retractions. Some members are very defensive when asked about retractions, others state they will never make updates to published works. How can we have a constructive conversation where the value of communicating updates appropriately is recognised?

Community engagement: Given the different scales, locations, disciplines, and technologies used by our members, it looks like one size will not fit all when it comes to updates. How can we get continual, representative feedback on new tools and processes?

Metadata assertions: Crossmark allows the deposit of metadata using custom field names, however this metadata seems to have low usefulness and is not highly valued by the community. Should we continue to collect it? Can we make some of the most-used field names part of our standard schema?

Changing the Crossmark UI: Although we didn’t specifically ask about it during the consultation, the look of the Crossref logo often came up, and concern that it is not recognised and not well-used. Can we change the look and behaviour so that it has more impact?

NISO Recommendations

Patrick Hargitt represented the NISO group on Communication of Retractions, Removals, and Expressions of Concern (CREC). The group’s recommendations were published earlier this year and cover how retractions are communicated. CREC arose from an earlier project, IRSRS. A large part of the motivation is that retracted works continued to be cited, with citing authors apparently unaware of the retraction. Patrick presented the CREC recommendations, which cover:

Metadata receipt, display, and distribution,
Which metadata elements to communicate,
How to implement the recommendations,
Discussion of some special cases,
Key stakeholders and their responsibilities.

The two presentations prompted discussion, which was taken into the first of two workshops.

First workshop: Improving collection of retractions and Crossmark

The first workshop looked at proposed changes to Crossmark and how to encourage more members to deposit their retractions, corrections, and other post-publication updates. Several important themes emerged.

First, the question of whose responsibility it should be to provide metadata on retractions and similar updates. Crossref has a responsibility to work with the community to obtain high quality and complete metadata; publishers should take responsibility for handling issues of research integrity and reporting them to relevant downstream services, like Crossref; and platforms need to provide tools that allow easy reporting of retractions.

The value of Crossmark appearing in PDFs was reiterated. The fact that a PDF can be downloaded, and years later there is a way to tell whether it has been retracted or not is highly valued. There was also the suggestion that the Crossmark logo on web pages can indicate a change before it has been clicked. This is something that we have been considering at Crossref and it was useful to have the idea reinforced. Another suggestion was that a browser plugin would make a good complement to Crossmark.

Implementation issues with Crossmark were raised, including that it’s difficult to validate whether a specific implementation is complete. There are a number of different changes (to metadata deposit and content, and websites) that need to work together to have Crossmark fully functional. There were several questions and a discussion about Retraction Watch data. Some were about understanding its collection and validation. A number of participants are actively using the data and it was great to see the variety of applications.

Second workshop: Community use of retraction metadata

The second workshop focused on a broader set of downstream organisations that might want to make use of retraction metadata. We looked at stakeholders and their needs, and attempted to match them up with existing tools. Several gaps were identified as a result, which may provide opportunities for new services or collaborations to fill them.

We identified a number of tools available for publishers, editorial systems, metadata researchers, and readers. A good example is reference managers, many of which are now highlighting retracted works to authors. This can help to reduce the number of retracted works being cited. Publishing platforms are also providing support to editors, using tools that include retraction metadata.

Some of the stakeholders identified have limited tools for identifying retractions that are relevant to them. These include funders, archives and repositories, journalists, and institutions.

Often, there are pathways for retraction data to be communicated but they are not being sufficiently used. There needs to be a concerted effort to improve the quality of retraction metadata for tools to function better. For example, a second author on a paper might not know that a correction or retraction is planned for their article. If their email or ORCID isn’t included in the metadata, an alerting tool wouldn’t be able to let them know. A similar argument can be made for institutions or funders if they are not well-identified in the metadata.

The question of standardisation of metadata was raised. It seems too early to implement a full set of standards at the moment. CREC and similar initiatives have documented and accommodated for a range of practices while providing guidance and principles to work towards. More discussion is needed in the community to work out paths that could be applied across the broad spectrum of scholarly communication.

Conclusion

The event was very valuable in bringing up a range of topics related to retraction and communication of post-publication changes to scholarly works. We are grateful to all of the participants for their contributions and sharing their diverse experience and opinions with us.

Research integrity is an area of flux, with significant changes over the past few years. While there has been progress, there remain gaps in metadata and tools to communicate retractions. This is something that Crossref will continue to contribute to, and Crossmark clearly still has a role to play.

Some of the ideas and suggestions from the discussion can be implemented in the near future. Others need further development, and we will continue to engage the community. Reading this, there may be topics where you feel you have a role to play. We are keen to partner with other organisations in this space as we continue to improve the transparency and communication of metadata for post-publication updates.

Participants

Many thanks to the participants. Here is the full list of those that attended:

Name	Role	Organisation
Aaron Wood	Head, Product & Content Management	American Psychological Association
Adya Misra	Associate Director, Research Integrity	Sage
Bianca Kramer		Sesame Open Science
Constanze Schelhorn	Head of Indexing	MDPI
Guillaume Cabanac	Full Professor	University of Toulouse
Hong Zhou	Director of AI Product	Wiley
Jennifer Wright	Head of Publication Ethics and Research Integrity	Cambridge University Press
Johanssen Obanda	Community Engagement Manager	Crossref
Joris van Rossum	Program Director	STM Solutions
Kathryn Weber-Boer	Data & Analytics	Digital Science
Kornelia Korzec	Director of Community	Crossref
Kruna Vukmirovic	Publisher- Journals	The Institution of Engineering and Technology
Lena Stoll	Product Manager	Crossref
Leslie McIntosh	VP, Research Integrity	Digital Science
Liying Yang	Professor	CAS Library
Luis Montilla	Technical Community Manager	Crossref
Madhura Amdekar	Community Engagement Manager	Crossref
Martyn Rittman	Progam Lead	Crossref
Maryna Kovalyova	Member Experience Manager	Crossref
Mina Roussenova	Project Manager, Strategic Projects	Karger
Osnat Vilenchik	VP Content Operations	Ex Libris, part of Clarivate
Patrick Hargitt	Senior Director of Product Management	Atypon/Wiley
Paul Davis	Tech Support & R&D Analyst	Crossref
Sami Benchekroun	CEO	Morressier
Scott Delman	Director of Publications	Association of Computing Machinery (ACM)
Shilpi Mehra	Head, Research Integrity & Paperpal Preflight	Cactus Communications
Sichao Tong		Chinese Academy of Sciences, Library

How good is your matching?

Dominika Tkaczyk — Wed, 06 Nov 2024 00:00:00 +0000

In our previous blog post in this series, we explained why no metadata matching strategy can return perfect results. Thankfully, however, this does not mean that it’s impossible to know anything about the quality of matching. Indeed, we can (and should!) measure how close (or far) we are from achieving perfection with our matching. Read on to learn how this can be done!

How about we start with a quiz? Imagine a database of scholarly metadata that needs to be enriched with identifiers, such as ORCIDs or ROR IDs. Hopefully, by this point in our series this is recognizable as a classic matching problem. In searching for a solution, you identify an externally-developed matching tool that makes one of the below claims. Which of the following would demonstrate satisfactory performance?

It is a cutting-edge, state-of-the-art, intelligent-as-they-come, bullet-proof technology! All the big players are using it. You won’t find anything better!
The tool was tested on the metadata of 10 articles we authored, and many identifiers were matched.
The quality of our matching is 98%.

Okay, okay, trick question. The correct answer here is to opt for secret answer #4: “I wouldn’t be satisfied by any of these claims!” Let’s dig in a bit more to why this is the correct response.

The importance of the evaluation

Before we decide to integrate a matching strategy, it is important to understand as much as possible about how it will perform. Whether it is used in a semi or fully automated fashion, metadata matching will result in the creation of new relationships between things like works, authors, funding sources, and institutions. Those relationships will then, in turn, be used by the consumers of this metadata to guide their understanding and perhaps even to make important decisions about those same entities. As organisations providing scholarly infrastructure, we must therefore take it as our paramount responsibility to understand any caveats or shortcomings of the scholarly metadata we make available, including that resulting from matching.

Proper evaluation is what allows us to do this, as it is impossible to know how well a given matching strategy will perform in its absence. This is true no matter how simple or complex a matching strategy may seem. Complex methods can be tailored to data with specific characteristics and might fail when faced with something different from this. Simple methods might be only appropriate for clean metadata or a narrow set of use cases.

Beyond complexity, matching strategies themselves vary widely in character, inheriting biases from their design, training data, or how a problem has been formulated. Some prioritise avoiding false negatives, while others focus on minimising false positives. Even a generally high-performing strategy might not be perfectly aligned with your specific needs or data. In some cases, the task also itself might be too challenging, or the available metadata too noisy, for any matching strategy to perform adequately.

Evaluation is, again, how we understand these nuances and make informed decisions about whether to implement matching or avoid it altogether. By now, it should also be clear that the notion “we don’t need to evaluate” is far from ideal! Given its importance, let’s explore how evaluation is actually done.

Evaluation process

In general, a proper evaluation procedure should follow the following steps:

Preparation of an evaluation dataset containing many examples of matching inputs and the corresponding expected outputs.
Applying the strategy to all inputs from the dataset and recording the responses.
Comparing the expected outputs with the outputs from the strategy.
Converting the results of the above comparison into evaluation metrics.

From this accounting, we can see that there are two primary components for the evaluation process: an evaluation dataset and metrics.

Evaluation dataset

It’s useful to conceive an evaluation dataset as the specification for an ideal matching strategy, describing what would be returned from our forever-elusive perfect matching. When creating such a dataset, what this means in practice is that it should contain a number of real-world, example inputs, along with the corresponding ideal or expected outputs, and that all data should be in the same format as the strategy is expected to process. The outputs should themselves also confirm the strategy’s overall requirements, for example, by being consistent with its cardinality, meaning whether zero, one, or multiple matches should be returned and under what circumstances. In terms of size, it’s generally useful to calculate the ideal number of evaluation examples using a sample size calculator or using standardised measures, but as a quick rule of thumb: less than 100 examples is probably insufficient, more than 1,000 or 2,000 is generally acceptable.

It is also important that the evaluation dataset be representative of the data to be matched in order to ensure reliable results. Using unrepresentative data, even if convenient, can lead to biassed or misleading evaluations. For example, if matching affiliations from various journals, building an evaluation dataset solely from one journal that already assigns ROR IDs to authors’ affiliations might be tempting. The data, having been already annotated, allow us to avoid the tedious work of labelling, and we might even know that it is produced by a high-quality source. This is still, unfortunately, a flawed approach. In practice, such datasets are unlikely to represent the entire range of affiliations to be matched, potentially leading to a significant discrepancy between the evaluated quality and the actual performance of the matching strategy, when applied to the full dataset. To assess a matching strategy’s effectiveness, we have to resist shortcuts and instead do our best to create truly representative evaluation datasets to be confident that we’ve accurately measured their performance.

Evaluation metrics

Evaluation metrics are what allow us to summarise the results of the evaluation into a single number. Metrics give us a quick way to get an estimation of how close the strategy was to achieving perfect results. They are also useful if we want to compare different strategies with each other or decide whether the strategy is sufficient for our use case, removing the need to compare countless evaluation examples from different strategies against one another.

The simplest metric is accuracy, which can be calculated as the fraction of the dataset examples that were matched correctly. While a commonsense benchmark, accuracy can be misleading, and we generally do not recommend using it. To understand why, let’s consider the following small dataset and the responses from two strategies:

Input	Expected output	Strategy 1	Strategy 2
string 1	ID 1	ID 1	ID 1
string 2	ID 2	ID 3	Empty output
string 3	Empty output	Empty output	Empty output

Both strategies achieved the same accuracy, 0.67, making one mistake each on the second affiliation string. However, a closer examination reveals that these error types are distinct. The first strategy matched to an incorrect identifier, while the second refused to return any value illustrating the limitation of accuracy as a measure: it generally fails to capture important nuances in strategy behaviour. In our example, the first strategy appears more permissive, returning matches even in unclear circumstances, while the second is more conservative, withholding them when uncertain. Although using such a small dataset would preclude drawing any definitive conclusions, it highlights how relying on accuracy alone can obscure differences in performance.

For evaluating matching strategies, we instead recommend using two metrics: precision and recall. To recap from our previous blog post:

Precision is calculated as the number of correctly matched relationships resulting from a strategy, divided by the total number of matched relationships. It can also be interpreted as the probability that a match is correct. Low precision indicates a high rate of false positives, which are incorrect relationships created by the strategy.
Recall is calculated as the number of correctly matched relationships resulting from a strategy, divided by the number of true (expected) relationships. It can also be interpreted as the probability that a true (correct) relationship will be created by the strategy. Low recall means a high rate of false negatives, which are relationships that should have been created by the strategy but were not made.

Applying these measures to our prior example, the strategies achieved the following results:

Strategy 1: accuracy 0.67, precision 0.5, recall 0.5
Strategy 2: accuracy 0.67, precision 1.0, recall 0.5

As we can see, while both strategies have the same accuracy, using precision and recall better describes the difference between the two sets of results. Strategy 1’s lower precision indicates it made false positive matches, while Strategy 2’s perfect precision shows that it made none. The identical recall scores show both identified half of the possible matches.

Of course, results calculated using such a small dataset are not very meaningful. If we obtained these scores from a large, representative evaluation dataset, it would indicate to us that Strategy 1 risks introducing many incorrect relationships, while Strategy 2 would be unlikely to do so. In both cases, we would still expect approximately half of the possible relationships to be missing from the strategies’ outputs.

Which one is more important to prioritise, precision or recall? It depends on the use case. As a general rule, if you want to use the strategy in a fully automated way, without any form of manual review or correction of the results, we recommend paying more attention to precision. Privileging precision will allow you to better control the number of incorrect relationships added to your data. If you want to use the strategy in a semi-automated fashion, where there is a manual examination of and a chance to correct the results, pay more attention to recall. Doing so will guarantee that enough options are presented during the manual review stage and fewer relationships will be missed as a result.

To get a more balanced estimation of performance, we can also consider both precision and recall at the same time using a measure called F-score. F-score combines precision and recall into a single number, with variable weight given to either aspect. There are three commonly used types, each calculated as the weighted harmonic mean of precision and recall:

F0.5: Precision is weighted more heavily. It can be understood as a score that is 50% more sensitive to precision than recall. A high F0.5 score indicates a measure of performance that minimises false positives.
F1: Equal weight is given to both precision and recall. It can be interpreted as the most balanced score in this set. High F1 indicates good overall performance, with both false positives and false negatives being minimised equally.
F2: Recall is weighted more heavily. It can be understood as a score that is 50% more sensitive to recall than precision. A high F2 score indicates a measure of performance where false negatives are minimised.

Each of these variants allows for fine-tuning the evaluation metric to align with your expectations for a specific matching task. Choose whichever reflects the relative importance of precision versus recall for your use case.

To summarise, to avoid falling prey to misleading sales pitches or silly quizzes, it is important to have a good understanding of the performance of any strategies you are building or integrating. With thorough evaluation, including a representative dataset and carefully considered metrics, we can estimate the quality of matching and, by extension, its resulting relationships.

Now that we’ve covered how to evaluate effectively, we can move on to some other aspects of metadata matching. Our next blog post will take a final, more holistic view of matching, exploring some complementary considerations to all of the preceding. Stay tuned for more!

Update on the Resourcing Crossref for Future Sustainability research

Kornelia Korzec — Mon, 28 Oct 2024 00:00:00 +0000

We’re in year two of the Resourcing Crossref for Future Sustainability (RCFS) research. This report provides an update on progress to date, specifically on research we’ve conducted to better understand the impact of our fees and possible changes.

Crossref is in a good financial position with our current fees, which haven’t increased in 20 years. This project is seeking to future-proof our fees by:

Making fees more equitable
Simplifying our complex fee schedule
Rebalancing revenue sources

In order to review all aspects of our fees, we’ve planned five projects to look into specific aspects of our current fees that may need to change to achieve the goals above. This is an update on the research and discussions that have been underway with our Membership & Fees Committee and our Board, and what we’ve learned so far in each of these areas.

Goal 1: More equitable fees.

To ensure our fees going into the future are more equitable, we’re carrying out two parallel projects: evaluation of the lowest membership tier, and the review of the basis for deciding the membership tiers and distribution of membership across them.

Project 1: Evaluate the lowest membership tier and propose a more equitable pricing structure.

All Crossref members pay an annual membership fee. These fees are tiered, and different members pay a different fee depending on the annual publishing revenue that their organisation receives (or publishing expenses if they don’t receive any publishing revenue).

We entered into this project recognising that we have too many membership tiers and the definition we use to size members is not consistent and can be confusing (e.g. different basis for funders than other organisations, and both are different still from subscribers to our Metadata Plus service). The idea of the membership tiers was to use publishing revenue as a proxy for “ability to pay”. We really want to develop proposals for a more equitable pricing structure. However we don’t know enough about our members’ capacity to pay to be able to model an alternative approach.

Our current lowest fee tier is $275 (USD) for any organisation with annual publishing revenue (or publishing expenses where the organisation doesn’t receive publishing revenue) of $0 to $1 million, and this is the tier where we focus our attention in our first project of the RCFS program. The difference between an organisation with revenue or expenses of USD 0, and an organisation with revenue or expenses of USD 1 million, is huge. Hardly any new members have joined in any other tier in the past several years. Of the 21,000 active members, more than 20,000 fall into the USD 275 tier - either directly (as an independent member) or indirectly (through a sponsor, where their fees would be lower). A fee structure that would fit better with the realities of our community might entail breaking our current $275 fee tier down into two or more more granular tiers.

At the moment, the majority of Crossref’s revenues come from the bottom membership tiers; 65% of membership revenues come from organisations in the USD 275 tier. We also know that many of those members (86%) are paying more in membership dues than in content registration, whereas other members have the inverse relationship between annual dues and content registration. Overall, the members in the USD 275 tier contributed 34% of Crossref’s revenue last year, and the members in the >USD 50 million tier – contributed 29%.

Members’ survey

Between April and May this year, we surveyed all independent members in the USD 275 tier. We asked questions about their operating size, how they’re funded, and how Crossref’s fees affect them. At the time of the survey, there were 8,027 members in this category. We received 1,054 responses; with a 13% response rate and broad representation globally, we are confident in the sample size. One-third of respondents said they were part of a larger organisation (such as a department or a library in a research institution).

Chart 1: Organisation revenue or funding The majority of respondents in this category (65%) have annual revenue or expense of less than USD 100,000; with 48% operating with less than USD 10,000.

Chart 2: Sources of funding When asked about the sources of funding (as an indicator of how stable these organisations might be and how readily accessible their funding is) the most frequent answer was public or government funding, and then article processing charges. If organisations relied on two sources of funding, the most common combination was public funding and article processing charges, and it was relatively rare for these organisations to have multiple sources of funding.

Chart 3: What percentage of expenses do you spend on Crossref fees?

The majority (61%) of respondents spend less than 5% of their expenses on Crossref fees. However, we have also learnt that for some volunteer-run publications, Crossref fees might be some of the only expenses they incur. Interestingly, the percentage of expenses spent on Crossref is fairly consistently spread across the continents.

Project 2: Review the basis and distribution of membership tiers

This project examines options for how we define the capacity to pay, how members are distributed across tiers, and the right levels of member fees.

There are currently a range of prices for our annual fees, based on an organisation’s ability to pay. We have used the metric of annual publishing expense or revenue as an indicator of that ability, but in some cases it doesn’t apply. As per our fee principles, we have not differentiated between organisation types. Nonprofit and commercial entities pay the same price (caveat: research funders still have a separate fee schedule, but that was intended to be temporary).

We conducted a review of other annual fee models to benchmark our approach against six like-minded organisations working in the context of scholarly communications and infrastructure. We looked at whether these organisations based their fees on one more more of the following:

Volume: e.g., research output, # of journals
Budget: e.g., total annual revenue or expenses
Relevant budget: e.g. publishing revenue
Organisation type: e.g. variance in fee based on publisher, institution, or funder
Country-level economic data: e.g., discounting based on World Bank classification, discounting based on purchasing power calculation.

Chart 4: Annual fee schedules comparisons between Crossref and CORE, DOAJ, Dryad, OA Switch-board, OpenCitations and ORCID.

There are three consistent themes among our peers: the total annual revenue and volume levels are the most common basis for membership fees among other organisations, and almost all offer discounted fees to accommodate country-based economic circumstances, utilising World Bank’s data (this is currently achieved at Crossref via the GEM program, which we have full intention of incorporating into our future fees whatever other decisions we might take). Only one other organisation uses publishing revenue or expenses as a basis for annual fees, while the potentially more transparent and less ambiguous data point of the total revenue factors in three other annual fee models.

For subscribers to our Metadata Plus service, the fee tier is selected based on whichever is the higher between their total annual revenue (including earned and fundraised, e.g. grants) or annual operating expenses (including staff and non-staff, e.g. occupancy, equipment, licences etc.). At present, we have limited understanding of the budgets of our members and how this may compare to their publishing revenues or expenses. We are looking to learn more about this as part of our annual membership data checking process, where we email all our members to ask them to confirm contact details for their organisation and the staff involved in managing their Crossref account. This year, we’re also asking all members about their organisation’s annual operating budget (or planned annual expenses) to help inform our discussions. In our case, the volume of outputs (in this case the number of items and associated metadata registered with Crossref) is recognised by the registration fees mechanism.

Consulting with organisations outside Crossref membership

To help us inform how our fees can be more equitable, it’s important to invite voices of organisations that may currently be unable to join us - due to fees or technical barriers. We hope that learning more about their circumstances will help us make sure that we improve accessibility of Crossref membership to all organisations that publish scholarly and professional works. We commissioned Accucoms to carry out a consultation on our behalf.

So far, from a handful of interviews with publishers from Nigeria, DRC, Canada and USA, we’ve learnt that while virtually all offer open access to their publications, the majority has no publishing income, and where the income is derived via APCs it’s modest and only applicable in rare circumstances. Through institutional funding and/or grants, these organisations have modest operational budgets, yet our respondents lacked clarity over the particulars. In terms of participation in professional networks and international publishing organisations, only one of the organisations we interviewed participates in DOAJ, and another is a member of OASPA, in both cases their participation is free. Among the interviewees, two organisations were interested in Crossref membership in the past but encountered technical barriers to joining.

With only five interviews to date, the consultation is still open and we’re keen to hear from more organisations that are not Crossref members but have considered our membership at some point.

Goal 2: Simplify complex fees

Projects 3 & 4: Review volume and back-year discounts for Content Registration

Along with our membership fees, our members also pay usage-based registration fees for records (scholarly works and grants) they register with us. Different content types render different costs for our members, and the fees are subject to discounts related to the age of publication and volume of registrations. Records for items older than two years have a lower fee associated with them, to help incentivise registration of such ‘back-year’ materials with great gains for the Research Nexus. There are also discounts related to the volume of transactions – which again depend on the content types.

These discounts are intended to encourage certain behaviours, specifically encouraging members to register older records in large quantities to better complete the scholarly record. Not all content types have back-year or volume discounts, and the rate of discount varies. This creates quite a complex system of fees. To the extent that the discount is successful in encouraging this behaviour, we want to preserve it, but in many cases these discounts see little to no activity.

Following the discussions of the Membership and Fees Committee, chaired by Vincas Grigas, Vilnius University, we are preparing to consult with the small number of members who currently receive volume discounts to discuss what the impact would be if we removed them.

We plan to identify and preserve the well-used back-year discounts, which encourage registration of old content, such as books, journal articles, grants. However, there are types of discounts that are hardly ever used and we are considering removing these to simplify the fees. This work will focus on the technical implications of removing some of the underused back-year discounts from the billing code and consulting with members to understand any impact .

Goal 3: Rebalance revenue sources

Project 5: Reflect increase in metadata usage and perceived shift of value toward metadata distribution

All Crossref metadata is made freely and openly available to everyone. However, some organisations may be looking for a service level agreement in delivery of the metadata, plus more regular snapshots and priority service/rate limits. For those organisations, we have an optional Metadata Plus service.

The final project is looking at the fees for this service. We are interested in making sure that Crossref metadata is available and used by the community where it can contribute to their objectives – related to discovery, analysis, integrity, and more. The optional paid service we offer aims to support the external tools that facilitate business and scholarly processes for the community. We are heartened to see that the appetite for the use of metadata seems to be growing, and the value of open research information is increasingly and widely recognised. We want to ensure that the users of metadata contribute proportionally to the maintenance of the records created and curated by our members.

Conclusion

At this point, most projects generate a lot of questions and the work is underway to deliver answers related to capacity to pay, discounts as well as available metadata usage, and barriers faced by organisations in our community.

What we have found so far is that two of our goals – simplification and equity – are often at odds with each other, and this is especially true with the $275 tier.

We welcome comments, suggestions and questions.

Meet the candidates and vote in our 2024 Board elections

Lucy Ofiesh — Tue, 24 Sep 2024 00:00:00 +0000

On behalf of the Nominating Committee, I’m pleased to share the slate of candidates for the 2024 board election.

Each year we do an open call for board interest. This year, the Nominating Committee received 53 submissions from members worldwide to fill four open board seats.

We maintain a balanced board of 8 large member seats and 8 small member seats. Size is determined based on the organisation’s membership tier (small members fall in the $0-$1,650 tiers and large members in the $3,900 - $50,000 tiers). We have two large member seats and two small member seats open for election in 2024.

We were pleased to see the diversity in candidates, with applicants from 24 countries. We also received three applications from research funders, which we specifically identified as a priority in the committee’s remit for this year. The committee was keen to prepare a diverse slate of organisation types, individual skills, and global representation.

The Nominating Committee presents the following slate.

The 2024 slate

Tier 1 candidates (electing two seats):

Katharina Rieck, Austrian Science Fund (FWF)
Lisa Schiff, California Digital Library
Ejaz Khan, Health Services Academy, Pakistan Journal of Public Health
Karthikeyan Ramalingam, MM Publishers

Tier 2 candidates (electing two seats):

Aaron Wood, American Psychological Association
Dan Shanahan, PLOS
Amanda Ward, Taylor and Francis

Please read the candidates’ statements

Every member has a vote

If your organisation is a voting member in good standing as of September 11th, 2024, you are eligible to vote.

The voting contact for your organisation will receive a ballot from eBallot, a third party election platform. You should receive your ballot by Wednesday, September 25th, and you will have until 15:00 UTC on October 29th to submit your ballot.

The election results will be announced at Crossref2024, our anual online meeting on October 29th, 2024.

If you have any questions about our election process, please contact me

Happy voting!

The myth of perfect metadata matching

Dominika Tkaczyk — Wed, 28 Aug 2024 00:00:00 +0000

In our previous instalments of the blog series about matching (see part 1 and part 2), we explained what metadata matching is, why it is important and described its basic terminology. In this entry, we will discuss a few common beliefs about metadata matching that are often encountered when interacting with users, developers, integrators, and other stakeholders. Spoiler alert: we are calling them myths because these beliefs are not true! Read on to learn why.

If you have stuck with us this far in our series, hopefully, you are at least a bit excited about the possibility of creating new relationships between the works, authors, institutions, preprints, datasets, and myriad other objects in our existing scholarly metadata. Who would not want all of these to be better connected?

We have to pause for a moment and be honest with you: metadata matching is a complex problem, and doing it correctly requires significant effort. What is worse, even if we do everything right, our matching won’t be perfect. This may be counterintuitive. Perhaps you’ve heard that matching is not a hard problem, or have encountered people surprised that a matching strategy returned a wrong or incomplete answer. Sometimes, it is obvious to a person from looking at some specific example that a match should (or should not) have been made, so they naturally assume that a change to account for this has to be simple.

Misconceptions like these can be problematic. They create confusion around matching, drive users’ expectations to unreasonable levels, and make people drastically underestimate the effort needed to build and integrate matching strategies. So let’s dive right in and debunk a few common myths about metadata matching.

Myth #1: A metadata matching strategy should be 100% correct

Anyone who has built or supported a matching strategy has likely encountered the following belief: it is possible to develop a perfect strategy, meaning one that always returns the correct results, no matter the inputs. The unfortunate truth is that while one’s aim should always be to design matching strategies that return correct results, once we move beyond the simplest class of problems or artificially clean data, no strategy can achieve this outcome. In thinking through why this is the case, some inherent constraints become obvious:

The inputs to matching are often strings in human-readable formats, which can vary wildly in their structure, order and completeness. Since they’re intended to be parsed by people, instead of machines, they’re inherently lossy and frequently unstructured, anticipating that a person can infer from the source context what is being referenced. Matching strategies, although built to make sense of unstructured data, unfortunately, don’t have the luxury of this flexibility. A strategy has to account for translating a messy, partial, or inconsistent input into a correct and structured match.

Consider, for example, the following inputs to an affiliation matching strategy:

“Department of Radiology, St. Mary’s Hospital, London W2 1NY, UK”
“Saint Mary’s Hospital, Manchester University NHS Foundation Trust”
“St. Mary’s Medical Center, San Francisco, CA”
“St Mary’s Hosp., Dublin”
“St Mary’s Hospital Imperial College Healthcare NHS Trust”
“聖マリア病院”

In order to correctly identify the organisations mentioned here, the matching strategy must be able to distinguish between different ways of representing the same institution, disambiguate multiple institutions that have similar names, and handle variant forms for the parts of each name (Saint/St./St), identify the same name in different languages (“聖マリア病院” is Japanese for “St. Mary’s Hospital”), and make assumptions about partial or ambiguous locations translating to more precise references. While a person reviewing each of these strings might be able to accomplish these tasks, even here there are some challenges. Does “St Mary’s Hosp., Dublin” refer to the hospital in Ireland or a separate hospital in one of the many cities that share this name? Should we presume that because “聖マリア病院” is in Japanese, this refers to a hospital in Japan? Would someone, by default, be aware that St. Mary’s Hospital in London is part of the Imperial College Healthcare NHS Trust, such that inputs one and five refer to the same organisation?

An additional challenge lies in the quality of the data, which in the context of matching, encompasses both the input and the dataset being matched against. In real world circumstances, no dataset is fully accurate, complete, or current and certainly not all three. As a result, there will always be functionally random differences between inputs to the strategy and the entities to be matched. A theoretically perfect matching strategy would thus need to distinguish between inconsequential discrepancies resulting from gaps, errors, and variable forms of reference and actual, meaningful differences indicating an incorrect match. As one might imagine, this would require near total knowledge of the meaning and context for all inputs and outputs, a nigh-on impossible task for any person or system!

As a consequence, no metadata matching strategy will ever be perfect. It is unreasonable for us to expect them to be. This does not mean, of course, that all strategies are equally flawed or destined to forever return middling results. Some are better than others and we can improve them over time. Which brings us to the next myth:

Myth #2: It is always a good idea to adapt the matching strategy to a specific input

Matching strategies are not static. They can - and should - be improved. There is, however, a deceptive trap that one can fall into when attempting to improve a matching strategy. Whenever we encounter an incorrect or missing result for a specific input, we treat this problem like a software bug and try to adapt the strategy to work better for it, without considering all other cases.

The more complicated reality is that the quality of matching results is controlled through a complex set of trade-offs between precision and recall that determine the kind and number of relationships created between items:

Precision is calculated as the number of correctly matched relationships resulting from a strategy, divided by the total number of matched relationships. It can also be interpreted as the probability that a match is correct. Low precision indicates a high rate of false positives, which are incorrect relationships created by the strategy.
Recall is calculated as the number of correctly matched relationships resulting from a strategy, divided by the number of true (expected) relationships. It can also be interpreted as the probability that a true (correct) relationship will be created by the strategy. Low recall means a high rate of false negatives, which are relationships that should have been created by the strategy but were not made.

The diagram depicts false negatives and false positives. The ideal outcome would be that the ellipses are identical, matched relationships are exactly the same as true relationships, and there are no false negatives or false positives. In practice, we try to make the intersection as big as possible.

The tradeoff between precision and recall roughly means that modifying the strategy to improve recall will decrease precision, and vice versa.

Imagine, for example, we received a report about a relationship that was missed by matching because of a partial, noisy, or ambiguous input. We might be tempted to resolve this issue by relaxing our matching criteria. Unfortunately, this will have a cost of a higher overall rate of false positive matches.

Conversely, if we encounter a case where the matching has returned an incorrect match, we might attempt to make the matching strategy stricter to avoid this result. We should remember, however, that this may have the consequence of causing the strategy to skip many perfectly valid matches.

The tradeoff between precision and recall. (a) A strict strategy prioritises precision over recall resulting in more false negatives. (b) A relaxed strategy prioritises recall over precision resulting in more false positives.

Striking this balance becomes even more difficult when attempting to address multiple issues at once, or considering constraints like the time and resources consumed by each aspect of the strategy. Each choice can compound the individual effects in unanticipated and expensive ways. The aim of matching ultimately then can’t be to achieve perfect results for every single case. Fixing one particular situation might not be desirable, as it can result in breaking multiple other cases. Instead, we have to find a locally optimal balance that optimises the strategy’s utility, relative to these inherent limitations. This means accepting some level of imperfection as not just inevitable, but necessary for implementing a workable strategy. When you consider all this, you might conclude that…

Myth #3: We shouldn’t do large-scale, unsupervised matching

Imperfect matching strategies, when applied automatically to real-world large datasets, might:

Fail to discover some relationships (false negatives), an outcome that may not be terribly problematic. In the worst case scenario, we have wasted a great deal of effort developing matching strategies that do not improve our metadata.
Create incorrect relationships between items (false positives), what seems like a potentially larger problem, where we have added incorrect relationships to the metadata.

Many have the instinct to avoid false positives at any cost, even if this means missing many additional correct relationships at the same time. They might come to the conclusion that if we cannot have 100% precision (see our previous myth), we simply should not allow matching strategies to act in an automated, unsupervised way on large datasets. While there might be circumstances where this belief is rational, in the context of the scholarly record, this notion is seriously flawed.

First, if you are dealing with any medium to large-sized dataset, it almost certainly contains errors, even before you apply any automated processing to it. Even if data is submitted and curated by users, they can still make mistakes, and might themselves be using automated tools for extracting the data from other sources, without your knowledge. It is thus not entirely obvious that applying an (imperfect) matching strategy to create more relationships would actually make the data quality worse.

Second, while we cannot eliminate all matching errors, we can place a high priority on precision when developing strategies, with the aim of keeping the number of incorrectly matched results as low as possible. We can also make use of additional mechanisms to easily correct for incorrectly matched results, for example doing so manually, in response to error reports.

Finally, the results of matching should always contain provenance information to distinguish them from those that have been manually curated. This way, the users can make their own decisions about whether to use and trust the matching results, relative to their use case.

By applying those additional checks, we can minimise the negative effects of incorrect matching, while at the same time reap the benefits of filling gaps in the scholarly record.

Myth #4: We can only ever guess at the accuracy of our matching results

In attempting to determine the correctness of our matching, we immediately encounter a number of inherent limitations. The sheer amount of entries in many datasets prevents a thorough, manual validation of the results, but if instead, we use too few or specific items as our benchmarks, these are unlikely to be representative of overall performance. The unpredictable nature of future data adds another wrinkle: will our matching always be as successful as when we first benchmarked it or will its performance degrade relative to some change in the data?

With so many unknowns, are we then doomed? No! We have rigorous and scientific tools at our disposal that can help us estimate how accurate our matching will be. How do we use them? Well, that is a big and fairly technical topic, so we will leave you with this little cliffhanger. See you in the next post!

Re-introducing Participation Reports to encourage best practices in open metadata

Lena Stoll — Thu, 25 Jul 2024 00:00:00 +0000

We’ve just released an update to our participation report, which provides a view for our members into how they are each working towards best practices in open metadata. Prompted by some of the signatories and organizers of the Barcelona Declaration, which Crossref supports, and with the help of our friends at CWTS Leiden, we have fast-tracked the work to include an updated set of metadata best practices in participation reports for our members. The reports now give a more complete picture of each member’s activity.

What do we mean by ‘participation’?

Crossref runs open infrastructure to link research objects, entities, and actions, creating a lasting and reusable scholarly record. As a not-for-profit with over 20,000 members in 160 countries, we drive metadata exchange and support nearly 2 billion monthly API queries, facilitating global research communication.

To make this system work, members strive to provide as much metadata as possible through Crossref to ensure it is openly distributed throughout the scholarly ecosystem at scale rather than bilaterally, thereby realizing the collective benefit of membership. Together, our membership provides and uses a rich nexus of information— known as the research nexus—on which the community can build tools to help progress knowledge.

Each member commits to certain terms, such as keeping metadata current, updating links for their DOIs to redirect to, linking references and other objects, and preserving their content in perpetuity. Beyond this, we also encourage members to register as much rich metadata as is relevant and possible.

Creating and providing richer metadata is a key part of participation in Crossref; we’ve long encouraged a more complete scholarly record, such as through Metadata 20/20, and through supporting or leading initiatives for specific metadata, like open citations (I4OC), open abstracts (I4OA), open contributors (ORCID), and open affiliations (ROR).

Which metadata elements are considered best practices?

Alongside basic bibliographic metadata such as title, authors, and publication date(s), we encourage members to register metadata in the following fields:

Example participation report for Crossref member University of Szeged

References

A list of all the references used by a work. This is particularly relevant for journal articles but the references can include any type of object, including datasets, versions, preprints, and more. Additionally, we encourage these to be added into relationships, where relevant.

Abstracts

A description of the work. These are particularly useful for discovery systems that will promote the work, and are often used in downstream analyses such as for detecting integrity issues.

Contributor IDs (ORCID)

All authors should be included in a work’s metadata, ideally alongside their verified ORCID identifier.

Affiliations / Affiliation IDs (ROR)

Members are able to register contributor affiliations as free text, but we are encouraging everyone to add ROR IDs for affiliations as the recommended best practice, as this differentiates and avoids mistyping. These two fields have newly been added to the participation reports interface in the most recent update.

Funder IDs (OFR)

Acknowledging the organisation(s) that funded the work. We encourage the inclusion of Open Funder Registry identifiers to make the funding metadata more usable. This will evolve into an additional use case for ROR over time.

Funding award numbers / Grant IDs (Crossref)

A number or identifier assigned by the funding organisation to identify the specific award of funding or other support such as use of equipment or facilities, prizes, tuition, etc. The Crossref Grant Linking System includes a unique persistent link that can be connected with outputs, activities, people, and organisations.

Crossmark

The Crossmark service gives readers quick and easy access to the current status of a record, including any corrections, retractions, or updates, via a button embedded on PDFs or a web article. Openly adding corrections, retractions, and errata is critical part of publishing, and the button provides readers with an easy in-context alert.

Similarity Check URLs

The Similarity Check service helps editors to identify text-based plagiarism through our collective agreement for the membership to access to Turnitin’s powerful text comparison tool, iThenticate. Specific full-text links are required to participate in this service.

License URLs

URLs pointing to a license that explains the terms and conditions under which readers can access content. These links are crucial to denote intended downstream use.

Text mining URLs

Full-text URLs that help researchers in meta-science easily locate your content for text and data mining.

What is a participation report?

Participation reports are are a visualization of the data representing members’ participation to the scholarly record which is available via our open REST API. There’s a separate participation report for each member, and each report shows what percentage of that member’s metadata records include 11 key metadata elements. These key elements add context and richness, and help to open up members’ work to easier discovery and wider and more varied use. As a member, you can use participation reports to see for yourself where the gaps in your organisation’s metadata are, and perhaps compare your performance to others. Participation reports are free and open to everyone - so you can also check the report for any other members you are interested in.

We first introduced participation reports in 2018. At the time, Anna Tolwinska and Kirsty Meddings wrote:

Metadata is at the heart of all our services. With a growing range of members participating in our community—often compiling or depositing metadata on behalf of each other—the need to educate and express obligations and best practice has increased. In addition, we’ve seen more and more researchers and tools making use of our APIs to harvest, analyze and re-purpose the metadata our members register, so we’ve been very aware of the need to be more explicit about what this metadata enables, why, how, and for whom.

All of that still rings true today. But as the research nexus continues to evolve, so should the tools that intend to reflect it. For example, in 2022, we removed the Open references field from participation reports after a board vote to change our policy and update the membership terms meant that all references deposited with Crossref would be open by default. And now we’ve expanded the list of fields again, adding coverage data for contributor affiliation text and ROR identifiers.

Putting it in practice

To find out how you measure up when it comes to participation, type the name of your member organisation into the search box. You may be surprised by what you find—we often speak to members who thought they were registering a certain type of metadata for all their records, only to learn from their participation report that something is getting lost along the way.

You can only address gaps in your metadata if you know that they exist.

More information, as well as a breakdown of the now 11 key metadata elements listed in every participation report and tips on improving your scores, is available in our documentation.

And if you have any questions or feedback, come talk to us on the community forum or request a metadata Health Check by emailing the community team.

Metadata schema development plans

Patricia Feeney — Mon, 22 Jul 2024 00:00:00 +0000

It’s been a while, here’s a metadata update and request for feedback

In Spring 2023 we sent out a survey to our community with a goal of assessing what our priorities for metadata development should be - what projects are our community ready to support? Where is the greatest need? What are the roadblocks?

The intention was to help prioritize our metadata development work. There’s a lot we want to do, a lot our community needs from us, but we really want to make sure we’re focusing on the projects that will have the most immediate impact for now.

Several projects were proposed, based on community demand over time. All are projects we intend to support long-term.

Projects

The projects included in the survey were:

Alternate names - We proposed adding a repeatable ‘name’ element to allow for names that aren’t separated by given/family/surname.
Updates to funding data -this update will be released in the near future and includes:
- Expand ROR support - Allow members to supply ROR ID instead of funder ID in funding data and grant records.
- Include Grant DOIs in funding metadata.
Publication typing in citations - Support citation type in citation metadata (for example article, preprint, data, software, etc.).
Expand contributor role support - Allow multiple contributor roles to be provided per contributor and add support for external vocabularies (like CRediT)
Expand abstract support - We currently require all abstracts to be formatted using JATS. We will be adding new abstract formats, including BITS and ONIX (which have been requested), as well as a generic abstract format (non-JATS).
Statements - Add support for free text statements such as data availability, acknowledgments, funding, and conflict of interest.
Contributor identifiers - Accept contributor identifiers such as ISNI (in addition to ORCID, which is already supported).
Conference event IDs - Identifiers for conference events.

What’s next?

There is a clear preference for publication types in citations and abstract markup, expanded support for multilingual metadata, followed by expanding contributor roles to support multiple roles and the CRediT taxonomy. The results have helped us prioritize our work and we’re advancing several projects soon based on our readiness to move forward.

First up is publication typing in citations and statements - we hope to be able to make this ready for registration in the coming months, but want to confirm a few things first, primarily the list of ‘types’ to apply to citations, so please review and comment: Metadata updates in need of feedback July 2024

We also have been discussing expansions to our support for preprints metadata with our Preprints Advisory Group and have a number of preprint-specific updates that will be rolled out in the coming months as well, including support for versions and status. These proposed changes are also available for comment.

And finally, we will be expanding support for contributor roles to include multiple roles per contributor, as well as adding support for the CRediT taxonomy. This update is yet to be scheduled but we do have the inputs and output planning done and welcome any comments on this as well.

We will also be continuing work on other projects highlighted in the survey that aren’t quite ready to go:

Multilingual metadata: Support for multilingual metadata in particular is very important and will require a fairly significant technical effort, so we want to be sure we get this right - at minimum we need to include repeatable fields flagged with language metadata for most items, there may be other considerations as well such as the scope of languages supported.

As we develop new metadata segments we’re keeping language metadata in mind, but I’d like to form a short-term working group to help shape this update - this group will be focused on the details of supporting multilingual metadata in our inputs and outputs, so conversations will be very XML and JSON heavy. If you are interested and available please contact pfeeney@crossref.org.
Abstract markup: we are currently in the research phase of this project but will be proposing updates and asking for input this fall. At the moment support for BITS and ONIX abstracts have been requested, as well as an agnostic format.
Expansion of name and contributor ID support: work is under way for this as well, and I should have inputs and outputs for feedback in the coming months.

We anticipate more developments and requests for feedback in the future as we still have other projects from the list above to get to. I’ve opened up a ‘Metadata Development’ section in our Community Forum to invite discussion and will be kicking off a renewed Metadata Interest Group in the fall.

Crossmark community consultation: What did we learn?

Martyn Rittman — Tue, 02 Jul 2024 00:00:00 +0000

In the first half of this year we’ve been talking to our community about post-publication changes and Crossmark. When a piece of research is published it isn’t the end of the journey—it is read, reused, and sometimes modified. That’s why we run Crossmark, as a way to provide notifications of important changes to research made after publication. Readers can see if the research they are looking at has updates by clicking the Crossmark logo. They also see useful information about the editorial process, and links to things like funding and registered clinical trials. All of this contributes to what we call the integrity of the scholarly record.

Crossmark has been around a long time and the context around it is constantly changing. It last had a major update in 2016 and in 2020 we removed fees for its use.

The past few years have seen a more intense focus on research integrity among the scholarly communications community, leading to more retractions and calling out large-scale manipulation of editorial processes. At the same time, we haven’t seen an increase in the uptake of Crossmark, which is still used by only a minority of our members. We would like to know why the uptake is low and whether there is more we can do in this area. To dig into this, in the first part of 2024 we reached out to members of our community.

What did we do?

We wanted to learn about attitudes towards Crossmark and related aspects of research integrity. This was done in several ways:

Structured interviews with eight of our members.
Round tables at Crossref LIVE events in Bogota and Nairobi
Surveying a selection of our members, which led to 94 responses.

The topics we asked about were related to how post-publication updates are made and communicated, and which metadata demonstrates good practice.

We are extremely grateful to the members who contributed. They provided valuable feedback and have helped to shape the future of Crossmark and our approach to the integrity of the scholarly record.

What did we find?

Across the various groups there were a few common themes, which fell into several areas.

Communication of updates is highly valued, and seen as the most important role that Crossmark can play. Some of those we spoke to would like readers to see if there is an update as soon as a page opens, without having to open a popup. This could be done by having a logo that changes colour, shape, or size.
Conversely, not as much enthusiasm was shown for the metadata assertions. These are additional fields that can be displayed to readers in the Crossmark popup. There wasn’t a strong consensus on which commonly-made assertions are the most important for research integrity.
There is diversity in attitudes towards making updates to published works, what research integrity means, and approaches to workflows for updates. Even within a single organisation, a number of different workflows and multiple staff members might be called on to update published research. This makes things complex and means that it can be difficult to fit Crossmark in.
There are technical challenges to getting started with Crossmark. Those responsible for implementing Crossmark are often technical staff who struggle with the documentation we provide in English. There is also no plugin for OJS, a widely-used open source editorial software. It is more difficult to deposit Crossmark metadata for books than journal articles, and many article types don’t permit Crossmark metadata at all. On the other hand, those who successfully installed Crossmark found it easy to use and low-maintenance.

Overall, it seems that Crossmark still has an important role to play but there are changes and improvements we can make.

What’s next?

Here are the main areas we intend to follow up on in the coming months.

Implementation

We need to look at how to make implementation more straight-forward. Can we provide multilingual documentation, plugins, run workshops or webinars, or make changes to Crossmark to lower the barrier to entry?

Understanding workflows

Can we collaborate with our members and other organisations to reach a better understanding of how to update published works? Are there alternative workflows we need to support? Have we made it too difficult to understand and implement the options we currently have?

While updates are always likely to be rare, we want to help members understand the benefits of making them. We talked to some members who were proud of never having published a retraction or correction, which left us wondering whether they are missing legitimate opportunities to correct the scholarly record. We also know that for some members and many work types (preprints, for example), updates are made without a separate published notification. Can we better understand the role that the published updates play and communicate updates even if there isn’t a published notice?

Ongoing feedback

Clearly one size doesn’t fit all when it comes to implementing and communicating updates. We need to find ways of keeping in touch with the community to test new solutions with as broad a range of members as possible. We want to avoid catering to a minority and leaving others struggling to find ways to implement a solution.

Custom metadata?

Is there an ongoing need for metadata assertions? Many of the assertions currently made are possible as standard metadata and others could be included in our deposit schema. We want to consider removing the option to add assertions. This needs more feedback from the community, especially those who currently make use of assertions.

Redesign the UI

Crossmark doesn’t have the recognition with readers we would like. Is there a way we can redesign it to make it more associated with Crossref and accurate metadata? We intend to explore different designs, and test them with members and readers.

Celebrating five years of Grant IDs: where are we with the Crossref Grant Linking System?

Kornelia Korzec — Mon, 01 Jul 2024 00:00:00 +0000

We’re happy to note that this month, we are marking five years since Crossref launched its Grant Linking System. The Grant Linking System (GLS) started life as a joint community effort to create ‘grant identifiers’ and support the needs of funders in the scholarly communications infrastructure.

The system includes a funder-designed metadata schema and a unique link for each award which enables connections with millions of research outputs, better reporting on the research and outcomes of funding, and a contribution to open science infrastructure. Our first activity to highlight the moment was to host a community call last week where around 30 existing and potential funder members joined to discuss the benefits and the steps to take to participate in the Grant Linking System (GLS).

Some organisations at the forefront of adopting Crossref’s Grant Linking System presented their challenges and how they overcame them, shared the benefits they are reaping from participating, and provided some tips about their processes and workflows.

The funding organisations whose experiences were shared included Wellcome, FCT (Foundation for Science and Technology, Portugal), and NWO (Dutch Research Council). They were joined by a new group of foundations, research councils, and private research funders from around the world—from Kenya to Singapore to Estonia—to have a first introduction to the GLS and connect them with colleagues who are further along on their journey.

We also heard about tools such as a new open source Crossref plugin for the Fluxx platform, grant management systems with in-built Crossref integrations such as ProposalCentral, Europe PMC GrantFinder which was first to implement the GLS on Wellcome’s behalf and hosts their grants, and one of the first publishers, eLife to start referencing Crossref grant links in their publications both online and in the open metadata for others to retrieve.

Read on for further information or watch the recording of the event.

What is the Crossref Grant Linking System?

The Crossref Grant Linking System, conceptualised in 2017, and launched in 2019, captures and helps clarify funding relationships for scholarly outputs. Thanks to interconnectedness with the 160 million metadata records collected and curated by Crossref members, it enables funders as well as scholars to track and analyse funding patterns and evaluate programmes, and it supports assertions about the integrity of scholarly records.

Features of the GLS

Globally unique persistent link and identifier for each grant
Connected with 160 million published outputs
Funder-designed metadata schema, including project, investigator, value, and award-type information
Programmatic or no-code methods to send metadata
- Thanks to the Gordon and Betty Moore Foundation who funded development of the online grant registration form
Open search and API for all to discover funding outcomes; all metadata is distributed openly to thousands of tools and services
Crossref-hosted landing pages
A global community of ~50 funder advisors and 35+ funders already in the Grant Linking System
Membership of Crossref; influence the foundational infrastructure powering open research

The last five years has seen the GLS grow through membership, metadata, and community contributions.

The momentum for this programme is building - as illustrated by increasing numbers of metadata records (and related relationships we’re seeing). The 35 funder members represent over 100 funding programmes and have created 125,000 grant records already.

During last week’s call, it was helpful to hear from the community what they see as key benefits of the Crossref Grant Linking System:

Meaningfully delivering on and supporting Open Science policies and mandates, and contributing ‘their bit’ to the transparency of the evidence trail in the scholarly ecosystem.
Reporting and evaluating the funding programmes, essential for the public funders who need to demonstrate the value for money in allocating their funds and other support.
Supporting a more holistic assessment of scholarship and scholars, especially as and when metadata becomes included with a full array of outputs, not limited to books and articles.

How the Crossref Grant Linking System supports Open Science policy

Since 2020, all the grant records are openly available through our REST API which is queried more than 1.8 billion times every month so these metadata records are distributed to thousands of systems across the research enteprise. In a 2022 blog, Ed Pentz and Ginny Hendricks laid out guidelines for research funders to meet open science guidelines using existing open infrastructure such as Crossref, ORCID, and ROR. Syman Stevens, a grantmaking and private philanthropy consultant, highlighted on the call that the funders he works with are increasingly interested in ways to deliver on their open science policy and that participation in the GLS is a tangible thing they can do to meet this goal.

As part of its open science policy, NWO will start participating in the Crossref Grant Linking System from July 2025. Research funders are a part of the scholarly communications system; we not only provide the funding to do the actual research but can also be the authoritative source of data about the projects we have funded and the outputs arising from that funding. Increasingly, all these elements – grants, researchers, outputs - are linked with metadata and unique identifiers to ensure that research is findable and accessible.

– Hans de Jonge, Director of Open Science NL, part of the Dutch Research Council (NWO)

How funders leverage the Grant Linking System in their reporting and assessment

Looking back to the origins of the system, it’s important to recognise the work of the initial working groups. Through their contribution, funders helped design the initial metadata schema for grants as well as establish the governance and fees for this service, and our Advisory Group continues to inform further developments. In this way, the Grant Linking System enables the needs and wishes of funders to contribute and see their data as part of the wider ecosystem.

An excellent example of that synergy in action is the use case presented by Cátia Laranjeira, manager of the PTCRIS programme at the Foundation for Science and Technology, Portugal (FCT). PTCRIS is the Foundation’s integrated national information ecosystem that supports scientific activity management. Cátia reflected on the relative fragmentation of spaces where the scientific outputs are found, and PTCRIS’s ambition for aggregating metadata in one place to be able to trace and evaluate programmes in light of the related outputs. At the start of the programme, they identified lack of a persistent identifier for grants as a major shortcoming of the system. Crossref GLS naturally fits in with their goals.

The initiative by FCT to assign unique DOIs to national public funding through Crossref is a game-changer for open science, linking funding directly to scientific outcomes and boosting transparency. Join us in this effort—let’s make every grant count and ensure open access to research information!"

– Cátia Laranjeira, PTCRIS Program Manager at Fundacao para a Ciencia e a Tecnologis (FCT Portugal)

FCT initially piloted a small subset of their grants (approximately 6,000 recent awards) at the end of 2023. Cátia pointed to researchers’ keen participation in this programme as one of its successes – and thanks to the word of mouth, FCT has already been approached by researchers requesting unique Crossref links for their grants! This appetite for grant IDs will soon be more fully satisfied, as FCT is readying to register all of their grants with Crossref, to enable further insights into funding and outcome flows, supporting them in demonstrating the value for money for the public resources they manage. Via interfaces for grant management and standardised online CVs, the system is also enabling researchers to use the system in their own future reporting and career development.

In the ensuing discussion, Rachel Bruce of UKRI mentioned that she’s hopeful that GLS will help funders ‘close the loop’ on more holistic reward and recognition, allowing for inclusion of evidence for a broader set of outputs in those processes.

How the community is working to integrate open infrastructure

Melissa Harrison, Team Leader at EMBL-EBI, manages Europe PMC and a complementary data science team, who were part of the initial FREYA project – supporting infrastructure delivery for unique identifiers for grants. The team has been adding grant records to Crossref on Wellcome’s behalf since 2019. Melissa highlighted the shortcomings of internal award numbers, which don’t tend to be understood outside of the ecosystem where they are produced (that is the funder’s administrative system), are almost certainly not unique, and don’t resolve to or connect with anything in the wider ecosystem. Therefore internal award numbers can’t signify relationships with other outputs or assets in the wider world. By contrast, Crossref’s Grant IDs are unique, persistent, resolvable, and interrelated with other Crossref metadata, whilst being retrievable for other systems to link to too.

Persistent identifiers for grants was the next logical step after identifiers for funders - open metadata registered with a PID in a central service like Crossref is invaluable to build the full picture of the research enterprise.

– Melissa Harrison, Team Leader, Literature Services at EMBL-EBI)

Ease of execution is important for scaling the Grant Linking System, and enabling its use in a diverse set of circumstances in the open science ecosystem. Altum was the trailblazer, first integrating its grant management platform Proposal Central with GLS. It was good to hear that others are now joining the integration efforts. Syman Stevens talked about the recent work initiated by Joe McArthur at OA Works, to develop a simple, open-source plug-in for any of the major grant management systems, to enable funders to deposit their grant metadata with Crossref GLS with a click of the button. Syman demonstrated the resulting interface in Fluxx, that allows for creating a record and sending grant metadata to Crossref as part of the regular grant management within the platform. He pointed out that, while this integration was developed for Fluxx, all code and documentation is openly available on GitHub and this can potentially be forked or adapted as necessary for reuse in other grant management systems.

It is heartening that others in the community are seeing such a need for this that they’re funding and creating their own tools to advance participation and use of the GLS.

Finally, Fred Atherden, Head of Production Operations at eLife, presented how they include Crossref grant identifiers in publication metadata for the version of record of the works published on their platform. eLife is the first publisher to fully integrate Crossref grant identifiers both within the article display and in the metadata. Fred shared that in addition to collecting the data from the authors, eLife also attempts matching, albeit using very restrictive methodology, to enable more grant metadata in their publication records. They recognise that so far there are very few publishers including persistent links for grants in this way, and talked about plans to start collecting and including this data further upstream, and including them in the future for reviewed preprints.

Acknowledgements and how to participate in the GLS

Reflecting on the last five years, thanks must go to the >35 funders who are already participating (see logo mashup below), to our current volunteers and to those partners working to promote and make use of the Grant Linking System. We also acknowledge that the GLS would not have been possible without the Crossref board members at the time, our staff including alumni Josh Brown, Jennifer Kemp, Rachael Lammey, and Geoffrey Bilder, or without the early dedicated time and input from the following people and organisations on our working groups for governance and fees, and for metadata modelling:

Yasushi Ogasaka and Ritsuko Nakajima, Japan Science & Technology Agency
Neil Thakur and Brian Haugen, US National Institutes of Health
Jo McEntyre and Michael Parkin, Europe PMC
Robert Kiley and Nina Frentop, Wellcome
Alexis-Michel Mugabushaka and Diego Chialva, European Research Council
Lance Vowell and Carly Robinson, OSTI/US Dept of Energy
Ashley Moore and Kevin Dolby, UKRI (Research Councils UK / Medical Research Council)
Salvo da Rosa, Children’s Tumor Foundation
Trisha Cruse, DataCite

To learn more about the Crossref Grant Linking System, the best place to start is our service page. And for the next step, please reach out to us for a conversation about any questions specific to your organisation and any questions that may need to be addressed in order to enable your full participation.

Grant DOIs enhance the discovery and accessibility of funded project information and are one of the important links in a connected research ecosystem. I’m grateful and proud to contribute to the robustness and interconnectedness of the research infrastructure. Few funders are currently participating in the Crossref Grant Linking System, and I encourage others to consider doing so. This adoption follows the “network effect,” where the value and utility increase as more people participate, encouraging even wider adoption.

– Kristin Eldon Whylly, Senior Grants Manager and Change Management Lead at Templeton World Charity Fund (TWCF)

You can email me via feedback@crossref.org or set up a call with me when it suits you (you can overlay your own calendar using the toggle at the top right). We look forward to welcoming even more funders and to see those relationships in the open science infrastructure grow even further in the coming years.

The anatomy of metadata matching

Dominika Tkaczyk — Thu, 27 Jun 2024 00:00:00 +0000

In our previous blog post about metadata matching, we discussed what it is and why we need it (tl;dr: to discover more relationships within the scholarly record). Here, we will describe some basic matching-related terminology and the components of a matching process. We will also pose some typical product questions to consider when developing or integrating matching solutions.

Basic terminology

Metadata matching is a high-level concept, with many different problems falling into this category. Indeed, no matter how much we like to focus on the similarities between different forms of matching, matching affiliation strings to ROR IDs or matching preprints to journal papers are still different in several important ways. At Crossref and ROR, we call these problems matching tasks.

Simply put, a matching task defines the kind or nature of the matching. Examples of matching tasks are bibliographic reference matching, affiliation matching, grant matching, or preprint matching.

Every matching task has an input, which is all the data that is needed to perform the matching. Input data can come in many shapes and forms, depending on the matching task. For example, all of the following could be inputs to a matching task:

Department of Molecular Medicine, Sapporo Medical University, Sapporo 060-8556, Japan

<fr:program xmlns:fr="http://www.crossref.org.pluma.sjfc.edu/fundref.xsd" name="fundref">
<fr:assertion name="fundgroup">
<fr:assertion name="funder_name">
European Union's Horizon 2020 Research and Innovation Program through Marie Sklodowska Curie
<fr:assertion name="funder_identifier">http://dx.doi.org.pluma.sjfc.edu/10.13039/501100000780</fr:assertion>
</fr:assertion>
<fr:assertion name="award_number">721624</fr:assertion>
</fr:assertion>
</fr:program>

Everitt, W. N., & Kalf, H. (2007). The Bessel differential equation and the Hankel transform. Journal of Computational and Applied Mathematics, 208(1), 3–19.

{
"title": "Functional single-cell genomics of human cytomegalovirus infection",
"issued": "2021-10-25",
"author": [
{"given": "Marco Y.", "family": "Hein"},
{"given": "Jonathan S.", "family": "Weissman", "ORCID": "http://orcid.org/0000-0003-2445-670X"}
]
}

Every matching task also has an output. For our purposes, this is almost exclusively zero or more matched identifiers. In the context of a specific matching task, output identifiers may be of a specific type (e.g. we might match to a ROR ID, and never to an ORCID ID). In some cases, there can be a certain target set as well (i.e. matching only to DataCite DOIs). The output identifiers can have different cardinality depending on the task, meaning that the matching task might allow for zero, one, or more identifiers as a result of matching to a single input.

A matching strategy defines how the matching is done. Multiple strategies can exist for a specific matching task. Compound strategies can run other strategies and combine their outcomes into a single result.

In some cases, we may also want the matching strategy to output a confidence score for each matched identifier. A confidence score represents the degree of certainty or likelihood that the matched identifier is correct, typically expressed as a value between 0 and 1. This score may help with post-processing or further interpretation of the results.

To summarise, the anatomy of the matching task can be diagrammed as follows:

How to specify a matching task

Whenever we plan the development or integration of a matching solution, it is good to begin by answering a few basic questions:

What problem do we plan to solve with our matching task? What would we call our matching task and how would we describe it?
What do we expect as the input for this matching task? Which input formats do we need to be able to accept? What information do we expect to find in this input?
What kind of identifiers should be output? Is there a target set of identifiers? Can our matching output zero/one/or multiple identifiers, and under what conditions might that occur?

These sound fairly simple, but the answers to these questions can be remarkably complex. Once one tries to apply these concepts to real-world problems, they might encounter several non-obvious challenges.

For example, one common concern is at what level we should define each matching task. Consider the following problems:

Matching bibliographic reference strings to DOIs. Example input:

Everitt, W. N., & Kalf, H. (2007). The Bessel differential equation and the Hankel transform. Journal of Computational and Applied Mathematics, 208(1), 3–19.

Matching structured bibliographic reference to DOIs. Example input:

{
volume: "208",
author: "Everitt",
journal-title: "J. Comput. Appl. Math.",
article-title: "The Bessel differential equation and the Hankel transform",
first-page: "3",
year: "2007",
issue: "1"
}

Are those discrete matching tasks (unstructured reference matching vs. structured reference matching), or are they the same task (reference matching) that can accept different types of inputs (unstructured or structured)?

Similarly, let’s compare the following tasks:

Matching affiliation strings to ROR IDs. Example input:

Department of Molecular Medicine, Sapporo Medical University, Sapporo 060-8556, Japan

Matching funder names to ROR IDs. Example input:

Alexander von Humboldt Foundation

Are these different matching tasks (affiliation matching vs. funder matching), or the same task with different inputs (organisation matching)?

Defining the boundaries of a matching task can also be difficult. Consider, for example, the need to obtain ROR IDs for organisations mentioned in the acknowledgements section of a full-text academic paper. To begin, one may first extract the acknowledgement section from the full text, then run something like a named entity recognition (NER) tool to isolate the organisation names from the extracted text, and finally match these names to ROR IDs. Is this entire process matching, with the input being the full text of a paper? Or perhaps matching starts with the acknowledgement section as the input? Instead, is it only the last phase, where we try to match the extracted name to the ROR ID, that constitutes the matching task, with the extraction phases being completely separate processes?

There are also important questions related to the expected behaviour of a matching strategy. Consider, for example, developing an affiliation matching strategy where we define our input as “an affiliation string”. What should happen when the strategy gets something else on the input, for example, song lyrics? Perhaps the strategy should simply return no matches, or an error, or we could say that in such a situation the behaviour is undefined and it simply doesn’t matter what is returned. But what should happen if in this input we have the lyrics of Street Life by Roxy Music, a song that mentions the names of a few universities that happen to have ROR IDs?

It is likewise important to consider what should happen if different parts of the input match to different identifiers, like in the following example:

Department of Haematology, Eastern Health and Monash University, Box Hill, Australia

Here, “Eastern Health” matches to https://ror.org/00vyyx863 and “Monash University” to https://ror.org/02bfwt286. Should the matching strategy return all the identifiers, one of them (if so, which one?), or nothing at all?

Similar questions arise when it is possible to match to multiple versions (or duplicates) in the target identifier set. This can happen, for example, in the context of bibliographic reference matching or preprint matching. Multiple matches may occur when there are different editions, reprints, or variations of the same publication in the target dataset, each with its own unique identifier.

If you are waiting for an answer to these questions, we unfortunately must disappoint you here. These can only be answered in the context of a specific problem, considering who the users are and what it is they need and expect.

Did you notice any other subtleties related to metadata matching and its concerns? Are there other non-obvious questions that should be considered when planning to develop or integrate metadata matching strategies? Let us know—we’d love to hear from you!

Drawing on the Research Nexus with Policy documents: Overton’s use of Crossref API

Luis Montilla — Sat, 15 Jun 2024 00:00:00 +0000

Update 2024-07-01: This post is based on an interview with Euan Adie, founder and director of Overton._

What is Overton?

Overton is a big database of government policy documents, also including sources like intergovernmental organisations, think tanks, and big NGOs and in general anyone who’s trying to influence a government policy maker. What we’re interested in is basically, taking all the good parts of the scholarly record and applying some of that to the policy world. By this we mean finding all the documents, finding what’s out there, collecting metadata for them consistently, fitting to our schema, extracting references from all the policy documents we find, adding links between them, and then we also do citation analysis.

What do you mean by the good parts of the scholarly record?

What I mean by the good parts of the scholarly record is, from a data perspective, having persistent open metadata for items on different stable, interoperable platforms and being able to build up layers of data to suit specific use cases. That’s a better approach than trying to do everything in a silo here and a silo there and trying to do stuff bit by bit or in a hundred different ways.

There’s also a bad part, which is less to do with metadata and more around citation analysis and responsible metrics. With all this data… as the famous Spiderman quote goes… with great power comes a great responsibility: once you start systematically collecting this data, it’s very easy to fall into the trap of thinking that if we can put numbers on it, and then maybe we could start reading meaning into those numbers, and then it spirals out of control. So the idea for Overton was: can we take the system, some of the infrastructure and apply those ideas? But then come at it already knowing where the later pitfalls are and try to avoid them.

What is your main use of Crossref resources?

We rely heavily on Crossref to link policy documents to the scholarly record. The question we’re trying to answer is: does this government document cite academic work? We work a lot with universities, think tanks, and IGOs. They’re asking where is the research we produce ending up? Is it being used by the government? In some countries, like the UK, there’s a big impact agenda where it’s quite important to demonstrate that for government funding. In the US as well, state universities for example aim to impact the local policy environment. Right? Are we producing things that went on to change life for local residents for the better? And that’s really what we’re trying to support. And so that’s one of the main use cases of the database.

Can you tell us a little bit more about the story of Overton, how did this idea start?

It really came from two things. The first one is that I’d always been interested in this area and before Overton, I founded a company called Altmetric.com, which was looking at kind of broader impact metrics for papers. And we looked at Twitter, and news, and blogs, and other things, including policy. But policy wasn’t a primary focus.

When I left Altmetric two things were happening in the UK – not that everything is about Brexit, but Brexit was happening, and then COVID happened as well. And in both cases, I think it just drove home to me that other people seemed to be very interested in the evidence that the government has used to make decisions. Be they good decisions like some of the evidence based initatives in COVID or bad decisions like Brexit. So, how can you find out what it was? And it is actually very difficult to do. You can’t really track back how this decision was made. I thought that there is a growing need for that kind of impact analysis. So the second thing was, can we do something that helps make it easy to see what evidence goes into policy? The scholarly evidence but also the other kind of policy influence that goes into any document or discussion.

What are the main challenges that you face when you are trying to retrieve these policy documents?

Well, first is another thing that the scholarly record does well, which is persistence. We have CLOCKSS and all the dark archives¹. So the whole idea is that if you have a DOI, if something moves, we can track it and it maintains the ID, and even if the publisher goes bust it’ll never disappear. For citing it, then there’s always going to be a copy of it somewhere available even if it’s in a library or a dark archive.

One of the biggest challenges with policy documents is that kind of persistence doesn’t exist… There are a lot of statistics about link rot², and they hold true for policy documents as much as anywhere else. Every year a percentage of the links everywhere basically break because websites are redesigned or a government changes, it’s even worse because it can be by design. If you think about it, a new government comes into power, they change… let’s say the Department of Agriculture and they merge it with the Department of Fisheries. That would refer to a completely new third thing. And the other two departments disappear or they start linking off, like, redirecting or whatever.

One of the challenges is just keeping track of all the changes in the landscape and constantly trying to stay on top of the data. And that’s a big part of what we do. Another challenge for us, and I think about it compared to journals, when you cite something in a scholarly document, you cite it in a given style, but there are no standards for referencing styles in policy documents. So even in the same document, we can see, like, four or five different ways of referring to something, and sometimes they’re missing important data and sometimes they’re not. And it means when we’re using Crossref search, we usually have much more unparsable text.

How has your experience been so far using our Crossref API or our services in general?

It’s been great. I would happily say this anywhere, I always talk about the Crossref API as being one of the best examples of a well-done scholarly infrastructure API. It’s well-documented. It’s fast. It’s clear. The rate limits are clear. It’s up when it should be up. I like that you can trust it. So the technical aspect is great. From an organisational aspect, in contrast with a lot of infrastructure in the scholarly world that you don’t know if it’s even going to be there in a given time, Crossref is pretty stable.

What would you say are the main challenges or things that we can improve in the future? What other expectations or suggestions do you have?

It depends, if we’re talking about how the service could be improved versus how the data could be improved. Data-wise, and I appreciate this is a publisher problem, not a Crossref one, but, we still have to pull other data from OpenAlex, for example, for things like affiliations just because it’s missing from so many articles. And then equally things like ORCID for authors. And in fact also disambiguation in general. This is a huge problem that either the user doesn’t solve or you end up using a hundred different author disambiguation systems. I don’t know if there’s necessarily something Crossref wants to get into, but there’s definitely not something out there generally accepted already.

Another kind of improvement I see is to make sure that changes in one API are reflected in the other, and they don’t get out of sync. When somebody updates their ORCID record, I’d like it reflected in the Crossref record if we’re using that as the “canonical” metadata record for the DOI. Retrospectively enriching records.

I think it’s harder than I expected to just find preprints because you can’t simply use the item type but I understand that this is maybe a bigger issue. So maybe it’s not for a short time.

Finally, this is very specific, but we experienced friction when going from the snapshots to having something useful, either in Elasticsearch or in, like, Postgres. It might be nice to have some open-source scripts to download and process everything, convert it to relational tables, or send it to an Elasticsearch cluster or something.

Rebalancing our REST API traffic

Stewart Houten — Tue, 04 Jun 2024 00:00:00 +0000

Since we first launched our REST API around 2013 as a Labs project, it has evolved well beyond a prototype into arguably Crossref’s most visible and valuable service. It is the result of 20,000 organisations around the world that have worked for many years to curate and share metadata about their various resources, from research grants to research articles and other component inputs and outputs of research.

The REST API is relied on by a large part of the research information community and beyond, seeing around 1.8 billion requests each month. Just five years ago, that average monthly number was 600 million. Our members are the heaviest users, using it for all kinds of information about their own records or picking up connections like citations and other relationships. Databases, discovery tools, libraries, and governments all use the API. Research groups use it for all sorts of things such as analysing trends in science or recording retractions and corrections.

So the chances are high that almost any tool you rely on in scientific research has somewhere incorporated metadata through us.

Optimising performance

For some time, we’ve been noticing reduced performance in a number of ways, and periodically we have a flurry of manually blocking/unblocking IP addresses from requesters that are hammering and degrading the service for everyone else, and this is of course only minimally effective and very short term. You can always watch our status page for alerts. This is the current one about REST API performance: https://status-crossref-org.pluma.sjfc.edu/incidents/d7k4ml9vvswv.

As the number of users and requests has grown, our strategies for serving those requests must evolve. This post discusses how we’re approaching balancing the growth in usage for the immediate term and provides some thoughts about things we could try in the future on which we’ll gladly take feedback and advice.

Load balancing

In 2018, we started routing users through three different pools (public, polite, and plus). This coincided with the launch of Metadata Plus, a paid-for service with monthly data dumps and very high rate limits. Note that all metadata is exactly the same and real-time across all pools. We also, more recently, introduced an internal pool. Here’s more about them:

Plus: This is the aforementioned premium option; it’s really for ‘enterprise-wide’ use in production services and is not really relevant here.
Public: This is the default and is the one that is struggling at the moment. You don’t have to identify yourself and, in theory, we don’t have to work through the night to support it if it’s struggling (although we often do). Public currently receives around 30,000 requests per minute.
Polite: Traffic is routed to polite simply by detecting a mailto in the header. Any system or person including an email is being routed to a currently-quieter pool, this means we can always get in touch for troubleshooting (and only troubleshooting). Polite currently receives around 5,000 requests per minute.
Internal: In 2021, we introduced a new pool just for our own tools where we can control and predict the traffic. Internal currently receives around 1,000 requests per minute.

The volumes of traffic across public, polite and internal pools are very different and yet each pool has always had similar resources. The purpose of each of these pools has been long-established but our efforts to ask the community to use polite by default have not been particularly successful and it is clear that we don’t have the right balance.

The internal pool has been dedicated to our internal services that have predictable usage and that have requests that are not initiated by external users. The internal pool has previously included reference matching but not Crossmark, Event Data, or search.crossref.org, which all use the polite pool instead, along with the community. We have the capacity on the internal pool to shift all of this “internal” traffic across, and in doing so we will create more capacity for genuine polite users and redefine what we consider to be “internal”.

Creating more capacity on polite will also give us the opportunity to load-balance requests to both polite and public across the two pools. We are at a point where we cannot eke more performance out of the API without architectural changes. In order to buy ourselves time to address this properly, we will modify the routing of polite and public and evenly distribute requests to the two pools 50/50.

The public and polite pools have equal resources at the moment yet handle very different volumes of traffic (30,000 req/min vs 5,000 req/min), and with the proposed changes to internal traffic the polite pool would handle a fraction of this. The result would look something like 31,000 req/min evenly distributed across public and polite.

Rate limiting

Our rate-limiting also needs review. We track a number of metrics in our web proxy but only deny requests on one of them - the number of requests per second. On public and polite we limit each IP address to sending 50 req/sec and if this rate is exceeded users are denied access for 10 seconds. These limits are generous and we cannot realistically support this volume of request for all users of the public or polite API.

However, when requests are taking a long time to return, we potentially have a separate problem of high concurrency as hundreds of requests could be sent before the first one has returned. We intend to identify and impose an appropriate rate limit on concurrent requests from each IP to prevent a small number of users from disproportionately affecting all users with long-running queries.

Longer-term

So, in the short-term we will revise our pool traffic as described above. We’ll do that this week. Then we will review the current rate limits and reduce them to something more reasonable for the majority of users. And we’ll identify and introduce a rate limit for concurrent requests from each user.

Longer-term, we need to rearchitect our Elasticsearch pools so that we can:

Reduce shard sizes to improve performance of queries
Balance data shards and replicas more evenly
Optimise our instance types for our workload

Want to help?

Thanks for asking!

Firstly, please, everyone, do always put an email in your API request headers - while the short term plan will help stabilise performance, this habit will always help us troubleshoot e.g. we can always contact you instead of blocking you!

Secondly, we know many of you incorporate Crossref metadata, add lots of value to it in order to deliver important services, and also develop APIs of your own. We’d love any comments or recommendations from those of you handling similar situations on scaling and optimising API performance. You can comment on this post which is managed via our Discourse forum. We’ll also be adding updates to this thread as well as on status.crossref.org. If you’d like to be in touch with any of us directly, all our emails are firstinitiallastname@crossref.org.

Metadata matching 101: what is it and why do we need it?

Dominika Tkaczyk — Thu, 16 May 2024 00:00:00 +0000

At Crossref and ROR, we develop and run processes that match metadata at scale, creating relationships between millions of entities in the scholarly record. Over the last few years, we’ve spent a lot of time diving into details about metadata matching strategies, evaluation, and integration. It is quite possibly our favourite thing to talk and write about! But sometimes it is good to step back and look at the problem from a wider perspective. In this blog, the first one in a series about metadata matching, we will cover the very basics of matching: what it is, how we do it, and why we devote so much effort to this problem.

What is metadata matching?

Would you be able to find the DOI for the work referenced in this citation?

Everitt, W. N., & Kalf, H. (2007). The Bessel differential equation and the Hankel transform. Journal of Computational and Applied Mathematics, 208(1), 3–19.

We bet you could! You might begin, for example, by pasting the whole citation, or only the title, into a search engine of your choice. This would probably return multiple results, which you would quickly skim. Then you might click on the links for a few of the top results, those that look promising. Some of the websites you visit might contain a DOI. Perhaps you would briefly compare the metadata provided on the website against what you see in the citation. If most of this information matches (see what we did there?), you would conclude that the DOI from that website is, in fact, the DOI for the cited paper.

Well done! You just performed metadata matching, specifically, bibliographic reference matching. Matching in general can be defined as the task or process of finding an identifier for an item based on its structured or unstructured “description” (in this case: finding a DOI of a cited article based on a citation string).

But matching doesn’t have to just be about citations and DOIs. There are many other instances of matching we can think of, for example:

finding the ROR ID for an organisation based on an affiliation string,
finding the ORCID ID for a researcher based on the person’s name and affiliation,
finding the ROR ID for a funder based on the acknowledgements section of a research paper,
finding the grant DOI based on an award number and a funder name.

Matching doesn’t have to be done manually. It is possible to develop fully automated strategies for metadata matching and employ them at scale. It is also possible to use a hybrid approach, where automated strategies assist users by providing suggestions.

Developing automated matching strategies is not a trivial task, and if we want to do it right, it takes a great deal of time and effort. This brings us to our next question: is it worth it?

Why do we need matching?

In short, metadata matching gives us a more complete picture of the research nexus by discovering missing relationships between various entities within and throughout the scholarly record:

These relationships are very powerful. They provide important context for any entity, whether it is a research output, a funder, a research institution, or an author. Imagine for a moment the scholarly record without any such relationships, where all bibliographic references, affiliations (institution names and addresses), and funding information (funder names and grant titles) are provided as unstructured strings only. In such a world, how would you calculate the number of times a particular research paper was cited? How would you get a list of research outputs supported by a specific funder? It would be incredibly challenging to navigate, summarise, and describe research activities, especially considering the scale. Thankfully, these and many other questions can be answered thanks to metadata matching that discovers relationships between entities in the scholarly record.

There are two primary ways we can use metadata matching in our workflows: as semi-automated tools that help users look up the appropriate identifiers or as fully automated processes that enrich the metadata in various scholarly databases.

The first approach is quite similar to the example we described at the beginning. If you are submitting scholarly metadata, for example of a new article to be published, you can use metadata matching to look up identifiers for the various entities and include these identifiers in the submission. For example, with the help of metadata matching, instead of submitting citation strings, you could provide the DOIs for works cited in the paper and instead of the name and address of your organisation, you could provide its ROR ID. To make this easier for people, metadata submission systems and applications sometimes integrate metadata matching tools into user interfaces.

The second approach allows large, existing sources of scholarly metadata to be enriched with identifiers in a fully automated way. For example, we can match affiliation strings to ROR IDs using a combination of machine learning models and ROR’s default matching service, effectively adding more relationships between people and organisations. We can also compare journal articles and preprints metadata in the Crossref database by calculating similarity scores for titles, authors, and years of publication to match them with each other and provide more relationships between preprints and journal articles. This automated enrichment can be done at any point in time, even after research outputs have been formally published.

There are fundamental differences between these two approaches. The first is done under the supervision of a user, and for the second, the matching strategy makes all the decisions autonomously. As a result, the first approach will typically (although not always) result in better quality matches. By contrast, the second approach is much faster, generally less expensive, and scales to even very large data sources.

In the end, no matter what approach is used, the goal is to achieve a more complete accounting of the relationships between entities in the scholarly record.

This blog is the first one in a series about metadata matching. In the coming weeks, we will cover more detail about the product features related to metadata matching, explain why metadata matching is not a trivial problem, and share how we can develop, assess, compare, and choose matching strategies. Stay tuned!

2024 public data file now available, featuring new experimental formats

Patrick Polischuk — Tue, 14 May 2024 00:00:00 +0000

This year’s public data file is now available, featuring over 156 million metadata records deposited with Crossref through the end of April 2024 from over 19,000 members. A full breakdown of Crossref metadata statistics is available here.

Like last year, you can download all of these records in one go via Academic Torrents or directly from Amazon S3 via the “requester pays” method.

Download the file: The torrent download can be initiated here. Instructions for downloading via the “requester pays” method, along with other tips for using these files, can be found on the “Tips for working with Crossref public data files and Plus snapshots” page.

In January, Martin Eve announced that we had been experimenting with alternative file formats meant to make our public data files easier to use by broader audiences. This year’s file will be published alongside the tools that can be used on the public data file to produce two experimental formats: JSON-lines and SQLite (and a bonus Rust version). You can read more about our thinking behind this work in Martin’s blog post, and we are keen to hear your thoughts on these alternatives.

Our annual public data file is meant to facilitate individuals and organisations interested in working with the entirety of our metadata corpus. Starting with the majority of our metadata records in one file should be much easier than starting from scratch with our API, but because Crossref metadata is always openly available, you can use the API to keep your local copy up to date with new and updated records.

If you’re curious about what you’ll get with the public data file, we’ve also published a sample version so that you can take a peek before committing to downloading the ~212 gb file. This file includes a random sample of JSON files and is available exclusively via torrent here.

We hope you find this public data file useful. Should you have any questions about how to access or use the file, please see the tips below, or share your questions below (you will be redirected to our community forum).

Tips for using the torrent and retrieving incremental updates

Use the public data file if you want all Crossref metadata records. Everyone is welcome to the metadata, but it will be much faster for you and much easier on our APIs to get so many records in one file. Here are some tips on how to work with the file.
Use the REST API to incrementally add new and updated records once you have the initial file. Here is how to get started (and avoid getting blocked in your enthusiasm to use all this great metadata!).
While bibliographic metadata is generally required, because lots of metadata is optional, records will vary in quality and completeness.

Questions, comments, and feedback are welcome at support@crossref.org.

Integrity of the Scholarly Record (ISR): what do research institutions think?

Madhura Amdekar — Thu, 09 May 2024 00:00:00 +0000

Earlier this year, we reported on the roundtable discussion event that we had organised in Frankfurt on the heels of the Frankfurt Book Fair 2023. This event was the second in the series of roundtable events that we are holding with our community to hear from you how we can all work together to preserve the integrity of the scholarly record - you can read more about insights from these events and about ISR in this series of blogs.

Research institutions are one of the most important stakeholders in the endeavour of research integrity, and any conversation around ISR is incomplete without the views of this key community. This fact was acknowledged at the second ISR roundtable event, and one of the main takeaways from the discussions was to make more focused efforts to hear the viewpoints of researchers and academics.

As the first step in this direction, we organised an online discussion on the integrity of the scholarly record, to which we invited: researchers and academics, research integrity experts based at academic institutions, Crossref members, as well as other organisations working on this topic such as COPE and Digital Science. The primary objective of this event was to hear from this community their perspectives on preserving and leveraging the integrity of the scholarly record and to identify opportunities for collaboration in this area. To ensure common ground, we also wanted to share information about Crossref metadata, the Research Nexus vision, and our position and role in the integrity of the scholarly record.

To facilitate this, the event started with an introduction by Kora Korzec, Head of Community Engagement and Communication at Crossref, to our mission and vision and the importance of capturing the relationships between the objects, people and places involved in research through the Research Nexus. Amanda Bartell, Head of Member Experience, was next and she spoke about the scholarly record and the role that Crossref plays in preserving the record’s integrity. In her presentation, Amanda emphasised that Crossref’s role is not to assess the quality of content deposited by the members but rather to provide infrastructure that enables the community to provide and use metadata about the scholarly content produced by members. It’s important not to put up barriers to entry, but to work with all publishers to encourage best practices.

Dominika Tkaczyk, Head of Strategic Initiatives, shared details of a few Crossref projects that focus on monitoring and improving metadata completeness, thereby supporting ISR. These projects include improving the Participation Reports, using metadata matching to discover new relationships (e.g., preprint published as work, work supported by funder, etc), and importing more retractions and other updates from the Retraction Watch database that was acquired and made openly available by Crossref. Dominika used these examples to highlight the ways in which open and complete metadata can help in uncovering large scale trends and systemic concerns. The final speaker was Amanda French, ROR Technical Community Manager, who introduced the audience to the Research Organization Registry, or ROR.

To accomplish the primary aim of the event, which was to hear the community’s viewpoints, the participants were divided into breakout groups for discussions and given three prompts to answer. The rest of the blog is a summary of what we heard from the participants.

1. Is Crossref’s role what you expected? What surprised you? What are we missing?

An overarching sentiment from the academics in the audience was that Crossref does so much more than is known to researchers! They were surprised by the range of activities underway at Crossref. At the same time, there were calls for Crossref to play a bigger role. Suggestions included playing a leadership role in deciding which metadata elements are a priority, providing guidance on the main metadata components important for signalling trust, playing a greater role in connecting various identifiers to ensure that relationships between different content types are preserved well, and to coordinate the efforts being taken by institutions, publishers and service providers around research integrity, by virtue of Crossref’s unique position in the community. There was a broad agreement that by providing the essential infrastructure, Crossref acts as the base upon which other actors in the scholarly community can build.

2. What metadata elements do you consider important for signalling trust?

Many participants spoke about the various ways in which author identity and affiliation are important as trust signals. Being able to identify when an author has changed institutions, or being able to make a distinction between authors who have the same name is important. Author affiliations that are authentic and verified would go a long way in establishing trust.

Multiple assertions, e.g. for affiliations, would be welcome. The use cases for this could be when research starts at one institution and is carried over to another, or when researchers affiliated with an institution may perform part of the research overseas. Some of the participants, who actively investigate research data, shared that abstracts are valuable because they can be used for large scale analyses related to research integrity.

Other metadata elements that came up during this discussion were data on peer review, ethics approval, patient and donor consent in medical research, editorial boards (especially of special issues), pre-registration, funding metadata, datasets and programming scripts.

3. What value do you see in the integrity and completeness of the scholarly record in the way you operate? How do you contribute to it? How can it support you to achieve your own goals?

Participants acknowledged that integrity of the metadata and the scholarly record is essential. Ensuring this integrity is a dynamic process, much akin to the concept of organised scepticism which is the notion that all scientific work should be trusted subject to its verification. Several ideas were shared on how to progress the integrity and completeness of the scholarly record. One recommendation was to use multiple metadata trust markers as that can make it harder for bad actors to game the system, but this may run the risk of making things complicated. Another suggestion was to make metadata part of the onboarding procedure- by gathering staff ORCID iDs during the onboarding process and sharing the institutional ROR ID with staff to promote its use, institutions can ensure that this information is routinely made available. The metadata deposited with Crossref should be integrated with downstream workflows to better facilitate the use of this rich metadata. An example of this is to integrate Crossmark with other research tools such as reference management software.

The participants acknowledged that this discussion underlined for them the fact that having identifiers in itself is not an indicator of quality and that the underlying metadata records and wider context is key to understanding trustworthiness of the content.

This event was a good first step towards engaging researchers and academics in the conversation about ISR. It connected folks working in different parts of the world who are united by their interest in research integrity. There was good engagement among all and commitment to continue these conversations in the future, with many participants planning to connect at the World Conference on Research Integrity in June (I’ll be attending as well, for anyone who wants to continue the conversation - along with my colleagues Fabienne and Evans).

At Crossref, we plan on continuing these conversations with all segments of the community to understand their needs and perceptions around metadata. The greater the awareness about the importance of metadata and its applications, including for research integrity, the richer the metadata that we are able to collect together. This will lead to building a comprehensive Research Nexus and emergence of more relationships therein. Please write in response to this post on our Community Forum if you have any thoughts on this as we’d love to hear from you.

List of participants


Manu Goyal	International Journal of Cancer
Panagiotis Kavouras	University of Oslo
Dorothy Bishop	University of Oxford
Zhesi (Phil) Shen	Centre of Scientometrics, National Science Library, Chinese Academy of Sciences
Wouter Vandevelde	KU Leuven
Leslie McIntosh	Digital Science
Elizabeth Noonan	University College Cork
Radek Gomola*	Masaryk University Press
	Queensland University of Technology
	London School of Hygiene & Tropical Medicine
	Vilnius Gediminas Technical University Library
	Committee on Publication Ethics (COPE)
Ginny Hendricks	Chif Program Officer, Crossref
Kornelia Korzec	Director of Community, Crossref
Amanda Bartell	Director of Membership, Crossref
Dominika Tkaczyk	Director of Data Science, Crossref
Amanda French	Technical Community Manager, Crossref
Madhura Amdekar	Community Engagement Manager, Crossref

*Note: name added 21-May-2024

Seeking consultancy: understanding joining obstacles for non-member journals

Ginny Hendricks — Wed, 01 May 2024 00:00:00 +0000

Crossref is undertaking a large program, dubbed 'RCFS' (Resourcing Crossref for Future Sustainability) that will initially tackle five specific issues with our fees. We haven’t increased any of our fees in nearly two decades, and while we’re still okay financially and do not have a revenue growth goal, we do have inclusion and simplification goals. This report from Research Consulting helped to narrow down the five priority projects for 2024-2025 around these three core goals:

Scope of the RCFS Program 2024-2025

GOAL: MORE EQUITABLE FEES

Project 1: Evaluate the USD $275 annual membership fee tier and propose a more equitable pricing structure, which might entail breaking this down into two or more different tiers.
Project 2: Define a new basis for sizing and tiering members for their capacity to pay

GOAL: SIMPLIFY COMPLEX FEES

Project 3: Address and adjust volume discounts for Content Registration
Project 4: Address and adjust back-year discounts for Content Registration

GOAL: REBALANCE REVENUE SOURCES

Project 5: Reflect the increasing value of Crossref as a metadata source, likely increasing Metadata Plus fees

Work to date

As part of the RCFS program, we are working closely with our Membership & Fees Committee to discuss insights, gather feedback, and make recommendations to the Board. As a first step, we have surveyed and received responses from around 1000 of the current 8000 Crossref members in our lowest membership fee tier (USD $275). We are now starting to distill that data and will discuss it on our community call on May 8th and subsequently with the M&F Committee to inform recommendations for fee changes that may going into effect in 2025 or 2026.

Request For Information (RFI) about community consultation project

While we have useful data from existing Crossref members, we know that there are many thousands of journals that are not (yet) members, and we need to understand this group better, in particular, to document and address the financial obstacles as well as the technical or social challenges.

We are looking for community facilitation expertise, with multiple language skills, to conduct a series of focus groups with non-member journals, with a summary and insights report (in English) provided by the end of June 2024.

All the data and documentation will be available publicly on the dedicated RCFS Program website

As well as designing, conducting, and summarising the results of some focus groups (participants for which will be gathered via our own contacts and those of partners such as DOAJ, EIFL, and the Free Journal Network) we would like the consultant to review work such as the DIAMAS institutional publishing report, and identify data relevant to Crossref’s fee model.

If you would like to respond, please provide the following information and send it to Kora Korzec at feedback@crossref.org by 15th May:

Your consultancy organisation and your role within it
Examples of similar market research undertaken
Languages spoken within your team
Confirmation that the timeline is workable
Approximate fee, likely range, or structure/basis for your fee

Equally, if you represent a journal or group of journals, such as Diamond Open Access journals, and are not yet using Crossref, please get in touch and we can include your group in the research.

Thank you!

This year's call for expressions of interest to join our board

Lucy Ofiesh — Fri, 26 Apr 2024 00:00:00 +0000

The Crossref Nominating Committee is inviting expressions of interest to join the Board of Directors of Crossref for the term starting in January 2025. The committee will gather responses from those interested and create the slate of candidates that our membership will vote on in an election in September.

Expressions of interest will be due Monday, May 27th, 2024

This is an exciting time to join the board, as we have a number of active projects underway: We are considering resourcing Crossref for a sustainable future and board members will be part of deciding any changes to our fees scheme and overseeing its implementation. We’re focusing on how our community and metadata can contribute to ensuring the integrity of the scholarly record. We’re broadening our metadata record to capture richer funding and institutional affiliations. We’re working towards a future where the scholarly record prioritizes relationships between research outputs to build a holistic research nexus. The board helps guide this work.

About the board elections

The board is elected through the “one member, one vote” policy wherein every member organisation of Crossref has a single vote to elect representatives to the Crossref board. Board terms are for three years, and this year, there are four seats open for election.

The board maintains a balance of seats, with eight seats for smaller members and eight seats for larger members (based on total revenue to Crossref). This is an effort to ensure that the scholarly community’s diversity of experiences and perspectives is represented in decisions made at Crossref.

This year, we will elect two of the larger member seats (membership tiers $3,900 and above) and two of the smaller member seats (membership tiers $1,650 and below). You don’t need to specify which seat you are applying for; we will provide that information to the nominating committee.

The online election will open in September, with results announced at the annual meeting on October 29th, 2024. New members will begin their term in January 2025.

About the Nominating Committee

The Nominating Committee reviews the expressions of interest and selects a slate of candidates for election. The slate put forward will exceed the total number of open seats. The committee considers the statements of interest, organisational size, geography, and experience.

2024 Nominating Committee

James Phillpotts*, Director of Content Transformation and Standards, Oxford University Press, committee chair
Oscar Donde*, Editor in Chief, Pan Africa Science Journal
Rose L’Huillier*, Senior Vice President Researcher Products, Elsevier
Ivy Mutambanengwe-Matanga, Chief Operating Officer, African Journals Online
Adam Sewell, Chief Technology Officer, IOP Publishing

(*) indicates Crossref board member

What is the committee looking for this year

The committee looks for skills and experience that will complement the rest of the board. Candidates from countries and regions not currently reflected on the board are strongly encouraged to apply. Successful candidates often have some or all of these characteristics:

Demonstrate a commitment to or understanding of our strategic agenda or the Principles of Open Scholarly Infrastructure;
Have expertise that may be underrepresented on the board currently;
Hold senior/director-level positions in their organisations;
Have experience with governance or community involvement;
Represent member organisations that are active in the scholarly communications ecosystem;
Demonstrate metadata best practices as shown in the member’s participation report

The board is also encouraging Crossref members who are research funders to apply.

Board roles and responsibilities

Crossref’s services provide a central infrastructure for scholarly communications. Crossref’s board helps shape the future of our services and by extension, impacts the broader scholarly ecosystem. We are looking for board members to contribute their experience and perspective.

The role of the board at Crossref is to provide strategic and financial oversight of the organisation, as well as guidance to the Executive Director and the staff leadership team, with the key responsibilities being:

Setting the strategic direction for the organisation;
Providing financial oversight; and
Approving new policies and services.

The board is representative of our membership base and guides the staff leadership team on trends affecting scholarly communications. The board sets strategic directions for the organisation while also providing oversight into policy changes and implementation. Board members have a fiduciary responsibility to ensure sound operations. They do this by attending board meetings as well as joining more specific board committees.

Who can apply to join the board?

Any active member of Crossref can apply to join the board. Crossref membership is open to organisations that produce content, such as academic presses, commercial publishers, standards organisations, and research funders.

What is expected of board members?

Board members attend four meetings each year that typically take place in January, March, July, and November. Meetings have taken place in a variety of international locations and travel support is provided when needed. January, March, and November board meetings are held virtually, and all committee meetings take place virtually. Each board member should sit on at least one Crossref committee. Care is taken to accommodate the wide range of time zones in which our board members live.

While the expressions of interest are specific to an individual, the seat that is elected to the board belongs to the member organisation. The primary board member also names an alternate who may attend meetings in the event that the primary board member is unable to. There is no personal financial obligation to sit on the board. The member organisation must remain in good standing.

Board members are expected to be comfortable assuming the responsibilities listed above and to prepare and participate in board meeting discussions.

How to apply

Please click here to submit your expression of interest. We ask for a brief statement about how your organisation could enhance the our board and a brief personal statement about your interest and experience with Crossref.

Please contact me with any questions at lofiesh@crossref.org

Common views and questions about metadata across Africa

Johanssen Obanda — Wed, 24 Apr 2024 00:00:00 +0000

This past year has been a captivating journey of immersion within the Crossref community, a mix of online interactions and meaningful in-person experiences. From the engaging Sustainability Research and Innovation Conference in Port Elizabeth, South Africa, to the impactful webinars conducted globally, this has been more than just a professional endeavour; it has been a personal exploration of collaboration, insights, and a shared commitment to pushing the boundaries of scholarly communication.

Working collaboratively with research funders and research organisations

Cocreation activity in smaller groups at the SRI conference.

The adventure began with a significant in-person event, the Sustainability Research and Innovation Conference. In the coastal city of Port Elizabeth, South Africa, I had the honour of hosting a parallel co-creation session titled “Connecting Science to Society: A Network Approach to Improving Science Communication in the Global South.” The co-creation session addressed research discoverability and accessibility among early-career researchers. Apart from some immediate feedback from the researchers in the room about how they might use co-creation beyond the conference to improve their research experience and outcome, I also had conversations with research funders from the Belmont Forum, Future Earth, and National Research Foundation - South Africa and the National Research Foundation - Mozambique about connecting their grants and grantees with their published outputs referencing Crossref’s Open Funder Registry and research grants registration. A different side conversation was about a community organisation in Botswana that is interested in registering patents with Crossref for proper referencing and protecting the intellectual property of their research on the indigenous communities’ innovations and the associated published work. These conversations are ongoing, unveiling a new understanding of unique needs and opportunities to pursue with research funders and research organisations working on indigenous knowledge and innovations.

Learning from organisations in GEM-eligible countries

The journey extended globally through a series of webinars conducted in Bangladesh, Tanzania, Nepal, and Ghana. Collaborating with dedicated Ambassadors and my colleagues leading the Global Equitable Membership (GEM) program, we witnessed an increase in Crossref membership from the GEM countries and initial metadata registration. The GEM Program offers relief from both Crossref membership and Crossref content registration fees for organisations in the least economically advantaged countries in the world, based on the World Bank’s IDA list. Susan, in her blog post, “The GEM Program: Year One”, elaborated on the significance of these efforts and their impact on fostering equitable access to scholarly resources and communication through the expansion of Crossref’s membership base in underrepresented regions, such as Bangladesh, Tanzania, Nepal, and Ghana. Specific concerns encountered while presenting the GEM program included feedback expressing reservations about the program’s approach, particularly in deciding on eligible countries, and advocating eligibility for the program to be extended to all the non-GEM countries in Africa. Additionally, a conversation with some organisations brought up concerns regarding the program’s sustainability, with inquiries about whether GEM was merely a free trial or freemium service, and seeking assurances against future fees. The audience found these sessions helpful, acknowledging that joining fees were no longer going to be a barrier, yet questions about the program’s longevity brought out the need for sustained support.

Discussing how The Research Nexus can support the community

My journey then led me to Makerere University in Uganda for the Consortium of Uganda University Libraries (CUUL 2023) conference and the Forum for Open Research in MENA (FORM 2023) in Abu Dhabi. In Uganda, I noticed the synergy between university libraries, institutional repositories, and the research and education network service provider formed a consortium that played a crucial role in bridging the digital gap and supporting the adoption of open infrastructure. The event was mainly attended by librarians from different universities in Uganda. Most of those I connected with needed more information about Crossref and had questions about how Crossref DOIs are different from ARKs, which they commonly use in their publishing workflows. At FORM 2023, in my presentation titled, “The Research Nexus: A Rich and Reusable Open Network of Relationships in the Scholarly Record,” I shared Crossref’s vision for a connected research ecosystem with the audience that comprised of researchers, research administrators, and funders, and a good number of big publishers like IEEE and Taylor & Francis. The Research Nexus seeks to reveal relationships beyond persistent identifiers, utilising rich metadata to connect various scholarly components. I also took the opportunity at both events to share about The Publishers Learning And Community Exchange (PLACE), an online forum promoting best practices in scholarly publishing. The goal was to show attendees how they can actively contribute to and benefit from this vision, fostering a robust and interconnected research community through Crossref’s open infrastructure.

Photo with Dr. Salwan Abdulateef, rossref Ambassador - Iraq

I enjoyed the opportunity to join the National Open Science Dialogue by TCC Africa, which provided crucial insights, emphasising the need for assessing awareness, implementing comprehensive policies, and fostering collaboration around Open Science. Higher education institutions were recognized as influencers in the global Open Science movement, while a call for an inclusive research environment was underscored through open access and data sharing. The dialogue emphasized a collective effort involving policymakers, educators, researchers, and institutions, focusing on inclusivity and collaboration to advance Open Science in East Africa.

Exploring how rich metadata can provide trust signals with members in Kenya

Reflecting on the Crossref Nairobi event that happened in February 2024, it was an enriching experience exploring key issues shaping scholarly publishing in Kenya. The discussions also touched on the role of metadata as a trust signal and a tool for the persistence of the scholarly record, particularly in regions where data protection challenges persist. This is exemplified by concerns raised during the event about the fear of data theft, misuse, or loss, especially in places with comparatively weaker data protection laws. The presence of robust metadata, particularly with detailed provenance information, becomes crucial in such contexts, as it enables better identification and handling of potential misuse. Thus, through effective metadata implementation and the persistence facilitated by identifiers, the management of data risks can be significantly improved.

The insights from existing Crossref members pointed out contextual challenges, regional differences, and the importance of effective post-publication processes. The conference served as a valuable platform for dialogue, emphasising the collective commitment to continuous improvement of scholarly communication in the country, and the need for continuous awareness and training on making the most of Crossref services. The roundtable discussions during the Crossmark service consultation brought to light various reflections and considerations regarding post-publication changes in publishing workflows. The Crossmark service was a new discovery for most participants, with potential value recognized in facilitating current updates on articles. However, there are existing barriers such as a lack of awareness and technical expertise, suggesting the need for further education to facilitate adoption. Overall, the consultation provided a platform for introspection and exploration of avenues for improving post-publication practices in scholarly publishing.

Crossref Nairobi group photo

We organised the Crossref Nairobi event with the help of colleagues from the community team and local Ambassadors, Mercury Shitindo of Kenya, Baraka Ngussa of Tanzania and our Board Members in Kenya, Oscar Donde. It was the first time I saw both my colleagues and Ambassadors in action and working closely together - making presentations and accommodating last-minute facilitation changes to the program. Compared to attending or speaking at an event, organising one was a unique experience requiring a lot of planning in advance for logistics and the event program, identifying and keeping in touch with important stakeholders, ushering guests and being on standby for any matters that come up about the event. All of that went very well thanks to the team on the ground and cooperative participants.

Exploring the role of open infrastructure for African universities

Attending the recent WACREN 2024 conference was an eye-opening experience, unfolding the role of open infrastructure in addressing challenges faced by African universities. A focus on open access systems and advocacy for decolonizing knowledge were voiced too, including challenges of affordability of DOIs and questions of local ownership amidst global initiatives. Global persistent identifier providers, including ORCID and DataCite too, had a presence at the conference, alongside passionate advocates for more locally managed, decentralised infrastructure. These are concerns that Crossref needs to understand better, as we seek to find effective ways of supporting equitable participation in the Research Nexus. The conference resonated with a call for continued work in fostering accessibility, sharing, and leveraging resources to accelerate research and innovation in Africa.

Photo with our Ambassadors from West Africa at WACREN 2024 event: Blessing Abumere - Nigeria, Audrey Kenni Nganmeni - Cameroon, Richard Lamptey - Ghana and Oumy Ndiaye - Senegal.

Conversations with Crossref Ambassadors brought about a shared narrative across universities in some African countries. These institutions are actively embracing digital shifts, setting up institutional repositories using platforms like DSpace and OJS. However, challenges persist, particularly in funding and technical capacity. It’s heartening to see how national and regional research and education networks step in to help in internet connectivity, opening up collaboration opportunities with other interoperable infrastructure, setting up repositories, providing hosting services and event managing content identifiers.

Deceptive publishing practices remain a shared concern, and we’ve had requests at these meetings for stricter inclusion criteria for membership of Crossref to ensure quality and trustworthiness of articles accessible through Crossref metadata.

We’ve explained to those we’ve met that Crossref doesn’t (and can’t) assess the quality of content or the integrity of the research process. We don’t have the people or the skills, and it isn’t our mission to be the gatekeepers of research quality. A DOI record is just an indication that something was published, it isn’t an indication of quality.

However, we do still have a vital role in preserving the integrity of the scholarly record. We provide the infrastructure which enables those who produce scholarly outputs to provide metadata (effectively evidence) about how they ensure the quality of content and how the outputs fit into the scholarly record. The scholarly record - that network of published outputs, inputs, relationships and contexts - is captured through the metadata records that our members register with us, and that we then distribute freely and openly through our API. The richer and more comprehensive Crossref records are, the more context there is for our members and for the whole scholarly research ecosystem to make their own decisions around trustworthiness. Blocking access to the infrastructure creates gaps in the scholarly record, but also potentially blocks legitimate newcomers.

“Crossref is focused on enriching metadata to provide more and better trust signals while keeping barriers to membership and participation as low as possible to enable an inclusive scholarly record.” Read more about Crossref’s role in preserving the integrity of the Scholarly record in the blog post by Amanda Bartell.

While the landscape of digital scholarly publication witnesses significant strides, a crucial need persists, the importance of preserving and interconnecting metadata to the global scholarly record. It’s not just about discoverability, a theme resonating strongly within the community, but about enabling reproducibility, upholding research and editorial integrity, and facilitating reporting and assessment.

The path forward

As I reflect on this year of immersing myself within the Crossref community, building awareness in new communities, and learning more about the different perceptions across the region, it feels like a personal progression of growth and discovery. From the captivating in-person moments to the global webinars and collaborative efforts to address challenges in scholarly communication, this journey is not just a professional pursuit; it’s a personal exploration. The path forward involves continued support, intensified awareness-building, and sustained dialogue, ensuring that the scholarly ecosystem continues to thrive, evolve, and leave a lasting impact.

Testing times

Martin Eve — Wed, 03 Apr 2024 00:00:00 +0000

One of the challenges that we face in Labs and Research at Crossref is that, as we prototype various tools, we need the community to be able to test them. Often, this involves asking for deposit to a different endpoint or changing the way that a platform works to incorporate a prototype.

The problem is that our community is hugely varied in its technical capacity and level of ability when it comes to modifying their platform. Some mega-publishers, for instance, outsource their platforms and so are dependent on third party developers/organisations when they want to make a change. Many smaller publishers, by contrast, use systems such as OJS, which come with Crossref plugins that make life very easy… but that require hard code changes to accommodate prototypes. Such changes are way beyond the technical capacity of most journal editors.

So how can we prototype new ideas and test them? One way is by creating new interstitial interfaces that allow people to manually supplement metadata or register for prototype services. Of course, this requires additional work on behalf of the user. Every time they wish to participate they have to visit an extra web page and re-input details that, surely, were included in the original deposit.

Another way would be for plugin developers to have an advanced option field that allowed end-users to change their deposit endpoint. It would be excellent to see this feature in OJS, Janeway, and also proprietary systems. This would allow us to work with the community to test new prototype mechanisms, without forcing anyone to edit code. Many systems already include the ability to switch between Crossref’s “test” system and our live deposit API. All I am really suggesting here is the logical next step: allow advanced users to specify a deposit endpoint of their own choosing so that we can give them access to prototype systems.

Of course, it’s not always that simple. Sometimes, prototype systems will require new data fields on submission, for example. In those cases, there is nothing for it except to modify the plugin or to provide a separate interface. But sometimes, as in the case of the Op Cit project (more on which soon), all the data is already in place; we just need to direct users to a different endpoint. Such changes would definitely make testing times less trying.

Mending Chesterton's fence: open source decision-making

Joe Wass — Mon, 18 Mar 2024 00:00:00 +0000

When each line of code is written it is surrounded by a sea of context: who in the community this is for, what problem we’re trying to solve, what technical assumptions we’re making, what we already tried but didn’t work, how much coffee we’ve had today. All of these have an effect on the software we write.

By the time the next person looks at that code, some of that context will have evaporated. There may be helpful code comments, tests, and specifications to explain how it should behave. But they don’t explain the path not taken, and why we didn’t take it. Or those occasions where the facts changed, so we changed our mind.

Some parts of our system are as old as Crossref itself. Whilst our process still involves coffee, it’s safe to say that most of our working assumptions have changed, and for good reasons! We have to be very careful when working with our oldest code. We always consider why it was written that way, and what might have changed since. We’re always on the look out for Chesterton’s Fence!

Leaving a Trail

We’re building a new generation of systems at Crossref, and as we go we’re being deliberate about supporting the people who will maintain it.

When our oldest code was written, the software development team all worked in an office with a whiteboard or three, and the code was proprietary. Twenty years later, things are very different. The software development team is spread over 8 timezones. Thanks to POSI, all the new code we write is open source, so the next people to read that code might not even be Crossref staff.

Working increasingly asynchronously, without that whiteboard, we need to record the options, collect evidence, and peer-review them within the team.

So for the past couple of years the software team has maintained a decision register. The first decision we recorded was that we should record decisions! Since then we have recorded the significant decisions as they arise. Plus some historical ones.

These aren’t functional specifications, which describe what the system should do. It’s the decisions and trade-offs we made along the way to get to the how. Look out for another blog post about specifications.

By leaving a trail of explanations as we go, we make it easier for people to understand why code was written, and what has changed. We’re writing the story of our new systems. This makes it easier to alter the system in future in response to changes in our community, and the metadata they use.

Difficult Decisions

There are some fun challenges to building systems at Crossref. We have a lot of data. Our schema is very diverse, and has a vast amount of domain knowledge embedded in it. It’s changed over time to accommodate 20 years of scholarly publishing innovations. Our community is diverse too, from small one-person publishers with a handful of articles, through to large ones that publish millions.

What might be an obvious decision for a database table with a thousand rows doesn’t always translate to a million. When you get to a billion, things change again. An initially sensible choice might not scale. And a scalable solution might look over-engineered if we had millions of DOIs, rather than hundreds of millions.

The diversity of the data also poses challenges. A very simple feature might get complicated or expensive when it meets the heterogeneity of our metadata and membership. What might scale for journal article or grant metadata might not work for book chapters.

The big decisions need careful discussion, experimentation, and justification.

2NF or not 2NF

One such recent decision was how we structure our SQL schema for the database that powers our new ‘relationships’ REST API endpoint, currently in development.

The data model is simple: we have a table of Relationships which connect pairs of Items. And each Item can have properties (such as a type). The way to model this is straightforward, following conventional normalization rules:

We built the API around it, and all was well.

We then added a feature which lets you look up relationships based on the properties of the subject or object. For example “find citations where the subject is an article and the object is a dataset”. This design worked well in our initial testing. We loaded more data into it, and it continued to work well.

And then, the context changed. Once we tested loading a billion relationships in the database, the performance dropped. The characteristics of the data: size, shape and distribution, reached a point where the database was unable to run queries in a timely way. The PostgreSQL query planner became unpredictable and occasionally produced some quite exciting query plans (to non-technical readers: databases are neither the time nor the place for excitement).

This is a normal experience in scaling up a system. We expected that something like this would happen at some point, but you don’t know when it will happen until you try. We bounced around some ideas and came up with a couple of alternatives. Each made trade-offs around processing time, data storage and query flexibility. The best way to evaluate them was to use real data at a representative scale.

One of the options was denormalisation. This is a conventional solution to this kind of problem, but was not our first choice as it involves extra machinery to keep the data up-to-date, and more storage. It would not have been the correct solution for a smaller dataset. But we had the evidence that the other two approaches would not scale predictably.

By combining the data into one table, we can serve up API requests much more predictably, and with much better performance. This code is now running with the right performance. Technical readers note that this diagram is simplified. The real SQL schema is a little different.

Without writing this history down, and explaining what we tried, someone might misunderstand the reason for the code and try to simplify it. Decision record DR-0500 guards against that.

But one day, when the context changes, future developers will be able to come back and modify the code, because they understand why it was like that in the first place.

Credential Checking at Crossref

Martin Eve — Fri, 15 Mar 2024 00:00:00 +0000

It turns out that one of the things that is really difficult at Crossref is checking whether a set of Crossref credentials has permission to act on a specific DOI prefix. This is the result of many legacy systems storing various mappings in various different software components, from our Content System through to our CRM.

To this end, I wrote a basic application, credcheck, that will allow you to test a Crossref credential against an API.

There are two modes of usage. First, a command-line interface that allows you to run a basic command and get feedback:

Usage: cli.py [OPTIONS] USERNAME PASSWORD DOI

Second, you can use it as a programmatic library in Python:

import cred
credential = cred.Credential(username=username, password=password, doi=doi)

if not credential.is_authenticated():
…

if credential.is_authorised():
…

The tool splits down authentication (whether the given username and password are valid) and authorisation (whether the valid credentials are usable against a specific DOI/prefix).

For technical information, the way this works is by attempting to run a report on the specific DOI in question and then scraping the response page. We hope, at some future point, that there will be a real API for this, but for now this solves the problem as a bridge.

Subject codes, incomplete and unreliable, have got to go

Patrick Polischuk — Wed, 13 Mar 2024 00:00:00 +0000

Subject classifications have been available via the REST API for many years but have not been complete or reliable from the start and will soon be deprecated. dfdfd

The subject metadata element was born out of a Labs experiment intended to enrich the metadata returned via Crossref Metadata Search with All Subject Journal Classification codes from Scopus. This feature was developed when the REST API was still fairly new, and we now recognize that the initial implementation worked its way into the service prematurely.

While subject classifications in Crossref metadata could be very useful, the current implementation in the REST API is problematic for three primary reasons:

They are misleadingly exposed in the API as a property of the work, when in fact they are a property of the container (e.g. a journal or conference proceeding). Just because a journal’s broad topic category is “X” doesn’t mean that a particular article in the journal is about “X.”

Existing works may have outdated subjects. Originally, subject codes were not updated periodically. However, subjects exposed in the /journals route are now updated once a day. Those exposed via the /works endpoint are indexed along with works, and so when a new subject list is ingested, new DOIs start getting new subjects, but existing works may have outdated subjects. We don’t have a mechanism for forcing updates when incorrect subject values are returned via the REST API, so this data can be stale and incorrect.

They are not applied to everything. This is because the Scopus list does not cover all the journals that Crossref has (conversely, the Scopus list contains some journals Crossref does not have), and does not contain other container types.

The Labs team investigated options for improving subject classification coverage but ultimately concluded that there are insufficient solutions to the coverage problem. For more, please see Esha Datta’s findings published at Force11’s Upstream: https://doi-org.pluma.sjfc.edu/10.54900/n6dnt-xpq48

Where does that leave us? Rather than continuing to supply unreliable and misleading subject category metadata, we will be deprecating this feature in the coming weeks. To minimize disruption and avoid breaking changes at this time, we will be removing this data from our index, so the subject element will simply be empty. We may remove the subject element in the future.

We know that the community’s desire for subject-based analysis of metadata is very strong, and we have supported efforts to establish a multidisciplinary taxonomy. Inaccurate codes in the meantime do not help but actually hinder these efforts, giving the false impression that they are correct.

We aim to deprecate the subject codes in April of this year.

Please let us know if you have any questions or concerns by leaving a comment below, which will start a thread in our community forum.

Frequently asked questions

Q. Will the subject field continue to be available and functional?
A. The subject metadata element will continue to be included in the JSON response but will not return any values.

Q. Will new subject codes be added in the future?
A. We do not have any current plans to add new subject codes in the future.

Q. I received a notification about this, but we don’t use subject codes. Do I need to do anything?
A. No, if you do not currently use the subject element, you do not need to do anything about this change.

Q. I noticed that wrong or inaccurate subject codes were assigned to my works. Is this a solution?
A. Yes. Until we can identify an accurate and sustainable system for assigning subject codes to Crossref metadata records, we want to stop assigning inaccurate subject codes and remove all existing assignments.

DOAJ and Crossref renew their partnership to support the least-resourced journals

Ginny Hendricks — Wed, 06 Mar 2024 00:00:00 +0000

Crossref and DOAJ share the aim to encourage the dissemination and use of scholarly research using online technologies and to work with and through regional and international networks, partners, and user communities for the achievement of their aims to build local institutional capacity and sustainability. Both organisations agreed to work together in 2021 in a variety of ways, but primarily to ‘encourage the dissemination and use of scholarly research using online technologies, and regional and international networks, partners and communities, helping to build local institutional capacity and sustainability around the world.’ Some of the fruits of this labour are:

DOAJ added support for Crossref XML to make it easier for publishers to upload metadata
Closer collaboration between customer/member support at both organisations, making it easier for publishers and journal editors to navigate both service’s technologies
the launch of PLACE: ‘a ‘one-stop shop’ for information to support publishers in adopting best practices the industry developed’ (together with other partners)
a pilot gap analysis of the journals in DOAJ with the possibility of helping them start to use and resolve DOIs.

The new agreement, signed earlier this month, will slightly shift focus to build upon existing collaborations, particularly around metadata. One of the primary sections of the MOU is enhancing support for the least-resourced journals by:

Assigning DOIs and depositing the metadata with Crossref
Finding ways to improve their DOAJ application experience to help them become indexed
Collect and ingest their Crossref metadata into DOAJ
Help them to get preserved via JASPER or similar initiatives
Help identify other local partners, such as Crossref Sponsoring Organisations, to support their use of Crossref services

It’s great that we can further underpin what is already a good working relationship. Both Crossref and DOAJ are central to discovery so it’s a natural partnership. Helping journals meet better standards and become indexed to make them more discoverable on a global scale is at the heart of our strategy. This agreement opens up a new avenue that allows the community to really focus on supporting those journals and the research they publish.’

– Joanna Ball, Managing Director of DOAJ

‘The collaborations with DOAJ so far only reconfirmed our shared goal to help make the global scholarly communications system more equitable wherever we can. Our joint projects aim to seek out and devise support for resource-constrained journals in multiple ways. DOAJ’s work is essential in helping journals to develop good practice, while Crossref offers an open infrastructure to ensure all journals can be included and discoverable in the global scholarly record.’

– Ginny Hendricks, Director of Member and Community Outreach at Crossref

——– END ——

About DOAJ

DOAJ is a community-curated online directory that indexes and provides access to high quality, open access, peer reviewed journals. DOAJ deploys around one hundred carefully selected volunteers from the community of library and other academic disciplines to assist in curating open access journals. This independent database contains over 20,400 peer-reviewed open access journals covering all areas of science, technology, medicine, social sciences, arts and humanities. DOAJ is financially supported worldwide by libraries, publishers and other like-minded organisations. DOAJ services (including the evaluation of journals) are free for all, and all data provided by DOAJ are harvestable via OAI/PMH and the API. See https://doaj.org/ for more information.

About Crossref

Crossref is a global community-governed open scholarly infrastructure that makes all kinds of research objects easy to find, assess, and reuse through a number of services critical to research communications, including an open metadata API that sees over 1.5 billion queries every month. Crossref’s ~20,000 members come from 155 countries and are made up of universities, publishers, funders, government bodies, libraries, and research groups. Their ~155 million DOI records contribute to the collective vision of a rich and reusable open network of relationships connecting research organisations, people, things, and actions; a scholarly record that the global community can build on forever, for the benefit of society.

For more information please contact: dominic@doaj.org and rclark@crossref.org

What do we know about DOIs

Martin Eve — Thu, 29 Feb 2024 00:00:00 +0000

Crossref holds metadata for approximately 150 million scholarly artifacts. These range from peer reviewed journal articles through to scholarly books through to scientific blog posts. In fact, amid such heterogeneity, the only singular factor that unites such items is that they have been assigned a document object identifier (DOI); a unique identification string that can be used to resolve to a resource pertaining to said metadata (often, but not always, a copy of the work identified by the metadata).

What, though, do we actually know about the state of persistence of these links? How many DOIs resolve correctly? How many landing pages, at the other end of the DOI resolution, contain the information that is supposed to be there, including the title and the DOI itself? How can we find out?

The first and seemingly most obvious way that we can obtain some of these data is by working through the most recent sample of DOIs and attempting to fetch metadata from each of them using a standard python script. This involves using the httpx library to attempt to resolve each of the DOIs to a resource, visiting that resource and seeing what the landing page yields.

Even this is not straightforward. Landing pages can be HTML resources or they can be PDF files, among other things. In the case of PDF files, to detect a run of text is not simple as a single line break can be enough to foil our search. Nonetheless, when using this strategy we find the following statistics:

Total DOI count in sample: 5000
Number of HTTP 200 response: 3301*
Percentage of HTTP 200 responses: 66.02%
Number of titles found on landing page: 1580
Percentage of titles found on landing page: 31.60%
Number of DOIs in recommended format found on landing page: 1410
Percentage of DOIs in recommended format found on landing page: 28.20%
Number of titles and DOIs found on landing page: 929
Percentage of titles and DOIs found on landing page: 18.58%
Number of PDFs found on landing page: 1469
Percentage of PDFs found on landing page: 29.38%
Percent of PDFs found on landing pages that loaded: 44.50%

* an HTTP 200 response means that the web page loaded correctly

While these numbers look quite low, the problem here is that a large number of scholarly publishers use Digital Rights Management techniques on their sites that block a crawl of this type. We can use systems like Playwright to remote control browsers to do the crawling, so that the request looks as much like a genuine user as possible and to evade such detection systems. However, lots of these sites detect headless browsers (where the browser is invisible and running on a server) and block them with a 403 Permission Denied error.

There’s a great Github javascript suite that aims to help evade headless detection. The tests it uses are:

User Agent: in a browser running with puppeteer in headless mode, user agent includes Headless.
App Version: same as User Agent above.
Plugins: headless browsers don’t have any plugins. So we can say that if it has plugin it’s headful, but not otherwise since some browsers, like Firefox, don’t have default plugins.
Plugins Prototype: check if the Plugin and PluginsArray prototype are correct.
Mime Type: similar to Plugins test, where headless browsers don’t have any mime type
Mime Type Prototype: check if the MimeType and MimeTypeArrayprototype are correct.
Languages: all headful browser has at least one language. So we can say that if it has no language it’s headless.
Webdriver: this property is true when running in a headless browser.
Time elapse: it pops an alert() on page and if it’s closed too fast, means that it’s headless.
Chrome element: it’s specific for chrome browser that has an element window.chrome.
Permission: in headless mode Notification.permission and navigator.permissions.query report contradictory values.
Devtool: puppeteer works on devtools protocol, this test checks if devtool is present or not.
Broken Image: all browser has a default nonzero broken image size, and this may not happen on a headless browser.
Outer Dimension: the attributes outerHeight and outerWidth have value 0 on headless browser.
Connection Rtt: The attribute navigator.connection.rtt,if present, has value 0 on headless browser.
Mouse Move: The attributes movementX and movementY on every MouseEvent have value 0 on headless browser.

Using the stealth plugin for Playwright also allows us to evade most of these checks. This just leaves Mouse Move and Broken Image detection, which I thought would not outweigh all the other factors. We can also jitter the connection with arbitrary delays so that it should appear to be coming at random intervals, rather than a robotic crawl.

Yet the basic fact is that we are still blocked from crawling many sites. This does not happen when we put the browser into headful mode, so current detection techniques have clearly evolved in the past half decade (since Detect Headless) was designed.

If, however, we run the browser in a headful mode, the results are somewhat stunningly different:

Total DOI count in sample: 5000
Number of HTTP 200 response: 4852
Percent of HTTP 200 responses: 97.04%
Number of titles found on landing page: 2547
Percentage of titles found on landing page: 50.94%
Number of DOIs in recommended format found on landing page: 2424
Percentage of DOIs in recommended format found on landing page: 48.48%
Number of titles and DOIs found on landing page: 1574
Percentage of titles and DOIs found on landing page: 31.48%
Number of PDFs found on landing page: 2085
Percentage of PDFs found on landing page: 41.70%
Percentage of PDFs found on landing pages that loaded: 42.97%

Let’s talk about the resolution statistics. Other studies, looking at general links on the web, have found a link-rot rate of about 60%-70% over a ten-year period (Lessig, Zittrain, and Albert 2014; Stox 2022). The DOI resolution rate that we have, with 97% of links resolving (or a 3% link-rot rate), is far better and more robust than a web link in general.

Is 3% a good or a bad number? It’s more robust than the web in general, but it still means that for every 100 DOIs, just under 3 will fail to resolve. We also cannot tell whether these DOIs are resolving to the correct target, except by using the metadata detection metrics (are the title and DOI on the landing page, which we could only detect at a far lower rate). It is entirely possible for a website to resolve with an HTTP 200 (OK) response, but for the page in question to be something very different to what the user expected, a phenomenon dubbed content drift. A good example is domain hijacking, where a domain name expires and spam companies buy them up. These still resolve to a web page, but instead of an article on RNA, for a hypothetical example, the user gets adverts for rubber welding hose. That said, other studies are also prone to this and there is no guarantee that content drift doesn’t affect a huge proportion of supposedly good links in the other studies, too.

Of course, one of the most frustrating elements of this exercise is having to work around publisher blocks on content when visiting using a server-only robot script. It’s important for us periodically to monitor the uptime rate of the DOI system. We also recognise, though, that publishers want to block malicious traffic. However, we can’t perform our monitoring in an easy, automatic way if headless scripts are blocked from resolving DOIs and visiting their respective landing pages. This is not even a call for open access; it’s just saying that current anti-bot techniques, sometimes implemented for legitimate reasons, stifle our ability to know the landscape. Even if the bot resolved a DOI to just a paywall, it would be easier for us to monitor this than it is now. Similarly, CAPTCHA systems such as Cloudflare that would seem to offer an easy way to distinguish between humans (good) and robots (bad) can make life very difficult at the monitoring end. We would certainly be grateful for any proposed solution that could help us to work around these mechanisms.

Conclusion

The context in which I wanted to know this information was so that we can take a snapshot of a page and then, at a later stage, determine whether it is down or has changed substantially. To do this, we are developing Shelob, an experimental content drift spider system; that’s what we’ve used so far to conduct this analysis. Over time, Shelob will evolve, we hope, to give us a way to detect when content has drifted or gone offline. If, however, we can’t detect whether an endpoint is good in the first place, then we likewise cannot detect when things have gone wrong. On the other hand, if, when we first visit, we find the DOI and title on the landing page, but at some future point this degrades, we might be able to say with some confidence that the original has died. I, personally, would encourage publishers not to block automated crawlers, because it’s good when we can determine these types of figures.

Works Cited

Lessig, Lawrence, Jonathan Zittrain, and Kendra Albert. 2014. ‘Perma: Scoping and Addressing the Problem of Link and Reference Rot in Legal Citations’. Harvard Law Review 127 (4). https://harvardlawreview.org/forum/vol-127/perma-scoping-and-addressing-the-problem-of-link-and-reference-rot-in-legal-citations/.(https://www.zotero.org/google-docs/?970bfS

Stox, Patrick. 2022. ‘Ahrefs Study on Link Rot’. SEO Blog by Ahrefs. 29 April 2022. https://ahrefs.com/blog/link-rot-study/.

The Lammey Effect

Crossref — Fri, 16 Feb 2024 00:00:00 +0000

We’re equally sad and proud to report that Rachael Lammey is moving on in her career to the very lucky team at 67Bricks. Her last day at Crossref is today, Friday 16th February. Which is too soon for us, but very exciting for her!

It’s hard to overstate Rachael’s impact on Crossref’s growth and success in her 12 years here. She started as a Product Manager where she developed that role into a broad and central function, and soon moved into the newly-formed community team as International Outreach Manager where she grew important programs such as Sponsors, Ambassadors, a series of ‘LIVE’ events around the world, and she went on to manage her own team and establish some of the most important strategic relationships that Crossref now feels fortunate to have.

Rachael was a significant part of the growth and adoption of new initiatives such as Crossmark, Similarity Check, the REST API, preprints, grants, data citation, and ROR. She's contributed to numerous organisations such as EASE, ALPSP, SSP, ISMTE, STM, and most recently co-Chaired the NISO working group on retractions and corrections.

As Head of Strategic Initiatives, and most recently, Director of Product, Rachael has shown dedication and leadership, supporting and strengthening not just her own teams but all of us across the organisation, encouraging us to do better while being one of the easiest people to work with.

The ‘butterfly effect’ is the notion that the world is deeply interconnected and that one small occurrence can influence a much larger complex system. Rachael embodies that notion, having created positive ripples and waves—and certainly many connections—in the scholarly record, in our organisation, and across the community.

Messages from colleagues

Rachael, I was saddened when I first heard the news that you were moving on to another opportunity. Your professionalism, work ethic, and positive attitude have been inspirational to work around. I have enjoyed the opportunities we have had to collaborate. As you move on to a new experience I wish you success and happiness in your future endeavors. Your presence will be missed at Crossref! Best Wishes.

– Ryan

I will miss you, Rachael. It has been great working with you for the few months that I have been at Crossref. I also cannot forget kayaking together with you and capsizing on the return to the shore, but almost professionally recovering. We would have made the best team this time around. I wish you all the best and many wins in your new role.

– Obanda

I feel like the luckiest human to have worked with Rachael over the last 4 years. She’s the perfect mix of smart and funny and knowing how to get things done. Rachael is a big part of what makes Crossref culture so special — I’ve never felt so supported in a role as when Rachael was my manager and for that I am very grateful. I will miss her wit and humor and her pragmatic approach to work and life!

– Sara

One of my first ‘Crossref LIVE’ events was with Rachael in Brazil in 2016. At the time, my role mostly focused on membership, and we had just started working more closely with ABEC, a large organisation in Brazil that sponsored quite a few members. Rachael managed the sponsor program then and thought this would be a good opportunity to collaborate with a sponsor on an event, and she asked me to join her. There is so much planning for these - venues, local partners, presentations, meetings - and she had all the details in order and made the event such a success. Rachael was supportive, encouraging, and I learned so much from working with her. The Brazil trip was such a positive experience that I realised I wanted to focus more closely on community engagement. Rachael encouraged me to do so.

She and I went on to partner on more LIVE events together. Our time in Indonesia was perhaps one of the most memorable for me - as well as our LIVE event, we had an unexpected tour of Yogyakarta with our Indonesian hosts, involving a tour of Prambanan Temple (see photo below), batik fabric shopping, visiting a few universities, and a stop at our hosts’ home. All the while trying not to let the winding car ride and traffic get the better of us. Our event the next day went perfectly, and I told her, half-jokingly, that the whole experience renewed my faith in humanity. Of note, we also drank the only bottle of wine available in the hotel bar.

Rachael was also my Crossref running buddy, and we spent quite a few miles together - in Brazil, NH, Maine, Oxford, and Spain. During our runs, topics ranged from Game of Thrones to Idris Elba to sportsing, but not so much about work. The next time I find myself in England, we will run a few more miles together, followed by a pint. Thank you for everything!

– Susan

Many have pointed out how talented, wise, or skilled you are and I certainly will not contradict a single word of it but that’s not what comes to mind first for me. Those traits, while true, pale in comparison to the person you are. Your positive, bright demeanor and the way everyone always feels better just being around you. I have dreaded some meetings from time to time. But whenever I’ve been involved in something with you, I’ve always left feeling better than when I started (no matter how grumpy I may have entered). You have been a consistent bright light in the Crossref constellation and you will truly be missed.

– Jon

Rachael! You are the best at cutting through all the bulls**t to get at what really needs to happen and why! Your knowledge is broad and deep, as is your institutional memory for all things Crossref and scholarly publishing. And your unflappability in pretty much any context is admirable and inspiring. We’ll miss you big time! Wishing you all the success at 67Bricks and otherwise.

– Shayn

Hey Rachael - I’m happy to be writing this note of Congratulations!! to you, particularly because it would be awkward to explain this bit of verklempt I’m feeling. Our interactions have been limited, but my impressions of you are of confidence, calm, capability, and collegiality. Thanks so much for your work with the Billing team. I’m sorry we are losing you, and am also so glad to know that you are out there at the forefront of inspiring others elsewhere, not only in the work you do, but also how you go about it.

– Laura

Hey Rachael, Just a big THANK YOU for helping me out all this time. I've had so many questions, and you've always been there to answer them. I always knew I could count on you. Thanks for those heartening chats when I needed a boost, and for including me in webinars and recordings - it really helped me improve. Remember that funny mistake I made on a recording when I called us 'Rochael'? We sure had a good laugh! I'm gonna miss those times and working with you. Can't wait to catch up with you over a drink the next time I'm in town. Wishing you all the very best and once again, thanks for everything!

– Rosa

I am happy we got to enjoy some delicious vegetarian/vegan meals and wine together. I guess I should also mention that I enjoyed recruiting, HR and business fun with you too. Thank you for being such a big part of Crossref for 12 years! Have fun conquering your new chapter. Congratulations!

– Michelle

Rachael! You will be missed. I have really enjoyed our chats and work together. I will miss our wide ranging talks about food, books, and your descriptions of all the sportsing, which I would admire because I can barely manage a short run. :-) Thanks so much for being you and let’s stay in touch! Congratulations on your new endeavor, you’re going to be great.

– Esha

When Rachel joined Crossref she brought a lot of enthusiasm and interest in learning about all that we were doing and also about what we could do. Her ideas and engaging leadership are wonderful for creating interest and drive to make projects happen. It has been wonderful to work with her over the 12 years here. I always look forward to seeing her and hearing what she has been doing outside of Crossref as well as inside. I will miss her but I know she will be doing great things wherever she may be.

– Tim

We’ve had a number of opportunities to reminisce, gassing each other up about how great it has been to work together, so I won’t do too much more of that here. But we will continue building on all of your contributions at Crossref and will carry forward your truth-telling and problem-solving approach to the work we do here. Best of luck with all the future has to offer, and we will certainly miss having you on the team.

– Patrick P

Rachael - I will miss you. I’ve really enjoyed working with you, hanging out while traveling, and getting recommendations on good books to read. Crossref won’t be the same without you. I think you have worked in the most different areas of Crossref and on the most projects of anybody, ever. Your commitment, professionalism and humour helped make Crossref what it is today. Your sportsing is also very impressive. All the best.

– Ed

Not all heroes wear capes! Rachael defines that saying so much with her ethic of getting things DONE! I know she loves to get things done but the speed and quality in which that happens is second-to-none. Rachael will be massively missed at Crossref and 67Bricks don’t yet know what they have found. I enjoyed working with Rachael throughout my tenure at Crossref, she has helped me a huge amount in developing my programming skills and has always been encouraging throughout, especially with the ’toil-bashing’ which is substantial and overwhelming at times.

On a more personal note, she is a great drinking buddy and always motivates me to be more active… by making me feel lazy. The number of hours Rachael would work was crazy, but then I always thought that anyone who gets up that early to go for runs must be a little crazy! AIl the best in your future endeavors and don’t be a stranger.

– Paul

When I started at Crossref in March 2015—at the UKSG conference in Glasgow—Rachael was leading a workshop on text mining, showing off in full glory her ‘unicorn’ mix of skills from her technical knowledge of metadata and APIs to her facilitation techniques with a large group of people, clearly a community whose needs she knew inside out. Later that evening, Rachael took it upon herself to induct me in the ways of Crossref. One of the most important things she thought I should know was that we were all trusted and treated like adults - there was no micromanagement and I was to feel completely free to challenge the status quo.

After one of the first ‘LIVE’ events, in Vilnius, I realised that it was Rachael who had created and embodied that trusted vibe through her own approach. She has been entrusted with so many programs, projects, teams, and tricky situations. Almost every launch, release, announcement, or achievement at Crossref very likely had Rachael’s eye on it at some stage, certainly the ‘actually-getting-it-done’ stage. Our close working relationship over the last nine years grew into a great friendship and I’m not quite sure how I’ll feel when the reality sets in and she’s not here for a quick chat, always a reality check. Working with Rachael has been inspiring, exciting, reassuring, and hilarious (that dry 'Norn Iron' humour!). 67Bricks is so fortunate and I can’t wait to watch her help them go from strength to strength, just like she has done for Crossref. See you soon, Ranty Rachael, no doubt putting the world to rights over a bottle of Malbec and many eyerolls 🙄.

– Ginny

Although our time working together only overlapped the short span of two years, I appreciate how much of a champion you were for ROR and everything else you did at Crossref! I’m sure you’ll continue to do the same, among many other great things, in your new journey at 67 Bricks. You will be missed!

– Adam

Rachael, It has been wonderful working with you!! You are truly a special person. I always looked forward to when we chatted over slack, had a call together, or got to spend time together in person. You are sure to do amazing things on your next adventure. You will truly be missed!! I hope we can stay in touch! Good luck, Rachael!!! Fondly, Amy.

– Amy

I am happy I got to meet Rachael when I joined Crossref in December 2023. We spoke generally about the Products team at Crossref, the differences and similarities between the African and British culture and upcoming projects on automation. You were really patient towards explaining and providing great information on metadata and research. Thank you so much for always responding swiftly to my requests pertaining to Finance issues. I have no doubt that you would be missed at Crossref and would keep doing great things into the future!!! Congratulations Rachael.

– Patience

I will greatly miss working with you, Rachael; you have been a stalwart of reliability and enthusiasm during my time at Crossref and the organisation will not be the same without you. That said, of course, I wish you all the best of luck and success in your future endeavours!

– Martin

Rachael- Congratulations on this new opportunity, I am thrilled for you! I am also very sorry that our time at Crossref did not overlap much and I am grateful for all the chances I had of interacting with you (including being able to meet you in London recently)- you were always very helpful and kind to me. I am hopeful that our paths will cross again in the future. We will definitely miss you here, and I wish you all the best for all the exciting things ahead.

– Madhura

My third week at Crossref back in 2017 was at the annual meeting in Singapore, and not getting into the timezone and not sleeping for 4 days was eased by our visit to a rooftop nightclub on the penultimate night - just before you headed off to Indonesia for a series of meetings with members and sponsors. I still don’t know where you get all your energy!
I’m so sorry you’re leaving - I’ll really miss your honesty, your approach to getting things done, and of course seeing Rosie on our zoom calls. Looking forward to seeing what’s next for 67 Bricks - exciting times!

– Amanda B

Rachael, it’s been a pleasure to work with you. You’re always ready to help and ever full of information. We’ve only just got coordinated on the perennial challenge of timelines! You took things on and got them done, as you said. The world of schol comms won’t even know how much it has to thank you for, probably chiefly for seeing the Retraction Watch data acquisition through and opening it up for all. I will miss your honesty and energy, and the opportunity to challenge you again on the amount of food consumed in one sitting… I don’t think you’ll need luck in your next place, but I wish you that it is all you want it to be.

– Kora

I’m so glad to have met you in person over these couple of days in London shortly after I joined Crossref and it’s such a shame we didn’t have much time to work together more and spend more time (not working) together. Thanks for the introduction to the Scampi Fries - you’ve changed my life forever (for the best obviously)!

– Maryna

Thank you for your collaboration and friendship over the past decade! You will be missed. We've worked on a long series of abbreviations, acronyms, and portmanteaus! Thanks for organizing countless things, from conference satellites to conference rooms. Your long record as fire warden was unblemished. 67bricks will benefit from your singular drive and attention to detail. All the best!

– Joe

Rachael! One thing I admire most in a person is a facility with metaphor accompanied by the ability to see to the heart of a matter, and hoo boy do you have those qualities in spades. I remember so clearly your talk at the Crossref team meeting in Spain in 2023 in which you clarified the Big Picture for us all in an extremely enlightening way, and then, in a smaller but equally impressive achievement, casually mentioning in a Funder Registry meeting that funders should start “stretching and warming up” for the transition to ROR – boy did I latch on to that terrific image. I wish you all the best at 67 Bricks.

– Amanda F

Rachael, thank you so much for all the support, patience, honesty, and determination. I will certainly miss our chats, work-related and non-work related. I wish you all the best in your new ventures!

– Dominika

Rachael - thank you for your boundless patience, generosity, and sense of humour. I’m very grateful I got to learn the Crossref ropes (cropes?) from you. Looking forward to randomly running into you on the Bristol karaoke circuit in 10 years’ time and performing an epic duet of Dancing in the Dark together. There’s a joke in there somewhere about you being the boss.

– Lena

Rachael, Congrats on your new opportunity. You will be greatly missed here. Through the years we have only been at the same events in person a handful of times but I will always remember your amazing personality and sense of humor. I am thankful to have spent some time with you at 2020 PIDapalooza.

– Maria

Thank you, Rachael. Thank you. I know everyone is telling you that they’re sad to see you go (I am too; we all are). I keep thinking if I delay telling you that, maybe the day won’t come when you walk out the Crossref doors. But here it is. Just wanted to you to know that I appreciate you. I appreciate you pushing us forward. I appreciate you being an advocate for all things Crossref. We’ll all miss you. Best of luck at 67bricks!

– Isaac

On one of our first meet ups together, I drove us from the Lynnfield office to Logan airport in rush hour, and we managed to survive the Bostonian road rage in one piece. We spent the ride talking through the intricacies of a sponsoring organisation’s agreement. Rachael has been a safe set of hands and an encyclopedia of institutional memory for Crossref for 12 years.
Rachael is one of those people who’s as equally competent as she is a pleasure to work with. She’s an innate leader because people want to get behind her. She shows her depth of understanding while also inviting input from everyone in the room. I’ll miss our Zoom calls, our marathon Friday sessions, and our post-meeting pub visits.

– Lucy

Hello! Here’s to hoping your new workplace appreciates you as much as you were here – they’re lucky to have you. I only wish we had the chance to interact more. Many hugs!

– Luis

Rachael, I will really really miss you, professionally and personally (but you know this already !). I'll miss all our work, dog, book and putting the world to rights chats. You'll be brilliant whatever you do and wherever you go (67Bricks have no idea how lucky they are !). Just keep 'getting stuff done' and have fun 😀

– Fabienne

You will be sorely missed but can be very proud of what you’ve done during your time at Crossref, I’m sure you’ll continue to have a big impact. You’ve always been a pleasure to work with: efficient, supportive, and always with a sense of fun and enjoyment. That’s probably one of the things that drew me to Crossref even before we worked together as colleagues. Thanks for the support and positivity you’ve brought on many, many occasions and best wishes for the future!

– Martyn

Hey Rachael! I might have not had the chance to meet with you much while still around but I’ll definitely miss your jokes and the good vibes you were bringing to each call! Looking forward to taking over your place for board games when around Bristol ;) Wishing you a great start in the new place!

– Panos

ThunderCats are on the move. ThunderCats are loose. Says it all, really. Best of luck in your new endeavours.

– Mike

Crossref won’t be the same without Rachael and we wish her well on her way to even greater things.

Good luck, Lammey!

Ed Pentz accepts the 2024 NISO Miles Conrad Award

Rosa Morais Clark — Tue, 13 Feb 2024 00:00:00 +0000

Great news to share: our Executive Director, Ed Pentz, has been selected as the 2024 recipient of the Miles Conrad Award from the USA’s National Information Standards organisation (NISO). The award is testament to an individual’s lifetime contribution to the information community, and we couldn’t be more delighted that Ed was voted to be this year’s well-deserved recipient.

During the NISO Plus conference this week in Baltimore, USA, Ed accepted his award and delivered the 2024 Miles Conrad lecture, reflecting on how far open scholarly infrastructure has come, and the part he has played in this at Crossref and through numerous other collaborative initiatives.

Established in 1965, the Miles Conrad Award gives recognition to those who’ve made substantial contributions to the information community over a lifetime. Named after the founder of the National Federation of Abstracting and Indexing Services (NFAIS)—an association that since merged with NISO—the award encourages innovation in content management and dissemination. Over the years, leaders and innovators who have significantly influenced the field of information exchange have been honored with the award. Ed has joined an illustrious group!

Ed’s leadership in collaboration and diplomacy has led to Crossref’s success in making research objects more accessible and useful to a wide global audience, including publishers, researchers, funders, societies, libraries, and more. Crossref’s founding purpose is stated as:

“To promote the development and cooperative use of new and innovative technologies to speed and facilitate scientific and other scholarly research”.

Acknowledging his privilege as a Western, university-educated, white man, which he comments has helped his career, Ed prioritises collaboration, open communication, teamwork, and equity in creating a positive, trusted environment that has brought together a diverse team of 49 colleagues from 11 countries. The organisation’s culture allows everyone to grow and contribute to the mission of a connected research nexus by including and developing solutions for community members across the globe.

Before his journey with Crossref, Ed held a number of roles at Harcourt Brace, including launching Academic Press’s first online journal. This experience led to his involvement with the DOI-X pilot project, which became the foundation for Crossref. Since its launch in 2000, under his leadership, Crossref has become an important component of the research ecosystem, an open scholarly infrastructure with nearly 20,000 members across more than 150 countries. Crossref is now the main source of >155 million records about all kinds of research objects and this open metadata registry is relied upon by thousands of tools and services across the whole research system.

Ed’s influence is also evident throughout the wider world of open scholarly infrastructure; aside from establishing Crossref, he co-founded ROR and was a founding member of ORCID, where he also served as board Chair. Further, he has engaged with the community by holding various advisory positions, including the DOI Foundation, the Digital Object Naming Authority (DONA), and the Coalition for Diversity in Scholarly Publishing (C4DISC).

Ed also emphasised that the long-term success of community initiatives lies in patience and the ability to agree on high-level principles of purpose and governance, which oil the wheels of collaboration, encourage participation, and enable more progressive change that builds and lasts over time. He says, “to solve collective problems it takes collaboration and diplomacy, bringing together a group of stakeholders, balancing their different concerns, building trust, and reaching consensus.”

The adoption of the Principles of Open Scholarly Infrastructure (POSI), along with (so far) 14 other organisations, was a key turning point for Crossref, Ed said, and one which has already paved the way for more openness of key metadata for the community, including references and retractions, as well as closer partnerships with many of the other POSI adoptees, given their shared understanding and experience.

Referencing the current “peak hype” around artificial intelligence (AI), Ed points to the challenge of research integrity and the “growing field of science sleuthing” as a forthcoming area that Crossref and open metadata may help tackle at scale, including through Crossref’s Integrity of the Scholarly Record (ISR) Program and—of course—community-wide collaboration.

In concluding his talk, Ed describes his hopes and dreams for scholarly communications in the future. He would like to see more balance in diversity in the leadership of open scholarly infrastructure, extended integrations among the various foundational infrastructures, and a fully connected system where the scholarly record is inclusive globally.

Ed, on behalf of all your proud colleagues at Crossref, thank you and congratulations!

ISR Roundtable 2023: The future of preserving the integrity of the scholarly record together

Madhura Amdekar — Tue, 06 Feb 2024 00:00:00 +0000

Metadata about research objects and the relationships between them form the basis of the scholarly record: rich metadata has the potential to provide a richer context for scholarly output, and in particular, can provide trust signals to indicate integrity. Information on who authored a research work, who funded it, which other research works it cites, and whether it was updated, can act as signals of trustworthiness. Crossref provides foundational infrastructure to connect and preserve these records, but the creation of these records is an ongoing and complex community effort. Crossref has always shown a deep commitment to preserving the integrity of the scholarly record in an open and scalable manner.

Given the increasing concerns in the community about matters of research integrity and integrity of the scholarly record (ISR), we at Crossref have been engaging with community members to understand what developments are needed. In 2022, we organised a roundtable discussion to talk about our role and the applicability of Crossref’s services in preserving and assessing the integrity of the scholarly record. We’ve acted on much of that feedback since, and so in October 2023, we organised a follow-up event, once more gathering representatives of publishers, research integrity experts, policy-makers, academic institutions, funders, and researchers (the full list of participants can be found in the appendix). This post aims to offer insight into the discussions at this event and the next steps. The objective of this event was to take the conversation forward by:

Sharing the progress made by Crossref on matters related to ISR since the last roundtable event.
Sharing information about how metadata contributes to the Research Nexus, and can act as trust markers for research outputs.
Apprising the community about the latest membership trends and examples of activities that we see, such as title transfer disputes, unregistered DOIs, requests for deleting records, and sneaked references .
Building upon the ideas discussed during the 2022 roundtable event to progress the conversation about issues related to ISR.
Learning from the participants about their experiences of pursuing research integrity initiatives.
Last but no less importantly, hearing from the participants their perspectives on strategies for preserving the integrity of the scholarly record, and opportunities for collaborating to leverage metadata to assess the integrity of the scholarly record.

The event was kicked off by Ed Pentz, who spoke to the participants about how integrity is key to Crossref’s mission, and Crossref’s vision of the Research Nexus. Next, Amanda Bartell, the Head of Member Experience at Crossref, shared the recent developments and trends in community behaviour. She expanded upon the actions taken by Crossref as part of its ISR program since the last roundtable event, which include:

Acquisition and opening of the Retraction Watch database, which makes it easier to access information on retractions and corrections.
Increased participation in the Global Equitable Membership (GEM) program, enabling a wider section of the community to provide and access trust signals.
Newer developments around metadata that act as trust signals: e.g. 120K grants or awards now have a Crossref DOI, and the planned transition of the Open Funder Registry into ROR.
Recruitment of a Community Manager to focus on working with publishers and editors, including on ISR (that’s me!), and recruitment of a Technical Community Manager to enable greater use of our APIs.

Amanda highlighted that all Crossref members should be using ROR IDs to provide affiliations for authors (along with ORCID iDs) in their Crossref metadata. She also shared some latest examples of community behaviours that we have seen, such as requests from authors to delete records of works that were published without their permission, title ownership disputes between publishers, and the recent instance of sneaked references.

Ivan Oransky, co-founder of Retraction Watch, and Lena Stoll, Product Manager at Crossref, were next, and they spoke about the future of the Retraction Watch database, and about the Crossmark service. After this, some of the other roundtable participants shared initiatives that they have undertaken that support ISR:

Jodi Schneider from the University of Illinois Urbana-Champaign spoke about NISO’s CREC Working Group that has created a Recommended Practice that should be followed by relevant stakeholders for communicating retracted research (Crossref’s Director of Product Rachael Lammey was the co-chair of that group).
Kihong Kim from the Korean Council of Science Editors shared information about the workshops that the Council has organised for researchers on publishing in journals.
Alberto Martín-Martín from Universidad de Granada presented his thoughts on how to reconcile the publishing system and the institutional view of tracking research outputs.
Bianca Kramer from Sesame Science spoke about her analysis of and the implications of sneaked references, duplicate references, and missing references for citation integrity.
Joris van Rossum from STM Solutions spoke about the STM Integrity Hub and the integrity tools that are being developed in collaboration with some publishers.

Some of the most valuable reflections stemmed from discussions in small groups on these three key questions:

What value do you see in the integrity and completeness of the scholarly record in the way you operate? How do you contribute to it? How can it support you to achieve your own goals?
Are you aware of Crossref services? What are the barriers to more uptake? What are the challenges and opportunities?
What information is essential and nice to have for you in the scholarly records to support trust signalling and ascertaining trustworthiness?

As groups shared their discussions, a few themes became apparent that I would like to elaborate on further.

What is “complete”?

Given the prompt to talk about the value of completeness of the scholarly record, an immediate reaction at most tables was: how much metadata qualifies as “complete” metadata? Can the scholarly record be considered complete if some publishers or journals do not use Crossref? What is the optimum level of metadata that should be deposited by members - should a minimum data standard be defined by disciplines, or should there be standard data requirements for all? The composition of metadata appears to change over time, too, as the processes change and our ability to record their facets increases. While there were spirited discussions about what constitutes a complete scholarly record, everyone agreed that “completeness” of metadata, as much as is possible, should be the aim. Unambiguous and consistent standards may help with this, for example, the Metadata 20/20 community creation of principles and best practices, and potentially also using a set of recognition standards and reproducibility badges.

Global participation is equally important for a truly “complete” scholarly record. In order to enable as many in the scholarly community as possible to participate in Crossref services and metadata, Crossref launched the Global Equitable Membership (GEM) program in 2023. Under this initiative, membership and content registration fees are waived off for members from the least economically advantaged countries. We are seeing first signs that this initiative meaningfully lowers the barriers to participation for organisations based in those countries, and allows the global community to contribute towards the building of a comprehensive research ecosystem.

At the end of the day, it is important to recognize that rich metadata is crucial because it can be used for all kinds of analysis, which in turn can drive decision-making. Even if some of the metadata components are sporadically missing, that could be acceptable, because every piece of data counts!

Corrections and Retractions

Similar to last year, retractions and corrections continued to be a topic of great interest in this year’s roundtable. This was not surprising given their relevance as trust indicators as well as the recent development with the acquisition of the Retraction Watch database by Crossref. Having heard from Ivan about the Retraction Watch taxonomy of reasons for retractions and the metadata included in the database, participants expressed the need to investigate this taxonomy as a community standard. While the Retraction Watch taxonomy is not widely known, we at Crossref are working to map the Crossmark taxonomy with the Retraction Watch taxonomy, which will enable complete integration of the Retraction Watch database with the Crossref database.

It would also be useful to add more information to retraction notices. Having more information about the reasons for retraction will not only destigmatize retractions, but certain additional information, such as submission dates for those outputs, might help with ethical investigations to determine whether manuscripts were being submitted to multiple publishers simultaneously.

On the topic of retractions, another aspect that came up in the room was about incentives for researchers to publish as much and as quickly as possible. If researchers indulge in unethical publishing practices due to this pressure to publish, that is hugely detrimental to the cause of research integrity and to the progress of scientific research in general. However, there is a distinction to be made between the integrity of the research and the integrity of the scholarly record - unethical research and publishing practices, including but not limited to data falsification, fabrication, and plagiarism, affect research integrity while integrity of the scholarly record is affected by unavailability of metadata, outdated metadata, incomplete metadata records, and incorrect metadata (e.g. as seen in the case of sneaked references).

There was a lot of discussion about Crossmark, a cross-platform service provided by Crossref that allows readers to discover whether an item has been updated, corrected, or retracted just by clicking a button that is standardised across publication platforms. While most participants acknowledged its importance, they also pointed out that its uptake has been limited and publishers do not use it as much, perhaps because it is difficult to implement and there’s a matter of providing more clarity about it to the readers. There were suggestions to add a notification system to Crossmark such that every time a published output is retracted, a notification goes out. This seemed of particular interest to funders, whose grievance was that they are usually the last to find out when research that they have funded is retracted. They would welcome notifications that would alert them to such events.

We already have plans to consult with the community more specifically about what changes they’d like to see to Crossmark that will enable them to implement it easily and use it more frequently. Take a look at this thread on our community forum and add your thoughts for our next steps on Crossmark.

The importance of education

There was an overwhelming sentiment that there was a need for collective arbitration of research integrity issues. However, everyone recognized that this is not a role for Crossref. We can act as a “trust broker” by bridging different metadata and identifiers that otherwise might not interact, creating a network of research outputs whose credibility can be verified by others. Many participants called for Crossref to increase its efforts in educating community members about the importance of metadata and how different pieces can be linked together to make meaningful connections.

Research practices vary between countries, and between institutions. Correspondingly, the metadata being provided by diverse Crossref members may also vary. There is an opportunity here for the global research community to work together to increase awareness about ethical standards, so that a lack of specific metadata or its variances (e.g. unusually formatted metadata, or non-standard metadata fields) may not be construed as “lower quality” metadata. Many felt that the greatest need for education about metadata is for the academic community – although individual researchers contribute a wealth of metadata associated with any published research output, they do not necessarily understand how metadata contributes to the completeness of the scholarly record. There is a further opportunity to talk to the academic community about how different metadata components link together to form a rich network, supporting visibility and confidence in their work. A greater awareness about these topics is likely to encourage researchers to provide more metadata and identifiers.

While most participants at the roundtable event agreed about the need for this conversation and the educational opportunities here, if Crossref were to lead these efforts, it would represent, in some eyes, a diversion from its mission. We do have several initiatives already to support our communities. As part of the Crossref Ambassadors program, volunteers from the international scholarly community who believe in Crossref’s mission liaise with our team to conduct training in their communities about using Crossref services and, generally, about the importance of metadata. In 2023, we also launched a new online public forum, the PLACE, in collaboration with the Committee on Publication Ethics (COPE), the Directory of Open Access Journals (DOAJ), and the Open Access Scholarly Publishers Association (OASPA). This forum is a place where new publishers can connect with these organisations and learn about best practices in scholarly publishing via discussion posts and by asking questions, as they get started. Another initiative that is designed to help new Crossref members is the “Managed Member Journey”: as members join and move through the various stages of membership, key information is shared with them during each of these stages in the form of triggered automated emails, web pages, and webinars.

While Crossref’s direct interactions with researchers are limited, we welcome the community’s recognition of the need to raise awareness about these matters. We have started engaging more closely with the reporters of metadata issues, in many cases investigators and ‘sleuths’ in the area of research integrity, and plan some closer collaborations with this group in 2024. We are open to supporting community efforts to inform other stakeholders about the importance and uses of metadata.

Incentives for the community

Another theme that was heard repeatedly was “incentives”: incentives for researchers to contribute to a “complete” scholarly record, incentives for publishers to improve metadata, and incentives for everyone to report on and register retractions.

As I mentioned before, a shared sentiment is that researchers may not be aware of the value of rich metadata. While more publications, increased citations, and greater grant funding are some examples of incentives that are part of the current academic settings, the right incentives probably do not exist for researchers to provide complete metadata. With the diverse set of participants present at this meeting, some groups also discussed how the current research assessment system can change to incorporate other metrics, perhaps those based on open science and open data.

What could be the incentives for publishers to improve the metadata collected and deposited by them? One suggestion was that clearly defined benefits of rich metadata can incentivise publishers. Being aware of what funders are mandating, can be another incentive. On the same note, funders will benefit from knowing what metadata is being provided by publishers. This metadata is available through our open API, and nine key checks on members’ activity are available through our public Participation Reports.

Retractions featured again in the discussion on the topic of incentives. As shared by Ivan, retractions are on the rise every year, with about 43k retractions currently in the Retraction Watch database. On the other hand, retractions registered in Crossmark at the time of the meeting numbered just 14k and have recently jumped up to 25k thanks to Hindawi/Wiley’s dedication to good open metadata. Besides the fact that the uptake of Crossmark by Crossref members is limited, another reason for the low number of retractions being registered is the associated stigma. Corrections and errata are usually conflated with retractions, and all these terms, which represent different kinds of updates that may happen to a published item, have a stigma associated with them in the academic community. There is a need to destigmatize retractions, and perhaps incentivize them by noting that these updates are essential to uphold the integrity of the scholarly record and to highlight the publishers that are showing leadership in addressing the issues openly through up-to-date Crossref metadata.

What metadata is nice to have in the scholarly record?

We asked everyone what information they think is essential as well as “nice to have” in the scholarly record to support trust signalling, and we heard a range of answers. Peer review information was recognized to be important. This would include data on who the peer reviewers were and standard peer review terminology that has been published by NISO. More generally, as much metadata as possible about the main actors of the peer review process was considered important - such as designating who the corresponding author is, and who the handling editor or the decision-making editor was.

As special issues led by guest editors in journals have been brought to the attention of late due to the uncovering of irregularities in some of them, one of the first suggestions in this context was more metadata about special issues. Participants thought that it would be useful to collect and distribute information on handling/guest editors of special issues, peer reviewers, as well as submission and acceptance dates. Recently, COPE has released guidance on “best practices for guest-edited collections” , highlighting that this topic looms at the forefront for the scholarly information industry.

Adding information on ethical approvals provided by institutional review boards would add more nuance to the research outputs. Metadata about clinical trials helps to add transparency to research in a field, where reproducibility is of primary importance. Conflicts of interest are another factor that could be a cause of concern if not reported accurately; these declarations were mentioned by the participants as important for signalling trust.

Recognizing that it is the relationships in the metadata that add context to research output, participants echoed that better interlinking between preprints and their published versions is required. To aid with all of this, it has been suggested that a complete list of all metadata that can be deposited with Crossref be made available in a simple format, so that members have more visibility about all the possibilities that exist for providing metadata.

Next steps

We asked all participants if the discussions prompted them to plan to take any actions in the near future. Several attendees reflected that the discussion encouraged them to go back and review the metadata that they are depositing with Crossref, and how they can make more use of the data openly available from Crossref. We also heard how some found training opportunities therein - discussion points from the event could be included in workshops for affiliated researchers, and in COPE guidance for members. As encouraged by members of the NISO’s CREC Working Group, some participants were looking to respond in the (then open) consultations of the draft Recommended Practice, NISO RP-45-202X, Communication of Retractions, Removals, and Expressions of Concern (CREC). One message resonated loud and clear: preserving the integrity of the scholarly record cannot be a lone endeavour and has to be a community effort. Attendees expressed their commitment to continue these conversations, with the next most opportune time being at the STM week. Everyone recognised that collaboration in this space is the need of the hour: facilitating information and data sharing across all the players in the ecosystem would be crucial to progressing this topic. As Bianca Kramer declared during her presentation, “I am committed to using only open data in my research, as access to data is important for the community to detect problems at scale”.

At our end, we are looking to act on suggestions that are specific to Crossref:

Consultation with the community about Crossmark

One of the first things that we are doing in early 2024 is to consult with our community about the developments needed in the Crossmark service. Our key aim with this exercise would be to understand how we can enable a more effective uptake of this service so that Crossref members can easily fulfil their obligation of keeping their records updated. We are keen to understand what we can do to help our members to send us metadata about updates to an output, and how we can help downstream services that use this data. Insights from this consultation will also help inform how the Retraction Watch data can be most effectively integrated into Crossmark and communicated to users. Please visit the discussion and add your thoughts here: https://community-crossref-org.pluma.sjfc.edu/t/communicating-post-publication-updates-inviting-feedback-on-the-next-steps-for-crossmark/.

Development of resources for using our API

As there is clearly no dearth of metadata components that the community thinks would be “nice to have” for signalling trust, it is equally important to equip users and downstream service providers to be able to access the rich metadata that is available with Crossref. This rich metadata opens up new avenues for the development of services and resources that can benefit the scholarly community. On account of this, we plan to prioritise development of resources for using Crossref APIs. These efforts would include making available workbooks with a variety of API use cases - ranging from how to use basic API queries, to how to use APIs for obtaining grant information or for obtaining citation data and so on, as well as retrieving corrections, retractions, and update information, especially when the Retraction Watch dataset merges in with the rest of the Crossref metadata.

Working group to facilitate community efforts for preserving ISR

We are looking to set up a working group that will facilitate the various stakeholders in the scholarly ecosystem to work together towards preserving the integrity of the scholarly record. One direction for the group could be to consider the role and impact of Crossref metadata in ISR. Another area of focus will be to enrich information about retractions, corrections, and expressions of concern. Raising industry-wide awareness about the current concerns in upholding the integrity of the scholarly record, and how comprehensive metadata can act as markers of trust about research output, would be another focal point.

Continued community outreach

We will continue our efforts to engage with the community on the very important issues surrounding ISR. We are particularly keen to redouble our efforts to include more funders and institutions in these conversations. Preserving the integrity of the scholarly record needs to be a truly inclusive effort and will benefit from diverse voices in the community. With that in mind, consulting with the community in Asia is next on our radar.

We look forward to working with the community further on this important topic - if you are keen to participate in these discussions and want to contribute towards preserving the integrity of the scholarly record, we would love to hear from you. Please write to us at feedback@crossref.org if you have any suggestions on this topic.

Appendix: Participant list

Name	Role	organisation
Ed Pentz	Executive Director	Crossref
Amanda Bartell	Head of Member Experience	Crossref
Madhura Amdekar	Community Engagement Manager	Crossref
Luis Montilla	Technical Community Manager	Crossref
Lena Stoll	Product Manager	Crossref
Kora Korzec	Head of Community Engagement and Communications	Crossref
Ivan Oransky	Co-Founder	Retraction Watch
Jennifer Wright	Research Integrity Manager	Cambridge University Press
Guntram Bauer	Director of Science Policy & Communications	Human Frontier Science Program
Wendy Patterson	Scientific Director	Beilstein-Institut
Sarah Jenkins	Director, Research Integrity & Publishing Ethics	Elsevier
Helene Stewart	Director, Editorial Relations Web of Science	Clarivate
Bianca Kramer	Advisor, Research Analyst, Facilitator	Sesame Open Science
Adya Misra	Research Integrity and Inclusion Manager	Sage
Andrew Joseph		Wits University Press
Theodora Bloom	Executive Editor	BMJ
Alberto Martín-Martín	Assistant Professor	Universidad de Granada
Aaron Wood	Head, Product & Content Management	American Psychological Association
Fred Atherden	Head of Production Operations	eLife
Kihong Kim		Korean Council of Science Editors
David Flanagan	Senior Director, Data Science	Wiley
Chiara Di Giambattista	Communications Director	OpenCitations
Scott Delman	Director of Publications	ACM
Chi Wai (Rick) Lee	General Manager	World Scientific Publishing Co (WSPC)
Leslie McIntosh	VP, Research Integrity	Digital Science
Adam Day	Director	Clear Skies
Damaris Critchlow	Project Manager	Karger
Tamara Welschot	Head of Research Integrity, Prevention	Springer Nature
Kathryn Dally	Research Integrity and Policy Lead	Research Services, University of Oxford
Masahiko Hayashi	Director, JSPS Bonn Office	Japan Society for the Promotion of Science
Simone Taylor	Chief, Publishing	American Psychiatric Association
Christna Chap	Head of Editorial Development	Karger Publishers
Coromoto Power Febres	Research Integrity Manager	Emerald Publishing
Carole Chapin	Project Manager	French Office for Research Integrity
Jodi Schneider	Associate Professor of Information Sciences	University of Illinois Urbana Champaign
Oliver Koepler	Head of Lab Linked Scientific Knowledge	TIB - Leibniz Information Centre for Science and Technology
Heather Staines		Delta Think
Eri Anno		JSPS Bonn office
Joris van Rossum		STM Solutions
Anita de Waard	VP Research Collaborations	Elsevier

RORing ahead: using ROR in place of the Open Funder Registry

Rachael Lammey — Tue, 30 Jan 2024 00:00:00 +0000

A few months ago we announced our plan to deprecate our support for the Open Funder Registry in favour of using the ROR Registry to support both affiliation and funder use cases. The feedback we’ve had from the community has been positive and supports our members, service providers and metadata users who are already starting to move in this direction.

We wanted to provide an update on work that’s underway to make this transition happen, and how you can get involved in working together with us on this.

Overall, we are building more comprehensive support for ROR into Crossref’s services. Some of this work is specifically to support using ROR to identify funding organisations in place of funder registry IDs. We have a number of parallel, complementary projects underway to support different elements of this work:

We are evolving our metadata schema so that we can collect ROR IDs in places where we currently support the collection of Funder IDs.
We are analysing the coverage of Funder ID to ROR ID mappings and testing the way we expose them in our APIs.
We are developing new matching strategies to match text strings to ROR IDs.

1. Schema updates

Everything flows from being able to get ROR IDs into the Crossref metadata!

We are evolving our metadata schema so that we can collect ROR IDs in places where we already support the collection of Funder IDs – for instance, in the funding section of the metadata for works and in the funder section for grants.

We’re working with members and service providers so that they can try sending us this data via a pipeline our Labs team has built to test schema updates before they go live. We are actively recruiting members to help us test our new pipeline by providing sample XML for registration. Planned metadata inputs and outputs are detailed in Including ROR as a funder identifier in your metadata (metadata prototyping instructions), we’d encourage you to provide feedback on these in the document, ideally in the next two weeks. We’re aiming to release an updated schema that supports these changes in Q1 2024.

2. Modelling ROR ID/Funder ID mappings in our metadata model

We have integrated the ROR registry into our evolving metadata model, and we have started work to integrate the Funder Registry. The aim is to create more flexibility in how Crossref’s metadata can be supplemented and queried, and give more clarity as to which party asserted or created a metadata element.

We’re working on an early iteration of how the model handles ROR IDs, funder IDs and their equivalencies. Once we have something to share, we’ll welcome community feedback on this approach and on the metadata model in general.

3. Developing new matching strategies to match text strings to ROR IDs

Ideally, everyone would always use persistent identifiers to exchange information about contributor and awardee affiliations, organisations related to works, as well as funders supporting the research. In practice, this information is often exchanged as data without identifiers, such as affiliation strings (e.g. “University of Virginia, Charlottesville, VA”), funder names, or even funding acknowledgements (e.g. “Funding and support generously provided by the Ford Foundation”). In such situations, a good metadata matching strategy can help map these to persistent identifiers.

Currently, we are focused on developing reliable strategies for matching affiliation strings to ROR IDs. In the future, we will adapt the strategies to support funder names and funding acknowledgements as well. All the strategies will be rigorously evaluated using real-life data. We will make the strategies, as well as the evaluation datasets and evaluation results, publicly available for anyone to use. If you are interested in collaborating on the development or the evaluation of the matching strategies, please get in touch!

In the future, we might also apply some of the new matching strategies at Crossref, to the metadata our members send us. This would allow us to insert matched identifiers to the metadata to better connect organisations with other items in the scholarly record. We already have a process that matches the names of funders supporting research against the Funder Registry and enriches the metadata with matched Funder Registry IDs. Developing and evaluating reliable matching strategies will allow us to modify this process to use ROR IDs instead, and extend it to support other use cases, such as contributor affiliations.

What will the transition mean for you?

We do recommend that you begin looking at what it will take to integrate ROR into your systems and workflows for identifying funders. Talk to your service providers about this to ready them for this change. To reiterate the point from the earlier post, in the short term, and even in the medium term, Funder IDs aren’t going away and the Funder IDs will continue to resolve – they are persistent, after all. Eventually, however, the Funder Registry will cease to be updated, so any new funders will only be registrable in Crossref metadata with ROR IDs. Legacy Funder IDs and their mapping to ROR IDs will be maintained, so if Crossref members submit a legacy Funder ID, it will get mapped to a ROR ID automatically. Note, too, that Crossref is committed to maintaining the current funder API endpoints until ROR IDs become the predominant identifier for newly registered content. We also know that there are questions that we’ll want to tackle with the community as we all make progress, some we know and some we don’t know. With that in mind:

Tell us what you need!

We want to hear from you! We have set up several channels of communication meant to ensure that you can tell both ROR and Crossref what will make this transition easier for you and that you can get answers to your questions.

First, we are conducting a series of Open Funder Registry user interviews designed to deepen our understanding of where Funder IDs are being used in workflows and systems. Write community@ror.org if you’d like to participate in these interviews to show and tell us how you’re using Funder IDs.

Second, in 2024, we will be running a follow-up to the funding data workshop we ran in June 2023. Please get in touch if your organisation would be interested in participating in the discussion.

Solving your technical support questions in a snap!

Isaac Farley — Thu, 25 Jan 2024 00:00:00 +0000

My name is Isaac Farley, Crossref Technical Support Manager. We’ve got a collective post here from our technical support team - staff members and contractors - since we all have what I think will be a helpful perspective to the question: ‘What’s that one thing that you wish you could snap your fingers and make clearer and easier for our members?’ Within, you’ll find us referencing our Community Forum, the open support platform where you can get answers from all of us and other Crossref members and users. We invite you to join us there; how about asking your next question of us there? Or, simply let us know how we did with this post. We’d love to hear from you!

A little about us and what drives the team

I’m fortunate to manage a great team - Evans, Kathleen, Paul, Poppy, and Shayn - who enjoy and are hardwired to guide. We have different strengths and interests, but the thing that unites us is that we are energized when we can unpick tricky problems for all of you, our members and users. In 2023, the technical support team answered around 11,000 questions from all of you. We do that with one-to-one requests sent to us via email and within our support center (using a closed-source software called Zendesk). And, we’ve been providing more and more support in our Community Forum, where we’re aiming for open interactions, so we can all learn from the rich exchanges with all of you (the Forum has an integration with Zendesk, so posts made in the Forum are delivered to us there, so our team won’t miss any of your questions).

We established in the previous paragraph that we have a great technical support team who all pride themselves on helping you. But we’re also human; the reality is that many of those ~11,000 technical support questions asked of us in 2023 were repetitive, and there are always trends in the questions asked. That’s another important reason why we’re hoping to have more and more of these questions asked and answered within our Community Forum; again, so we can all learn from one another. We know certain parts of content registration, metadata retrieval, and everything in between are, well, complicated. The Crossref learning curve can be steep for all of us. Collectively, our technical support team has more than 25 years of Crossref experience, and we’re continuously learning new things about the Research Nexus and the scholarly ecosystem from one another and all of you.

Learning through this complexity is one of the most enriching parts of our days. Our daily stand-up, modeled off of different software development methodologies, where together we troubleshoot tangly questions from all of you, share ideas, and just keep up-to-date on the latest from across the organisation leads to a lot of knowledge exchange. So, years ago, we decided to transform the issues we discuss in those stand-ups into public-facing posts in our Community Forum. It gave us the opportunity to share much-needed examples in a new community space; and, we knew, since these were the issues we were all discussing and learning from ourselves, that many of you would also benefit from us surfacing the topics openly. We call these posts tickets of the month, since the majority of topics we discuss have originated from tickets in our support center.

Examples of some of the most popular topics in the last two-plus years have been:

Snapping our fingers

Like I said, these posts originated from real-life questions of us from our community members. In most cases, we’ve been asked these questions by many of you. These Community Forum posts are our attempts to unlock understanding of our services, rich metadata, or the larger Research Nexus. Said another way: we all see value in putting in the effort to post one more example or answer that nuanced question. Perhaps one of our posts will include an example that really resonates with you and/or your work.

In that spirit, I asked Evans, Kathleen, Paul, Poppy, and Shayn to answer this question below (yes, I’m going to weigh in, too):

What’s that one thing that you wish you could snap your fingers and make clearer and easier for our members?

Evans, Technical Support Specialist

As a publisher and a Crossref member, at one point or another, you might have made a mistake in the metadata deposited for a given DOI. I’m sure after the slight ‘shock’, the next question you had in mind was, ‘How can I correct this mistake?’ Well, here is a simplified guide on how to do that correction/update!

Can I modify/ update the metadata of a registered DOI? As indicated by my colleague Shayn below in this blog post, Crossref DOIs are designed to be persistent (and cannot be changed/deleted once registered). And YES, you can update the metadata associated with any of your registered DOIs whenever necessary, at no additional fee.

How can I perform a standard metadata update? To add, change, or remove any metadata element from your existing records, you generally just need to resubmit your complete metadata record with the correct/new changes included. How you choose to update a DOI metadata record is highly dependent on the content registration tool/platform you are using/comfortable working with, as described below:

OJS: Navigate to the article record you wish to update, add in your new metadata/delete relevant metadata fields, and deposit it again using the Crossref import/export plugin. You must be running at least OJS 3.1.2 and have the Crossref import/export plugin enabled.
Web deposit form: Open the web deposit form, and re-enter all the metadata, including the new changes - leave the relevant field blank to delete it, or add in your new metadata to update it - and resubmit the form (note: there are a handful of exceptions to this for the web deposit form).
Depositing XML files with Crossref: Make changes to the relevant XML file and resubmit it to Crossref via the admin tool. When making an update, you must supply all the bibliographic metadata for the record being updated, not just the fields that need to be changed. During the update process, we overwrite the existing metadata with the new information you submit, and insert null values for any fields not supplied in the update. This means, for example, that if you’ve supplied an online publication date in your initial deposit, you’ll need to include that date in subsequent deposits if you wish to retain it. Note that the value included in the element must be incremented each time a DOI is updated.

If you’re looking for real-life examples of other members who have updated their metadata, the Community Forum is a great starting point. If you have follow-up questions on any of the existing threads, I invite you to post a message today.

Kathleen, Technical Support Specialist

One of my favorite types of queries to tackle are those regarding content registration problems. I love a good mystery and getting to the bottom of why that pesky submission just didn’t succeed. Sometimes members come to us with an error message and specific questions about what has gone awry. But, in fact, two of the most common questions we receive are: 1) I deposited something; did it work? and 2) I deposited something; why isn’t it showing up?!

To address the first question of whether your submission went through or not, I wrote a forum post back last June talking about how to use the admin tool to see whether your registration was successful or not. We know there are also email alerts and perhaps status messages within your own registration platform, but using the admin tool is a great way to concretely check where your submission has ended up. If it’s not there, we didn’t get it!

Using the admin tool is also a great way to get more details about the submission and more information in case the submission happened to fail. You may have had the experience in which you contacted us with a question about a failed deposit, and we asked you for the submission ID. You can find that info in the admin tool! And we ask for that, because that helps us get to the bottom of those error message mysteries.

And, as for the second question of when will your DOI be active, my colleague, Paul, wrote a fantastic post on the forum (with an excellent flowchart and all!), explaining when you can expect to see your DOI up and running. Often members will submit a deposit and expect the DOI to resolve immediately. When that doesn’t happen, many think that something has gone wrong or perhaps there is an error, but, in fact, our systems may still be updating and processing the metadata.

I recommend giving these two posts a read if you’re at all concerned about whether you’re depositing your content correctly or not. Hopefully, they’ll help ease your content-registration worries.

Isaac, Technical Support Manager

Oh, thanks for asking! Many of our members, after receiving one of our reports, will respond to us in support with a message similar to: ‘What did I do wrong? Please help me fix this. I don’t want to be out of compliance!’

The receipt of one of our reports does not necessarily mean that you’ve done anything wrong. In truth, the reports we send to our official member contacts are produced using very simple logic. It’s true that they may signal larger, more complicated problems, but we really need your help to determine next steps (and, in some cases, no action is needed because there is no issue for members to fix (e.g., many failed resolutions within the resolution reports)).

Let’s look at the conflict and resolution reports since those are the reports we get the most questions about:

Conflict reports are the most complicated of our reports to navigate. But, the reports are generated using simple logic: if you register two or more DOIs with matching bibliographic metadata, we’ll flag those DOIs as being in conflict, which will generate a warning message at the time of registration and a subsequent conflict report. When members receive this report, we often get the sense that members simply want us, the technical support team, to tell them how to fix it. The problem is we don’t know your content, so we don’t know if the two DOIs do represent a duplicate, or if both DOIs, while having very similar bibliographic metadata, are legitimate and will be maintained going forward (e.g., for errata). Paul wrote a great post in our community forum about what conflicts are and how to resolve them.

Resolution reports, like conflict reports, are generated using simple logic: a resolution is the result of a click on that DOI. If a DOI has been registered, that click results in a successful resolution. If that DOI has not been registered, that click results in a failed resolution. Our monthly report is a count of those resolutions - successful and failed. Failures can represent content registration errors in a member’s workflow. Or, they can signal that an end user has made a mistake when attempting to click the DOI in question. So, for example, an end user perhaps added an extra period onto their DOI link. Instead of trying to resolve https://doi-org.pluma.sjfc.edu/10.5555/cupnfcm2wj, a legitimate DOI, they added a period to the end and tried to resolve https://doi-org.pluma.sjfc.edu/10.5555/cupnfcm2wj. instead. That extra period at the end of the DOI has made it a completely different DOI that is not registered with us, thus they get a failed resolution. This is pretty common. For members with content being regularly clicked, there will be user errors in the logs appearing as failed resolutions. The first question members should ask themselves when reviewing the failed .csv report within the resolution report is: ‘are any of these DOIs legitimate DOIs that I thought we had registered?’ We have more on the basics of resolution reports also over in our Community Forum.

Paul, Technical Support Specialist & R&D Support Analyst

I know we were asked to name “one thing” but I have two that are closely related. May I snap my fingers twice and fix two issues? [Of course, Paul! Take it away!]

Paul’s first snap

One of the most asked questions we get in support is “why is my DOI not working?” 90% of the time it is down to a failed submission. A good proportion of those failures are a result of title mismatches between the deposited container title and the one we have stored on the system here. There are other error messages that occur, too, which I wrote about back in 2020.

So, “why do we fail submissions because of title differences?” You might ask.

Well, the title and ISSN/ISBN and/or the title level DOIs act like locks to the title record, which need the right keys to unlock the title so that you can add or update the records against it. So if you don’t match what was in the original submission, you get a failure. Without that stringent check, we would have way too many iterations of titles and matching to those would be a nightmare. Not to mention sorting those DOIs into one container in the REST API.

Isaac wrote a great forum post about these title-level issues as well.

If a title update is required due to an error with an original title deposit, then these need to be made by the support team, so get in touch with us on the Community Forum.

And, a second

Permissions against titles and DOIs: Lots of our members don’t realise that each DOI has its own permissions against the prefix that currently ‘owns’ or is associated with that DOI in the background.

It would be fair to assume you can tell just by looking at a DOI who the current publisher is, based on the prefix at the start —but that’s not always the case. Things can (and often do) change. Individual journals get purchased by other publishers, and whole organisations get bought and sold.

What you can tell from looking at a DOI prefix is who originally registered it, but not necessarily who it currently belongs to. That’s because if a journal (or whole organisation) is acquired, DOIs don’t get deleted and re-registered to the new owner. The update will of course be reflected in the relevant metadata, but the prefix on the DOI will stay the same. It never changes—and that’s the whole point, that’s what makes the DOI persistent. Isaac also wrote this in much more detail and explains the internal Crossref processes in his blog “What can often change, but always stays the same?“

These permissions are very important to understand when it comes to title transfers and working with updating your metadata against transferred DOIs to prevent duplicate DOIs for the same work.

Poppy, Technical Support Contractor

As a researcher myself, I’d like to talk about references in a journal article, book, conference paper, etc. (I’ll just use ‘article’ going forward for simplicity). These are the references included in an article by the author. References in one article result in citations for another article. It’s the thing every author dreams of and accruing citations can be a big deal for authors, journals, and publishers.

For readers, articles with no references can be less discoverable using systems that use citation links for relevance, and that discoverability is of critical importance for our members who decide to register references with us. We all want your content to be shared, cited, linked, and used far and wide.

We receive many questions from authors asking why citations don’t show up; it’s usually due to metadata deposits with no references included. There may be an assumption that our process is like Google Scholar, which crawls full text and websites. This misunderstanding has a big impact on references and citation counts. However, as we do not store a copy of the paper, our intake system does not extract references from the article, regardless if they have a DOI. This is one of the main reasons that Crossref citation counts are lower than services that use extraction methods. We only store the data that a publisher registers and maintains with us. On deposit of a metadata record that includes references, our system performs a matching process - if there is a match, a cited-by connection is applied to the metadata. With deposits with no references, however, there is no data to match to other articles (and, therefore limitations on the discoverability and no cited-by count increase).

An article with no references has big impacts for the authors, the journal, the publisher, researchers, and ultimately, the readers. This can mean decreased distribution of the content itself, reduced citation counts for cited articles, lower impact metrics for journals, and can ultimately affect value for publishers. For example, researchers just don’t include articles without references for scientometric analysis.

Our documentation on references includes the elements for both structured and unstructured data. Including the DOI in the structured data is best practice as it provides a precise location with rich data for matching. If the matcher does not see a link between the deposited DOI and the cited DOI at the time of deposit, then the references are stored to be crawled with other matching algorithms later. So, we’re always working to create those rich cited-by linkages between works (raising the content’s profile and overall discoverability), no matter when you register reference metadata. You can also see how your publisher is doing on depositing references by viewing their Participation Report. If you are an author, you can check if your DOIs that were registered contained any references by using our REST API. Don’t see them? You can always contact the editor of the journal or the publisher that published your paper and ask them to add them. Didn’t hear back? Just drop us a line in the Community Forum, we’re happy to help.

Shayn, Technical Support Specialist

Let’s ‘zoom out’ to the big picture. What are DOIs for? What makes them useful? What are we all doing here anyway?!

There are a lot of different answers to those questions. It’s a complex picture. But, way back in the late ‘90s, the DOI system was designed in order to allow for the creation of unique and persistent identifiers. Crossref members use these identifiers to represent their research outputs and publications. This allows for reliable linking to those items, and the ability to identify and communicate the relationships between them, notably (but not exclusively!) citation relationships.

So, what do we mean when we say that Crossref DOIs should be unique and persistent? In basic terms, unique means that there is only a single Crossref DOI registered for a given citable research output. And, persistent meaning that the DOI associated with a given research output today will continue to be associated with, and link to, that same research output indefinitely into the future.

Yes, there are some grey areas, and we know that everything doesn’t always work 100% perfectly all the time. But, the more deviations from persistence and uniqueness, the harder it becomes for end-users, publishers, Crossref, and other services which make use of our metadata to reliably find research outputs and reliably relate them to one another. It weakens the value and utility of DOIs for everyone.

So, what does this mean in practice?

Be certain that every item you register with Crossref is something you can maintain in the long-term.
- Have an arrangement with an archive that can take responsibility for your content if your organisation stops hosting it or ceases to exist.
- Don’t register things that you know will only exist for a short time.
When you’re about to register new content, be absolutely sure that it hasn’t been registered already, either by your organisation or any other organisation.
- If you acquire a new journal from another publisher, have a process in place to check what content has already been registered and adopt the use of the DOIs registered by the prior publisher for that content. We can always provide a list of the existing DOIs for a journal.
- If you publish books, and have a co-publishing agreement with another publisher, distributor, or hosting platform, be aware that one of those other parties may have already registered DOIs for your books. Adopt the use of those DOIs rather than assigning and registering new ones. And, if you don’t want them to do that going forward, communicate that to your co-publishing partners.
When mistakes happen, inadvertently resulting in duplicate DOIs for a single item, identify them quickly. Alias the new duplicate DOI to the long-standing original DOI, and remove all instances of the new DOI from your website or platform.
Ensure that your publishing software, platform, or journal management system can accommodate DOIs with various prefixes for the same publication. You should be able to use (display, link, update metadata and URLs for) the DOIs registered for older content by any prior publishers as easily as you use the DOIs that you registered yourself for more recent content.

Things like persistence and uniqueness can sound like theoretical abstractions, but they actually play an important role in the day-to-day grind of your publishing operations. Their impact on linking, citing, discovery, and analysis of your content is concrete and important. Thus, it’s not surprising that we often hear from members and others in the research community who share this commitment to persistence, uniqueness, and overall rich, accurate metadata. You’ll see that play out in the Community Forum where members and users get involved to troubleshoot issues, compare notes, and share ideas with us and one another. We appreciate the commitment to the Research Nexus and the overall spirit to serve in this growing community. Like we said at the top, we’re all wired to contribute in this way, so building an open, welcoming space that moves us forward excites us.

Again, we invite you to join the discussion on this and many other Crossref-related topics over in our Community Forum.

The GEM program - year one

Susan Collins — Wed, 24 Jan 2024 00:00:00 +0000

In January 2023, we began our Global Equitable Membership (GEM) Program to provide greater membership equitability and accessibility to organisations located in the least economically advantaged countries in the world. Eligibility for the program is based on a member’s country; our list of countries is predominantly based on the International Development Association (IDA). Eligible members pay no membership or content registration fees.

The list undergoes periodic reviews, as countries may be added or removed over time as economic situations change. Sri Lanka was added to the GEM program in March 2023 as they were recategorised to the IDA classification by the World Bank.

When the program launched, we had 214 existing members eligible for the program who then were no longer charged for membership or content registration. Since the program began, we have welcomed an additional 131 new members into the program, including our first members from Cambodia and Togo.

Country	As of 1/1/2023 (start of GEM)	Additions in 2023 (end of first year of GEM)	Total
Afghanistan	6	4	10
Bangladesh	56	33	89
Benin	1	1	2
Bhutan	4	2	6
Burkina Faso	2	0	2
Burundi	1	0	1
Cambodia	0	2	2
Central African Republic	1	0	1
Congo, Democratic Republic	1	11	12
Ethiopia	4	6	10
Ghana	14	7	21
Guyana	1	1	2
Haiti	1	0	1
Kosovo	2	2	4
Kyrgyz Republic	22	3	25
Laos	1	0	1
Madagascar	1	1	2
Malawi	1	0	1
Maldives	1	0	1
Mali	2	0	2
Mauritania	1	0	1
Myanmar	1	0	1
Nepal	20	18	38
Nicaragua	1	0	1
Rwanda	4	1	5
Senegal	3	3	6
Somalia	2	2	4
Sri Lanka	13	5	18
Sudan	9	2	11
Tajikistan	5	1	6
Tanzania	9	7	16
Togo	0	1	1
Uganda	3	6	9
Yemen	16	12	28
Zambia	5	0	5

With help from our ambassadors based in GEM countries, we organised and co-hosted several webinars to introduce the program, along with an introduction to Crossref, and the benefits of including all kinds of research objects in the Research Nexus.

In April, our team, together with ambassador Binayak Raj Pandey, provided an overview of Crossref for members and organisations in Nepal.
Our team and ambassadors, Dr Md Jahangir Alam and Shaharima Parvin hosted two webinars in May for members and organisations in Bangladesh. The first webinar provided an introduction to Crossref, our services, and the GEM Program. The second webinar focused on the methods to register content and how to add and update metadata.
In September, ambassador Baraka Manjale Ngussa joined us for an introductory webinar aimed at organisations in Tanzania
In November, CARLIGH (the Consortium of Academic and Research Libraries in Ghana), Crossref, and EIFL co-hosted a webinar for librarians and journal editors in Ghana with a discussion on the GEM program and Crossref services.

In 2024, we will continue to collaborate with our ambassadors and other members of the community to offer more opportunities for organisations in GEM-eligible countries to learn about the program and the benefits of membership for content discovery.

The program was initially met with scepticism by some organisations in GEM-eligible countries, who wanted to be certain that it wasn’t a free trial, that there are no hidden fees, or that they would be required to pay later for other services. Others expressed concern that Crossref would introduce fees after a year or two. Though we were able to clarify these aspects of the program, we understand the concerns and are working to ensure we provide clarity and transparency about the program. Additionally, we will be conducting a complete review of our fees in 2024, and we will ensure that GEM-eligible members will have input.

Although the program offers relief from fees, many organisations require technical assistance and language support. The GEM program would benefit from an increase in local Sponsors to facilitate membership and provide support, particularly In countries with the highest growth, such as Bangladesh, Nepal, Yemen, Kyrgyz Republic, and Ghana. Though we have Sponsors working with members who are in GEM countries (e.g. PKP), we do not yet have any Sponsors who are based in a GEM country.

We will be working with relevant like-minded organisations, such as PKP, DOAJ, INASP, OASPA, EIFL, and others, to help identify suitable candidates for new Sponsors in underserved regions and engage them proactively. Additionally, we will consult with our ambassadors in GEM countries to help identify potential Sponsors. We are beginning the year by making the most of the momentum created in African countries (Uganda, Ghana, Tanzania) and looking to develop new networks in other parts of the world in Q2-Q4 of this year.

Increasing Crossref Data Reusability With Format Experiments

Martin Eve — Fri, 19 Jan 2024 00:00:00 +0000

Every year, Crossref releases a full public data file of all of our metadata. This is partly a commitment to POSI and partly just what we do. We want the community to re-use our metadata and to find interesting ends to which they can be put!

However, we have also recognized, for some time, that 170GB of compressed .tar.gz files, spread over 27,000 items, is not the easiest of formats with which to work. For instance, there’s no indexing capacity on these files, meaning that it is virtually impossible simply to pull out the record for a DOI. Decompressing the .tar.gz files takes a good three hours or more even on high-end hardware, without any additional processing.

To that end, the Crossref Labs team has been experimenting with different formats for trial release that might allow us to reach broader audiences, including those who have not previously worked with our metadata files. The two new formats, alongside the existing data file format, with which we have been experimenting, are JSON lines and SQLite.

JSON-L

The first format with which we’ve been experimenting is JSON-L (JSON lines). With one JSON entry per line, as opposed to one giant JSON file/block, JSON-L lends itself to better parallelisation in systems such as SPARC, because the data can easily be partitioned.

This data format also has the benefit of being appendable, one line at a time. Unlike conventional JSON, which requires the entire structure to be parsed in-memory before an append is possible, JSON-L can simply be written to and updated. It’s also possible to do multi-threaded write operations on the file, without each thread having to parse the entire JSON structure and then sync with other threads.

In our experiments, JSON-L came with substantial parallelisation benefits. Our routines to calculate citation counts can be completed in ~20-25 minutes. Calculating the number of resolutions per container title takes less than half an hour.

SQLite

SQLite is a library written in C with client bindings for Python, Java, C#, and many other languages that produces an on-disk, portable, single-file SQL database. You can produce the SQLite file using our openly available Rust program, rustsqlitepacker. We also have a Python script that can produce the final SQLite file, for those happier working in this language.

The resultant SQLite file is approximately 900GB in size, so it requires quite a lot of free disk space to create in the first place (alongside storage of the data file that is needed to build it). However, queries are snappy when looking up by DOI and other indexes can be constructed (the indexing part of the procedure takes about 1.5 hours per field).

The database structure, at present, is the bare minimum that will work. It contains a list of fields for searching/indexing – DOI, URL, member, prefix, type, created, and deposited – and a metadata field that contains the JSON response that would be returned by the API for this value.

This allows for the processing and extraction of individual JSON elements using SQLite’s built-in json_extract method. For example, to get just the title of an item, you can use:

SELECT json_extract(metadata, ‘$.title’) from works WHERE doi=“10.1080/10436928.2020.1709713”;

The balance that we have had to strike here is between flattening the JSON so that more fields are indexable and searchable, as against the trade-off in time and processing that this takes to create the database in the first place. The first draft version of our experiment was wildly ambitious in flattening all the records and using an Object Relation Mapper (ORM) to present Python models of the database. Like painting the Forth Bridge, this initial attempt would not finish in any sane length of time. Indeed, by the time we’d created this year’s data file, we’d need to begin work on the next.

What are the anticipated use cases here? When people need to do an offline metadata search on an embedded device, for instance, the portability and indexed lookup of the SQLite database can be very appealing. One of our team has even got the database running on a Raspberry Pi 5. You can also load the database into Datasette if you want to explore it visually.

Where do we go from here with this? It would be good to flatten a few more fields, but we would welcome feedback on use cases that we haven’t anticipated for SQLite and we’d love to hear whether this is already too unwieldy (at 900GB).

Data Files

As usual, we will be releasing the annual data file in the next few months. As an experiment this year, we will also be releasing the tools that can be used on that file to produce these alternative file formats. We will consider releasing the final data files for each of these formats, too.

What we would like to hear from the community is whether there are other data file formats that you might wish to use. Are there use cases that we haven’t anticipated? What would you ideally like in terms of file formats?

I4OA Hall of Fame - 2023 edition

Bianca Kramer — Tue, 09 Jan 2024 00:00:00 +0000

The Initiative for Open Abstracts (I4OA) was launched in September 2020 to advocate and promote the unrestricted availability of the abstracts of the world’s scholarly publications, particularly journal articles and book chapters, in trusted repositories where they are open and machine-accessible. I4OA calls on all scholarly publishers to open the abstracts of their published works and, where possible, to submit them to Crossref.

Since the launch of I4OA, we have been tracking the openness of abstracts for all Crossref members over time (for data and code, see this GitHub repository). For a subset of 40+, mostly larger, publishers, the proportion of current journal articles (published in the current year and preceding two years) that have abstracts deposited in Crossref is shown in a chart on the I4OA website, which is updated quarterly (Figure 1).

Figure 1: Proportion of current journal articles from selected publishers that have open abstracts in Crossref. Data collected on January 1, 2024 for publication years 2021-2023. Publishers already supporting I4OA are shown in orange.

These longitudinal data and accompanying visualisations allow us to identify and highlight good examples from 2023: publishers (both large and small) who newly started to make abstracts openly available last year and/or who managed to get the proportion of their articles with open abstracts close to 100%¹.

While we highlight some of these examples below in our ‘Hall of Fame’, it’s important to also acknowledge all the publishers that already were depositing abstracts to Crossref for most or all of their journal articles prior to 2023, thereby contributing to the availability of abstracts as part of a rich ecosystem of open metadata, for others to use and build upon.

Hall of Fame - Part 1: publishers included in I4OA visualisation

For the set of (mostly larger) publishers included in the visualisation on the I4OA website, Figure 2 shows the difference in the proportion of abstracts available in Crossref between January and December 2023 for journal articles published in 2021-2023.

A number of publishers stand out from this figure:

Wiley announced in October 2022 that it was joining I4OA and would be making abstracts available through Crossref. In August 2023, Wiley started to deposit abstracts to Crossref, and at the end of 2023, the proportion of current journal articles with open abstracts was 77%.

This makes Wiley the first of the four largest traditional commercial publishers to deposit abstracts for the majority of journal articles they publish. Springer Nature does this only for their current open access articles, while Elsevier and Taylor & Francis² do not yet provide abstracts to Crossref at all. SAGE, the fifth largest traditional commercial publisher, was a founding member of I4OA and has open abstracts for 85% of current journal articles.
Among society publishers, the American Geophysical Union (AGU) went from 7% to 99% open abstracts for current journal articles last year, which is a great achievement. The publishing arm of the American Institute of Physics (AIP Publishing) joins them in reaching close to 100% open abstracts, going from 41% to 95% in 2023.

¹Depending on the type of journal(s) of a given publisher, the maximal coverage of open abstracts will often be somewhat below 100%, as in Crossref, all journal content is assigned the type ‘journal article’. This includes e.g. editorials, letters to the editor and other publication types that are not always expected to have abstracts.

²The numbers for Wiley and Taylor & Francis do not include Hindawi and F1000 Research, respectively, as these have separate Crossref member IDs. As most full open access publishers, both Hindawi and F1000 Research have high proportions of open abstracts (81% and 98%, respectively).

CAIRN and Project Muse, two publishing platforms in the humanities and social sciences representing a number of individual publishers, both started including abstracts in the metadata they provide to Crossref in 2023. At the end of 2023, CAIRN had abstracts available for 41% of current journal articles, while Project Muse was just starting out at 5%. Both will hopefully increase further this coming year.
Returning to traditional commercial publishers, Wolters Kluwer Health, part of Wolters Kluwer, had seen a slow growth in the proportion of journal articles with open abstracts in the years prior to 2023, going from 2% to 10%. However, they showed a rapid increase in 2023, ending the year with 52% open abstracts.

While it is good to see publishers who have publicly committed their support for I4OA follow through with opening their abstracts (like Wiley and AIP), it is also very encouraging to see publishers who are not (yet) listed as I4OA supporters do so. This shows a growing awareness and action on this issue beyond advocacy through I4OA alone. And of course, we would love to list these publishers on our website as official supporters of I4OA!

Figure 2 also shows some cases where the proportion of open abstracts has gone down during the year. This can be due to temporary technical issues in depositing abstracts (as was the case for Hindawi). Theoretically, the proportion of open abstracts can also go down when publishers stop providing abstracts altogether during the year, but we have not observed that to be the case.

Figure 2: Development in the proportion of open abstracts in 2023 for current journal articles (publication years 2021-2023) from selected publishers. Publishers already supporting I4OA are shown in orange. Light orange/blue dots show the proportion of open abstracts in January 2023, and dark orange/blue dots in December 2023.

Hall of Fame - Part 2: other publishers

Among the many publishers not included in the limited selection shown in the I4OA visualisation, there are also some interesting highlights of publishers either starting out to deposit abstracts (and reaching a sizeable proportion) or having deposited open abstracts for almost all their current journal articles in 2023. The examples below drew our attention in 2023; they include a number of medium-sized publishers as well as a group of smaller publishers that deserve special attention.

The European Molecular Biology organisation (EMBO) went from 0% to 42% open abstracts in 2023. However, from January 2024 onwards, several EMBO journals were transferred to Springer Nature, so EMBO can no longer be tracked at publisher level in Crossref. It will still be possible to look at the development of open abstracts for individual EMBO journals.
The Institution of Engineering and Technology (IET), a medium-sized publisher, started to deposit abstracts in 2023, reaching 33% open abstracts for current journal articles at the end of the year.
The Acoustical Society of America (ASA) had open abstracts for almost all their current journal articles at the end of 2023, increasing from 50% to 97%.
Finally, in the second quarter of 2023, a group of over 200 smaller Turkish publishers saw large increases in their coverage of open abstracts, resulting in open abstracts for 95%-100% of their current journal articles. Consultation with Crossref pointed to the potential supporting role of DergiPark, one of the largest Crossref sponsors in Turkey. This is a great example of developments in open metadata at smaller publishers.

Looking forward

At the beginning of 2024, the proportion of current journal articles published by Crossref members with open abstracts has reached 49.7%, up from 20.7% when I4OA was launched in September 2020. This is thanks to a growing number of publishers who are depositing abstracts to Crossref, often depositing open abstracts for close to 100% of their journal articles.

This blog post has highlighted a number of publishers who contributed to this growth in the availability of open abstracts in 2023. We hope these examples will inspire other publishers to start doing the same and are looking forward to following the growth in the availability of open abstracts in 2024.

For publishers that started to deposit abstracts in recent years and are doing so for newly published articles only, our data on open abstracts for current journal articles will look better in 2024 than in 2023, as only articles published in the current year and two preceding years are taken into account.

However, the benefits of having abstracts openly available from a central location such as Crossref (both for direct usage and for integration in other open scholarly infrastructures) are not limited to recent publications only. Hopefully, publishers currently depositing abstracts to Crossref will continue to do so both for newly published articles as well as for the backfiles of journal articles already published.

Publishers who would like to be added to the list of I4OA supporters, or who would like more information on how to deposit abstracts for both new and existing journal articles, are very welcome to reach out to I4OA. More information about open abstracts in general, and I4OA in particular, can also be found in the FAQ on the I4OA website.

The author would like to thank Ludo Waltman (CWTS) and Ginny Hendricks (Crossref) for useful feedback on an earlier draft of this post.

This blog post is published under a CC BY 4.0 license. The header image is an adaptation of an image by Adam Jones available from Wikimedia Commons (https://commons.wikimedia.org/wiki/File:Interior_02_of_Rock_%26_Roll_Hall_of_Fame_and_Museum,_Cleveland_%28by_Adam_Jones%29.jpg) and is shared under a CC BY-SA license.