Relationships on Crossref

An Advisory Group for Preprints

Martyn Rittman — Wed, 09 Jun 2021 00:00:00 +0000

We are delighted to announce the formation of a new Advisory Group to support us in improving preprint metadata. Preprints have grown in popularity over the last few years, with increasing focus brought by the need to rapidly disseminate knowledge in the midst of a global pandemic. We have supported metadata deposits for preprints under the record type ‘posted content’ since 2016, and members currently register a total of around 17,000 new preprints metadata records each month.

As preprints develop and different practices arise, we are keen to re-examine the metadata schema: to do this properly we need community input. We want to ensure that the schema is fit for purpose and supports the diversity of ways in which preprints are posted, linked with other objects, and used. Metadata schema need regular review, and this is just one example of a number of areas we are looking to update. Several topics we see as a high priority for preprints are better notification for when a preprint has been withdrawn or removed, accurate recording of versioning, and better indication of preprint server names.

We have invited a number of organisations we know to be active in this area, and are looking forward to some very positive discussions. Participants span five continents and include members who post preprints, indexing services, and others with significant experience in the area of preprints. The first meeting took place earlier this week and brought up a diverse range of themes that will be tackled in future meetings.

Service Provider perspectives: A few minutes with our publisher hosting platforms

Jennifer Kemp — Mon, 24 May 2021 00:00:00 +0000

Service Providers work on behalf of our members by creating, registering, querying and/or displaying metadata. We rely on this group to support our schema as it evolves, to roll out new and updated services to members and to work closely with us on a variety of matters of mutual interest. Many of our Service Providers have been with us since the early days of Crossref. Others have joined as scholarly communications has grown and services have evolved. Though fewer than 20 in number, their impact far outweighs the size of the group.

They, like us, work with a great variety of members and have a broad view into publishing trends. In this post, we focus on views from some of the publishing hosting platform Service Providers, who’ve taken the time to share their thoughts on a few questions:

What is the biggest change you’ve experienced working with publisher metadata over the last few years and how have you adapted to it?

It has become more and more important that not only the DOIs are registered with the minimum of necessary metadata to get the DOIs registered, but that a most complete set of metadata is being sent along – including author identifiers, funding information, abstracts, licenses, to support other Crossref services and improve discoverability.

– de Gruyter

Our clients are increasingly aware of the key role metadata plays in the effective dissemination of research. With an increasing number of published articles and a clear domination of “search engines” and aggregation of content, metadata is the primary means of making sure that publications reach the right audience. Publishers’ value-add includes not just copy editing, formatting, and packaging, but also now creating journal articles for the digital age that are discoverable and well linked to the research corpus. Furthermore, we sense a clear move toward standardization, which goes beyond the structure to introduce standardized semantics: adopting common taxonomies for classifying content in different dimensions. Our response is to introduce effective, automated and consistent services that capture, and surface metadata throughout the value chain from authoring to publication and search.

– Atypon

Highwire’s publishers are always looking to use the latest DTD (Document Type Definition) for the content to stay up to current standards. Currently this would be JATS 1.2. They are choosing to remain current so that they can stay on top of all or new metadata that can enrich their deposits. We have handled this well and offer support for the latest version of DTD when they are released, but some publishers are not always familiar with what can/should be deposited with their content and this can be a learning process for them.

– MPS Limited

How do you explain to clients (and others!) why correct, quality metadata is important?

In the digital age, metadata is the key to enabling effective content consumption. Publications that cannot be effectively discovered are of little value. We can only increase the impact of research with “discoverable” and “machine readable” publications. So ensuring correct and quality metadata is the key to optimizing not only the processing (finding the right journal, editor, reviewers) but also to positioning each publication properly. As the volume of published scientific research increases, article metadata is the way forward — it brings “order” and enables our community to manage this volume.

– Atypon

Highwire always positions itself as “good content in” means “good content out”. This is true for our own content stores. Strong and valid metadata will result in valid and strong deposits. We explain this to all new clients on-boarded with Highwire and the use of current standards and for current client projects where content should/can be enriched through re-load.

– MPS Limited

Getting our journals to care about metadata is a two step process: First, make sure they understand how metadata will help their journal succeed (i.e. why it matters to them). Second, make it easy for them to produce metadata while minimizing the cost, time, or complexity of their workflow. The first step – making a case for why metadata matters – is often easier than you’d think. At the very least, most journal editors understand that metadata, e.g., JATS or DOI registration, is an important signifier of professionalism / prestige. In other words, they see that top journals publish metadata and want the same for their journal. From a more technical standpoint, metadata is important because that’s the format computers understand and, like it or not, the publishing ecosystem relies on computers to deliver all sorts of critical services – such as indexing, archiving, and discoverability. So, if you’re not publishing metadata, you’re likely missing the benefit of these services. The second step – making it easy to produce metadata – is more difficult. Journal editors generally understand metadata matters but often lack the technical skills or resources necessary to create metadata. This is where a platform, such as Scholastica, can be very helpful. Because platforms work with many journals, they can invest in tools to automate the creation of metadata, reducing costs for all their clients. For example, most platforms offer integrations to support automatic DOI registration. At Scholastica, we’re pushing this idea even further with automatic integration to more complicated services such as PubMed Central. By reducing cost and complexity, we can help new or small-budget journals have the same quality metadata normally reserved for large, established journals.

– Scholastica

We are sending other publishers’ metadata to academic libraries and distribution channels. Erroneous metadata will have a direct impact on how discoverable a title may be. The more uniform and correct the metadata, the better it will be indexed in other places.

– de Gruyter

What is the one industry development or trend you’re most excited about for the near future and why?

Open Science and the ability to deliver research with the tools for reproducing it is the most exciting and game changing trend. Technology has enabled the output of science to transition from two-dimensional printed text delivery into globally accessible and responsive web-based delivery. We are now taking the next steps to further leverage web technology to enhance research output with rich assets ranging from audio and video, datasets, executable code, high-resolution imagery, interactive applications and more. As more assets accompany research publications, viewing these assets as modular, individually citable, and reusable becomes a requirement. We are reviewing the whole research output flow from authoring to publishing, and most importantly to its dissemination through the myriad of discovery tools now available.

– Atypon

The move of everything to the cloud – this is changing and improving our infrastructure, our possibility to scale and to stay on top of technological development.

– de Gruyter

Thanks very much to the interviewees for their time and thoughts. We look forward to working with our entire Service Provider group on questions like these and many more. If you’d like more details, you can read about our Service Provider program or contact me for more information.

Doing more with relationships - via Event Data

Martyn Rittman — Fri, 14 May 2021 00:00:00 +0000

Crossref aims to link research together, making related items more findable, increasing transparency, and showing how ideas spread and develop. There are a number of moving parts in this effort: some related to capturing and storing linking information, others to making it available.

By including relationship metadata in Event Data, we are taking a big step to improve the visibility of a large number of links between metadata. We know this is long-promised and we’re pleased that making this valuable metadata available supports a number of important initiatives. We will also be backfilling, so all previously deposited relationships will eventually become available as events. The first step will be to add relationships between items that have DOIs, such as between a research article and a related review report or dataset.

What are relationships?

When members register metadata with us, they have the possibility to identify other works, items, and websites that they know are related. This might be supplementary material or previous versions of a work (especially for preprints and working papers). Equally, identifiers for a protein, gene, or organism used in the research can be included. These are recorded as ‘relationships’ and can be accessed in the same way as the rest of the metadata we hold about registered content.

Some examples

Relationships in the metadata show links to the published article from this bioRxiv preprint. In the Crossref Rest API:

"relation": {
 "is-preprint-of": [
 {
 "id-type": "doi",
 "id": "10.1038/s41467-020-17892-0",
 "asserted-by": "subject"
 }
 ],
 "cites": []
},

And now in Event Data:

"subj": {
 "pid": "https://doi-org.pluma.sjfc.edu/10.1101/2020.05.21.109546",
 "url": "https://doi-org.pluma.sjfc.edu/10.1101/2020.05.21.109546",
 "work_type_id": "posted-content"
},
"obj": {
 "pid": "https://doi-org.pluma.sjfc.edu/10.1038/s41467-020-17892-0",
 "url": "https://doi-org.pluma.sjfc.edu/10.1038/s41467-020-17892-0",
 "method": "doi-literal",
 "verification": "literal",
 "work-type-id": "journal-article"
},

Linking to a dataset in the Dryad Digital Repository by a recent eLife article. In the Crossref metadata:

"relation": {
 "is-supplemented-by": [
 {
 "id-type": "doi",
 "id": "10.5061/dryad.s58qh",
 "asserted-by": "subject"
 }
 ],
 "references": [
 {
 "id-type": "doi",
 "id": "10.5061/dryad.s58qh",
 "asserted-by": "subject"
 }
 ],
 "cites": []
},

And now in Event Data:

"subj": {
 "pid": "https://doi-org.pluma.sjfc.edu/10.7554/elife.19920",
 "url": "https://doi-org.pluma.sjfc.edu/10.7554/elife.19920",
 "work_type_id": "journal-article"
},
"obj": {
 "pid": "https://doi-org.pluma.sjfc.edu/10.5061/dryad.s58qh",
 "url": "https://doi-org.pluma.sjfc.edu/10.5061/dryad.s58qh",
 "method": "doi-literal",
 "verification": "literal",
 "work-type-id": "Dataset"
},

If you are interested in relationships for a single DOI, we still recommend checking the metadata of that record, however Event Data is a great option for looking across multiple records. For example, to check for relationships across a prefix, in a given time period, or for a specific type of relationship.

Data citation

Data citations can be included in data deposits in relationship metadata, usually using the ‘is-supplemented-by’ relationship. By creating an event from each relationship, the links between journal articles and books, and the data they rely on are more visible. This makes the data much easier to locate.

Many datasets have DOIs which are usually recorded with DataCite, meaning you are unlikely to find them via searches of Crossref metadata. Making data citation relationship metadata available in Event Data means it will be available in the same format as citations from datasets to articles (which DataCite sends to Event Data) and citations from articles to datasets from Crossref reference metadata (more to come on this later this year). It also means we will convert this information into Scholix format so that it can be harvested and combined with other sets of Scholix-compliant article/data links. Data citations will therefore be available for the community to identify, share, link and recognise research data. We’re working with initiatives like Make Data Count and STM’s research data program to support the growing uptake of good data citation practices. This is a big step forward in making data citation happen for the community; we have more to do, but Crossref is committed to completing this work as a strategic priority.

What’s next?

In this first stage we are adding relationships that link two objects with a DOI, and later this year we will bring in relationships using other identifiers such as accession numbers and URIs. That will make it more straightforward to ask questions of Event Data such as which organisms have relationships to which works with a DOI.

More info and staying in touch

Find out more about Event Data in our support documentation or check out tickets in the GitLab repo.
Keep informed and ask us anything via our community forum for Event Data discussion

A transparent record of life after publication

Madeleine Watson — Wed, 01 Nov 2017 00:00:00 +0000

Crossref Event Data and the importance of understanding what lies beneath the data.

Some things in life are better left a mystery. There is an argument for opaqueness when the act of full disclosure only limits your level of enjoyment: in my case, I need a complete lack of transparency to enjoy both chicken nuggets and David Lynch films. And that works for me. But metrics are not nuggets. Because in order to consume them, you really need to know how they’re made. Knowing the provenance of data, along with the context with which it was derived, provides everyone with the best chance of creating indicators which are fit for purpose. This is just one of the reasons why we built the Event Data infrastructure with transparency in mind.

The transparency problem

For the scholarly community, alternative metrics to citation count (‘altmetrics’) are becoming increasingly popular as they can offer rich and expedited insight into today’s diverse and dynamic research environment. Research artifacts undergo an extended life online as they’re linked, shared, saved and discussed in forums both within and beyond the traditional academic ecosystem. Data on these interactions are initially fragmented and buried within platforms like social media, blogs and news sites. Downstream, there are several value-add services that collate and present that data as a single, aggregated count. We see individual data points like ‘paper X was tweeted 22 times’, and ‘paper X is referenced 16 times on Wikipedia’ being combined, homogenised, weighted and expressed as a single figure, a calculated number serving as a proxy for value. But altmetrics alone don’t tell the whole story, and how they are calculated is not without idiosyncrasy or politics. As we each have our own unique voice and perspective, we need to ensure we understand the lenses through which these metrics are made in order to consume them effectively.

The 2015 Metric Tide report highlighted transparency as one of the five dimensions of responsible metrics. Having access to the context used to create a metric — the provenance of the original data as well as full transparency around its extraction, processing and aggregation — helps consumers to use the data meaningfully and allows for comparison across third-party vendors. But transparency is difficult to achieve when, as the report notes, the systems and infrastructure for collecting and curating altmetrics-style data are fragmented and have limited interoperability.

In the academic community, underlying centralised systems include ORCIDs to identify people and DOIs to identify items. But we’re missing a transparent, centralised infrastructure for describing and recording the relationships between objects and resources¹. These relationships, or links, occur outside publisher platforms and can provide valuable information about the interconnectivity and dissemination of research. Dedicated infrastructure for collecting these relationships would provide a data source for those interested in altmetrics to build upon.

Figure 1.1 Example of some relationships between articles and activity on the web

At Crossref, we call these relationships Events. An Event is the record of a claim made about the existence of a relationship between a registered content item (i.e. a DOI) and a specific activity on the web. Events include:

a DataCite dataset DOI contains a link to a Crossref article DOI
an article was referenced in Wikipedia
an article was mentioned on Twitter
an article has a Hypothes.is annotation
a blog contains a link to an article

In collaboration with DataCite, we are collecting Events for the DOIs registered with our organisations and are making that data available for others in the community to use. This is the Event Data infrastructure, with which we’re plugging the gap in open scholarly relationships infrastructure. https://doi-org.pluma.sjfc.edu/10.64000/kattr-5k219

The Event Data infrastructure

Crossref and DataCite have for many years provided a centralised location for bibliographic metadata and links, and a facility to help our members register Persistent Identifiers (DOIs) for their content. With nearly 100 million DOIs registered with Crossref, we know where research lives. Which got us thinking — could we use these links to find out more about the journey research undertakes after publication? Could we express these interactions as links without any aggregation or counts so it could be maximally reused? And if so, could we then provide this data in an open, centralised, structured format? The answer was yes, subject to some challenges:

Querying for individual DOIs wasn’t scalable for our full corpus of 100 million items, so we had to find something else.
Not everyone uses the DOI link (not a surprise!). Most people will link directly to the publisher’s site. This means we need to look for links using both the DOI and article landing page URLs.
When we find people referring to registered content using its landing page, we find the DOI for that content item so that the link can be referenced in our data set in a stable, link-rot-proof way.
We don’t always know the article landing page URL for every DOI upfront because like many relationships, the one between DOIs and URLs is complicated.

We began by asking the wrong questions and as a result we got the wrong type of data back: instead of returning a record of individual actions, we were returning aggregated counts. Aside from not meeting our use case, aggregation requires the curation of an ever-churning dataset in order to keep totals updated, which is not scalable for the number of DOIs in our corpus.

We soon learnt to ask the right questions. One pivotal change in approach was that instead of counts, we asked instead ‘what activity is happening on Twitter for this article?’. Our data went from ‘DOI X was mentioned 20 times on Twitter as of this date’ to ‘tweet X mentions DOI X on this date’. The data are now represented as a subject-verb-object triple:

Figure 1.2 Triple table.

Ultimately this has allowed us to represent actions like Wikipedia page edits as individual atomic actions (i.e an Event) rather than as a dataset that changes over time. Being open about the provenance of altmetrics with Event Data Crossref Event Data (the Crossref-specific service powered by the shared Event Data infrastructure) has evolved beyond a link store to become a continual stream of Events; each Event tells a new part of the story. Rather than constantly updating an Event whenever a new action takes place, we add a new one instead:

Figure 1.3 A Wikipedia Event.

Events answer a whole range of questions, such as:

What links to what?
How was the link made?
Which Agent collected the Event?
Which data source?
When was the link observed?
When do we think the link actually happened?
What algorithms were used to collect it?
Where’s the evidence?

We’re collecting data from a diverse range of platforms including Twitter, Wikipedia, blogs and news sites, Reddit, StackExchange, Wordpress.com and Hypothes.is. This means that when we observe a link in these platforms to what we think is a DOI, we create an Event and a corresponding Evidence Record to represent our observation. We also have Events to represent the links between research items registered with Crossref and DataCite - for example, when a Crossref DOI cites a DataCite DOI and vice versa.

The provenance of the data is fully transparent and is made available to everyone via an open API. We call this the evidence trail. The record of each link (‘Events’) as well as the corresponding evidence can then be used to feed into tools for impact measurement, discoverability, collaboration and network analysis.

Therefore, one application of Event Data is as an underlying, transparent data source for altmetrics calculations. For example, you might want to know the total number of times your paper has been mentioned on Twitter to date. If I told you that the number was 22, what does that actually mean? Do you know whether I counted both tweets and retweets? Do you consider both of these actions as equal? Is the sentiment of the tweet important to you? Was it a human or a bot that initiated a tweet? Are you interested in tweets containing links to multiple representations of your paper or do you only want to track mentions of your version of record (the final published copy)? With Event Data as your underlying data source, you can answer these questions.

Not only transparent in data, transparent by design

The National Information Standards Organisation (NISO), a US organisation responsible for technical standards for publishing, bibliographic and library applications, has developed a set of recommendations for transparency in their Alternative Assessment Metrics Project report, as well as a Code of Conduct for both altmetric practitioners and aggregators that aims to help improve the quality of altmetrics data. The working groups recognised that without transparency and conforming to a recognised standard, altmetric indicators “are difficult to assess, and thus may be seen as less reliable for purposes of measuring influence or evaluation”¹.

Crossref Event Data is one of the example altmetric data providers listed in the NISO recommendations. My colleague Joe Wass participated in the development and specification of the NISO “Altmetrics Recommended Practices on Data Metrics, Alternative Outputs, and Persistent Identifiers” at the same time as we were working with DataCite on Event Data, so they have mutually informed one another.

Figure 1.4 Martin Fenner (DataCite) and Joe Wass (Crossref) drawing plans for the Event Data infrastructure.

The outcome of our involvement in the NISO recommendations is that Crossref Event Data is a service that is transparent by design. We have opened up our entire extraction and processing workflow so that we can clearly demonstrate the context and environment that was used to generate an Event. This evidence is a core component of our transparency-first principle.

Building services on Event Data

There are some really exciting ways that people are already using Event Data, and we’re still only in beta. Our aim has always been to create an open, portable, transparent data set that can be used by our diverse community including researchers, application developers, publishers, funders and third-party service providers. We have already seen data from our service used in recent research studies, impact reports and even a front-end tool. Launched recently as a prototype, ImpactStory’s Paperbuzz.org uses Event Data as one of its data sources for tracking the online buzz around scholarly articles. Jason Priem, cofounder of ImpactStory, notes:

“Because Crossref Event Data is completely open data, we believe it’s a game-changer for altmetrics. Our latest project, Paperbuzz.org, is just the first of a whole constellation of upcoming tools that will add value on top of Crossref’s open data.”

We are working towards launching Crossref Event Data as a production service. In the meantime though, please do take a look at our comprehensive User Guide. Hopefully you’ll be inspired to go make something cool using the data! Events are being collected constantly; take a look below as they stream in from our data sources or visit our live stream demo site to watch in real time.

Figure 1.5 Screen capture of Crossref Event Data live stream demo.

As the service matures, we’ll continue to add new platforms to track and I also encourage anyone with article link data to get in touch to discuss how we can share it with the community via Event Data.

For researchers in particular, I’m really keen to hear your thoughts on our data model and about the things we could additionally provide you with from an infrastructure perspective that would best support your research needs.

And if you’re a publisher, take a look at our Event Data best practice guidelines — there’s some really important information in there about how you can help give us the best chance possible of collecting Events for your registered content.

And finally, if you’re a consumer of altmetrics data, I encourage you to ask questions. Ask your altmetrics vendors about how they gather their data and what context they apply to the aggregation of the metrics they supply. Ask yourself what behaviours you are interested in tracking and equally those you are not. Think about the endgame; about the type of impact you’re truly trying to measure and the story you want to tell. Because it’s these questions that will help you choose indicators that are the best fit for your own unique narrative.

This content is cross-posted on eLife Labs.

References

¹ Bilder, Geoffrey; Lin, Jennifer; Neylon, Cameron (2015): What exactly is infrastructure? Seeing the leopard’s spots. Retrieved: Oct 16, 2017; https://doi-org.pluma.sjfc.edu/10.6084/m9.figshare.1520432.v1

² NISO, Outputs of the NISO Alternative Assessment Metrics Project. Retrieved: 6th October 2017; https://www.niso.org/publications/rp-25-2016-altmetrics , p.2.