Linking on Crossref

On metadata enrichment

Dominika Tkaczyk — Thu, 19 Mar 2026 00:00:00 +0000

Metadata is communication; it can tell a story about research and paint a picture for others to respond to and learn from, across the world and throughout the forthcoming generations. Metadata can feel technical with words like ‘infrastructure’ and ‘schema’, and sometimes, like tech in general, it comes with hyperbole. But metadata really is part art (storytelling and pictures) and part science (structured models and standards) with both aspects being equally important, and requiring people as well as systems. That necessary combination of human and machine involvement also makes metadata challenging.

Crossref, as the earliest adopter of DOIs specialising in scholarly research, became synonymous with DOIs in this community. However, not everyone realises that DOIs can be registered with any one of nine different agencies, which are all separate organisations with entirely separate systems that do not at present integrate or connect. And what’s more – there isn’t a central or shared “DOI schema” – each agency develops the metadata for the purposes of their organisation or community. In Crossref’s case, with our vision to create the research nexus as a complete and robust network of relationships between objects, people, and institutions of scholarship – that community encompasses the whole of the research enterprise.

The immense 180 million records of research outputs in Crossref are maintained in a system that 24,000 member organisations have already invested in. Those records benefit from rich and format-appropriate metadata schema, developed in close collaboration with the community, which makes it possible for our members to offer contextual information about each object they register. We have a long history of working with our members on recording that context, creating tools, and providing support to adopt standard metadata, enriching the context for the benefit of the scholarly community, and society at large.

Of course, those metadata records are not perfect, both in terms of quality and completeness, and the frustration around gaps in metadata is particularly strong. We are working to improve the quality and completeness of the metadata from many angles: by working with the community to understand their needs and obstacles, by identifying and analysing potential sources for additional metadata, by maintaining and adopting the existing system to changing environment, and by planning a new flexible system that will allow third-party assertions and automated enrichment workflows.

In 2020, we published a paper for the inaugural issue of Quantitative Science Studies on Crossref: The Sustainable Source of Community-Owned Scholarly Metadata and blogged an introduction to it under Crossref Metadata for Bibliometrics. One of the things our analyses in 2019 showed was that over 80% of records between 2013-2016 had been updated. Reviewing the numbers recently, we continue to see this stewardship and maintenance of metadata, amounting to almost 70% of records from the past decade being updated at least once. On the dawn of reaching 2 billion citation links, we’d like to share our experience, plans, and views on this ubiquitous activity of updating and connecting metadata – by our members and by automations built into the system by us. Altogether, these constitute the enrichment process to improve the usability of the information for the community.

Metadata available through Crossref

Crossref collects, processes, stores, and shares metadata records for a wide range of research outputs. While each record describes an individual research output, it also mentions other entities and their attributes - and, most importantly, the relationships between them. Two works identified by DOIs, for example, may be linked by a citation relationship. A person identified by an ORCID may be connected to an institution identified by a ROR ID through an affiliation relationship. A preprint and its corresponding journal article, each with its own DOI, can be linked by an “is preprint of” relationship. A research output may be associated with a grant through a “financed by” relationship. Together, these entities and relationships form the foundational building blocks of the research nexus.

As of March 14, 2026, the Crossref database contains 180,034,490 metadata records describing research outputs. You can download all the records and examine them yourself in the latest public data file. The plot below illustrates how the number of works has changed over time, showing that the rate of growth is accelerating.

The metadata records describe research outputs of various types, including:

journal articles
books and book chapters
conference proceedings
peer reviews
reports
datasets
preprints
dissertations
grants
and more

The majority of works in the Crossref database (67%) are journal articles. However, the distribution of record types has changed considerably over time. Newer types, such as components, datasets, and posted content, are growing more quickly than more traditional ways of communicating research:

Research outputs in the Crossref database are represented by rich metadata records, which may include:

basic bibliographic metadata (title, publication dates, contributors, journal title, conference name, volume and issue numbers)
authors’ affiliations and ORCID identifiers
abstracts and links to full text
funding metadata, including funders and grant details
license metadata
bibliographic reference lists
clinical trial numbers
updates such as corrections or retractions
relationships between works and other entities, such as “is translation of”, “is review of”, “is preprint of”, or “is version of”
components associated with the work, such as figures, tables, and supplemental materials

All metadata is freely available through the Crossref REST API, and additional services, such as Crossref Search, are also provided.

A natural question is: where does all this metadata come from? This is important for two main reasons. First, it helps address the question of trust, as understanding the origin of the metadata allows users to better assess its reliability. Second, it points us to the right place when investigating or addressing issues or gaps in the data.

At first glance, the answer might seem straightforward: from Crossref members. Crossref members, such as publishers, research institutions, universities, funders, museums, libraries, data and subject repositories, and conference providers, register metadata for the outputs they publish. Crossref stores this metadata and makes it available to the community.

In reality, however, the story is more complicated.

Metadata enrichment layers

The initial metadata deposit is only the beginning of what can become a long and rather fascinating journey. What users can see in our REST API is often the result of a series of updates and additions that occur over time, sometimes coming from multiple sources and happening in different ways. We can think of these ways as enrichment layers.

Each enrichment layer offers opportunities to improve the metadata while also introducing its own considerations and challenges. Rather than forming a sequence of clearly separated stages, these layers intertwine, overlap, and affect one another, collectively shaping how a research output is represented within the research nexus.

Enrichment layers are essential for completeness of the research nexus. If we relied solely on the original, one-off deposits from members, the metadata would be full of gaps, limiting the usefulness of any analysis or assessment based on it. While the scholarly metadata will never be perfectly complete, applying these enrichment layers is how we gradually and collectively build a fuller, more accurate picture of the research nexus.

One important caveat is that more metadata doesn’t magically equal better metadata. In fact, there’s often a delicate tradeoff between completeness and quality: the harder one pushes to fill every gap, the greater the chance of introducing errors. At Crossref, we believe quality comes first. We recognise that no dataset will ever be perfect, but we’re equally unwilling to apply enrichment processes without quality control. Any enrichment we introduce must meet a high bar for accuracy — no exceptions, no shortcuts.

The order of the enrichment layers discussed here loosely reflects how established they are within the scholarly ecosystem. There also might be a correlation, or at least a perceived one, between this ordering and the reliability of the underlying processes. That said, one must tread carefully when making such interpretations: perceived reliability is not the same as actual reliability.

Layer 1: Member updates

Crossref members not only deposit metadata, but also update it over time. This is an essential part of the system for several reasons. There may be errors in the originally deposited metadata that need to be corrected. Also, the initial record may contain gaps that can be filled later as more information becomes available. In addition, many changes naturally occur: landing page URLs may change, works may be archived in new locations, or identifiers for affiliated organisations may become available. Those situations also ideally result in an update.

This update process is well established. Over 24,000 Crossref members form a large global community that operates under shared membership terms. As part of these terms, members are responsible for maintaining and updating their metadata records. In this governance framework it is clearly defined who owns and stewards the metadata associated with each record, and who is responsible for the quality level and issues.

Member updates are very common. As an example, over 80% of works deposited between 2013 and 2020 were updated at least once. This demonstrates the community’s commitment to improving completeness and quality of the scholarly record. The plot below shows the percentage of works created in a given month that were updated at least once.

However, this layer also comes with challenges. It relies on members actively meeting their obligations to maintain and improve their metadata. As a result, gaps and inconsistencies can remain, and overall metadata quality is never perfect.

Our plans for the future in this area largely build on what is already happening. This includes developing and maintaining effective user interfaces for updating metadata, evolving the input metadata schema to keep pace with changes in the scholarly landscape, offering regular workshops on metadata improvements, and collaboratively establishing best practices while educating members on how to apply them.

Layer 2: Community feedback loop

Crossref metadata is widely used and examined by a large community of consumers. As a result, issues with metadata are sometimes identified by community members and reported back to us. When this happens, Crossref does not directly correct the metadata records. Instead, we contact the relevant member responsible for the record and able to deposit an update.

In this layer, the stewardship of metadata remains with the member, while responsibility for metadata quality broadens to include other actors in the community. This creates significant potential for scaling by involving a large community in identifying and reporting metadata issues.

At present, however, this process is not automated. Crossref staff effectively act as intermediaries between those reporting issues and the responsible member. As a result, the process has limited scalability. It also depends on the willingness of members to act on the reports they receive, as they are not obligated to respond to such reports.

In the future, we may explore automating portions of this workflow to handle community feedback more efficiently and lighten the load on everyone involved.

Layer 3: Metadata matching

Metadata matching is the task of finding an identifier for an item based on a structured or unstructured description of it. Matching strategies run as fully automated processes that analyse information deposited and updated by members and add identifiers, filling gaps in the metadata.

There are many instances of metadata matching problems, for example:

bibliographic reference matching: finding a DOI for a cited paper based on a bibliographic reference,
funder matching: finding the ROR ID for a funder based on its name,
affiliation matching: finding the ROR ID for an organisation based on an affiliation string,
preprint matching: finding the DOI for a preprint that precedes a given journal article,
grant matching: finding the grant DOI based on an award number and a funder name.

This layer is unique, as it focuses on a crucial type of gap in the scholarly record: the missing relationships between entities. Indeed, adding an identifier for an entity mentioned within a metadata record of a research output is typically an equivalent of asserting a relationship between that output and the matched entity. For example, bibliographic reference matching inserts citation relationships, and funder name matching - funding relationships between a research output and a funding organisation. These relationships form the foundation of the research nexus.

Currently, at Crossref, we perform two types of matching. We match bibliographic references to the DOIs of cited outputs, and funder names to Funder IDs. Both processes rely on fuzzy comparisons and other heuristic approaches to identify likely matches.

In the case of bibliographic reference matching, as it turns out, more than half of the cited DOIs (1 billion) available in the Crossref database originate from automated metadata matching:

In the case of funder name matching, the distribution is very different, but the matching strategy was still able to fill in some of the gap:

Metadata matching is a particularly valuable form of enrichment for several reasons. Matching strategies can often achieve high levels of accuracy while working in a fully automated way. This makes them highly scalable and drastically reduces the need for human oversight. Their focus on relationships also strengthens the foundations of the research nexus.

At the same time, this enrichment layer presents a number of challenges.

Its most fundamental limitation to remember is that metadata matching can only fill gaps when there is at least some useful information to work with. For example, it can identify a cited document only using structured or unstructured citation data, and the funding organisation can only be identified if some funding information is available. But if citation information, or funding information, is completely absent, as is the case for 101M (56%) records and 166M (92%) records respectively, then matching simply isn’t possible.

Matching strategies can also be complex and time-consuming to research, develop, and maintain. They require additional considerations of issues such as openness, explainability, complexity, flexibility, and cost.

Perhaps most importantly, in the case of matching, it becomes less clear who is responsible for the information introduced through the matching process. This is particularly important because matching results are never perfect, meaning there is always a risk of introducing errors. The risk is further amplified by the fact that matching strategies typically operate in a fully automated, unsupervised manner. As a result, careful evaluation of matching performance, as well as maintaining accurate provenance records, becomes increasingly important.

At Crossref, we have ambitious plans in this area. We intend to rebuild Crossref’s metadata matching workflows using modern software development and data science practices. The goal is to create a dedicated, consolidated matching service that will eventually replace all existing production matching processes, with results made available through the REST API. This project will cover six matching tasks: bibliographic reference matching, funder name matching, preprint matching, affiliation matching, grant matching, and title matching. You can learn more about metadata matching at Crossref at a dedicated project page.

Layer 4: Third-party datasets

There are many databases containing scholarly data, and one way to fill gaps in Crossref member-provided metadata is to incorporate additional metadata from those external sources.

We already have one example of this. Crossref ingests data from the Retraction Watch database to supplement information about retractions and other updates to records:

This layer has several advantages. It draws on subject-specific and metadata-specific expertise, avoids reinventing work that has already been done elsewhere, and reflects a collaborative community-driven approach to improving the scholarly record.

However, there are also important challenges to consider. Integrating external data often involves multiple data licenses or acquisition arrangements, and there may be less control over data quality compared to metadata that comes directly from members. There is also a risk that relying too heavily on external sources could shift responsibility away from the member stewards of the metadata. Finally, it can be difficult to determine which external datasets provide sufficient value and longevity to justify long-term integration.

Looking ahead, we plan to explore further opportunities to incorporate third-party datasets, carefully considering the value they bring, as well as issues of licensing, sustainability, and data quality.

Layer 5: Unstructured content scraping

A significant amount of scholarly information still exists in fully unstructured forms, such as full-text PDF documents and web pages. In principle, extracting information from these sources could help fill many gaps in existing metadata.

In a lighter-touch approach, analysing full-text documents can also help verify existing metadata elements. If such a check fails, the unverified element may be removed from the record — which, perhaps counterintuitively, can also count as enrichment, since improving accuracy is every bit as important as adding new information.

There are also important challenges to consider. Extracting metadata directly from unstructured sources could substantially shift responsibility away from the original data stewards or owners, weakening the current stewardship model. The results of automated extraction may also be inconsistent or of relatively low quality. In addition, there are potential legal and rights-related concerns, particularly when processing full-text materials. Finally, developing reliable extraction methods would require substantial research and engineering effort.

For all these reasons, the practical usefulness of this approach remains uncertain, and Crossref currently has no plans to run such processes in production. We will, however, keep a close eye on emerging extraction technologies and may consider adopting them in some form if future evaluations show clear value.

Summary

Metadata is far more than a technical afterthought of the publishing process. It is the connective tissue of the scholarly ecosystem, linking research objects, people, and institutions into a coherent, navigable network. At Crossref, this takes the form of a vast and continually evolving corpus of more than 180 million metadata records, all contributing to the emerging research nexus, being built through collective community effort to help the global research community discover, interpret, and reuse knowledge effectively.

The initial metadata record deposited by members is only the beginning. Its quality and completeness can improve over time through multiple enrichment layers: member-driven updates, community feedback, automated metadata matching, and the incorporation of third-party datasets. These processes help fill gaps and strengthen the reliability of the scholarly record, all while upholding a firm commitment to accuracy and stewardship.

Taken together, these layers reflect a long-term, collaborative effort across technology developments, community participation, and responsible automation, to ensure that scholarly metadata becomes richer, more interconnected, and more useful for everyone who relies on it.

Enhancing repository integration with Crossref

Johanssen Obanda — Mon, 13 Oct 2025 00:00:00 +0000

Repositories are home to a wide range of scholarly content; they often archive theses, dissertations, preprints, datasets, and other valuable outputs. These records are an important part of the research ecosystem and should be connected to the broader scholarly record. But to truly serve their purpose, repository records need to be connected to each other, to the broader research ecosystem, and to the people behind the research. Metadata is what makes that possible. Enhancing metadata is a way to tell a fuller, more accurate story of research. It helps surface relationships between works, people, funders, and institutions, and allows us as a community to build and use a more connected, more useful network of knowledge - what Crossref calls the ‘Research Nexus’.

The challenge many repositories face is that metadata can be incomplete, inconsistent, or disconnected. Think of references without DOIs, authors without ORCID iDs, or research outputs that aren’t linked to funding. To address this, Crossref provides a range of services that repositories can use to improve the quality and interoperability of their metadata. Our REST API, which is openly and publicly accessible, allows repositories to retrieve structured metadata, such as DOIs, references, abstracts, contributors, ORCID iDs, and funder information, that can be used to enrich and update their local records. For repository members, with the Cited-by service and reference linking, repositories can also show how works are being cited and interconnect related content. The Grant Linking System (GLS) enables the clear indication of which research outputs are linked to specific grants, and funding bodies themselves are connected using Open Funder Registry and ROR, adding another layer of context. With Crossmark, repositories can flag updates, corrections, or retractions to ensure transparency and trust in the scholarly content they host.

Enriching repository metadata using Crossref is a practical and empowering step toward making your records more discoverable, complete, and connected. The process is simple, and you don’t need to be a developer to get started. Repositories can query the Crossref REST API using a DOI or basic metadata like a title or author name, and receive structured, reliable information. This can include full author lists, ORCID iDs, reference lists, funding data, and licensing terms. You can then match and merge this data into your repository records. Adding Crossref DOIs to your metadata enables persistent linking, helping users trace research outputs back to their stewards. It also helps create rich relationships between articles, datasets, software, grants, and other research objects. All of this supports the FAIR principles and contributes to a more connected and reusable scholarly record. And because Crossref’s infrastructure is open, any repository can access and use this metadata to improve the quality, visibility, and long-term value of their collections.

Steps to enrich repository metadata with Crossref:

Query the REST API using DOIs or basic metadata (visit our API learning hub to learn how to use the Crossref API)
Retrieve structured metadata like authors, ORCID iDs, funders, affiliations, ROR IDs, licenses, grants, and references
Map and merge with your local records
Display persistent links to all kinds of research objects using Crossref DOIs
Support FAIR by including open, structured, and complete metadata

Across the repository community, several institutions are already integrating Crossref metadata in meaningful ways to enrich their records and improve discoverability. DSpace users can enrich their deposits by using the platform’s “Live Import” feature, which allows them to pull in Crossref metadata, such as titles, authors, and DOIs, directly into items during the submission process. A deeper integration between DSpace and Crossref is currently in development. HAL in France uses the Crossref API to complete and standardise references, making its content more consistent and connected (hal.archives-ouvertes.fr). SciELO, a key open access platform in Latin America, leverages Crossref DOI links and citation metadata to strengthen the visibility of its journals (scielo.org). In Canada, the University of Saskatchewan’s eCommons repository queries the Crossref API to enhance metadata accuracy and link records to the broader scholarly graph (ecommons.usask.ca). The Apollo repository at the University of Cambridge uses Crossref to connect theses and articles to their published versions, creating a clearer picture of research outcomes (repository.cam.ac.uk). Zenodo, hosted by CERN, draws on Crossref metadata to link deposited datasets and software with related publications, supporting transparency and reuse (zenodo.org).

These examples show how even modest integrations with Crossref can lead to substantial gains in metadata quality, interoperability, and global discoverability. Altogether, these activities and organisations are enhancing the Research Nexus, enriching a scholarly graph for the benefit of all.

Want to learn more? You can explore the presentation slides (PDF) from Open Repositories 2025, which cover the Crossref API and its capabilities, how repositories can use it to query and enrich metadata, the benefits for repository managers, researchers, and funders, as well as recent updates to our metadata schema.

Metadata matching: beyond correctness

Dominika Tkaczyk — Wed, 08 Jan 2025 00:00:00 +0000

In our previous entry, we explained that thorough evaluation is key to understanding a matching strategy’s performance. While evaluation is what allows us to assess the correctness of matching, choosing the best matching strategy is, unfortunately, not as simple as selecting the one that yields the best matches. Instead, these decisions usually depend on weighing multiple factors based on your particular circumstances. This is true not only for metadata matching, but for many technical choices that require navigating trade-offs. In this blog post, the last one in the metadata matching series, we outline a subjective set of criteria we would recommend you consider when making decisions about matching.

Openness

Matching tools come in many different shapes and sizes: web applications, APIs, command-line tools, sometimes even enchanted crystal balls showing matched identifiers emerging from a mysterious mist! No matter what form they take, an important consideration is whether the source code and all the related resources for the matching are openly available.

Matching strategies that are either closed-source, or rely on closed-source services for their matching logic, make it difficult to fully understand and explain matching processes. This lack of transparency also makes it impossible to adjust or improve the matching logic, since we cannot understand or improve code we cannot see.

Users are similarly impeded from identifying flaws or suggesting improvements to processes they are unable to examine. By blocking this community participation, we also lose the proven cycle of real-world testing, refinement, and validation that has strengthened myriad of open source projects. The cumulative impact of both minor and major community-driven refinements over time is incredibly valuable and should not be underestimated.

Using open source matching will also help build trust in the matching workflows and results. This is one reason why open source is one of the tenets of the Principles of Open Scholarly Infrastructure, adopted by Crossref, DataCite, ROR, and other organisations who build and maintain open scholarly infrastructure.

When evaluating matching strategies, we strongly recommend prioritizing those that are fully open source. This not only ensures their transparency and trustworthiness, but also allows for the kind of continuous improvement that results from this visibility and community engagement.

Explainability

In terms of our ability to understand and improve a matching strategy, using an open source model is only the first step. What typically matters most in the context of building and maintaining matching services is that we are able to understand their underlying code and have a clear model of how matches are derived from their corresponding inputs. Even if the matching code itself and all of the resources used in the matching are open, if they are poorly documented, lack reproducibility or tests, or are otherwise opaque, there is no guarantee that it will be possible to understand or improve the strategy. Striving for a high level of interpretability in our matching plays a determinative role in how well we can understand and modify our strategies in the future.

Being able to explain the behaviour of the matching will also help you to respond to and incorporate user feedback. When users encounter errors, you will be able to do things like advise them on how to modify or clean their inputs so that the results are better. Conversely, examining the behaviour of the strategy relative to user inputs and feedback can provide you with ideas for improving the matching.

Typically, heuristic-based strategies, such as those that use forms of search or string similarity measures, like edit distance, are easier to explain than, say, machine learning models. If a strategy uses machine learning, at least some internal decisions might be made by passing data through a complex network of algebraic equations. Those can be mysterious, non-deterministic, and are famous for being hard to interpret. This doesn’t mean they should be avoided entirely - we have built and use many machine-learning based tools ourselves! Instead, it is a good idea to weigh how their inherent lack of explainability could affect your ability to continue work on the strategy and respond to user needs, relative to all the available options.

Complexity

Complexity is another aspect that can greatly affect how easy it is to maintain the strategy. Complexity is related to how many different components the strategy has and how difficult they are to use and maintain. When a strategy has multiple interconnected parts, each component becomes a potential failure point that requires discrete assessment and maintenance.

Consider, for example, two different approaches to a matching strategy: one that uses a single machine learning model versus another that uses an ensemble of models. A single model requires maintaining one set of training data, a single training pipeline, and one deployment process. If the model’s performance unexpectedly deteriorates, whether because of an issue with the training data, a configuration error, or the need for additional input sanitization, the source of the problem is easier to isolate and fix.

The ensemble, by contrast, combines multiple, specialized models, each requiring its own training data, tests, updates, and deployments. If one model in the ensemble is found to reduce the performance of the strategy, the interdependence between models can cause this degradation to cascade through the entire system and undermine its overall reliability. Correcting for these errors becomes more challenging. If fixing one model’s performance requires retraining or adjusting its outputs, this could require recalibrating the entire ensemble to maintain the balance between models, identify regressions, and prevent new errors from emerging.

In general, preferring simpler strategies not only reduces operational overhead, but also makes it easier to diagnose issues, test changes, and iterate on user feedback. When problems arise, having fewer moving parts means less places to look for the root cause and fewer components that could be affected by any fixes.

Flexibility

The metadata to which we match grows and changes over time. New records are created, existing ones are updated, with schemas changing and evolving alongside. The resources that underlie our matching are also not static. The libraries we depend on may deprecate features between versions or the taxonomies we used to categorize results might undergo significant revisions. We thus rarely have the luxury of deploying a matching strategy once and using it forever without any changes. A good strategy has to be flexible enough to adapt to such changes, with this adaptation also being both technically feasible and practical to implement.

Much of this flexibility is also determined by a matching strategy’s ability to incorporate new data. Strategies that use continuously updated databases or indices can immediately match against new metadata as it appears in the system. By contrast, some machine learning-based approaches require training on target matches and can thus be limited in flexibility and face more constraints. While some models can be incrementally updated to recognize new matches, others require retraining from scratch to incorporate these changes - a process that can be both time-consuming and resource-intensive.

Paying close attention to a strategy’s flexibility and favoring this aspect, when possible, can significantly impact its long-term viability. When comparing different matching strategies, flexibility should thus be a primary concern in your decision-making process.

Resources

Matching strategies can vary significantly in their resource requirements, including things like CPU and GPU utilization, memory consumption, storage capacity, and network bandwidth. These requirements are directly related to infrastructure costs and energy consumption, so when evaluating a matching strategy, it is necessary to assess its resource demands across all phases of the matching lifecycle. This includes things like initial model training, re-training, index construction, updates and management for all aspects of the strategy, as well as the real-world processing of matching requests. It is a good idea to measure and monitor resource usage carefully in considering which strategies to use, as the best performing strategy may also be too resource intensive to run as a service or might grow to this state over time with additional utilization.

Speed

Matching strategies can operate at a wide range of speeds, from milliseconds to minutes per match. Since the overall response time of a strategy can affect both system scalability and user experience, we should always assess the strategy’s performance for different usage scenarios and scales of data. While some strategies might perform adequately with small datasets, they can also exhibit exponential slowdowns as data volume and complexity increases or as concurrent requests grow in number. We should therefore consider carefully how requirements for matching speed might evolve with increased usage, data complexity, and total anticipated growth. The fastest matching strategy might not always be the best choice if it comes at the cost of reduced accuracy or requires large amounts of resources, but unacceptable latency can make an otherwise excellent strategy unusable in practice for many use cases.

Putting it all together

The typical life cycle of developing a metadata matching strategy is as follows:

Scoping: we define the matching task, along with its inputs and outputs.
Research: we research what existing strategies are available for our task and/or we develop our own.
Evaluation: we evaluate all available strategies, internally or externally-developed, exploring all of the aspects described above.
Decision: we choose which strategy (if any) we want to use in our production system.
Production setup: we prepare the production models, indexes, and other resources needed for the matching.
Maintenance: we monitor and adapt the strategy relative to changing data, user feedback, and new resource requirements.

In practice, these phases do not happen all at once, nor in this strict order. Often we need to proceed through multiple iterations of them to arrive at the best strategy. For example, if initial evaluation of a strategy yields poor results, we might return to the research phase to investigate other strategies or refine our understanding of the task. Often, during the maintenance phase, we receive feedback from users that indicates potential areas of improvement and then pursue them with a new round of research and evaluation.

As we cycle through these phases, ideally all the aspects described in this entry, along with the results of the evaluation, would be taken into account. Of course, this means that these decisions have to be based on multiple criteria and by making trade-offs between their performance and all other considerations. In making these complex and difficult choices, it is useful to consider two primary questions:

Are any of the considered matching strategies good enough for our use case?
Out of all the considered strategies that are sufficient for our use case, which would be the best?

The first question requires us to create clear and quantifiable criteria that allow for eliminating some of the potential strategies. As we have indicated, these could include things like the strategy being open source, minimum performance baselines using measures like precision or recall, and operational thresholds, like the strategy being able to return results quickly, relative to user expectations or the volume of data to be processed. It should be fairly easy to test these requirements and eliminate any strategies that fall short of them. If the strategies are difficult to assess, that is likely a mark against them.

If no strategies meet these criteria, we have two options: either to abandon matching entirely or to reassess and relax our criteria to align with the available options. While the former is always an option, adopting a more pragmatic lens, framing in terms of potential value (or harm) to the users, might be beneficial. Sometimes we approach matching tasks with too high expectations and a dose of realism helps us to re-center our perspectives. After more consideration, you might decide that your criteria were too stringent or realize that you need to better define and decompose the tasks to fit the available options.

When multiple strategies appear viable, the selection process becomes more nuanced. When evaluating strategies across these various dimensions, we should try to avoid placing undue weight on minor performance differences. Evaluation metrics are useful estimates of performance, but do not always translate to real-world applications and changing data. In cases where a more complex strategy offers only marginal improvements over a simpler alternative, the maintenance and operational benefits of the simpler solution often outweigh small performance gains.

This concludes our series on metadata matching, where we described the conceptual, product, and technical aspects of matching and its applications. We hope this overview was instructive and helps you to make better decisions about the use of matching in your own tools and services!

How good is your matching?

Dominika Tkaczyk — Wed, 06 Nov 2024 00:00:00 +0000

In our previous blog post in this series, we explained why no metadata matching strategy can return perfect results. Thankfully, however, this does not mean that it’s impossible to know anything about the quality of matching. Indeed, we can (and should!) measure how close (or far) we are from achieving perfection with our matching. Read on to learn how this can be done!

How about we start with a quiz? Imagine a database of scholarly metadata that needs to be enriched with identifiers, such as ORCIDs or ROR IDs. Hopefully, by this point in our series this is recognizable as a classic matching problem. In searching for a solution, you identify an externally-developed matching tool that makes one of the below claims. Which of the following would demonstrate satisfactory performance?

It is a cutting-edge, state-of-the-art, intelligent-as-they-come, bullet-proof technology! All the big players are using it. You won’t find anything better!
The tool was tested on the metadata of 10 articles we authored, and many identifiers were matched.
The quality of our matching is 98%.

Okay, okay, trick question. The correct answer here is to opt for secret answer #4: “I wouldn’t be satisfied by any of these claims!” Let’s dig in a bit more to why this is the correct response.

The importance of the evaluation

Before we decide to integrate a matching strategy, it is important to understand as much as possible about how it will perform. Whether it is used in a semi or fully automated fashion, metadata matching will result in the creation of new relationships between things like works, authors, funding sources, and institutions. Those relationships will then, in turn, be used by the consumers of this metadata to guide their understanding and perhaps even to make important decisions about those same entities. As organisations providing scholarly infrastructure, we must therefore take it as our paramount responsibility to understand any caveats or shortcomings of the scholarly metadata we make available, including that resulting from matching.

Proper evaluation is what allows us to do this, as it is impossible to know how well a given matching strategy will perform in its absence. This is true no matter how simple or complex a matching strategy may seem. Complex methods can be tailored to data with specific characteristics and might fail when faced with something different from this. Simple methods might be only appropriate for clean metadata or a narrow set of use cases.

Beyond complexity, matching strategies themselves vary widely in character, inheriting biases from their design, training data, or how a problem has been formulated. Some prioritise avoiding false negatives, while others focus on minimising false positives. Even a generally high-performing strategy might not be perfectly aligned with your specific needs or data. In some cases, the task also itself might be too challenging, or the available metadata too noisy, for any matching strategy to perform adequately.

Evaluation is, again, how we understand these nuances and make informed decisions about whether to implement matching or avoid it altogether. By now, it should also be clear that the notion “we don’t need to evaluate” is far from ideal! Given its importance, let’s explore how evaluation is actually done.

Evaluation process

In general, a proper evaluation procedure should follow the following steps:

Preparation of an evaluation dataset containing many examples of matching inputs and the corresponding expected outputs.
Applying the strategy to all inputs from the dataset and recording the responses.
Comparing the expected outputs with the outputs from the strategy.
Converting the results of the above comparison into evaluation metrics.

From this accounting, we can see that there are two primary components for the evaluation process: an evaluation dataset and metrics.

Evaluation dataset

It’s useful to conceive an evaluation dataset as the specification for an ideal matching strategy, describing what would be returned from our forever-elusive perfect matching. When creating such a dataset, what this means in practice is that it should contain a number of real-world, example inputs, along with the corresponding ideal or expected outputs, and that all data should be in the same format as the strategy is expected to process. The outputs should themselves also confirm the strategy’s overall requirements, for example, by being consistent with its cardinality, meaning whether zero, one, or multiple matches should be returned and under what circumstances. In terms of size, it’s generally useful to calculate the ideal number of evaluation examples using a sample size calculator or using standardised measures, but as a quick rule of thumb: less than 100 examples is probably insufficient, more than 1,000 or 2,000 is generally acceptable.

It is also important that the evaluation dataset be representative of the data to be matched in order to ensure reliable results. Using unrepresentative data, even if convenient, can lead to biassed or misleading evaluations. For example, if matching affiliations from various journals, building an evaluation dataset solely from one journal that already assigns ROR IDs to authors’ affiliations might be tempting. The data, having been already annotated, allow us to avoid the tedious work of labelling, and we might even know that it is produced by a high-quality source. This is still, unfortunately, a flawed approach. In practice, such datasets are unlikely to represent the entire range of affiliations to be matched, potentially leading to a significant discrepancy between the evaluated quality and the actual performance of the matching strategy, when applied to the full dataset. To assess a matching strategy’s effectiveness, we have to resist shortcuts and instead do our best to create truly representative evaluation datasets to be confident that we’ve accurately measured their performance.

Evaluation metrics

Evaluation metrics are what allow us to summarise the results of the evaluation into a single number. Metrics give us a quick way to get an estimation of how close the strategy was to achieving perfect results. They are also useful if we want to compare different strategies with each other or decide whether the strategy is sufficient for our use case, removing the need to compare countless evaluation examples from different strategies against one another.

The simplest metric is accuracy, which can be calculated as the fraction of the dataset examples that were matched correctly. While a commonsense benchmark, accuracy can be misleading, and we generally do not recommend using it. To understand why, let’s consider the following small dataset and the responses from two strategies:

Input	Expected output	Strategy 1	Strategy 2
string 1	ID 1	ID 1	ID 1
string 2	ID 2	ID 3	Empty output
string 3	Empty output	Empty output	Empty output

Both strategies achieved the same accuracy, 0.67, making one mistake each on the second affiliation string. However, a closer examination reveals that these error types are distinct. The first strategy matched to an incorrect identifier, while the second refused to return any value illustrating the limitation of accuracy as a measure: it generally fails to capture important nuances in strategy behaviour. In our example, the first strategy appears more permissive, returning matches even in unclear circumstances, while the second is more conservative, withholding them when uncertain. Although using such a small dataset would preclude drawing any definitive conclusions, it highlights how relying on accuracy alone can obscure differences in performance.

For evaluating matching strategies, we instead recommend using two metrics: precision and recall. To recap from our previous blog post:

Precision is calculated as the number of correctly matched relationships resulting from a strategy, divided by the total number of matched relationships. It can also be interpreted as the probability that a match is correct. Low precision indicates a high rate of false positives, which are incorrect relationships created by the strategy.
Recall is calculated as the number of correctly matched relationships resulting from a strategy, divided by the number of true (expected) relationships. It can also be interpreted as the probability that a true (correct) relationship will be created by the strategy. Low recall means a high rate of false negatives, which are relationships that should have been created by the strategy but were not made.

Applying these measures to our prior example, the strategies achieved the following results:

Strategy 1: accuracy 0.67, precision 0.5, recall 0.5
Strategy 2: accuracy 0.67, precision 1.0, recall 0.5

As we can see, while both strategies have the same accuracy, using precision and recall better describes the difference between the two sets of results. Strategy 1’s lower precision indicates it made false positive matches, while Strategy 2’s perfect precision shows that it made none. The identical recall scores show both identified half of the possible matches.

Of course, results calculated using such a small dataset are not very meaningful. If we obtained these scores from a large, representative evaluation dataset, it would indicate to us that Strategy 1 risks introducing many incorrect relationships, while Strategy 2 would be unlikely to do so. In both cases, we would still expect approximately half of the possible relationships to be missing from the strategies’ outputs.

Which one is more important to prioritise, precision or recall? It depends on the use case. As a general rule, if you want to use the strategy in a fully automated way, without any form of manual review or correction of the results, we recommend paying more attention to precision. Privileging precision will allow you to better control the number of incorrect relationships added to your data. If you want to use the strategy in a semi-automated fashion, where there is a manual examination of and a chance to correct the results, pay more attention to recall. Doing so will guarantee that enough options are presented during the manual review stage and fewer relationships will be missed as a result.

To get a more balanced estimation of performance, we can also consider both precision and recall at the same time using a measure called F-score. F-score combines precision and recall into a single number, with variable weight given to either aspect. There are three commonly used types, each calculated as the weighted harmonic mean of precision and recall:

F0.5: Precision is weighted more heavily. It can be understood as a score that is 50% more sensitive to precision than recall. A high F0.5 score indicates a measure of performance that minimises false positives.
F1: Equal weight is given to both precision and recall. It can be interpreted as the most balanced score in this set. High F1 indicates good overall performance, with both false positives and false negatives being minimised equally.
F2: Recall is weighted more heavily. It can be understood as a score that is 50% more sensitive to recall than precision. A high F2 score indicates a measure of performance where false negatives are minimised.

Each of these variants allows for fine-tuning the evaluation metric to align with your expectations for a specific matching task. Choose whichever reflects the relative importance of precision versus recall for your use case.

To summarise, to avoid falling prey to misleading sales pitches or silly quizzes, it is important to have a good understanding of the performance of any strategies you are building or integrating. With thorough evaluation, including a representative dataset and carefully considered metrics, we can estimate the quality of matching and, by extension, its resulting relationships.

Now that we’ve covered how to evaluate effectively, we can move on to some other aspects of metadata matching. Our next blog post will take a final, more holistic view of matching, exploring some complementary considerations to all of the preceding. Stay tuned for more!

The myth of perfect metadata matching

Dominika Tkaczyk — Wed, 28 Aug 2024 00:00:00 +0000

In our previous instalments of the blog series about matching (see part 1 and part 2), we explained what metadata matching is, why it is important and described its basic terminology. In this entry, we will discuss a few common beliefs about metadata matching that are often encountered when interacting with users, developers, integrators, and other stakeholders. Spoiler alert: we are calling them myths because these beliefs are not true! Read on to learn why.

If you have stuck with us this far in our series, hopefully, you are at least a bit excited about the possibility of creating new relationships between the works, authors, institutions, preprints, datasets, and myriad other objects in our existing scholarly metadata. Who would not want all of these to be better connected?

We have to pause for a moment and be honest with you: metadata matching is a complex problem, and doing it correctly requires significant effort. What is worse, even if we do everything right, our matching won’t be perfect. This may be counterintuitive. Perhaps you’ve heard that matching is not a hard problem, or have encountered people surprised that a matching strategy returned a wrong or incomplete answer. Sometimes, it is obvious to a person from looking at some specific example that a match should (or should not) have been made, so they naturally assume that a change to account for this has to be simple.

Misconceptions like these can be problematic. They create confusion around matching, drive users’ expectations to unreasonable levels, and make people drastically underestimate the effort needed to build and integrate matching strategies. So let’s dive right in and debunk a few common myths about metadata matching.

Myth #1: A metadata matching strategy should be 100% correct

Anyone who has built or supported a matching strategy has likely encountered the following belief: it is possible to develop a perfect strategy, meaning one that always returns the correct results, no matter the inputs. The unfortunate truth is that while one’s aim should always be to design matching strategies that return correct results, once we move beyond the simplest class of problems or artificially clean data, no strategy can achieve this outcome. In thinking through why this is the case, some inherent constraints become obvious:

The inputs to matching are often strings in human-readable formats, which can vary wildly in their structure, order and completeness. Since they’re intended to be parsed by people, instead of machines, they’re inherently lossy and frequently unstructured, anticipating that a person can infer from the source context what is being referenced. Matching strategies, although built to make sense of unstructured data, unfortunately, don’t have the luxury of this flexibility. A strategy has to account for translating a messy, partial, or inconsistent input into a correct and structured match.

Consider, for example, the following inputs to an affiliation matching strategy:

“Department of Radiology, St. Mary’s Hospital, London W2 1NY, UK”
“Saint Mary’s Hospital, Manchester University NHS Foundation Trust”
“St. Mary’s Medical Center, San Francisco, CA”
“St Mary’s Hosp., Dublin”
“St Mary’s Hospital Imperial College Healthcare NHS Trust”
“聖マリア病院”

In order to correctly identify the organisations mentioned here, the matching strategy must be able to distinguish between different ways of representing the same institution, disambiguate multiple institutions that have similar names, and handle variant forms for the parts of each name (Saint/St./St), identify the same name in different languages (“聖マリア病院” is Japanese for “St. Mary’s Hospital”), and make assumptions about partial or ambiguous locations translating to more precise references. While a person reviewing each of these strings might be able to accomplish these tasks, even here there are some challenges. Does “St Mary’s Hosp., Dublin” refer to the hospital in Ireland or a separate hospital in one of the many cities that share this name? Should we presume that because “聖マリア病院” is in Japanese, this refers to a hospital in Japan? Would someone, by default, be aware that St. Mary’s Hospital in London is part of the Imperial College Healthcare NHS Trust, such that inputs one and five refer to the same organisation?

An additional challenge lies in the quality of the data, which in the context of matching, encompasses both the input and the dataset being matched against. In real world circumstances, no dataset is fully accurate, complete, or current and certainly not all three. As a result, there will always be functionally random differences between inputs to the strategy and the entities to be matched. A theoretically perfect matching strategy would thus need to distinguish between inconsequential discrepancies resulting from gaps, errors, and variable forms of reference and actual, meaningful differences indicating an incorrect match. As one might imagine, this would require near total knowledge of the meaning and context for all inputs and outputs, a nigh-on impossible task for any person or system!

As a consequence, no metadata matching strategy will ever be perfect. It is unreasonable for us to expect them to be. This does not mean, of course, that all strategies are equally flawed or destined to forever return middling results. Some are better than others and we can improve them over time. Which brings us to the next myth:

Myth #2: It is always a good idea to adapt the matching strategy to a specific input

Matching strategies are not static. They can - and should - be improved. There is, however, a deceptive trap that one can fall into when attempting to improve a matching strategy. Whenever we encounter an incorrect or missing result for a specific input, we treat this problem like a software bug and try to adapt the strategy to work better for it, without considering all other cases.

The more complicated reality is that the quality of matching results is controlled through a complex set of trade-offs between precision and recall that determine the kind and number of relationships created between items:

Precision is calculated as the number of correctly matched relationships resulting from a strategy, divided by the total number of matched relationships. It can also be interpreted as the probability that a match is correct. Low precision indicates a high rate of false positives, which are incorrect relationships created by the strategy.
Recall is calculated as the number of correctly matched relationships resulting from a strategy, divided by the number of true (expected) relationships. It can also be interpreted as the probability that a true (correct) relationship will be created by the strategy. Low recall means a high rate of false negatives, which are relationships that should have been created by the strategy but were not made.

The diagram depicts false negatives and false positives. The ideal outcome would be that the ellipses are identical, matched relationships are exactly the same as true relationships, and there are no false negatives or false positives. In practice, we try to make the intersection as big as possible.

The tradeoff between precision and recall roughly means that modifying the strategy to improve recall will decrease precision, and vice versa.

Imagine, for example, we received a report about a relationship that was missed by matching because of a partial, noisy, or ambiguous input. We might be tempted to resolve this issue by relaxing our matching criteria. Unfortunately, this will have a cost of a higher overall rate of false positive matches.

Conversely, if we encounter a case where the matching has returned an incorrect match, we might attempt to make the matching strategy stricter to avoid this result. We should remember, however, that this may have the consequence of causing the strategy to skip many perfectly valid matches.

The tradeoff between precision and recall. (a) A strict strategy prioritises precision over recall resulting in more false negatives. (b) A relaxed strategy prioritises recall over precision resulting in more false positives.

Striking this balance becomes even more difficult when attempting to address multiple issues at once, or considering constraints like the time and resources consumed by each aspect of the strategy. Each choice can compound the individual effects in unanticipated and expensive ways. The aim of matching ultimately then can’t be to achieve perfect results for every single case. Fixing one particular situation might not be desirable, as it can result in breaking multiple other cases. Instead, we have to find a locally optimal balance that optimises the strategy’s utility, relative to these inherent limitations. This means accepting some level of imperfection as not just inevitable, but necessary for implementing a workable strategy. When you consider all this, you might conclude that…

Myth #3: We shouldn’t do large-scale, unsupervised matching

Imperfect matching strategies, when applied automatically to real-world large datasets, might:

Fail to discover some relationships (false negatives), an outcome that may not be terribly problematic. In the worst case scenario, we have wasted a great deal of effort developing matching strategies that do not improve our metadata.
Create incorrect relationships between items (false positives), what seems like a potentially larger problem, where we have added incorrect relationships to the metadata.

Many have the instinct to avoid false positives at any cost, even if this means missing many additional correct relationships at the same time. They might come to the conclusion that if we cannot have 100% precision (see our previous myth), we simply should not allow matching strategies to act in an automated, unsupervised way on large datasets. While there might be circumstances where this belief is rational, in the context of the scholarly record, this notion is seriously flawed.

First, if you are dealing with any medium to large-sized dataset, it almost certainly contains errors, even before you apply any automated processing to it. Even if data is submitted and curated by users, they can still make mistakes, and might themselves be using automated tools for extracting the data from other sources, without your knowledge. It is thus not entirely obvious that applying an (imperfect) matching strategy to create more relationships would actually make the data quality worse.

Second, while we cannot eliminate all matching errors, we can place a high priority on precision when developing strategies, with the aim of keeping the number of incorrectly matched results as low as possible. We can also make use of additional mechanisms to easily correct for incorrectly matched results, for example doing so manually, in response to error reports.

Finally, the results of matching should always contain provenance information to distinguish them from those that have been manually curated. This way, the users can make their own decisions about whether to use and trust the matching results, relative to their use case.

By applying those additional checks, we can minimise the negative effects of incorrect matching, while at the same time reap the benefits of filling gaps in the scholarly record.

Myth #4: We can only ever guess at the accuracy of our matching results

In attempting to determine the correctness of our matching, we immediately encounter a number of inherent limitations. The sheer amount of entries in many datasets prevents a thorough, manual validation of the results, but if instead, we use too few or specific items as our benchmarks, these are unlikely to be representative of overall performance. The unpredictable nature of future data adds another wrinkle: will our matching always be as successful as when we first benchmarked it or will its performance degrade relative to some change in the data?

With so many unknowns, are we then doomed? No! We have rigorous and scientific tools at our disposal that can help us estimate how accurate our matching will be. How do we use them? Well, that is a big and fairly technical topic, so we will leave you with this little cliffhanger. See you in the next post!

The anatomy of metadata matching

Dominika Tkaczyk — Thu, 27 Jun 2024 00:00:00 +0000

In our previous blog post about metadata matching, we discussed what it is and why we need it (tl;dr: to discover more relationships within the scholarly record). Here, we will describe some basic matching-related terminology and the components of a matching process. We will also pose some typical product questions to consider when developing or integrating matching solutions.

Basic terminology

Metadata matching is a high-level concept, with many different problems falling into this category. Indeed, no matter how much we like to focus on the similarities between different forms of matching, matching affiliation strings to ROR IDs or matching preprints to journal papers are still different in several important ways. At Crossref and ROR, we call these problems matching tasks.

Simply put, a matching task defines the kind or nature of the matching. Examples of matching tasks are bibliographic reference matching, affiliation matching, grant matching, or preprint matching.

Every matching task has an input, which is all the data that is needed to perform the matching. Input data can come in many shapes and forms, depending on the matching task. For example, all of the following could be inputs to a matching task:

Department of Molecular Medicine, Sapporo Medical University, Sapporo 060-8556, Japan

<fr:program xmlns:fr="http://www.crossref.org.pluma.sjfc.edu/fundref.xsd" name="fundref">
<fr:assertion name="fundgroup">
<fr:assertion name="funder_name">
European Union's Horizon 2020 Research and Innovation Program through Marie Sklodowska Curie
<fr:assertion name="funder_identifier">http://dx.doi.org.pluma.sjfc.edu/10.13039/501100000780</fr:assertion>
</fr:assertion>
<fr:assertion name="award_number">721624</fr:assertion>
</fr:assertion>
</fr:program>

Everitt, W. N., & Kalf, H. (2007). The Bessel differential equation and the Hankel transform. Journal of Computational and Applied Mathematics, 208(1), 3–19.

{
"title": "Functional single-cell genomics of human cytomegalovirus infection",
"issued": "2021-10-25",
"author": [
{"given": "Marco Y.", "family": "Hein"},
{"given": "Jonathan S.", "family": "Weissman", "ORCID": "http://orcid.org/0000-0003-2445-670X"}
]
}

Every matching task also has an output. For our purposes, this is almost exclusively zero or more matched identifiers. In the context of a specific matching task, output identifiers may be of a specific type (e.g. we might match to a ROR ID, and never to an ORCID ID). In some cases, there can be a certain target set as well (i.e. matching only to DataCite DOIs). The output identifiers can have different cardinality depending on the task, meaning that the matching task might allow for zero, one, or more identifiers as a result of matching to a single input.

A matching strategy defines how the matching is done. Multiple strategies can exist for a specific matching task. Compound strategies can run other strategies and combine their outcomes into a single result.

In some cases, we may also want the matching strategy to output a confidence score for each matched identifier. A confidence score represents the degree of certainty or likelihood that the matched identifier is correct, typically expressed as a value between 0 and 1. This score may help with post-processing or further interpretation of the results.

To summarise, the anatomy of the matching task can be diagrammed as follows:

How to specify a matching task

Whenever we plan the development or integration of a matching solution, it is good to begin by answering a few basic questions:

What problem do we plan to solve with our matching task? What would we call our matching task and how would we describe it?
What do we expect as the input for this matching task? Which input formats do we need to be able to accept? What information do we expect to find in this input?
What kind of identifiers should be output? Is there a target set of identifiers? Can our matching output zero/one/or multiple identifiers, and under what conditions might that occur?

These sound fairly simple, but the answers to these questions can be remarkably complex. Once one tries to apply these concepts to real-world problems, they might encounter several non-obvious challenges.

For example, one common concern is at what level we should define each matching task. Consider the following problems:

Matching bibliographic reference strings to DOIs. Example input:

Everitt, W. N., & Kalf, H. (2007). The Bessel differential equation and the Hankel transform. Journal of Computational and Applied Mathematics, 208(1), 3–19.

Matching structured bibliographic reference to DOIs. Example input:

{
volume: "208",
author: "Everitt",
journal-title: "J. Comput. Appl. Math.",
article-title: "The Bessel differential equation and the Hankel transform",
first-page: "3",
year: "2007",
issue: "1"
}

Are those discrete matching tasks (unstructured reference matching vs. structured reference matching), or are they the same task (reference matching) that can accept different types of inputs (unstructured or structured)?

Similarly, let’s compare the following tasks:

Matching affiliation strings to ROR IDs. Example input:

Department of Molecular Medicine, Sapporo Medical University, Sapporo 060-8556, Japan

Matching funder names to ROR IDs. Example input:

Alexander von Humboldt Foundation

Are these different matching tasks (affiliation matching vs. funder matching), or the same task with different inputs (organisation matching)?

Defining the boundaries of a matching task can also be difficult. Consider, for example, the need to obtain ROR IDs for organisations mentioned in the acknowledgements section of a full-text academic paper. To begin, one may first extract the acknowledgement section from the full text, then run something like a named entity recognition (NER) tool to isolate the organisation names from the extracted text, and finally match these names to ROR IDs. Is this entire process matching, with the input being the full text of a paper? Or perhaps matching starts with the acknowledgement section as the input? Instead, is it only the last phase, where we try to match the extracted name to the ROR ID, that constitutes the matching task, with the extraction phases being completely separate processes?

There are also important questions related to the expected behaviour of a matching strategy. Consider, for example, developing an affiliation matching strategy where we define our input as “an affiliation string”. What should happen when the strategy gets something else on the input, for example, song lyrics? Perhaps the strategy should simply return no matches, or an error, or we could say that in such a situation the behaviour is undefined and it simply doesn’t matter what is returned. But what should happen if in this input we have the lyrics of Street Life by Roxy Music, a song that mentions the names of a few universities that happen to have ROR IDs?

It is likewise important to consider what should happen if different parts of the input match to different identifiers, like in the following example:

Department of Haematology, Eastern Health and Monash University, Box Hill, Australia

Here, “Eastern Health” matches to https://ror.org/00vyyx863 and “Monash University” to https://ror.org/02bfwt286. Should the matching strategy return all the identifiers, one of them (if so, which one?), or nothing at all?

Similar questions arise when it is possible to match to multiple versions (or duplicates) in the target identifier set. This can happen, for example, in the context of bibliographic reference matching or preprint matching. Multiple matches may occur when there are different editions, reprints, or variations of the same publication in the target dataset, each with its own unique identifier.

If you are waiting for an answer to these questions, we unfortunately must disappoint you here. These can only be answered in the context of a specific problem, considering who the users are and what it is they need and expect.

Did you notice any other subtleties related to metadata matching and its concerns? Are there other non-obvious questions that should be considered when planning to develop or integrate metadata matching strategies? Let us know—we’d love to hear from you!

Metadata matching 101: what is it and why do we need it?

Dominika Tkaczyk — Thu, 16 May 2024 00:00:00 +0000

At Crossref and ROR, we develop and run processes that match metadata at scale, creating relationships between millions of entities in the scholarly record. Over the last few years, we’ve spent a lot of time diving into details about metadata matching strategies, evaluation, and integration. It is quite possibly our favourite thing to talk and write about! But sometimes it is good to step back and look at the problem from a wider perspective. In this blog, the first one in a series about metadata matching, we will cover the very basics of matching: what it is, how we do it, and why we devote so much effort to this problem.

What is metadata matching?

Would you be able to find the DOI for the work referenced in this citation?

Everitt, W. N., & Kalf, H. (2007). The Bessel differential equation and the Hankel transform. Journal of Computational and Applied Mathematics, 208(1), 3–19.

We bet you could! You might begin, for example, by pasting the whole citation, or only the title, into a search engine of your choice. This would probably return multiple results, which you would quickly skim. Then you might click on the links for a few of the top results, those that look promising. Some of the websites you visit might contain a DOI. Perhaps you would briefly compare the metadata provided on the website against what you see in the citation. If most of this information matches (see what we did there?), you would conclude that the DOI from that website is, in fact, the DOI for the cited paper.

Well done! You just performed metadata matching, specifically, bibliographic reference matching. Matching in general can be defined as the task or process of finding an identifier for an item based on its structured or unstructured “description” (in this case: finding a DOI of a cited article based on a citation string).

But matching doesn’t have to just be about citations and DOIs. There are many other instances of matching we can think of, for example:

finding the ROR ID for an organisation based on an affiliation string,
finding the ORCID ID for a researcher based on the person’s name and affiliation,
finding the ROR ID for a funder based on the acknowledgements section of a research paper,
finding the grant DOI based on an award number and a funder name.

Matching doesn’t have to be done manually. It is possible to develop fully automated strategies for metadata matching and employ them at scale. It is also possible to use a hybrid approach, where automated strategies assist users by providing suggestions.

Developing automated matching strategies is not a trivial task, and if we want to do it right, it takes a great deal of time and effort. This brings us to our next question: is it worth it?

Why do we need matching?

In short, metadata matching gives us a more complete picture of the research nexus by discovering missing relationships between various entities within and throughout the scholarly record:

These relationships are very powerful. They provide important context for any entity, whether it is a research output, a funder, a research institution, or an author. Imagine for a moment the scholarly record without any such relationships, where all bibliographic references, affiliations (institution names and addresses), and funding information (funder names and grant titles) are provided as unstructured strings only. In such a world, how would you calculate the number of times a particular research paper was cited? How would you get a list of research outputs supported by a specific funder? It would be incredibly challenging to navigate, summarise, and describe research activities, especially considering the scale. Thankfully, these and many other questions can be answered thanks to metadata matching that discovers relationships between entities in the scholarly record.

There are two primary ways we can use metadata matching in our workflows: as semi-automated tools that help users look up the appropriate identifiers or as fully automated processes that enrich the metadata in various scholarly databases.

The first approach is quite similar to the example we described at the beginning. If you are submitting scholarly metadata, for example of a new article to be published, you can use metadata matching to look up identifiers for the various entities and include these identifiers in the submission. For example, with the help of metadata matching, instead of submitting citation strings, you could provide the DOIs for works cited in the paper and instead of the name and address of your organisation, you could provide its ROR ID. To make this easier for people, metadata submission systems and applications sometimes integrate metadata matching tools into user interfaces.

The second approach allows large, existing sources of scholarly metadata to be enriched with identifiers in a fully automated way. For example, we can match affiliation strings to ROR IDs using a combination of machine learning models and ROR’s default matching service, effectively adding more relationships between people and organisations. We can also compare journal articles and preprints metadata in the Crossref database by calculating similarity scores for titles, authors, and years of publication to match them with each other and provide more relationships between preprints and journal articles. This automated enrichment can be done at any point in time, even after research outputs have been formally published.

There are fundamental differences between these two approaches. The first is done under the supervision of a user, and for the second, the matching strategy makes all the decisions autonomously. As a result, the first approach will typically (although not always) result in better quality matches. By contrast, the second approach is much faster, generally less expensive, and scales to even very large data sources.

In the end, no matter what approach is used, the goal is to achieve a more complete accounting of the relationships between entities in the scholarly record.

This blog is the first one in a series about metadata matching. In the coming weeks, we will cover more detail about the product features related to metadata matching, explain why metadata matching is not a trivial problem, and share how we can develop, assess, compare, and choose matching strategies. Stay tuned!

Discovering relationships between preprints and journal articles

Dominika Tkaczyk — Thu, 07 Dec 2023 00:00:00 +0000

In the scholarly communications environment, the evolution of a journal article can be traced by the relationships it has with its preprints. Those preprint–journal article relationships are an important component of the research nexus. Some of those relationships are provided by Crossref members (including publishers, universities, research groups, funders, etc.) when they deposit metadata with Crossref, but we know that a significant number of them are missing. To fill this gap, we developed a new automated strategy for discovering relationships between preprints and journal articles and applied it to all the preprints in the Crossref database. We made the resulting dataset, containing both publisher-asserted and automatically discovered relationships, publicly available for anyone to analyse.

TL;DR

We have developed a new, heuristic-based strategy for matching journal articles to their preprints. It achieved the following results on the evaluation dataset: precision 0.99, recall 0.95, F0.5 0.98. The code is available here.
We applied the strategy to all the preprints in the Crossref database. It discovered 627K preprint–journal article relationships.
We gathered all preprint–journal article relationships deposited by Crossref members, merged them with those discovered by the new strategy, and made everything available as a dataset. There are 642K relationships in the dataset, including:
- 296K provided by the publisher and discovered by the strategy,
- 331K new relationships discovered by the strategy only,
- 15K provided by the publisher only.
In the future, we plan to replace our current matching strategy with the new one and make all discovered relationships available through the Crossref REST API.

Introduction

Relationships between preprints and journal articles link different versions of research outputs and allow one to follow the evolution of a publication over time. The Crossref deposit schema allows Crossref members to provide these relationships for new publications, either as a has-preprint relationship deposited with a journal article, or an is-preprint-of relationship deposited with a preprint.

To assist members who deposit preprints, we also try to connect deposited journal articles with preprints. The current method looks for an exact match between the title and first authors. We send possible matches as suggestions to the preprint server, which decides whether to update the metadata with the relationship.

At the time of writing, 137,837 journal articles in the Crossref database have a has-preprint relationship¹, and 562,225 works of type posted-content (preprints belong to this type) have an is-preprint-of relationship².

We suspected that many preprint–journal article relationships are missing, as some members inevitably fail to deposit them, even after suggestions from the current matching strategy. Another factor is that the current strategy is fairly conservative, and probably misses a significant number of relationships. For these reasons, we decided to investigate whether we could improve on the current process. Doing so would allow us to infer missing relationships on a large scale, similar to how we automatically match bibliographic references to DOIs.

This preprint matching task can be defined in two directions:

We start with a journal article and we want to find all its preprints.
We start with a preprint and we want to find a subsequently published journal article.

On the one hand, matching from journal articles to preprints would allow us to enrich the database continually with new relationships, either periodically or every time new content is added. Since journal articles tend to appear in the database later than their preprints, it makes sense for a new journal article to trigger the matching and not the other way round. This way we can expect the potential matches to be already in the database at the time of matching.

On the other hand, matching from preprints to journal articles can be useful in a situation where we want to add relationships in an existing database retrospectively. In our case, the database contains many more journal articles than preprints, so for performance reasons it is better to start with preprints.

In both cases we are dealing with structured matching, meaning that we match a metadata record of a work (preprint or journal article), rather than unstructured text.

As a result of matching a single preprint or a single journal article, we should expect zero or more matched journal articles/preprints. Multiple matches occur when:

there are multiple versions of the matched preprint and/or
matched works have duplicates.

The image shows the result of matching a journal article to two versions of a preprint:

Matching strategy

Our matching strategy uses the following workflow:

Gathering a short list of candidates using the Crossref REST API.
Scoring the similarity between the input item and each candidate.
A final decision about which candidates, if any, should be returned as matches.

Gathering candidates is done using the Crossref REST API’s query.bibliographic parameter. The query is a concatenation of the title and authors’ last names of the input item. We filter the candidates based on their type, to leave only preprints or only journal articles, depending on the direction of the matching. In the future, instead of getting the candidates from the REST API, we will be using a dedicated search engine, optimised for preprint matching.

Scoring candidates is heuristic-based. Similarities between titles, authors, and years are scored independently, and the final score is their average. Titles are compared in a fuzzy way using the rapidfuzz library. Authors are compared pairwise using the ORCID ID, or first/last names if ORCID ID is not available. The similarity score between issued years is 1 if the article was published no earlier than one year before the preprint and no later than three years after the preprint, or 0 otherwise.

The final decision is made based on two parameters: minimum score and maximum score difference, both chosen based on a validation dataset. The following diagram depicts the results of applying these two parameters in all possible scenarios. First, any candidate scoring below the minimum score is rejected (grey area in the diagram). Second, the scores of the remaining candidates are compared with the score of the top candidate. If the score of a candidate is close enough to the score of the top candidate, it is returned as a match (blue area).

This process can result in the following scenarios:

Scenario A: there is no candidate above the minimum score. This means nothing matches sufficiently, so nothing is returned.
Scenario B: there is only one candidate above the minimum score. This means it is the best match and we don’t have much of a choice, so it is returned.
Scenario C: there are multiple candidates above the minimum score, and they all have similar scores. This means they all are similarly good matches, so all are returned.
Scenario D: there are multiple candidates above the minimum score, but their scores differ a lot. In this case, we don’t want to return all of them, but only those that are close to the top match. Intuitively, we don’t want to return less-than-great matches if we have really great ones. This is when the maximum score difference comes into play: we return the candidates with the “score distance” to the top candidate lower than the maximum score difference.

We evaluated this strategy on a test set sampled from the Crossref metadata records. The test set contains 3,000 pairs (journal article, set of corresponding preprints). Half of the journal articles have known preprints and the other half don’t. The test set can be accessed here.

We used precision, recall, and F0.5 as evaluation metrics:

Precision measures the fraction of the matched relationships that are correct.
Recall measures the fraction of the true relationships that were matched.
F0.5 combines precision and recall in a way that favours precision.

The strategy achieved the following results: precision 0.9921, recall 0.9474, F0.5 0.9828. The average processing time was 0.96s.

We have made this strategy (journal article -> preprints) available through the (experimental) API: https://marple-research-crossref-org.pluma.sjfc.edu/match?task=preprint-matching&strategy=preprint-sbmv&input=10.1109/access.2022.3213707. The input is the DOI of a journal article we want to match to preprints, and the output is a list of matches found, along with the score for each.

We have investigated other approaches to making decisions about which candidates to return as matches (step 3 above), including using machine learning. At present none have outperformed the heuristic approach described above. The heuristic method is also preferred because of its fast performance.

Preprint–journal article relationship dataset

We applied the strategy to the entire Crossref database:

We selected all preprints published until the end of August 2023. This included only works with type posted-content and subtype preprint, as reported by the REST API. There were 1,050,247 of them.
We ran the matching strategy (preprint -> journal article) on them. This resulted in 627,011 preprint–journal article relationships.
The resulting relationships were combined with the relationships deposited by the Crossref members. We included relationships of types has-preprint or is-preprint-of, where both sides of the relationship exist in our database, were published until the end of August 2023, and are of proper types and subtypes (type=journal-article for the journal article and type=posted-content, subtype=preprint for the preprint).

The resulting dataset is a single CSV file with the following fields:

preprint DOI (string)
journal article DOI (string)
whether the publisher of the journal article deposited this relationship (boolean)
whether the publisher of the preprint deposited this relationship (boolean)
the confidence score returned by the strategy (float, empty if the strategy did not discover this relationship)

The dataset contains:

641,950 relationships in total, including 580,532 preprints and 565,129 journal articles,
14,939 of them were deposited by the Crossref members, but not discovered by the strategy,
330,826 of them were discovered by the strategy, but not provided by any Crossref member,
296,185 of them were both deposited by a Crossref member and discovered by the strategy.

The dataset can be downloaded here.

Conclusions and what’s next

Overall, based on the number of existing and newly discovered preprint–journal article relationships, it seems that employing automated matching strategies would approximately double the number of these relationships in the Crossref database. In the future, we would like to match new journal articles on an ongoing basis. We also plan to make all discovered relationships available through the REST API.

In the meantime, we will be publishing the discovered relationships in the form of datasets, and we invite anyone interested to further analyse this data. And if you find out something interesting about preprints and their relationships, do let us know!

Forming new relationships: contributing to open source

Patrick Vale — Wed, 19 Oct 2022 00:00:00 +0000

TL;DR

One of the things that makes me glad to work at Crossref is the principles to which we hold ourselves, and the most public and measurable of those must be the Principles of Open Scholarly Infrastructure, or POSI, for short. These ambitions lay out how we want to operate - to be open in our governance, in our membership and also in our source code and data. And it’s that openness of source code that’s the reason for my post today - on 26th September 2022, our first collaboration with the JSON Forms open-source project was released into the wild.

Like most organisations, we depend heavily on open-source software for our operations - the software is universally available, generally high quality and ‘free’. And it’s easy to take that dependency, and the associated dependency on free time and effort on the part of the maintainers, for granted - but that’s not very sustainable. In fact, we believe relying on open-source software without helping to sustain it is an anti-pattern, and this project marks the start of our efforts to make funding open-source software a standard part of our technology budget.

This isn’t the first time we’ve supported or released open-source software. Indeed for the past few years, all our new software is open source, and we’re in the process of replacing old closed code with new, so that eventually all our code will be open source. But this is the first time we’ve contributed extensively to something that isn’t focussed primarily on us, and our services. This is a project that we will find very useful, but it is a general purpose tool, and it’s already gaining traction in the community.

Background and motivations

A while back, I was tasked to do a quick spike of work on testing the theory that we could use automated form generation tools to bring new interfaces to our users more quickly, and make them easier for “people who aren’t devs” to adapt and manage. We wanted to build a new user interface for registering content, and especially we wanted to make it easier for funders to register the grants they were awarding. As well as being more approachable by a less-technical audience, we also wanted these forms to be accessible (in terms of a11y and users of assistive technology) and localisable - we wanted a solution that would cater to the needs of our rapidly diversifying membership.

Enter JSON Schema

We were clear about one side of the puzzle - we knew that we had to look beyond the XML ecosystem upon which much of our existing system is built - and landed on JSON Schema. JSON Schema is a ‘vocabulary that allows you to annotate and validate JSON documents’. This means you can describe the shape you expect your data to take, and apply constraints-based validation to that. Which means, in terms of a form library, that you can infer the structure of the form and test that the data entered into it matches what you expect. More than that, you can use that built-in validation to provide error messages to help people get the data right, first time.

Working backwards from the outcome, the argument for adopting JSON Schema is compelling. It provides a mechanism for checking that data you are handling (for example, receiving input from a form) conforms to the constraints that you declare, but also allows you to tell people up-front, in a human and machine-readable way, what structure and format you will accept. This closed-loop of data annotation and validation gets more appealing when you look at the wide adoption of JSON Schema across languages and libraries. You can pretty much guarantee that for whatever client or server -side technology you are using, there will be a JSON Schema validator for it. Being able to share schemas across your systems (and equally importantly, with third parties) moves JSON schema from ‘just’ being about data validation, to a key supportive technology.

Building a form derived from a JSON Schema is an equally attractive prospect. JSON Schema was conceived during the AjaxWorld conference in 2007 as a ‘JSON-based format for defining the structure of JSON data’, and its use as a form-generation tool is relatively new, but there is growing community interest. There is even a discussion about how to best create a JSON Schema vocabulary, specifically geared towards addressing some of the needs of form generation users. However, even in its current form, a JSON Schema can be passed to a library, and a very serviceable user interface appears. The devil is always in the detail, and the client-side libraries differ in their abilities to customise areas such as layout (you may not always want your form fields to appear in exactly the same order as they do in your JSON Schema), custom elements (you might want something that wasn’t a form input, or that changes based on user input) and localisation. The ability to flexibly customise the appearance and behaviour of the interface was a key factor in our selection of a client-side form generation library.

Choosing a library

The other side of the puzzle was less clear - choosing a UI library that would take this JSON Schema, and turn it into a useful, and usable, form. I made the prototype using the venerable React JSON Schema form. This worked well as a proof of concept, but veered dramatically off our chosen Frontend stack of VueJS and Vuetify, and had some architectural constraints that would limit the scope of customisations we could make to our forms. So I went off looking for libraries that would work with our stack and came up with Vuetify JSON Schema Form, and JSON Forms.

Vuetify JSON Schema Form matched our stack perfectly, but made some interesting decisions about the layout of data within the form, and that wouldn’t suit our purposes without dramatic modification.

JSON Forms was an abstracted library, with a core handling the JSON Schema transformation and validation, and separate rendering libraries to handle the form generation. This was great - they had renderers for Angular, React, and even some support for VueJS. But not Vuetify.

Clearly, we were going to have to make something.

We made contact with the maintainers of both short-listed libraries to see how we could collaborate in creating a tool that would meet all of our (and hopefully, much of the wider community’s) requirements. Both maintainers were very helpful, and we had constructive discussions in both cases. In the end, we decided that the abstracted nature of the JSON Forms project was a better fit for our needs, providing a flexible platform on which we - and others - could extend. We were fortunate to receive funding from the Gordon and Betty Moore Foundation (Grant Agreement #10485) in order to accelerate this work, so we could provide a Grant Registration UI more quickly. We paid a large portion of that funding to the library maintainers, and Crossref contributed a portion of my time on the project. This allowed us to enter into an agreement with EclipseSource, the maintainers of JSON Forms, to collaboratively develop the new VueJS and Vuetify renderer library. Stefan Dirix, the lead maintainer, worked with me to build it.

We didn’t forget about Vuetify JSON Schema Form though, and by way of appreciation for their help in the early stages, Crossref made a contribution towards the continued development of that library.

JSON Forms - now with Vuetify

Work started on the JSON Forms Vuetify renderer set in September 2021 - Stefan quickly created the first early prototypes of the new form renderers - but then we had a stroke of luck. Our repository received more input from the community. The one that made us sit up and take real notice was the news that someone else had already ported the JSON Forms React renderer set to Vue/Vuetify - and was offering this as a contribution. Krasimir Chobantonov’s fantastic first contribution got merged in at the end of the month. This propelled the project forward massively, and was an early validation of the value of working in the open. Needless to say, we were very grateful. Another example of the open source value chain was that Stefan - as the maintainer - could take the time to carefully review and tidy up the incoming code, so what was merged was the product of two great developers.

Having this great head start meant we could turn our attention to one of the other big areas we wanted to get right - localisation. Traditionally, JSON Schema -generated forms have handled localisation (translation of text and adjustment of date and numerical formats) by wholesale duplication and translation of the schema. This is cumbersome, and doesn’t integrate very well with custom error messages, nor external sources of interface messages (think form labels, descriptions, placeholders). So Stefan came up with a proposal, which we accepted, to add complete i18n support to the library. We now have a mechanism by which you can hook up a translation engine of your choice, and JSON forms will use that to lookup messages, before falling back to the validator (also localised!) and finally, the JSON Schema’s defaults. This gives much stronger integration and allows the community to plug in their existing localisation methods - no wasted effort.

Since the localisation addition, we’ve been working on fine-tuning the layout engine, making bug fixes, and integrating more closely with the underlying Vuetify library. This allows developers to more easily use the existing Vuetify parameters to change the style and behaviour of their form widgets. Again, no wasted effort.

We’re lucky to have an active community - @kchobantonov continues to make great contributions and push the library forward in unexpected ways - and the library is gaining popularity, with an average of a few hundred downloads per day.

Some of our funder members have already seen this work in action, and given their feedback on early iterations of the user interface that supports registering grant records. We’ll be releasing this publicly very soon to get feedback from members - and then using that feedback to iterate on the grants registration form, and look towards extending it to other record types.

Open source POSItivity

A continuous theme throughout this project has been the willingness of people working on these open source projects to be generous with their time and experience. Whether it has been form generation libraries, the JSON Schema project or maintainers of localisation plug-ins - help, advice and encouragement have never been far away. And that’s appreciated. But it’s not something that we, or any other organisation who relies on the software they produce, should take for granted. Open source software helps everyone who uses it, and there’s a real opportunity within our community to make meaningful steps towards supporting its sustainability. Ironically, it’s often the most-used general purpose tools that get the least attention. We can change that.

Look out for more

Look out for more posts from the engineering team, coming soon!

References

JSON Binpack: A space-efficient schema-driven and schema-less binary serialization specification based on JSON Schema (Chapter 3.2.1 History and Relevance)

https://web.archive.org/web/20071026190426/http://www.json.com/2007/09/27/json-schema-proposal-collaboration/

Accessibility for Crossref DOI Links: Call for comments on proposed new guidelines

Jennifer Kemp — Tue, 06 Sep 2022 00:00:00 +0000

Our entire community – members, metadata users, service providers, community organisations and researchers – create and/or use DOIs in some way so making them more accessible is a worthy and overdue effort.

For the first time in five years and only the second time ever, we are recommending some changes to our DOI display guidelines (the changes aren’t really for display but more on that below). We don’t take such changes lightly, because we know it means updating established workflows. We appreciate the questions that prompted us to make this recommendation and we know it’s critical that we get community input on the proposed updates.

TL;DR

Here is a quick overview:

DOIs and URLs themselves don’t really tell readers much. People with visual impairments rely on screen readers to read out loud the contents of a page. We’re asking for the title of each DOI to be added, in an ARIA (Accessible Rich Internet Applications) attribute, so these users understand what these links are for.
Accessible text, as this kind of description is known, should be included for all links, but at this time, we’re specifically recommending it for landing pages of newly registered records.
It’s not required, yet. We’re proposing a 2 year recommendation period and we want your feedback on the particulars, including timing and how we can help. Please take a short survey and/or get in touch and share your thoughts.
We’ll finalize these recommendations after assessing the feedback. Please check back for updates.

What is changing, when and why

The proposed updates are meant to improve overall usability, particularly for people with visual impairments, by aligning our guidelines with modern accessibility requirements such as the new W3C recommendations and the European Accessibility Act. This means that assistive technologies such as screen readers can interpret DOI links.

Why are changes being recommended?

DOIs are unique and persistent links to items in the scholarly record so it makes sense that they link to the full URLs for the associated content –for example, a journal article. The issue for people who rely on screen readers is that a DOI link doesn’t provide title or other information to give that link context. Users of screen readers need to know what the destination of a link is.

These users often lack the context that other users have; in fact, they may be presented with links in a document as a list. That’s why all links, not just DOI links, need what is called “accessible text.” Providing additional information for links requires ARIA (Accessible Rich Internet Applications) techniques. This speaks to the Web Content Accessibility Guidelines (WCAG), the standard guidelines for accessibility across the web, specifically success criterion 2.4.4 - Link Purpose (In Context), which aims to ‘help users understand the purpose of each link so they can decide whether they want to follow the link.’

For your feedback: recommended draft changes

We recommend the addition of an aria-label attribute for DOI links, containing as its value the descriptive title of the content represented by the DOI, so that screen readers can interpret DOI links. This means that, while the DOI display itself doesn’t actually change, the link is enhanced with additional, contextual information for the user of assistive technology, in one of two ways, either:

an aria-label attribute, described as ‘a way to place a descriptive text label on an object,’ identifying the destination, or
an aria-describedby attribute pointing to where the destination is identified in the surrounding text.

The updated HTML for a journal article*, for example, would be:

<a href="https://doi-org.pluma.sjfc.edu/10.5555/12345678" aria-label="DOI for Toward a Unified Theory of High-Energy Metaphysics: Silly String Theory">https://doi-org.pluma.sjfc.edu/10.5555/12345678</a>

Here the aria-label has been set to the value of the ‘title’ property as retrieved from the Crossref REST API at https://api-crossref-org.pluma.sjfc.edu/v1/works/10.5555/12345678.

*Note that fields may vary slightly for different record types.

This proposed solution allows screen readers to read aloud to users the value of the aria-label attribute, instead of the full DOI in the link text.

At this time, we are recommending the change for landing pages in particular, but it can and should be applied to wherever DOI links appear, whenever feasible (more on this below).

Our guidelines will continue to state that the DOI should always be displayed as a full URL link–that will not change. Neither will content registration–we are not asking for additional information in your deposits.

It’s not perfect, but it’s very worthwhile

This recommendation has some limitations worth noting but it must be said that there is no perfect solution.

DOI links appear in lots of places - PDFs for one notable example. We reviewed and tested the recommendation with Bill Kasdorf, Principal, Kasdorf & Associates, LLC, Richard Orme, CEO, DAISY Consortium, and George Kerscher, Chief Innovations Officer, DAISY Consortium-Senior Officer, Global Literacy, Benetech, who graciously provided their time and expertise. EPUBs and websites proved to be easy to update; other formats, notably PDFs, less so. Widespread adoption of accessible DOIs is so important and we don’t want confusion or frustration to get in the way of making progress. We support and welcome efforts to include an ARIA attribute wherever DOI links appear, but we recommend focusing on landing pages, for now.

Patrick Vale, Crossref Senior Front End Developer, explains that:

”DOI links serve a very specific purpose: to provide the persistent link to an item in the scholarly record. And as such, they present an unusual set of requirements when balancing accurately presenting the information they encode - the persistent link - and making that link accessible, and understandable. With these proposed changes, we hope to strike this balance.“

We know it will be a challenge (more on that below) but we think it’s absolutely a worthwhile effort. Indeed, we are undertaking a project to update our own website to meet these recommendations and to review overall accessibility.

As Bill Kasdorf notes:

“Most people have no idea how many people with visual impairments there are. Not only is it unfair to those people not to provide accessible text for links, the authors and publishers of the linked resource are missing a lot of readers. This update is a great move by Crossref, and every bit aligned with its mission to make scholarly content discoverable and consumable.”

We propose the following timeline, also for your feedback

Once finalized, following community feedback, the updated guidelines will be issued as a recommendation for a suggested period of two years starting next year, 2023. Beginning in 2025, the changes will be required for landing pages of newly registered content (and strongly recommended for existing registered content). Feedback on this approach and timeline is also encouraged.

Help us help you

We are conscious that adding descriptive information to DOI links places a significant responsibility on the members and Service Providers creating and hosting these links. Therefore, we are also considering the creation of a tool to help with implementation. Initial discussions suggest this could be a JavaScript helper tool, which could be included on member websites. We also welcome feedback as to how such a tool might be implemented, and how it would best integrate with existing sites and workflows.

Call for comments - by 1st November

We hope that this proposal is a welcome one and that the timing is good for moving forward together toward greater accessibility of the scholarly record. We welcome questions, feedback and suggestions through 1st November via the survey below or by email to feedback@crossref.org

Small changes, big impact

We’re excited to make changes that improve accessibility and we look forward to the community’s response to our proposal. We will share aggregated feedback in an updated post later this year.

A note on language

Multiple sources were consulted to find the most appropriate and inclusive term(s) for users of screen readers in this context. “Print disabled,” for example, seemed to be a good candidate but was ultimately deemed likely to be confusing to a very global publishing audience, who often don’t physically print anything. Sources differ slightly, for example between the US and UK and of course, this English text may well be translated into other languages. Feedback on the terms used here is also very welcome.

Additional resources

The Inclusive Publishing Hub (DAISY Consortium)
National Center on Disability and Journalism (Arizona State University, US)
Inclusive Language guidance (UK government)
The American Psychological Association (APA) Bias-Free Language Disability Guide
The Open Access Books Network (OABN)

With a little help from your Crossref friends: Better metadata

Jennifer Kemp — Thu, 31 Mar 2022 00:00:00 +0000

We talk so much about more and better metadata that a reasonable question might be: what is Crossref doing to help?

Members and their service partners do the heavy lifting to provide Crossref with metadata and we don’t change what is supplied to us. One reason we don’t is because members can and often do change their records (important note: updated records do not incur fees!). However, we do a fair amount of behind the scenes work to check and report on the metadata as well as to add context and relationships. As a result, some of what you see in the metadata (and some of what you don’t) is facilitated, added or updated by Crossref.

Much of the work is automated but some of it still requires manual intervention (sound familiar?). Here’s an overview:

Before registration

Our open APIs allow for Crossref metadata to be used throughout research and scholarly communications systems and services, before and after records are registered with us. Those who have used a search function in something like a manuscript submission system, rather than having to hand key or copy and paste the information, will appreciate how these integrations reduce time, effort and the likelihood of errors in collecting metadata well before it gets to Crossref.

For one example, it’s very common for members to use the metadata to add DOIs to reference lists when preparing deposits. Of course, new members first need a prefix (and a memberID and name, but more on that later) in order to register content. We also provide a suffix generator for help in constructing DOIs. If you’re not sure how best to make use of existing metadata in deposits, we’ve got a few options for you. Questions are welcome.

We don’t often put it this way but we should: Crossref members rely on the metadata as much, if not more, than the rest of the community. More and better metadata directly benefits our members.

Upon registration

There are a number of ways we work with the metadata when deposits are received.

Checking for uniqueness In order to avoid duplicate records, we check to make sure that a title or work hasn’t been registered before. Depending on what we find, a conflict report or failed registration may result.
Adding DOIs to references When references come to us without DOIs, we’ll try to match and add them.
ORCID auto-update We automatically update authors’ ORCID records (with their permission of course) whenever deposits include their ORCID iDs.
Preprint to VoR reports We compare title information and provide notifications of matching records to members depositing preprints, to help them fulfill their obligation to link to Versions of Record (VoRs), where they exist.
Relationships Like preprint to VoR links, components are another kind of relationship. These might be supplementary material such as figures we can link to the ‘parent’ record.
Funding data When members register only a funder name as part of the information on who funded the work, we’ll try to match it to its identifier from the Funder Registry, to support better linking between funders and works.
Timestamps We add date-times for first created and last updated to member-supplied timestamps.
Count of references That’s right, we count all the references for each record that includes them and add the total to the metadata.

After registration

Once registered, we check, report on and update metadata in a few ways.

Link checking We email each member a monthly Resolution Report with details of the number of failed and successful resolutions for their DOIs. If someone in the community reports a DOI that isn’t registered, we email the member a DOI Error Report.
Citation counts and matches Citation counts for records of members participating in our Cited-by service are openly available in our REST API. The matching citations themselves are available to members, for their own records only.
Title transfers Title, prefix and DOI transfers are common and require assistance from our team.
MemberID It’s not uncommon for members to have more than one prefix. The memberID means users of the REST API can query for records associated with all of a member’s prefixes.
Digital preservation We handle the infrequent but critical update of URLs that are necessary when titles are triggered for digital preservation. We also preserve the metadata itself, with both CLOCKSS and Portico.

Of course, since records are often redeposited with updates (note, deposit fees are only charged once per record), some of these processes on our side are repeated as necessary.

This list isn’t exhaustive and other needs and opportunities will emerge. For example, we are looking at matching to add ROR IDs, as we do for funderIDs, and doing some research into how we might determine and assert subject classifications at the work-level. If you’re interested in more about this kind of work, you’ll want to read this recent post by my Labs colleague Dominika on matching grants to outputs.

Get in touch if you have questions or for more information.

Follow the money, or how to link grants to research outputs

Dominika Tkaczyk — Tue, 22 Mar 2022 00:00:00 +0000

The ecosystem of scholarly metadata is filled with relationships between items of various types: a person authored a paper, a paper cites a book, a funder funded research. Those relationships are absolutely essential: an item without them is missing the most basic context about its structure, origin, and impact. No wonder that finding and exposing such relationships is considered very important by virtually all parties involved. Probably the most famous instance of this problem is finding citation links between research outputs. Lately, another instance has been drawing more and more attention: linking research outputs with grants used as their funding source. How can this be done and how many such links can we observe?

TL;DR

We looked for links between research outputs and grants registered with Crossref.
Grant DOIs alone are not enough for linking research outputs with grants, because the funding information in research outputs typically does not contain grant DOIs (yet). Award numbers alone are also not enough because they are not globally unique.
We used either grant DOIs (if available) or the combination of award number and funder information to match grants to research outputs.
In total, we found 20,834 links between research outputs and registered grants, involving 17,082 research outputs and 3,858 grants (10% of all registered grants)¹.
Erroneous and incomplete metadata, especially involving award numbers, is the main factor that prevents linking research outputs to grants.

Introduction

The ecosystem of scholarly metadata is filled with relationships between items of various types: a person authored a paper, an author works at a university, a paper cites a book, a book contains a chapter, a funder funded research. Those relationships are absolutely essential: an item without them is missing the most basic context about its structure, origin, and impact.

No wonder that finding and exposing relationships between items in the scientific ecosystem is considered very important by virtually all parties involved. Probably the most famous instance of this problem is finding citation links between research outputs. Another, relatively new example, is linking research outputs with grants used as their funding source.

At Crossref, for some time now we have been seeing a steady growth of funder membership and grant registration. We are aware that the possibility of finding relationships between grants and research outputs is a big reason why funders are registering grants with us in the first place. Being able to see which research outputs are being supported by which grants helps reduce the reporting burden on researchers, funders, and institutions alike, especially now with the addition of ROR IDs to help complete the picture. Exposing relationships between research outputs and grants also increases the transparency of funding sources of the research, making it easier to assess and trust scientific findings.

But how can we find those relationships and how many of them can we already observe? Thankfully our REST API, recently equipped with the grant metadata, can help us answer these questions.

The perfect scenario

Imagine a world where the metadata of any scientific output states all relationships with other items existing in the scientific ecosystem, and those related items are always referred to by their persistent identifiers, allowing all this information to be accessed in a fully machine-readable way… Lovely, right?

In the case of citations, in such a perfect world every bibliographic reference has a DOI of the cited item. And in the case of funding information, a scientific paper contains grant DOIs, stating the funded-by relationships between the paper and the grants.

But, as the last two years have painfully taught us all, life is not all rainbows and unicorns.

The reality kicks in

We know that around 71% of bibliographic references are deposited with Crossref without a DOI of the cited item. This means that if we want to establish citation links between items, we need to match the bibliographic references using the provided metadata, which is not a trivial task and can potentially introduce errors.

And the situation with the funding information and grant DOIs is even worse.

Problem #1: our schema does not allow the publishers to attach grant DOIs to research outputs

This issue is 100% on us. Because grant DOIs are relatively new, our deposit schema does not yet allow to specify the grant DOI in the funding information of a research output, even if the publisher wanted to. We are working on changing this.

Interestingly, it looks like persistent identifiers always find a way. Within over 7.4 million research outputs with funding information, we noticed 6 cases where a grant DOI was provided as an award number. For example in 10.1093/nar/gkaa994 we have the following:

funder: [
{
name: "Wellcome Trust",
award: ["10.35802/108758"],
doi-asserted-by: "publisher",
DOI: "10.13039/100010269"
}, ...
]

This may not be 100% correct from the schema perspective, but it is very useful when one is interested in linking grants to research outputs!

But those cases are extremely rare outliers. For the vast majority of the outputs, grant DOIs are not present in the metadata. This means that, just like in the case of bibliographic references, we have to use the metadata to match funding information to grants.

Funding information is typically given as a pair: award number, funder information. Grants contain similar metadata. One might be tempted to use only the award number for linking, as in some cases it can look like a grant identifier.

Let’s consider an example. We want to find all papers funded by grant 10.37807/gbmf7622. The award number is GBMF7622. A simple approach might be to search for items with this award number in Crossref’s REST API, which returns 12 results². However, one of the resulting items is the grant itself³. So excluding that, it seems like there are 12-1=11 research outputs funded by this grant.

Simple and easy, right? Well, think again.

Problem #2: award numbers are not unique

Let’s look at another example grant: 10.25585/60000600. Its award number is 2817 and the funder is the US Department of Energy.

When we search for this award we get 10 results⁴. Like before, one of them is our grant. After examining the remaining 9 we will see that:

3 items have been funded by the Joint Genome Institute, which according to the Funder Registry has been incorporated into Basic Energy Sciences, which is a descendant of the US Department of Energy
2 items have been funded by International Rett Syndrome Foundation from the US
2 items have been funded by Agencia Nacional de Promoción Científica y Tecnológica from Argentina
1 item has been funded by Arak University of Medical Sciences from Iran
1 item has been funded by Shahrekord University also from Iran

So among only 9 items mentioning the same award number we have in fact 5 different grants. Our input grant should probably be linked only to the three items mentioning Joint Genome Institute. The main problem illustrated here is that the award numbers are not globally unique, and thus should not be treated like identifiers.

Indeed, within 38,326 grants registered so far, we have 37,608 distinct award numbers, and among those, there are 716 award numbers, each of which appears in multiple grants. This issue comes in two flavours: conflicts between and within funders.

Between-funder award number conflicts

A conflict between funders is when more than one funder uses the same award number for one of their grants. This is expected - award numbers are assigned by funders internally and are not designed to be a globally unique identifier.

Out of 716 award numbers that appear in multiple grants, 12 are numbers that appear in grants of different funders. For example, there are two grants with the award number 105626:

Systemic MFG-E8 Blockade as Melanoma Therapy funded by Melanoma Research Alliance
Institutional Strategic Support Fund Phase2 FY2014/16 funded by Wellcome Trust

Because of those conflicts, we cannot simply rely on the award numbers for linking grants to research outputs. Instead, we have to use more information to be sure that the links are correctly established.

Within-funder award number conflicts

To our big surprise, it turns out that the majority of the award number conflicts happen not between different funders, but within the grants of a single funder. Out of 716 award numbers that appear in multiple grants, 704 appear in multiple grants of a single funder only. Such situations are not expected and could indicate an error or some other systematic issue with the data.

Interestingly, out of those 704 award numbers, 700 are associated with the US Department of Energy. We’ve followed up with them in order to clarify or resolve this. The US Department of Energy pointed out a fundamental issue with the data model: currently a grant deposited with Crossref has to have at least one funder DOI, and no other way of identifying the associated organisation is allowed. At the same time, some of the facilities that should appear in their grants’ metadata are not funders at all and thus cannot be identified by a funder DOI. In the future, they plan to identify those facilities in their grant metadata by providing ROR IDs.

Because of within-funder award number conflicts, in some cases it might be difficult to distinguish between two grants with the same award number and funder. A solution might be to use additional information or simply not accept any links if a research output cannot be reliably linked to one grant only.

Our linking approach

Based on all those observations, we adopted the following approach:

We iterated over all registered grants, for each we performed the following steps:
- We used award.number:<grant DOI> filter in the REST API to find all items listing a given grant’s DOI as the award number. Because this is based on the grant’s persistent identifier, we recorded those links without any further verification.
- We used the award.number:<grant award number> filter in the REST API to find all items listing grant’s award number in the funding information. Each resulting item was then verified by comparing the funder information in the item to the funder information in the grant. We recorded the link between the grant and the candidate item only if the verification succeeded.
In the final step, we examined all recorded links to make sure that each pair (research output, award number) is linked to at most one grant. Links violating this rule were flagged as not reliable.

We used different techniques to verify the funder information between the research output (item) and the grant, depending on what information is available. Grants always have the funder DOI. The item, however, can have the funder DOI, the funder name, or both.

If the funder DOI was available on both sides, the following rules were used for the funder verification (ordered by decreasing confidence):

Both the item and the grant contain the same funder DOI, for example, 10.35802/089928 and 10.1242/jcs.196758
The funder in the item replaced or was replaced by the funder in the grant (according to the Funder Registry), for example, 10.35802/104848 and 10.1136/medethics-2020-106821
The funder in the paper is an ancestor or a descendant of the funder in the grant (according to the Funder Registry), for example, 10.46936/sthm.proj.2010.40084/60004575 and 10.1016/j.heliyon.2018.e00629

If the funder DOI was not available in the item, the following rules were used for the funder verification (ordered by decreasing confidence):

The funder name in the paper is the same (ignoring the case) as the funder name in the grant, for example, 10.35802/110166 and 10.12688/wellcomeopenres.14645.4
The funder name in the item is the same (ignoring the case) as the name of the funder that replaced/was replaced by the funder in the grant, for example, 10.35802/206194 and 10.1172/jci.insight.96381
The funder name in the item is the same (ignoring the case) as the name of the ancestor/descendant of the funder in the grant, for example, 10.46936/cpbl.proj.2001.2191/60002922 and 10.1109/tkde.2016.2628180

Note that this is in fact very similar to our reference matching approach. In both cases, first we search for candidate items, and then verify the candidates by comparing the metadata. The actual metadata used for the verification varies, because different information is typically given in the bibliographic reference and the funding information.

What we found

This procedure applied to the entire Crossref dataset resulted in 20,846 links between research outputs and grants⁵. Of those, 12 were flagged as unreliable, because they involved more than one grant linked to the same item and award number. The rest of this section focuses on the remaining 20,834 links.

Within the 20,834 links, we have 17,082 research outputs and 3,858 (10.1%) grants.

Here is the breakdown into the verification approaches used:

Verification	#links	%links
The item contains grant DOI - no verification	6	<0.1%
Funder DOIs are the same	8,364	40.1%
Funder DOIs are related with a replaced/was replaced by relationship	3,704	17.8%
Funder DOIs are related with an ancestor/descendant relationship	7,718	37.0%
Funder names are the same	591	2.8%
The name of the funder in the item is the same as the name of the funder that replaced/was replaced by the funder in the grant	364	1.7%
The name of the funder in the item is the same as the name of the ancestor or descendant of the funder in the grant	87	0.4%

In most cases, just using the funder DOIs for the verification was enough. Verifying by the funder name added 1,042 links, which is 5% of all links.

And here are statistics for individual funders. Only funders with at least 10 deposited grants are listed in the table. The table shows the number of detected links, the number of distinct research outputs linked, the total number of outputs mentioning the given funder DOI, and the number of grants.

Funder	#links	#linked research outputs	#total outputs with funder DOI	#grants
Japan Science and Technology Agency	11,922	10,411	25,779	9,383
Wellcome Trust (including both funder DOIs 10.13039/100004440 and 10.13039/100010269)	8,001	6,246	49,492	17,534
James S. McDonnell Foundation	463	457	2,534	557
Melanoma Research Alliance	152	150	894	392
Asia-Pacific Network for Global Change Research	100	100	838	539
ALS Association	84	78	909	434
U.S. Department of Energy	56	52	97,482	8,462
Gordon and Betty Moore Foundation	51	50	5,928	94
American Cancer Society	3	3	7,276	107
Children’s Tumor Foundation	1	1	759	630
American Parkinson Disease Association	0	0	181	12
Neurofibromatosis Therapeutic Acceleration Program	0	0	101	68
International Anesthesia Research Society	0	0	94	34
Australian National Data Service	0	0	92	67

Note that the fourth column reports the total number of outputs registered with Crossref and mentioning the given funder DOI, including grants, journal papers and all other record types.

It is interesting to compare the number of linked research outputs for a given funder with the total number of research outputs mentioning a given funder DOI. In general, for a funder that registers grants, the more research outputs mentioning this funder, the more links we should be able to find.

And for some funders (Japan Science and Technology Agency, Melanoma Research Alliance, Asia-Pacific Network for Global Change Research, Wellcome Trust, James S. McDonnell Foundation), the number of linked outputs is indeed high, as compared with how many outputs mention the funder in the first place. This suggests our procedure was quite successful in linking outputs funded by these funders, meaning that in general the metadata in their grants and the funding information in the research outputs match.

On the other hand, we have a few funders for which we managed to link only a very small fraction of research outputs. There are several potential explanations here. A simple one is that not all relevant grants have been deposited yet. For example, a funder might be registering new grants only, whereas many research outputs mention older, not yet registered grants. It is also possible that there are systematic differences in how the publishers deposit the funding information in articles and other outputs, and how it is given in grants. Such differences might prevent us from establishing links, contributing to the overall low percentage of linked grants.

The importance of being precise

Here are some examples of existing links that should’ve been found, but were not.

The award number in grant 10.48105/pc.gr.93156 is CTF-2020-01-004. This article: 10.3390/ijms22094716 mentions award number 2020‐01‐004 and the same funder (Children’s Tumor Foundation). It is very probable that this is the same grant, but our procedure expects exactly the same award number, and so the two were not linked.

Paper 10.1128/genomea.00159-18 contains award number 1931 and U.S. Department of Energy as the funder. There are two grants with the same award number and funder: 10.46936/10.25585/60001053 and 10.46936/genr.proj.2000.1931/60002530. It is difficult to choose between them, and these links were marked as unreliable.

These examples could be signs of systematic errors and/or discrepancies that effectively prevent linking of those funders’ grants.

What’s next

In problems such as linking grants to research outputs, there are typically two key ingredients of the success, which at the same time are the main areas of improvement: the quality of the metadata, and the strength of the linking approach.

The metadata could be improved greatly by addressing existing discrepancies between grants and research outputs and allowing (and encouraging!) the publishers to provide grant DOIs in the funding information. Thankfully, we are not alone in those efforts. Both this recent Upstream blog from Alexis-Michel Mugabushaka, and this Scholarly Kitchen post from Robert Harrington call for the development and adoption of grant DOIs in scholarly metadata.

In terms of the linking approach, there are some ideas that could be used to further improve the linking accuracy and completeness:

The verification by funder name could be fuzzy and allow for minor variations like typos or additional words.
Apart from replaced/replaced by and ancestor/descendant, there are other relationships between funders in the Funder Registry: continuation of, incorporates/incorporated into, merged with, renamed as, split into/split from. We could also consider those relationships during the funder validation.
Apart from the funder information, there is other information that could be potentially used for verification, for example, the names of the authors and the investigators, the domain, or keywords.

If you have any questions, do get in touch!

All numbers are as of March 8, 2022 ↩︎
https://api-crossref-org.pluma.sjfc.edu/works?filter=award.number:gbmf7622 ↩︎
https://api-crossref-org.pluma.sjfc.edu/works?filter=award.number:gbmf7622,type:grant ↩︎
https://api-crossref-org.pluma.sjfc.edu/works?filter=award.number:2817 ↩︎
The code and data available here: https://gitlab.com/crossref/labs_data_analyses/-/tree/master/analyses/22-01-26-grants-matching ↩︎

Fast, citable feedback: Peer reviews for preprints and other record types

Martyn Rittman — Wed, 09 Dec 2020 00:00:00 +0000

Crossref has supported depositing metadata for preprints since 2016 and peer reviews since 2018. Now we are putting the two together, in fact we will permit peer reviews to be registered for any record type.

Currently, peer reviews can be registered for journal articles, but that means that they can only be related to some of the content our members deposit. Preprints, books, chapters, working papers, dissertations, and a host of other works can also be registered with Crossref. A number of these frequently undergo some form of review and many of our members and voices in the community have called for us to widen the net on peer reviews, including journal publishers, book publishers, review platforms, and preprint servers. We’ve listened and taken action, and from now on Crossref members can add relationship metadata that links peer reviews to any record type. The metadata will also contain the type of review, stating whether it is a referee report, author response, or community comment, etc. This allows accurate reporting on whether the peer review is happening within a traditional editorial process or elsewhere.

Reviews for preprints

In the last decade there has been an increase in the number of disciplines using preprints. Since enabling registration of preprint metadata, it has become our fastest-growing record type. Preprints, working papers, and other forms of early publication help to accelerate dissemination of the latest research and discovery. They can also promote discussion on important topics, and help authors to improve papers before an editorial decision for journal publication. During the COVID-19 pandemic, preprints have become invaluable for speeding the publication of vital research and case studies.

On the other hand, preprints do not undergo formal review and editorial approval, leading to concerns about the dissemination of false information. While the issue of misinformation in preprints has been discussed for some time, the COVID-19 pandemic has brought it more sharply into focus. organisations that post preprints need to balance the benefits of rapid dissemination with promoting their responsible use.

To support the feedback process, preprint servers along with a growing number of other platforms and services offer scholars the opportunity to post public comments on preprints. By doing this, they give extra context for readers, provide suggestions for authors, and raise awareness of work that could be flawed or too preliminary.

Another growing trend is journal publishers adopting editorial processes that involve preprint-first options and open peer review. As Dr. Stephanie Dawson from ScienceOpen says:

“We have long believed in rewarding reviewers by assigning Crossref DOIs to their open reviews to make them citable objects and we were one of the first users of Crossref’s peer review schema. However, a large percentage of the articles reviewed on ScienceOpen are publicly available preprints. The UCL Open: Environment journal hosted on the platform, for example, is based on a workflow of open peer review of preprints. Our customers, editors, reviewers and authors are therefore extremely happy that these reviews can now also be assigned a Crossref peer review DOI for more accountability and transparency in scholarly publishing.”

At Crossref, we’re continually looking to support more record types and relations between them to build trust, support reproducibility and increase discoverability of content. This is another small step in building the research nexus and we look forward to working with members depositing peer reviews of preprints.

What if I told you that bibliographic references can be structured?

Dominika Tkaczyk — Mon, 08 Jul 2019 00:00:00 +0000

Last year I spent several weeks studying how to automatically match unstructured references to DOIs (you can read about these experiments in my previous blog posts). But what about references that are not in the form of an unstructured string, but rather a structured collection of metadata fields? Are we matching them, and how? Let’s find out.

TL;DR

43% of open/limited references deposited with Crossref have no publisher-asserted DOI and no unstructured string. This means they need a matching approach suitable for structured references. [EDIT 6th June 2022 - all references are now open by default].
I adapted our new matching algorithms: Search-Based Matching (SBM) and Search-Based Matching with Validation (SMBV) to work with both structured and unstructured references.
I compared three matching algorithms: Crossref’s current (legacy) algorithm, SBM and SBMV, using a dataset of 2,000 structured references randomly chosen from Crossref’s references.
SBMV and the legacy algorithm performed almost the same. SBMV’s F1 was slightly better (0.9660 vs. 0.9593).
Similarly as in the case of unstructured references, SBMV achieved slightly lower precision and better recall than the legacy algorithm.

Introduction

Those of you who often read scientific papers are probably used to bibliographic references in the form of unstructured strings, as they appear in the bibliography, for example:

[5] Elizabeth Lundberg, “Humanism on Gallifrey,” Science Fiction Studies, vol. 40, no. 2, p. 382, 2013.

This form, however, is not the only way we can store the information about the referenced paper. An alternative is a structured, more machine-readable form, for example using BibTeX format:

@article{Elizabeth_Lundberg_2013,
year = 2013,
publisher = {{SF}-{TH}, Inc.},
volume = {40},
number = {2},
pages = {382},
author = {Elizabeth Lundberg},
title = {Humanism on Gallifrey},
journal = {Science Fiction Studies}
}

Probably the most concise way to provide the information about the referenced document is to use its identifier, for example (🥁drum roll…) the DOI:

<https://doi-org.pluma.sjfc.edu/10.5621/sciefictstud.40.2.0382>

It is important to understand that these three representations (DOI, structured reference and unstructured reference) are not equivalent. The amount of information they carry varies:

The DOI, by definition, provides the full information about the referenced document, because it identifies it without a doubt. Even though the metadata and content are not directly present in the DOI string, they can be easily and deterministically accessed. It is by far the preferred representation of the referenced document.
The structured reference contains the metadata of the referenced object, but it doesn’t identify the referenced object without a doubt. In our example, we know that the paper was published in 2013 by Elizabeth Lundberg, but we might not know exactly which paper it is, especially if there are more than one document with the same or similar metadata.
The unstructured reference contains the metadata field values, but without the names of the fields. This also doesn’t identify the referenced document, and even its metadata is not known without a doubt. In our example, we know that the word “Science” appears somewhere in the metadata, but we don’t know for sure whether it is a part of the title, journal title, or maybe the author’s (very cool) name.

The diagram presents the relationships between all these three forms:

The arrows show actions that Crossref has to perform to transform one form to another.

Green transformations are in general easy and can be done without introducing any errors. The reason is that green arrows go from more information to less information. We all know how easy it is to forget important stuff!

Green transformations are typically performed when the publication is being created. At the beginning the author can access the DOI of the referenced document, because they know exactly which document it is. Then, they can extract the bibliographic metadata (the structured form) of the document based on the DOI, for example by following the DOI to the document’s webpage or retrieving the metadata from Crossref’s REST API. Finally, the structured form can be formatted into an unstructured string using, for example, the CiteProc tool.

We’ve also automated it further and these two green transformation (getting the document’s metadata based on the DOI and formatting it into a string) can be done in one go using Crossref’s content negotiation.

Red transformations are often done in systems that store bibliographic metadata (like our own metadata collection), often at a large scale. In these systems, we typically want to have DOIs (or other unique identifiers) of the referenced documents, but in practise we often have only structured and/or unstructured form. To fix this, we match references. Some systems also perform reference parsing (thankfully, we discovered we do not need to do this in our case).

In general, red transformations are difficult, because we have to go from less information to more information, effectively recreating the information that has been lost during paper writing. This requires a bit of reasoning, educated guessing, and juggling probabilities. Data errors, noise, and sparsity make the situation even more dire. As a result, we do not expect any matching or parsing algorithm to be always correct. Instead, we perform evaluations (like in this blog post) to capture how well they perform on average.

My previous blog post focused on matching unstructured references to DOIs (long red “matching” arrow). In this one, I analyse how well we can match structured references to DOIs (short red “matching” arrow).

References in Crossref

You might be asking yourself how important it is to have the matching algorithm working for both structured and unstructured references. Let’s look more closely at the references our matching algorithm has to deal with.

29% of open/limited references deposited with Crossref already have the DOI provided by the publisher member. At Crossref, when we come across those references, we start dancing on a rainbow to the tunes of Linkin Park, while the references holding their DOIs sprinkle from the sky. Some of us sing along. We live for those moments, so if you care about us, please provide as many DOIs in your references as possible!

You might be wondering how we are sure these publisher-provided DOIs are correct. The short answer is that we are not. After all, the publisher might have used an automated matcher to insert the DOIs before depositing the metadata. Nevertheless, our current workflow assumes these publisher-provided DOIs are correct and we simply accept them as they are.

Unfortunately, the remaining 71% of references are deposited without a DOI. Those are the references we try to match ourselves.

Here is the distribution of all the open/limited references:

17% of the references are deposited with no DOI and both structured and unstructured form. 11% have no DOI and only an unstructured form, and 43% have no DOI and only a structured form. These 43% cannot be directly processed by the unstructured matching algorithm.

This distribution clearly shows that we need a matching algorithm able to process both structured and unstructured references. If our algorithm worked only with one type, we would miss a large percentage of the input references, and the quality of our citation metadata would be questionable.

The analysis

Let’s get to the point. I evaluated and compared three matching algorithms, focusing on the structured references.

The first algorithm is one of the legacy algorithms currently used in Crossref. It uses fuzzy querying in a relational database to find the best matching DOI for the given structured reference. It can be accessed through a Crossref OpenURL query.

The second algorithm is an adaptation of the Search-Based Matching (SBM) algorithm for structured references. In this algorithm, we concatenate all metadata fields of the reference and use it to search in the Crossref’s REST API. The first hit is returned as the target DOI if its relevance score exceeds the predefined threshold.

The third algorithm is an adaptation of the Search-Based Matching with Validation (SBMV) for structured references. Similarly as in the case of SBM, we also concatenate all metadata fields of the input reference and use it to search in the Crossref’s REST API. Next, a number of top hits are considered as candidates and their similarity score with the input reference is calculated. The candidate with the highest similarity score is returned as the target DOI if its score exceeds the predefined threshold. The similarity score is based on fuzzy comparison of the metadata field values between the candidate and the input reference.

I compared these three algorithms on a test set composed of 2,000 structured bibliographic references randomly chosen from Crossref’s metadata. For each reference, I manually checked the output of all matching algorithms, and in some cases performed additional manual searching. This resulted in the true target DOI (or null) assigned to each reference.

The metrics are the same as in the previous evaluations: precision, recall and F1 calculated over the set of input references.

The thresholds for SBM and SBMV algorithms were chosen on a separate validation dataset. The validation dataset also contains 2,000 structured references with manually-verified target DOIs.

The results

The plot shows the results of the evaluation of all three algorithms:

The vertical black lines on top of the bars represent the confidence intervals.

As we can see, SBMV and the legacy approach achieved very similar results. SBMV slightly outperforms the legacy approach in F1: 0.9660 vs. 0.9593.

SBMV is slightly worse that the legacy approach in precision (0.9831 vs. 0.9929) and better in recall (0.9495 vs. 0.9280).

The SBM algorithm performs the worst, especially in precision. Why is there such a huge difference between SBM and SBMV? The algorithms differ in the post-processing validation stage. SBM relies on the ability of the search engine to select the best target DOI, while SBMV re-scores a number of candidates obtained from the search engine using custom similarity. The results here suggest that in the case of structured references, the right target DOI is usually somewhere close to the top of the search results, but often it is not in the first position. One of the reasons might be missing titles in 76% of the structured references, which can confuse the search engine.

Let’s look more closely at a few interesting cases in our test set:

first-page = 1000
article-title = Sequence capture using PCR-generated probes: a cost-effective method of targeted high-throughput sequencing for nonmodel organisms
volume = 14
author = Peñalba
year = 2014
journal-title = Molecular Ecology Resources

The reference above was successfully matched by SBMV to https://doi-org.pluma.sjfc.edu/10.1111/1755-0998.12249, even though the document’s volume and pages are missing from Crossref’s metadata.

issue = 2
first-page = 101
volume = 6
author = Abraham
year = 1987
journal-title = Promoter: An Automated Promotion Evaluation System

Here the structure incorrectly labels article title as journal title. Despite this, the reference was correctly matched by our brave SBMV to https://doi-org.pluma.sjfc.edu/10.1287/mksc.6.2.101.

author = Marshall Day C.
volume = 39
first-page = 572
year = 1949
journal-title = India. J. A. D. A.

Above we have most likely a parsing error. A part of the article title appears in the journal name, and the main journal name is abbreviated. ‘I see what you did there, my old friend Parsing Algorithm! Only a minor obstacle!’ said SBMV, and matched the reference to https://doi-org.pluma.sjfc.edu/10.14219/jada.archive.1949.0114.

volume = 5
year = 2015
article-title = A retrospective analysis of the effect of discussion in teleconference and face-to-face scientific peer-review panels
journal-title = BMJ Open

Here the the page number and author are not in the structure, but our invincible SBMV jumped over the holes left by the missing metadata and gracefully grabbed the right DOI https://doi-org.pluma.sjfc.edu/10.1136/bmjopen-2015-009138.

issue = 2
first-page = 533
volume = 30
author = Uthman BM
year = 1989
journal-title = Epilepsia

In this case we have a mismatch in the page number (“533” vs. “S33”). But did SBMV give up and burst into tears? I think we already know the answer! Of course, it conquered the nasty typo with the sword made of fuzzy comparisons (yes, it’s a thing!) and brought us back the correct DOI https://doi-org.pluma.sjfc.edu/10.1111/j.1528-1157.1989.tb05823.x.

Structured vs. unstructured

How does matching structured references compare to matching unstructured references?

The general trends are the same. For both structured and unstructured references, SBMV outperforms the legacy approach in F1, achieving worse precision and better recall. This tells us that our legacy algorithms are more strict and as a result they miss some links.

Structured reference matching seems easier than unstructured reference matching. The reason is that when we have the structure, we can compare the input reference to the candidate field by field, which is more precise than using the unstructured string.

Structured matching, however, in practise brings new challenges. One big problem is data sparsity. 15% of structured references without DOIs have fewer than four metadata fields. This is not always enough to identify the DOI. Also, 76% of the structured references without DOIs do not contain the article title, which poses a problem for candidate selection using the search engine.

What’s next?

So far, I have focused on evaluating SBMV for unstructured and structured references separately. 17% of the open/limited references at Crossref, however, have both unstructured and structured form. In those cases, it might be beneficial to use the information from both forms. I plan to perform some experiments on this soon.

The data and code for this evaluation can be found at https://github.com/CrossRef/reference-matching-evaluation. The Java version of SBMV (for both structured and unstructured references) can be found at https://gitlab.com/crossref/search-based-reference-matcher.

A simpler text query form

Isaac Farley — Tue, 30 Apr 2019 00:00:00 +0000

The Simple Text Query form (STQ) allows users to retrieve existing DOIs for journal articles, books, and chapters by cutting and pasting a reference or reference list into a simple query box. For years the service has been heavily used by students, editors, researchers, and publishers eager to match and link references.

We had changes to the service planned for the first half of this year - an upgraded reference matching algorithm, a more modern interface, etc. In the spirit of openness and transparency, part of our project plan was to communicate these pending changes to STQ users well in advance of our 30 April completion date. What would users think? Could they help us improve upon our plans?

About a month ago, I reached out to the 21,000 plus users we had on record of using STQ since January 2018. We received nearly 85 responses from the messages we sent. Questions ranged from: if we were making changes, would PubMed ID matching be supported? To: What about the reliability of the returned reference links? And: Could we better accommodate larger reference lists?

Many of the users we heard from told us how STQ was critical to their work. I read all these messages. The concerns raised by users were legitimate and much appreciated. We reassessed our project timeline and plans, and decided to shift course. So, what are we doing?

What’s changing?

The previous hurdle of having to register your email address simply to return reference links was confusing and unnecessary. We removed it.
We previously limited the number of monthly reference links to 5,000 per email address. Most didn’t reach the limit, but those who did were frustrated by it and/or found ways around it. We want you to match and register as many references as possible, so we removed the monthly limit too.
Many of you with long reference lists found that you were occasionally reaching our limit of 30,000 characters per submission. Once again, we want you to match and register as many references as possible so we removed the character limit altogether and instead are just looking at the number of references per submission. We now provide space for 1,000 references per submission (We checked. The most references we have ever received via the STQ form in one submission was around 750. Thus, we rounded up.).
We did make a change to the backend of the service. We updated the algorithm we use to return reference links. We think it’s an improvement. Let us know how you find it.

What’s remaining the same?

Core functionality. It’s all in the name. Retrieve DOIs for journal articles, books, and chapters by cutting and pasting a reference or reference list into a simple query box.
PubMed ID matching. You use it. You need it. We’re keeping it.
Deposits. You’ll still need an email address for this, but we won’t ask for it until you’re at the deposit screen.
The interface. We’re still eager to give the user interface a much-needed refresh, but, as many users pointed out to us, there’s still some core functionality that’s important that we need to retain with any interface update. For instance, you need to be able to easily copy and paste reference links into your reference list. That functionality isn’t going anywhere.
Resetting reference links. Submit references, match, reset, and repeat. Many users like the reset button. It’s not going anywhere either.

XML queries

The change to the backend of the service that I mentioned above is not confined to reference matching and depositing for STQ users. XML queries for reference matching are also now powered by that new backend. We think it’s a seamless transition, but if you find it is not, please let us know.

I’m excited for these changes and hope you are too. I invite you to try the simpler and improved STQ form, and let us know what you think.

Reference matching: for real this time

Dominika Tkaczyk — Tue, 18 Dec 2018 00:00:00 +0000

In my previous blog post, Matchmaker, matchmaker, make me a match, I compared four approaches for reference matching. The comparison was done using a dataset composed of automatically-generated reference strings. Now it’s time for the matching algorithms to face the real enemy: the unstructured reference strings deposited with Crossref by some members. Are the matching algorithms ready for this challenge? Which algorithm will prove worthy of becoming the guardian of the mighty citation network? Buckle up and enjoy our second matching battle!

TL;DR

I evaluated and compared four reference matching approaches: the legacy approach based on reference parsing, and three variants of search-based matching.
The dataset comprises 2,000 unstructured reference strings from the Crossref metadata.
The metrics are precision and recall calculated over the citation links. I also use F1 as a standard single-number metric that combines precision and recall, weighing them equally.
The best variant of search-based matching outperforms the legacy approach in F1 (96.3% vs. 92.5%), with the precision worse by only 0.9% (98.09% vs. 98.95%), and the recall better by 8.9% (94.56% vs. 86.85%).
Common causes of SBMV’s errors are: incomplete/erroneous metadata of the target documents, and noise in the reference strings.
The results reported here generalize to the subset of references in Crossref that are deposited without the target DOI and are present in the form of unstructured strings.

Introduction

In reference matching, we try to find the DOI of the document referenced by a given input reference. The input reference can have a structured form (a collection of metadata fields) and/or an unstructured form (a string formatted in a certain citation style).

In my previous blog post, I used reference strings generated automatically to compare four matching algorithms: Crossref’s legacy approach based on reference parsing and three variations of search-based matching. The best algorithm turned out to be Search-Based Matching with Validation (SBMV). SBMV uses our REST API’s bibliographic search function to select the candidate target documents, and a separate validation-scoring procedure to choose the final target document. The legacy approach and SBMV achieved very similar average precision, and SBMV was much better in average recall.

This comparison had important limitations, which affect the interpretation of these results.

First of all, the reference strings in the dataset might be too perfect. Since they were generated automatically from the Crossref metadata records, any piece of information present in the string, such as the title or the name of the author, will exactly match the information in Crossref’s metadata. In such a case, a matcher comparing the string against the record can simply apply exact matching and everything should be fine.

In real life, however, we should expect all sorts of errors and noise in the reference strings. For example, a string might have been manually typed by a human, so it can have typos. The string might have been scraped from the PDF file, in which case it could have unusual unicode characters, ligatures or missing and extra spaces. A string can also have typical OCR errors, if it was extracted from a scan.

These problems are typical for messy real-life data, and our matching algorithms should be robust enough to handle them. However, when we evaluate and compare approaches using the perfect reference strings, the results won’t tell us how well the algorithms handle harder, noisy cases. After all, even if you repeatedly win chess games against your father, it doesn’t mean you will likely defeat Garry Kasparov (unless, of course, you are Garry Kasparov’s child, in which case, please pass on our regards to your dad!).

Even though I attempted to make the data more similar to the noisy real-life data by simulating some of the possible errors (typos, missing/extra spaces) in two styles, this might not be enough. We simply don’t know the typical distribution of the errors, or even what all the possible errors are, so our data was probably still far from the real, noisy reference strings.

The differences in the distributions are a second major issue with the previous experiment. To build the dataset, I used a random sample from Crossref metadata, so the distribution of the cited item types (journal paper, conference proceeding, book chapter, etc.) reflects the overall distribution in our collection. However, the distribution in real life might be different if, for example, journal papers are on average cited more often than conference proceedings.

Similarly, the distribution of the citation styles is most likely different. To generate the reference strings, I used 11 styles distributed uniformly, while the real distribution most likely contains more styles and is skewed.

All these issues can be summarized as: the data used in my previous experiment is different from the data our matching algorithms have to deal with in the production system. Why is this important? Because in such a case, the evaluation results do not reflect the real performance in our system, just like the child’s score on the math exam says nothing about their score on the history test. We can hope my previous results accurately showed the strengths and weaknesses of each algorithm, but the estimations could be far off.

So, can we do better? Sure!

This time, instead of automatically-generated reference strings, I will use real reference strings found in the Crossref metadata. This will give us a much better picture of the matching algorithms and their real-life performance.

Evaluation

This time the evaluation dataset is composed of 2,000 unstructured reference strings from the Crossref metadata, along with the target true DOIs. The dataset was prepared mostly manually:

First, I drew a random sample of 100,000 metadata records from the system.
Second, I iterated over all sampled items, and extracted those unstructured reference strings, that do not have the DOI provided by the member.
Next, I randomly sampled 2,000 reference strings.
Finally, I assigned a target DOI (or null) to each reference string. This was done by verifying DOIs returned by the algorithms and/or manual searching.

The metrics this time are based on the citation links. A citation link points from the reference (or the document containing the reference) to the referenced (target) document.

When we apply a matching algorithm to a set of reference strings in our collection, we get a set of citation links between our documents. I will call those citation links returned links.

On the other hand, in our collection we have real, true links between the documents. In the best-case scenario, the set of true links and the set of returned links are identical. But we don’t live in a perfect world and our matching algorithms make mistakes.

To measure how close the returned links are to the true links, I used precision, recall and F1. This time they are calculated over all citation links in the dataset. More specifically:

Precision is the fraction of the returned links that are correct. Precision answers the question: if I see a citation link A->B in the output of a matcher, how certain can I be that paper A actually cites paper B?
Recall is the percentage of true links that were returned by the algorithm. Recall answers the question: if paper A cites paper B and B is in the collection, how certain can I be that the matcher’s output contains the citation link A->B?
F1 is the harmonic mean of precision and recall.

In the previous experiment, I also used precision, recall and F1, but they were calculated for each target document and then averaged. This time precision, recall and F1 are not averaged but simply calculated over all citation links. This is a more natural approach, since now the dataset comprises isolated reference strings rather than target documents, and in practice each target document has at most one incoming reference.

I tested the same four approaches as before:

the legacy approach, based on reference parsing
SBM with a simple threshold, which searches for the reference string in the search engine and returns the first hit, if its relevance score exceeds the predefined threshold
SBM with a normalized threshold, which searches for the reference string in the search engine and returns the first hit, if its relevance score divided by the string length exceeds the predefined threshold
SBMV, which first applies SBM with a normalized threshold to select a number of candidate items, and a separate validation procedure is used to select the final target item

All the thresholds are parameters which have to be set prior to the matching. The thresholds used in the experiments were chosen using a separate dataset, as the values maximizing the F1 of each algorithm.

Results

The plot shows the overall results of all tested approaches:

The exact values are also given in the table (the best result for each metric is bolded):

	precision	recall	F1
legacy approach	0.9895	0.8685	0.9251
SBM (simple threshold)	0.8686	0.8191	0.8431
SBM (normalized threshold)	0.7712	0.9121	0.8358
SBMV	0.9809	0.9456	0.9629

As we can see, the legacy approach is the best in precision, slightly outperforming SBMV. In recall, SBMV is clearly the best, which also decided about its victory over the legacy approach in F1.

How do these results compare to the results from my previous blog post? The overall trends (the legacy approach slightly outperforms SBMV in precision, and SBMV outperforms the legacy approach in recall and F1) are the same. The most important differences are: 1) on the real dataset SBM without validation is worse than the legacy approach, and 2) this time the algorithms achieved much higher recall. These differences are most likely related to the difference in data distributions explained before.

SBMV’s strengths and weaknesses

Let’s look at a few example cases where SBMV successfully returned the correct DOI, while the legacy approach failed.

Lundqvist D, Flykt A, Ohman A: The Karolinska Directed Emotional Faces - KDEF, CD ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet. 1998

matched to https://doi-org.pluma.sjfc.edu/10.1037/t27732-000

The target item is a dataset, which means unusual metadata fields and an unusual reference string.

Schminck, A. , ‘The Beginnings and Origins of the “Macedonian” Dynasty’ in J. Burke and R. Scott , eds., Byzantine Macedonia: Identity, Image and History (Melbourne, 2000), 61–8.

matched to https://doi-org.pluma.sjfc.edu/10.1163/9789004344730_006

This is an example of a book chapter. The reference string contains special quotes and dash characters.

R. Schneider,On the Aleksandrov-Fenchel inequality, inDiscrete Geometry and Convexity (J. E. Goodman, E. Lutwak, J. Malkevitch and R. Pollack, eds.), Annals of the New York Academy of Sciences440 (1985), 132–141.

matched to https://doi-org.pluma.sjfc.edu/10.1111/j.1749-6632.1985.tb14547.x

In this case, spaces are missing in the reference string, which might be problematic for the parsing.

R. B. Husar andE. M. Sparrow, Int. J. Heat Mass Transfer11, 1206 (1968).

matched to https://doi-org.pluma.sjfc.edu/10.1016/0017-9310(68)90036-7

This is another example of a reference string with missing spaces.

F. Cappello, A. Geist, W. Gropp, S. Kale, B. Kramer, and M. Snir. Toward exascale resilience: 2014 update. Supercomputing frontiers and innovations, 1(1), 2014.

matched to https://doi-org.pluma.sjfc.edu/10.14529/jsfi140101

In this case authors are missing in the Crossref metadata.

Li KZ, Shen XT, Li HJ, Zhang SY, Feng T, Zhang LL. Ablation of the Carbon/carbon Composite Nozzle-throats in a Small Solid Rocket Motor[J]. Carbon, 2011, 49: 1 208–1 215

matched to https://doi-org.pluma.sjfc.edu/10.1016/j.carbon.2010.11.037

Here we have unexpected spaces inside page numbers.

N. Kaloper, A. Lawrence and L. Sorbo, An Ignoble Approach to Large Field Inflation, JCAP 03 (2011) 023 [ arXiv:1101.0026 ] [ INSPIRE ].

matched to https://doi-org.pluma.sjfc.edu/10.1088/1475-7516/2011/03/023

In this case we have an acronym of the journal name and additional arXiv id.

KrönerE. ?Stress space and strain space continuum mechanics?, Phys. Stat. Sol. (b), 144 (1987) 39?44.

matched to https://doi-org.pluma.sjfc.edu/10.1002/pssb.2221440104

This reference string has a missing space, a missing word in the title, and incorrectly encoded special characters.

Suyemoto K. L., (1998) The functions of self-mutilationClinical Psychology Review 18(5): 531–554

matched to https://doi-org.pluma.sjfc.edu/10.1016/s0272-7358(97)00105-0

In this case the space is missing between the title and the journal name.

Ono , N. 2011 Stable and fast update rules for independent vector analysis based on auxiliary function technique Proceedings of IEEE Workshop on Applications of Signal Processing to Audio and Acoustics 189 192

matched to https://doi-org.pluma.sjfc.edu/10.1109/aspaa.2011.6082320

The parsing can also have problems with missing punctuation, like in this case.

Hybertsen M.S., Witzigmann B., Alam M.A., Smith R.K. (2002) 1 113

matched to https://doi-org.pluma.sjfc.edu/10.1023/a:1020732215449

In this case both title and journal name are missing from the reference string.

We can see from these examples that SBMV is fairly robust and able to deal with a small amount of noise in the metadata and reference strings.

What about the errors SBMV made? From the perspective of citation links, we have two types of errors:

False positives: incorrect links returned by the algorithm.
False negatives: links that should have been returned but weren’t.

When we apply SBMV instead of the legacy approach, the fraction of false positives within the returned links increases from 1.05% to 1.91%, and the fraction of false negatives within the true links decreases from 13.15% to 5.44%. This means with SBMV:

1.91% of the links in the algorithm’s output are incorrect
5.44% of the true links are not returned by the algorithm

We can also classify all the references in the dataset into several categories, based on the values of true and returned DOIs:

We have the following categories:

References matched to correct DOIs (1129 cases, returned and true blue)
References correctly not matched to anything (791 cases, returned and true white)
References not matched to anything, when they should be (58 cases, returned white, true grey)
References matched to wrong DOIs (7 cases, returned red, true yellow)
References matched to something, when they shouldn’t be matched to anything (15 cases, returned black, true white)

Note that in terms of these categories, precision is equal to:

And recall is equal to:

What are the most common causes of SBMV’s errors?

Incomplete or incorrect Crossref metadata. Even a perfect reference string formatted in the most popular citation style will not be matched, if the target record in the Crossref collection has many missing or incorrect fields.
Similarly, missing or incorrect information in the reference string is very problematic for the matchers.
Errors/noise in the reference string, such as:
- HTML/XML markup not stripped from the string
- multiple references mixed in one string
- spacing issues and typos
In a few cases a document related to the real target was matched, such as the book instead of its chapter, or the conference proceedings paper instead of the thesis.

Limitations

The most important limitation is the size of the dataset. Every item had to be verified manually, which significantly limited the possibility of creating a large set and also using a lot of independent sets.

Finally, the numbers reported here still don’t reflect the overall precision and recall of the current links in the Crossref metadata. This is because:

we still use the legacy approach for matching,
some references are deposited along with the target DOIs and are not matched by Crossref, these links are not analyzed here, and
in Crossref we have both unstructured and structured references, and in this experiment only the unstructured ones were tested.

What’s next?

The next experiment will be related to the structured references. Similarly as here, I will try to estimate the performance of the search-based matching approach and compare it to the performance of the legacy approach.

The evaluation framework, evaluation data and experiments related to the reference matching are available in the repository https://github.com/CrossRef/reference-matching-evaluation. Future experiments will be added there as well.

https://github.com/CrossRef/reference-matching-evaluation also contains the Python implementation of the SBMV algorithm. The Java implementation of SBMV is available in the repository https://gitlab.com/crossref/search_based_reference_matcher.

Matchmaker, matchmaker, make me a match

Dominika Tkaczyk — Mon, 12 Nov 2018 00:00:00 +0000

Matching (or resolving) bibliographic references to target records in the collection is a crucial algorithm in the Crossref ecosystem. Automatic reference matching lets us discover citation relations in large document collections, calculate citation counts, H-indexes, impact factors, etc. At Crossref, we currently use a matching approach based on reference string parsing. Some time ago we realized there is a much simpler approach. And now it is finally battle time: which of the two approaches is better?

TL;DR

I evaluated and compared four approaches to reference matching: the legacy approach based on reference parsing, and three variants of the new idea called search-based matching.
A large automatically generated dataset was used for the experiments. It is composed of 7,374 metadata records from the Crossref collection, each of which was formatted automatically into reference strings using 11 citation styles.
The main metrics used for the evaluation are precision and recall. I also use F1 as a standard metric that combines precision and recall into a single number, weighing them equally. All values are calculated for each metadata record separately and averaged over the dataset.
In general, search-based matching is better than the legacy approach in F1 and recall, but worse in precision.
The best variant of search-based matching outperforms the legacy approach in average F1 (84.5% vs. 52.9%), with the average precision worse by only 0.1% (99.2% vs 99.3%), and the average recall better by 88% (79.0% vs. 42.0%).
The best variant of search-based matching also outperforms the legacy approach in average F1 for each one of the 11 styles.
A weak spot of the parsing-based approach is degraded/noisy reference strings, which do not appear to use any of the known citation styles.
A weak spot of search-based approach is short reference strings, and in particular citation styles that do not include the title in the reference string.

Introduction

In reference matching, on the input we have a bibliographic reference. It can have the form of an unstructured string, such as:

(1) Adamo, S. H.; Cain, M. S.; Mitroff, S. R. Psychological Science 2013, 24, 2569–2574.

The input can also have the form of a structured reference, such as (BibTex format):

 @article{adamo2013,
 author = {Stephen H. Adamo and Matthew S. Cain and Stephen R. Mitroff},
 title = {Self-Induced Attentional Blink: A Cause of Errors in Multiple-Target Search},
 journal = {Psychological Science},
 volume = {24},
 number = {12},
 pages = {2569-2574},
 year = {2013}
}

The goal of matching is to find the document, which the input reference points to.

Matching algorithms

Matching references is not a trivial task even for a human, not to mention the machines, which are still a bit less intelligent than us (or so they want us to believe…). A typical meta-approach to reference matching might be to score the similarity between the input reference and the candidate target documents. The document most similar to the input is then returned as the target.

Of course, still a lot can go wrong here. We can have more than one potential target record with the same score (which one do we choose?). We can have only documents with low to medium scores (is the actual target even present in our collection?). We can also have errors in the input string (are the similarity scores robust enough?). Life’s tough!

The main difference between various matching algorithms is in fact how the similarity is calculated. For example, one idea might be to compare the records field by field (how similar is the title/author/journal in the input reference to the title/author/journal of our candidate target record?). This is roughly how the matching works currently at Crossref.

The main problem with this approach is that it requires a structured reference, and in practise, often all we have is a plain reference string. In such a case we need to extract the metadata fields from the reference string (this is called parsing). Parsing introduces errors, since no parser is omniscient. The errors propagate further and affect the scoring… you get the picture.

Luckily, as we have known for some time now, this is not the only approach. Instead of comparing structured objects, we could calculate the similarity between them using their unstructured textual form. This effectively eliminates the need for parsing, since the unstructured form is either already available on the input or can be easily generated from the structured form.

What about the similarity scores? We already know a powerful method for scoring the similarities between texts. Those are (you guessed it!) scoring algorithms used by search engines. Most of them, including Crossref’s, do not need a structured representation of the object, they are perfectly happy with just a plain text query.

So all we need to do is to pass the original reference string (or some concatenation of the reference fields, if only a structured reference is available) to the search engine and let it score the similarity for us. It will also conveniently sort the results so that it is easy to find the top hit.

Evaluation

So far so good. But which strategy is better? Is it better to develop an accurate parser, or just rely on the search engine? I don’t feel like guessing. Let’s try to answer this using (data) science. But first, we need to decompose our question into smaller pieces.

Question 1. How can I measure the quality of a reference matcher?

Generally speaking, this can be done by checking the resulting citation links. Simply put, the better the links, the better the matching approach must have been.

A few standard metrics can be applied here, including accuracy, precision, recall and F1. We decided to calculate precision, recall and F1 separately for each document in the dataset, and then average those numbers over the entire dataset.

When I say “documents”, I really mean “target documents”:

precision for a document X tells us, what percentage of links to X in the system are correct
recall for a document X tells us, what percentage of true links to X are present in the system
F1 is the harmonic mean of precision and recall

F1 is a single-number metric combining precision and recall. In F1 precision and recall are weighted equally. It is also possible to combine precision and recall using different weights, to place more emphasis on one of those metrics.

We decided to look at links from the target document’s perspective, because this is what the academic world cares about (i.e. how accurate the citation counts of academic papers are).

Calculating separate numbers for individual documents and averaging them within a dataset is the best way to have reliable confidence intervals (which makes the whole analysis look much smarter!).

Question 2. Which approaches should be compared?

In total we tested four reference matching approaches.

The first approach, called the legacy approach, is the approach currently used in Crossref ecosystem. It uses a parser and matches the extracted metadata fields against the records in the collection.

The second approach is the search-based matching (SBM) with a simple threshold. It queries the search engine using the reference string and returns the top hit from the results, if its relevance score exceeds the threshold.

The third approach is the search-based matching (SBM) with a normalized threshold. Similarly as in the simplest SBM, in this approach we query the search engine using the reference string. In this case the first hit is returned if its normalized score (the score divided by the reference length) exceeds the threshold.

Finally, the fourth approach is a variation of the search based matching, called search-based matching with validation (SBMV). In this algorithm we use additional validation procedure on top of SBM. First, SBM with a normalized threshold is applied and the search results with the scores exceeding the normalized threshold are selected as candidate target documents. Second, we calculate validation similarity between the input string and each of the candidates. This validation similarity is based on the presence of the candidate record’s metadata fields (year, volume, issue, pages, the last name of the first author, etc.) in the input reference string, as well as the relevance score returned by the search engine. Finally, the most similar candidate is returned as the final target document, if its validation similarity exceeds the validation threshold.

By adding the validation stage to the search-based matching we make sure that the same bibliographic numbers (year, volume, etc.) are present in both the input reference and the returned document. We also don’t simply take the first result, but rather use this validation similarity to choose from results scored similarly by the search engine.

All the thresholds are parameters which have to be set prior to the matching. The thresholds used in these experiments were chosen using a separate dataset, as the values maximizing the F1 of each algorithm.

Question 3. How to create the dataset?

Results

We could try to calculate our metrics for every single document in the system. Since we currently have over 100M of them, this would take a while, and we already felt impatient…

A faster strategy was to use sampling with all the tools statistics was so generous to provide. And this is exactly what we did. We used a random sample of 2500 items from our system, which is big enough to give reliable results and, as we will see later, produces quite narrow confidence intervals.

Apart from the sample, we needed some input reference strings. We generated those automatically by formatting the metadata of the chosen items using various citation styles. (Similarly to what happens when you automatically format the bibliography section for your article. Or at least we hope you don’t produce those reference strings manually…)

For each record in our sample, we generated 11 citation strings, using the following styles:

Well known citation styles from various disciplines:
- american-chemical-society (acs)
- american-institute-of-physics (aip)
- elsevier-without-titles (ewt)
- apa
- chicago-author-date
- modern-language-association (mla)
Known styles + random noise. To simulate not-so-clean data, we randomly added noise (additional spaces, deleted spaces, typos) to the generated strings of the following styles:
- american-institute-of-physics
- apa
Custom degraded “styles”:
- degraded: a simple concatenation of authors’ names, title, container title, year, volume, issue and pages
- one author: a simple concatenation of the first author’s name, title, container title, year, volume, issue and pages
- title scrambled: same as degraded, but with title words randomly shuffled

Some styles include the DOI in the reference string. In such cases we stripped the DOI from the string, to make the matching problem non-trivial.

An ideal matching algorithm will match every generated string to the record it was generated from. In practise, some of the expected matches will be missing, which will lower the recall of the tested matching approach. On the other hand, it is very probable that we will get the precision of 100%. To have the precision lower than 100%, we would have to have some unexpected matches to our sampled documents, which is unlikely. This is obviously not great, because we are missing a very important piece of information.

What can we do to “encourage” such mismatches to our sampled documents? We could generate additional reference strings of documents that are not in our sample, but are similar to the documents in our sample. Hopefully, we will see some incorrect links from those similar strings to our sampled documents.

For each sampled document I added up to 2 similar documents (I used, surprise surprise, our search engine to find the most similar documents). I ended up with 7,374 items in total (2,500 originally sampled and 4,874 similar items). For each item, 11 different reference strings were generated. Each reference string was then matched using the tested approaches and I could finally look at some results.

Results

First, let’s compare the overall results averaged over the entire dataset:

The small vertical black lines at the top of the boxes show the confidence intervals at the confidence level 95%. The table gives the exact values and the same confidence intervals. The best result for each metric is bolded.

	average precision	average recall	average F1
legacy approach	0.9933 (0.9910 - 0.9956)	0.4203 (0.4095 - 0.4312)	0.5289 (0.5164 - 0.5413)
SBM (simple threshold)	0.9890 (0.9863 - 0.9917)	0.7127 (0.7021 - 0.7233)	0.7866 (0.7763 - 0.7968)
SBM (normalized threshold)	0.9872 (0.9844 - 0.9901)	0.7905 (0.7796 - 0.8015)	0.8354 (0.8249 - 0.8458)
SBMV	0.9923 (0.9902 - 0.9945)	0.7902 (0.7802 - 0.8002)	0.8448 (0.8352 - 0.8544)

The confidence intervals given in the table are the ranges, in which it is 95% likely to have the real average precision, recall and F1. For example, we are 95% sure that the real F1 for SBMV in our entire collection is within the range 0.8352 - 0.8544.

As we can see, each metric has a different winner.

The legacy approach is the best in precision. This suggests the legacy approach is quite conservative and outputs a match only if it is very sure about it. This might also result in missing a number of true matches (false negatives).

According to the paired Student’s t-test, the difference between the average precision of the legacy approach and the average precision of the second best SBMV is not statistically significant. This means we cannot rule out that this difference is simply the effect of the randomness in sampling, and not the sign of the true difference.

SBM with a normalized threshold is the best in recall. This suggests that it is fairly tolerant and returns a lot of matches, which might also result in returning more incorrect matches (false positives). Also in this case the difference between the winner and the second best (SBMV) is not statistically significant.

SBMV is the best in F1. This shows that this approach balances precision and recall the best, despite being only the second best in both of those metrics. According to the paired Student’s t-test, the difference between SBMV and the second best approach (SBM with a normalized threshold) is statistically significant.

All variants of the search-based matching outperform the parsing-based approach in terms of F1, with statistically significant differences. This shows that in search based-matching it is possible to keep precision almost as good as in the legacy approach, and still include many more true positives.

Let’s also look at the same results split by the citation style:

For all styles the precision values are very high, and the legacy approach is slightly better than all variations of the search-based approach.

In terms of recall and F1 SBM with a simple threshold is better than the legacy approach in 8 out of 11 styles. The three styles for which the legacy approach outperforms SBM with a simple threshold are styles that do not include the title in the reference strings (acs, aip and ewt). The reason for this is that the simple threshold cannot be well calibrated for shorter and longer reference strings at the same time.

SBM with a normalized threshold and SBMV is better than the legacy approach in recall and F1 for all 11 styles.

The weak spot of the legacy approach is degraded and noisy reference strings, which do not appear to use any of the known citation styles.

The weak spot of the search-based matching is short reference strings, and in particular citation styles that do not include the title in the string.

Limitations

The limitations are related mostly to the method of building the dataset.

All the numbers reported here are estimates, since they were calculated on a sample.
The numbers show strengths and weaknesses of each approach, but they do not reflect the real precision and recall in the system:
- Since we included only 2 similar documents for each document in the sample, precision is most likely lower in the real data.
- We used a number of styles distributed uniformly. Of course in the real system the styles and their distribution might be different, which affects all the calculated numbers.

What does the sample say?

Dominika Tkaczyk — Fri, 09 Nov 2018 00:00:00 +0000

At Crossref Labs, we often come across interesting research questions and try to answer them by analyzing our data. Depending on the nature of the experiment, processing over 100M records might be time-consuming or even impossible. In those dark moments we turn to sampling and statistical tools. But what can we infer from only a sample of the data?

Imagine you are cooking soup. You just put some salt in it and now you are wondering if it is salty enough. What do you do next?

Option #1: Since you carefully measured 1/7 of a teaspoon of salt per 0.13 litres of soup (as always), you already know the soup is fine. Everyone else better stop asking silly questions and eat their soup.
Option #2: You stir everything carefully and taste a tablespoon. If it is not salty enough, you put more salt in the soup and repeat the tasting procedure.
Option #3: You eat a tablespoon of soup and it tastes fine. But wait, there’s more soup in the pot, what if the sip you’ve just tasted was somehow different than the rest? You decide it’s better to eat another spoon of soup (which tastes fine). Still, a lot of soup left, who knows what that tastes like? It might be safer to eat an entire bowl of soup. Hmm, still not sure, you’ve eaten such a small fraction of the soup, who can guarantee the rest tastes the same? You have no choice but to eat another bowl, and then some more… Ooops, now you have eaten the entire pot of soup! At least you can be 100% sure now that the soup was indeed salty enough. The problem is, there is no soup left, and also, you don’t feel so good. But people are getting hungry, so you start cooking a new batch…

If your answer was option #3, read on. Your life is going to get easier!

TL;DR

Sampling and confidence intervals can be used to estimate the mean of a certain feature, or the proportion of items passing a certain test, by calculating it only for a random sample of items, instead of the entire large set of items. Note that estimating =/= guessing.
Confidence intervals are a way of controlling the amount of uncertainty related to randomness in sampling.
The confidence interval has a form (estimated value - something, estimated value + something). Confidence interval at the confidence level 95% is interpreted as follows: we are 95% sure that the real value that we are estimating is within our calculated confidence interval.
The higher the confidence level (i.e. the more certain we want to be about the interval), the wider the interval has to be.
The larger the sample, the narrower the confidence interval.
We are never 100% sure that the value we are estimating is actually within our calculated confidence interval. By setting the confidence level high, we only make sure this is a very likely event.

The problem

Sampling and estimating drew my attention while I was working on the evaluation of the reference matching algorithms. In Crossref’s case, reference matching is the task of finding the target document DOI for the given input reference string, such as:

(1) Adamo, S. H.; Cain, M. S.; Mitroff, S. R. Psychological Science 2013, 24, 2569–2574.

Accurate reference matching is very important for the scientific community. Thanks to automatic reference matching we are able to find citing relations in large document sets, calculate citation counts, H-indexes, impact factors, etc.

For several weeks now I have been investigating a simple reference matching algorithm based on the search engine. In this algorithm, we use the input reference string as the query in the search engine, and we return the first item from the results as the target document. Luckily, at Crossref we already have a good search engine in place, so all the pieces are there.

I was interested in how well this simple algorithm works, i.e. how often the correct target document is found. For example, let’s say we have a reference string in APA citation style generated for a specific record in Crossref system. How certain can I be that it will be correctly matched to the record’s DOI?

I could calculate this directly by generating the APA reference string for every record in the system and trying to match those strings to DOIs. Since we already have over 100M records, this would take a while and I was getting impatient. So instead of eating the whole pot of soup, I decided to stir and taste just a little bit of it, or, academically speaking, use sampling and confidence intervals.

These statistical tools are useful in situations, where we have a large set of items, and we want to know the average of a certain feature of an item in our set, or the proportion of items passing a certain test, but calculating it directly is impossible or difficult. For example, we might want to know the average height of all women living in USA, the average salary of a Java programmer in London, or the proportion of book records in the Crossref collection. The entire set we are interested in is called a population and the value we are interested in is called a population average or a population proportion. Sampling and confidence intervals let us estimate the population average or proportion using only a sample of items, in a reliable and controlled way.

Experiments

In general I wanted to see, how well I can estimate the population proportion of records passing a certain test, using only a sample.

In the following experiments, the population is 1 million metadata records from the Crossref collection. I didn’t use the entire collection as the population, because I wanted to be able to calculate the real proportion and compare it to the estimates.

The test for a single record is: whether the APA reference string generated from said record is correctly matched to the record’s original DOI. In other words: if I generate the APA reference string from my record and use it as the query in Crossref’s search, will the record be the first element in the result list? Note that this proportion can also be interpreted as the probability that the APA reference string will be correctly matched to the target DOI.

Estimating from a sample

I took a random sample of size 100 from my population and calculated the proportion of the records correctly matched - this is called a sample proportion. In my case, the sample proportion is 0.92. This means that in my sample 92 reference strings were successfully matched to the right DOIs. Not too bad.

I could now treat this number as the estimate and assume that 0.92 is close to the population proportion. On the other hand, this is only a sample, and a rather small one, which raises doubts. What if our 92 correct matches happen to be the only correct matches in the entire 1M population? In such a case, our estimate of 0.92 would be very far from the population proportion. This uncertainty related to sampling randomness can be captured by the confidence interval.

Confidence interval

The confidence interval for my 100-point sample, at the confidence level 95%, is 0.8668-0.9732. This is interpreted as follows: we are 95% sure that the real population proportion is within the range 0.8668-0.9732. Note that the sample average (0.92) is exactly in the middle of this range.

100 items is not a big sample. Let’s calculate the confidence interval for a sample 10 times larger. From a sample of size 1000 I got the estimate 0.932, and the confidence interval 0.9164-0.9476. Based on this, we can be 95% sure that the real population proportion is within the range 0.9164-0.9476.

It seems the our interval got smaller when we increased the sample size. Let’s plot the intervals for a variety of sample sizes:

The blue line represents the estimated proportion for samples of different sizes, and the grey vertical lines are confidence intervals. The estimated proportion varies, because for each size a different sample was drawn.

We can see that increasing the sample size decreases the interval. This should make intuitive sense: if we have more data to estimate from, we can expect our estimate to be more reliable (i.e. closer to the population proportion).

What about the confidence level? By setting the confidence level we specify, how certain we want to be about our confidence interval. So far I used 95%. What happens if I calculate the confidence intervals for my original sample of 100 records, but with varying confidence level?

In this case the average is always the same, because only one sample was used.

As we can see, increasing the confidence level widens the interval. In other words, the more certain we want to be about the interval containing the real population average, the wider the interval has to be.

Sampling distribution

So far so good, but where does this magic confidence interval actually come from? It is calculated by the theoretical analysis of the sampling distribution (not to be confused with sample distribution):

Sample distribution is when we collect one sample of size k and calculate a certain feature for every element in the sample. It is a distribution of k values of the feature in one sample.
Sampling distribution is when we independently collect n samples, each of size k, and calculate the sample proportion for each sample. It is the distribution of n sample proportions.

Imagine I collect all samples of size 100 from my population and I calculate the sample proportion for each sample. This is the sampling distribution. Now I randomly choose one number from this sampling distribution. Note that this is equivalent to what I did before: choosing one random sample of size 100 and calculating its sample proportion.

According to Central Limit Theorem, sampling distribution is approximately normal with the mean equal to the population proportion. Here is the visualisation of the sampling distribution:

The black vertical line shows the mean of the sampling distribution. This is also the real population proportion. The grey area covers the middle 95% of the distribution mass (within 2 standard deviations from the mean).

When we choose one sample and calculate the sample proportion, there are two possibilities:

With 95% probability, we were lucky and the sample proportion is within the grey area. In that case, the real population proportion is not further than 2 standard deviations from our estimate.
With 5% probability, we were unlucky and the sample proportion is outside the grey area. In that case, the real population proportion is further than 2 standard deviations from our estimate.

So with the confidence of 95% we can say that the real population proportion is within 2 standard deviations from our sample proportion. We can see now that these 2 standard deviations of the sampling distribution define our confidence interval at the confidence level of 95%.

Smaller confidence level would make the grey area narrower, and the confidence interval would shrink as well. Larger confidence level makes the grey area, and the confidence interval, larger.

To look more closely at the sampling distribution, I generated sampling distributions for all combinations of “n samples of size k”, where n and k are the elements of the set {25, 50, 100, 200, 400, 800, 1600, 3200}. This is only an approximation, since the real sampling distributions would contain many more samples.

Here is the heatmap showing the mean of each sampling distribution (this should be approximately the same as the real population proportion):

We can see that there is some variability in the top left part of the heatmap, which corresponds to small sample sizes and small numbers of samples. The bottom right part of the heatmap shows much less variability. As we increase the sample size and number of samples, the mean of the sampling distribution approaches numbers around 0.933.

Here is the heatmap showing the standard deviation for each sampling distribution:

We can clearly see how the standard deviation decreases when we increase the sample size. This is consistent with the previous observation, that the confidence interval decreases when the sample size is increased.

Let’s also see the histograms of all the sampling distributions:

Here we can see the following patterns:

All histograms indeed seem to be centered around approximately the same number.
The more samples we include, the more normal the sampling distribution appears. This happens because with more samples the real sampling distribution is better approximated.
The larger the sample size, the narrower the sampling distribution (i.e. smaller standard deviation).

The estimation vs. the real value

Let’s go back to my original question. What is the proportion of reference strings in APA style, that are successfully matched to the original DOIs of the records they were generated from? So far we observed the following:

A small sample of 100 gave the estimate 0.92 (confidence interval 0.8668-0.9732)
A larger samples of 1000 gave the estimate 0.932 (confidence interval 0.9164-0.9476)
The means of sampling distributions seem to slowly approach 0.933

So what is the real population proportion in my case? It is 0.933005. As we can see, the estimations were fairly close, and the intervals indeed contain the real value.

Now I can also calculate the confidence interval for each sample in my sampling distributions, and then the fraction of the intervals that contain the real population proportion (I expect these numbers to be close to the confidence level 95%). Here is the heatmap:

We can see that for larger sample sizes indeed the fractions are high. The fraction is not always above 95%, as we would expect, especially for smaller sample sizes. One of the reasons is that when we calculate the confidence interval, we approximate the standard deviation of the population with the standard deviation of the sample. This is not always a reliable estimate, especially for small samples. This suggests that sample sizes of at least 1000-2000 should be used.

Be careful

Some important things to remember:

Aggregate functions. As mentioned before, apart from estimating the proportion, a similar procedure can be applied for estimating the average of a certain numeric feature.
(Lack of) certainty. Remember that the confidence level < 1. This means that we are never sure that our confidence interval contains the true population proportion. If for any reason you need to be 100% sure, just process the entire dataset.
Randomness, a.k.a. “stirring before tasting”. The sample has to be chosen randomly. Beware of assuming that the dataset is shuffled and taking the first 1000 rows!
Sample size. We know already that the larger the sample, the better. As a rule of thumb, using sample sizes < 30 makes the estimates, including the interval, rather unreliable.
Skewness. In general, the more skewed the original feature distribution, the larger sample we need. In case of the proportion, the sample should contain at least 5 data points of each value of the feature (passes/doesn’t pass the test).
Generalization. The sample average/proportion can be used as an estimate for the population average/proportion, but only the population it was drawn from. This means that if we applied any filters before sampling (which is equivalent to sampling from a subset passing the filter), we can reason only about the filtered subset of the data.
Reproducibility. This is more of an engineering concern. In short, all the analyses we do should be reproducible. In the context of sampling it means, at the very least, that we should record the samples we use.

Linking references is different from registering references

Anna Tolwinska — Wed, 30 May 2018 00:00:00 +0000

From time to time we get questions from members asking what the difference is between reference linking and registering references as part the Content Registration process.

Here’s the distinction:

Linking out to other articles from your reference lists is a key part of being a Crossref members - it’s an obligation in the membership agreement and it levels the playing field when all members link their references to one another.

Registering references when you register your content is completely different. It’s enriching the metadata record that describes your content, and it allows Crossref and others—including non-members—to use them.

Reference Linking

A research article usually includes a reference list of citations to other works that helped inform it. The original function of Crossref was to provide a central service for publishers that enabled them to link to each others’ content from these reference lists—using a DOI as a persistent link. This meant that members of all sizes and in all disciplines could easily link to one another without having to sign hundreds of bilateral agreements.

We made Reference Linking obligatory for Crossref members because it’s fundamental to making content discoverable, and because when everyone links their references, research travels further and benefits everyone.

Registering references

Every single day hundreds of members register and update their metadata with us—and every single day hundreds of organisations search for, extract and use it. To make sure your content is discovered in this process, it’s important to make the metadata you register with us as rich as possible. Rich metadata includes information such as journal title, article author, publication date, page numbers, ISSN, abstracts, ORCID iDs, funding information, clinical trials numbers, license information, and of course—references.

Additionally, registering references is ~~a prerequisite~~ recommended for participating in our Cited-by service—which provides citation counts and lists, and ultimately makes your content more discoverable. [EDIT 7th February 2024 - it is no longer required but highly recommended.]

We know it’s not easy for smaller publishers to deposit references. Read more on how to here. Our upcoming Metadata Manager tool will allow you to register your references at the same time as the rest of your content. This service is currently in development but let us know if you want to try it out. [EDIT 7th February 2024 - Metadata Manager has been deprecated. More info about it here.]

Reference Linking

Reference Linking means adding Crossref DOI links to the reference list for journal articles on your article pages as per this example: https://doi-org.pluma.sjfc.edu/10.1088/1367-2630/1/1/006.

How it works

First retrieve DOIs for all available references either through our human or machine interfaces. Then make sure you use the DOI link in your references and on your article landing page using the Crossref DOI display guidelines.

Why it’s useful

Reference Linking:

Enables you to link to more than 10,000 publishers without having to sign multiple agreements
Helps with discoverability, because DOIs don’t break if implemented correctly
Displays your DOIs as URLs so that anyone can copy and share them
Makes your content more useful to readers
Drives traffic to your website from other publishers.

Is it obligatory?

Yes, within a short time after becoming a member you should be including references.

Registering References

Registering references means submitting them as part of your Crossref metadata deposit as per this example: https://www-crossref-org.pluma.sjfc.edu/xml-samples/article_with_references.xml.

How it works

Whenever you register content with us, make sure you include your references in the submission. You can also add references to your existing content via a metadata redeposit, or our resource-only deposit, or our Simple Text Query form.

Why it’s useful

References registered as part of your metadata:

Make your content more discoverable
Make your content richer and more useful
Are required to participate in our Cited-by service (this service shows what articles cite your article)
Enables discovery of research
Enables evaluation of research
Highlights your contents’ provenance
Helps with citation counts.

Is it obligatory?

No, it’s optional, but strongly encouraged. It is ~~required~~ recommended if you are participating in our Cited-by service. [EDIT 7th February 2024 - it is no longer required but highly recommended].

If you have any questions about reference linking or registering your references please get in touch.

Revised Crossref DOI display guidelines are now active

Ed Pentz — Wed, 15 Mar 2017 00:00:00 +0000

We have updated our DOI display guidelines as of March 2017, this month! I described the what and the why in my previous blog post New Crossref DOI display guidelines are on the way and in an email I wrote to all our members in September 2016. I’m pleased to say that the updated Crossref DOI display guidelines are available via this fantastic new website and are now active. Here is the URL of the full set of guidelines in case you want to bookmark it (https://doi-org.pluma.sjfc.edu/10.13003/5jchdy) and a shareable image to spread the word on social media.

This blog is a quick reminder that all Crossref members should now be displaying DOIs in the recommended new format from this month, on any new content you publish online. Please note these guidelines are for Crossref DOIs only, we have nearly 90 million registered but there are others, and not all DOIs are made equal.

The main changes are to display the DOI as a full, linked URL using HTTPS:

https://doi-org.pluma.sjfc.edu/10.xxxx/xxxxx

For background on the HTTPS issue please read Geoffrey Bilder’s blog post, Linking DOIs using HTTPS.

What will happen if you don’t update your Crossref DOI display?

We tell members that they should be working towards making the change even if they can’t do it until later - we recognize that it is not always an easy change to make.

However, if members don’t make the change, nothing immediate will happen (Crossref won’t fine you!) although as more members make the change your display will look odd and out of place compared with other members’ content.

If you have any questions please do not hesitate to contact us.

Included, registered, available: let the preprint linking commence.

Rachael Lammey — Mon, 05 Dec 2016 00:00:00 +0000

We began accepting preprints as a new record type last month (in a category known as “posted content” in our XML schema). Over 1,000 records have already been registered in the first few weeks since we launched the service.

By extending our existing services to preprints, we want to help make sure that:

links to these publications persist over time
they are connected to the full history of the shared research
the citation record is clear and up-to-date.

It’s not just collecting the metadata however, it’s also making it available so that it can be as widely used as possible. Preprint metadata is no different. As with all record types, we make the metadata available for machine and human access, across multiple interfaces (e.g. REST API, OAI-PMH, Crossref Metadata Search)

For example, you can see information on the preprint https://doi-org.pluma.sjfc.edu/10.20944/preprints201608.0191.v1 in a number of ways:

If you want to see all the preprint metadata deposited so far, try https://api-crossref-org.pluma.sjfc.edu/v1/types/posted-content/works. Over 1,000 records have already been registered in the first few weeks since we launched the service.

Crossref members depositing preprints need to make sure they:

Register content using the posted content metadata schema.
Respond to our match notifications that a manuscript / version of record (AM/VOR) has been registered and link to that within seven days.
Label the manuscript as a preprint clearly, above the scroll on the preprint landing page, and ensure that any link to the AM/VOR is also prominently displayed above the scroll.

It’s important to clearly label the record type so we can ensure that the connections between preprints and the associated literature are clearly visible, to both humans and machines.

As with other record types, there is a registration fee to include content in the Crossref system. For preprints, it’s $0.25 fee for current preprint files and $0.15 for back-year records.

Are you an existing Crossref member who wants to assign preprint DOIs? Let’s talk about getting started or migrating any existing content over to the dedicated preprint deposit schema.

Interested in becoming a Crossref member to assign DOIs to your preprints? Contact our membership specialist so we can answer any questions and get you set up as a member.

New Crossref DOI display guidelines are on the way

Ed Pentz — Tue, 27 Sep 2016 00:00:00 +0000

TL;DR

Crossref will be updating its DOI Display Guidelines within the next couple of weeks. This is a big deal. We last made a change in 2011 so it’s not something that happens often or that we take lightly. In short, the changes are to drop “dx” from DOI links and to use “https:” rather than “http:”. An example of the new best practice in displaying a Crossref DOI link is: https://doi-org.pluma.sjfc.edu/10.1629/22161

Hey Ho, “doi:” and “dx” have got to go

The updated Crossref DOI Display guidelines recommend that https://doi-org.pluma.sjfc.edu/ be used and not http://dx.doi.org.pluma.sjfc.edu/ in DOI links. Originally the “dx” separated the DOI resolver from the International DOI Foundation (IDF) website but this has changed and the IDF has already updated its recommendations so we are bringing ours in line with theirs.

We are also recommending the use of HTTPS because it makes for more secure browsing. When you use an HTTPS link, the connection between the person who clicks the DOI and the DOI resolver is secure. This means it can’t be tampered with or eavesdropped on. The DOI resolver will redirect to both HTTP and HTTPS URLs.

Timing and backwards compatibility

We are requesting all Crossref member publishers and anyone using Crossref DOIs to start following the updated guidelines as soon as possible. But realistically we are setting a goal of six months for implementation; we realize that updating systems and websites can take time. We at Crossref will also be updating our systems within six months - we already use HTTPS for some of our services and our new website (coming very soon!) will use HTTPS.

An important point about backwards compatibility is that “http://dx.doi.org.pluma.sjfc.edu/” and “http://doi.org.pluma.sjfc.edu/” are valid and will continue to work forever-or as long as Crossref DOIs continue to work-and we plan to be around a long time.

We need to do better

Reflecting on the 2011 update to the display guidelines it’s fair to say that we have been disappointed. It is still much too common to see unlinked DOIs in the form doi:10.1063/1.3599050 or DOI: 10.1629/22161 or even unlinked in this form: http://dx.doi.org.pluma.sjfc.edu/10.1002/poc.3551

What’s so wrong with this approach? To demonstrate, please click on this DOI doi:10.1063/1.3599050 - oh, you can’t click on it? How about I send you to a real example of a publisher page. What I’d like you to do is click the following link and then copy the DOI you find there and come back - http://dx.doi.org.pluma.sjfc.edu/10.1002/poc.3551.

Are you back? I expect you had to carefully highlight the “10.1063/1.3599050” and then do “edit”, “copy”. That wasn’t too bad but the next step is to put the DOI into an email and send it to someone. But wait - what are they going to do with “10.1063/1.3599050”? It’s useless. If you want it to be useful you’ll have to add “http://doi.org.pluma.sjfc.edu” or https://doi-org.pluma.sjfc.edu/ in the front.

When publishers follow the guidelines it makes things easier - if you go to https://doi-org.pluma.sjfc.edu/10.1063/1.3599050 you’ll note that you can just right click on the full DOI link on the page and get a full menu of options of what to do with it. One of which is to copy the link and then you can easily paste into an email or anywhere else.

However-putting a positive spin on the spotty adherence to the 2011 update to the DOI display guidelines-everyone has another chance with the latest set of updates to make all the changes at once!

More on HTTPS (future-proofing scholarly linking)

We take providing the central linking infrastructure for scholarly publishing seriously. Because we form the link between publisher sites all over the web, it’s important that we do our bit to enable secure browsing from start to finish. In addition, HTTPS is now a ranking signal for Google who gives sites using HTTPS a small ranking boost.

The process of enabling HTTPS on publisher sites will be a long one and, given the number of members we have, it may a while before everyone’s made the transition. But by using HTTPS we are future-proofing scholarly linking on the web.

Some years ago we started the process of making our new services available exclusively over HTTPS. The Crossref Metadata API is HTTPS enabled, and Crossmark and our Assets CDN use HTTPS exclusively. Last year we collaborated with Wikipedia to make all of their DOI links HTTPS. We hope that we’ll start to see more of the scholarly publishing industry doing the same.

So-it’s simple-always make the DOI a full link - https://doi-org.pluma.sjfc.edu/10.1006/jmbi.1995.0238 - even when it’s on the abstract or full text page of the content that the DOI identifies - and use “https://doi-org.pluma.sjfc.edu/”.

Linked Clinical Trials initiative gathers momentum

Kirsty Meddings — Tue, 21 Jun 2016 00:00:00 +0000

We now have linked clinical trials deposits coming in from five publishers: BioMedCentral, BMJ, Elsevier, National Institute for Health Research and PLOS. It’s still a relatively small pool of metadata - around 4000 DOIs with associated clinical trial numbers - but we’re delighted to see that “threads” of publications are already starting to form.

If you look at this article in The Lancet and click on the Crossmark button you will see that in the Clinical Trials section there are links to three other articles reporting on the same trial: two from the American Heart Journal and one from BMJ’s Heart. Readers can navigate between these four articles in three separate journals using the Crossmark functionality- a new set of links and routes for discovery have appeared.

In another example, three articles from PLOS ONE are threaded together around a trial for the treatment of Type 1 diabetes. And here another PLOS journal, Neglected Tropical Diseases links through to a PLOS ONE article about the same trial.

If you publish in the health sciences please do consider joining this exciting initiative so that we can expand these threads and build up the metadata. Read the tech specs here or drop me an email if you have questions.

Where do DOI clicks come from?

Joe Wass — Thu, 19 May 2016 00:00:00 +0000

As part of our Event Data work we’ve been investigating where DOI resolutions come from. A resolution could be someone clicking a DOI hyperlink, or a search engine spider gathering data or a publisher’s system performing its duties. Our server logs tell us every time a DOI was resolved and, if it was by someone using a web browser, which website they were on when they clicked the DOI. This is called a referral.

This information is interesting because it shows not only where DOI hyperlinks are found across the web, but also when they are actually followed. This data allows us a glimpse into scholarly citation beyond references in traditional literature.

Last year Crossref Labs announced Chronograph, an experimental system for browsing some of this data. We’re working toward a new version, but in the meantime I’d like to share the results for 2015 and some of 2016. We have filtered out domains that belong to Crossref member publishers to highlight citations beyond traditional publications.

Top 10 DOI referrals from websites in 2015

This chart shows the top 10 referring non-primary-publisher domains of DOIs per month. Note that if browsers don’t send the referrer (e.g. from an HTTPS page), we don’t get to find out. Because the top 10 can be different month to month, the total number of domains mentioned can be more than 10. Subdomains are combined, which means that, for example, the wikipedia.org entry covers all Wikipedia languages. This chart covers all of 2015 and the first two months of 2016.

The top 10 referring domains for the period:

webofknowledge.com
baidu.com
serialssolutions.com
scopus.com
exlibrisgroup.com
wikipedia.org
google.com
uni-trier.de
ebsco.com
google.co.uk

It’s not surprising to see some of these domains here: for example serialssolutions.com and exlibrisgroup.com are effectively proxies for link resolvers, Baidu and Google are incredibly popular search engines which would show up anywhere. But it is exciting to see Wikipedia ranked amongst these. For more detail look out for the new Chronograph.

HTTP vs HTTPS in 2015

We’ve also seen a steady increase in HTTPS referral traffic, i.e. people clicking on DOIs from sites that are using HTTPS. While it is still dwarfed by HTTP, there was a steady uptick throughout 2015.

This chart shows HTTP vs HTTPS referrals per day, which shows up the weekly spikes. It doesn’t include resolutions where we don’t know the referrer.

Increasing numbers of people are moving to HTTPS for reasons of security, privacy and protection from tampering. Google has announced plans to take HTTPS into account when ranking search results. Wikipedia has moved exclusively to HTTPS, and I’ll be telling the story of how Crossref and Wikipedia collaborated in an upcoming blog post.

Chronograph

Another version of Chronograph will be available soon. It will contain full data for all non-primary-publisher referring domains. Stay tuned!

Getting Started with Crossref DOIs, courtesy of Scholastica

Anna Tolwinska — Mon, 25 Apr 2016 00:00:00 +0000

I had a great chat with Danielle Padula of Scholastica, a journals platform with an integrated peer-review process that was founded in 2011. We talked about how journals get started with Crossref, and she turned our conversation into a blog post that describes the steps to begin registering content and depositing metadata with us. Since the result is a really useful description of our new member on-boarding process, I want to share it with you here as well. As always, comments and questions are welcome here, at member@Crossref.org, and @CrossrefOrg. - Anna_

The internet is in a constant state of change, with new content being added to the web by the minute and old content sometimes getting moved around. While the benefit of publishing scholarly outputs online is that it’s possible to update them at any moment, moving or modifying content can also …

The Wikipedia Library: A Partnership of Wikipedia and Publishers to Enhance Research and Discovery

Rachael Lammey — Mon, 04 Apr 2016 00:00:00 +0000

Back in 2014, Geoffrey Bilder blogged about the kick-off of an initiative between Crossref and Wikimedia to better integrate scholarly literature into the world’s largest knowledge space, Wikipedia. Since then, Crossref has been working to coordinate activities with Wikimedia: Joe Wass has worked with them to create a live stream of content being cited in Wikipedia; and we’re including Wikipedia in Event Data, a new service to launch later this year. In that time, we’ve also seen Wikipedia importance grow in terms of the volume of DOI referrals.

Alex Stinson, Project Manager for the Wikipedia Library, and our guest blogger! This file is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported license (Source: Myleen Hollero Photography)

Alex Stinson, Project Manager for the Wikipedia Library, and guest blogger! This file is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported license (Source: Myleen Hollero Photography)

How can we keep this momentum going and continue to improve the way we link Wikipedia articles with the formal literature? We invited Alex Stinson, a project manager at The Wikipedia Library (and one of our first guest bloggers) to explain more:

Wikipedia provides the most public gateway to academic and scholarly research. With millions of citations to academic as well as non-academic but reliable sources, like those produced by newspapers, its ecosystem of 5 million English Wikipedia articles and 35 million articles in hundreds of languages provides the first stop for researchers in both scholarly and informal research situations. The practice of “checking Wikipedia” has become ubiquitous in a number of fields; for example, Wikipedia is the most visited source of medical information online, even providing the first stop for many medical students and medical practitioners when looking for medical literature.

The Wikipedia Library program helps Wikipedia’s volunteer editors access and use the best sources in their research and citations. Through partnerships with over fifty leading publishers and aggregators, like JSTOR, Project Muse, Elsevier, Newspapers.com, Highbeam, Oxford University Press and others, we have been able to give over 3000 of our most prolific volunteers access to over 5500 accounts. These are clear, win-win relationships where Wikipedia editors get to use these databases to improve Wikipedia, while in turn linking to authoritative resources and enhancing their discovery.

JSTOR has been working with us since 2012, providing over 500 accounts to our editors. Kristen Garlock at JSTOR writes:

“We’re very happy to collaborate with the Wikipedia Library to provide JSTOR access to Wikipedia editors. Supporting the initiative to increase editor access to scholarly resources and improve the quality of information and sources on Wikipedia has the potential to help all Wikipedia readers. In addition to providing more discoverability for our institutional subscribers, introducing new audiences to the scholarship on JSTOR them discover access opportunities like our Register & Read program.”

There are strong signals that Wikipedia’s role in the citation ecosystem helps ensure the best materials reach the public through its over 400 million monthly readers:

The latest estimates by Crossref show that Wikipedia has risen from the 8th most prolific referrer to DOIs to the 5th.

Two of our access partners have found that around half of the referrals arriving from Wikipedia were able to authenticate into their subscription resources, suggesting that a large portion of our readers can take advantage of subscriptions provided by scholarly institutions.

Wikipedia is highly influential in the open access ecosystem as well, with a recent study showing higher citation rates for OA materials than those behind a paywall.

Altmetrics tools (such as Altmetric.com, ImpactStory or Plum Analytics) are recognizing Wikipedia’s importance by including Wikipedia citations in their impact metrics.

Despite these advances, we think this is only the beginning of Wikipedia’s impact on the landscape of scholarly research and discovery. Wikipedia can become a highly integrated research platform within the broader research ecosystem, where the best scholarship is summarized and discoverable-where Wikipedia effectively becomes the front matter to all research.

However, there are some clear barriers to fulfilling this vision. Currently, most citations on Wikipedia are stored in free-text and not readily available in machine-readable formats; our community is working to fix this. Wikipedia also has major systematic gaps in topics where either we lack volunteer interest or Wikipedia reflects larger systemic biases within society or scholarship.We need the help of volunteers, experts, industry partners, and information technologists to grow Wikipedia’s collection of citations, especially around key missing areas, and to transform existing citations into structured formats.

WikiData, Wikipedia’s sister project which crowdsources structured metadata, offers an excellent opportunity for improving the impact of Wikipedia in research. Having Wikipedia citations stored in this structured ecosystem, connecting metadata with semantic meaning, would allow the citations in Wikipedia to become the backbone for discovery tools which emphasize the hand-curated interrelationships between authoritative sources and the knowledge collected by Wikipedia and Wikidata editors.

We need more collaborators to realize the full vision of Wikipedia supporting research in the most effective ways:

We need help from publishers with subscription databases, to help us give our editors access to the databases through The Wikipedia Library’s access partnership program. These high-quality source materials allow our editors to expose that research in a number of languages and for millions of readers.

We need help from the open access community, to figure out how to better support increased citation and strategic use of open access materials within Wikipedia and other Wikimedia projects. Our community has some ideas, but we need your input and collaboration.

We need your expertise to build our structured metadata ecosystem, by helping Wikidata map and collect citation data.

We need the larger research community to promote Wikipedia as a scholarly communications tool and make contributing to Wikipedia an important part of the social responsibility of experts. Wider citation of sources in Wikipedia ensures widespread discovery and dissemination of that research.

If you think you can help, we invite you to contact us at wikipedialibrary@wikimedia.org or via Twitter @WikiLibrary.

Linking clinical trials = enriched metadata and increased transparency

Kirsty Meddings — Mon, 18 Jan 2016 00:00:00 +0000

We will shortly be adding a new feature to Crossmark. In a section called “Clinical Trials” we will be using new metadata fields to link together all of the publications we know about that reference a particular clinical trial.

Most medical journals make clinical trial registration a prerequisite for publication. Trials should be registered with one of the fifteen WHO-approved public trial registries , or with clinicaltrials.gov which is run by the US National Library of Medicine. Once registered, a trial is assigned a clinical trial number (CTN) which is subsequently used to identify that trial in any publications that report on it.

Publications that result from any one trial are likely to be released in multiple journals from different publishers and at different times, for example secondary analyses coming some time after the publication of the initial results. Cross-publisher collaboration is paramount to linking all of these publications together so that researchers, funders, and regulatory agencies can understand the whole set of results from clinical trials. With this in mind, a group of medical publishers, led by BioMedCentral, approached Crossref to establish a working group, and here, they designed an approach to address this problem: “thread” all the various documents together surrounding a clinical trial.

Updated upstream

To implement threaded publications, publishers extract clinical trial numbers from papers, or ask authors to submit those numbers to them. Publishers add the CTNs to the Crossref DOI metadata via three new fields: clinical trial number, clinical trial registry where trial is registered, and trial stage (pre-results, results or post-results of the trial). Crossref has assigned unique IDs to each trial registry (much the same as we have done for funders in our Funder Registry and for the same reason - trial registry names and URIs can change over time and we need a persistent identifier). Using a combination of trial registry ID and clinical trial number, we can easily identify other content in the Crossref database that cites the same trial. Finally, Crossref displays the clinical trial metadata on the respective papers for all participating Crossmark publishers. Crossmark is a convenient place for readers to access the clinical trial information and is readily accessible directly from the journal article (online and PDF versions). And of course all of the data also goes into our open API so that anyone can make use of it.

The reporting of clinical trial results is notoriously inconsistent, something that the AllTrials initiative is also seeking to address. Publishers can help by collecting this information upstream and disseminating it using the existing Crossref infrastructure.

We ask all publishers to deposit the clinical trial data which is so critical to transparency in this area of research, and have already had the first data in from Crossref member the National Institute of Health Research. Once we launch the initial set of linked clinical trials, we will expand coverage of the threaded publications to include all content that reports on or references a clinical trial, from protocol to results to supporting data and systematic reviews.

Stay tuned and watch this space as threaded publications rolls out to journal articles across publishers!

Rehashing PIDs without stabbing myself in the eyeball

Geoffrey Bilder — Thu, 11 Jun 2015 00:00:00 +0000

Anybody who knows me or reads this blog is probably aware that I don’t exactly hold back when discussing problems with the DOI system. But just occasionally I find myself actually defending the thing…

About once a year somebody suggests that we could replace existing persistent citation identifiers (e.g. DOIs) with some new technology that would fix some of the weaknesses of the current systems. Usually said person is unhappy that current systems like

DOI, Handle, Ark, perma.cc, etc. depend largely on a social element to update the pointers between the identifier and the current location of the resource being identified. It just seems manifestly old-fashioned and ridiculous that we should still depend on bags of meat to keep our digital linking infrastructure from falling apart.

In the past, I’ve threatened to stab myself in the eyeball if I was forced to have the discussion again. But the dirty little secret is that I play this game myself sometimes. After all, the best thing a mission-driven membership organisation could do for its members would be to fulfil its mission and put itself out of business. If we could come up with a technical fix that didn’t require the social component, it would save our members a lot of money and effort.

When one of these ideas is posed, there is a brief flurry of activity as another generation goes through the same thought processes and (so far) comes to the same conclusions.

The proposals I’ve seen generally fall into one of the following groups:

Replace persistent identifiers (PIDs) with hashes, checksums, etc.
Just use search (often, but not always coupled with 1 above)
Automagically create PIDs out of metadata.
Automagically redirect broken citations to archived versions of the content identified
And more recently… use the blockchain

I thought it might help advance the discussion and avoid a bunch of dead ends if I summarised (rehashed?) some of the issues that should be considered when exploring these options.

Warning: Refers to FRBR terminology. Those of a sensitive disposition might want to turn away now.

DOIs, PMIDs, etc. and other persistent identifiers are primarily used by our community as “citation identifiers”. We generally cite at the “expression” level.
Consider the difference between how a “citation identifier” a “work identifier” and a “content verification identifier” might function.
How do you deal with “equivalent manifestations” of the same expression. For example the ePub, PDF and HTML representations of the same article are intellectually equivalent and interchangeable when citing. The same applies to csv & tsv representations of the same dataset. So, for example, how do hashes work here as a citation identifier?
Content can be changed in ways that typically doesn’t effect the interpretation or crediting of the work. For example, by reformatting, correcting spelling, etc. In these cases the copies should share the same citation identifier, but the hashes will be different.
Content that is virtually identical (and shares the same hash) might be republished in different venues (e.g. a normal issue and a thematic issue). Context in citation is important. How do you point somebody at the copy in the correct context?
Some copies of an article or dataset are stewarded by publishers. That is, if there is an update, errata, corrigenda, retraction/withdrawal, they can reflect that on the stewarded copy, not on copies they don’t host or control. Location is, in fact, important here.
Some copies of content will be nearly identical, but will differ in ways that would affect the interpretation and/or crediting of the work. A corrected number in a table for example. How would you create a citation form a search that would differentiate the correct version from the incorrect version?
Some content might be restricted, private or under embargo. For example private patient data, sensitive data about archaeological finds or the migratory patterns of endangered animals.
Some content is behind paywalls (cue jeremiads)
Content is increasingly composed of static and dynamic elements. How do you identify the parts that can be hashed?
How do you create an identifier out of metadata and not have them look like this?

This list is a starting point that should allow people to avoid a lot of blind alleys.

In the mean time, good luck to those seeking alternatives to the current crop of persistent citation identifier systems. I’m not convinced it is possible to replace them, but if it is- I hope I beat you to it. 🙂 And I hope I can avoid stabbing myself in the eye.

Real-time Stream of DOIs being cited in Wikipedia

Joe Wass — Tue, 03 Mar 2015 00:00:00 +0000

TL;DR

Watch a real-time stream of DOIs being cited (and “un-cited!” ) in Wikipedia articles across the world: https://live-eventdata-crossref-org.pluma.sjfc.edu/live.html

Background

For years we’ve known that the Wikipedia was a major referrer of Crossref DOIs and about a year ago we confirmed that, in fact, the Wikipedia is the 8th largest refer of Crossref DOIs. We know that people follow the DOIs, too. This despite a fraction of Wikipedia citations to the scholarly literature even using DOIs. So back in August we decided to create a Wikimedia Ambassador programme. The goal of the programme was to promote the use of persistent identifiers in citation and attribution in Wikipedia articles. We would do this through outreach and through the development of better citation-related tools.

Remember when we originally wrote about our experiments with the PLOS ALM code and how that has transitioned into the DOI Event Tracking Pilot? In those posts we mentioned that one of the hurdles in gathering information about DOI events is the actual process of polling third party APIs for activity related to millions of DOIs. Most parties simply wouldn’t be willing handle the load of a 100K API calls an hour. Besides, polling is a tremendously inefficient process, only a fraction of DOIs are ever going to generate events, but we’d have to poll for each of them, repeatedly, forever, to get an accurate picture of DOI activity. We needed a better way. We needed to see if we could reverse this process and convince some parties to instead “push” us information whenever they saw DOI related events (e.g. citations, downloads, shares, etc). If only we could convince somebody to try this…

Wikipedia DOI Events

In December 2014 we took the opportunity of the 2014 PLOS/Crossref ALM Workshop in San Francisco too meet with Max Klein and Anthony Di Franco where we kicked off a very exciting project.

There’s always someone editing a Wikipedia somewhere in the world. In fact, you can see a dizzying live stream of edits. We thought that given that there are so many DOIs in Wikipedia, that live stream may contain some diamonds (DOIs are made of diamond, that’s how they can be persistent). Max and Anthony went away and came back with a demo that contains a surprising amount of DOI activity.

That demo is evolving into a concrete service, called Cocytus. It is running at Wikimedia Labs monitoring live edits as you read this.

For now we’re feeding that data into the DOI Events Collection app (which is an off-shoot of the Chronograph project). We are in the process of modifying the Lagotto code so that we can instead push those events into the DOI Event Tracking Instance.

The first DOI event we noticed was delightfully prosaic: The DOI for “The polymath project” is cited by the Wikipedia page for “Polymath Project”. Prosaic perhaps, but the authors of that paper probably want to know. Maybe they can help edit the page.

Or how about this. Someone wrote a a paper about why people edit Wikipedia and then it was cited by Wikipedia. And then the citation was removed. The plot thickens…

We’re interested in seeing how DOIs are used outside of the formal scholarly literature. What does that mean? We don’t fully know, that’s the point. We have retractions in scholarly literature (and our Crossmark metadata and service allow publishers to record that), but it’s a bit different on Wikipedia. Edit wars are fought over … well you can see for yourself.

Citations can slip in and out of articles. We saw the DOI 10.1001/archpediatrics.2011.832 deleted from “Bipolar disorder in children”. If we’d not been monitoring the live feed (we had considered analysing snapshots of the Wikipedia in bulk) we might never have seen that. This is part of what non-traditional citations means, and it wasn’t obvious until we’d seen it.

You can see this activity on the Chronograph’s stream. Or check your favourite DOI. Please be aware that we’re only collecting newly added citations as of today. We do intend to go back and back-fill, but that may take some time- as it * cough * requires polling again.

Some Technical Things

A few interesting things that happened as a result of all this:

Secure URLs

SSL and HTTPS were invented so you could do things like banking on the web without fear of interception or tampering. As the web becomes a more important part of life, many sites are upgrading from HTTP to HTTPS, the secure version. This is not only because your confidential details may be tampered with, but because certain governments might not like you reading certain materials.

Because of this, some time ago, Wikipedia decided to embark on an upgrade to HTTPS last year, and they are a certain way along the path. The IDF, who are responsible for running the DOI system, upgraded to HTTPS this Summer, although most DOIs are referred to by HTTP still.

We met with Dario Taraborelli at the ALM workshop and discussed the DOI referral data that is fed into the Chronograph. We put two and two together and realised that Wikipedia was linking to DOIs (which are mostly HTTP) from pages which might be served over HTTPS. New policies in HTML5 specify that referrer URL headers shouldn’t be sent from HTTPS to HTTP (in case there was something secret in them). The upshot of this is, if someone’s browsing Wikipedia via HTTPS and click on a normal DOI, we won’t know that the user came from Wikipedia. Not a huge problem today, but as Wikipedia switches over to entirely secure, we’re going to miss out on very useful information.

Fortunately, the HTML5 specification includes a way to fix this (without leaking sensitive information). We discussed this with Dario, and he did some research, and came up with a suggestion, which got discussed. It’s fascinating to watch a democratic process like this take place and take part in it.

We’re waiting to see how the discussion turns out, and hope that it all works out so we can continue to report on how amazing Wikipedia is at sending people to scholarly literature.

How shall I cite thee?

Another discussion grew out of that process, and we started talking to a Wikipedian called Nemo (note to Latin scholars: we weren’t just talking to ourselves). Nemo (real name Federico Leva) had a few suggestions of his own. Another way to solve the referrer problem is by using HTTPS URLs (HTML5 allows browsers to send the referrer domain when going from HTTPS to HTTPS).

This means going back to all the articles that use DOIs and change them from HTTP to HTTPS. Not as simple as it sounds, and it doesn’t sound simple. We started looking into how DOIs were cited on Wikipedia.

After some research we found that there are more ways that we expected to cite DOIs.

First, there’s the URL. You can see it in action in this article. URLs can take various forms.

Second there’s the official template tag, seen in action here:

<ref name="SCI-20140731">{{cite journal |title=Sustained miniaturization and anatomical innovation in the dinosaurian ancestors of birds |url=http://www.sciencemag.org.pluma.sjfc.edu/content/345/6196/562 |date=1 August 2014 |journal=[[Science (journal)|Science]] |volume=345 |issue=6196 |pages=562–566 |doi=10.1126/science.1252243 |accessdate=2 August 2014 |last1=Lee |first1=Michael S. Y. |first2=Andrea|last2=Cau |first3=Darren|last3=Naish|first4=Gareth J.|last4=Dyke}}</ref>

There’s a DOI in there somewhere. This is the best way to cite DOIs, firstly as it’s actually a proper traditional citation and there’s nothing magic about DOIs, secondly because it’s a template tag and can be re-rendered to look slightly different if needed.

Third there’s the old official DOI template tag that’s now discouraged:

<ref name="Example2006">{{Cite doi|10.1146/annurev.earth.33.092203.122621}}</ref>

And then there’s another one.

{{doi|10.5555/123456789}}

Knowing all this helps us find DOIs. But if we want to convert DOIs links in Wikipedia to use HTTPS, it means that there are more template tags to modify and more pages to re-render.

Nemo also put DOIs on the Interwiki Map which should make automatically changing some of the URLs a lot easier.

We’re very grateful to Nemo for his suggestions and work on this. We’ll report back!

The elephant in the room

Those of you who know how DOIs work will have spotted an unsecured elephant in the room. When you visit a DOI, you visit the URL, which hits the DOI resolver proxy server, which returns a message to your browser to redirect to the landing page on the publisher’s site.

Securely talking to the DOI resolver by using HTTPS instead of HTTP means that no-one can eavesdrop and see which DOI you are visiting, or tamper with the result and send you off to a different page. But the page you are sent to will be, in nearly all cases, still HTTP. Upgrading infrastructure isn’t trivial, and, with over 4000 members (mostly publishers), most Crossref DOIs will still redirect to standard HTTP pages for the foreseeable future.

You can keep as secure as possible by using HTTPS Everywhere.

Fin

There’s lots going on, watch this space to see developments. Thanks for reading this, and all the links. We’d love to know what you think.

Bootnote

Not long after this blog post was published we saw something very interesting.

That’s no DOI. We like interesting things, but they can panic us. This turned out to be a great example of why this kind of thing can be useful. A minute’s digging and we found the article edit:

It turns out that this was a typo: someone put a title when they should have put in a DOI. And, as the event shows, this was removed from the Wikipedia article.

Crossref’s DOI Event Tracker Pilot

Geoffrey Bilder — Mon, 02 Mar 2015 00:00:00 +0000

TL;DR

Crossref’s “DOI Event Tracker Pilot”- 11 million+ DOIs & 64 million+ events. You can play with it at: http://goo.gl/OxImJa

Tracking DOI Events

So have you been wondering what we’ve been doing since we posted about the experiments we were conducting using PLOS’s open source ALM code? A lot, it turns out. About a week after our post, we were contacted by a group of our members from OASPA who expressed an interest in working with the system. Apparently they were all about to conduct similar experiments using the ALM code, and they thought that it might be more efficient and interesting if they did so together using our installation. Yippee. Publishers working together. That’s what we’re all about.

So we convened the interested parties and had a meeting to discuss what problems they were trying to solve and how Crossref might be able to help them. That early meeting came to a consensus on a number of issues:

The group was interested in exploring the role Crossref could play in providing an open, common infrastructure to track activities around DOIs, they were not interested in having Crossref play a role in the value-add services of reporting on an interpreting the meaning of said activities.
The working group needed representatives from multiple stakeholders in the industry. Not just open access publishers from OASPA, but from subscription based publishers, funders, researchers and third party service providers as well.
That it was desirable to conduct a pilot to see if the proposed approach was both technically feasible and financially sustainable.

And so after that meeting, the “experiment” graduated to becoming a “pilot.” This Crossref pilot is based on the premise that the infrastructure involved in tracking common information about “DOI events” can be usefully separated from the value-added services of analysing and presenting these events in the form of qualitative indicators. There are many forms of events and interactions which may be of interest. Service providers will wish to analyse, aggregate and present those in a range of different ways depending on the customer and their problem. The capture of the underlying events can be kept separate from those services.

In order to ensure that the Crossref pilot is not mistaken for some sub rosa attempt to establish new metrics for evaluating scholarly output, we also decided eschew any moniker that includes the word “metrics” or synonyms. So the “ALM Experiment” is dead. Long live the “”DOI Event Tracker” (DET) pilot. Similarly PLOS’s open source “ALM software” has been resurrected under the name “Lagotto.”

The Technical Issues

Crossref members are interested in knowing about “events” relating to the DOIs that identify their content. But our members face a now-classic problem. There are a large number of sources for scholarly publications (3k+ Crossref members) and that list is still growing. Similarly, there are an unbounded number of potential sources for usage information. For example:

Supplemental and grey literature (e.g. data, software, working papers)
Orthogonal professional literature (e.g. patents, legal documents, governmental/NGO/IGO reports, consultation reports, professional trade literature).
Scholarly tools (e.g. citation management systems, text and data mining applications).
Secondary outlets for scholarly literature (institutional and disciplinary repositories, A&I services).
Mainstream media (e.g. BBC, New York Times).
Social media (e.g. Wikipedia, Twitter, Facebook, Blogs, Yo).

Finally, there is a broad and growing audience of stakeholders who are interested in seeing how the literature is being used. The audience includes publishers themselves as well as funders, researchers, institutions, policy makers and citizens.

Publishers (or other stakeholders) could conceivably each choose to run their own system to collect this information and redistribute it to interested parties. Or they can work with a vendor to do the same. But either case, they would face the following problems:

The N sources will change. New ones will emerge. Old ones will vanish.
The N audiences will change. New ones will emerge. Old ones will vanish.
Each publisher/vendor will need to deal with N source’s different APIs, rate limits, T&Cs, data licenses, etc. This is a logistical headache for both the publishers/vendors and for the sources.
Each audience will need to deal with N publisher/vendor APIs, rate limits, T&Cs, data licenses, etc. This is a logistical headache for both the audiences and for the publishers.
If publishers/vendors use different systems which in turn look at different sources, it will be difficult to compare or audit results across publishers/vendors.
If a journal moves from one publisher to another, then how are the metrics for that journal’s articles going to follow the journal?

And then there is the simple issue of scale. Most parties will be interested in comparing the data that they collect for their own content, with data about their competitors. Hence, if they all run their own system, they will each be querying much more than their own data. If, for example, just the commercial third-party providers were interested in collecting data covering the formal scholarly literature, they would each find themselves querying the same sources for the same 80 million DOIs. To put this into perspective, to refresh the data for 10 million DOIs once a month, would require sources to support ~ 14K API calls an hour. 60 million DOIs would require 100K API calls an hour. Current standard API caps for many of the sources that people are interested in querying hover around 2K per hour. We may see these sources lift that cap for exceptional cases, but they are unlikely to do so for many different clients all of whom are querying essentially the same thing.

These issues typify the “multiple bilateral relationships” problem that Crossref was founded to try and ameliorate. When we have many organisations trying to access the exact same APIs to process the exact same data (albeit to different ends), then it seems likely that Crossref could help make the process more efficient.

Piloting A Proposed Solution

The Crossref DET pilot aims to show the feasibility of providing a hub for the collection, storage and propagation of DOI events from multiple sources to multiple audiences.

Data Collection

Pull: DET will collect DOI event data from sources that are of common interest to the membership, but which are unlikely to make special efforts to accommodate the scholarly communications industry. Examples of this class of source include large, broadly popular services like FaceBook, Twitter, VK, Sina Weibo, etc.
Push: DET will allow sources to send DOI event data directly to Crossref in one of three ways:
- Standard Linkback: Using standards that are widely used on the web. This will automatically enable linkback-aware systems like WordPress, Moveable Type, etc. to alert DET to DOI events.
- Scholarly Linkback: A to-be-defined augmented linkback-style API which will be optimized to work with scholarly resources and which will allow for more sophisticated payloads including other identifiers (e.g. ORCIDs, FundRefs), metadata, provenance information and authorization information. This system could be used by tools designed for scholarly communications. So, for example, it could be used by publisher platforms to distribute events related to downloads or comments within their discussion forums. It could also be used by third party scholarly apps like Zotero, Mendeley, Papers, Authorea, IRUS-UK, etc. in order to alert interested parties in events related to specific DOIs.
- Redirect: DET will also be able to serve as a service discovery layer that will allow sources to push DOI event data directly to an appropriate publisher-controlled endpoint using the above scholarly linkback mechanism. This can be used by sources like repositories in order to send sensitive usage data directly to the relevant publishers.

Data Propagation

Parties may want to use the DET in order to propagate information about DOI events. The system will support two broad data propagation patterns:

one-to-many: DOI events that are commonly harvested (pulled) by the DET system from a single source will be distributed freely to anybody who queries the DET API. Similarly, sources that push DOI events via the standard or scholarly linkback mechanisms, will also propagate their DOI events openly to anybody who queries the DET API. DOI events that are propagated in either of these cases will be kept and logged by the DET system along with appropriate provenance information. This will be the most common, default propagation model for the DET system.
one-to-one: Sources of DOI events can also report (push) DOI event data directly to owner of the relevant DOI if the DOI owner provides & registers a suitable end-point with the DET system. In these cases, data sources seeking to report information relating to a DOI, will be redirected (with a suitable 30X HTTP status and relevant headers) to the end-point specified by the DOI owner. The DET system will not keep the request or provenance information. One-to-one propagation model is designed to handle use cases where the source of the DOI event has put restrictions on the data and will only share the DOI events with the owner (registrant) of the DOI. This use case may be used, for example, by aggregators or A&I services that want to report confidential data directly back to a publisher. The advantage of the redirect mechanism is that Crossref is not put into the position of having to secure sensitive data as said data will never reside on Crossref systems.

Note that the two patterns can be combined. So, for example, a publisher might want to have public social media events reported to the DET and propagated accordingly, but to also to private third parties report confidential information directly to the publisher.

So Where Are We?

So to start with, the DET Working Group has grown substantially since the early days and we have representatives from a wide variety of stakeholders. The group includes:

Cameron Neylon, PLOS
Chris Shillum, Elsevier
Dom Mitchell, Co-action Publishing
Euan Adie, Altmetric
Jennifer Lin, PLOS
Juan Pablo Alperin, PKP
Kevin Dolby, Wellcome Trust
Liz Ferguson, Wiley
Maciej Rymarz, Mendeley
Mark Patterson, eLife
Martin Fenner, PLOS
Mike Thelwell, U Wolverhampton
Rachel Craven, BMC
Richard O’Beirne, OUP
Ruth Ivimey-Cook, eLife
Victoria Rao, Elsevier

As well as the usual contingent of Crossref cat-herders including: Geoffrey Bilder, Rachael Lammey & Joe Wass.

When we announced the then-DET experiment, we said that one of the biggest challenges would be to create something that scaled to industry levels. At launch, we only loaded in about 317,500+ Crossref DOIs representing publications from 2014 and we could see the system was going to struggle. Since then Martin Fenner and Jennifer Lin at PLOS have been focusing on making sure that the Lagotto code scales appropriately and now it is currently humming along with just over 11.5 million DOIs for which we’ve gathered over 64 million “events.” We aren’t worried about scalability on that front any more.

We’ve also shown that third parties should be able to access the API to provide value added reporting and metrics. As a demonstration of this, PLOS configured a copy of its reporting software “Parascope” to point at the Crossref DET instance. The next step we’re taking is to start testing the “push” API mechanism and the “point-to-point redirect” API mechanism. For the push API, we should have a really exciting demo available to show within the next few days. And on the point-to-point redirect, we have a sub-group exploring how the point-to-point redirect mechanism could potentially be used for reporting COUNTER stats as a compliment to the Sushi initiative.

The other major outstanding task we have before us is to calculate what the costs will be of running the DET system as a production service. In this case we expect to have some pretty accurate data to go on as we will have had close to half a year of running the pilot with a non-trivial number of DOIs and sources. Note that the work group is concerned to ensure that the underlying data from the system remains open to all. Keeping this raw data open as seen as critical to establishing trust in the metrics and reporting systems that third parties build on the data. The group has also committed to leaving the creation of value-add services to third parties. As such we have been focusing on exploring business models based around service-level-agreement backed versions of the API to complement the free version of the same API. The free API will come with no guarantees of uptime, performance characteristics or support. For those users that depend on the API in order to deliver their services, we will offer paid-for SLA-backed versions of the free APIs. We can then configure our systems so that we can independently scale these SLA-backed APIs in order to meet SLA agreements.

Our goal is to have these calculations complete in time for the working group to make a recommendation to the Crossref board meeting in July 2015.

Until then, we’ll use CrossTech as a venue for notifying people when we’ve hit new milestones or added new capabilities to the DET Pilot system.

Introducing the Crossref Labs DOI Chronograph

Joe Wass — Mon, 12 Jan 2015 00:00:00 +0000

tl;dr http://chronograph.labs.crossref.org.pluma.sjfc.edu

At Crossref we mint DOIs for publications and send them out into the world, but we like to hear how they’re getting on out there. Obviously, DOIs are used heavily within the formal scholarly literature and for citations, but they’re increasingly being used outside of formal publications in places we didn’t expect. With our DOI Event Tracking / ALM pilot project we’re collecting information about how DOIs are mentioned on the open web to try and build a picture about new methods of citation.

As part of the preparation for collaborating with Wikipedia, we looked at our statistics about when DOIs are clicked and discovered that Wikipedia was, over a two year period from 2012, the eighth largest referrer of DOIs. This means that not only does Wikipedia have a lot of DOIs, but people click them too. This bit of one-off data analysis (which surprised us) gave us enough of a prod to kickstart our collaboration with Wikipedia.

At the ALM Workshop 2014 in San Francisco we talked to some Wikipedians and bibliometricians and realised that we were sitting on a really interesting data-set and that it would be churlish not to share it. At the hackathon (read the report here) we started work on a service to gather information about DOIs and, a month later, we’re ready to unveil the DOI Chronograph.

Show me the goods

You can see:

Daily referrals (clicks) from top level domains, e.g. Wikipedia.org: http://chronograph.labs.crossref.org.pluma.sjfc.edu/domain.html?domain=wikipedia.org

Daily referrals from specific subdomains, e.g. fr.wikipedia.org: http://chronograph.labs.crossref.org.pluma.sjfc.edu/domain.html?domain=fr.wikipedia.org

Daily resolutions per DOI: http://chronograph.labs.crossref.org.pluma.sjfc.edu/doi.html?doi=10.1787%2F20752288

And, the chart that kicked this all off: DOI referring domains league tables. This shows that Wikipedia is the 3rd or 4th non-traditional referrer of DOIs (i.e. excluding referrals from Publishers’ domains): http://chronograph.labs.crossref.org.pluma.sjfc.edu/top.html

Try it out

Visit the Chronograph and give it a try chronograph.labs.crossref.org on your favourite DOI (everyone has one).

More data

Talking to a bibliometrician we also realised we can correlate other data for DOIs. We’re getting the issue date (approximately the publication date) from our own metadata, as well as the date that the Crossref metadata was updated. This gives interesting results, like the resolutions for 10.1038/ncomms2953, which peak after publication and then tails off. We are attempting to collect the following information:

daily resolution counts
day on which resolution was first successful
day on which it’s possible to resolve the DOI (we’ve got a bot running for new publications)
day on which the publisher says the article was published
day on which the metadata was most recently deposited with us
day on which the metadata was first deposited with us

We’re not there yet, but we’ve made a start and we’ve already got some pretty interesting data!

Weasel words

It’s a labs project so the usual weasel words apply. Specifically, we currently have the logs for 2012 to 2014 (we’re working at digging out the rest), and the referral information for 50 million DOIs (out of 71 million). That number will be higher by the time you read this. If your page is slow to load, be patient, as it’s currently working hard crunching numbers.

This project is focused on exploring the use of DOIs outside of the formal literature. As such, we are only looking at referrals from domains that do not appear to belong to primary publishers (i.e. our members). If you try a domain and it doesn’t work, it could be that the domain belongs to one of our members. If you’ve notice any mistakes, please email us at labs@crossref.org .

Finally, these numbers contain all DOI resolutions. That’s human clicks but also content negotiation to retrieve metadata, robots etc. We might try to filter them in future, but for now be aware that not every visitor is a human.

I’ll detail some of the the technical stuff (it’s very interesting) and what happened next with Wikipedia in a future post. Watch this space.

A Christmas Reading List… with DOIs

Geoffrey Bilder — Sun, 13 Dec 2009 00:00:00 +0000

Was outraged (outraged, I tell you) that one of my favorite online comics, PhD, didn’t include DOIs in their recent bibliography of Christmas-related citations.. So I’ve compiled them below.

We care about these things so that you don’t have to. Bet you will sleep better at night knowing this.

Or perhaps not…

A Christmas Reading List… with DOIs.

Citation: Biggs, R, Douglas, A, Macfarlane, R, Dacie, J, Pitney, W, Merskey, C & O’Brien, J, 1952, ‘Christmas Disease’, BMJ, vol. 2, no. 4799, pp. 1378-1382.

Crossref DOI: http://dx.doi.org.pluma.sjfc.edu/10.1136/bmj.2.4799.1378

Title: More Than a Labor of Love: Gender Roles and Christmas Gift Shopping

Citation: Fischer, E & Arnold, S, 1990, ‘More Than a Labor of Love: Gender Roles and Christmas Gift Shopping’, Journal of Consumer Research, vol. 17, no. 3, p. 333.

Crossref DOI: http://dx.doi.org.pluma.sjfc.edu/10.1086/208561

Title: Looking at Christmas trees in the nucleolus

Citation: Scheer, U, Xia, B, Merkert, H & Weisenberger, D, 1997, ‘Looking at Christmas trees in the nucleolus’, Chromosoma, vol. 105, no. 7-8, pp. 470-480.

Crossref DOI: http://dx.doi.org.pluma.sjfc.edu/10.1007/s004120050209

Title: The Vela glitch of Christmas 1988

Citation: McCulloch, P, Hamilton, P, McConnell, D & King, E, 1990, ‘The Vela glitch of Christmas 1988’, Nature, vol. 346, no. 6287, pp. 822-824.

Crossref DOI: http://dx.doi.org.pluma.sjfc.edu/10.1038/346822a0

Title: Cardiac Mortality Is Higher Around Christmas and New Year’s Than at Any Other Time: The Holidays as a Risk Factor for Death

Citation: Phillips, D, 2004, ‘Cardiac Mortality Is Higher Around Christmas and New Year’s Than at Any Other Time: The Holidays as a Risk Factor for Death’, Circulation, vol. 110, no. 25, pp. 3781-3788.

Crossref DOI: http://dx.doi.org.pluma.sjfc.edu/10.1161/01.CIR.0000151424.02045.F7

Title: Red Crabs in Rain Forest, Christmas Island: Biotic Resistance to Invasion by an Exotic Snail

Citation: Lake, P & O’Dowd, D, 1991, ‘Red Crabs in Rain Forest, Christmas Island: Biotic Resistance to Invasion by an Exotic Snail’, Oikos, vol. 62, no. 1, p. 25.

Crossref DOI: http://dx.doi.org.pluma.sjfc.edu/10.2307/3545442

Title: The Carvedilol Hibernation Reversible Ischaemia Trial, Marker of Success (CHRISTMAS) study Methodology of a randomised, placebo controlled, multicentre study of carvedilol in hibernation and heart failure

Citation: Pennell, D, 2000, ‘The Carvedilol Hibernation Reversible Ischaemia Trial, Marker of Success (CHRISTMAS) study Methodology of a randomised, placebo controlled, multicentre study of carvedilol in hibernation and heart failure’, International Journal of Cardiology, vol. 72, no. 3, pp. 265-274.

Crossref DOI: http://dx.doi.org.pluma.sjfc.edu/10.1016/S0167-5273(99)00198-9

QR Codes and DOIs

Geoffrey Bilder — Tue, 08 Dec 2009 00:00:00 +0000

Inspired by Google’s recent promotion of QR Codes, I thought it might be fun to experiment with encoding a Crossref DOI and a bit of metadata into one of the critters. I’ve put a short write-up of the experiment on the Crossref Labs site, which includes a demonstration of how you can generate a QR Code for any given Crossref DOI. Put them on postcards and send them to your friends for the holidays. Tattoo them on your pets. The possibilities are endless.

Citation Typing Ontology

Geoffrey Bilder — Fri, 20 Mar 2009 00:00:00 +0000

I was happy to read David Shotton’s recent Learned Publishing article, Semantic Publishing: The Coming Revolution in scientific journal publishing, and see that he and his team have drafted a Citation Typing Ontology.^*

Anybody who has seen me speak at conferences knows that I often like to proselytize about the concept of the “typed link”, a notion that hypertext pioneer, Randy Trigg, discussed extensively in his 1983 Ph.D. thesis.. Basically, Trigg points out something that should be fairly obvious- a citation (i.e. “a link”) is not always a “vote” in favor of the thing being cited.
In fact, there are all sorts of reasons that an author might want to cite something. They might be elaborating on the item cited, they might be critiquing the item cited, they might even be trying to refute the item cited (For an exhaustive and entertaining survey of the use and abuse of citations in the humanities, Anthony Grafton‘s, The Footnote: A Curious History, is a rich source of examples)
Unfortunately, the naive assumption that a citation is tantamount to a vote of confidence has become inshrined in everything from the way in which we measure scholarly reputation, to the way in which we fund universities and the way in which search engines rank their results. The distorting affect of this assumption is profound. If nothing else, it leads to a perverse situation in which people will often discuss books, articles, and blog postings that they disagree with without actually citing the relevant content, just so that they can avoid inadvertently conferring “wuffie” on the item being discussed. This can’t be right.
Having said that, there has been a half-hearted attempt to introduce a gross level of link typology with the introduction of the “nofollow” link attribute- an initiative started by Google in order to try to address the increasing problem of “Spamdexing”. But this is a pretty ham-fisted form of link typing- particularly in the way it is implemented by the Wikipedia where Crossref DOI links to formally published scholarly literature have a “nofollow” attribute attached to them but, inexplicably, items with a PMID are not so hobbled (view the HTML source of this page, for example). Essentially, this means that, the Wikipedia is a black-hole of reputation. That is, it absorbs reputation (through links too the Wikipedia), but it doesn’t let reputation back out again. Hell, I feel dirty for even linking to it here ;-).
Anyway, scholarly publishers should certainly read Shotton’s article because it is full of good, and practical ideas about what can can be done with today’s technology in order to help us move beyond the “digital incunabula” that the industry is currently churning out. The sample semantic article that Shotton’s team created is inspirational and I particularly encourage people to look at the source file for the ontology-enhanced bibliography which reveals just how much more useful metadata can be associated with the humble citation.
And now I wonder whether CiteULike, Connotea, 2Collab or Zotero will consider adding support for the CItation Typing Ontology into their respective services?
* Disclosure:
a) I am on the editorial board of Learned Publishing
b) Crossref has consulted with David Shotton on the subject of semantically enhancing journal articles

DOIs in an iPhone application

Geoffrey Bilder — Thu, 12 Feb 2009 00:00:00 +0000

Very cool to see Alexander Griekspoor releasing an iPhone version of his award-winning Papers application. A while ago Alex intigrated DOI metadata lookup into the Mac version of papers and now I can get a silly thrill from seeing Crossref DOIs integrated in an iPhone app. Alex has just posted a preview video of the iPhone application and it includes a cameo appearance by a DOI. Yay.

ORE/POWDER: Remarks on Ratings

Tony Hammond — Sat, 06 Dec 2008 00:00:00 +0000

I wanted to make some remarks about the “Ease of use” and “Learn curve” ratings which I gave in the ORE/POWDER comparison table that I blogged about here the other day. It may seem that I came out a little harsh on ORE and a little easy on POWDER. I just wanted to rationalize the justification for calling it that way. (By the way, the revised comparison table includes a qualification to those ratings.)

My primary interest was from the perspective of a data provider rather than a data consumer. What does it take to get a resource description document (“resource map”, “description resource” or “sitemap”) ready for publication?

(Continues)

To look at POWDER first, it defines two sets of semantics: an “operational semantics” which is embodied in the simple XML that is intended as the primary publication vehicle, and a “formal semantics” embodied in the RDF/OWL document that would typically be generated by a POWDER processor.

The operational semantics (XML) document requires minimal RDF understanding (and arguably none at all): it only requires that URI resources be organized into groups by pattern matching, and that metadata be attached to those groups using groups.

URI patterns are specified using any of the following XML elements for inclusive patterns:

> **<includeschemes>**, **<includehosts>**, **<includeexactpaths>**, **<includepathcontains>**, **<includepathstartswith>**, **<includepathendswith>**, **<includeports>**, **<includequerycontains>**, **<includeiripattern>**, **<includeregex>**, **<includeresources>**

and their exclusive counterparts

**<excludeschemes>**, …

These are turned into corresponding regular expressions by a POWDER processor which then emits RDF/OWL classes using those expressions as property restrictions on set membership. But a publisher is not required to understand this transformation nor the formal semantics generated from the simple XML document that was authored.

Now, as to metadata. Resource group descriptors are either free text (tags) or properties from a published namespace. For example, the property name from a namespace ex: would be added in one of two ways, depending on whether it were a simple literal string (“value”, say) or a resource URI:

http://example.org/value

<ex:name rdf:resource=”http://example.org/value“/>

While technically this is RDF/XML it hardly qualifies, I think, as requiring any great knowledge of RDF, more a knowledge of XML namespaces alone would be sufficient.

And that’s about it – all that is required for publication of a POWDER “description resource” document. (The guidelines for discovery mechanisms of a POWDER document might also need to be consulted.)

So, on that basis I would judge POWDER to be at most “medium” on the “Learn curve”. However, as soon as the mapping to the formal semantics (POWDER-S) using RDF/OWL is considered, then that learn curve rating would automatically swing to “high”.

Now, ORE on the other hand is a straightforward RDF application. What does make ORE a bit of a challenge are the following two aspects:

 1. concept of named aggregation
* abstract data model - no fixed bindings</ol>
Well, the first aspect is what ORE is all about &ndash; its USP &ndash; and what it gives us beyond the simpler POWDER approach of merely describing resource bundles. Still, it’s a concept that needs to be grokked. All too easy to take it for granted.
It is the second aspect that may make ORE appear to be &#8220;difficult&#8221;. It does not prescribe a single binding or set of bindings but provides an abstract data model. That means that a prospective user must endeavour to understand something of the model before deploying.
But enough of that. Because who really reads instruction manuals anyway? So to deploy there are user guides available for one standalone document format (RDF/XML), and two carrier document formats (Atom, RDFa). That means right there that the publisher must either embrace RDF/XML or learn how to weave it into an existing document markup. (By the way, it should be remarked that there is an excellent [primer][3] available - as there is also for POWDER - and user guides for each of the formats.)
So that I think warrants the &#8220;high&#8221; rating for ORE on the learn curve, and the corresponding &#8220;low&#8221; ease of use. But that is not to say that the two initiatives are in any competition and that one should be favoured over the other. They serve different purposes. Any yet they may also have compatibilities as the previous [mapping of ORE in POWDER][4] attempts to show. I’ll leave that task for other commentators.

Resource Maps Encoded in POWDER

Tony Hammond — Fri, 05 Dec 2008 00:00:00 +0000

Following right on from yesterday’s post on ORE and POWDER, I’ve attempted to map the worked examples in the ORE User Guide for RDF/XML (specifically Sect. 3) to POWDER to show that POWDER can be used to model ORE, see

Resource Maps Encoded in POWDER

(A full explanation for each example is given in the RDF/XML Guide, Sect. 3 which should be consulted.)

This could just all be sheer doolally or might possibly turn out to have a modicum of instructional value – I don’t know. (It would be real good to get some feedback here.) There are, however, a couple points to note in mapping ORE to POWDER:

The POWDER form is actually more long-winded because it splits the RDF triples into subject and predicate/object divisions, with the first listed within an iriset and the second within a descriptorset. The net effect, however, may be somewhat cleaner since POWDER uses a simple XML format rather than RDF/XML.
- POWDER only supports simple object types (literals or resources) so the blank nodes in the RDF/XML examples for dcterms:creator cannot be mapped as such. I have chosen here to use either foaf:name or foaf:page value.
  - Likewise, and as far as I am aware, POWDER does not support datatyping but I could be wrong on this. I have thus dropped the datatypes on dcterms:created and dcterms:modified.

This is a fairly naïve mapping. POWDER’s real strength comes in defining groups of resources with its powerful pattern matching capabilities, whereas here I am using a named single resource in each iriset through the includeresource element. I think, though, this does show how the abstract ORE data model can be serialized in yet another format.

Describing Resource Sets: ORE vs POWDER

Tony Hammond — Thu, 04 Dec 2008 00:00:00 +0000

I’ve been reading up on POWDER recently (the W3C Protocol for Web Description Resources) which is currently in last call status (with comments due in tomorrow). This is an effort to describe groups of Web resources and as such has clear similarities to the Open Archives Initiative ORE data model, which has been blogged about here before.

In an attempt to better understand the similarities (and differences) between the two data models, I’ve put up the table which directly compares the two heavyweight contendors OAI-ORE and POWDER and also (unfairly) places them alongside the featherweight Sitemaps Protocol for reference.

This is very much a draft document and I will aim to update the table based on my own further reading and on any feedback that I may get (contributions gratefully received). I’m all too aware that my understanding of the respective data models is painfully limited and I, for one, hope to profit through this exercise. There will be certainly errors which I will aim to fix as soon as I get wind of them. 🙂

By the way, the ORE work especially is of interest to Crossref members and has obvious synergies with the multiple resolution potential that DOI has long promised but not quite delivered on.

Ubiquity commands for Crossref services

Geoffrey Bilder — Wed, 03 Dec 2008 00:00:00 +0000

So the other day Noel O’Boyle made me feel guilty when he pinged me and asked about the possibility using one of the Crossref APIs for creating a Ubiquity extension. You see, I had played with the idea myself and had not gotten around to doing much about it. This seemed inexcusable- particularly given how easy it is to build such extensions using the API we developed for the WordPress and Moveable Type plugins that we announced earlier in the year. So I dug up my half-finished code, cleaned it up a bit and have posted the results.

Note that the back-end that supports the plugins has been moved to more stable machines and the index is now being automatically updated with journal and conference proceeding deposits (sorry, no books yet).

Also note that we are hoping that others will look at the code for the WordPress, Moveable Type and Ubiquity plugins and create more such extensions. If you do, please let us know about them at citation-plugin@crossref.org.

Knols and Citations

Tony Hammond — Thu, 24 Jul 2008 00:00:00 +0000

So, Google’s Knol is now live (see this announcement on Google’s Blog). There’ll be comment aplenty about the merits of this service and how it compares to other user contributed content sites. But one curious detail struck me. In terms of citeability, compare how a Knol contribution (or “knol”) may be linked to as may be a corresponding entry in Wikipedia (here I’ve chosen the subject “Eclipse”):

Knol

https://web.archive.org/web/20080730124803/http://knol.google.com/k/jay-pasachoff/eclipse/IDZ0Z-SC/wTLUGw

Wikipedia: http://en.wikipedia.org/wiki/Eclipse

The Knol link includes author name, subject, and service gunk, while the Wikipedia link includes only the subject. That makes the Wikipedia link both more readily citeable as well as being to some degree discoverable. I wonder what Google’s intentions, if any, are with respect to the citing of their pages (or “knols”) as authoritative sources of information. They don’t seem to be doing themselves many favours.

I am minded of this post on Jeff Young’s Q6 which cites this passage from the HTTP spec (see RFC 2616, Sect. 3.2):

“As far as HTTP is concerned, Uniform Resource Identifiers are simply formatted strings which identify-via name, location, or any other characteristic-a resource.”

URIs bearing these so-called “characteristics” are what I would call a service URI in contrast to a name URI (something that I will elaborate on in a separate post). For now, however, I would just note that the Knol URI looks more like a service URI and the Wikipedia URI more like a name URI. I know which URI form I would prefer to cite.

Library APIs

Tony Hammond — Mon, 21 Jul 2008 00:00:00 +0000

Roy Tennant in a post to XML4Lib announces a new list of library APIs hosted at

https://web.archive.org/web/20080730080413/http://techessence.info/apis//

A useful rough guide for us publishers to consider as we begin cultivating the multiple access routes into our own content platforms and tending to the “alphabet soup” that taken together comprises our public interfaces.

Q6

Tony Hammond — Thu, 03 Jul 2008 00:00:00 +0000

For anybody interested in the why’s and wherefore’s of OpenURL, Jeff Young at OCLC has started posting over on his blog Q6: 6 Questions - A simpler way to understand OpenURL 1.0: Who, What, Where, When, Why, and How (note: no longer available online). He’s already amassing quite a collection of thought provoking posts. His latest is The Potential of OpenURL (note: no longer available online), from which:

OpenURL has effectively cornered the niche market where Referrers need to be decoupled from Resolvers.

Blog has UML diags, definitions, musings, etc. - something for everybody. Definitely worth checking out.

CLADDIER Final Report

Crossref — Tue, 15 Jan 2008 00:00:00 +0000

I just ran across the final report from the CLADDIER project. CLADDIER comes from the JISC and stands for “CITATION, LOCATION, And DEPOSITION IN DISCIPLINE & INSTITUTIONAL REPOSITORIES”. I suspect JISC has an entire department dedicated to creating impossible acronyms (the JISC Acronym Preparation Executive?)

Anyhoo- the report describes a distributed citation location and updating service based on the linkback mechanism that is widely used in the blogging community.

I think this is an interesting approach and is one that I talked about briefly (PDF) at the UKSG’s Measure for Measure seminar last June. I think that, like most proponents of p2p distributed architectures, they massively underestimate the problem of trust in the network. They fully knowledge the problem of linkback spam, but their hand-wavy-solution(tm) of using whitelists just means the system effectively becomes semi-centralized again (you have to have trusted keepers of the whitelists).

And of course I was mildly exasperated by the report’s characterization of one of the perceived “disadvantages” of the Crossref architectural model being a :

“Centralised service hosting a large persistent store – with the need for a (possibly commercial) business model to justify providing the service.”

Though DOI registries like Bowker and Nielsen Bookdata are commercial, Crossref, the organisation that services the industry that the JISC is concerned with, is *not* a commercial service.

Also if you replaced the phrase “justify providing” with the word “sustain”, the sentence wouldn’t sound like such a “disadvantage.”

But aside from these quibbles, the report makes an interesting (if technical) read.

NLM Blog Citation Guidelines

Geoffrey Bilder — Mon, 15 Oct 2007 00:00:00 +0000

I’ve just returned from Frankfurt Book fair and noticed that there has been some recent in the The NLM Style Guide for Authors, Editors and Publishers recommendations concerning citing blogs.

Which reminds me of an issue that has periodically been raised here at Crossref- should we be doing something to try and provide a service for reliably citing more ephemeral content such as blogs, wikis, etc.?

Personally, I cringe when I see people include plain old URLs (POUs?) in citations. What’s the point? They are almost guaranteed to fail to resolve after a few years. In citing them, you are hardly helping to preserve the scholarly record. You might as well just record the metadata associated with the content.

So why don’t we simply allow individuals to assign DOIs to their content?

As Chuck Koscher says, “Crossref DOIs are only as persistent as Crossref staff.” Crossref depends on its ability to chase down and berate member publishers when they fail to update their DOI records. Its hard enough doing this with publishers, so just imagine what it would be like trying to chase down individuals. In short, it just wouldn’t scale.

But what if we provided a different service for more informal content? Recently we have been in talking with Gunther Eysenbach, the creator of the very cool WebCite service about whether Crossref could/should operate a citation caching service for ephemera.

As I said, I think WebCite is wonderful, but I do see a few problems with it in its current incarnation.

The first is that, the way it works now, it seems to effectively leech usage statistics away from the source of the content. If I have a blog entry that gets cited frequently, I certainly don’t want all the links (and their associated Google-juice) redirected away from my blog. As long as my blog is working, I want traffic coming to my copy of the content, not some cached copy of the content (gee- the same problem publishers face, no?). I would also, ideally, like that traffic to continue to come to to my blog if I move hosting providers, platforms (WordPress, Moveable Type) , blog conglomerates (Gawker, Weblogs, Inc.), etc.

The second issue I have with WebCite is simpler. I don’t really fancy having to actually recreate and run a web-caching infrastructure when there is already a formidable one in existence.

So what if we ran a service for individuals that worked like this:

For a fee, you can assign DOIs to your ephemeral, CC-licensed content.
When you assign a DOI to an item of content (or update an existing DOI), we will immediately archive said content with the Internet Archive (who, incidentally, charges for this service)
We will direct those DOIs to your web site as long as you are both:
Paying the fee
Updating your URLs to point to the correct content
If you fail in either “a” or “b”, we will then redirect said DOIs to the cached version of the content on the Internet Archive (after having warned you repeatedly via automated e-mail).

(Note, as an aside, that we could in theory provide a similar dark-archive service for publishers with non free content using something like JStore as the archive)

This approach would help to ensure that a blogger’s version of content was always linked to as long it was available. It would also preserve the “persistence” of Crossref DOIs by making sure that we could always resolve the DOI even if we were not able to get the owner of said DOI to update it.

So back to the NLM guidelines… On the one hand, I’m delighted to see that the NLM has issued guidelines on citing blogs. It seems glaringly obvious that informal (and ephemeral) content such as blogs and wikis are increasingly becoming vital parts of the scholarly record. On the other hand, it also seems to me that recommending that somebody “cite” with a broken pointer (i.e. a URL) to content verges on tokenism. This isn’t the NLM’s fault- there just isn’t a reliable mechanism for citing informal content in a manner that ensures you can then retrieve and look at said content in the future.

And this is no longer a problem confined to the Scholarly/Professional publishing space. As Jon Udell has occasionally pointed out, citation is increasingly an important currency for *any* professional writer on the web. It seems to me that a system for reliably citing blogs and wikis would benefit many communities. I could easily see commercial hosted Blog services (Blogger, WordPress) offering a “Cached-DOI” feature as a premium service to their clients.

So what do you think? What am I missing? is this something we should be looking at?

Handle Plugin: Some Notes

Tony Hammond — Thu, 02 Aug 2007 00:00:00 +0000

The first thing to note is that this demo (the Acrobat plugin) is an application. And that comes with its own baggage, i.e. this is a Windows only plugin and is targeted at Acrobat Reader 8. On a wider purview the application merely bridges an identifier embedded in the media file and the handle record filed against that identifier and delivers some relevant functionality. The data (or metadata) declared in the PDF and in the associated handle if rich enough and structured openly can also be used by other applications. I think this is a key point worth bearing in mind, that the demo besides showing off new functionalities is also demonstrating how data (or metadata) can be embedded at the respective endpoints (PDF, handle).

Some initial observations follow below.

Install problems

As noted in my previous post I had to haul out the old HP laptop and engage in a dialog with our IT folks to get both Acrobat Reader 8 and the plugin installed as I did not have admin privileges on my own machine. Wasn’t pretty but they were kind.

Useability

I don’t know what’s happening here but from our network it seems as if the first attempts to contact the handle server are timing out and the handle client in the plugin is failing over to an alternate route (HTTP?). So, the plugin doesn’t work as expected since the user has to wait an untenably long time (somewhere between 60s and 90s). Of course, if a certain network access policy is required that would need to be specified and implemented by institutions for their users.

I used both Firefox and Internet Explorer browsers and ran into occasional Acrobat plugin crashes which would lock up the browser. Due to the severe network access problems noted above I wasn’t able to rigorously test this further apart from to note that it was “buggy”.

Functionality

I tested most of the demo cases, but was hampered by the useability restrictions noted above. I didn’t see the “Related Links” or get the “Collections” to work but did see all the other cases and tried the buttons provided.

One thing of note is that the Crossref metadata record was spoofed and returned from a stored data file rather than an active query to Crossref. A real query would have been been interesting to guage the impact of network latency, although the lookup point is made by hardwiring a response.

PDF Metadata

OK, so the document DOI is embedded in the PDF both in the document information dictionary and in the (document) metadata stream within an XMP packet. This is great although I do have some specific comments about how the DOI is actually disclosed. See my Metadata in PDF: 2. Use Cases post for details.

Handle Data

Handle types are generally a matter for the handle administrators to oversee, although the unregulated use of new types is not going to help foster interoperability between handle applications. In passing I note that the handles used in this demo

10.5555/pdftest-collection
10.5555/pdftest-collection-item1
10.5555/pdftest-collection-item2
10.5555/pdftest-collection-item3
10.5555/pdftest-crossref
10.5555/pdftest-kernelmetadata
10.5555/pdftest-multires
10.5555/pdftest-rights
10.5555/pdftest-version

make use of the following handle types (periods and underscores used as below)

COLLECTION
COLLECTION_ITEM
HS_ADMIN
HS_MODIFIED
HDL_MD
HDL.RIGHTS
HDL.XREF

There is some degree of variability here which presumably will be managed better with a central handle type registry.

DOI/Handle

And lastly, this demo raises questions again about DOI and handle boundaries. From a handle viewpoint a DOI is nothing more than a branded handle, whereas from a DOI viewpoint a DOI is a specific handle profile with governance and policies, and its own service portfolio. The two terms should not be used interchangeably which I fear is where some of the demo details would lead us. As a very crude analogy (and with apologies to Bob Kahn) but I would see the relationship between DOI and handle as not being dissimilar from that between TCP and IP.

OAI-ORE Presentation at OAI5

Tony Hammond — Wed, 02 May 2007 00:00:00 +0000

I posted here about an initial meeting of the OAI-ORE Technical WG back in January. ORE is the “Object Reuse and Exchange” initiative which is aiming to provide a formalism for describing scholarly works as complete units (or packages) of information on the Web using resource maps which would be available from public access points. From a DOI perspective this work is intimately connected with multiple resolution. For further updates on this work, see here for a presentation by Herbert Van de Sompel on OAI-ORE at the OAI5 Workshop (5th Workshop on Innovations in Scholarly Communication) held a couple weeks back at CERN, Geneva, Switzerland.

The presentation gives an insight regarding the problem domain in which ORE operates, and in the evolving thinking regarding potential solutions. The presentation was recorded on video and is available for both streaming and download (slides, streaming video, video download).

Note that Michael Nelson of Old Dominion University also presented on behalf of the ORE effort at the recent CNI Task Force Meeting and at the DLF Forum.

A Modest Proposal

Tony Hammond — Wed, 11 Apr 2007 00:00:00 +0000

Was just reminded (thanks, Tim) of the possibility of using a special tag in bookmarking services to tag links to documents of interest to a given community. I think this is a fairly well-established practice. Note that e.g. the OAI-ORE project is using Connotea to bookmark pages of interest and tagging them “oaiore” which can then be easily retrieved using the link http://web.archive.org/web/20160402182544/http://www.connotea.org/.

I would suggest that Crossref members might like to consider using the tag “crosstech” in bookmarking pages about publishing technology, so that the following links might be used to retrieve documents of interest to this readership:

del.icio.us - <https://web.archive.org/web/20071206033322/https://del.icio.us/
- CiteULike - http://www.citeulike.org/tag/crosstech
  - Connotea - http://web.archive.org/web/20160402182544/http://www.connotea.org/
    - etc.

Markup for DOIs

Tony Hammond — Thu, 29 Mar 2007 00:00:00 +0000

Following up on his earlier post (which was also blogged to CrossTech here), Leigh Dodds is now [Following up on his earlier post (which was also blogged to CrossTech here), Leigh Dodds is now]3 the possibility of using machine-readable auto-discovery type links for DOIs of the form

These LINK tags are placed in the document HEAD section and could be used by crawlers and agents to recognize the work represented by the current document. This sounds like a great idea and we’d like to hear feedback on it.

Concurrently at Nature we have also been considering how best to mark up in a machine-readable way DOIs appearing within a document page BODY. Current thinking is to do something along the following lines:

doi: 10.1038/nprot.2007.43

which allows the DOI to be presented in the preferred Crossref citation format (doi:10.1038/nprot.2007.43), to be hyperlinked to the handle proxy server (<a href="http://dx.doi.org.pluma.sjfc.edu/10.1038/nprot.2007.43">http://dx.doi.org.pluma.sjfc.edu/10.1038/nprot.2007.43</a>), and to refer to a validly registered URI form for the DOI (info:doi/10.1038/nprot.2007.43). Again, we would be real interested to hear any opinions on this proposal for inline DOI markup as well as on Leigh’s proposal for document-level DOI markup.

(Oh, and btw many congrats to Leigh on his recent promotion to CTO, Ingenta.)

Indexing URLs

Tony Hammond — Thu, 08 Mar 2007 00:00:00 +0000

Leigh Dodds proposes in this post some solutions to persistent linking using web crawlers and social bookmarking.

“When I use del.icio.us, CiteULike, or Connotea or other social bookmarking service, I end up bookmarking the URL of the site I’m currently using. Its this specific URL that goes into their database and associated with user-assigned tags, etc.

…

A more generally applicable approach to addressing this issue, one that is not specific to academic publishing, would be to include, in each article page, embedded metadata that indicates the preferred bookmark link. The DOI could again be pressed into service as the preferred bookmarking link.”

He’s inviting feedback. I’d certainly like to hear what others may think of these suggestions.

OpenURL Podcast

Tony Hammond — Sat, 17 Feb 2007 00:00:00 +0000

Jon Udell interviews Dan Chudnov about OpenURL, see his blog entry: “A conversation with Dan Chudnov about OpenURL, context-sensitive linking, and digital archiving”. The podcast of the interview is available here.

Interesting to see these kind of subjects beginning to be covered by a respected technology writer like Jon. As he says in his post:

“I have ventured into this confusing landscape because I think that the issues that libraries and academic publishers are wrestling with — persistent long-term storage, permanent URLs, reliable citation indexing and analysis — are ones that will matter to many businesses and individuals. As we project our corporate, professional, and personal identities onto the web, we’ll start to see that the long-term stability of those projections is valuable and worth paying for.”

What’s My Link?

Tony Hammond — Mon, 05 Feb 2007 00:00:00 +0000

Simon Willison has a great piece here about disambiguating URLs. Best practice on creating and publishing URLs is obviously something of interest to any publisher. See this excerpt from Simon’s post:

_“Here’s a random example, plucked from today’s del.icio.us popular. convinceme.net is a new online debating site (tag clouds, gradient fills, rounded corners). It’s listed in del.icio.us a total of four times!

https://web.archive.org/web/20070203050251/http://www.convinceme.net/ has 36 saves
https://web.archive.org/web/20070202182238/http://www.convinceme.net/index.php has 148 saves
https://web.archive.org/web/20070203050251/http://www.convinceme.net/ has 211 saves
https://web.archive.org/web/20070202182238/http://www.convinceme.net/index.php has 38 saves

Combined that’s 433 saves; much more impressive, and more likely to end up at the top of a social sharing sites.”_