Programming on Crossref

Redirecting redirection

Geoffrey Bilder — Tue, 24 Apr 2018 00:00:00 +0000

Crossref has decided to change the HTTP redirect code used by our DOIs from 303 back to the more commonly used 302. Our implementation of 303 redirects back in 2010 was based on recommended best practice for supporting linked data identifiers. Unfortunately, very few other parties have adopted this practice.

What’s more, because using a 303 redirect is still unusual, it tends to throw SEO tools into a tizzy- and we spend a lot of time fielding SEO questions from our members about our use of 303s.

TL;DR

At this point, we need to emphasise that we have never seen our use of 303s actually affect page rankings. But at the same time, use of 303 redirects has not had wider uptake. Maintaining this quixotic behaviour just isn’t worth the effort. We hope that, in the future, we can use other techniques (e.g. signposting & cite-as) to achieve some of the things that 303 was supposed to do.

Note that these changes will not affect users or machines using DOIs. The change should be entirely transparent.

Below we provide some background to our decision and after that we provide some detailed technical notes from Jonathan Rees and Henry Thompson who have been very kind in helping to provide Crossref technical guidance on how we can help DOIs best support linked open data and adhere to HTTP best practice.

Background

Back in 2010, Crossref, DataCite (and later, several other RAs) responded to concerns that DOIs were not “linked-data friendly.” There were three problems with DOIs at that time:

It was not clear that DOIs could be used and expressed as HTTP URIs.
There was no standard way to ask a DOI to return a machine-readable representation of the data.
It wasn’t always clear if the DOI resolved to “the thing” (e.g. an article) or “something about the thing” (e.g. a landing page).

On the advice of several people in the linked data community, we proposed some options for fixing this. And we finally settled on:

Recommending that Crossref DOIs be expressed and displayed as HTTP (now HTTPS) URIs. This made it clear that DOIs could be used with HTTP applications.
Enabling DOI registration agencies to support content negotiation. This allowed RAs to support providing machine-readable representations of the data associated with a DOI.
Changing the underlying redirect code from the normal 302 to 303. This was designed to clarify what, at the time, was true- that most DOIs resolved to a landing page, not the article itself.

By any practical measure, machine use of DOIs has exploded since we made these decisions back in 2010. Crossref’s APIs and content negotiation handle over 800 million requests for machine readable data a month. Our sibling organisation, DataCite, has also seen a huge growth in machine use of DOIs. Many applications, from bibliographic management tools, to authoring systems and CRIS systems, make use of machine actionable DOIs all the time. So clearly our work to promote DOIs as machine actionable identifiers is working, but we are certain that our current use of 303 redirects has nothing to do with this growth.

First of all, as we said, very few parties have actually subscribed to the notion of using 303s to help distinguish “the thing” from “something about the thing”.

Secondly, even if they did try to rely on 303s to make this distinction, they would quickly get confused because the DOI is so often just the first in a chain of redirects which do not implement the same semantic distinction. At this point we should be clear - Crossref thinks these kinds of long redirect chains are a bad idea for two main reasons:

They slow down resolution.
They increase the number of potential failure points between the DOI and the item it resolves to.

But we also cannot legislate them away. They exist. And in the real world you will find plenty of DOIs that do a 303 redirect to a system that, in turn, does a 302 redirect to a system that does a 301 redirect and…eventually ends up someplace returning a 200. You get the picture. How on earth is a machine supposed to interpret a 303->302->301->302 redirect chain?

Furthermore - nowadays, after following this chain of redirects, you will often find yourself on a “page” that is both a landing page and the article itself. Dynamic, one-page applications can simply morph the one into the other without the use of additional HTTP requests.

In other words, using 303s is not helping machines interpret what the DOI is pointing at. And yet, people seem to be making good use of machine actionable DOIs and they are not complaining much about it.

Personally, I’d might have just been happy to switch back to using 302s simply so that I could cut down on my conversations with SEO hacks. But that wouldn’t be a principled approach. In 2010 we spent a lot of time considering the initial switch to 303s- we needed to consult with the LOD community on a potential switch back to 302s. At the January 2018 PIDapalooza I had a chance to talk to Henry Thomson about the 302/303 dilemma we faced, and he along with Jonathan Rees very generously provided the following feedback.

Best practices for HTTP redirection by persistent identifier resolvers: 302 vs. 303

Jonathan Rees (MIT CSAIL, https://orcid.org/0000-0001-7694-8250)
Henry Thompson (University of Edinburgh, School of Informatics, https://orcid.org/0000-0001-5490-1347)

If one goes to the trouble to organize an identifier system, then the desire that such a system should last as long as possible leads one to aspirationally say it’s a persistent identifier (PID) system. The unwillingness of the major browser suppliers to implement new URI schemes for PIDs initially hindered their use on the Web and this in turn inhibited widespread adoption. More recently a number of PID approaches have enjoyed very rapid growth as a result of a compromise: these PIDs participate in the World Wide Web by defining simple conversion rules mapping identifiers to actionable (‘http:’ and/or ‘https:’) forms and providing resolution servers that redirect requests for such forms to the appropriate destination.This approach has been widely adopted and is very successful, because it is so useful. An identifier’s actionable form leads, via the HTTP protocol and one or more redirections, to a web page that bears on the ground identity of the associated entity – or perhaps even directly to the entity itself, if the system is one for document entities that are naturally provided as web pages. The nature of the retrieved web page varies from one system to the next.

A confusion arose, however, over claims in various technical specifications (URIs, HTTP, Web Architecture) that the normal case is for the protocol to yield a “representation” of the “resource” “identified” by the URI. None of these terms is adequately defined by the specifications, and initially the language was not taken as normative. Those deploying identifier systems took the HTTP “resource” to be the entity associated with an identifier, and understood the “resource” as being “identified” by the URI, but it was never clear what was, or wasn’t, a “representation” of a given entity/resource: a description of the resource, the resource itself, a version of the resource, instructions on how to find the resource, etc. Sixteen years ago, in an attempt to clarify the intent of this part of the theory of URIs, and to allow applications to usefully and uniformly exploit the idea that an HTTP 200 response must deliver a “representation” of the “resource”, Tim Berners-Lee asked the W3C Technical Architecture Group to consider what came to be known as the httpRange-14 issue. It’s now 13 years after the TAG gave advice which almost no one was happy with, and 5 years after work on issue httpRedirections-57 (which superseded httpRange-14) ground to a halt. There’s still no consensus on whether it’s OK to return landing pages with a 200 status in response to requests for pictures or publications, but the Web seems to be working nonetheless, and no one seems to be bothered much anymore.

The provision of HTTP-based resolution services has stimulated widespread support for the use of identifier systems with Web resolution, particularly in the scholarly journal publication context. Those setting up HTTP resolvers responsible for identifier systems must decide which HTTP response code should be used. The TAG’s advice sows doubt on the use of the 200 response code when the response would have been a landing page, and many resolvers avoid 200 regardless and use redirection for administrative purposes, for example

‘https://dx-doi-org.pluma.sjfc.edu/10.1109/5.771073’ to

‘http://ieeexplore.ieee.org/document/771073/?reload=true’ for the DOI

‘10.1109/5.771073’, or ‘https://identifiers.org/uniprot/A0A022YWF9’ to

‘http://www.uniprot.org/uniprot/A0A022YWF9’ for the Uniprot identifier

‘A0A022YWF9’.

So the response should be a redirection, but what kind, 301, 302, or 303? (Or 307, which is almost the same as 302.) A 301 redirect seems to say that the URI is not persistent (since its target is deemed “more persistent”). A 302 redirect seems to say that the response could have come via a 200, and so suffers the same fate as 200. That leaves 303, as hinted at in the TAG’s advice. This idea got some traction: Ten years ago a Semantic Web interest group promoted the TAG’s advice in a published note, and seven years ago one of us wrote a blog post giving the same advice for resolvers for PIDs in publishing.

However, not only is there neither consensus nor general utility around this strict understanding of the use of the various response codes – that is, that resolution to a landing page is inconsistent with a 200 (and a posteriori therefore with a 302) – but also the range of usage patterns for redirection of HTTP requests has grown and ramified over time as the Web has grown and become more complex. It’s on the face of it unlikely that a mere three response codes can capture all the resulting complexity or cover the space of outcomes (in terms of e.g. what ends up in the browser address bar or what search engines index a page under) that a page owner might like to signal.

We find in practice that some PID redirections are ending up (usually after further publisher-local redirects) at the “identified” document, some at landing pages, and some at one or the other depending on the requesting site, for example in the case of paywalled material.

In the absence of a rethinking of the whole 3xx space, it seems to us that only the 301 vs. 302 distinct ion (roughly, 301 = permanent = please fix the link, and 302 = temporary = don’t change the link) is well understood and more or less consistently treated, whereas for 303, web servers are not very consistent and both search engine and citation crawler behaviours are at best inconsistent and at worst downright unhelpful.

So, we believe it is in both users’ and publishers’ interests for resolvers of actionable-form PIDs to use 302 redirects, not 303.

If we want to help machines better understand the resource that a DOI points at, we have to explore using more nuanced mechanisms.

Just using 302 for the first redirect doesn’t do everything necessary to effectively support the emerging PID+redirection architecture. It’s at the end of the redirect chains that we need more: a standardised way to find the PID back at the start of the chain. The ‘cite-as’ proposal does exactly this, and we hope it’s quickly approved and widely adopted. Once that happens a proposal for augmenting browser (and API) behaviour to prefer, or at least offer, the ‘cite-as’ link for bookmarking and copying will be needed.

Using AWS S3 as a large key-value store for Chronograph

Joe Wass — Tue, 02 Aug 2016 00:00:00 +0000

One of the cool things about working in Crossref Labs is that interesting experiments come up from time to time. One experiment, entitled “what happens if you plot DOI referral domains on a chart?” turned into the Chronograph project. In case you missed it, Chronograph analyses our DOI resolution logs and shows how many times each DOI link was resolved per month, and also how many times a given domain referred traffic to DOI links per day.

We’ve released a new version of Chronograph. This post explains how it was put together. One for the programmers out there.

Big enough to be annoying

Chronograph sits on the boundary between normal-sized data and large-enough-to-be-annoying-size data. It doesn’t store data for all DOIs (it includes only those that are used on average once a day), but it has information on up to 1 million DOIs per month over about 5 years, and about 500 million data points in total.

Storing 500 million data points is within the capabilities of a well-configured database. In the first iteration of Chronograph a MySQL database was used. But that kind of data starts to get tricky to back up, move around and index.

Every month or two new data comes in for processing, and it needs to be uploaded and merged into the database. Indexes need to be updated. Disk space needs to be monitored. This can be tedious.

Key values

Because the data for a DOI is all retrieved at once, it can be stored together. So instead of a table that looks like

<td>
<span >2010-01-01</span>
</td>
<td>
<span >5</span>
</td>

<td>
<span >2010-02-01</span>
</td>
<td>
<span >7</span>
</td>

<td>
<span >2010-03-01</span>
</td>
<td>
<span >3</span>
</td>

10.5555/12345678

Instead we can store

<td>
{&#8220;2010-01-01&#8221;: 5, &#8220;2010-02-01&#8221;: 7, &#8220;2010-03-01&#8221;: 3}
</td>

10.5555/12345678

This is much lighter on the indexes and takes much less space to store. However, it means that adding new data is expensive. Every time there’s new data for a month, the structure must be parsed, merged with the new data, serialised and stored again millions of times over.

After trials with MySql, MongoDB and MapDB, this approach was taken with MySQL in the original Chronograph.

Keep it Simple Storage Service Stupid

In the original version of Chronograph the data was processed using Apache Spark. There are various solutions for storing this kind of data, including Cassandra, time-series databases and so on.

The flip side of being able to do interesting experiments is wanting them to stick around without having to bother a sysadmin. The data is important to us, but we’d rather not have to worry about running another server and database if possible.

Chronograph fits into the category of ‘interesting’ rather than ‘mission-critical’ projects, so we’d rather not have to maintain expensive infrastructure if possible.

I decided to look into using Amazon Web Services Simple Storage Service (AWS S3) to store the data. AWS itself is a key-value store, so it seems like a good fit. S3 is a great service because, as the name suggests, it’s a simple service for storing a large number of files. It’s cheap and its capabilities and cost scale well.

However, storing and updating up to 80 million very small keys (one per DOI) isn’t very clever, and certainly isn’t practical. I looked at DynamoDB, but we still face the overhead of making a large number of small updates.

Is it weird?

In these days of plentiful databases with cheap indexes (and by ‘these days’ I mean the 1970s onward) it seems somehow wrong to use plain old text files. However, the whole Hadoop “Big Data” movement was predicated on a return to batch processing files. Commoditisation of services like S3 and the shift to do more in the browser have precipitated a bit of a rethink. The movement to abandon LAMP stacks and use static site generators is picking up pace. The term ‘serverless architecture’ is hard to avoid if you read certain news sites.

Using Apache Spark (with its brilliant RDD concept) was useful for bootstrapping the data processing for Chronograph, but the new code has an entirely flat-file workflow. The simplicity of not having to unnecessarily maintain a Hadoop HDFS instance seems to be the right choice in this case.

Repurposing the Wheel

The solution was to use S3 as a big hash table to store the final data that’s served to users.

The processing pipeline uses flat files all the way through from input log files to projections to aggregations. At the penultimate stage of the pipeline blocks of CSV per DOI are produced that represent date-value pairs.

<td>
2010-01
</td>
<td>
2010-01-01,05<br /> 2010-02-01,02<br /> 2010-01-03,08<br /> &#8230;
</td>

<td>
2010-02
</td>
<td>
2010-02-1,10<br /> 2010-02-01,7<br /> 2010-02-03,22<br /> &#8230;
</td>

10.5555/12345678

At the last stage, these are combined into blocks of all dates for a DOI

<td>
2010-01
</td>
<td>
2010-01-01,05<br /> 2010-02-01,02<br /> 2010-01-03,08<br /> &#8230;<br /> 2010-02-1,10<br /> 2010-02-01,7<br /> 2010-02-03,22<br /> &#8230;
</td>

10.5555/12345678

The DOIs are then hashed into 12 bits and stored as chunks of CSV

day-doi.csv-chunks_8841:

10.1038/ng.3020
2014-06-24,4
2014-06-25,4
2014-06-26,3
...
10.1007/978-94-007-2869-1_7
2012-06-01,12
2012-06-02,8
...
10.1371/journal.pone.0145509
2016-02-01,13
2016-02-02,75
2016-02-03,30
...

There are 65,536 (0x000 to 0xFFFF) possible files, each with about a thousand DOIs worth of data in each.

When the browser requests data for a DOI, it is hashed and then the request for the appropriate file in S3 is made. The browser then has to perform a linear scan of the file to find the DOI it is looking for.

This is the simplest possible form of hash table: simple addressing with separate linear chaining. The hash function is a 16-bit mask of MD5, chosen because of availability in the browser. It does a great job of evenly distributing the DOIs over all 65,536 possible files.

Striking the balance

In any data structure implementation, there are balances to be struck. Traditionally these concern memory layout, the shape of the data, practicalities of disk access and CPU cost.

In this instance, the factors in play included the number of buckets that need to be uploaded and the cost of the browser downloading an over-large bucket. The size of the bucket doesn’t matter much for CPU (as far as the user is concerned it takes about the same time to scan 10 entries as it does 10,000), but it does make a difference asking user to download a 10kb bucket or a 10MB one.

I struck the balance at 4096 buckets, resulting in files of around 100k, which is the size of a medium sized image.

It works

The result is a simple system that allows people to look up data for millions of DOIs, without having to look after another server. It’s also portable to any other file storage service.

The approach isn’t groundbreaking, but it works.

Python and Ruby Libraries for accessing the Crossref API

Scott Chamberlain — Fri, 04 Mar 2016 00:00:00 +0000

I’m a co-founder with rOpenSci, a non-profit that focuses on making software to facilitate reproducible and open science. Back in 2013 we started to make an R client working with various Crossref web services. I was lucky enough to attend last year’s Crossref annual meeting in Boston, and gave one talk on details of the programmatic clients, and another higher level talk on text mining and use of metadata for research.

Crossref has a newish API encompassing works, journals, members, funders and more (check out the API docs), as well as a few other services. Essential to making the Crossref APIs easily accessible—and facilitating easy tool/app creation and exploration—are programmatic clients for popular languages. I’ve maintained an R client for a while now, and have been working on Python and Ruby clients for the past four months or so.

The R client falls squarely into the analytics/research use cases, while the Python and Ruby clients are ideal for general data access and use in web applications (the Javascript library below as well).

I’ve strived to make each client in idiomatic fashion according to the language. Due to this fact, there is not generally correspondence between the different clients with respect to data outputs. However, I’ve tried to make method names similar across Ruby and Python; although the R client is quite a bit older, so method names differ from the other clients and I’m resistant to changing them so as not to break current users’ projects. In addition, R users are likely to want a data.frame (i.e., table) of results, so we give back that - whereas with Python and Ruby we give back dictionaries and hashes, respectively.

Crossref clients

Python:
- Source: https://github.com/sckott/habanero
- Pypi: https://pypi.python.org/pypi/habanero
Ruby:
- Source: https://github.com/sckott/serrano
- Rubygems: https://rubygems.org/gems/serrano
- serrano also comes with a command line tool of the same name that’s installed when you install serrano (examples below)
R:
- Source: https://github.com/ropensci/rcrossref
- CRAN: https://cran.rstudio.com/web/packages/rcrossref/
Javascript:
- Source: https://github.com/scienceai/crossref
- NPM: https://www.npmjs.com/package/crossref

I’ll cover the Python, Ruby, and R libraries below.

Installation

Python

on the command line

pip install habanero

Ruby

on the command line

gem install serrano

in an R session

install.packages("rcrossref")

Examples

Output is indicated by the syntax #> in all examples below.

Python

in a Python REPL (e.g. iPython)

Import the Crossref module from within habanero, and initialize a client

from habanero import Crossref
cr = Crossref()

Query for the phrase “ecology”

x = cr.works(query = "ecology", limit = 5)

Index to various parts of the output

x['message']['total-results']
#> 276188

Extract similar data items from each result. The records are in the “items” slot

[ z['DOI'] for z in x['message']['items'] ]
#> [u'10.1002/(issn)1939-9170',
#> u'10.4996/fireecology',
#> u'10.5402/ecology',
#> u'10.1155/8641',
#> u'10.1111/(issn)1439-0485']

In habanero for some methods we require you to instantiate a client.

You can set a base URL and API key. This is a future looking feature

as Crossref API does not require an API key.

Note: I’ve tried to make sure habanero is Python 2 and 3 compatible. Hopefully you’ll find that’s true.

Ruby

in a Ruby repl (e.g., pry), load serrano

require 'serrano'

Query for “peerj” on the journals route

x = Serrano.journals(query: "peerj")

Collect just ISSN’s from each result

x['message']['items'].collect { |z| z['ISSN'] }
#> => [["2376-5992"], ["2167-8359"]]

Shell

The serrano command line tool is quite powerful if you are used to doing things there.

Here, search for one article; summary data is shown.

serrano works 10.1371/journal.pone.0033693
#> DOI: 10.1371/journal.pone.0033693
#> type: journal-article
#> title: Methylphenidate Exposure Induces Dopamine Neuron Loss and Activation of Microglia in the Basal Ganglia of Mice

There’s also a -json flag to give back JSON data, which can be parsed with the command line tool jq.

serrano works --filter=has_full_text:true --json --limit=5 | jq '.message.items[].link[].URL'
#> "http://api.wiley.com.pluma.sjfc.edu/onlinelibrary/tdm/v1/articles/10.1002%2F9781119208082.ch9"
#> "http://api.wiley.com.pluma.sjfc.edu/onlinelibrary/tdm/v1/articles/10.1002%2F9781119208082.index"
#> "http://api.wiley.com.pluma.sjfc.edu/onlinelibrary/tdm/v1/articles/10.1002%2F9781119208082.ch11"
#> "http://api.wiley.com.pluma.sjfc.edu/onlinelibrary/tdm/v1/articles/10.1002%2F9781119208082.ch15"
#> "http://api.wiley.com.pluma.sjfc.edu/onlinelibrary/tdm/v1/articles/10.1002%2F9781119208082.ch4"

In an R session, load rcrossref

library("rcrossref")

Search the works route for the phrase “science”

res <- cr_works(query = "science", limit = 5)
#> $meta
#> total_results search_terms start_index items_per_page
#> 1 4333827 science 0 5
#>
#> $data
#> Source: local data frame [5 x 23]
#>
#> alternative.id container.title created deposited DOI funder indexed
#> (chr) (chr) (chr) (chr) (chr) (chr) (chr)
#> 1 2013-11-21 2013-11-21 10.1126/science <NULL> 2015-12-27
#> 2 Science Askew 2004-11-26 2013-12-16 10.1887/0750307145/b426c18 <NULL> 2015-12-24
#> 3 2006-04-10 2010-07-30 10.1002/(issn)1557-6833 <NULL> 2015-12-25
#> 4 2013-08-27 2013-08-27 10.1002/(issn)1469-896x <NULL> 2015-12-27
#> 5 2013-12-19 2013-12-19 10.5152/bs. <NULL> 2015-12-28
#> Variables not shown: ISBN (chr), ISSN (chr), issued (chr), link (chr), member (chr), prefix (chr), publisher
#> (chr), reference.count (chr), score (chr), source (chr), subject (chr), title (chr), type (chr), URL
#> (chr), assertion (chr), author (chr)
#>
#> $facets
#> NULL

Index through to get the DOIs

res$data$DOI
#> [1] "10.1126/science" "10.1887/0750307145/b426c18" "10.1002/(issn)1557-6833"
#> [4] "10.1002/(issn)1469-896x" "10.5152/bs."

rcrossref also has faster versions of most functions with an underscore at the end (_) which only do the http request and give back json (e.g., cr_works_())

Comparison of Crossref Client Methods

After installation and loading the libraries above, the below methods are available

<th>
<span >Python</span>
</th>
<th>
<span >Ruby</span>
</th>
<th>
<span >R</span>
</th>

<td>
<span ><code>cr.works()</code></span>
</td>
<td>
<span ><code>Serrano.works()</code></span>
</td>
<td>
<span ><code>cr_works()</code></span>
</td>

<td>
<span ><code>cr.members()</code></span>
</td>
<td>
<span ><code>Serrano.members()</code></span>
</td>
<td>
<span ><code>cr_members()</code></span>
</td>

<td>
<span ><code>cr.funders()</code></span>
</td>
<td>
<span ><code>Serrano.funders()</code></span>
</td>
<td>
<span ><code>cr_funders()</code></span>
</td>

<td>
<span ><code>cr.types()</code></span>
</td>
<td>
<span ><code>Serrano.types()</code></span>
</td>
<td>
<span ><code>cr_types()</code></span>
</td>

<td>
<span ><code>cr.licenses()</code></span>
</td>
<td>
<span ><code>Serrano.licenses()</code></span>
</td>
<td>
<span ><code>cr_licenses()</code></span>
</td>

<td>
<span ><code>cr.journals()</code></span>
</td>
<td>
<span ><code>Serrano.journals()</code></span>
</td>
<td>
<span ><code>cr_journals()</code></span>
</td>

<td>
<span ><code>cr.members()</code></span>
</td>
<td>
<span ><code>Serrano.members()</code></span>
</td>
<td>
<span ><code>cr_members()</code></span>
</td>

<td>
<span ><code>cr.registration_agency()</code></span>
</td>
<td>
<span ><code>Serrano.registration_agency()</code></span>
</td>
<td>
<span ><code>cr_agency()</code></span>
</td>

<td>
<span ><code>cr.random_dois()</code></span>
</td>
<td>
<span ><code>Serrano.random_dois()</code></span>
</td>
<td>
<span ><code>cr_r()</code></span>
</td>

API Route
works
members
funders
types
licenses
journals
members
registration agency
random DOIs

Other Crossref Services

<th>
<span >Python</span>
</th>
<th>
<span >Ruby</span>
</th>
<th>
<span >R</span>
</th>

<td>
<span ><code>cn.content_negotiation()</code><a href="#footnote-1">[1]</a></span>
</td>
<td>
<span ><code>Serrano.content_negotiation()</code></span>
</td>
<td>
<span ><code>cr_cn()</code></span>
</td>

<td>
<span ><code>cn.csl_styles()</code><a href="#footnote-1">[1]</a></span>
</td>
<td>
<span ><code>Serrano.csl_styles()</code></span>
</td>
<td>
<span ><code>get_styles()</code></span>
</td>

<td>
<span ><code>counts.citation_count()</code><a href="#footnote-2">[2]</a></span>
</td>
<td>
<span ><code>Serrano.citation_count()</code></span>
</td>
<td>
<span ><code>cr_citation_count()</code></span>
</td>

Service
content negotiation
CSL styles
citation count

[1] from habanero import cn

[2] from habanero import counts

Features

These are supported in all 3 libraries:

Filters (see below)
Deep paging (see below)
Pagination
Verbose curl output

Filters

Filters (see API docs for details) are a powerful way to get closer to exactly what you want in your queries. In the Crossref API filters are passed as query parameters, and are comma-separated like filter=has-orcid:true,is-update:true . In the client libraries, filters are passed in idiomatic fashion according to the language.

Python

from habanero import Crossref
cr = Crossref()
cr.works(filter = {'award_number': 'CBET-0756451', 'award_funder': '10.13039/100000001'})

Ruby

require 'serrano'
Serrano.works(filter: {award_number: 'CBET-0756451', award_funder: '10.13039/100000001'})

library("rcrossref")
cr_works(filter=c(award_number=TRUE, award_funder='10.13039/100000001'))

Note how syntax is quite similar among languages, though keys don’t have to be quoted in Ruby and R, and in R you pass in a vector or list instead of a hash as in the other two.

All 3 clients have helper functions to show you what filters are available and what the options are for each filter.

<th>
<span >Python</span>
</th>
<th>
<span >Ruby</span>
</th>
<th>
<span >R</span>
</th>

<td>
<span ><code>filters.filter_names</code><a href="#footnote-3">[3]</a></span>
</td>
<td>
<span ><code>Serrano::Filters.names</code></span>
</td>
<td>
<span ><code>filter_names()</code></span>
</td>

<td>
<span ><code>filters.filter_details</code><a href="#footnote-3">[3]</a></span>
</td>
<td>
<span ><code>Serrano::Filters.filters</code></span>
</td>
<td>
<span ><code>filter_details()</code></span>
</td>

Action
Filter names
Filter details

[3] from habanero import filters

Deep paging

Sometimes you want a lot of data. The Crossref API has parameters for paging (see rows and offset), but large values of either can lead to long response times and potentially timeouts (i.e., request failure). The API has a deep paging feature that can be used when large data volumes are desired. This is made possible via Solr’s cursor feature (e.g., blog post on it). Here’s a run down of how to use it:

cursor: each method in each client library that allows deep paging has a cursor parameter that if you set to * will tell the Crossref API you want deep paging.
cursor_max: for boring reasons we need to have feedback from the user when they want to stop, since each request comes back with a cursor value that we can make the next request with, thus, an additional parameter cursor_max is used to indicate the number of results you want back.
limit: this parameter when not using deep paging determines number of results to get back. however, when deep paging, this parameter sets the chunk size. (note that the max. value for this parameter is 1000)

For example, cursor=”*” states that you want deep paging, cursor_max states maximum results you want back, and limit determines how many results per request to fetch.

Python

from habanero import Crossref
cr = Crossref()
cr.works(query = "widget", cursor = "*", cursor_max = 500)

Ruby

require 'serrano'
Serrano.works(query: "widget", cursor: "*", cursor_max: 500)

library("rcrossref")
cr_works(query = "widget", cursor = "*", cursor_max = 500)

Text mining clients

Just a quick note that I’ve begun a few text-mining clients for Python and Ruby, focused on using the low level clients discussed above.

Python: https://github.com/sckott/pyminer
Ruby: https://github.com/sckott/textminer

Do try them out!

DOIs and matching regular expressions

Andrew Gilmartin — Tue, 11 Aug 2015 00:00:00 +0000

We regularly see developers using regular expressions to validate or scrape for DOIs. For modern Crossref DOIs the regular expression is short

/^10.\d{4,9}/[-._;()/:A-Z0-9]+$/i

For the 74.9M DOIs we have seen this matches 74.4M of them. If you need to use only one pattern then use this one.

The other 500K are mostly from Crossref’s early days when the battle between “human-readable” identifiers and “opaque” identifiers was still being fought, the web was still new, and it was expected that “doi” would become as well a supported URI schema name as “gopher”, “wais”, …. Ok, that didn’t go so well.

An early Crossref’s member was John Wiley & Sons. They faced the need to design DOIs without much prior work to lean on. Many of those early DOIs are not expression friendly. Nevertheless, they are still valid and valuable permanent links to the work’s version of record. You can catch 300K more DOIs with

/^10.1002/[^\s]+$/i

While the DOI caught is likely to be the DOI within the text it may also contain trailing characters that, due to the lack of a space, are caught up with the DOI. Even the recommended expression catches DOIs ending with periods, colons, semicolons, hyphens, and underscores. Most DOIs found in the wild are presented within some visual design program. While pleasant to look at the visual design can misdirect machines. Is the period at the end of the line part of the DOI or part of the design? Is that endash actually a hyphen? These issues lead to a DOI bycatch.

Adding the following 3 expressions with the previous 2 leaves only 72K DOIs uncaught. To catch these 72K would require a dozen or more additional patterns. Each additional pattern, unfortunately, weakens the overall precision of the catch. More bycatch.

/^10.\d{4}/\d+-\d+X?(\d+)\d+<[\d\w]+:[\d\w]*>\d+.\d+.\w+;\d$/i

/^10.1021/\w\w\d++$/i

/^10.1207/[\w\d]+\&\d+_\d+$/i

Crossref is not the only DOI Registration Agency and while our members account for 65-75% of all registered DOIs this means there are tens of millions of DOIs that we have not seen. Luckily, the newer RAs and their publishers can copy our successes and avoid our mistakes.

Content Negotiation for Crossref DOIs

Geoffrey Bilder — Tue, 19 Apr 2011 00:00:00 +0000

So does anybody remember the posting DOIs and Linked Data: Some Concrete Proposals?

Well, we went with option “D.”

From now on, DOIs, expressed as HTTP URIs, can be used with content-negotiation.

Let’s get straight to the point. If you have curl installed, you can start playing with content-negotiation and Crossref DOIs right away:

curl -D - -L -H   “Accept: application/rdf+xml” “http://dx.doi.org.pluma.sjfc.edu/10.1126/science.1157784”

curl -D - -L -H   “Accept: text/turtle” “http://dx.doi.org.pluma.sjfc.edu/10.1126/science.1157784”

curl -D - -L -H   “Accept: application/atom+xml” “http://dx.doi.org.pluma.sjfc.edu/10.1126/science.1157784”

Or if you are already using Crossref’s “unixref” format:

curl -D - -L -H “Accept: application/unixref+xml” “http://dx.doi.org.pluma.sjfc.edu/10.1126/science.1157784&#8221;

This will work with over 46 million Crossref DOIs as of today, but the beauty of the setup is that from now on, any DOI registration agency can enable content negotiation for their constituencies as well. DataCite- we’re looking at you 😉 .

It also means that, as registration agency members (Crossref publishers, for instance) start providing more complete and richer representations of their content, we can simply redirect content-negotiated requests directly to them.

We expect that that this development will round-out Crossref’s efforts to support standard APIs including OpenURL and OAI_PMH and we look forward to seeing DOIs increasingly used in linked data applications.

Finally, Crossref would just like to thank the IDF and CNRI for their hard work on this as well as Tony Hammond and Leigh Dodds for their valuable advice and persistent goading.

Ubiquity commands for Crossref services

Geoffrey Bilder — Wed, 03 Dec 2008 00:00:00 +0000

So the other day Noel O’Boyle made me feel guilty when he pinged me and asked about the possibility using one of the Crossref APIs for creating a Ubiquity extension. You see, I had played with the idea myself and had not gotten around to doing much about it. This seemed inexcusable- particularly given how easy it is to build such extensions using the API we developed for the WordPress and Moveable Type plugins that we announced earlier in the year. So I dug up my half-finished code, cleaned it up a bit and have posted the results.

Note that the back-end that supports the plugins has been moved to more stable machines and the index is now being automatically updated with journal and conference proceeding deposits (sorry, no books yet).

Also note that we are hoping that others will look at the code for the WordPress, Moveable Type and Ubiquity plugins and create more such extensions. If you do, please let us know about them at citation-plugin@crossref.org.

Kay Sera Sera

Crossref — Tue, 20 Feb 2007 00:00:00 +0000

Not specifically publishing-related, but here is a fun ~~rant~~ interview with Alan Kay titled The PC Must Be Revamped—Now.

My favorite bit…

“…in the last few years I’ve been asking computer scientists and programmers whether they’ve ever typed E-N-G-E-L-B-A-R-T into Google-and none of them have. I don’t think you could find a physicist who has not gone back and tried to find out what Newton actually did. It’s unimaginable. Yet the computing profession acts as if there isn’t anything to learn from the past, so most people haven’t gone back and referenced what Engelbart thought. ”

Stick this in your pipe…

Crossref — Mon, 19 Feb 2007 00:00:00 +0000

Rob Cornelius has a practical little demo of using Yahoo! pipes against some Ingenta feeds.

Like Tony, I keep experiencing speed/stability problems while accessing pipes so I haven’t yet become a crack-pipes-head.

Ruby Makes A-List

Tony Hammond — Thu, 12 Oct 2006 00:00:00 +0000

Um, well. Seems according to O’Reilly Ruby that Ruby is now a mainstream language.

“The Ruby programming language just made the A-list on the TIOBE Programming Community Index, and Ruby is now listed as a mainstream programming language. For the past three or four years Ruby has consistently placed in the high 20’s in this index, but is now placed as the 13th most popular programming language!”

(No language wars, but I am, I will confess, a big admirer - for some time.)