Wikipedia on Crossref

A transparent record of life after publication

Madeleine Watson — Wed, 01 Nov 2017 00:00:00 +0000

Crossref Event Data and the importance of understanding what lies beneath the data.

Some things in life are better left a mystery. There is an argument for opaqueness when the act of full disclosure only limits your level of enjoyment: in my case, I need a complete lack of transparency to enjoy both chicken nuggets and David Lynch films. And that works for me. But metrics are not nuggets. Because in order to consume them, you really need to know how they’re made. Knowing the provenance of data, along with the context with which it was derived, provides everyone with the best chance of creating indicators which are fit for purpose. This is just one of the reasons why we built the Event Data infrastructure with transparency in mind.

The transparency problem

For the scholarly community, alternative metrics to citation count (‘altmetrics’) are becoming increasingly popular as they can offer rich and expedited insight into today’s diverse and dynamic research environment. Research artifacts undergo an extended life online as they’re linked, shared, saved and discussed in forums both within and beyond the traditional academic ecosystem. Data on these interactions are initially fragmented and buried within platforms like social media, blogs and news sites. Downstream, there are several value-add services that collate and present that data as a single, aggregated count. We see individual data points like ‘paper X was tweeted 22 times’, and ‘paper X is referenced 16 times on Wikipedia’ being combined, homogenised, weighted and expressed as a single figure, a calculated number serving as a proxy for value. But altmetrics alone don’t tell the whole story, and how they are calculated is not without idiosyncrasy or politics. As we each have our own unique voice and perspective, we need to ensure we understand the lenses through which these metrics are made in order to consume them effectively.

The 2015 Metric Tide report highlighted transparency as one of the five dimensions of responsible metrics. Having access to the context used to create a metric — the provenance of the original data as well as full transparency around its extraction, processing and aggregation — helps consumers to use the data meaningfully and allows for comparison across third-party vendors. But transparency is difficult to achieve when, as the report notes, the systems and infrastructure for collecting and curating altmetrics-style data are fragmented and have limited interoperability.

In the academic community, underlying centralised systems include ORCIDs to identify people and DOIs to identify items. But we’re missing a transparent, centralised infrastructure for describing and recording the relationships between objects and resources¹. These relationships, or links, occur outside publisher platforms and can provide valuable information about the interconnectivity and dissemination of research. Dedicated infrastructure for collecting these relationships would provide a data source for those interested in altmetrics to build upon.

Figure 1.1 Example of some relationships between articles and activity on the web

At Crossref, we call these relationships Events. An Event is the record of a claim made about the existence of a relationship between a registered content item (i.e. a DOI) and a specific activity on the web. Events include:

a DataCite dataset DOI contains a link to a Crossref article DOI
an article was referenced in Wikipedia
an article was mentioned on Twitter
an article has a Hypothes.is annotation
a blog contains a link to an article

In collaboration with DataCite, we are collecting Events for the DOIs registered with our organisations and are making that data available for others in the community to use. This is the Event Data infrastructure, with which we’re plugging the gap in open scholarly relationships infrastructure. https://doi-org.pluma.sjfc.edu/10.64000/kattr-5k219

The Event Data infrastructure

Crossref and DataCite have for many years provided a centralised location for bibliographic metadata and links, and a facility to help our members register Persistent Identifiers (DOIs) for their content. With nearly 100 million DOIs registered with Crossref, we know where research lives. Which got us thinking — could we use these links to find out more about the journey research undertakes after publication? Could we express these interactions as links without any aggregation or counts so it could be maximally reused? And if so, could we then provide this data in an open, centralised, structured format? The answer was yes, subject to some challenges:

Querying for individual DOIs wasn’t scalable for our full corpus of 100 million items, so we had to find something else.
Not everyone uses the DOI link (not a surprise!). Most people will link directly to the publisher’s site. This means we need to look for links using both the DOI and article landing page URLs.
When we find people referring to registered content using its landing page, we find the DOI for that content item so that the link can be referenced in our data set in a stable, link-rot-proof way.
We don’t always know the article landing page URL for every DOI upfront because like many relationships, the one between DOIs and URLs is complicated.

We began by asking the wrong questions and as a result we got the wrong type of data back: instead of returning a record of individual actions, we were returning aggregated counts. Aside from not meeting our use case, aggregation requires the curation of an ever-churning dataset in order to keep totals updated, which is not scalable for the number of DOIs in our corpus.

We soon learnt to ask the right questions. One pivotal change in approach was that instead of counts, we asked instead ‘what activity is happening on Twitter for this article?’. Our data went from ‘DOI X was mentioned 20 times on Twitter as of this date’ to ‘tweet X mentions DOI X on this date’. The data are now represented as a subject-verb-object triple:

Figure 1.2 Triple table.

Ultimately this has allowed us to represent actions like Wikipedia page edits as individual atomic actions (i.e an Event) rather than as a dataset that changes over time. Being open about the provenance of altmetrics with Event Data Crossref Event Data (the Crossref-specific service powered by the shared Event Data infrastructure) has evolved beyond a link store to become a continual stream of Events; each Event tells a new part of the story. Rather than constantly updating an Event whenever a new action takes place, we add a new one instead:

Figure 1.3 A Wikipedia Event.

Events answer a whole range of questions, such as:

What links to what?
How was the link made?
Which Agent collected the Event?
Which data source?
When was the link observed?
When do we think the link actually happened?
What algorithms were used to collect it?
Where’s the evidence?

We’re collecting data from a diverse range of platforms including Twitter, Wikipedia, blogs and news sites, Reddit, StackExchange, Wordpress.com and Hypothes.is. This means that when we observe a link in these platforms to what we think is a DOI, we create an Event and a corresponding Evidence Record to represent our observation. We also have Events to represent the links between research items registered with Crossref and DataCite - for example, when a Crossref DOI cites a DataCite DOI and vice versa.

The provenance of the data is fully transparent and is made available to everyone via an open API. We call this the evidence trail. The record of each link (‘Events’) as well as the corresponding evidence can then be used to feed into tools for impact measurement, discoverability, collaboration and network analysis.

Therefore, one application of Event Data is as an underlying, transparent data source for altmetrics calculations. For example, you might want to know the total number of times your paper has been mentioned on Twitter to date. If I told you that the number was 22, what does that actually mean? Do you know whether I counted both tweets and retweets? Do you consider both of these actions as equal? Is the sentiment of the tweet important to you? Was it a human or a bot that initiated a tweet? Are you interested in tweets containing links to multiple representations of your paper or do you only want to track mentions of your version of record (the final published copy)? With Event Data as your underlying data source, you can answer these questions.

Not only transparent in data, transparent by design

The National Information Standards Organisation (NISO), a US organisation responsible for technical standards for publishing, bibliographic and library applications, has developed a set of recommendations for transparency in their Alternative Assessment Metrics Project report, as well as a Code of Conduct for both altmetric practitioners and aggregators that aims to help improve the quality of altmetrics data. The working groups recognised that without transparency and conforming to a recognised standard, altmetric indicators “are difficult to assess, and thus may be seen as less reliable for purposes of measuring influence or evaluation”¹.

Crossref Event Data is one of the example altmetric data providers listed in the NISO recommendations. My colleague Joe Wass participated in the development and specification of the NISO “Altmetrics Recommended Practices on Data Metrics, Alternative Outputs, and Persistent Identifiers” at the same time as we were working with DataCite on Event Data, so they have mutually informed one another.

Figure 1.4 Martin Fenner (DataCite) and Joe Wass (Crossref) drawing plans for the Event Data infrastructure.

The outcome of our involvement in the NISO recommendations is that Crossref Event Data is a service that is transparent by design. We have opened up our entire extraction and processing workflow so that we can clearly demonstrate the context and environment that was used to generate an Event. This evidence is a core component of our transparency-first principle.

Building services on Event Data

There are some really exciting ways that people are already using Event Data, and we’re still only in beta. Our aim has always been to create an open, portable, transparent data set that can be used by our diverse community including researchers, application developers, publishers, funders and third-party service providers. We have already seen data from our service used in recent research studies, impact reports and even a front-end tool. Launched recently as a prototype, ImpactStory’s Paperbuzz.org uses Event Data as one of its data sources for tracking the online buzz around scholarly articles. Jason Priem, cofounder of ImpactStory, notes:

“Because Crossref Event Data is completely open data, we believe it’s a game-changer for altmetrics. Our latest project, Paperbuzz.org, is just the first of a whole constellation of upcoming tools that will add value on top of Crossref’s open data.”

We are working towards launching Crossref Event Data as a production service. In the meantime though, please do take a look at our comprehensive User Guide. Hopefully you’ll be inspired to go make something cool using the data! Events are being collected constantly; take a look below as they stream in from our data sources or visit our live stream demo site to watch in real time.

Figure 1.5 Screen capture of Crossref Event Data live stream demo.

As the service matures, we’ll continue to add new platforms to track and I also encourage anyone with article link data to get in touch to discuss how we can share it with the community via Event Data.

For researchers in particular, I’m really keen to hear your thoughts on our data model and about the things we could additionally provide you with from an infrastructure perspective that would best support your research needs.

And if you’re a publisher, take a look at our Event Data best practice guidelines — there’s some really important information in there about how you can help give us the best chance possible of collecting Events for your registered content.

And finally, if you’re a consumer of altmetrics data, I encourage you to ask questions. Ask your altmetrics vendors about how they gather their data and what context they apply to the aggregation of the metrics they supply. Ask yourself what behaviours you are interested in tracking and equally those you are not. Think about the endgame; about the type of impact you’re truly trying to measure and the story you want to tell. Because it’s these questions that will help you choose indicators that are the best fit for your own unique narrative.

This content is cross-posted on eLife Labs.

References

¹ Bilder, Geoffrey; Lin, Jennifer; Neylon, Cameron (2015): What exactly is infrastructure? Seeing the leopard’s spots. Retrieved: Oct 16, 2017; https://doi-org.pluma.sjfc.edu/10.6084/m9.figshare.1520432.v1

² NISO, Outputs of the NISO Alternative Assessment Metrics Project. Retrieved: 6th October 2017; https://www.niso.org/publications/rp-25-2016-altmetrics , p.2.

Event Data enters Beta

Jennifer Kemp — Wed, 05 Jul 2017 00:00:00 +0000

We’ve been talking about it at events, blogging about it on our site, living it, breathing it, and even sometimes dreaming about it, and now we are delighted to announce that Crossref Event Data has entered Beta.

A collaborative initiative by Crossref and DataCite, Event Data offers transparency around the way interactions with scholarly research occur online, allowing you to discover where it’s bookmarked, linked, liked, shared, referenced, commented on etc., across the web, and beyond publisher platforms.

The name Event Data reflects the nature of the service, as it collects and stores digital actions that occur on the web, from the quick and simple, such as bookmarking and referencing, through to deeper interconnectivity such as exposing the links between research artifacts. Each individual action is timestamped and recorded in our system as an Event, and made available to the community via an API.

Event Data will be available for absolutely anyone to use; publishers, third party vendors, editors, bibliometricans, researchers, authors, funders etc., and with tens of thousands of events occurring every day, there’s a wealth of insight to be gained for those interested in analyzing and interpreting the data.

It’s important to note that Event Data does not provide metrics. What is does provide is the raw data to help you facilitate your own analysis, giving you the freedom to integrate the data into your own systems.

We are currently working very closely with a few organisations with specific use cases who are helping us to test and refine Beta before we launch our production service later this year. If you decide to take a look at Beta yourself, all the data you collect from Event Data is licensed for public sharing and reuse according to our Terms of Use.

Until Event Data is in production mode, we do not recommend building any commercial or customer-based tools off the data. If you are not in the Beta test group but are interested in participating, please contact me below. For more information about Event Data, please see our user guide.

Please contact me, Jennifer Kemp—Outreach Manager for Event Data—with any questions.

Using AWS S3 as a large key-value store for Chronograph

Joe Wass — Tue, 02 Aug 2016 00:00:00 +0000

One of the cool things about working in Crossref Labs is that interesting experiments come up from time to time. One experiment, entitled “what happens if you plot DOI referral domains on a chart?” turned into the Chronograph project. In case you missed it, Chronograph analyses our DOI resolution logs and shows how many times each DOI link was resolved per month, and also how many times a given domain referred traffic to DOI links per day.

We’ve released a new version of Chronograph. This post explains how it was put together. One for the programmers out there.

Big enough to be annoying

Chronograph sits on the boundary between normal-sized data and large-enough-to-be-annoying-size data. It doesn’t store data for all DOIs (it includes only those that are used on average once a day), but it has information on up to 1 million DOIs per month over about 5 years, and about 500 million data points in total.

Storing 500 million data points is within the capabilities of a well-configured database. In the first iteration of Chronograph a MySQL database was used. But that kind of data starts to get tricky to back up, move around and index.

Every month or two new data comes in for processing, and it needs to be uploaded and merged into the database. Indexes need to be updated. Disk space needs to be monitored. This can be tedious.

Key values

Because the data for a DOI is all retrieved at once, it can be stored together. So instead of a table that looks like

<td>
<span >2010-01-01</span>
</td>
<td>
<span >5</span>
</td>

<td>
<span >2010-02-01</span>
</td>
<td>
<span >7</span>
</td>

<td>
<span >2010-03-01</span>
</td>
<td>
<span >3</span>
</td>

10.5555/12345678

Instead we can store

<td>
{&#8220;2010-01-01&#8221;: 5, &#8220;2010-02-01&#8221;: 7, &#8220;2010-03-01&#8221;: 3}
</td>

10.5555/12345678

This is much lighter on the indexes and takes much less space to store. However, it means that adding new data is expensive. Every time there’s new data for a month, the structure must be parsed, merged with the new data, serialised and stored again millions of times over.

After trials with MySql, MongoDB and MapDB, this approach was taken with MySQL in the original Chronograph.

Keep it Simple Storage Service Stupid

In the original version of Chronograph the data was processed using Apache Spark. There are various solutions for storing this kind of data, including Cassandra, time-series databases and so on.

The flip side of being able to do interesting experiments is wanting them to stick around without having to bother a sysadmin. The data is important to us, but we’d rather not have to worry about running another server and database if possible.

Chronograph fits into the category of ‘interesting’ rather than ‘mission-critical’ projects, so we’d rather not have to maintain expensive infrastructure if possible.

I decided to look into using Amazon Web Services Simple Storage Service (AWS S3) to store the data. AWS itself is a key-value store, so it seems like a good fit. S3 is a great service because, as the name suggests, it’s a simple service for storing a large number of files. It’s cheap and its capabilities and cost scale well.

However, storing and updating up to 80 million very small keys (one per DOI) isn’t very clever, and certainly isn’t practical. I looked at DynamoDB, but we still face the overhead of making a large number of small updates.

Is it weird?

In these days of plentiful databases with cheap indexes (and by ‘these days’ I mean the 1970s onward) it seems somehow wrong to use plain old text files. However, the whole Hadoop “Big Data” movement was predicated on a return to batch processing files. Commoditisation of services like S3 and the shift to do more in the browser have precipitated a bit of a rethink. The movement to abandon LAMP stacks and use static site generators is picking up pace. The term ‘serverless architecture’ is hard to avoid if you read certain news sites.

Using Apache Spark (with its brilliant RDD concept) was useful for bootstrapping the data processing for Chronograph, but the new code has an entirely flat-file workflow. The simplicity of not having to unnecessarily maintain a Hadoop HDFS instance seems to be the right choice in this case.

Repurposing the Wheel

The solution was to use S3 as a big hash table to store the final data that’s served to users.

The processing pipeline uses flat files all the way through from input log files to projections to aggregations. At the penultimate stage of the pipeline blocks of CSV per DOI are produced that represent date-value pairs.

<td>
2010-01
</td>
<td>
2010-01-01,05<br /> 2010-02-01,02<br /> 2010-01-03,08<br /> &#8230;
</td>

<td>
2010-02
</td>
<td>
2010-02-1,10<br /> 2010-02-01,7<br /> 2010-02-03,22<br /> &#8230;
</td>

10.5555/12345678

At the last stage, these are combined into blocks of all dates for a DOI

<td>
2010-01
</td>
<td>
2010-01-01,05<br /> 2010-02-01,02<br /> 2010-01-03,08<br /> &#8230;<br /> 2010-02-1,10<br /> 2010-02-01,7<br /> 2010-02-03,22<br /> &#8230;
</td>

10.5555/12345678

The DOIs are then hashed into 12 bits and stored as chunks of CSV

day-doi.csv-chunks_8841:

10.1038/ng.3020
2014-06-24,4
2014-06-25,4
2014-06-26,3
...
10.1007/978-94-007-2869-1_7
2012-06-01,12
2012-06-02,8
...
10.1371/journal.pone.0145509
2016-02-01,13
2016-02-02,75
2016-02-03,30
...

There are 65,536 (0x000 to 0xFFFF) possible files, each with about a thousand DOIs worth of data in each.

When the browser requests data for a DOI, it is hashed and then the request for the appropriate file in S3 is made. The browser then has to perform a linear scan of the file to find the DOI it is looking for.

This is the simplest possible form of hash table: simple addressing with separate linear chaining. The hash function is a 16-bit mask of MD5, chosen because of availability in the browser. It does a great job of evenly distributing the DOIs over all 65,536 possible files.

Striking the balance

In any data structure implementation, there are balances to be struck. Traditionally these concern memory layout, the shape of the data, practicalities of disk access and CPU cost.

In this instance, the factors in play included the number of buckets that need to be uploaded and the cost of the browser downloading an over-large bucket. The size of the bucket doesn’t matter much for CPU (as far as the user is concerned it takes about the same time to scan 10 entries as it does 10,000), but it does make a difference asking user to download a 10kb bucket or a 10MB one.

I struck the balance at 4096 buckets, resulting in files of around 100k, which is the size of a medium sized image.

It works

The result is a simple system that allows people to look up data for millions of DOIs, without having to look after another server. It’s also portable to any other file storage service.

The approach isn’t groundbreaking, but it works.

HTTPS and Wikipedia

Joe Wass — Tue, 31 May 2016 00:00:00 +0000

This is a joint blog post with Dario Taraborelli, coming from WikiCite 2016.

In 2014 we were taking our first steps along the path that would lead us to Crossref Event Data. At this time I started looking into the DOI resolution logs to see if we could get any interesting information out of them. This project, which became Chronograph, showed which domains were driving traffic to Crossref DOIs.

You can read about the latest results from this analysis in the “Where do DOI Clicks Come From” blog post.

Having this data tells us, amongst other things:

where people are using DOIs in unexpected places
where people are using DOIs in unexpected ways
where we knew people were using DOIs but the links are more popular than we realised

By the time the ALM Workshop 2014 rolled around there was some preliminary data and we realised that Wikipedia came into the third category. There are lots of DOIs in Wikipedia and people click them!

I met with Dario Taraborelli, head of research at the Wikimedia Foundation, and shared the data. Dario — who co-authored in 2010 the Altmetrics Manifesto — has been interested in understanding how scholarly citations are used in Wikipedia. Over the years, Wikipedia contributors have made extensive use of references to the scientific literature using DOIs, and by doing so they have created a resource that represents today in many ways the “front matter to all research”. There is growing interest in the community in understanding how DOIs are being used in Wikipedia and in non traditional scholarship.

During our discussions the subject of Wikipedia’s gradual transition to HTTPS was raised: we anticipated that this change would affect our data gathering.

Changes

When you’re reading webpage and click on a link to another page, your web browser will usually tell the server of that second page the last page you were on. This forms the basis of trackers like Google Analytics.

In the days before HTTPS, the next site would know the full URL that you were previously on. With the change to HTTPS, this was reduced to just sending the domain name and not the full URL, or no data at all if you click from an HTTPS page to HTTP.

DOI hyperlinks are just like any other hyperlink, and are mostly HTTP not HTTPS.

Up until 2015, Wikipedia was served over HTTP, only switching to HTTPS when users were logged in or if they requested it. The Wikimedia Foundation started planning to move to HTTPS and we knew that if they did that, and continued to use HTTP DOIs then we would lose valuable research data.

A Plan

We decided that the best course of action was to try and change the DOIs in Wikipedia to use HTTPS. Simple, right?

After some further research, Dario posted a proposal on how to mitigate the impact of the HTTPS rollout, to make sure that Wikipedia can still signal its importance as a traffic source, while preserving the privacy of its users. Discussion followed and the conclusion was to change the format of every single DOI on Wikipedia, which fortunately could be done without having to edit millions of pages. You can read the full story in this post from a year ago.

The result of this effort was that well in advance of the HTTPS switchover, the DOI links were ready to continue reporting referral data.

The Switch

In June 2015 the Wikimedia foundation made the announcement that they were finalising the switch, and that within a few weeks all traffic would be HTTPS.

We held our breath. Would it work? Would we lose all referral data from Wikipedia sites? In February 2016 the last piece of the puzzle fell into place as Wikipedia gained a ‘meta referrer’ tag to explicitly specify how they would like referrers to be sent: a detailed report on the effect of this change is coming up on the Wikimedia Foundation’s blog.

The results

As detailed in the last blog post the traffic that we measured coming from Wikipedia doesn’t seem to have slowed down during 2015:

I’d call that a success! Over the period covered in the graph, Wikipedia remained prominent as a non-publisher referral of traffic to DOIs.

Looking at the balance of HTTP vs HTTPS traffic coming from wikipedia.org, the switchover was dramatic:

Thank you to Dario Taraborelli, Nemo (Federico Leva), Aaron Halfaker, Alex Stinson and everyone who put in this effort.

I’ll leave the last word to Dario:

It’s great to see this data. It shows that the switchover happened successfully, which better protects the privacy of our users whilst still reporting the fact that Wikipedia is a prominent source of traffic. This is important validation of the increasing role that Wikipedia plays in the education and scientific community.

Where do DOI clicks come from?

Joe Wass — Thu, 19 May 2016 00:00:00 +0000

As part of our Event Data work we’ve been investigating where DOI resolutions come from. A resolution could be someone clicking a DOI hyperlink, or a search engine spider gathering data or a publisher’s system performing its duties. Our server logs tell us every time a DOI was resolved and, if it was by someone using a web browser, which website they were on when they clicked the DOI. This is called a referral.

This information is interesting because it shows not only where DOI hyperlinks are found across the web, but also when they are actually followed. This data allows us a glimpse into scholarly citation beyond references in traditional literature.

Last year Crossref Labs announced Chronograph, an experimental system for browsing some of this data. We’re working toward a new version, but in the meantime I’d like to share the results for 2015 and some of 2016. We have filtered out domains that belong to Crossref member publishers to highlight citations beyond traditional publications.

Top 10 DOI referrals from websites in 2015

This chart shows the top 10 referring non-primary-publisher domains of DOIs per month. Note that if browsers don’t send the referrer (e.g. from an HTTPS page), we don’t get to find out. Because the top 10 can be different month to month, the total number of domains mentioned can be more than 10. Subdomains are combined, which means that, for example, the wikipedia.org entry covers all Wikipedia languages. This chart covers all of 2015 and the first two months of 2016.

The top 10 referring domains for the period:

webofknowledge.com
baidu.com
serialssolutions.com
scopus.com
exlibrisgroup.com
wikipedia.org
google.com
uni-trier.de
ebsco.com
google.co.uk

It’s not surprising to see some of these domains here: for example serialssolutions.com and exlibrisgroup.com are effectively proxies for link resolvers, Baidu and Google are incredibly popular search engines which would show up anywhere. But it is exciting to see Wikipedia ranked amongst these. For more detail look out for the new Chronograph.

HTTP vs HTTPS in 2015

We’ve also seen a steady increase in HTTPS referral traffic, i.e. people clicking on DOIs from sites that are using HTTPS. While it is still dwarfed by HTTP, there was a steady uptick throughout 2015.

This chart shows HTTP vs HTTPS referrals per day, which shows up the weekly spikes. It doesn’t include resolutions where we don’t know the referrer.

Increasing numbers of people are moving to HTTPS for reasons of security, privacy and protection from tampering. Google has announced plans to take HTTPS into account when ranking search results. Wikipedia has moved exclusively to HTTPS, and I’ll be telling the story of how Crossref and Wikipedia collaborated in an upcoming blog post.

Chronograph

Another version of Chronograph will be available soon. It will contain full data for all non-primary-publisher referring domains. Stay tuned!

Crossref Event Data: early preview now available

Madeleine Watson — Mon, 18 Apr 2016 00:00:00 +0000

Test out the early preview of Event Data while we continue to develop it. Share your thoughts. And be warned: we may break a few eggs from time to time!

Chicken by anbileru adaleru from the The Noun Project

Want to discover which research works are being shared, liked and commented on? What about the number of times a scholarly item is referenced? Starting today, you can whet your appetite with an early preview of the forthcoming Crossref Event Data service. We invite you to start exploring the activity of DOIs as they permeate and interact with the world after publication.

But first, a bit of background

Discussion around scholarly research increasingly occurs online after publication, for example on blogs, sharing services, social media, and wikis. These ‘events’ occur across the web on numerous platforms and are a critical part of the scholarly enterprise. We are developing an infrastructure service (Crossref Event Data) that collects, stores, and delivers raw data of the events occurring with Crossref DOIs. We will store the data in an open, auditable and portable form for the community to access. Publishers, platforms, funders, bibliometricians and service providers may benefit from access to this raw data, and it can be used to feed into research records or proprietary tools and services that offer aggregation and analysis.

For more information, see our pilot blog post and description of potential use cases.

Collaborative, transparent development

Developers Martin Fenner (DataCite) and Joe Wass (Crossref) enjoy a tofu break

Lagotto, the software originally developed at PLOS, has been extended and improved in a joint effort between DataCite and Crossref. The two DOI Registration Agencies have partnered to envision, build and release the service. On the 13th of April, after a year of collaboration, we jointly released Lagotto 5.0. You can read about the collaboration on the DataCite blog post.

Crossref and DataCite will continue to work closely together to develop Lagotto and the Event Data service. Although Crossref Event Data has mostly Crossref DOIs at launch, you will be able to find DataCite DOIs if they are cited in Crossref or Wikipedia.

All of the software that runs Event Data, including Lagotto, is developed in the open and is open source. Please refer to the Crossref Event Data Technical User Guide for full details.

Preview the data

This service is currently under development with a full launch expected the second half of 2016. Before it is launched however, we invite you to take a look around and preview a subset of the data sources we plan to include. You may experience occasional hiccups while we continue building the service.

At this stage, we are working with data from three sources although we will greatly expand the variety of platforms from which we collect data as development progresses. At this stage, you can view Mendeley bookmarks, Wikipedia references, and Crossref to DataCite links.

Mendeley

Mendeley is a reference manager and academic social network for scholars. View the number of social bookmarks from scholars or groups on Mendeley.

For example, doi.org/10.1016/J.JIP.2016.03.007 currently has 8 readers on Mendeley to date.

Wikipedia

Wikipedia is an online encyclopaedia, the Internet’s largest and most popular general reference work. View references in Wikipedia of Crossref publications in Wikipedia articles in all languages.

For example, doi.org/10.3897/ZOOKEYS.565.7185 was referenced in the Russian Wikipedia page on Oxyscelio

Crossref to DataCite links

DataCite is a global consortium that assigns DOIs to research data. This enables people to find, share, use, and cite data. You can view all the data citations to DataCite research outputs found in Crossref publications (work is underway to make the links found in DataCite metadata available in Event Data).

For example, Global, Regional, and National Fossil-Fuel CO2 Emissions (doi.org/10.3334/CDIAC/00001) dataset has been referenced by six Crossref publications to date. Software links are also included. Another example is PGOPHER (doi.org/10.5523/bris.huflggvpcuc1zvliqed497r2), a general purpose software for simulating and fitting rotational, vibrational and electronic spectra, which has been referenced by seven Crossref publications to date.

Ready to take a spin?

You can explore the Crossref Event Data early preview by visiting http://eventdata.crossref.org.pluma.sjfc.edu and following the links to featured examples within our interim application for inspecting the data, technical documentation, and our Quick Start guide.

This service is currently under development and as such we welcome your thoughts and feedback on the data we are collecting currently from our three active sources. As a reminder, we expect to include the following sources as part of our full service launch later this year (pending confirmation):

[table id=1 /]

We’re also on the lookout for new data sources to investigate for future inclusion in the Event Data service so please do get in touch with requests and recommendations. As we continue to build the service throughout 2016, we will be committing to a model of continuous development so that we can make new sources available as they are completed.

Watch this blog for regular updates on our progress, or subscribe to receive new blog posts by email (just add your details to the upper right side of this page).

The Wikipedia Library: A Partnership of Wikipedia and Publishers to Enhance Research and Discovery

Rachael Lammey — Mon, 04 Apr 2016 00:00:00 +0000

Back in 2014, Geoffrey Bilder blogged about the kick-off of an initiative between Crossref and Wikimedia to better integrate scholarly literature into the world’s largest knowledge space, Wikipedia. Since then, Crossref has been working to coordinate activities with Wikimedia: Joe Wass has worked with them to create a live stream of content being cited in Wikipedia; and we’re including Wikipedia in Event Data, a new service to launch later this year. In that time, we’ve also seen Wikipedia importance grow in terms of the volume of DOI referrals.

Alex Stinson, Project Manager for the Wikipedia Library, and our guest blogger! This file is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported license (Source: Myleen Hollero Photography)

Alex Stinson, Project Manager for the Wikipedia Library, and guest blogger! This file is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported license (Source: Myleen Hollero Photography)

How can we keep this momentum going and continue to improve the way we link Wikipedia articles with the formal literature? We invited Alex Stinson, a project manager at The Wikipedia Library (and one of our first guest bloggers) to explain more:

Wikipedia provides the most public gateway to academic and scholarly research. With millions of citations to academic as well as non-academic but reliable sources, like those produced by newspapers, its ecosystem of 5 million English Wikipedia articles and 35 million articles in hundreds of languages provides the first stop for researchers in both scholarly and informal research situations. The practice of “checking Wikipedia” has become ubiquitous in a number of fields; for example, Wikipedia is the most visited source of medical information online, even providing the first stop for many medical students and medical practitioners when looking for medical literature.

The Wikipedia Library program helps Wikipedia’s volunteer editors access and use the best sources in their research and citations. Through partnerships with over fifty leading publishers and aggregators, like JSTOR, Project Muse, Elsevier, Newspapers.com, Highbeam, Oxford University Press and others, we have been able to give over 3000 of our most prolific volunteers access to over 5500 accounts. These are clear, win-win relationships where Wikipedia editors get to use these databases to improve Wikipedia, while in turn linking to authoritative resources and enhancing their discovery.

JSTOR has been working with us since 2012, providing over 500 accounts to our editors. Kristen Garlock at JSTOR writes:

“We’re very happy to collaborate with the Wikipedia Library to provide JSTOR access to Wikipedia editors. Supporting the initiative to increase editor access to scholarly resources and improve the quality of information and sources on Wikipedia has the potential to help all Wikipedia readers. In addition to providing more discoverability for our institutional subscribers, introducing new audiences to the scholarship on JSTOR them discover access opportunities like our Register & Read program.”

There are strong signals that Wikipedia’s role in the citation ecosystem helps ensure the best materials reach the public through its over 400 million monthly readers:

The latest estimates by Crossref show that Wikipedia has risen from the 8th most prolific referrer to DOIs to the 5th.

Two of our access partners have found that around half of the referrals arriving from Wikipedia were able to authenticate into their subscription resources, suggesting that a large portion of our readers can take advantage of subscriptions provided by scholarly institutions.

Wikipedia is highly influential in the open access ecosystem as well, with a recent study showing higher citation rates for OA materials than those behind a paywall.

Altmetrics tools (such as Altmetric.com, ImpactStory or Plum Analytics) are recognizing Wikipedia’s importance by including Wikipedia citations in their impact metrics.

Despite these advances, we think this is only the beginning of Wikipedia’s impact on the landscape of scholarly research and discovery. Wikipedia can become a highly integrated research platform within the broader research ecosystem, where the best scholarship is summarized and discoverable-where Wikipedia effectively becomes the front matter to all research.

However, there are some clear barriers to fulfilling this vision. Currently, most citations on Wikipedia are stored in free-text and not readily available in machine-readable formats; our community is working to fix this. Wikipedia also has major systematic gaps in topics where either we lack volunteer interest or Wikipedia reflects larger systemic biases within society or scholarship.We need the help of volunteers, experts, industry partners, and information technologists to grow Wikipedia’s collection of citations, especially around key missing areas, and to transform existing citations into structured formats.

WikiData, Wikipedia’s sister project which crowdsources structured metadata, offers an excellent opportunity for improving the impact of Wikipedia in research. Having Wikipedia citations stored in this structured ecosystem, connecting metadata with semantic meaning, would allow the citations in Wikipedia to become the backbone for discovery tools which emphasize the hand-curated interrelationships between authoritative sources and the knowledge collected by Wikipedia and Wikidata editors.

We need more collaborators to realize the full vision of Wikipedia supporting research in the most effective ways:

We need help from publishers with subscription databases, to help us give our editors access to the databases through The Wikipedia Library’s access partnership program. These high-quality source materials allow our editors to expose that research in a number of languages and for millions of readers.

We need help from the open access community, to figure out how to better support increased citation and strategic use of open access materials within Wikipedia and other Wikimedia projects. Our community has some ideas, but we need your input and collaboration.

We need your expertise to build our structured metadata ecosystem, by helping Wikidata map and collect citation data.

We need the larger research community to promote Wikipedia as a scholarly communications tool and make contributing to Wikipedia an important part of the social responsibility of experts. Wider citation of sources in Wikipedia ensures widespread discovery and dissemination of that research.

If you think you can help, we invite you to contact us at wikipedialibrary@wikimedia.org or via Twitter @WikiLibrary.

Crossref Labs plays with the Raspberry Pi Zero

Joe Wass — Wed, 02 Dec 2015 00:00:00 +0000

If you’re anything like us at Crossref Labs (and we know some of you are) you would have been very excited about the launch of the Raspberry Pi Zero a couple of days ago. In case you missed it, this is a new edition of the tiny low-priced Raspberry Pi computer. Very tiny and very low-priced. At $5 we just had to have one, and ordered one before we knew exactly what we want to do with it. You would have done the same. Bad luck if it was out of stock.

We love the way DOIs are being used in Wikipedia, but you probably already know that by now. Not only is it a brilliant source of information, mostly well cited, it’s also an organic living thing, with countless people and bots working together on countless articles. Our live stream of edits that cite (or uncite) DOIs shows new scholarly literature unfold, as it happens. From new articles to new references to improved citations to edit wars to bots cleaning up all the mess, it captivates everyone we show it to. The latest version has a live chart to show exactly how much activity is going on.

Crossref works in five ways: Rally, Tag, Run, Play, and Make and this definitely comes under ‘Play’. By the time our Raspberry Pi Zero arrived it was clear what we had to do. We ordered a servo, a driver board and a wireless adapter and got to work.

We have some new neighbours in the basement. Oxford Hackspace is a community of people who want to work on projects from electronics to metalwork, hack things to improve them or find out how they work. A diverse bunch who at the last visit were working on squeezing unprecedented color capabilities from the 30 year old ZX Spectrum, a nixie tube display, a smartphone controlled doorbell and a robotic glockenspiel. They let us use their soldering iron to solder a few header pins.

A bit of hacky Python, a pictureframe and lots of duck tape later, we have a live display of how many DOIs are cited and uncited per hour. It updates live every minute, fetches the latest numbers from the Wikipedia DOI citation stream and moves the hand.

(For the worried engineers amongst you, rest assured that sufficient duck tape was added after this picture)

It’s extraordinary to think that a fully fledged computer with very capable specifications can be manufactured and sold for $5. Within the space of a lunchtime we had it up and running, all connected and fetching data over the internet via wireless. A generation ago you would have had to use punched cards, send them by post and load them in by hand. The live stream would have been at least a month behind.

It now sits in our Oxford office reminding us that DOIs Aren’t Just for Traditional Bibliographies. Below Geoff Bilder’s reminder about what happens when you have too many standards (they’re telephone plugs from round the world).

You can find source code and instructions on the github repository so you can make your own if you want.

DOI Event Tracker (DET): Pilot progresses and is poised for launch

Jennifer Lin — Tue, 15 Sep 2015 00:00:00 +0000

Publishers, researchers, funders, institutions and technology providers are all interested in better understanding how scholarly research is used. Scholarly content has always been discussed by scholars outside the formal literature and by others beyond the academic community. We need a way to monitor and distribute this valuable information.

The Crossref DOI Event Tracker (DET)

To meet this need, Crossref will be introducing a new service that tracks activity surrounding a research work from potentially any web source where an event is associated with a DOI. Following a successful pilot run started Spring 2014, the service has been approved to move toward production and is expected to launch in 2016. Any party wishing to join this phase is welcome to contact Jennifer Lin. The DOI Event Tracker (DET) registers a wide variety of events such as bookmarks, comments, social shares, citations, and links to other research entities, from a growing list of online sources. DET aggregates them, and stores and delivers the data in many ways.

Open, portable, and licensed for maximum reuse

Crossref has long served as the citation linking and metadata infrastructure provider for scholarly communication; the new DOI Event Tracker is a natural next step, providing a practical solution as a resource for the whole community. The tracker offers the following features:

Data on event activity across a common pool of online channels.
Near real-time alerting for select sources with push notifications to the system.
Cross-publisher monitoring to enable benchmarking and provide context to the data.
Common format for normalizing data results across the diverse set of sources via modern REST API.
Secure and regularly refreshed backups of critical data for long term data preservation.
Transparency of data collection so as to ensure auditable, replicable, and trustworthy results.
Query-initiated retrieval or real-time alerts when an event of interest occurs.
CC-0 license for open and flexible propagation of data.

A number of platforms are already confirmed and more parties are welcomed at any stage. So far we have confirmation to track DOI events on the following platforms:

[table id=1 /]

This set of sources reflects our initial focus on parties willing to allow their data to be redistributed in the common pool. Efforts are underway to expand the source list to include Twitter and MyScienceWork, among others. Publishers can also act as sources by publishing and distributing DOI event data via the DET when an event occurs on its platform (for example, when a PDF is downloaded, or when a comment mentions a DOI in a locally hosted discussion forum, etc.). This would make local DOI activity globally available to funders, researchers, institutions, etc.

DET provides benefits of scale and ease of access as a central point for collecting and propagating data to the community. As a single point of access, it overcomes the business and technical hurdles that are a part of managing multiple online sources where scholarly activity occurs, in a rapidly changing landscape of online channels. This resource covers content across publishers and serves as a strong foundation to support the development of tools and services by any party. DET users will always be able to combine the DET data with those individually collected via negotiated or paid access. DET remains a utility separate from any value-added amenities, such as analytics, presentation, and reporting.

DET Service-Level Agreement

For those who seek the highest level of service and a more flexible range of access options, Crossref will provide a Service-Level Agreement (SLA) service for the DOI Event Tracker. The DET SLA includes the following additional features on top of the common data offering:

Access to the complete suite of sources, which includes restricted and/or paid sources in addition to common data, providing the fullest picture of DOI usage activity possible.
Guaranteed uptime and response time to the latest raw data on the aggregate activity surrounding a DOI.
Guaranteed support response time to questions and issues surrounding data and data delivery.
Flexible data access options: on-demand real time data access and scheduled bulk downloads for processing batch analytics.
Optimum retrieval rates and accelerated delivery speeds with the dedicated SLA API.
Access to a webhook API for events of interest as an alternative to polling DET.
Standardized and enhanced linkback service for the difficult-to-track, grey literature.

The DET SLA service has a simple, value-based pricing model based on subscriber size. Register your interest in Crossref’s DOI Event Tracker and the DET SLA service if you would like stay informed of the upcoming launch. Please contact Jennifer Lin for more information.

Image modified from “Radar” icon by Karsten Barnett from the Noun Project.

Coming to you Live from Wikipedia

Joe Wass — Wed, 20 May 2015 00:00:00 +0000

We’ve been collecting citation events from Wikipedia for some time. We’re now pleased to announce a live stream of citations, as they happen, when they happen. Project this on your wall and watch live DOI citations as people edit Wikipedia, round the world.

View live stream »

In the hours since this feature launched, there are events from Indonesian, Portugese, Ukrainian, Serbian and English Wikipedias (in that order).

The usual weasel words apply. This is a labs project and so may not be 100% stable. If you experience any problems please email labs@crossref.org .

Real-time Stream of DOIs being cited in Wikipedia

Joe Wass — Tue, 03 Mar 2015 00:00:00 +0000

TL;DR

Watch a real-time stream of DOIs being cited (and “un-cited!” ) in Wikipedia articles across the world: https://live-eventdata-crossref-org.pluma.sjfc.edu/live.html

Background

For years we’ve known that the Wikipedia was a major referrer of Crossref DOIs and about a year ago we confirmed that, in fact, the Wikipedia is the 8th largest refer of Crossref DOIs. We know that people follow the DOIs, too. This despite a fraction of Wikipedia citations to the scholarly literature even using DOIs. So back in August we decided to create a Wikimedia Ambassador programme. The goal of the programme was to promote the use of persistent identifiers in citation and attribution in Wikipedia articles. We would do this through outreach and through the development of better citation-related tools.

Remember when we originally wrote about our experiments with the PLOS ALM code and how that has transitioned into the DOI Event Tracking Pilot? In those posts we mentioned that one of the hurdles in gathering information about DOI events is the actual process of polling third party APIs for activity related to millions of DOIs. Most parties simply wouldn’t be willing handle the load of a 100K API calls an hour. Besides, polling is a tremendously inefficient process, only a fraction of DOIs are ever going to generate events, but we’d have to poll for each of them, repeatedly, forever, to get an accurate picture of DOI activity. We needed a better way. We needed to see if we could reverse this process and convince some parties to instead “push” us information whenever they saw DOI related events (e.g. citations, downloads, shares, etc). If only we could convince somebody to try this…

Wikipedia DOI Events

In December 2014 we took the opportunity of the 2014 PLOS/Crossref ALM Workshop in San Francisco too meet with Max Klein and Anthony Di Franco where we kicked off a very exciting project.

There’s always someone editing a Wikipedia somewhere in the world. In fact, you can see a dizzying live stream of edits. We thought that given that there are so many DOIs in Wikipedia, that live stream may contain some diamonds (DOIs are made of diamond, that’s how they can be persistent). Max and Anthony went away and came back with a demo that contains a surprising amount of DOI activity.

That demo is evolving into a concrete service, called Cocytus. It is running at Wikimedia Labs monitoring live edits as you read this.

For now we’re feeding that data into the DOI Events Collection app (which is an off-shoot of the Chronograph project). We are in the process of modifying the Lagotto code so that we can instead push those events into the DOI Event Tracking Instance.

The first DOI event we noticed was delightfully prosaic: The DOI for “The polymath project” is cited by the Wikipedia page for “Polymath Project”. Prosaic perhaps, but the authors of that paper probably want to know. Maybe they can help edit the page.

Or how about this. Someone wrote a a paper about why people edit Wikipedia and then it was cited by Wikipedia. And then the citation was removed. The plot thickens…

We’re interested in seeing how DOIs are used outside of the formal scholarly literature. What does that mean? We don’t fully know, that’s the point. We have retractions in scholarly literature (and our Crossmark metadata and service allow publishers to record that), but it’s a bit different on Wikipedia. Edit wars are fought over … well you can see for yourself.

Citations can slip in and out of articles. We saw the DOI 10.1001/archpediatrics.2011.832 deleted from “Bipolar disorder in children”. If we’d not been monitoring the live feed (we had considered analysing snapshots of the Wikipedia in bulk) we might never have seen that. This is part of what non-traditional citations means, and it wasn’t obvious until we’d seen it.

You can see this activity on the Chronograph’s stream. Or check your favourite DOI. Please be aware that we’re only collecting newly added citations as of today. We do intend to go back and back-fill, but that may take some time- as it * cough * requires polling again.

Some Technical Things

A few interesting things that happened as a result of all this:

Secure URLs

SSL and HTTPS were invented so you could do things like banking on the web without fear of interception or tampering. As the web becomes a more important part of life, many sites are upgrading from HTTP to HTTPS, the secure version. This is not only because your confidential details may be tampered with, but because certain governments might not like you reading certain materials.

Because of this, some time ago, Wikipedia decided to embark on an upgrade to HTTPS last year, and they are a certain way along the path. The IDF, who are responsible for running the DOI system, upgraded to HTTPS this Summer, although most DOIs are referred to by HTTP still.

We met with Dario Taraborelli at the ALM workshop and discussed the DOI referral data that is fed into the Chronograph. We put two and two together and realised that Wikipedia was linking to DOIs (which are mostly HTTP) from pages which might be served over HTTPS. New policies in HTML5 specify that referrer URL headers shouldn’t be sent from HTTPS to HTTP (in case there was something secret in them). The upshot of this is, if someone’s browsing Wikipedia via HTTPS and click on a normal DOI, we won’t know that the user came from Wikipedia. Not a huge problem today, but as Wikipedia switches over to entirely secure, we’re going to miss out on very useful information.

Fortunately, the HTML5 specification includes a way to fix this (without leaking sensitive information). We discussed this with Dario, and he did some research, and came up with a suggestion, which got discussed. It’s fascinating to watch a democratic process like this take place and take part in it.

We’re waiting to see how the discussion turns out, and hope that it all works out so we can continue to report on how amazing Wikipedia is at sending people to scholarly literature.

How shall I cite thee?

Another discussion grew out of that process, and we started talking to a Wikipedian called Nemo (note to Latin scholars: we weren’t just talking to ourselves). Nemo (real name Federico Leva) had a few suggestions of his own. Another way to solve the referrer problem is by using HTTPS URLs (HTML5 allows browsers to send the referrer domain when going from HTTPS to HTTPS).

This means going back to all the articles that use DOIs and change them from HTTP to HTTPS. Not as simple as it sounds, and it doesn’t sound simple. We started looking into how DOIs were cited on Wikipedia.

After some research we found that there are more ways that we expected to cite DOIs.

First, there’s the URL. You can see it in action in this article. URLs can take various forms.

Second there’s the official template tag, seen in action here:

<ref name="SCI-20140731">{{cite journal |title=Sustained miniaturization and anatomical innovation in the dinosaurian ancestors of birds |url=http://www.sciencemag.org.pluma.sjfc.edu/content/345/6196/562 |date=1 August 2014 |journal=[[Science (journal)|Science]] |volume=345 |issue=6196 |pages=562–566 |doi=10.1126/science.1252243 |accessdate=2 August 2014 |last1=Lee |first1=Michael S. Y. |first2=Andrea|last2=Cau |first3=Darren|last3=Naish|first4=Gareth J.|last4=Dyke}}</ref>

There’s a DOI in there somewhere. This is the best way to cite DOIs, firstly as it’s actually a proper traditional citation and there’s nothing magic about DOIs, secondly because it’s a template tag and can be re-rendered to look slightly different if needed.

Third there’s the old official DOI template tag that’s now discouraged:

<ref name="Example2006">{{Cite doi|10.1146/annurev.earth.33.092203.122621}}</ref>

And then there’s another one.

{{doi|10.5555/123456789}}

Knowing all this helps us find DOIs. But if we want to convert DOIs links in Wikipedia to use HTTPS, it means that there are more template tags to modify and more pages to re-render.

Nemo also put DOIs on the Interwiki Map which should make automatically changing some of the URLs a lot easier.

We’re very grateful to Nemo for his suggestions and work on this. We’ll report back!

The elephant in the room

Those of you who know how DOIs work will have spotted an unsecured elephant in the room. When you visit a DOI, you visit the URL, which hits the DOI resolver proxy server, which returns a message to your browser to redirect to the landing page on the publisher’s site.

Securely talking to the DOI resolver by using HTTPS instead of HTTP means that no-one can eavesdrop and see which DOI you are visiting, or tamper with the result and send you off to a different page. But the page you are sent to will be, in nearly all cases, still HTTP. Upgrading infrastructure isn’t trivial, and, with over 4000 members (mostly publishers), most Crossref DOIs will still redirect to standard HTTP pages for the foreseeable future.

You can keep as secure as possible by using HTTPS Everywhere.

Fin

There’s lots going on, watch this space to see developments. Thanks for reading this, and all the links. We’d love to know what you think.

Bootnote

Not long after this blog post was published we saw something very interesting.

That’s no DOI. We like interesting things, but they can panic us. This turned out to be a great example of why this kind of thing can be useful. A minute’s digging and we found the article edit:

It turns out that this was a typo: someone put a title when they should have put in a DOI. And, as the event shows, this was removed from the Wikipedia article.

Introducing the Crossref Labs DOI Chronograph

Joe Wass — Mon, 12 Jan 2015 00:00:00 +0000

tl;dr http://chronograph.labs.crossref.org.pluma.sjfc.edu

At Crossref we mint DOIs for publications and send them out into the world, but we like to hear how they’re getting on out there. Obviously, DOIs are used heavily within the formal scholarly literature and for citations, but they’re increasingly being used outside of formal publications in places we didn’t expect. With our DOI Event Tracking / ALM pilot project we’re collecting information about how DOIs are mentioned on the open web to try and build a picture about new methods of citation.

As part of the preparation for collaborating with Wikipedia, we looked at our statistics about when DOIs are clicked and discovered that Wikipedia was, over a two year period from 2012, the eighth largest referrer of DOIs. This means that not only does Wikipedia have a lot of DOIs, but people click them too. This bit of one-off data analysis (which surprised us) gave us enough of a prod to kickstart our collaboration with Wikipedia.

At the ALM Workshop 2014 in San Francisco we talked to some Wikipedians and bibliometricians and realised that we were sitting on a really interesting data-set and that it would be churlish not to share it. At the hackathon (read the report here) we started work on a service to gather information about DOIs and, a month later, we’re ready to unveil the DOI Chronograph.

Show me the goods

You can see:

Daily referrals (clicks) from top level domains, e.g. Wikipedia.org: http://chronograph.labs.crossref.org.pluma.sjfc.edu/domain.html?domain=wikipedia.org

Daily referrals from specific subdomains, e.g. fr.wikipedia.org: http://chronograph.labs.crossref.org.pluma.sjfc.edu/domain.html?domain=fr.wikipedia.org

Daily resolutions per DOI: http://chronograph.labs.crossref.org.pluma.sjfc.edu/doi.html?doi=10.1787%2F20752288

And, the chart that kicked this all off: DOI referring domains league tables. This shows that Wikipedia is the 3rd or 4th non-traditional referrer of DOIs (i.e. excluding referrals from Publishers’ domains): http://chronograph.labs.crossref.org.pluma.sjfc.edu/top.html

Try it out

Visit the Chronograph and give it a try chronograph.labs.crossref.org on your favourite DOI (everyone has one).

More data

Talking to a bibliometrician we also realised we can correlate other data for DOIs. We’re getting the issue date (approximately the publication date) from our own metadata, as well as the date that the Crossref metadata was updated. This gives interesting results, like the resolutions for 10.1038/ncomms2953, which peak after publication and then tails off. We are attempting to collect the following information:

daily resolution counts
day on which resolution was first successful
day on which it’s possible to resolve the DOI (we’ve got a bot running for new publications)
day on which the publisher says the article was published
day on which the metadata was most recently deposited with us
day on which the metadata was first deposited with us

We’re not there yet, but we’ve made a start and we’ve already got some pretty interesting data!

Weasel words

It’s a labs project so the usual weasel words apply. Specifically, we currently have the logs for 2012 to 2014 (we’re working at digging out the rest), and the referral information for 50 million DOIs (out of 71 million). That number will be higher by the time you read this. If your page is slow to load, be patient, as it’s currently working hard crunching numbers.

This project is focused on exploring the use of DOIs outside of the formal literature. As such, we are only looking at referrals from domains that do not appear to belong to primary publishers (i.e. our members). If you try a domain and it doesn’t work, it could be that the domain belongs to one of our members. If you’ve notice any mistakes, please email us at labs@crossref.org .

Finally, these numbers contain all DOI resolutions. That’s human clicks but also content negotiation to retrieve metadata, robots etc. We might try to filter them in future, but for now be aware that not every visitor is a human.

I’ll detail some of the the technical stuff (it’s very interesting) and what happened next with Wikipedia in a future post. Watch this space.

Citation needed

Geoffrey Bilder — Thu, 07 Aug 2014 00:00:00 +0000

Remember when I said that the Wikipedia was the 8th largest referrer of DOI links to published research? This despite only a fraction of eligible references in the free encyclopaedia using DOIs.

We aim to fix that. Crossref and Wikimedia are launching a new initiative to better integrate scholarly literature in the world’s largest public knowledge space, Wikipedia.

This work will help promote standard links to scholarly references within Wikipedia, which persist over time by ensuring consistent use of DOIs and other citation identifiers in Wikipedia references. Crossref will support the development and maintenance of Wikipedia’s citation tools on Wikipedia. This work will include bug fixes and performance improvements for existing tools, extending the tools to enable Wikipedia contributors to more easily look up and insert DOIs, and providing a “linkback” mechanism that alerts relevant parties when a persistent identifier is used in a Wikipedia reference.

In addition, Crossref is creating the role of Wikimedia Ambassador (modeled after Wikimedian-in-Residence) to act as liaison with the Wikimedia community, promote use of scholarly references on Wikipedia, and educate about DOIs and other scholarly identifiers (ORCIDs, PubMed IDs, DataCite DOIs, etc) across Wikimedia projects.

Starting today, Crossref will be working with Daniel Mietchen to coordinate Crossref’s Wikimedia-related activities. Daniel’s team will be composed of Max Klein and Matt Senate, who will work to enhance Wikimedia citation tools, and will share the role of Wikipedia ambassador with Dorothy Howard.

Since the beginnings of Wikipedia, Daniel Mietchen has worked to integrate scholarly content into Wikimedia projects. He is part of an impressive community of active Wikipedians and developers who have worked extensively on linking Wikipedia articles to the formal literature and other scholarly resources. We’ve been talking to him about this project for nearly a year, and are happy to finally get it off the ground.

-G

]7 Matt, Max and Daniel at #wikimania2014. Photo by Dorothy.

wikimania2014

Many Metrics. Such Data. Wow.

Geoffrey Bilder — Mon, 24 Feb 2014 00:00:00 +0000

[

Crossref Labs loves to be the last to jump on an internet trend, so what better than than to combine the Doge meme with altmetrics?

Note: The API calls below have been superceeded with the development of the Event Data project. See the latest API documentation for equivalent functionality

Want to know how many times a Crossref DOI is cited by the Wikipedia?

http://det.labs.crossref.org.pluma.sjfc.edu/works/doi/10.1371/journal.pone.0086859

Or how many times one has been mentioned in Europe PubMed Central?

http://det.labs.crossref.org.pluma.sjfc.edu/works/doi/10.1016/j.neuropsychologia.2013.10.021

Or DataCite?

http://det.labs.crossref.org.pluma.sjfc.edu/works/doi/10.1111/jeb.12289

Background

Back in 2011 PLOS released its awesome ALM system as open source software (OSS). At Crossref Labs, we thought it might be interesting to see what would happen if we ran our own instance of the system and loaded it up with a few Crossref DOIs. So we did. And the code fell over. Oops. Somehow it didn’t like dealing with 10 million DOIs. Funny that.

But the beauty of OSS is that we were able to work with PLOS to scale the code to handle our volume of data. Crossref contracted with Cottage Labs and we both worked with PLOS to make changes to the system. These eventually got fed back into the main ALM source on Github. Now everybody benefits from our work. Yay for OSS.

So if you want to know technical details, skip to Details for Propellerheads. But if you want to know why we did this, and what we plan to do with it, read on.

Why?

There are (cough) some problems in our industry that we can best solve with shared infrastructure. When publishers first put scholarly content online, they used to make bilateral reference linking agreements. These agreements allowed them to link citations using each other’s proprietary reference linking APIs. But this system didn’t scale. It was too time-consuming to negotiate all the agreements needed to link to other publishers. And linking through many proprietary citation APIs was too complex and too fragile. So the industry founded Crossref to create a common, cross-publisher citation linking API. Crossref has since obviated the need for bilateral linking arrangements.

So-called altmetrics look like they might have similar characteristics. You have ~4000 Crossref member publishers and N sources (e.g. Twitter, Mendeley, Facebook, CiteULike, etc.) where people use (e.g. discuss, bookmark, annotate, etc.) scholarly publications. Publishers could conceivably each choose to run their own system to collect this information. But if they did, they would face the following problems:

The N sources will be volatile. New ones will emerge. Old ones will vanish.
Each publisher will need to deal with each source’s different APIs, rate limits, T&Cs, data licenses, etc. This is a logistical headache for both the publishers and for the sources.
If publishers use different systems which in turn look at different sources, it will be difficult to compare results across publishers.
If a journal moves from one publisher to another, then how are the metrics for that journal’s articles going to follow the journal? This isn’t a complete list, but it shows that there might be some virtue in publishers sharing an infrastructure for collecting this data. But what about commercial providers? Couldn’t they provide these ALM services? Of course - and some of them currently do. But normally they look on the actual collection of this data as a means to an end. The real value they provide is in the analysis, reporting and tools that they build on top of the data. Crossref has no interest in building front-ends to this data. If there is a role for us to play here, it is simply in the collection and distribution of the data.

No, really, WHY?

Aren’t these altmetrics an ill-conceived and meretricious idea? By providing this kind of information, isn’t Crossref just encouraging feckless, neoliberal university administrators to hasten academia’s slide into a Stakhanovite dystopia? Can’t these systems be gamed?

FOR THE LOVE OF FSM, WHY IS CROSSREF DABBLING IN SOMETHING OF SUCH QUESTIONABLE VALUE?

takes deep breath. wipes spittle from beard

These are all serious concerns. Goodhart’s Law and all that… If a university’s appointments and promotion committee is largely swayed by Impact Factor, it won’t improve a thing if they substitute or supplement Impact Factor with altmetrics. Amy Brand has repeatedly pointed out, the best institutions simply don’t use metrics this way at all (PowerPoint presentation). They know better.

But yes, it is still likely that some powerful people will come to lazy conclusions based on altmetrics. And following that, other lazy, unscrupulous and opportunistic people will attempt to game said metrics. We may even see an industry emerge to exploit this mess and provide the scholarly equivalent of SEO. Feh. Now I’m depressed and I need a drink.

So again, why is Crossref doing this? Though we have our doubts about how effective altmetrics will be in evaluating the quality of content, we do believe that they are a useful tool for understanding how scholarly content is used and interpreted. The most eloquent arguments against altmetrics for measuring quality, inadvertently make the case for altmetrics as a tool for monitoring attention.

Critics of altmetrics point out that much of the attention that research receives outside of formal scholarly communications channels can be ascribed to:

Puffery. Researchers and/or university/publisher “PR wonks” over-promoting research results.
Innocent misinterpretation. A lay audience simply doesn’t understand the research results.
Deliberate misinterpretation. Ideologues misrepresent research results to support their agendas.
Salaciousness. The research appears to be about sex, drugs, crime, video games or other popular bogeymen.
Neurobollocks. A category unto itself these days.

In short, scholarly research might be misinterpreted. Shock horror. Ban all metrics. Whew. That won’t happen again.

Scholarly research has always been discussed outside of formal scholarly venues. Both by scholars themselves and by interested laity. Sometimes these discussions advance the scientific cause. Sometimes they undermine it. The University of Utah didn’t depend on widespread Internet access or social networks to promote yet-to-be peer-reviewed claims about cold fusion. That was just old-fashioned analogue puffery. And the Internet played no role in the Laetrile or DMSO crazes of the 1980s. You see, there were once these things called “newspapers.” And another thing called “television.” And a sophisticated meatspace-based social network called a “town square.”

But there are critical differences between then and now. As citizens get more access to the scholarly literature, it is far more likely that research is going to be discussed outside of formal scholarly venues. Now we can build tools to help researchers track these discussions. Now researchers can, if they need to, engage in the conversations as well. One would think that conscientious researchers would see it as their responsibility to remain engaged, to know how their research is being used. And especially to know when it is being misused.

That isn’t to say that we expect researchers will welcome this task. We are no Pollyannas. Researchers are already famously overstretched. They barely have time to keep up with the formally published literature. It seems cruel to expect them to keep up with the firehose of the Internet as well.

Which gets us back to the value of altmetrics tools. Our hope is that, as altmetrics tools evolve, they will provide publishers and researchers with an efficient mechanism for monitoring the use of their content in non-traditional venues. Just in the way that citations were used before they were distorted into proxies for credit and kudos.

We don’t think altmetrics are there yet. Partly because some parties are still tantalized by the prospect of usurping one metric for another. But mostly because the entire field is still nascent. People don’t yet know how the information can be combined and used effectively. So we still make naive assumptions such as “link=like” and “more=better.” Surely it will eventually occur to somebody that, instead, there may be a connection between repeated headline-grabbing research and academic fraud. A neuroscientist might be interested in a tool that alerts them if the MRI scans in their research paper are being misinterpreted on the web to promote neurobollocks. An immunologist may want to know if their research is being misused by the anti-vaccination movement. Perhaps the real value in gathering this data will be seen when somebody builds tools to help researchers DETECT puffery, social-citation cabals, and misinterpretation of research results?

But Crossref won’t be building those tools. What we might be able to do is help others overcome another hurdle that blocks the development of more sophisticated tools; getting hold of the needed data in the first place. This is why we are dabbling in altmetrics.

Wikipedia is already the 8th largest referrer of Crossref DOIs. Note that this doesn’t just mean that the Wikipedia cites lots of Crossref DOIs, it means that people actually click on and follow those DOIs to the scholarly literature. As scholarly communication transcends traditional outlets and as the audience for scholarly research broadens, we think that it will be more important for publishers and researcher to be aware of how their research is being discussed and used. They may even need to engage more with non-scholarly audiences. In order to do this, they need to be aware of the conversations. Crossref is providing this experimental data source in the hope that we can spur the development of more sophisticated tools for detecting and analyzing these conversations. Thankfully, this is an inexpensive experiment to conduct - largely thanks to the decision on the part of PLOS to open source its ALM code.

What Now?

Crossref’s instance of PLOS’s ALM code is an experiment. We mentioned that we had encountered scalability problems and that we had resolved some of them. But there are still big scalability issues to address. For example, assuming a response time of 1 second, if we wanted to poll the English-language version of the Wikipedia to see what had cited each of the 65 million DOIs held in Crossref, the process would take years to complete. But this is how the system is designed to work at the moment. It polls various source APIs to see if a particular DOI is “mentioned”. Parallelizing the queries might reduce the amount of time it takes to poll the Wikipedia, but it doesn’t reduce the work. Another obvious way in which we could improve the scalability of the system is to add a push mechanism to supplement the pull mechanism. Instead of going out and polling the Wikipedia 65 million times, we could establish a “scholarly linkback” mechanism that would allow third parties to alert us when DOIs and other scholarly identifiers are referenced (e.g. cited, bookmarked, shared). If the Wikipedia used this, then even in an extreme case scenario (i.e. everything in Wikipedia cites at least one Crossref DOI), this would mean that we would only need to process ~ 4 million trackbacks.

The other significant advantage of adding a push API is that it would take the burden off of Crossref to know what sources we want to poll. At the moment, if a new source comes online, we’d need to know about it and build a custom plugin to poll their data. This needlessly disadvantages new tools and services as it means that their data will not be gathered until they are big enough for us to pay attention to. If the service in question addresses a niche of the scholarly ecosystem, they may never become big enough. But if we allow sources to push data to us using a common infrastructure, then new sources do not need to wait for us to take notice before they can participate in the system.

Supporting (potentially) many new sources will raise another technical issue- tracking and maintaining the provenance of the data that we gather. The current ALM system does a pretty good job of keeping data, but if we ever want third parties to be able to rely on the system, we probably need to extend the provenance information so that the data is cheaply and easily auditable.

Perhaps the most important thing we want to learn from running this experimental ALM instance is: what it would take to run the system as a production service? What technical resources would it require? How could they be supported? And from this we hope to gain enough information to decide whether the service is worth running and, if so, by whom. Crossref is just one of several organisations that could run such a service, but it is not clear if it would be the best one. We hope that as we work with PLOS, our members and the rest of the scholarly community, we’ll get a better idea of how such a service should be governed and sustained.

Details for Propellerheads

Warning, Caveats and Weasel Words

The Crossref ALM instance is a Crossref Labs project. It is running on R&D equipment in a non-production environment administered by an orangutang on a diet of Redbulls and vodka.

So what is working?

The system has been initially loaded with 317,500+ Crossref DOIs representing publications from 2014. We will load more DOIs in reverse chronological order until we get bored or until the system falls over again.

We have activated the following sources:

PubMed

DataCite

PubMedCentral Europe Citations and Usage

We have data from the following sources but will need some work to achieve stability:

Facebook

Wikipedia

CiteULike

Twitter

Some of them are faster than others. Some are more temperamental than others. WordPress, for example, seems to go into a sulk and shut itself off after approximately 1,300 API calls.

In any case, we will be monitoring and tweaking the sources as we gather data. We will also add new sources as we get requested API keys. We will probably even create one or two new sources ourselves. Watch this blog and we’ll update you as we add/tweak sources.

Dammit, shut up already and tell me how to query stuff.

You can login to the Crossref ALM instance simply using a Mozilla Persona (yes, we’d eventually like to support ORCID too). Once logged-in, your account page will list an API key. Using the API key, you can do things like:

http://det.labs.crossref.org.pluma.sjfc.edu/api/v5/articles?ids=10.1038/nature12990

And you will see that (as of this writing), said Nature article has been cited by the Wikipedia article here:

https://en.wikipedia.org/wiki/HE0107-5240#cite_ref-Keller2014_4-0;

PLOS has provided lovely detailed instructions for using the API- So, please, play with the API and see what you make of it. On our side we will be looking at how we can improve performance and expand coverage. We don’t promise much- the logistics here are formidable. As we said above, once you start working with millions of documents, the polling process starts to hit API walls quickly. But that is all part of the experiment. We appreciate your helping us and would like your feedback. We can be contacted at: