Software on Crossref

Mending Chesterton's fence: open source decision-making

Joe Wass — Mon, 18 Mar 2024 00:00:00 +0000

When each line of code is written it is surrounded by a sea of context: who in the community this is for, what problem we’re trying to solve, what technical assumptions we’re making, what we already tried but didn’t work, how much coffee we’ve had today. All of these have an effect on the software we write.

By the time the next person looks at that code, some of that context will have evaporated. There may be helpful code comments, tests, and specifications to explain how it should behave. But they don’t explain the path not taken, and why we didn’t take it. Or those occasions where the facts changed, so we changed our mind.

Some parts of our system are as old as Crossref itself. Whilst our process still involves coffee, it’s safe to say that most of our working assumptions have changed, and for good reasons! We have to be very careful when working with our oldest code. We always consider why it was written that way, and what might have changed since. We’re always on the look out for Chesterton’s Fence!

Leaving a Trail

We’re building a new generation of systems at Crossref, and as we go we’re being deliberate about supporting the people who will maintain it.

When our oldest code was written, the software development team all worked in an office with a whiteboard or three, and the code was proprietary. Twenty years later, things are very different. The software development team is spread over 8 timezones. Thanks to POSI, all the new code we write is open source, so the next people to read that code might not even be Crossref staff.

Working increasingly asynchronously, without that whiteboard, we need to record the options, collect evidence, and peer-review them within the team.

So for the past couple of years the software team has maintained a decision register. The first decision we recorded was that we should record decisions! Since then we have recorded the significant decisions as they arise. Plus some historical ones.

These aren’t functional specifications, which describe what the system should do. It’s the decisions and trade-offs we made along the way to get to the how. Look out for another blog post about specifications.

By leaving a trail of explanations as we go, we make it easier for people to understand why code was written, and what has changed. We’re writing the story of our new systems. This makes it easier to alter the system in future in response to changes in our community, and the metadata they use.

Difficult Decisions

There are some fun challenges to building systems at Crossref. We have a lot of data. Our schema is very diverse, and has a vast amount of domain knowledge embedded in it. It’s changed over time to accommodate 20 years of scholarly publishing innovations. Our community is diverse too, from small one-person publishers with a handful of articles, through to large ones that publish millions.

What might be an obvious decision for a database table with a thousand rows doesn’t always translate to a million. When you get to a billion, things change again. An initially sensible choice might not scale. And a scalable solution might look over-engineered if we had millions of DOIs, rather than hundreds of millions.

The diversity of the data also poses challenges. A very simple feature might get complicated or expensive when it meets the heterogeneity of our metadata and membership. What might scale for journal article or grant metadata might not work for book chapters.

The big decisions need careful discussion, experimentation, and justification.

2NF or not 2NF

One such recent decision was how we structure our SQL schema for the database that powers our new ‘relationships’ REST API endpoint, currently in development.

The data model is simple: we have a table of Relationships which connect pairs of Items. And each Item can have properties (such as a type). The way to model this is straightforward, following conventional normalization rules:

We built the API around it, and all was well.

We then added a feature which lets you look up relationships based on the properties of the subject or object. For example “find citations where the subject is an article and the object is a dataset”. This design worked well in our initial testing. We loaded more data into it, and it continued to work well.

And then, the context changed. Once we tested loading a billion relationships in the database, the performance dropped. The characteristics of the data: size, shape and distribution, reached a point where the database was unable to run queries in a timely way. The PostgreSQL query planner became unpredictable and occasionally produced some quite exciting query plans (to non-technical readers: databases are neither the time nor the place for excitement).

This is a normal experience in scaling up a system. We expected that something like this would happen at some point, but you don’t know when it will happen until you try. We bounced around some ideas and came up with a couple of alternatives. Each made trade-offs around processing time, data storage and query flexibility. The best way to evaluate them was to use real data at a representative scale.

One of the options was denormalisation. This is a conventional solution to this kind of problem, but was not our first choice as it involves extra machinery to keep the data up-to-date, and more storage. It would not have been the correct solution for a smaller dataset. But we had the evidence that the other two approaches would not scale predictably.

By combining the data into one table, we can serve up API requests much more predictably, and with much better performance. This code is now running with the right performance. Technical readers note that this diagram is simplified. The real SQL schema is a little different.

Without writing this history down, and explaining what we tried, someone might misunderstand the reason for the code and try to simplify it. Decision record DR-0500 guards against that.

But one day, when the context changes, future developers will be able to come back and modify the code, because they understand why it was like that in the first place.

Renewed persistence

Joe Wass — Sat, 01 Apr 2023 00:00:00 +0000

We believe in Persistent Identifiers. We believe in defence in depth. Today we’re excited to announce an upgrade to our data resilience strategy.

Defence in depth means layers of security and resilience, and that means layers of backups. For some years now, our last line of defence has been a reliable, tried-and-tested technology. One that’s been around for a while. Yes, I’m talking about the humble 5¼ inch floppy disk.

This may come as surprise to some. When things go well, you’re probably never aware of them. In day to day use, the only time a typical Crossref user sees a floppy disk is when they click ‘save’ (yes, some journals still require submissions in Microsoft Word).

History

But why?

Let me take you back to the early days of Crossref. The technology scene was different. This data was too important to trust to new and unproven technologies like Zip disks, CD-Rs or USB Thumb Drives. So we started with punched cards.

IBM 5081-style punched card.

Punched cards are reliable and durable as long as you don’t fold, spindle or mutilate them. But even in 2001 we knew that punched cards’ days were numbered. The capacity of 80 characters kept DOIs short. Translating DOIs into EBCDIC made ASCII a challenge, let alone SICIs. We kept a close eye on the nascent Unicode.

Breathing Room

In 2017 the change of DOI display guidelines from http://dx.doi.org.pluma.sjfc.edu to https://doi-org.pluma.sjfc.edu shortened each DOI by 2 characters, buying us some time. But eventually we knew we had to upgrade to something more modern.

So we migrated to 5¼ inch floppy disks.

5¼ Floppy disk in drive

At 640 KB per disk these were a huge improvement. We could fit around 20,000 DOIs on one floppy. Today we only need around 10,000 floppy disks to store all of our DOIs (not the metadata, just the DOIs). Surprisingly this only takes about 20 metres of shelf space to store.

Typical work from home setup. Getting ready to backup some DOIs!

The move to working-from-home brought an unexpected benefit. Staff mail floppy disks to each other and keep them in constant rotation, which produces a distributed fault tolerant system.

Persistence Means Change

But it can’t last forever. DOIs registration shows no sign of slowing down. It’s clear we need a new, compact storage medium. So, after months of research, we’ve invested in new equipment.

Today we announce our migration to 3½ inch floppies.

If it goes to plan you won’t even notice the change.

Image credits

Punched card: IBM 5081-style punched card. Derived from public domain by Gwern.

Forming new relationships: contributing to open source

Patrick Vale — Wed, 19 Oct 2022 00:00:00 +0000

TL;DR

One of the things that makes me glad to work at Crossref is the principles to which we hold ourselves, and the most public and measurable of those must be the Principles of Open Scholarly Infrastructure, or POSI, for short. These ambitions lay out how we want to operate - to be open in our governance, in our membership and also in our source code and data. And it’s that openness of source code that’s the reason for my post today - on 26th September 2022, our first collaboration with the JSON Forms open-source project was released into the wild.

Like most organisations, we depend heavily on open-source software for our operations - the software is universally available, generally high quality and ‘free’. And it’s easy to take that dependency, and the associated dependency on free time and effort on the part of the maintainers, for granted - but that’s not very sustainable. In fact, we believe relying on open-source software without helping to sustain it is an anti-pattern, and this project marks the start of our efforts to make funding open-source software a standard part of our technology budget.

This isn’t the first time we’ve supported or released open-source software. Indeed for the past few years, all our new software is open source, and we’re in the process of replacing old closed code with new, so that eventually all our code will be open source. But this is the first time we’ve contributed extensively to something that isn’t focussed primarily on us, and our services. This is a project that we will find very useful, but it is a general purpose tool, and it’s already gaining traction in the community.

Background and motivations

A while back, I was tasked to do a quick spike of work on testing the theory that we could use automated form generation tools to bring new interfaces to our users more quickly, and make them easier for “people who aren’t devs” to adapt and manage. We wanted to build a new user interface for registering content, and especially we wanted to make it easier for funders to register the grants they were awarding. As well as being more approachable by a less-technical audience, we also wanted these forms to be accessible (in terms of a11y and users of assistive technology) and localisable - we wanted a solution that would cater to the needs of our rapidly diversifying membership.

Enter JSON Schema

We were clear about one side of the puzzle - we knew that we had to look beyond the XML ecosystem upon which much of our existing system is built - and landed on JSON Schema. JSON Schema is a ‘vocabulary that allows you to annotate and validate JSON documents’. This means you can describe the shape you expect your data to take, and apply constraints-based validation to that. Which means, in terms of a form library, that you can infer the structure of the form and test that the data entered into it matches what you expect. More than that, you can use that built-in validation to provide error messages to help people get the data right, first time.

Working backwards from the outcome, the argument for adopting JSON Schema is compelling. It provides a mechanism for checking that data you are handling (for example, receiving input from a form) conforms to the constraints that you declare, but also allows you to tell people up-front, in a human and machine-readable way, what structure and format you will accept. This closed-loop of data annotation and validation gets more appealing when you look at the wide adoption of JSON Schema across languages and libraries. You can pretty much guarantee that for whatever client or server -side technology you are using, there will be a JSON Schema validator for it. Being able to share schemas across your systems (and equally importantly, with third parties) moves JSON schema from ‘just’ being about data validation, to a key supportive technology.

Building a form derived from a JSON Schema is an equally attractive prospect. JSON Schema was conceived during the AjaxWorld conference in 2007 as a ‘JSON-based format for defining the structure of JSON data’, and its use as a form-generation tool is relatively new, but there is growing community interest. There is even a discussion about how to best create a JSON Schema vocabulary, specifically geared towards addressing some of the needs of form generation users. However, even in its current form, a JSON Schema can be passed to a library, and a very serviceable user interface appears. The devil is always in the detail, and the client-side libraries differ in their abilities to customise areas such as layout (you may not always want your form fields to appear in exactly the same order as they do in your JSON Schema), custom elements (you might want something that wasn’t a form input, or that changes based on user input) and localisation. The ability to flexibly customise the appearance and behaviour of the interface was a key factor in our selection of a client-side form generation library.

Choosing a library

The other side of the puzzle was less clear - choosing a UI library that would take this JSON Schema, and turn it into a useful, and usable, form. I made the prototype using the venerable React JSON Schema form. This worked well as a proof of concept, but veered dramatically off our chosen Frontend stack of VueJS and Vuetify, and had some architectural constraints that would limit the scope of customisations we could make to our forms. So I went off looking for libraries that would work with our stack and came up with Vuetify JSON Schema Form, and JSON Forms.

Vuetify JSON Schema Form matched our stack perfectly, but made some interesting decisions about the layout of data within the form, and that wouldn’t suit our purposes without dramatic modification.

JSON Forms was an abstracted library, with a core handling the JSON Schema transformation and validation, and separate rendering libraries to handle the form generation. This was great - they had renderers for Angular, React, and even some support for VueJS. But not Vuetify.

Clearly, we were going to have to make something.

We made contact with the maintainers of both short-listed libraries to see how we could collaborate in creating a tool that would meet all of our (and hopefully, much of the wider community’s) requirements. Both maintainers were very helpful, and we had constructive discussions in both cases. In the end, we decided that the abstracted nature of the JSON Forms project was a better fit for our needs, providing a flexible platform on which we - and others - could extend. We were fortunate to receive funding from the Gordon and Betty Moore Foundation (Grant Agreement #10485) in order to accelerate this work, so we could provide a Grant Registration UI more quickly. We paid a large portion of that funding to the library maintainers, and Crossref contributed a portion of my time on the project. This allowed us to enter into an agreement with EclipseSource, the maintainers of JSON Forms, to collaboratively develop the new VueJS and Vuetify renderer library. Stefan Dirix, the lead maintainer, worked with me to build it.

We didn’t forget about Vuetify JSON Schema Form though, and by way of appreciation for their help in the early stages, Crossref made a contribution towards the continued development of that library.

JSON Forms - now with Vuetify

Work started on the JSON Forms Vuetify renderer set in September 2021 - Stefan quickly created the first early prototypes of the new form renderers - but then we had a stroke of luck. Our repository received more input from the community. The one that made us sit up and take real notice was the news that someone else had already ported the JSON Forms React renderer set to Vue/Vuetify - and was offering this as a contribution. Krasimir Chobantonov’s fantastic first contribution got merged in at the end of the month. This propelled the project forward massively, and was an early validation of the value of working in the open. Needless to say, we were very grateful. Another example of the open source value chain was that Stefan - as the maintainer - could take the time to carefully review and tidy up the incoming code, so what was merged was the product of two great developers.

Having this great head start meant we could turn our attention to one of the other big areas we wanted to get right - localisation. Traditionally, JSON Schema -generated forms have handled localisation (translation of text and adjustment of date and numerical formats) by wholesale duplication and translation of the schema. This is cumbersome, and doesn’t integrate very well with custom error messages, nor external sources of interface messages (think form labels, descriptions, placeholders). So Stefan came up with a proposal, which we accepted, to add complete i18n support to the library. We now have a mechanism by which you can hook up a translation engine of your choice, and JSON forms will use that to lookup messages, before falling back to the validator (also localised!) and finally, the JSON Schema’s defaults. This gives much stronger integration and allows the community to plug in their existing localisation methods - no wasted effort.

Since the localisation addition, we’ve been working on fine-tuning the layout engine, making bug fixes, and integrating more closely with the underlying Vuetify library. This allows developers to more easily use the existing Vuetify parameters to change the style and behaviour of their form widgets. Again, no wasted effort.

We’re lucky to have an active community - @kchobantonov continues to make great contributions and push the library forward in unexpected ways - and the library is gaining popularity, with an average of a few hundred downloads per day.

Some of our funder members have already seen this work in action, and given their feedback on early iterations of the user interface that supports registering grant records. We’ll be releasing this publicly very soon to get feedback from members - and then using that feedback to iterate on the grants registration form, and look towards extending it to other record types.

Open source POSItivity

A continuous theme throughout this project has been the willingness of people working on these open source projects to be generous with their time and experience. Whether it has been form generation libraries, the JSON Schema project or maintainers of localisation plug-ins - help, advice and encouragement have never been far away. And that’s appreciated. But it’s not something that we, or any other organisation who relies on the software they produce, should take for granted. Open source software helps everyone who uses it, and there’s a real opportunity within our community to make meaningful steps towards supporting its sustainability. Ironically, it’s often the most-used general purpose tools that get the least attention. We can change that.

Look out for more

Look out for more posts from the engineering team, coming soon!

References

JSON Binpack: A space-efficient schema-driven and schema-less binary serialization specification based on JSON Schema (Chapter 3.2.1 History and Relevance)

https://web.archive.org/web/20071026190426/http://www.json.com/2007/09/27/json-schema-proposal-collaboration/

Linking Publications to Data and Software

Jennifer Lin — Wed, 07 Sep 2016 00:00:00 +0000

TL;DR

Crossref and Datacite provide a service to link publications and data. The easiest way for Crossref members to participate in this is to cite data using DataCite DOIs and to include them in the references within the metadata deposit. These data citations are automatically detected. Alternatively and/or additionally, Crossref members can deposit data citations (regardless of identifier) as a relation type in the metadata. Data & software citations from both methods are freely propagated. This blog post also describes how to retrieve the links collected between publication and data & software.

Data & software citation is good research practice (DataCite-STM Joint Statement and FORCE11 Joint Declaration of Data Citation Principles) and is part of the scholarly ecosystem supporting research validation and reproducibility. Data & software citation is also instrumental in enabling the reuse and verification of these research outputs, tracking their impact, and creating a scholarly structure that recognises and rewards those involved in producing them.

Crossref supports the propagation of data & software citations alongside a publisher’s standard bibliographic metadata. members deposit the data citation link as part of the overall publication metadata when registering their content. Crossref partners with DataCite and together, we jointly provide a clearinghouse for the citations collected. These are all made freely available to the community as open data.

Citation practices are evolving across different communities of practice. Crossref’s offering is flexible and easily accommodates variations and changes, since it does not rely on a specific set of citation metadata elements, citation format, nor manner of credit and attribution. Publishers deposit data & software citations in their metadata deposit via a) references and/or b) relation type.

Method A: Bibliographic references

Crossref and DataCite have partnered to provide automatic linking between publications registered with Crossref and datasets bearing DataCite DOIs. This is the most efficient and effective way to ensure that data citations are fully integrated into the scholarly research information network with full and accurate metadata.

All data & software citations that include datasets bearing a DataCite DOI are eligible for auto-update linking with Crossref. In this method: authors cite the dataset or software containing the DataCite DOI per journal article submission guidelines and add it to the article citation list (c.f. FORCE11 citation placement, FORCE11 Software Citation Principles). Publishers then deposit references as part of their standard practice when registering content. Crossref checks every reference deposited for a DOI. If the DOI is identified as DataCite’s, we automatically link it to the article. With this method, no additional action is needed when publishers register their content with Crossref.

Data citation links to non-DataCite DOIs can only be exposed in the references if the publisher makes references openly available. Even in the event that the data citation is shared, it remains undifferentiated from other references. Method B described below offers another approach.

Method B: Relation type

Publishers can link their publication to a variety of associated research objects as part of the article metadata directly in the metadata deposited to Crossref, including data & software, protocols, videos, published peer reviews, preprints, conference papers, etc. Doing so not only groups digital objects together, but formally associates them with the publication. Each link is a relationship and the sum of all these relationships constitutes a ‘research article nexus.’ Data & software citations are a valuable part of this.

To tag the citation in the metadata deposit, we ask for:

description of dataset or software (optional)

dataset or software identifier

identifier type

relationship type.

Crossref can accommodate research outputs with any identifier, though we currently only validate DOI relationships during metadata processing. Technical details are documented in the [Data & Software Citations Deposit Guide][4].

Combining methods increases total available citations

The two methods are independent and can be used exclusively or jointly. Each caters to a different set of conditions and their practical considerations. See the comparison of benefits and limitations for each method in the deposit guide. We recommend that publishers use both methods where possible at this time for optimum specificity and coverage.

How to access data & software citations

Crossref and DataCite make the data & software citations deposited by Crossref members and DataCite data repositories openly available to a wide host of parties, including both Crossref and DataCite communities as well as the extended research ecosystem (funders, research organisations, technology and service providers, research data frameworks such as Scholix, etc.).

Data & software citations from references can be accessed via the Crossref Event Data API Citations included directly into the metadata by relation type can be accessed via Crossref’s APIs in a number of formats (REST, OAI-PMH, OpenURL). (A single channel containing data & software citations across interfaces is in development and will be released next year.)

Publishers, visit our detailed guide on how to deposit data and software citations. We welcome your questions and concerns at feedback@crossref.org.

Special thanks to the following who provided valuable feedback in developing the guide: Martin Fenner (DataCite), Amye Kenall (Springer Nature), Brooks Hanson (AGU), Shelley Stall (AGU), and the FORCE11 Data Citation Implementation Pilot publisher’s subgroup.