Code on Crossref

Mending Chesterton's fence: open source decision-making

Joe Wass — Mon, 18 Mar 2024 00:00:00 +0000

When each line of code is written it is surrounded by a sea of context: who in the community this is for, what problem we’re trying to solve, what technical assumptions we’re making, what we already tried but didn’t work, how much coffee we’ve had today. All of these have an effect on the software we write.

By the time the next person looks at that code, some of that context will have evaporated. There may be helpful code comments, tests, and specifications to explain how it should behave. But they don’t explain the path not taken, and why we didn’t take it. Or those occasions where the facts changed, so we changed our mind.

Some parts of our system are as old as Crossref itself. Whilst our process still involves coffee, it’s safe to say that most of our working assumptions have changed, and for good reasons! We have to be very careful when working with our oldest code. We always consider why it was written that way, and what might have changed since. We’re always on the look out for Chesterton’s Fence!

Leaving a Trail

We’re building a new generation of systems at Crossref, and as we go we’re being deliberate about supporting the people who will maintain it.

When our oldest code was written, the software development team all worked in an office with a whiteboard or three, and the code was proprietary. Twenty years later, things are very different. The software development team is spread over 8 timezones. Thanks to POSI, all the new code we write is open source, so the next people to read that code might not even be Crossref staff.

Working increasingly asynchronously, without that whiteboard, we need to record the options, collect evidence, and peer-review them within the team.

So for the past couple of years the software team has maintained a decision register. The first decision we recorded was that we should record decisions! Since then we have recorded the significant decisions as they arise. Plus some historical ones.

These aren’t functional specifications, which describe what the system should do. It’s the decisions and trade-offs we made along the way to get to the how. Look out for another blog post about specifications.

By leaving a trail of explanations as we go, we make it easier for people to understand why code was written, and what has changed. We’re writing the story of our new systems. This makes it easier to alter the system in future in response to changes in our community, and the metadata they use.

Difficult Decisions

There are some fun challenges to building systems at Crossref. We have a lot of data. Our schema is very diverse, and has a vast amount of domain knowledge embedded in it. It’s changed over time to accommodate 20 years of scholarly publishing innovations. Our community is diverse too, from small one-person publishers with a handful of articles, through to large ones that publish millions.

What might be an obvious decision for a database table with a thousand rows doesn’t always translate to a million. When you get to a billion, things change again. An initially sensible choice might not scale. And a scalable solution might look over-engineered if we had millions of DOIs, rather than hundreds of millions.

The diversity of the data also poses challenges. A very simple feature might get complicated or expensive when it meets the heterogeneity of our metadata and membership. What might scale for journal article or grant metadata might not work for book chapters.

The big decisions need careful discussion, experimentation, and justification.

2NF or not 2NF

One such recent decision was how we structure our SQL schema for the database that powers our new ‘relationships’ REST API endpoint, currently in development.

The data model is simple: we have a table of Relationships which connect pairs of Items. And each Item can have properties (such as a type). The way to model this is straightforward, following conventional normalization rules:

We built the API around it, and all was well.

We then added a feature which lets you look up relationships based on the properties of the subject or object. For example “find citations where the subject is an article and the object is a dataset”. This design worked well in our initial testing. We loaded more data into it, and it continued to work well.

And then, the context changed. Once we tested loading a billion relationships in the database, the performance dropped. The characteristics of the data: size, shape and distribution, reached a point where the database was unable to run queries in a timely way. The PostgreSQL query planner became unpredictable and occasionally produced some quite exciting query plans (to non-technical readers: databases are neither the time nor the place for excitement).

This is a normal experience in scaling up a system. We expected that something like this would happen at some point, but you don’t know when it will happen until you try. We bounced around some ideas and came up with a couple of alternatives. Each made trade-offs around processing time, data storage and query flexibility. The best way to evaluate them was to use real data at a representative scale.

One of the options was denormalisation. This is a conventional solution to this kind of problem, but was not our first choice as it involves extra machinery to keep the data up-to-date, and more storage. It would not have been the correct solution for a smaller dataset. But we had the evidence that the other two approaches would not scale predictably.

By combining the data into one table, we can serve up API requests much more predictably, and with much better performance. This code is now running with the right performance. Technical readers note that this diagram is simplified. The real SQL schema is a little different.

Without writing this history down, and explaining what we tried, someone might misunderstand the reason for the code and try to simplify it. Decision record DR-0500 guards against that.

But one day, when the context changes, future developers will be able to come back and modify the code, because they understand why it was like that in the first place.

Renewed persistence

Joe Wass — Sat, 01 Apr 2023 00:00:00 +0000

We believe in Persistent Identifiers. We believe in defence in depth. Today we’re excited to announce an upgrade to our data resilience strategy.

Defence in depth means layers of security and resilience, and that means layers of backups. For some years now, our last line of defence has been a reliable, tried-and-tested technology. One that’s been around for a while. Yes, I’m talking about the humble 5¼ inch floppy disk.

This may come as surprise to some. When things go well, you’re probably never aware of them. In day to day use, the only time a typical Crossref user sees a floppy disk is when they click ‘save’ (yes, some journals still require submissions in Microsoft Word).

History

But why?

Let me take you back to the early days of Crossref. The technology scene was different. This data was too important to trust to new and unproven technologies like Zip disks, CD-Rs or USB Thumb Drives. So we started with punched cards.

IBM 5081-style punched card.

Punched cards are reliable and durable as long as you don’t fold, spindle or mutilate them. But even in 2001 we knew that punched cards’ days were numbered. The capacity of 80 characters kept DOIs short. Translating DOIs into EBCDIC made ASCII a challenge, let alone SICIs. We kept a close eye on the nascent Unicode.

Breathing Room

In 2017 the change of DOI display guidelines from http://dx.doi.org.pluma.sjfc.edu to https://doi-org.pluma.sjfc.edu shortened each DOI by 2 characters, buying us some time. But eventually we knew we had to upgrade to something more modern.

So we migrated to 5¼ inch floppy disks.

5¼ Floppy disk in drive

At 640 KB per disk these were a huge improvement. We could fit around 20,000 DOIs on one floppy. Today we only need around 10,000 floppy disks to store all of our DOIs (not the metadata, just the DOIs). Surprisingly this only takes about 20 metres of shelf space to store.

Typical work from home setup. Getting ready to backup some DOIs!

The move to working-from-home brought an unexpected benefit. Staff mail floppy disks to each other and keep them in constant rotation, which produces a distributed fault tolerant system.

Persistence Means Change

But it can’t last forever. DOIs registration shows no sign of slowing down. It’s clear we need a new, compact storage medium. So, after months of research, we’ve invested in new equipment.

Today we announce our migration to 3½ inch floppies.

If it goes to plan you won’t even notice the change.

Image credits

Punched card: IBM 5081-style punched card. Derived from public domain by Gwern.

Accidental release of internal passwords, & API tokens for the Crossref system

Geoffrey Bilder — Fri, 04 Oct 2019 00:00:00 +0000

TL;DR

On Wednesday, October 2nd, 2019 we discovered that we had accidentally pushed the main Crossref system as part of a docker image into a developer’s account on Docker Hub. The binaries and configuration files that made up the docker image included embedded passwords and API tokens that could have been used to compromise our systems and infrastructure. When we discovered this, we immediately secured the repo, changed all the passwords and secrets, and redeployed the system code. We have since been scanning all of our logs and systems to see if there has been any unusual activity that could be related to the exposure of the container.

Please note that no external data e.g. member passwords or personal information were exposed; our source code contains only internal passwords and ‘secrets’ such as API tokens.

Thankfully, the way in which these secrets were exposed (in compressed, binary files which were, in turn, in a Docker image) means that they were probably overlooked by the automated exploitation tools which focus on scanning source code. And, so far, we have seen nothing that would indicate that these passwords and secrets have been exploited. We will, of course, inform our members directly (and update this blog) if that changes.

More than you probably want to know

If you are continuing to read this, my guess is that you might have questions like:

Why are you doing something as silly as embedding secrets and passwords in your code?
And wait a minute… I thought Crossref code was open source?
And why is the director of strategic initiatives announcing this?

Let me answer these questions in random order.

In March 2019 I took over Crossref’s technical teams when Chuck Koscher announced that he would be retiring at the end of the year. I’m now the director of technology & research.

A few months earlier we had already concluded that a major portion of the Crossref system had accumulated 20 years of technical debt and that we were going to spend a significant portion of 2019 and 2020 paying down that debt.

Specifically, a lot of the code that runs Crossref was inherited from a third party who developed it back in the early 2000s. This means that, even though any new systems that we’ve developed since 2007 have been open-source, the code for the oldest parts of the system has remained closed because it contained potentially proprietary code as well as a lot of deprecated coding practices. Also - the architecture, the tooling, and the development processes behind the Crossref system had not changed much in those twenty years. It was fantastic architecture, tooling, and code for its time. But architectures that scale to millions of records need to change to handle hundreds of millions of records. Processes that work for configuring one service need to change when you are managing dozens of services. And support tools that work for a few hundred members break down when you are dealing with tens of thousands of members.

These parts of the Crossref system were decidedly not 12 factor. We were not using DevOps or SRE working practices to run them. And the bulk of that part of the system is still being run in a traditional data center.

But since March we have been slowly fixing that. In incremental steps. Some of which are visible as a side effect of the security incident that precipitated this blog post. For example, one of our first moves was to move our development to Gitlab. Even though a big chunk of the base Crossref code is still closed source, we saw moving to Gitlab as a priority because Gitlab offers a fantastic suite of tools to help automate and manage our deployments. Similarly, we have been Dockerizing the Crossref system so that it is easier to scale and run in different environments. And as part of this effort, we have spent a lot of time on the issue of how to best handle secrets. We knew our secrets management in this part of the codebase was horrible. We have been developing some experiments and infrastructure for handling these secrets securely. But we haven’t finished this work yet. And so the system slipped out into a public repo too early. Ironically, this too illustrates a fundamental change in the way we develop things. Our default is to be open and transparent. This case is currently an exception. An exception we want to eliminate, but one we are not ready to do yet. We have to audit and scrub the code first.

Yes, this incident has been embarrassing. But not nearly as embarrassing as the fact that Crossref has succumbed to a technology industry cliche. That we spent so much time growing and focusing on new features for our members, that we neglected some of the creaking infrastructure of our infrastructure.

And I should be clear about two things:

First, not all of our code is like this. We have, for a long time, been building open source software and using modern best practices for secrets management in our newer subsystems and services. The problems described above are confined to twenty-year-old-code that we didn’t write in the first place and that we had been avoiding refactoring.

And second, the technology team has been marvelous at responding to the challenge we face. They have adopted new processes and tools. They are learning new techniques. We are steadily chipping away at these problems.

It is generally considered bad practice to praise or reward technology teams for fire-fighting instead of fire prevention, but this may be the exception that proves the rule.

I was blown away by how the technology, product, and support teams worked together. When we discovered this problem, I sat at my desk in rural France and watched as staff from the UK, and all three US time zones shut down this problem in just a couple of hours. Obviously, I wish we hadn’t had the problem in the first place, but seeing their response did a great deal to encourage me that we are on the right track.

In any case, it looks like we’ve been lucky. And we’ll be working even harder to refactor our code, tools, and processes so that this kind of thing doesn’t happen again.