When the first railways were built in the nineteenth century, engineers laid tracks with varying widths; trains and carriages were manufactured to fit their local set. Later, when owners wanted to connect different railways together, their trains and rails proved incompatible. Most countries had to standardise their railways at great expense.

Today, for those building the infrastructure that connects the digital world, standardisation is just as important. As well as physical resources, we now move information from place to place, and to do this properly, we need data from different sources to be compiled to consistent standards.

To achieve this, organisations and individuals around the world are creating agreed frameworks for how to compile data. At face value, it’s straightforward work, fixing the basic plumbing that allows new digital services and products to be created. But getting people to agree can often be more difficult than aligning the technology.

• For more like this, see our digital government newsfeed.

Why standards?

Sharing data between organisations — public, private and academic — lies at the heart of most digital work. But often data isn’t compatible. Information in one dataset can be defined differently in another. The way data is organised in one set can make it unrecognisable in another system. It can be something as simple as the way you write a date.

Standards are documented agreements between digital organisations to define and organise data in the same way. Though they may seem obscure to non-specialists, they allow information to get from place to place. Without standards, developers have to spend their time and resources collecting and cleaning data, and not building services.

Though the wonders of AI and “the blockchain” capture the headlines, standards are the basic infrastructural reforms that mean systems can work together. Very often, the outcome isn’t particularly flashy.

In 2014, Steve Bennett, an open data advocate, wrote a set of standards to enable councils in Australia to open their data. They were adopted: they are now used widely across the state of Victoria to publish council information and are spreading to councils elsewhere in Australia. But for Bennett, communicating the benefits of effective standards is not straightforward.

“People look for some kind of high impact, high value stories,” said Bennett. “I think the reality is that you have a lot of very low impact ones. There’s a lot of organisations that use [the data], like some little company that’s running some simulation and just needs to know where the fire hydrants are.”

Standard deviation

While the benefits of data standards are clear enough, both the private and public sector often ignore them or resist them.

“When I first started talking to people about the need for standards, I actually met a surprising amount of ignorance among technologists,” said Greg Bloom, Chief Organising Officer of the Open Referral Initiative, a group which develops ways for digital organisations in the public and private sector to work together.

What Bloom found was that many technologists assumed that it was up to the largest and most dominant organisation within a field to set standards. “[Many] tended to insist that standards are not possible, that the only way to have a standard is for some organisation to win the market, and then to set the standard — we’ve seen over and over again how harmful that outcome is.”

If market leaders are the ones setting standards, smaller companies can be locked out. Adapting or “cleaning” data so that it can operate with other systems is costly and time-consuming. For small businesses, NGOs or community organisations, a lack of fair and open standards can put potentially lucrative work beyond their remit.

Dumping data

A lack of standards also hampers those working with government data. In recent years, ever more national and local governments have committed to publishing data on their sites, aiding transparency, and giving businesses or community groups the chance to work with it.

The benefits are clear — businesses can create valuable services, for instance journey planners, that public bodies don’t have the expertise or time to. But this doesn't happen if governments publish data without thinking about whether it can be used.

“People look for some kind of high impact, high value stories — I think the reality is that you have a lot of very low impact ones"

“We know very well that if you have an open data portal, very few people actually go and use the data,” said Jean-Noé Landry, executive director of Open North, a Canadian not-for-profit which helps government work with open data. “There's a gap between the promises of open data, which are principle and value-driven, and then the actual reality — being able to demonstrate returns on the investment.”

Setting the standard

Standards require consensus. There’s debate between whether standards should be set by one organisation, or made in the open, allowing all organisations with skin in the game to participate. When standards are open, all organisations can access them, and understand what they need to do to make their data and systems cross-compatible. When they aren’t, they can lock organisations out.

For smaller companies and government bodies which lack resources or influence, participating in negotiations can be difficult. Leigh Dodds, data infrastructure programme lead at the Open Data Institute (ODI), said that means the process needs to be open.

“[Open standards] are standards that are not just openly licensed, but are developed in the open — that means all kinds of organisations can participate in the development process and then be able to freely and easily implement the standard afterwards,” said Dodds.

Open negotiation over standards, in which all organisations can participate, is a means to make sure small organisations and individual developers aren’t excluded. With the open standards for data guidebook, the ODI has written a living document, helping organisations to navigate standards and contribute to its guidelines. It advises research — working out what standards exist so that they aren’t needlessly replicated.

“We know very well that if you have an open data portal, very few people actually go and use the data"

Often, creating standards works best through an ebbing process of consultation and negotiation. With Open Referral, Bloom has built up a community of technologists who work together to set standards. Negotiations are held online, and everyone is able to contribute to a standard set of documents. Landry’s Open North group similarly works with local governments in Ontario to create a set of open data standards in an effort to help them release more useful data that technologists can work with.

And what comes out at the end, after standards have been set, and people are able to exchange data? For Bennett, it can be hard to know. “One of the depressing things about working in this area is that you have very little visibility into what — if anything — people are doing with the data,” he said.

Setting standards that everyone can see and help shape is often difficult work, but necessary. Sometimes those who hold data are the least able to see the potential value in it: finding ways to get it into the right hands will only increase and improve the work that comes out of it. — Anoush Darabi

This piece previously said that the Steve Bennett's standards are widely used across South Australia. It has been amended to reflect the fact that they apply to all councils in Australia, and the majority of uptake has been in Victoria.

(Picture credit: Unsplash/Marcus Spiske)