This isn’t about being critical but about educating—providing examples of how not to do something may be one of the best ways of showing how to do it right. It also provides a source of good practice material for budding data wranglers!

Each “bad” dataset on gets a simple page that shows what’s wrong along with a preview or screenshot.

We’ve started to stock the site with some of the better examples of bad data that we’ve come across over the years. This includes machine-unreadable Transport for London passenger numbers from the London Datastore and a classic “ASCII spreadsheet” from the US Bureau of Labor Statistics.

We welcome contributions of new examples! Submit them here.

read original post on Rufus Pollock's Site

The post Bad Data: real-world examples of how not to do data appeared first on Shuttleworth Foundation.

Back on April 17 2012 the Northern Mariana Islands Retirement Fund attempted to file for bankruptcy under Chapter 11. There was some pretty interesting reading in their petition for bankruptcy including this section (para 10) which suggests some pretty bad public financial management (emphasis added):

“Debtor has had difficulty maintaining healthy funding levels due to a combination of factors, including: the failure of the Commonwealth’s central government and autonomous agencies to remit full employer contributions; a difficult investing climate over the most recent three to four years; and a benefit structure that has been continuously increased and made more generous by the Commonwealth government without a corresponding increase in funding to the Debtor to cover increased costs.”

The Government then made things even worse:

“These issues have combined to place the fund in a dire financial situation. Compounding this perfect storm, the Commonwealth Government has passed laws declaring payment holidays, diverting earmarked revenues from the Debtor and reducing contribution rates for the Commonwealth Government, its agencies and political subdivisions.”

And they are now very bankrupt:

“As a result, as of the date of Debtor’s last actuarial report as of October 1, 2009, Debtor’s actuary determined that Debtor’s obligation was approximately 38.8% funded. While the Debtor is currently engaged in compiling information necessary for an updated actuarial study, the Debtor estimates that it is currently 32% funded. Debtor’s financial condition continues to deteriorate, and, by management’s figures, the Debtor will deplete its assets by July, 2014 and thereafter be unable to provide any level of benefits to current and future Beneficiaries“

Remember what it means when a pension fund goes bankrupt: it means all those people who worked, often for many years, will suddenly receive nothing, often in a period of their life when they most need support.

Note that ultimately Marianas petition for bankdruptcy got refused – as a governmental unit they are not eligible for Chapter 11. More on this and its relation to Puerto Rico situation at http://www.distressed-debt-investing.com/2013/10/distressed-debt-investing-puerto-rico.html

read original post on Rufus Pollock's Site

The post Northern Mariana Islands Retirement Fund Bankruptcy appeared first on Shuttleworth Foundation.

We’ve all come across “issues” with data, whether it’s “data” that turns out to be provided as a PDF, the many ways to badly format tabular data (empty rows, empty columns, inlined metadata …), “ASCII spreadsheets”, or simply erroneous data.

Key to starting to improve data quality is a way to report and record these issues.

We’ve thought about ways to address this for quite some time and, led by Labs member Friedrich Lindenberg, even experimented with building our own service. But recently, thanks to a comment from Labs member David Miller, we were hit with a blinding insight: why not do the simplest thing possible and just use an existing bug tracker tool? And so was born the current version of Data Issues based on a github issue tracker!

Aside: Before you decide we were completely crazy not to see this in the first place, it should be said that doing data issues “properly” (in the medium term) probably does require something a bit more than a normal bug tracker. For example, it would be nice to be able to both pinpoint an issue precisely (e.g. the date in column 5 on line 3751 is invalid) and group similar issues (e.g. all amounts in column 7 have a commas in them). Doing this would require a tracker that was customized for data. The solution described in this post, however, seems like a great way to get started.

Introducing Data Issues

Given the existence of so many excellent issue-tracking systems, we thought the best way to start is to reuse one—in the simplest possible way.

With Data Issues, we’re using GitHub Issues to track issues with datasets. Data Issues is essentially just a GitHub repository whose Issues are used to report problems on open datasets. Any problem with any dataset can be reported on Data Issues.

To report an issue with some data, just open an issue in the tracker, add relevant info on the data (its URL, who’s responsible for it, the line number of the bug, etc.), and explain the problem. You can add labels to group related issues—for example, if multiple datasets from the same site have problems, you can add a label that identifies the dataset’s site of origin.

Straightaway, the issue you raise becomes a public notice of the problem with the dataset. Everyone interested in the dataset has access to the issue. The issue is also actionable: each issue contains a thread of comments that can be used to track the issue’s status, and the issue can be closed when it has been fixed. All issues submitted to Data Issues are visible in a central list, which can be filtered by keyword or label to zoom in on relevant issues. All of these great features come for free because we’re using GitHub Issues.

Get Involved

For Data Issues to work, people need to use it. If civic hackers, journalists, and other data wranglers learn about Data Issues and start using it to track their work on datasets, we might find that the problem of tracking issues with datasets has already been solved.

You can also contribute by helping develop the project into something richer than a simple Issues page. One limitation of Data Issues is that raising an issue does not actually contact the parties responsible for the data. Our next goal is to automate sending along feedback from Data Issues, making it a more effective bug tracker.

If you want to discuss new directions for Data Issues or point out something you’ve built that contributes to the project, get in touch via the Labs mailing list.

read original post on Rufus Pollock's Site

The post Tracking Issues with Data the Simple Way appeared first on Shuttleworth Foundation.

New data

New data contributed over the last couple of months is now validated and live – this includes new data for Switzerland, Greece, Brazil and the US. Huge thank-you to contributors here including Hannes, Charalampos, Augusto and Todd.

We also have pending data for Italy and China to get in once it has been reviewed, and we have data in progress for Canada!

We’d love to have more data – if you’re interested in contributing see https://github.com/okfn/publicbodies#contribute-data

Updated Schema for Data

Thanks to input from James McKinney and others we’ve reworked the schema quite extensively to match up as much as possible with the Popolo spec. You can see the new schema in the datapackage.json.

Or if you don’t like raw JSON so much in prettier HTML version on: http://data.okfn.org/community/okfn/publicbodies

Search support

We now have basic search support via google custom search: http://publicbodies.org/search

Get Involved

As always we’d love help! There is a full list of issues here and example items:

• Getting Descriptions for 130 EU Public Bodies
• Support for sending corrections / additions on the website
• Support for reconciliation (e.g. via nomenklatura)

read original post on Rufus Pollock's Site

The post PublicBodies.org – Update no. 2 appeared first on Shuttleworth Foundation.

I was delighted to see this post as it makes many points I deeply agree with – and have for some time. In fact, reading it gave me something a sense of (very positive) deja-vu since it made similar points to several posts I and others had written several years ago – suggesting that perhaps we’re now getting close to the critical mass we need to create a real distributed and collaborative open data ecosystem!

It also suggested it was worth dusting off and recapping some of this earlier material as much of it was written more than 6 years ago, a period which, in tech terms, can seem like the stone age.

Previous Thinking

For example, there is this essay from 2007 on Componentization and Open Data that Jo Walsh and I wrote for our XTech talk that year on CKAN. It emphasized analogies with code and the importance of componentization and packaging.

This, in turn, was based on Four principes for Open Knowledge Development and What do we mean by componentization for knowledge. We also emphasized the importance of “version control” in facilitating distributed collaboration, for example in Collaborative Development of Data (2006/2007) and, more recently, in Distributed Revision Control for Data (2010) and this year in Git (and GitHub) for Data.

Package Managers and CKAN

This also brings me to a point relevant both to Ben’s post and Michal’s comment: the original purpose (and design) of CKAN was precisely to be a package manager a la rubygems, pypi, debian etc. It has evolved a lot from that into more of a “wordpress for data” – i.e. a platform for publishing, managing (and storing) data because of user demand. (Note that in early CKAN “datasets” were called packages in both the interface and code – a poor UX decision that illustrated we were definitely ahead of our time – or just wrong!)

Some sense of what was intended is evidenced by the fact that in 2007 we were writing a command line tool called datapkg (since changed to dpm for data package manager) to act at the command equivalent of gem / pip / apt-get – see this Introducting DataPkg post which included this diagram illustrating how things were supposed to work.

Recent Developments

As CKAN has evolved into a more general-purpose tool – with less of a focus on just being a registry supported automated access – we’ve continued to develop those ideas. For example:

• The basic “package” idea from CKAN has evolved into the Data Package spec – and Simple Data Format
• We’ve explored storing data using code tools like git – with a dedicated datasets organization on Github
• We’ve re-booted the idea of a simple registry and storage mechanism in the form of http://data.okfn.org/ – with data stored in simple data format in git repos on github, displayed in a very simple registry with good tool integration, and curated by a dedicated group of maintainers
• We’ve booted the “Frictionless Data” initiative as a way to bring together these different activities in one coherent vision of how we can do something simple to make progress

Data Packages and Frictionless Data – from data.okfn.org

read original post on Rufus Pollock's Site

The post Data as Code Deja-Vu appeared first on Shuttleworth Foundation.

An example

This takes the input data (sourced from this original Greater London Authority financial data), slices out the first 50 rows (head), deletes the first column (its blank!) (cut), deletes rows 1 through 7 (delete) and finally renders the result as HTML (html).

http://datapipes.okfn.labs.org/csv/head -n 50/cut 0/delete 1:7/html?url=https://raw.github.com/okfn/datapipes/master/test/data/gla.csv

Before

After

Motivation – Data Wrangling, Pipes, NodeJS and the Unix Philosophy

When you find data in the wild you usually need to poke around in it and then to some cleaning for it to be usable.

Much of the inspiration for Data Pipes comes from our experience using Unix command lines tools like grep, sed, and head to do this kind of work. These tools a powerful way to operate on streams of text (or more precisely streams of lines of text, since Unix tools process text files line by line). By using streams, they can scale to large files easily (they don’t load the whole file but process it bit by bit) and, more importantly, allow “piping” – that is, direct connection of the output of one command with the input of another.

This already provides quite a powerful way to do data wrangling (see here for more). But there are limits: data isn’t always line-oriented, plus command line tools aren’t online, so it’s difficult to share and repeat what you are doing. Inspired by a combination of Unix pipes and the possibilities of NodeJS’s great streaming capabilities, we wanted to take the pipes online for data processing – and so Data Pipes was born.

We wanted to use the Unix philosophy that teaches us to solve problems with cascades of simple, composable operations that manipulate streams, an approach which has proven almost universally effective.

Data Pipes brings the Unix philosophy and the Unix pipes style to online data. Any CSV data can be piped through a cascade of transformations to produce a modified dataset, without ever downloading the data and with no need for your own backend. Being online means that the operations are immediately shareable and linkable.

More Examples

Take, for example, this copy of a set of Greater London Authority financial data. It’s unusable for most purposes, simply because it doesn’t abide by the CSV convention that the first line should contain the headers of the table. The header is preceded by six lines of useless commentary. Another problem is that the first column is totally empty.

First of all, let’s use the Data Pipes html operation to get a nicer-looking view of the table.

GET /csv/html/?url=http://static.london.gov.uk/gla/expenditure/docs/2012-13-P12-250.csv

Now let’s get rid of those first six lines and the empty column. We can do this by chaining together the delete operation and the cut operation:

GET /csv/delete 0:6/cut 0/html/?url=http://static.london.gov.uk/gla/expenditure/docs/2012-13-P12-250.csv

And just like that, we’ve got a well-formed CSV!

But why stop there? Why not take the output of that transformation and, say, search it for the string “LONDON” with the grep transform, then take just the first 20 entries with head?

GET /csv/delete 0:6/cut 0/grep LONDON/head -n 20/html/?url=http://static.london.gov.uk/gla/expenditure/docs/2012-13-P12-250.csv

Awesome!

What’s next?

Data Pipes already supports a useful collection of operations, but it’s still in development, and more are yet to come, including find-and-replace operation sed plus support for arbitrary map and filter functions.

You can see the full list on the Data Pipes site, and you can suggest more transforms to implement by raising an issue.

Data Pipes needs more operations for its toolkit. That means its developers need to know what you do with data – and to think about how it can be broken down in the grand old Unix fashion. To join in, check out Data Pipes on GitHub and let us know what you think.

read original post on Rufus Pollock's Site

The post Data Pipes – streaming online data transformations appeared first on Shuttleworth Foundation.

Today we’re pleased to officially launch Open Knowledge Foundation Labs, a community home for civic hackers, data wranglers and anyone else intrigued and excited by the possibilities of combining technology and open information for good – making government more accountable, culture more accessible and science more efficient.

Labs is about “making” – whether that’s apps, insights or tools – using open data, open content and free / open source software. And you don’t need to be an uber-geek to participate: interest and a willingness to get your hands dirty (digitally), be that with making, testing or helping, is all that’s needed – although we do allow lurking on the mailing list

Join in now! Sign up on the mailing list, follow us on twitter or read more about what we’re up to and how you can get involved.

Follow @OKFNLabs

Find out more

For the full picture of what Labs is up to, check out its projects page and the list of ideas for projects. Highlights include:

• ReclineJS, a library for building data-driven web applications in pure JavaScript
• Annotator, an open-source JavaScript library and tool that can be added to any webpage to make it annotatable
• Nomenklatura, a simple service that makes it easy to maintain a canonical list of entities such as persons, companies or streets, and to match messy input against that list
• PyBossa, a platform for crowd-sourcing online volunteer assistance on tasks that require human intelligence, which powers CrowdCrafting

Labs is part of the Open Knowledge Foundation Network and operates as a collaborative community, which anyone can join. You can also take a look at the:

• About Page
• Community principles
• FAQs

While some of you may have noticed we’ve been operating unannounced and somewhat under the radar for some time, we recently revamped the website and decided that it was high time to officially cut the ribbon and open our doors.

If you’re interested in making things with open data or open content, we hope you’ll come and say hello.

read original post on Rufus Pollock's Site

The post Introducing Open Knowledge Foundation Labs appeared first on Shuttleworth Foundation.

• When: Tuesday 16th July 2013
• Where: Centre for Creative Collaboration, 16 Acton Street, London.
• Signup: on Meetup page (optional but nice to know numbers!)

Look forward to seeing folks there!

What

Open Data Maker Nights are informal events focused on “making” with open data – whether that’s creating apps or insights. They aren’t a general meetup – if you come, expect to get pulled into actually building something, though we won’t force you!

Who

The events usually have short introductory talks about specific projects and suggestions for things to work on – it’s absolutely fine to turn up knowing nothing about data or openness or tech as, there’ll an activity for you to help and someone to guide you in contributing!

Organize your own!

Not in London? Why not organize your own Open Data Maker night in your city? Anyone can and it’s easy to do – find out more »

read original post on Rufus Pollock's Site

The post Open Data Maker Night London No 3 – Tuesday 16th July appeared first on Shuttleworth Foundation.

This Autumn the Open Government Partnership Annual Conference is coming to London and will place on the 31st October and 1st November. As a lead into the main event, OGP is planning a 1-day civil society
Pre-Conference event on Wednesday 30th October and we here at the Open Knowledge Foundation will be collaborating with them on it.

An informal group discussion on open government data

The aim is for this to be informal with lots of open space and a collaboratively organized schedule with activities and discussions like:

• What does civil society want from OGP?
• What’s next for open government data and open government?
• Small group conversations about challenges and what can be learnt from other initiatives like EITI, IATI and the like
• Workshops and data expeditions
• Space for individual communities groups to meet, share and plan
• Your suggestion here

If you’re interested you can pre-register now so as to notified once registration opens and more information becomes available.

Pre-register now »

Further details coming soon!

read original post on Rufus Pollock's Site

The post Save the Date – OGP Pre-Conference, London Wednesday 30th October appeared first on Shuttleworth Foundation.

The ability to do “version control” for data is a big deal. There are various options but one of the most attractive is to reuse existing tools for doing this with code, like git and mercurial. This post describes a simple “data pattern” for storing and versioning data using those tools which we’ve been using for some time and found to be very effective.

Introduction

The ability to do revisioning and versioning data – store changes made and share them with others – especially in a distributed way would be a huge benefit to the (open) data community. I’ve discussed why at some length before (see also this earlier post) but to summarize:

• It allows effective distributed collaboration – you can take my dataset, make changes, and share those back with me (and different people can do this at once!)
• It allows one to track provenance better (i.e. what changes came from where)
• It allows for sharing updates and synchronizing datasets in a simple, effective, way – e.g. an automated way to get the last months GDP or employment data without pulling the whole file again

There are several ways to address the “revision control for data” problem. The approach here is to get data in a form that means we can take existing powerful distributed version control systems designed for code like git and mercurial and apply them to the data. As such, the best github for data may, in fact, be github (of course, you may want to layer data-specific interfaces on on top of git(hub) – this is what we do with http://data.okfn.org/).

There are limitations to this approach and I discuss some of these and alternative models below. In particular, it’s best for “small (or even micro) data” – say, under 10Mb or 100k rows. (One alternative model can be found in the very interesting Dat project recently started by Max Ogden — with whom I’ve talked many times on this topic).

However, given the maturity and power of the tooling – and its likely evolution – and the fact that so much data is small we think this approach is very attractive.

The Pattern

The essence of the pattern is:

1. Storing data as line-oriented text and specifically as CSV¹ (comma-separated variable) files. “Line oriented text” just indicates that individual units of the data such as a row of a table (or an individual cell) corresponds to one line².

2. Use best of breed (code) versioning like git mercurial to store and manage the data.

Line-oriented text is important because it enables the powerful distributed version control tools like git and mercurial to work effectively (this, in turn, is because those tools are built for code which is (usually) line-oriented text). It’s not just version control though: there is a large and mature set of tools for managing and manipulating these types of files (from grep to Excel!).

In addition to the basic pattern, there are several a few optional extras you can add:

• Store the data in GitHub (or Gitorious or Bitbucket or …) – all the examples below follow this approach
• Turn the collection of data into a Simple Data Format data package by adding a datapackage.json file which provides a small set of essential information like the license, sources, and schema (this column is a number, this one is a string)
• Add the scripts you used to process and manage data — that way everything is nicely together in one repository

What’s good about this approach?

The set of tools that exists for managing and manipulating line-oriented files is huge and mature. In particular, powerful distributed version control systems like git and mercurial are already extremely robust ways to do distributed, peer-to-peer collaboration around code, and this pattern takes that model and makes it applicable to data. Here are some concrete examples of why its good.

Provenance tracking

Git and mercurial provide a complete history of individual contributions with “simple” provenance via commit messages and diffs.

Peer-to-peer collaboration

Forking and pulling data allows independent contributors to work on it simultaneously.

Data review

By using git or mercurial, tools for code review can be repurposed for data review.

Simple packaging

The repo model provides a simple way to store data, code, and metadata in a single place.

Accessibility

This method of storing and versioning data is very low-tech. The format and tools are both very mature and are ubiquitous. For example, every spreadsheet and every relational database can handle CSV. Every unix platform has a suite of tools like grep, sed, cut that can be used on these kind of files.

Examples

We’ve been using with this approach for a long-time: in 2005 we first stored CSV’s in subversion, then in mercurial, and then when we switched to git (and github) 3 years ago we started storing them there. In 2011 we started the datasets organization on github which contains a whole list of of datasets managed according to the pattern above. Here are a couple of specific examples:

• Country, regional, and world GDP, collected from the World Bank and turned into a normalized CSV. The Python script used to clean the data is included in scripts.

• List of companies in Standard & Poor’s 500, an index of the top 500 publicly listed stocks in the US. Includes Python scripts to process data and instructions on how to replicate cleaned data from source. This dataset provides interesting examples of what diffs look like, including bad diffs (e.g.. a column rearrange).

Note Most of these examples not only show CSVs being managed in github but are also simple data format data packages – see the datapackage.json they contain.

Appendix

Limitations and Alternatives

Line-oriented text and its tools are, of course, far from perfect solutions to data storage and versioning. They will not work for datasets of every shape and size, and in some respects they are awkward tools for tracking and merging changes to tabular data. For example:

• Simple actions on data stored as line-oriented text can lead to a very large changeset. For example, swapping the order of two fields (= columns) leads to a change in every single line. Given that diffs, merges, etc. are line-oriented, this is unfortunate.³
• It works best for smallish data (e.g. < 100k rows, < 50mb files, optimally < 5mb files). git and mercurial don’t handle big files that well, and features like diffs get more cumbersome with larger files.⁴
• It works best for data made up of lots of similar records, ideally tabular data. In order for line-oriented storage and tools to be appropriate, you need the record structure of the data to fit with the CSV line-oriented structure. The pattern is less good if your CSV is not very line-oriented (e.g. you have a lot of fields with line breaks in them), causing problems for diff and merge.
• CSV lacks a lot of information, e.g. information on the types of fields (everything is a string). There is no way to add metadata to a CSV without compromising its simplicity or making it no longer usable as pure data. You can, however, add this kind of information in a separate file, and this exactly what the Data Package standard provides with its datapackage.json file.

The most fundamental limitations above all arise from applying line-oriented diffs and merges to structured data whose atomic unit is not a line (its a cell, or a transform of some kind like swapping two columns)

The first issue discussed below, where a simple change to a table is treated as a change to every line of the file, is a clear example. In a perfect world, we’d have both a convenient structure and a whole set of robust tools to support it, e.g. tools that recognize swapping two columns of a CSV as a single, simple change or that work at the level of individual cells.

Fundamentally a revision system is built around a diff format and a merge protocol. Get these right and much of the rest follows. The basic 3 options you have are:
* Serialize to line-oriented text and use the great tools like git (what’s we’ve described above)
* Identify atomic structure (e.g. document) and apply diff at that level (think CouchDB or standard copy-on-write for RDBMS at row level)
* Recording transforms (e.g. Refine)

At the Open Knowledge Foundation we built a system along the lines of (2) and been involved in exploring and researching both (2) and (3) – see changes and syncing for data on on dataprotocols.org. These options are definitely worth exploring — and, for example, Max Ogden, with whom I’ve had many great discussions on this topic, is currently working on an exciting project called Dat, a collaborative data tool which will use the “sleep” protocol.

However, our experience so far is that the line-oriented approach beats any currently available options along those other lines (at least for smaller sized files!).

data.okfn.org

Having already been storing data in github like this for several years, we recently launched http://data.okfn.org/ which is explicitly based on this approach:

• Data is CSV stored in git repos on GitHub at https://github.com/datasets
• All datasets are data packages with datapackage.json metadata
• Frontend site is ultra-simple – it just provides catalog and API and pulls data directly from github

Why line-oriented

Line-oriented text is the natural form of code and so is supported by a huge number of excellent tools. But line-oriented text is also the simplest and most parsimonious form for storing general record-oriented data—and most data can be turned into records.

At its most basic, structured data requires a delimiter for fields and a delimiter for records. Comma- or tab-separated values (CSV, TSV) files are a very simple and natural implementation of this encoding. They delimit records with the most natural separation character besides the space, the line break. For a field delimiter, since spaces are too common in values to be appropriate, they naturally resort to commas or tabs.

Version control systems require an atomic unit to operate on. A versioning system for data can quite usefully treat records as the atomic units. Using line-oriented text as the encoding for record-oriented data automatically gives us a record-oriented versioning system in the form of existing tools built for versioning code.

read original post on Rufus Pollock's Site

The post Git (and Github) for Data appeared first on Shuttleworth Foundation.