• Where: Fountain Inn, Cambridge, UK (OpenStreetMap)
• When: Feb 2nd 2012, from 7:30pm

Abstract

There has been growing interest, especially in government, in ‘open
data’. This talk will explain what open data is, why it is important
and go on to cover some of the Open Knowledge Foundation’s recent work
in this area.

About the Open Knowledge Foundation

The Open Knowledge Foundation has been a pioneer in the field of open
data since its inception in 2004. It works in a wide array of areas
from sonnets to statistics, genes to geodata. Its open-source CKAN software powers http://data.gov.uk/,
http://thedatahub.org/ and dozens of other open data hubs around the
world. For more information about the Foundation see http://okfn.org/
and http://okfn.org/projects.

read original post on Rufus Pollock's Site

Availability

• All week (in London Wednesday)

Last Week

• Lots of CKAN
  • Especially work on Recline Data Explorer. This is now fully functional and integrated into the DataHub, see e.g. this gold prices dataset
• Work on BibServer and Public Domain Works and its Public Domain Calculators
• Open Economics Hackday — it was great!
  • Working on YourTopia code – clean rewrite in pure js
  • Data wrangling: Economics datasets on the DataHub — up 10 in the last week
• Post on Dreams of a Unified Text
• Various administration

This Week

• Some core admin, including getting a standard spreadsheet template for budgeting and (simple) tasks/todos
• More on OKFN Labs with Tom Rees and others
• CKAN work and planning
• More work on YourTopia code with Open Economics team — hope to get it to the point where we can upgrade curr
  ent YourTopia to new codebase
• NESTA round-table on innovation on Wednesday
• Talking at Data Insights meetup in Cambridge on Thursday

read original post on Rufus Pollock's Site

• All week

Last Week

• Away until Friday when did a lot of email catching up
• Quite a bit of coding on Recline Data Explorer while away. Check out the demo

This Week

• CKAN – finishing and integrating Recline
• OKFN Labs
• BibServer Sprint

read original post on Rufus Pollock's Site

Availability

• Mon-Thurs: but in Boston Mon-Tues and travelling back to UK Wednesday

Week of 21st November

• In South Africa all week
• More meetings and coding with people in South Africa including Cape Town meetup
• More work on Community Dashboard with Tom Rees
• Integrated Data Explorer with CKAN

Week of 28th November

• Tues-Friday at: Bridging Transparency and Technology Session on Natural Resources
• Saturday: Open Data Day and RHOK. Specifically took part in the DataHub and OpenSpending related Seize the Data Day segment
  • Seize the data day: updated Data Patterns with a piece on using the DataHub and Webstore. Wrangling on Exchange Rate data and sqlite data pattern
  • In the RHOK part I started work on using the PyBossa microtask framework to integrate microtasking into existing crowdsource workflows with specific focus on a geocoding microtasker.
  • Code and README here https://github.com/rgrp/pybossa-geodemo
• Open Government Data Bulletin planning
• Start planning the Open Data Census

This Week

• Meetings in Boston
• Finish off PyBossa demo
• Recline Data Explorer
• OpenSpending work especially re finalizing dataset import process

read original post on Rufus Pollock's Site

CKAN and the theDataHub

• Finalized separation of software and site: CKAN = data hub software, theDataHub.org = community data hub site
• Worked heavily to develop CKAN as a product e.g. much improved website http://ckan.org/, nascent partner programme and more
• Major milestone with CKAN v1.5 release just 2 weeks ago. Result of 6 months of development with major new features.
• theDataHub has also been much improved with move to new servers, a new theme (default in CKAN v1.5) and deployment of ever more extensions
• Some personal items:
  • DataExplorer
  • UX work – http://wiki.ckan.org/UX
  • data package manager (dpm) and work on Data Packages spec

OpenSpending

• Two major point releases of OpenSpending software v0.10 and v0.11 (v0.11 just last week!). Huge maturing and development of the system. Backend architecture now finalized after a major refactor and reworking.
• Community has grown significantly with now almost 50 OpenSpending datasets on theDataHub.org and growing group of core “data wranglers”
• Spending Stories was a winner of the Knight News Challenge. Spending Stories will build on and extend OpenSpending.

Open Bibliography and the Public Domain

• The Public Domain Review goes from strength to strength
• BibServer has had a ground-up rewrite and can be seen online at http://bibsoup.net/
• Additional support from JISC for a second phase of JISC Open Bibliography project

Open Knowledge Foundation and the Community

• In September we received a 3 year grant from the Omidyar Network to help the Open Knowledge Foundation sustain and expand its community especially in the formation of new chapters
• Completed a major recruitment process in (Summer-Autumn 2011) to bring on more paid OKFN team members including community coordinators, foundation coordinator and developers
• The Foundation participated in launch of Open Government Partnership and CSO events surrounding the meeting
• Working groups continuing to develop. Too much activity to summarize it all here but some highlights include:
  • WG Science Coordinator Jenny Molloy travelling to OSS2011 in SF to present Open Research Reports with Peter Murray-Rust
  • Open Economics WG developing and Open Knowledge Index in August
  • Open Bibliography working group’s work on an Metadata guide.
  • Open Humanities / Open Literature working group winning Inventare Il Futuro competition with their idea to use the Annotator
• Development of new Local Groups and Chapters
  • Lots of ongoing activities in existing local groups and chapters such as those in Germany and Italy have
  • In addition, interest from a variety of areas in the establishment of new chapters and local groups, for example in Brazil and Belgium
• Start of work on OKFN labs
  • Alpha website at http://labs.okfn.org/
  • Dashboard sprint

Meetups and Events

• Regular OKFN organized Open Data meetup in london
• Open Government Data Camp

Talks and Events

• Attended Open Government Partnership meeting in July in Washington DC and launch event in New York in September
• Attended Chaos Computer Camp with other OKFNers in August near Berlin
• September: Spoke at PICNIC in Amsterdam
• October: Code for America Summit in San Francisco (plus meetings) – see partial writeup
• October: Open Government Data Camp in Warsaw (organized by Open Knowledge Foundation)
• November: South Africa – see this post on Africa@Home and Open Knowledge meetup in Cape Town

General

• Partcipation in meetings of the UK Public Sector Transparency Board
• Annotator has seen ongoing coding and growing use – including very promising integration of Annotator with textbooks thanks to Ewald Zietsman of Siyavula
• Co-coded PyBossa – an open source platform for crowd-sourcing online (volunteer) assistance to perform tasks that require human cognition, knowledge or intelligence (e.g. image classification, transcription, information location etc)

read original post on Rufus Pollock's Site

• All week
• In South Africa Mon-Thurs (return to UK Friday)

Activity

• All last week in South Africa for Shuttleworth Fellows meetup
• Post about meetups and hackdays in South Africa
• Humanities working group meetup
• Limited coding and contribution to CKAN, specifically on:
  • data package manager
  • Recline Data Explorer (with Max Ogden) – cf this ticket http://trac.ckan.org/ticket/1357
• Work on v0.11 release of OpenSpending. In particular updating and improving openspendingjs so that we have a proper bubbletree explorer. Check out, for example this bubbletree of UK gov expenditure in 2010 – which also includes a breakdown (see the donut ring) by region
• OKFN Dashboard hackday
• Africa at Home hackday where I worked on PyBossa – an opensource crowdsourcing engine. PyBossa is a port the BOSSA crowdsourcing engine to python (a bit like an opensource mechanical turk but without money!) and with an initial focus on helping scientists and other researchers crowd-source human problem-solving skills!

This Week

• More meetings and coding with people in South Africa including Cape Town meetup
• OpenSpending especially coding on ETL and loading system and getting data loaded!
• Finish off Community Dashboard with Tom Rees
• Data Explorer integrated into new CKAN resource pages

read original post on Rufus Pollock's Site

The event is free, but unfortunately invite only due to limited capacity -(email richard.steele@icaew.com to attend). So, if you don’t have anything planned and would like to find out more about the intersection of accountancy and open data (think OpenSpending) then why not come along!

Open Data: What, Why, How

There has been growing interest in many circles, and especially in government, regarding ‘open data’. In this lecture Rufus will explain what open data is, why it’s of interest to government and others, and finally explain how governments and others can ‘go open’.

Access to government data is essential to many of the webapps and digital services we’d like to see, from planning a journey to work to knowing where your taxes get spent. As well as covering the basic what, why, how of open data this lecture will look at examples of some of the most interesting work in this area and provide a vision for what the nascent open data ecosystem could look like.

read original post on Rufus Pollock's Site

• All week (though may be at Moz Festival at end of the week)

Activity

• Back last week from Open Government Data Camp
  • Mini-hackday with Pedro Markun and others on OKFN Community Dashboard
• OKFN Hiring
• Planning of OKFN Labs
• Posts about Shuttleworth Fellowship renewal and plans for next year
• Limited coding and contribution to CKAN, specifically v0.9 release of datapkg (now renamed to dpm) and work on Recline / Data Explorer. Also created several datasets on theDataHub
• Getting up to speed on OpenSpending so I can help out with upcoming revamp, plus work on issues and User Stories
• sysadmin work – helped migrate some community CKANs

This Week

• Administration of various kinds (fun, fun, fun!)
• OpenSpending plus more Data Package Manager and Recline / Data Explorer
• More planning of OKFN Labs with Tom Rees
• Mozilla Festival

read original post on Rufus Pollock's Site

Not only was it great to have the resource to dedicate myself full-time to this area, but as a Fellow you get the (rare) chance to be part of a small, tight-knit group of really smart, interested and, well, just, incredibly nice set of people who share your interests and are able to offer constructive, critical feedback and support on what you are doing — the other place I know like this is the Open Knowledge Foundation .

(By the way, deadline for Spring 2012 Fellowship applications closes on the 1st of Nov and it’s really easy to apply so if you’re interested it’s not too late, just head over to Shuttleworth Fellowship application page right now).

I’ve really enjoyed my time as a Fellow over the last year and I’m delighted to report that Shuttleworth have been kind enough to accept my reapplication and renew my fellowship for a further year. Reflecting the evolution and maturing of the open data ecosystem this year’s proposal was entitled “Scaling the Open Data Ecosystem” to contrast with last year’s “Bootstrapping”.

In a follow up post next week I’ll be outlining what I’m planning to get up to in the coming year with Shuttleworth’s support.

read original post on Rufus Pollock's Site

• Very limited due to Open Government Data Camp

Activity

• In San Francisco last week Tuesday – Saturday. Attended Code for America Summit and had a variety of useful meetings including with Wikimedia Foundation folks including Erik Möller and Dario Taraborelli, Dan Whaley of Hypothes.is and Max Ogden.
  • Below are some written up notes from excellent chat with Max

This Week

• Open Government Data Camp – come join us in Warsaw for the world’s biggest open government data event!

Max Ogden Chat

1. Changes protocol

Want a general changes protocol for data (can we generalize CouchDB’s _changes to lots of other stuff including the Webstore)

• A changes protocol => syncing between DBs => support for updates and distributed systems

2. Diffing and merging

• We lack a diff (patch) format and a merge protocol (see previous posts We Need Distributed Revision Control for Data and (older) Collaborative Development of Data)
• Note these issues are related but not identical — full-on merging, as in e.g. git or mercurial is about more than simple patch application.
• Diff format options (are there more?)
  1. Brute force: e.g. serialize to text and use git
  2. Identify atomic structure (e.g. document) and apply diff at that level (think CouchDB or standard copy-on-write for RDBMS at row level)
  3. Recording transforms (e.g. Refine)
• Capturing diffs at document level in a given system e.g. CouchDB (trivial as can just provide new and old document) or an SQL database (approx = Write-Ahead Log) isn’t that hard though often not immediately exposed (as with SQL) and, more importantly, specific to the data store and, often, type of data
• Aside: if using something like CouchDB how would we capture edits when editing ‘offline’ e.g. on CSV version locally

3. Micro Schemas

Micro-schemas = just enough of standardized schema to apps and tools together with data in a minimal way

• Micro-schemas = small schemas (“conventions”) for data that would support interoperability and allow for generic tools and apps.
• Simple example: one would have a geodata micro-schema for tabular data that required every dataset with a long / lat for points to have a field named geometry containing geojson representing that point. That way one could build a data browser to present any such dataset.
• Not only should these schemas be very simple, they would be combinable so a given dataset could support many of these.
• Aside: I’d sort of prefer the term knowledge or information API here as it would suggest all the nice analogies with code: schemas are just liked APIs but for information: they provide a standard way for other systems to interact with that material. Plus, things like the combinability I just referred to in the previous point, have a nice analogy with MixIns or Abstract interfaces in code where a given Class can implement many MixIns or Abstract interfaces. However, I fear the term API may just be too technical and “geeky”.

read original post on Rufus Pollock's Site