Highlights

• Substantial new project support from several funders including support for Science working group and Economics working group
• Our CKAN Data Management System selected in 2 major new data portal initatives
• Continuing advance of projects across the board with several projects reaching key milestones (v1.0 or beta release, adoption by third parties)
• Rapid expansion of chapters and local groups — e.g. London Meetup now has more than 100 participants, new chapters in Belgium and Switzerland are nearly finalized
• Completion of major upgrade of core web-presence with new branding and theme used on http://okfn.org/ and across our network of sites (now numbering more than 40)
• Announcement of School of Data which drew huge attention from the community. This is will be a joint Open Knowledge Foundation / P2PU project.
• Major strengthening of organizational capacity with new staff

Projects

Major new project support including:

• $124k grant from Alfred P. Sloan Foundation to support the development of the Open Economics Working Group
• Grant from Open Society Foundations for work associated to the OpenSpending project
• Grant from Open Society Foundation for Panton Fellowships
• Part of new European Union funded DM2E (Digital Manuscripts to Europeana) project

CKAN and the DataHub

• Won several competitive bids to deliver data portals with the Open Knowledge Foundation’s CKAN Data management System

  • Part of winning bid to deliver the European Commission’s new Data Portal which will be powered by CKAN
  • Together with new CKAN Partner New Amsterdam Ideas won bid to develop CMAP’s data portal in Illinois

• CKAN v1.6 Completed

  • New Javascript Data Explorer (Recline) – see it in action on this Dataset of ISO Currency Codes
  • New DataStore providing a rich Data API (NB: this will ship as part of v1.7 but is already live on the DataHub)
  • Continuing extensive UX improvements started 6m ago

• The DataHub passed 3000 datasets in early January and is nearing 3300

OpenSpending

• Major breakthrough with achievement of simple data upload and management process – result of more than 9 months of work
• OpenSpending now contains more than 30 datasets with ~7 million spending items (up from 2 datasets and ~200k items a year ago, and under 10 datasets a 1.5m items just 4 months ago)
• Substantial expansion in set of collaborators and a variety of new funding opportunities

Other Projects

• BibServer and BibSoup, our bibliogrpahic software and service, reached beta and have been receiving increasing attention

• Public Domain Review celebrated its 1st Birthday. Some stats:

  • The Review now has more than 800+ email subscribers, ~800 followers on Twitter
  • 20k visitors with over 40k page views per month
  • An increasing number of supporters making a monthly donation

• Initiated a substantive collaboration on the PyBossa crowdsourcing platform with Shuttleworth Fellow Emeritus Francois Grey and his Citizen Cyberscience Centre

  • New dedicated website: http://pybossa.com/

• Annotator and AnnotateIt v1.0 Completed and Released

  • Annotator is now seeing uptake from several third-party projects and developers
  • Project components now have more than 100 followers on GitHub (up from ~20 in December)

Working Groups and Local Groups and Chapters

Working groups have continued to develop well:

• New dedicated Working Group coordinator (Laura Newman)
• Panton Fellowships run under auspices of Science Working Group
• Funding of Economics Working Group

Rapid Chapter and local group development:

• Lots of new Open Knowledge Foundation meetup groups
• Belgium and Swiss groups nearly at Chapter status
• New Local Groups in several other countries – see http://okfn.org/chapters/ for overview
• New meetup system with meetups in more than a dozen countries. Last London Open Data Meetup attended by more than 100 people

Additional items

• Hired key new staff including developers, community coordinators and a new management team
• Begun a series of hackathons and hackdays under the auspices of the Open Knowledge Foundation Working Group on Economics
• Open Data Handbook v1.0 Released – http://opendatahandbook.org/
• In collaboration with P2PU and Philipp Schmidt announced the School of Data — huge interest from the community with the announce post having the most shares so far of any on the Open Knowledge Foundation blog
• Finalized Open Knowledge Foundation Labs proposal and launched basic website: http://okfnlabs.org/
• Invited to become a Fellow of the Royal Society of Arts
• Invited to be an Adviser to Canadian Government on Open Data

Events and Meetings

Participated in numerous events and meetings including:

• TA Tech (Transparency and Accountability Technology) meeting in New York
• LIFT 2012 in Geneva
• Dev8d
• Citizen Cyberscience Summit
• Transparency Board

read original post on Rufus Pollock's Site

CKAN and the theDataHub

• Finalized separation of software and site: CKAN = data hub software, theDataHub.org = community data hub site
• Worked heavily to develop CKAN as a product e.g. much improved website http://ckan.org/, nascent partner programme and more
• Major milestone with CKAN v1.5 release just 2 weeks ago. Result of 6 months of development with major new features.
• theDataHub has also been much improved with move to new servers, a new theme (default in CKAN v1.5) and deployment of ever more extensions
• Some personal items:
  • DataExplorer
  • UX work – http://wiki.ckan.org/UX
  • data package manager (dpm) and work on Data Packages spec

OpenSpending

• Two major point releases of OpenSpending software v0.10 and v0.11 (v0.11 just last week!). Huge maturing and development of the system. Backend architecture now finalized after a major refactor and reworking.
• Community has grown significantly with now almost 50 OpenSpending datasets on theDataHub.org and growing group of core “data wranglers”
• Spending Stories was a winner of the Knight News Challenge. Spending Stories will build on and extend OpenSpending.

Open Bibliography and the Public Domain

• The Public Domain Review goes from strength to strength
• BibServer has had a ground-up rewrite and can be seen online at http://bibsoup.net/
• Additional support from JISC for a second phase of JISC Open Bibliography project

Open Knowledge Foundation and the Community

• In September we received a 3 year grant from the Omidyar Network to help the Open Knowledge Foundation sustain and expand its community especially in the formation of new chapters
• Completed a major recruitment process in (Summer-Autumn 2011) to bring on more paid OKFN team members including community coordinators, foundation coordinator and developers
• The Foundation participated in launch of Open Government Partnership and CSO events surrounding the meeting
• Working groups continuing to develop. Too much activity to summarize it all here but some highlights include:
  • WG Science Coordinator Jenny Molloy travelling to OSS2011 in SF to present Open Research Reports with Peter Murray-Rust
  • Open Economics WG developing and Open Knowledge Index in August
  • Open Bibliography working group’s work on an Metadata guide.
  • Open Humanities / Open Literature working group winning Inventare Il Futuro competition with their idea to use the Annotator
• Development of new Local Groups and Chapters
  • Lots of ongoing activities in existing local groups and chapters such as those in Germany and Italy have
  • In addition, interest from a variety of areas in the establishment of new chapters and local groups, for example in Brazil and Belgium
• Start of work on OKFN labs
  • Alpha website at http://labs.okfn.org/
  • Dashboard sprint

Meetups and Events

• Regular OKFN organized Open Data meetup in london
• Open Government Data Camp

Talks and Events

• Attended Open Government Partnership meeting in July in Washington DC and launch event in New York in September
• Attended Chaos Computer Camp with other OKFNers in August near Berlin
• September: Spoke at PICNIC in Amsterdam
• October: Code for America Summit in San Francisco (plus meetings) – see partial writeup
• October: Open Government Data Camp in Warsaw (organized by Open Knowledge Foundation)
• November: South Africa – see this post on Africa@Home and Open Knowledge meetup in Cape Town

General

• Partcipation in meetings of the UK Public Sector Transparency Board
• Annotator has seen ongoing coding and growing use – including very promising integration of Annotator with textbooks thanks to Ewald Zietsman of Siyavula
• Co-coded PyBossa – an open source platform for crowd-sourcing online (volunteer) assistance to perform tasks that require human cognition, knowledge or intelligence (e.g. image classification, transcription, information location etc)

read original post on Rufus Pollock's Site

I’m contributing to a “primer” session on The Perspective of Open Data Communities and then giving a conference talk on Collective Costs and Benefits in Opening PSI for Re-use in a session on PSI Re-use: a Tool for Enhancing Competitive Markets where I’ll be covering work by myself and others on pricing and regulation of PSI (see e.g. the “Cambridge Study” and the paper on the Economics of the Public Sector of Information).

Update: slides are up.

Community, Openness And Technology

PSI: Costs And Benefits Of Openness

read original post on Rufus Pollock's Site

read original post on Rufus Pollock's Site

Creative Commons as a brand has been fantastically successful and is now very widely recognized. While in many ways this success has been beneficial for those interested in free/open material it has also raised some issues that are worth highlighting.

Creative Commons is not a Commons

Ironically, despite its name, Creative Commons, or more precisely its licenses, do not produce a commons. The CC licenses are not mutually compatible, for example, material with a CC Attribution-Sharealike (by-sa) license cannot be intermixed with material licensed with any of the CC NonCommercial licenses (e.g. Attribution-NonCommercial, Attribution-Sharealike-Noncommercial).

Given that a) the majority of CC licenses in use are ‘non-commercial’ b) there is also large usage of ShareAlike (e.g. Wikipedia), this is an issue affects a large set of ‘Creative Commons’ material.

Unfortunately, the presence of the word ‘Commons’ in CC’s name and the prominence of ‘remix’ in the advocacy around CC tends to make people think, falsely, that all CC licenses as in some way similar or substitutable.

The ‘Brand’ versus the Licenses

More and more frequently I hear people say (or more significantly write) things like: “This material is CC-licensed”. But as just discussed there is large, and very significant, variation in the terms of the different CC licenses. It appears that for many people the overall ‘Brand’ dominates the actual specifics of the licenses.

This is in marked contrast to the Free/Open Source software community, where even in the case of the Free Software Foundation’s licenses people tend to specify the exact license they are talking about.

Standards and interoperability are what really matter for licenses (cf the “Commons” terminology). Licensing and rights discussions are pretty dull for most people — and should be. They are important only because they determine what you and I can and can’t do, and specifically what material you and I can ‘intermix’ — possible only where licenses are ‘interoperable’.

To put it the other way round: licenses are interoperable if you can intermix freely material licensed under one of those licenses with material licensed under another. This interoperability is crucial and it is, in license terms, what underlies a true commons.

More broadly we are interested in a ‘license standard’, in knowing, not only that a set of licenses are interoperable, but that they all allow certain things, for example for anyone to use, reuse and redistribute the licensed material (or to put in terms of freedom, that they guarantee those freedoms to users). This very need for a standard is why we created the Open Definition for content and data building directly on the work on a similar standard (the Open Source Definition) in the Free/Open Source software community.

The existence of non-commercial

CC took a crucial decision in including NonCommercial licenses in their suite. Given the ‘Brand’ success of Creative Commons the inclusion of NC licenses has been to give them a status close to, if not identical, with the truly open, commons-supporting, licenses in the CC suite.

This is a noticeable difference here with the software world, where NC is also active, but under the ‘freeware’ and ’shareware’ names (these terms aren’t always used consistently), and with this material clearly distinguished from the Free/Open Source software community.

As the CC brand has grown, there is a desire by some individuals and institutions to use CC licenses simply because they are CC licenses (this is also encouraged by the baking in of CC licenses to many products and services). Faced with choosing a license, many people, and certainly many institutions, tend to go for the more restrictive option available (especially when the word commercial is in there — who wants to sanction exploitation for gain of their work by some third-party!). Thus, it is no surprise that non-commercial licenses appear to be by far the most popular.

Without the NC option, some of these people would have chosen one of the open CC licenses instead. Of course, some would not have licensed at all (or, at least not with a CC license), sticking with pure copyright or some other set of terms. Nevertheless, the benefit in gaining a clear dividing line, and in creating brand-pressure for a real commons, and real openness would have been substantial, and worth, in my opinion, the loss of the non-commercial option.

Structure and community

It is notable in the F/OSS community that most licenses, especially the most popular, are either not ‘owned’ by anyone (MIT/BSD) or are run by an organization with a strong community base (e.g. the Free Software Foundation). Creative Commons seem rather different. While there are public mailing lists ultimately decisions regarding the licenses, and about crucial features thereof such as compatibility with 3rd party licenses, remains with CC central based in San Francisco.

Originally, there was a fair amount of autonomy given to country projects but over time this autonomy has gradually been reduced (there are good reasons for this — such as a need for greater standardization across licenses). This has concrete affects for the terms in licenses.

For example, for v3.0 the Netherlands were requested to remove their provisions which included things like DB rights in their share-alike provision and instead standardize on a waiver for these additional rights (rights which are pretty important if you are doing data(base) licensing). Most crucially the CC licenses reserve the right to Creative Commons as an organization to determine compatibility decisions. This is arguably the single most important aspect of licensing, at least in respect of interoperability and the Commons.

Creative Commons and Data

From my first involvement in the ‘free/open’ area, I’d been interested in data licensing, both because of personal projects and requests from other people.

When first asked how to deal with this I’d recommended ‘modding’ a specific CC license (e.g. Attribution-Sharealike) to include provisions for data and data(bases). However, starting from 2006 there was a strong push from John Wilbanks, then at Science Commons but with the apparent backing of CC generally, against this practice as part of a general argument for ‘PD-only’ for data(bases) (with the associated implication that the existing CC licenses were content-only). While I respect John, I didn’t really agree with his arguments about PD-only and furthermore it was clear that there was a need in the community for open but non-PD licenses for data(bases).

In late 2007 I spoke with Jordan Hatcher and discovered about the work he and Charlotte Waelde were doing for Talis, to draft a new ‘open’ license for data(bases). I was delighted and started helping Jordan with these licenses — licenses that became the Open Data Commons PDDL and the ODbL. We sought input from CC during the drafting of these licenses, specifically the ODbL, but the primary response we had (from John Wilbanks and colleagues) was just “don’t do this”.

Once the ODbL was finalized we then contacted CC further about potential compatibility issues.

The initial response then was that, as CC did not recommend use of its licenses (other than CCZero) for data(bases), there should not be an issue since, as with CC licenses and software, there should be an ‘orthogonality’ of activity — CC licenses would license content, F/OSS licenses would license code, and data(base) licenses (such as the ODC ones) would license data. We pressed about this and had a phone con about this with Diane Peters and John Wilbanks in January 2010, with a follow-up email detailing the issues a bit later.

We’ve also explained on several occasions to senior members of CC central our desire to hear from CC on this issue and our willingness to look at ways to make any necessary amendments to ODC licenses (though obviously such changes would be conditional on full scrutiny by the Advisory Council and consultation with the community).

No response has been forthcoming. To this date, over a year later, we are yet to receive any response from CC despite having though we have now been promised a response at least 3 times (we’ve basically given up asking).

Further to this lack response, without any notice or discussion to ODC, CC recently put out a blog post in which they stated, in marked contrast to previous statements, that CC licenses were entirely suited to data. In many ways this is a welcome step (cf. my original efforts to use CC licenses for data above) but CC have made no statement about a) how they would seek to address data properly b) mention of the relationship of these efforts to existing work in Open Data Commons and especially re. the ODbL. One can only assume, at least in the latter case, that the omission was intentional.

All of this has led me, at least, to wonder what exactly CC’s aims are here. In particular, is CC genuinely concerned with interoperability (beyond a simple ‘everyone uses CC’) and the broader interests of the community who use and apply their licenses?

Conclusion

Creating a true commons for content and data is incredibly important (it’s one of the main things I work on day to day). Creative Commons have done amazing work in this area but as I outline above there is an important distinction between the (open) commons and CC licenses.

Many organisations, institutions, governments and individuals are currently making important decisions about licensing and legal tools – in relation to opening up everything from scientific information, to library catalogues to government data. CC could play an important role in the creation of an interoperable commons of open material. The open CC licenses (CC0, CC-BY and CC-BY-SA) are an important part of the legal toolbox which enables this commons.

I hope that CC will be willing to engage constructively with others in the ‘open’ community to promote licenses and standards which enable a true commons, particularly in relation to data where interoperability is especially crucial.

read original post on Rufus Pollock's Site

I’ve posted the slides online and iframed below.

Outline

Over the past few years, there has an explosive growth in open data
with significant uptake in government, research and elsewhere.

Bibliographic records are a key part of our shared cultural heritage.
They too should therefore be open, that is made available to the
public for access and re-use under an open license which permits use
and reuse without restriction (http://opendefinition.org/). Doing this
promises a variety of benefits.

First, it would allow libraries and other managers of bibliographic
data to share records more efficiently and improve quality more
rapidly through better, easier feedback. Second, through increased
innovation in bibliographic services and applications generating
benefits for the producers and users of bibliographic data and the
wider community.

This talk will cover the what, why and how of open bibliographica
data, drawing on direct recent experience such as the development of
the Open Biblio Principles and the work of the Bibliographica and JISC
OpenBib projects to make the 3 million records of the British
Library’s British National Bibliography (BNB) into linked open data.

With a growing number of Government agencies and public institutions
making data open, is it now time for the publishing and library
community to do likewise?

read original post on Rufus Pollock's Site

read original post on Rufus Pollock's Site

• Managing Public Sector Information – update: slides
• The NCVO (National Council for Voluntary Organizations) Annual Conference

read original post on Rufus Pollock's Site

Update: Slides

read original post on Rufus Pollock's Site

• Release: http://pypi.python.org/pypi/datapkg
• Docs: http://packages.python.org/datapkg/

There’s a quick getting started section below (also see the docs).

About the release

This release brings substantial improvements to the download functionality of datapkg including support for extending the download system via plugins. The full changelog below has more details and here’s an example of the new download system being used to download material selectively from the COFOG package on CKAN.

# download metadata and all resources from cofog package to current directory
# Resources to retrieve will be selected interactively
download ckan://cofog .

# download all resources
# Note need to quote *
download ckan://name path-on-disk "*"

# download only those resources that have format 'csv' (or 'CSV')
download ckan://name path-on-disk csv

For more details see the documentation of the download command:

datapkg help download

Get started fast

# 1. Install: (requires python and easy_install)
$ easy_install datapkg
# Or, if you don't like easy_install
$ pip install datapkg or even the raw source!

# 2. [optional] Take a look at the manual
$ datapkg man

# 3. Search for something
$ datapkg search ckan:// gold
gold-prices -- Gold Prices in London 1950-2008 (Monthly)

# 4. Get some data
# This will result in a csv file at /tmp/gold-prices/data
$ datapkg download ckan://gold-prices /tmp

Find out more » — including how to create, register and distribute your own ‘data packages’.

Changelog

• ResourceDownloader objects and plugin point (#964)
• Refactor PackageDownloader to use ResourceDownloader and support Resource
  filtering
• Retrieval options for package resourcs (#405). Support selection of
  resources to download (on command line or API) via glob style patterns or
  user interaction.

read original post on Rufus Pollock's Site