The UK is ranked number one in the world for its open government data and is now driving hard to reap the benefits for public sector transformation.

Public Sector Leadership and Convergence

In the UK government, similarly to New Zealand, a range of agencies have lead roles that are converging in the data ecosystem. This creates both collaboration and creative tensions.

A Government Data Partnership (GDP) has been established within the public sector as a coordinating mechanism for lead agencies, whose responsibilities include:

  • Cabinet Office: setting policy on open data and ethics
  • Government Digital Service (GDS): driving technology and use
  • Government Office of Science: promoting data use to enable government policy aims
  • UK Statistical Authority and Office of National Statistics: collecting data, publishing data and statistics, governing the broader statistics system
  • Heads of Professions ? the UK has a wide number of these roles including heads of operational research, of analysis, of statistics, of science and of engineering.


In my last blog, I shared some reflections on my findings at the Cabinet Office and GDS. In this one, I'll write about changes underway at my sister agency the Office of National Statistics.

Unleashing the power of data to enable better decisions ? UKSA/ONS

Like Statistics New Zealand, the UK Statistics Authority (UKSA) and its operational arm the Office of National Statistics (ONS) are well placed to take a key role in data stewardship for the public sector due both to their extensive data holdings and to their data management and data analytics expertise. The data published regularly by the ONS makes up a large portion of the country's open government data. As one person in another agency told me "they employ all the data scientists ? we need them!".

That said, the statutory independence of government watchdogs has led national statistical offices to hold themselves at arm's length from other parts of government and be seen as conservative. Beth Noveck speaking at the International Open Data Conference in Ottawa last month observed that there is a major divide between the statistics community and the open data community internationally, with statistical agencies focused on the ‘old way of doing things’ rather than co-creating solutions with citizens, businesses and governments.

To lead effectively in the broader data ecosystem, statistical agencies need to make a strategic and culture shift to embrace the ‘data revolution’ (as recommended last year by the UN). The UKSA and ONS have made major steps in this direction over the past year. John Pullinger was appointed CE and National Statistician in July 2014. He has set a new strategic direction, which makes ‘helpful’ and ‘innovative’ key perspectives for ONS to apply to its objectives (alongside ‘capable’, ‘professional’ and ‘efficient’). He has also turned the ONS outwards the introduction of a new set of leadership roles including four new deputies each with external responsibilities.

Technology has been brought into the centre of ONS with the appointment of Heather Savory, previous chair of the Open Data Users Group, as DCE responsible for Data Capability. A new Chief Data Officer Bill Oates reports to Heather. Bill's role will have a strong external focus ? responsible for driving the analytics profession to get data science on the front foot across government, as well as driving internal cultural change so that ONS can deliver on its external vision.

Decision making is being made more effective. The traditional service approach was risk averse, bureaucratic, overused project management methodologies, and nonetheless most tech projects failed. Now the aim is ‘first-response decision making’, empowering individual decision making rather than decision making by committee.

Rob Bumpstead, John Pullinger, Evelyn, Bill Oates, Heather Savory

Chief of Staff Rob Bumpstead, CE and National Statistician John Pullinger, Evelyn, Chief Data Officer Bill Oates, and Director General of Data Capability ONS Strategy Heather Savory

User-centred service design

The ONS is also charging ahead with more customer-centric, more innovative delivery of its data services. I met with Matt Jukes and his team at the ONS office in Newport (South Wales) to hear about this work. Matt is world famous at Statistics NZ thanks to his blog on the changes underway at in ONS digital data publishing.

As in many cases, a crisis created the burning platform for this innovation. Last January, the ONS website crashed badly 11 hours before the critical GDP release. Although staff were able to resurrect it just in time for the release, this caused shock waves across the organisation and empowered the relevant manager to create a new website and digital data publishing services. Matt was brought in to lead this work, bringing wide experience in running digital innovation teams.

The development is following the approach in the GDS Government Service Design Manual, moving through four steps:

  1. discovery (researching users' needs)
  2. alpha (rapid prototyping)
  3. beta (build and scale, put out for public)
  4. live (into production, following an intensive GDS review to check for people centred design).

The discovery phase engaged with genuinely publicly critical people ? including those who had written pieces criticising ONS services in major papers. They turned out to be normal, nice people who were very supportive when ONS team went to talk with them. The team also engaged with 60+ organisations that were reliant on ONS data ? meeting with them, asking for early feedback, and running the development openly with people invited to criticise as the services developed. This worked well, revealing that a lot of things that ONS has perceived as problems were not problems for users, while some simple but lower priority things were critical.

Matt's team aims to work in lean start-up mode ? with a high focus on agile and on usability. They are undertaking iterative development, bringing in users early and adjusting development to fit their needs as they feed back. The motto is ‘Done is better than perfect’.

ONS Strategy

ONE strategy

Evelyn at ONS digital publishing design room

ONS digital publishing design room

ONS digital publishing design

ONS digital publishing design

ONS digital publishing design

ONS digital publishing design

To share or not to share

One challenge faced by the ONS is the lack of sharing of core datasets between UK government departments. This is an area where we have been able to make much more progress in New Zealand thanks to the Analysis for Outcomes Cabinet decision and the operation of the Integrated Data Infrastructure (IDI) under enabling provisions in the Statistics Act 1975 and Privacy Act 1993.

I was told often that IDI wouldn't be possible in UK ? as it would be seen as ‘big brother’, and privacy considerations would mean this information was locked within individual departments. Privacy was a high concern for the government officials I met with on my fellowship. It is clearly a high profile issue, with a strong lobby and frequent page one stories generating public concern. Although I was also reminded that there is a mixed relationship with privacy: UK agencies cannot share data for policy purposes, yet the UK has the world's highest concentration of CCTV cameras.

The Cabinet Office and UKSA have been working to develop enabling legislation for data sharing for statistical purposes, as the Statistics Act 2008 requires any data sharing proposal to go through parliament for an order to have ‘affirmative resolution’. It is difficult to get parliamentary time for such orders. Moreover, the requirement that if any part of an order falls, the whole order falls, breeds a culture of conservatism in the drafting of orders. This means that the ONS cannot get access to data to do feasibility work let alone to enable statistical production or policy research.

Some progress in data sharing has been made with the establishment of the Administrative Data Research Network (ADRN). This became operational in November 2014, with four research centred ? one in a university in each constituent country of UK ? and a coordinating service at the University of Essex, which hosts the UK Data Service. The UKSA provides independent governance for the ADRN, and supports quarterly board meetings. The aim of centres is to link person level information, and provide safe access in anonymised form for research use. Researchers have to provide the public good value of their research project, and an Approvals Panel judges each project to check that they meet criteria. Data is linked for specific projects and destroyed when the research use is complete. The ADRN is funded by government through the social and economic research council.

In contrast, the Government Digital Service (GDS) is using open government data published online by agencies to enable develop of public service registers and user-centric apps that bypass the problems of lack of joined up systems and data. The data landscape in government is fragmented, with a lot of copying across data, and a lot of lists that are not shared. GDS wants to fix this and enable government to reuse its own data, not just in the context of one agency but across full government starting with open data, then looking at more sensitive data. This was the aim of the Registers Hackday that I sat in on. It's also an example of the use of an agile, design centred approach to data and tech ? why invest in expensive, slow, monolithic systems when you can use APIs and fast prototyping to develop the tools the public need?

newspaper headline