A Domesday Book for public service contracts – better data, better value for money

A fair amount of data about tendering exists, but information about public service contracts is sparse and disjointed.

No systematic monitoring of outsourcing takes place within the NHS, policing, or local government, and data flows within Whitehall are unreliable.

The lack of coherent data constrains decisions by departments, NHS trusts and councils because they simply don’t know enough about companies and contract performance.

Better information would not just improve contract letting, it would make public bodies more accountable, improve productivity and help scrutineers of spending.

This paper proposes a central depository or office to collect data – a Domesday Book with contract details, company information, their remuneration and employment policies and contract performance. This depository would actively disseminate this data to all public bodies for use prior to any ‘make or buy’ decisions.

This paper will set out three actions for this new function:

1. Improve data collection on outsourced contracts

The Cabinet Office and Treasury have begun making improvements, but progress must be faster. The Cabinet Office should be given a greater role in collecting data from Whitehall, local authorities, police and crime commissioners, and the NHS.

2. A ‘Domesday Book’ for all contracts

A new public body should be set up that operates at arm’s length from central government. It would have statutory powers to require both commissioners and contractors from across the public sector to supply it with data. And it should maintain a ‘Domesday Book’ of all contracts including their performance on the outsourcing of services.

3. Major reforms to improve value for taxpayers from outsourcing

The National Audit Office should become audit supervisor for the whole public sector. And a new Office for Equity, Efficiency and Effectiveness should be established to maximise social and public value This new Office of the 3Es and the NAO should have a clear mandate to assess value for money, efficiency, effectiveness and equity when services are outsourced.

The management model imposed on British public services in recent decades has fallen apart. According to the Institute for Government (IfG), “notions of how to pursue efficiency, the role of the private sector and competition, and a view of the public as consumers, not citizens – which have enjoyed a remarkable consensus through Labour, Coalition and Conservative governments – are now being questioned”. [1] Shockwaves from the bankruptcy of Carillion plc still reverberate across the sector, rocking established ideas about the audit and governance of companies and the inherent risks of externalising vital parts of public business to them. Northamptonshire County Council’s near collapse has further undermined the claim that outsourcing automatically brings budgetary savings.

Central, devolved and local bodies should be more aware of what it takes to deal with commercial suppliers without risking service failure. In-house and communal provision of services are being reappraised more positively, but in costing their options departments and councils face a yawning information gap regarding contracts and contractors. Government procurement data simply is not good enough because “Critical information that public bodies could use to make more informed spending decisions is often unavailable or incomplete. While government’s performance on collecting and publishing data in some areas has improved, in others it has gone backwards.”[2]

The UK government procures £195bn on goods and services in a year. [3] The NAO estimates half of this is spent on services, e.g. outsourcing and privately-run public services.[4] Private firms now train RAF pilots, cut the grass in royal parks, run IT systems, collect household refuse, staff front-desk receptions, and contract for community health and care for the elderly. Amazingly, the exact figure for public sector outsourcing is unknown and no breakdown of procurement spending has yet been able to value it. The data has not been collected, little or no research has been done, and commentary is often anecdotal or self-interested. This ignorance isn’t inconsequential: it inhibits fully informed service delivery decisions. Tenders, contracts and budgets can’t be reliably linked and accountability suffers as a result.

Early stage data does exist. A fair amount is known about tendering (though the process is still far from transparent), but much of that data is uncodified, segmented and incomplete. It is rarely connected to what is subsequently delivered, let alone the unintended and systemic effects of outsourcing. Data is not shared or disseminated between public bodies, meaning decisions about services (especially the balance between outsourcing and in-house provision) are data deficient. Data gets locked away inside contractor companies and much is sequestered in consultancies, which sell their knowledge at a premium and have a vested interest in promoting outsourcing.

The UK government is also not joined up, with the NHS, central government and local bodies operating in silos. Their commercial dealings are kept hidden and intelligence about markets and contractors is not shared. Knowledge is therefore asymmetric. Companies can compare and contrast the various councils and departments they deal with (Carillion had contracts with hundreds of them), but public bodies don’t share details about a contractor – certainly not Carillion. Nor do they evaluate contractor performance (or, if they do, it is not shared).

A great learning opportunity is going begging. The devolved governments of Scotland, Wales and Northern Ireland can do things differently – all would benefit (as would the Westminster government) if they shared their contracting experience. At a local level even neighbouring councils often hide what they know about a contractor from the authority next door. Of course local authorities have a degree of autonomy in deciding whether to outsource or not. But the effectiveness of their public services depends on better reporting on how contractors have performed in the past.

This paper sets out some remedial proposals, some of which could be put into effect quickly without legislation or new structures. For example, mechanisms already exist to collect and collate the missing data. Councilors, ministers, police commissioners and members of boards could insist on better reporting and share the results.

The benefits of outsourcing are being challenged. Better data – comprehensive, coherent, consistent and readable data – is needed to evaluate existing contracts, as well as to decide the future shape of services. William of Normandy famously determined that effective rule depended on knowing how the land lay, literally, after his conquest of Anglo-Saxon England. The vast terrain of public service contracting needs its own Domesday Book.

Procurement and contract management generate intelligence on suppliers and contractors. Maximising that knowledge would help to:

• improve decision-making, assessment of risk and make for fairer calculation of cost and advantage among different modes of service delivery
• make departments and councils better customers and contract-letters, improving their make-or-buy decisions
• assist scrutineers and auditors of spending secure better value for money
• understand supply markets and thereby assist regulators (including the Competition and Markets Authority, the Bank of England and the Department of Business, Energy and Industrial Strategy) and provide insight into UK productivity and corporate performance
• promote joined-up government
• make public bodies more transparent and accountable.

According to the Commons Public Administration Committee, the evidence behind decisions about services and whether to use the private sector was “thin or non-existent”.[1] Carillion’s implosion showed the scale of exposure to contractual risk. Clients were simply unaware of its financial liabilities across government or its commercial exposure in the UK and abroad. Nor did councils and departments log their dealings with the firm. The Cabinet Office started registering their vulnerability after Carillion gave cause for concern, but even then did not grasp the extent of the company’s reach into local government, the NHS, the Highways Agency and other bodies. The Cabinet Office even had to ask Carillion itself for details. [2] Without the necessary data, there was no way of assessing how dependent public bodies were on this one company – or to anticipate the costs of its demise. Research by Unite suggests this could eventually total more than £150m, including redundancy of £65m and lawyers’ and liquidation fees of £70m. [3]

Similarly, in residential care, it was only after Southern Cross collapsed that its market dominance was disclosed, along with the numbers of vulnerable people who were put in jeopardy as local authorities faced major operational and financial difficulties. Councils did not register their dealings with the firm: neither the Department of Health, the Local Government Association (LGA) nor the Care Quality Commission (CQC) maintained a database.

Probation, primary care support and refugee housing show other recent examples of contract failure. Each has specific features, but their commonalities are characterised by “the failure to deliver on promised savings and revenue, underperformance against contract requirements and ineffective risk transfer”. [4] Repetitions only become visible when contract data is widely available – and visible across all sectors.

We do know more about aspects of the commitments entered into by the state. A process known as Whole of Government Accounting, inaugurated under the Brown government, and the Office of Budget Responsibility, created by the Cameron coalition, were important reforms. They show multi-year trends in spending and revenue and allocations to departments but at high level. Poor data make it hard, maybe impossible, to assess the marketisation of services in recent years.[1] At a greater level of detail, public bodies are now required to publish details of their external spending and list procurements on their websites. But the picture is far from clear. After the tendering stage, contracts do not have to be put into common reporting templates. As a result, what public bodies know is not compared and contrasted with what neighbours or other sectors may have learnt. MPs complain that it is “almost impossible” to find out what is being spent on particular services, [2] including contracted services.

Since the early 1980s, the government has massively expanded the purchase of services from private companies. We know this from the fortunes enjoyed by the companies themselves – from their stock exchange performance and their executives’ pay. These new ‘markets’ in contracting are dominated by a handful of firms, including Capita and Serco – companies specially created to exploit burgeoning profit opportunities. Yet the phenomenon has been little studied. There is no taxonomy of service contracts; it is often hard to tell where the purchase of goods (desks, energy, vehicles, laptops) merges into outsourcing (service contracts).

The Treasury exercises strong control over revenue and budget aggregates. It knows how much is spent in total but pays surprisingly little attention to the how of spending – to performance and efficiency. The UK government falters when it comes to gauging whether policies make a difference and how spending decisions affect households and communities, or on whether councils, trusts and departments are efficient, effective and equitable. Because of these gaps, accountability suffers: parliament and citizens lack the tools to hold decision-makers responsible.

Yet the Treasury’s own spending manual says that spending decisions, including outsourcing, should be conceptualised and measured in the round. [3] The ‘extra-contractual’ effects of letting a contract should be counted in, it says. A council should not let a contract with the result that staff are made redundant without factoring in the net cost to the public purse of any resulting spike in benefits payments to jobless workers – and any effects on their health and wellbeing, which may not push spending up immediately but may have longer-run consequences. A council may not bear these unintended contract costs itself but the Department of Work and Pension’s budget will. A risk here is ‘cost shunting’, where savings in one programme produce costs elsewhere. Even if the Department for Health and Social Care knew about NHS contracting (and it’s unclear whether it knows much at all), it does not necessarily share information either with the Treasury and Cabinet Office or other departments. The same happens with the Department of Housing, Communities and Local Government (which has the smallest commercial unit of all Whitehall departments) and council contracting.

In June 2018 David Lidington, the Cabinet Office minister, said the ‘social value’ of contracts should be counted in (Gov.uk 2018). Social value entailed, for example, challenging contractors on their equality and diversity record. That surely means publishing data on areas such as ethnic minority staffing or the gender pay gap. To avoid, as the minister put it, “too narrow a focus on value for money” entails knowing much more about contractors – who they are, the extent of their public sector involvement, their performance as well as specific information about their workforce. To the minister’s list could be added trade union recognition, location of headquarters and regional presence and other indicators of social value.

Tendering is widely reported. Councils and departments publish details of what they pay external suppliers. But the columns of data are not linked together, nor is it prepared in standardised formats. A critical next phase – how contracts are carried out – is therefore missed. Either the data is not collected at all or not published.

As of now the initiation of contracts is visible in the Official Journal of the European Union (OJEU) Tenders Electronic Daily . Cost thresholds are £118,133 for UK central government, £181,302 for other contracting bodies and £4,551,413 for public works contracts. Other contracts may still be visible on Contracts Finder . On this UK government website, Whitehall departments and agencies are required to advertise deals worth over £10,000. Other bodies are supposed to advertise contracts worth more than £25,000, but the picture is incomplete. The Institute for Government estimates that in 2018 only two in five tenders were published on Contracts Finder. [1] Leases are exempt from notification. Separate notification schemes apply in defence and in the devolved administrations. After April 2019, when the UK is to leave the EU, these arrangements will supposedly remain in place. The government says that with no agreement with the EU the fallback would be a UK specific e-notification service into which all contracts now published on OJEU/TED would go.

As well as these official sites, tender data is available – for a fee – from private sector sources. Tenders Direct , a division of the company Proactis, scans OJEU and other sources. Such data is primarily of use to would-be contractors. Reciprocal data about them – relevant to the contract decision – is much more dispersed or locked away, either inside companies themselves or conserved by consultants and auditors. Post- tender, data disappears into a black hole. Some private suppliers offer information, culled from annual reports and market intelligence, such as Tussell. Its clients can buy market analysis at a certain level of aggregation – for example, the number of public sector contracts held by a firm.

No systematic monitoring takes place inside the NHS, either of contracts between NHS trusts or let by trusts to private and third sector suppliers. Procurement has itself been outsourced, which has complicated and perhaps compromised the flow and quality of data. Regulators demand lots of information from trusts and could in theory find out how contractors perform. Yet in a recent appraisal, the King’s Fund did not identify any oversight of contracting by a major regulator, the Care Quality Commission. [1] NHS improvement (NHSI) publishes ad hoc studies, which incorporate some data on contracting (for example, Lord Carter’s analyses of procurement). The boards of Foundation Trusts may see local data on contracts but have no access to systemic data from within the NHS or from other local bodies.

In other domains, it is a similar, fragmented picture. HM Inspectorate of Constabulary and Fire and Rescue Services, the criminal justice inspectorates and Ofsted are all concerned with performance and so, implicitly, with the work done by contractors. But none focus on or instigate the collection of data about their performance.

External auditors of public bodies possess knowledge about contracting practice and performance but they report it only intermittently, if at all. Outside central government, audit is now carried out by private firms. Most carry a responsibility to report on value for money or ‘use of resources’ but not, it seems, contracting. Local authorities publish raw contracts data (but schools and others bodies don’t). Raw is the word: it is not easy to make sense of the undifferentiated list of items on the spreadsheets made available by a council.  

The solution is self-evident. It is what we call a Domesday Book: the central collection of data on outsourcing. A repository or registry could collect and disseminate knowledge, compiling and updating regularly. Into it would go:

+ all new contracts above the EU value threshold, to be notified after a start date

+ then, over a two-year period, details of all existing outsourcing contracts

+ all PFI contracts (which often involve services and outsourcing contractors)

+ contracts with voluntary and community groups, charities and social enterprises.

To secure this data, a new duty of contract reporting could be laid on public bodies in England. Welsh, Scottish and Northern Irish governments would be invited to join in and share data with the registry, and would in turn benefit from access and analysis. This new duty would consolidate work that is going on. A reciprocal duty of reporting would be laid on contractors – doing work above the value threshold. The duty could take the form of a contract clause, requiring the company to lodge a copy of the agreement with the central registry.

A central registry

The database would eventually list:

• the names of suppliers of services above the OJEU threshold. The database would feature the names of the contractor, its ownership details, its major shareholders if a quoted company, and the names of partners if a consortium. These details would be cross-tabulated with contracts above the threshold, the identity of principal sub-contractors and the contract terms.

Corporate data would include:

• the value and percentage of corporate turnover formed by public sector contracts
• the gender pay gap, trade union recognition, the location of corporate headquarters, the geographical distribution of staff and proportion of turnover in the UK and contractors’ staff diversity
• company tax policy, Fair Tax Mark qualification and the estimates of UK tax paid
• remuneration policies and practice.

The database would archive and analyse:

• copies of the contract and any significant renegotiated terms; the length, service specification, value and annual contractual payments, and the contract default and cancellation terms.

It would establish:

• a means of identifying suppliers, for example by their company or charity numbers.[1]

Through discussion with councils, trusts and departments, the registry would be able to assemble and appraise:

• key objectives set by them in letting a contract.

The registry could compare and contrast contracts, examining contractors and processes and promoting discussion. It might consider establishing a What Works centre on outsourcing as part of the existing network of What Works centres under the Cabinet Office. Its staff would include specialists and people with cross-sector experience who knew the tricks of the trade. Its database would include not only contracts but also material on people’s attitudes towards services and on how they are delivered.

The registry would establish lines of communication to audit, risk, scrutiny and oversight committees to deliver updates on contracting trends and alerts about contractors, ensuring that its material was in an accessible and intelligible format to citizens at large. The government could use its leverage over the big management consultancies to try to unlock the knowledge about contractors and contracting they have hidden away.

Under new protocols:

+ public bodies and contractors would ensure data was kept up to date

+ data should be verifiable and open to the maximum degree, with commercial confidentiality tightly defined and exceptional

+ every body would be expected to have a named lead official responsible for the database and for enquiries from other public bodies

+ Domesday Book data should be accessible via an internet portal, after agreements on access, security, applicability and Freedom of Information

+ data would be subject to external audit on a managed basis.

Over time further knowledge might be sought about the governance and employment standards of contracting companies across their entire business. The registry would analyse trends and markets, presenting data by sector and region to compile an accurate figure for the value of outsourcing. The aim is also practical, usable knowledge. For example, in a Whitehall department registry data would be available to ministers and permanent secretaries as a central part of the evidence on which policy and delivery should be based. This data should be monitored by parliamentary select committees and reported in annual and accountability statements. Similarly, in local government councillors and executives should seek assurance that evidence and data from the Domesday Book are used in policy and operational recommendations. Audit and scrutiny committees should check they do so.

Those are the objectives for reform. Here follows three actions to secure them.

Action 1

Improve data collection on outsourced contracts

The government recently assured MPs that it was on the case. [1] It is developing a contracts ‘playbook’, bringing together multiple sources of ‘actionable information’. The Cabinet Office, which houses the Infrastructure and Projects Authority (IPA) and Crown Commercial Service (CCS), can access contracts let by Whitehall departments and arm’s length bodies. And the Treasury recently established a unit to look at ‘end-to-end’ costs of decisions, which should encompass how services are delivered. This is welcome, if overdue, and it’s also doubtful that the Cabinet Office will seek information from outside Whitehall, from the NHS and local government.

This work could be accelerated and expanded by extending the Cabinet Office mobilisation over Carillion into a structured capacity to bring the CCS and IPA closer together and pool data. Other ‘joiners’ should include the government finance function, whose head, Mike Driver, recently claimed was doing more horizon scanning and becoming more proactive. [2] Intra-Whitehall collaboration could tap the data held by the Competition and Markets Authority, the Bank of England and Department of Business, Enterprise and Industrial Strategy, all of which monitor sectors, companies and markets. The IfG recommends the Cabinet Office collaborate with Companies House in requiring accounts to be submitted electronically and in machine-readable forms.[3]

Regulators could be pushed to do more. NHS Improvement and NHS England could acquire contract data through existing reporting requirements on trusts and commissioners. The then health minister Steve Barclay recently said “a modern health service shouldn’t involve 234 separate trusts spending time and money negotiating different contracts and prices for the same thing. That’s why our work to centralise how the NHS buys goods and services is crucial”.[4] A precondition of cost-effective contracting is the centralisation of data: how many firms, what prices, with what sort of record. After another contracting failure – this time over the disposal of clinical waste – the Cabinet Office, Environment Agency and Department for Health and Social Care said they were working together. Again, sharing data about contractors and contractors is vital.[5]

The CQC and inspectorates could insist on contracting data as part of their supervision of service quality.

The Cabinet Office is the UK point of contact for the devolved administrations and could work with them on a common template for reporting on contracts and contract performance. Much of the above is doable with existing staff numbers and functions. It would require more intense collaboration, always an issue in Whitehall (especially between the historical rivals in the Cabinet Office and Treasury). The attitude of the CBI to contracting information appears to have been changing. It could be enlisted to encourage companies voluntarily to share contracts data with the government.

But this voluntary (non-statutory) option is deficient. The Cabinet Office has traditionally been weak, relative to other Whitehall departments, especially the Treasury. Does it have the requisite ‘push’? As observed above, this option misses out local government and the NHS. Councils value their autonomy, but that has meant in practice they do not share even basic information with each other, let alone other levels of government. Historically the Cabinet Office has focused on Whitehall. Its relationship with councils and the NHS (in England) has been mediated through the departments sponsoring local government and health, which have not always been collaborative.

Action 2

A ‘Domesday Book’ for all contracts

A statutory body should be given power to wrest the information needed. If this were at arm’s length from central government, local authorities might be more willing to collaborate. They might nominate representatives to its governing board, along with representatives from the NHS, police and crime commissioners, arm’s length bodies and possibly the devolved administrations.

The size and cost of this new body are up for discussion. It could be funded through a levy on contracts, which might add marginally to the cost of contracting. It would be subject to Freedom of Information to encourage more contracts to be public documents, dispelling the suspicion that has grown up around contracting letting. The body might also absorb the Crown Commercial Service, making its expertise available to public bodies outside Whitehall.

It would not dictate to council or NHS trusts how to organise services and whether to outsource or not (and if so with which provider). But it would be proactive, trying to persuade public bodies to utilise its information about firms, markets and previous contract performance. Its target audience would be auditors, scrutiny committees and interest groups, as well as elected decision-makers and boards. These in turn would have to collect and curate data that is not presently visible, report during the life of contracts and publish assessments at their end.

It should maintain a ‘Domesday Book’ of all contracts on the outsourcing of services. This Domesday Book would record all contracts above the EU procurement threshold, details of contractors, contact values and performance data. The agency would use this date to issue analysis to the public sector on outsourcing, its impact and contractors’s performance and track record.

Action 3

Radical institutional reform

This option calls for a comprehensive restructuring of how public bodies achieve value for money, including in their use of contractors. The machinery concerned with securing value for money in public services would be completely remodelled following a re-examination of existing arrangements for public audit, scrutiny and accountability, especially those that function at arm’s length from democratically elected ministers and councillors. The Accountability System Statements inaugurated under the Cameron coalition has outlined areas of concern at academy schools and local enterprise partnerships. NAO reports have also found multiple instances of inadequate supervision of public money. [6]

After the demise of the Audit Commission, arrangements for supervising councils have been criticised, especially the silence of auditors as austerity has squeezed local budgets and capacity. Multiple, overlapping regulators are held responsible for equity and efficiency in the NHS, while the NAO’s remit has not been comprehensively reviewed since its establishment. The NAO has done vital work in adding to available data on outsourcing, but it has also been patchy. And only some of its reports are followed up by the PAC, which has a mixed record in securing its recommendations. Under this option, comprehensive questions would need to be asked about parliamentary scrutiny, probably going well beyond the inquiry recently undertaken by the PACAC.

A programme of radical reform might consider separating public audit from the pursuit of value for money. The NAO might become the audit supervisor for the whole public sector (in England) while a new agency – an Office for Equity, Efficiency and Effectiveness, perhaps – is charged with rationalising and streamlining the work of sector regulators in maximising the social and public value of spending. Such an office would of course be deeply interested in how public services are delivered and it would have specific mandate to assess the efficiency, effectiveness and equity of outsourcing, procurement and contract performance across the public sector.

The proposals above flow from an existing stream. In his 2017 report to the Treasury, the former head of Tony Blair’s Delivery Unit Sir Michael Barber prescribed “a tool for measuring, tracking and delivering [productivity] improvements while permanently embedding a new dialogue around expenditure”.[1] His “public value framework” demands the knowledge that this report is calling for: that is, data on the costs and benefits of contracting and the relationship, if any, between outsourcing and achieving goals of efficiency, effectiveness and equity.

The government now seems to accept that contracting can only be evaluated by reference to what used to be called ‘extra contractual’ parameters. It has signalled its assent to the reappraisal of the ‘big four’ consultancies, which have played a large part in promoting outsourcing. [2] The big four are also public auditors, which is another reason why reform is so overdue.

Audit is important. Better-informed auditors would be in a position to query decisions around contracting and closely examine risk assessments. Channels to be exploited include external auditor reports, professional networks and messages from regulators. The pursuit of fair value for money should become more coherent and expanded to include residents, citizens and service users. Critically, this depends on better data on outsourcing.

This proposal to compile and maintain a new registry of public service contracts is not about creating new powers to direct any organisation. It will not challenge the rights of any local authority or NHS trust to make their own decisions about how best to deliver a service. Such decisions should and would remain decentralised. But a Domesday Book for public service contracts will help to inform these organisations by providing contextual knowledge about contract performance across the public sector and, critically, about the contractors themselves.