UK Gov't Spending Details Now Online

krou writes "The UK government has released a treasure trove of public spending data in an effort to lift what Prime Minister David Cameron calls the government's 'cloak of secrecy.' 'The first two tranches of data are from 2008/09 and 2009/10. The Combined Online Information System (Coins) includes what departments were authorised to spend, what they actually spent and what they are forecast to spend in future.' Since the government admits that 'some degree of technical competence' will be needed to use the files, they have asked the Open Knowledge Foundation to help make it 'more accessible,' and have also promised 'more accessible formats' by August. The datasets can be downloaded from data.gov.uk." And on a similarly happy note, reader mccalli writes "Bletchley Park's archive is to be digitised and put online. It seems HP made an offer to help out with scanners and expertise, and the result is that these texts will be made available to all."

Comments

Advanced

[jez9999]

"Bletchley Park's archive is to be digitised and put online. It seems HP made an offer to help out with scanners and expertise, and the result is that these texts will be made available to all."

Wow, they texted at Bletchley Park? They were truly ahead of their time!

The Guardian have a web app already...

[levell]

The Guardian newspaper have already built an interesting tool for exploring the data.

Re:The Guardian have a web app already...

[d_54321]

Would be a little better if, instead of alphabetical order, they put months in chronological order.

nice to see Torrent links

[ikoleverhate]

if Her Majesty's Government uses torrents, they must be OK, right?

Re:nice to see Torrent links

[Wowsers]

What they should have done is rename the file something like "Britney Spears entire album collection", then waste the RIAA's bandwidth on them downloading it, and lots of their time decompressing it and checking the files out to see if it contains WAV / FLAC / AAC / MP3 files.

Re:nice to see Torrent links

[Anonymous Coward]

Just use an obscure music format. Like ogg

Re:nice to see Torrent links

[Anonymous Coward]

What they should have done is rename the file something like "Britney Spears entire album collection", then waste the RIAA's bandwidth on them downloading it, and lots of their time decompressing it and checking the files out to see if it contains WAV / FLAC / AAC / MP3 files.

But first you have to register at some dubious website to get the password for the RAR file. Then you find out the files are empty.

Splendid

[Anonymous Coward]

Just a few days ago I watched David Cameron give a TED talk about the need for government transparency and accountability. He emphasized the importance of the Web as a tool for achieving those ends. Looks like the Tories are actually going to put this into practice.

Re:Splendid

[Wowsers]

When the new coalition speaks of all things being open, I don't think the people were thinking about their private details being among the information being "open".....

Private medical details open for all to read via the NHS copmputer system

A controversial database of personal health records will continue to be rolled out despite Government pledges on civil liberties. Campaigners accused the coalition of a U-turn over a system they say people are bullied into signing up to and is too hard to opt out of. The Tories insist they never made a commitment to end it. Director of Big Brother Watch, said patients should be told the truth about its drawbacks and, like ID cards, it should be scrapped. This is a disgraceful U-turn by the coalition. The Government wants us to believe that they are serious about privacy and civil liberties - this is their first test and they have failed it.

Private tax information being used illegally by the public sector workforce.

More and more town hall bureaucrats have been caught snooping on private details held on a giant 'Big Brother' tax and benefits database. Instances of unlawful hacking of the Customer Information System, which belongs to the Department of Work and Pensions and holds the personal records of 85 million people, have increased sixfold in a single year to more than two a week. Council staff have looked at accounts belonging to their friends, family members, neighbours and even celebrities.

Quick pop quiz on the second article. There's 65 million people in the UK, so why are there 85 million tax records?

Re:Splendid

[jabithew]

It's the Daily Mail. They probably just made it up.

Re:Splendid

[PolygamousRanchKid]

Quick pop quiz on the second article. There's 65 million people in the UK, so why are there 85 million tax records?

How about 20 million non-person entities, like corporations, churches, your local bowling club, the fox hunter club . . . oh, I guess that one is out.

Re:Splendid

[Anonymous Coward]

Most of those 20 million are deceased. Their records are held because of their relationship to a living citizen, which is needed to correctly assess eligibility for certain benefits.

However, there are also some duplicates - people who have for some reason assigned multiple tax records under different National Insurance numbers. There are legitimate reasons for this (for example, foreigners working in the UK are first assigned a temporary record, and later created a permanent record - with these two records then being linked together). However, there are also instances where this is due to fraudulent reasons. One of the key benefits for CIS (the system in question) is fraud detection: before CIS, the same data was scattered across 26 different systems, which naturally makes fraud detection very difficult.

(Disclosure: I was one of the architects for this system)

Re:Splendid

[Anonymous Coward]

HURRAH FOR THE BLACKSHIRTS

Re:Splendid

[timeOday]

Privacy in health care is not an easy issue. Yes, personal medical information needs to be private. But at the same time, for medicine to advance as a science and to become more cost-effective, requires data on what is being done, why, and what is being charged. Part of the reason US health care is so expensive is because it's a "free market" with no transparency, so everybody is flying in the dark.

Re:Splendid

[Anonymous Coward]

(Disclosure: I was one of the architects for this system)

Good thing you ticked AC then: most of the DWP folk I know are far from satisfied with CIS.

Re:Splendid

[DrXym]

The Daily Mail usually doesn't make stuff up. Instead it is just extremely selective about what it reports and toploads its articles with a heavily distorted version of events burying some more moderate thinking / mitigating reason right at the end. Usually stories are selected to push the outrage button on their middle England, closet facist readers. Popular topics might be immigrants, councils, "nanny state" rules, speed cameras, alternate lifestyles etc.

For example, this story is crafted to push buttons on how some barmy council is wasting taxpayers money mandating the number of holes in salt shakers while the entirely reasonable rationale for a pilot study is buried right at the bottom.

Re:Splendid

[drsmithy]

Quick pop quiz on the second article. There's 65 million people in the UK, so why are there 85 million tax records?

Corporations, duplicates, the deceased, non-resident taxpayers...

Quite frankly I'm surprised the difference is _only_ 20 million, for a country that's one of the world's economic powers.

Re:Splendid

[Anonymous Coward]

More like one of the world's economic basket cases at the moment.

Re:Splendid

[Anonymous Coward]

Interesting.

Given that CIS is a backend system - essentially an Oracle database with a set of web service and batch interfaces to the data - with no direct user access, I'm not sure how an end user would be able to have an informed opinion - good or bad - about the quality of the system itself.

They may have a view on the quality of the data held in the database and this was definitely a problem back when I was involved: the usual "garbage in, garbage out" applies. However, in my view this tells more about the processes and data quality governance applied at DWP than about the architecture of the IT system holding the data: there's only so much you can do to fix the data at the backend once it has been entered incorrectly at the frontend.

Anyway - my involvement with CIS ceased years ago - but I do wish them well. At least back then, it was definitely one of the better government IT projects that I have experienced (e.g. it had a sensible business case and was being delivered on time and on budget).

UK data release vs US annual reports?

[AHuxley]

How does the UK spending report shape up vs a US Comprehensive Annual Financial Report (CAFR)?
http://en.wikipedia.org/wiki/Comprehensive_annual_financial_report
That would show an accurate picture of the UKs institutional funds, financial holdings, assets and total investment incomes, for the government.

Re:UK data release vs US annual reports?

[jabithew]

This is raw data, not a spending report. We also have spending reviews; before the general election they were on the HM Treasury Site but now they've been archived. Without the legal requirements for clarity associated with private sector financial reporting, civil servants are able to hide key data in impenetrable waffle. It is also a rather different kettle of fish to the US; our government is ludicrously centralised and almost all spending is from Whitehall. The report is thus so broad in scope as to almost be meaningless. It also makes wading through this raw transaction data much more daunting.

Re:UK data release vs US annual reports?

[Burnhard]

In the UK it's called The Red Book and has been in the public domain for many years. COIN isn't the same thing - this is a full (mostly) breakdown of public spending. In the US you even get to see the Presidents tax return, so we've still got some way to go before we're on a par with the US. It's a good start.

Re:UK data release vs US annual reports?

[DaveGod]

I gather the forthcoming Whole of Government Accounts will be broadly similar. Most importantly, they will be using International Financial Reporting Standards (IFRS) and hopefully understandable if you're comfortable enough with the consolidated accounts of a major UK company (and yes even the US is crawling towards adopting IFRS). I'm not sure what the score is here, I thought these were meant to be done for 2006/07 but apparently 2009/10 will be the first set, I think they spent a few years just moving in this direction.

Publication of this Coins data is something different but is a major disclosure. It's almost certainly not going to be of any direct use to the taxpaying public, but for journalists and anybody wanting details on something specific then wow.

I am a little hesitant however. Not many people understand financial information and even fewer are able of putting it in the context of an organisation so massive as the UK government, and UK newspapers and other interested bodies are well versed in manipulating ignorance to their own ends. Even with the best of intentions, the volume of detail here is such that not even a team of researchers has much chance of putting it into context.

Files available in bittorrent

[Anonymous Coward]

OMG Pirates!!!

Torrrents.

[caluml]

I'm just really impressed that the .gov.uk is using torrents. I'd have expected you'd have had to apply, and it would have been posted to you on a stack of DVDs.

Re:Torrrents.

[mvar]

Thats impressive indeed. Now lets just hope the RIAA doesn't sue them

Re:Torrrents.

[TheRaven64]

No, let's hope the RIAA does. It might make them less keen to pass RIAA (or BPI)-friendly legislation in the future...

Re:Torrrents.

[RDW]

'I'd have expected you'd have had to apply, and it would have been posted to you on a stack of DVDs.'

They tried that previously, but ran into some minor difficulties:

http://www.independent.co.uk/news/uk/politics/lost-in-the-post-the-personal-details-of-25-million-people-758867.html

It is, of course, a complete coincidence that this data, which naturally only covers the expenditure of the previous government, is now being released by the new government just as it starts to slash public spending by several billion pounds. Somehow I suspect the 'cloak of secrecy' hasn't been sent to Oxfam, but is neatly folded in a cupboard at 10 Downing St ready for future use.

Re:Torrrents.

[Anonymous Coward]

zipped table ... 67MiB (4.28GiB uncompressed)

What the hell? How does that work?

I have just started a research project on this

[MarcoF]

It is a contract job for an Italian University, funded with EU money. I am looking for real world examples of the availability of public data like these have actually been good for local businesses, making them sustainable or cheaper to run. Every feedback is welcome. Here are more details on the project: Open Data, Open Society: a research project about openness of public data in EU local administrations. Again, thanks to all who will provide relevant pointers & info!

Re:I have just started a research project on this

[Anonymous Coward]

It is a contract job for an Italian University, funded with EU money.

So that's where MY money goes to!

The downside of all this

[Anonymous Coward]

I am all for transparency but it will also have a negative effect on the efficiency of the government

I work for a non elected government body and there is already that much red tape that it takes two weeks to get a purchase order for items over less than £1000 (having spent up to half a day doing the purchase paperwork and justification).

If we have to document it and everything it adds another layer of junior civil servants who's job it will be to document all the spending for public consumption. This at a time when my budget has been cut by over 1/3 (£200K). Do you really need to know I spent £2K on 64GB Ram for a blade ? Will it enrich your life. We are always being asked to do more with less, but then we look at all these mad hat ideas that drain cash. Cash that could be used to provide better services to you the public.

Most people don't realise but there is an extra ordinary amount of oversight even now. Even our small government dept has 3 levels of external auditing.

To purchase expensive kit (Like i do regularly -ie 40K at a time it can take several months to get it all sorted.

If they want to interfere in every minor purchase and put it online it will cause almost paralysis in most government organisations.

Just my 2ps worth. Posting as AC for obvious reasons.

Re:The downside of all this

[Anonymous Coward]

If you are spending public money then you have a duty of account for it. If you think you are entitled to spend £2000 on memory without having the relevant paperwork, then please leave public service now.

Re:The downside of all this

[Joce640k]

Presumably that purchase order goes into some sort of computer system even now, the amount spent and a code for type of purchase.

Once it's in there the rest can be automated, no need for a 'layer of junior civil servants' to re-type the data.

If I was paranoid I might suspect there will be another layer of people censoring it and fiddling with it to look good, but that's another story...

Re:The downside of all this

[Joce640k]

Yep, 2k is hardly a "minor purchase". How many taxpayers do you suppose it takes to pay for that (plus your wages)?

Re:The downside of all this

[PatrickThomson]

Oh COME ON. The situation is: "here's some money, make our computer infrastructure work. But wait! We'll micromanage even though we know fuck-all." It's not like there isn't already accountability in the total budget, and if nothing else just dump all the data in a public repository somewhere and let paranoid netizens crawl it. Some red tape is an appropriate thing, but this isn't going to stop typical budgetary BS politics like blowing it all before the end of the fiscal year.

Re:The downside of all this

[Anonymous Coward]

What's really something is that the poster works for a "non-elected government body"

In Soviet Russia...aw never mind

Re:The downside of all this

[VShael]

Non Elected Government Body = Civil Service.

It's hardly Soviet Russia.

Re:The downside of all this

[rich_r]

You have to account for the money you spend anyway. From your point of view as a buyer, nothing will change.

What will happen is the way the accounts get reported up will change, and this won't affect you in the slightest.

Re:The downside of all this

[scamper_22]

This is ultimately why government should not do very much.

The idea that you can have honest, efficient, accountable government is impossible.

It is far better to let people offer their services and have other people choose to pay for those services... aka... a free market.
I don't know if the restaurant chain down the street is run efficiently and I certainly don't need to know the details of how they run their business. I really don't care. What I do care about is that they make amazing food with amazing service for $8 a plate. The other restaurants are not a high quality. What better accountability than people handing over their own money?

The UK is the ultimate in creating bureaucratic nonsense. Their best teachers spend more time writing reports and going through red tape than teaching... all to justify the costs in education.

So like you, I'm not too impressed with the extra information.

The more the government can leave to free people making free choices, the better society is.
Yes, the legal system and various monopolies (roads,electricity) will always involve government. And I'm always willing to just accept 'reasonable' spending in those areas. Even if everything is not accounted for.

The problem comes when government is expected to run everything from healthcare to education to obesity problem... now there is way too much money in the hands of bureaucrats who ultimately do not have the accountability of people putting up their own money.

Re:The downside of all this

[UBfusion]

Transparency is very desirable until tabloids start calling this a major scandal - Amazon sells 64GB for £760, where's the rest £1240 ????

Re:The downside of all this

[Anonymous Coward]

Transparency is very desirable until tabloids start calling this a major scandal - Amazon sells 64GB for £760, where's the rest £1240 ????

At a complete guess, he's remembering a price from last year.

Re:The downside of all this

[hughk]

I am all for transparency but it will also have a negative effect on the efficiency of the government

Who is paying for it, who pays your salary? Remember we aren't talking about including the public in the purchase decision. We are talking about review.

Do you really need to know I spent £2K on 64GB Ram for a blade ?

If I find that you are paying 2x the cost of memory elsewhere or an extraordinary amount with one supplier - that is a very big red flag and I'm interested. We can all name government projects that have gone spectacularly wrong even with oversight and this is another form of review.

Re:The downside of all this

[Anonymous Coward]

Non Elected Government Body = Civil Service.

It's hardly Soviet Russia.

It is. Its SOCIALISM!!!

Transparency

[Anonymous Coward]

Good. Transparency, and honesty, make for better government. I was shocked at how our school district was consistently ignoring the budget that had been set and choosing to spend more despite repeated voter instructions to stop this behavior.

Re:Transparency

[PatrickThomson]

Were these the same voters that asked for reduced class sizes and better grades for their precious snowflake? The same voters that sued the school for sports injuries?

Nice to see a government reducing it's own privacy

[calmofthestorm]

rather than mine.

I know it's early days...

[VShael]

but is it at all possible that the coalition is doing GOOD instead of evil?

I do believe that's unprecedented.

Here We Go Again, Hollywood...

[pandrijeczko]

Bletchley Park's archive is to be digitised and put online. It seems HP made an offer to help out with scanners and expertise ...which will result in yet more "authentic" WW2 movies starring Matthew McConaughy as the Yanks credit themselves with creating the technology that our glorious "British Boffins at Bletchley" created.

Did they ever make that movie about The Battle Of Britain where the top Allied fighter ace was going to be played by an American, despite the fact they wouldn't be entering the war for another year or so?

Questionable Accounting

[Iyonesco]

From the site: "The ‘fact tables’ are approximately 70MB. With a fast broadband link of 8mbps, it will take approximately 10 minutes to download this file."

70MB at 1MB/second = 600 seconds!?!? This left me rather concerned as to the reliability of the figures on the site. I never went on to look at the data but I can imagine it would be something like this:

Expenses: £100,000,000,000.00
Bureaucracy: £500,000,000,000.00
Propaganda: £25,000,000,000.00
Big Brother: £50,000,000,000.00
Foreign Wars: £10,000,000,000.00

Total: £1.23

Re:Questionable Accounting

[pablodiazgutierrez]

70 megabytes at 1 megabit/s take 573 44 seconds, which can be honestly rounded up to 10 minutes.

UK national debt

[UBfusion]

I would appreciate if any of the economists in the /. crowd could point to figures in these data explaining the height of the UK national debt...

This is great

[Anonymous Coward]

This is great, the UK government has never had open-ness with regards to its financial matters. Generally you would have to ask the Prime Minster during Prime Minister Question Time, and he obviously wouldn't have to answer (I don't recall, I don't have it on me right now, etc).

Actually you don't know what you are talking about

[Anonymous Coward]

actually 2k IS a minor purchase. This isn't some home-users crap and you think they buy this stuff off of ebay or are able to get their credit cards out to buy it and claim it back?

They will have recommended suppliers or they will have required suppliers who will be able to charge a premium.

Unless you've worked in a place like this, you really cannot believe how much things cost when you have little say over who supplies your product.