Obama Administration Places $200 Million Bet On Big Data

wiredmikey writes "As the Federal Government aims to make use of the massive volume of digital data being generated on a daily basis, the Obama Administration today announced a 'Big Data Research and Development Initiative' backed by more than $200 million in commitments to start. Through the new Big Data initiative and associated monetary investments, the Obama Administration promises to greatly improve the tools and techniques needed to access, organize, and glean discoveries from huge volumes of digital data. Interestingly, as part of a number of government announcements on big data today, The National Institutes of Health announced that the world's largest set of data on human genetic variation – produced by the international 1000 Genomes Project (At 200 terabytes so far) is now freely available on the Amazon Web Services (AWS) cloud. Additionally, the Department of Defense (DoD) said it would invest approximately $250 million annually across the Military Departments in a series of programs. 'We also want to challenge industry, research universities, and non-profits to join with the Administration to make the most of the opportunities created by Big Data,' Tom Kalil, Deputy Director for Policy at OSTP noted in a blog post. 'Clearly, the government can't do this on its own. We need what the President calls an 'all hands on deck' effort.'"

Comments

Great QOTD

[smwny]

The QOTD at the bottom of the page so perfectly matched this story.

All the taxes paid over a lifetime by the average American are spent by the government in less than a second. -- Jim Fiebig

Re:Great QOTD

[the_Bionic_lemming]

It isn't possible to buy anything without paying taxes.

Public aid cards are used by people who do not pay taxes with money they earned - the money they use and the money spent on taxes are earned by others.

Ergo, it is possible for people to buy things without actually paying for it themselves - or paying taxes themselves.

Re:Great QOTD

[owlnation]

It isn't possible to buy anything without paying taxes.

You can buy politicians without paying taxes.

Re:Great QOTD

[Anonymous Coward]

I understand your joke but in reality there are about 40-50% that pay no federal income tax at all and a decent percentage of that group get back more than they paid during the year (they MAKE money when they file).

On a side note.. I really don't think most people understand the tax refund concept and tax withholdings from your paycheck. They don't realize that they are getting money back that they gave to the government already with the exception of those I mentioned above that get even more back then they paid in.

Re:Great QOTD

[the_Bionic_lemming]

No, it doesn't.

If you are a leech on society, and produce noting - then you cannot claim to have done anything other than being a leech.

The taxes are paid by the people who are forced to allow the leeches to live.

If you are a leech and think you are paying taxes - or trying to convince others that you are doing anything other than being a leech, rest assured, we know better.

Re:Great QOTD

[FrootLoops]

That's kind of a silly statistic to use, since it's sensitive to population size. There are about 310 million Americans, and 1 year contains about 31 million seconds. You'd expect (roughly) 10 years of an average taxpayer's taxes to be spent every 1 second if revenues == expenditures, everything is inflation adjusted, and everyone pays taxes. If the country instead had only 31 million people you'd expect 1 year per second. The implied point--that government spending is out of control and/or hugely wasteful--depends crucially on factors not mentioned in that one-liner. Without further analysis all it does is illustrate how huge and unintuitive national scales are.

I wonder if the original quote is inflation adjusted. It would be an easy mistake to make to forget that today's dollars and dollars of the past aren't directly comparable and should not be aggregated naively. It would also account for a large part of the deviation between my very rough estimate and the quote (~1 lifetime's taxes / 4 seconds vs. 1 lifetime's taxes / <1 second).

Privacy?

[GeneralTurgidson]

When it comes to big data, there's going to be little privacy.

Re:Privacy?

[Tsingi]

Yeah. Considering what the data is that they want to process.

Re:Privacy?

[Sarten-X]

That's an absolutely unfounded concern.

I worked at a Big Data company. About 90% of my job was improving privacy while maintaining the integrity of medical data. The patient's zip code was reduced to 3 digits. Any references to states were removed and forgotten (because there are some zip codes that cross state lines). Any names were removed, as were any user-entered comments (doctor's notes, etc.) that might possibly contain personal information. Any personal information that is necessary for the system but might be identifiable is salted and hashed twice before it ever leaves the source (hospital, insurance provider, etc).

That in itself isn't good enough for privacy, so we then used some proprietary methods (that was kinda outside my job, so I don't know much about them) to intentionally screw up the data we provided to our users. A user could find out, for example, that between one and fifty people in the vicinity of Denver had a particular medical condition on a particular date, and received a particular drug. Narrow down results more than that, and my company's system simply wouldn't fulfill the request.

This isn't really the exception to how many Big Data companies treat their data. Believe it or not, Big Data providers take privacy seriously, and are willing to sacrifice perfect accuracy to run an ethical operation. Anyone interested in Big Data is running on statistics anyway, so statistically-insignificant methods are easy to preserve privacy.

Re:Privacy?

[Bravoc]

Yea but.... we're not talking about a company where they could become the target of civil litigation. We're talking about the US Federal Govurn-munt. Need I say more? I don't feel good about this at all.

Doesn't this bother anyone?

[Anonymous Coward]

Clearly, the government can't do this on its own. We need what the President calls an 'all hands on deck' effort

So the Obama wants to pick and choose how this will be handled but he wants everyone else to do it? Whatever happened to representation?

Re:Doesn't this bother anyone?

[Anonymous Coward]

Why does every criticism of Obama lead to howls of racism? You spent 70% of a post shouting about racism when not a racist word was said. And yes, some of us were against Bush just as much as Obama. We don't all fit into tiny little slots, some of us don't wear partisan labels. How about you?

Re:Doesn't this bother anyone?

[LandoCalrizzian]

I'm black, and I think that even if Obama is "my nigga", he cannot be trusted, and he betrayed most of those who voted for him with all his false promises.

I intend to vote against "my nigga" next time.

I guess you can call me an Uncle Tom, but I'd rather be called that than be called a sucker or an idiot.

Posting as AC and talking about trust is an irony in itself. Anyone naive enough to think that the President can drastically change the way a nation works in 4 years is a sucker and an idiot. Government is slow to work and slow to react because it is run by a committee of people who have their own agenda that doesn't always align with the President's agenda. It doesn't matter who is in office, if the kids can't play nice then nothing gets done. Let's set the record straight, a President has the least control on how a nation is run, the body of Congress has most of the power. If you want one man to run the country move to Syria. If you want change, you should be more selective of your Congressional representatives instead of only paying attention to 2 people for ~6 months every 4 years.

Right investment, right time

[macwhizkid]

I'm a hard science/computer science guy who's livelihood is working on various NIH/NSF projects. A common thread talking to other scientists the past few years has been the theme that the tools for data analysis have not kept pace with the tools for data acquisition. Companies like National Instruments sell sub-$1000 USB DAQ boards with resolution and bandwidth that would make a scientist from the early 1990's weep for joy. But most data analysis is done the same way it's been done since that same era: with a desktop application working with discrete files, and maybe some ad-hoc scripts. (Only now the scripts are Python instead of C...)

The funny thing is, most researchers haven't yet wrapped their brains around the notion of offloading data onto cloud computing solutions like Amazon AWS. I was at an AWS presentation a couple months ago, and the university's office of research gave an intro talking about their new supercomputer that has 2000 cores, only to get upstaged 10 minutes later when the Amazon guys introduced their 17000 core virtual supercomputer (#42 on the top 500 list, IIRC). There's a lot of untapped potential right now for using that infrastructure to crunch big data.

And the cost to move the data?

[oneiros27]

Amazon is using the idle time of their huge cloud when it's not being used for christmas shopping ... so the cost of CPU is relatively cheap. Bandwidth and storage is *not* with most cloud sevices.

So, say I need to calibrate a year's worth of SDO/AIA data ... that'd mean pushing to them somewhere in the range of 500TB of data, and then pulling it back again. They've changed their pricing so the transfer in is now free ... but if I'm doing the math right, that'd cost somewhere on the order of $30k for the transfers, and if we assume we're pushing it in and deleting it as soon as it's done, we don't need a lot of storage. For other processes, people *do* need the storage, which runs around $100/TB/month, so $50k ... per month.

It's not as impressive, but it's more cost effective in the long run to build in your own processing near the data. Would it be nice to redo two years of calibration in a day, rather than the ~3hrs to process 1 day's data that it takes now? Yes, but we don't have the funding to pay for it. (every launch delay costs money (gotta keep the scientists employed, store satellites in machine rooms, pay for offices, etc.) ... and that money, without fail, gets taken from the actual running of the mission and the data analysis.

What I'd personally like to see is more large scale infrastructure coordination, and for any project where the PI team's composed entirely of physicists yet they're designing and implementing their own data system be immediately de-funded.

I'm not going to say that everyone should be using iRODS or OODT or whatever the next new sexy thing is ... but a physicist writing the drivers that run the tape drives? That's a sign something's gone horribly wrong, and yet it's still happening.