Slashdot Mirror
NSA Mimics Google, Angers Senate
An anonymous reader writes "In a bizarre turn of events, the Senate would prefer that the DoD use software not written by the government for the government. Quoting: 'Like Google, the agency needed a way of storing and retrieving massive amounts of data across an army of servers, but it also needed extra tools for protecting all that data from prying eyes. They added 'cell level' software controls that could separate various classifications of data, ensuring that each user could only access the information they were authorized to access. It was a key part of the NSA’s effort to improve the security of its own networks. But the NSA also saw the database as something that could improve security across the federal government — and beyond. Last September, the agency open sourced its Google mimic, releasing the code as the Accumulo project. It's a common open source story — except that the Senate Armed Services Committee wants to put the brakes on the project. In a bill recently introduced on Capitol Hill, the committee questions whether Accumulo runs afoul of a government policy that prevents federal agencies from building their own software when they have access to commercial alternatives. The bill could ban the Department of Defense from using the NSA's database — and it could force the NSA to meld the project's security tools with other open source projects that mimic Google's BigTable.'"
This seems like a result of the conservative cry to shrink the size of the federal gubmint. "Gubmint shouldn't be allowed to do internally what they can outsource to some private company" possibly owned by China. THis is sad
Why should we get something for free when we can pay for it? Wait a minute....
The simple truth is that interstellar distances will not fit into the human imagination
- Douglas Adams
Accumulo runs afoul of a government policy that prevents federal agencies from building their own software when they have access to commercial alternatives
Just arrange to sell it to Google, make them the maintainers, and buy it back for $1.
All ideas^H^H^H^H^Hprocesses in this post are Patent Pending. (as well as the process of patenting all postings)
Will they call the merge of Hbase and Accumulo Humulo or Acbase?
America is backwards! the NSA was doing their best to save the US money, and instead it should have used commercial software?!?
backwards thinking, you should be praising the NSA for once (saving people money...)
It is the result of private corporations lobbying for more privatisation. "Shrink the Government" is the voter-friendly PR spin on it. We have the same in the UK...fortunately the privatised "security" company G4S has just screwed up so massively that the agenda must have been put back a year or so. Personally, I think that any and all national security functions, whether physical or cyber, shouldn't be provided by anybody whose managers I cannot vote out of office.
From scarped cliff or quarried stone she cries "A thousand types are gone, I care for nothing, no not one."
Some congress person's fund needs slushing. If the NSA writes its own software no money changes hands, therefore no graft can take place. It's a serious problem.
for bills &c.
They're created w/ a tool named ACOMP.EXE (which the GPO used to use to make their style manual --- which typeset exactly like a printed copy I have from 1943 --- the new version is done w/ Adobe InDesign CS3 though).
If the Senate can use a special software tool for so prosaic a function, why can't other parts of the government?
William
(who recently had to download the successor to NIH (National Institute of Health) Image to make a reasonably-sized bitmap for placement into an automated pagination system when Adobe PhotoShop insisted on wrapping it up in all sorts of metadata, resulting in a several KB file, when JImage was able to write it out in a mere 480 bytes.)
Sphinx of black quartz, judge my vow.
The NSA's job is to spy on other countries communications and break their codes.
It is also the NSA's job to protect US govt communications & systems by developing secure cryptosystems.
Set Metadata to none when saving in Photoshop
Several years ago when I was a young service member and working for around $25K a year to develop software for the military, I was told that the military was moving away from GOTS solutions and was mandating that everyone move to COTS software. They replaced my position with contractors that made $75K a year and ultimately with multi hundred million dollar contracts with contracting firms who "integrate" in COTS solutions. Granted having become one of those contractors myself and having over doubled my pay in that time frame, I do have to admit I appreciate that cheaper COTS solution. Though I do often times wonder to myself if the Government centralized their development efforts, tracked industry standards for producing secure code, and further developed some of the charming projects they have worked on (like SELinux) what the world would be like today. Just think, instead of knowing a huge ass hole is in your current revision of router code, you could simply send it off to the developers to repair. No lack of a $100K+ support contract to prevent you from getting a patch...
Select from tblFriends where interesting >= 4;
This sort of crap happens all of the time. A few years ago I was tasked to a project to re-implement an existing part of a government support system. Great. Wonderful project to be on. We'd get the hardware needed, had an existing system to base the new system on, had a bunch of user suggestions and complaints, a whole bunch of documentation to base the new system on and a core team willing and capable.
Management then said we would have to buy a product which would provide most of the new system would eventually do and tailor the rest. Management even had an existing vendor and software licence.
Here is where the project went right down the toilet. After several months it was obvious that the chosen vendor's software would not do what was needed. It had to be tossed. Then the wonderful vendor produced another software 'solution'. After a short time of wrangling with this crap our dev threatened to quit and I started looking for a new job.
Funny thing was that in the last six months as the project disintegrated and it all went south and other projects took priority in the background our dev knocked up a good prototype from scratch and had a semi functional system on the go. I did an estimate and figured that if we had started development on our own inhouse solution instead of trying to use third party software then we would have produced the system required. I was so annoyed at the end of all this that I wrote all of this up including a timeline of project comparison of buy vs build showing costs and time.
TFA does not surprise me at all. So much time and money wasted because of this type of stupidity .
I suppose I'll be moderated "troll" if I suggest that the government shouldn't waste time and money rewriting software that already exists and can be licensed in the commercial market. Not that necessarily there's a tool that can support the NSA's massive data-sharing needs, but still.
Accumulo runs afoul of a government policy that prevents federal agencies from building their own software when they have access to commercial alternatives.
This is to keep the government from wasting millions of dollars maintaining in-house software versus just buying it cheaper in the commercial sector.
The law needs updating to state that the cost to write and maintain should be cheaper versus the commercial alternative if you are going to do it in-house.
OMG! We must stop this severe threat to world freedom! Instead of in-house semi-secure software let's require all government agencies dealing with classified information to buy commercial software with built-in Chinese back-doors. How dare they try and keep this information safe from our oriental overlords!
In a bill recently introduced on Capitol Hill, the committee questions whether Accumulo runs afoul of a government policy that prevents federal agencies from building their own software when they have access to commercial alternatives
I work at a large defense contractor, so obviously I'm posting anon. My thoughts on this are as follows: indeed there are requirements to use as much COTS and/or FOSS as possible for things that already exist (and so long as the use of any does not/cannot cause no future licensing issues that can be reasonably foreseen.)
Is in an effort to avoid the "not invented here" syndrome that plagues commercial and government enterprises alike. But the operative idea is that we should use a COTS if it provides the functionality that we need. If there is some type of deviation in the type of functionality that a project needs, it is perfectly reasonable to add new logic around it (or build one from scratch altogether.)
The NSA requirements for retrieving and storing massive amounts of data, when taken as is, do sound like something that Google already does. However, there are other requirements a Google-like COTS might or might not meet or might not meet efficiently (.ie. "tweaking the COTS will cause substantial operational costs down the road", just as a hypothetical example.)
There are needs to attach security label classifiers (TS,S,R,C,SBU,U), and compartment/silos to meet "need-to-know" requirements. There can be security-related non-functional requirements that say the mechanisms for storing/retrieving information above a certain security label be also be labeled with a classifier as strict as the data being handled. Part of the software system might be required to exist within Type 1 cryptography products, with physical shielding and all. It might be required to provide interfaces and protocols aware of sneakernet and airwalls.
Things like that do not get solved by deployment schemes and configuration alone. So "mimicking google" might not be descriptive to what's really going on here.
Furthermore, it looks incredibly stupid for Congress to be telling the NSA to shelve their own FOSS and to look for a COTS alternative. Sometimes, for some types of operations, you simply do not want a COTS. Fine for building government owned systems that handles, say, tax or immigration/nationalization records. Not so fine for TS-level material.
The NSA has been guilty of some major pork-barrel mishaps, and needs fiscal supervision. Hell, the whole defense sector is plagued by inefficiencies. However, this particular action by Congress, it's not a solution.
Wasn't the big reason why we didn't stop the 9/11 attackers was a lack of cross departmental sharing of data. Public or Private systems, it seems like they are going back to the old ways of doing things again.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
Clearly, someone must have paid for this charming little legislative tidbit. But who?
I mean, I could understand if Lockheed-Martin had a proprietary solution that they were offering (with just a few change orders needed to satisfy NSA's requirements, of course), but the beneficiaries here seem to be the Cassandra and HBase projects, neither of which seem likely to have much of a lobbying budget. Was it their forebears at Facebook? Could they possibly care enough?
And blaming it on "conservatives-want-smaller-government" seems pretty silly, too. Sure, turfing Accumulo might conceivably further that goal in some tiny, tiny way, but it's not like some senator was likely to have figured this out by himself. No, clearly someone put them on to it, but who and why?
It's an intriguing mystery. Any ideas?
Who are they kidding? they are going to let the NSA do what they like, but the Senate doesn't want the public to know that.
Hmm. if you worked for a government research lab, you'd run into this all the time. Everytime you want to do something, you have to show that you, and only you (or your lab) can do this work, and that it cannot be contracted out to industry. Want to build a Mars rover? Better show that no industrial partner is capable of doing it. Want to build a prototype widget for some new idea? Better be able to show that nobody else is making something similar, nor that they've patented, nor that they've thought about it and might have mentioned it as a future product, etc. First step these days is to run a google search on what you want to do, and make sure that you can address the top 10 hits with specific reasons why your thing is different.
By this we don't unfairly compete with industry (because we don't have profit, in theory, our costs are lower), and we avoid duplication of services. By this we are removing "waste, fraud, and abuse" from government
>prevents federal agencies from building their own software when they have access to commercial alternatives.
I don't think so, all government software that was written on government time should be released in the public domain just like all research is.
... if my campaign donors make a profit on it."
Just yesterday I was reminded of this old chestnut:
"There is no distinctly American criminal class except Congress." -- Mark Twain
AFAICT that's only an option when using ``Save for Web and Devices...'' --- the file in question needed to be a .tif.
Sphinx of black quartz, judge my vow.
I heard about G4S when Have I Got News For You was still fresh, how many decades ago is that?
In Holland, there was/is a parliamentary inquiry into whether the efforts of privatization in the last decades has produced any positive results or at least any non-negative ones... they are still searching.
But AT THE SAME TIME, the CDA and VVD, Christian and capitalist filth, were advocating MORE of it, despite being able to name a single success or even a single non-disaster. Their solution to the mess the public rail network is in? MORE privatization.
It is often recognized that socialism and communism are ideologies, closely related to religion with believers who truly believe and refuse to take facts into account.
Here is a secret, capitalism is a religion as well and none come more blind.
You can delay the agenda but no matter what happens, no matter how big the disaster, the privatization will continue because without it, if it was recognized it did not work, capitalism would be as dead as communism.
MMO Quests are like orgasms:
You may solo them, I prefer them in a group.
...the committee questions whether Accumulo runs afoul of a government policy that prevents federal agencies from building their own software when they have access to commercial alternatives.
Is this a joke?
I thought Doug Cutting, creator of Hadoop, did a lot of the work on Accumulo too. And they open-sourced it for more people to use, how can that possibly be bad? This seems backwards, it seems the NSA is doing something good here in making up some nice software and releasing it to the world. I think the real question is what sort of vested interest these senators have in the businesses that would "sell similar technology" to the gov't.
Vertically integrating your own software stack isn't necessarily a bad idea. At some scale, if you have enough internal resources, supporting your own code stack becomes more effective than dealing with a large number of third party contractors that are often competing with each-other and not 100% mission focused (think profit motivation). While it makes sense to use a COTS (commercial-of-the-shelf) application for certain problems, the problem of National Security I don't think should be corporatized. I think they should be using the best tools, whether internal or externally developed.
I ordered my Nexus 7 on June 28th and saw almost nothing but Nexus 7 ads for the next week. I received my Nexus 7 yesterday and haven't done a lot of browsing on it (I'm actually rather disappointed in the device - I'm not feeling the "butter", especially in scrolling, and the UI is still inconsistent and does unpredictable things like rotating the display to display certain images).
The STOCK Act needs to be amended to plug the loophole allowing legislators to own stock in industries that they can affect with their political power.
Have gnu, will travel.
Remember $500 hammers? Back in the 1980s, there was a big push to reduce government purchasing costs, especially for military projects, through the use of "Commercial Off-The-Shelf" technology, so whenever possible you'd buy COTS products instead of specially-made customized government-market products. It didn't always make sense, but in many cases it could save a huge amount of money, and realistically a large fraction of the stuff the government bought had commercial equivalents that already had economies of scale keeping the costs down. Sometimes the hammer costs $500 because it's made of MIL-SPEC Titanium, sometimes it's because you spend $490 setting up your hammer-making machine to run off two Left-Handed Jet Engine Hammers for the Air Force, sometimes it's because you spend $600 in contact-lawyer time writing an addendum to a ten-year-old contract to sell two more off-the-shelf hammers to replace the MIL-SPEC ones that got lost.
Government procurement has always had a lot of "check the box on the contract" requirements. Sometimes they make sense, like using COTS to save money when there are commercial products available (especially if that means forcing the organization that wants the stuff to be realistic about what they need.) Sometimes they're theoretically required, but in practice the agency can get a waiver (so everything needs IPv6, but they actually use IPv4, and POSIX was required from mid-80s on but everybody got a waiver and used MS-DOS for office equipment.) Sometimes they increase the costs because the purchasing department puts all that stuff in the contract even though the users don't actually need it.
I did work on some projects where COTS didn't make sense. We were bidding on a communications system that used X.25 (which wasn't yet obsolete :-), but the civilian agency that wanted it had asked the NSA for help specifying a system that would be secure. So yes, it was X.25, but with dozens of special options that no commercial equipment used more than a few of. And the contract specified COTS. How do you reconcile the problem and let the agency check off the "COTS" box on their contract? Make the device, offer it for sale to the market, have a couple of your subcontractors buy boxes from you for "testing" or "evaluation".
Another part of that project not only wanted special-flavor X.25 off the shelf, and POSIX, but also wanted a B1-secure operating system (but it was communication gear, so it would have to be Red Book B1, which was still way-future research, and we had one of the first Orange Book B1 Unix boxes), and GOSIP (the OSI networking stack, though nobody had a GOSIP stack that worked with that particular flavor of X.25 options.) A later project I worked on wanted B1 Secure, POSIX, Ada, POSIX Real-Time (even though the spec wasn't baked yet, and the B1 Secure Unix system didn't support it, and getting that re-evaluated would cost $250K even if we could figure out how to make it work :-)
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
If it's free it will unfairly compete with the open source company I just formed to sell Accumulo service contracts to government agencies.
"And, No! No! You nodareto cut loose from the corporate leash!" [Apologies to ol' Quickdraw]
A specific Air Force agency paid my company to develop a SharePoint application to manage the staffing of Electronic Documents. The agency owns the product (aka code) when we are done. NO annual license fee and can share with whomever they want.
Another AF agency has the exact same requirement, even follows the same regulations, can they use the application. No the license something called TMT, pay to have it customized and have to pay annual licensing in the tens of millions to continue to use the produce. All because of procurement rules designed to protect the private company.
Yes I know that SharePoint is a Microsoft product (BAD) that is licensed but the point is no additional costs for the staffing product they already own.
It appears to me that it's some dumb attempt to create outside jobs instead of in-house programming... a "Make Work" program. if you will.. For the Senate to insist on that when dealing with a Department of Defense contract, though is (IMHO) stupid.. the more people in the general public know the ins and outs of military files, the less secure they are.. And for the military to 'open source' this is INSANE... as now there'll be teams of crackers trying to find out how to gain access into the DoD files, courtesy of the DoD themselves... Sheesh.....