Slashdot Mirror
US Government Upgrades RAM
Deep Throat writes "Techworld has the scoop on a new super-sized RAM disk that the US government has just bought for a few million dollars in order to speed up searching through huge databases. It's 2.5TB! The VP of the company that made it says it is for Washington DC and searching databases but won't say who. Techworld explains why it reckons it's the Department of Homeland Security searching in the NSA and Pentagon databases for terrorists. And apparently the government is 'very happy' with the purchase and thinking about getting more."
How are they going to save the data if the power goes out? That's a lot of data to store on hard copies.
However, not that many departments could possibly want to run such vast queries regularly. It would also be extremely difficult to justify a $4.7 million investment unless that work was seen as vital and speed was a main consideration in that work. It is also peculiar that such a large purchase could be approved at a time of tightening belts.
Honestly, I wonder what the author was smoking?
* However, not that many departments could possibly want to run such vast queries regularly.
You don't think so? I think *every* DBA would like to have a few extra TB of RAM. Maybe the Department of Transportation just wants a more efficient way to keep track of US Highway routes?
* It would also be extremely difficult to justify a $4.7 million investment...
What country is this guy living in? If you're high up enough, it's trivially easy to justify $5 million. That's hardly enough to build one Interstate highway intersection.
* It is also peculiar that such a large purchase could be approved at a time of tightening belts.
Oh, now I know the problem. The author has been in a coma for the past 18 months. Wake up, dude, and smell the money!
Stressed? Me? Of course not. Stress is what a rubber band feels before it breaks, silly.
This was bought by the Internal Revenue Service in order to improve the auditing of tax returns. They say that the additional revenue brought in will easily pay for the device many times over.
If it was for the DHS or NSA you would not have heard about the purchase.
Could be memory for pattern matching genomic alignments : Health and Human Services.
Could be memory for modeling Nuclear Blasts : Department of Energy.
Could be everybody's tax returns since the Truman administration : Internal Revenue Service.
WTFK.
that's about 8KB for every person in the US, including the 'terrorist' bit. You can put a lot of personal data in 8KB.
try { do() || do_not(); } catch (JediException err) { yoda(err); }
The parent post isn't just funny, the poster has a very good point. Google does this on a regular basis and, from what I understand, does it very well. And without lots of really expensive and specialized hardware. They've got a lot of really, really good graph theorists and other such people working for them, too, so I'd expect that whatever they do can be generalized quite nicely.
...exactly what they're going to use this for. Believe me, the US government isn't doing anything sinister now that there's a department of homeland security and we have the "post 9-11" buzzword that is any worse than what they were doing before. You just hadn't heard about it yet back then.
Let me tell you a little story.
January 2001. There are still two very tall buildings at the southern end of Manhattan. A certain three letter agency paid a *large* sum of money for me to signifigantly alter the virtual memory subsystem of the linux kernel for alpha processors. The validation testing and application of the modifications inolved benchmarks that compared large quantities of text against a large word list. The modification changed the amount of data they could pump through their (very large) cluster of machines by about 15%. Why would their motive today be any different? If this has been going on all this time, how can you say for sure what they're going to use 2.5TB of fast storage for?
Google wouldn't need the clearances to be asked to supply the technology. The government just goes and says "we need to search some data quickly, what can you sell us?" and Google gives them some algorithms and code and such. Which they then pore over to look for security holes and then isolate nicely from everything else on the planet.
No classification req'd.
they didn't just throw money at it, they also threw brainpower at it.
The Government likes to waste money. This was their best solution.
This cautionary tale is a USENET fable:
VAXen, My Children, Just Don't Belong In Some Places
Usenet Apocrypha
VAXen, my children, just don't belong some places. In my business, I am frequently called by small sites and startups having VAX problems. So when a friend of mine in an Extremely Large Financial Institution (ELFI) called me one day to ask for help, I was intrigued because this outfit is a really major VAX user--they have several large herds of VAXen--and plenty of sharp VAXherds to take care of them.
So I went to see what sort of an ELFI mess they had gotten into. It seems they had shoved a small 750 with two RA60's running a single application, PC style, into a data center with two IBM 3090's and just about all the rest of the disk drives in the world. The computer room was so big it had three street addresses. The operators had only IBM experience and, to quote my friend, they were having ``a little trouble adjusting to the VAX,'' were a bit hostile towards it and probably needed some help with system management. Hmmm, Hostility.... Sigh.
Well, I thought it was pretty ridiculous for an outfit with all that VAX muscle elsewhere to isolate a dinky old 750 in their Big Blue Country, and said so bluntly. But my friend patiently explained that although small, it was an ``extremely sensitive and confidential application.'' It seems that the 750 had originally been properly clustered with the rest of a herd and in the care of one of their best VAXherds. But the trouble started when the Chief User went to visit his computer and its VAXherd.
He came away visibly disturbed and immediately complained to the ELFI's Director of Data Processing that, ``There are some very strange people in there with the computers.'' Now since this user person was the Comptroller of this Extremely Large Financial Institution, the 750 had been promptly hustled over to the IBM data center which the Comptroller said, ``was a more suitable place.'' The people there wore shirts and ties and didn't wear head bands or cowboy hats.
So my friend introduced me to the Comptroller, who turned out to be five feet tall, 85 and a former gnome of Zurich. He had a young apprentice gnome who was about 65. The two gnomes interviewed me in whispers for about an hour before they decided my modes of dress and speech were suitable for managing their system and I got the assignment.
There was some confusion, understandably, when I explained that I would immediately establish a procedure for nightly backups. The senior gnome seemed to think I was going to put the computer in reverse, but the apprentice's son had an IBM PC and he quickly whispered that ``backup'' meant making a copy of a program borrowed from a friend and why was I doing that? Sigh.
I was shortly introduced to the manager of the IBM data center, who greeted me with joy and anything but hostility. And the operators really weren't hostile--it just seemed that way. It's like the driver of a Mack 18 wheeler, with a condo behind the cab, who was doing 75 when he ran over a moped doing it's best to get away at 45. He explained sadly, ``I really warn't mad at mopeds but to keep from runnin' over that'n, I'da had to slow down or change lanes!''
Now the only operation they had figured out how to do on the 750 was reboot it. This was their universal cure for any and all problems. After all it works on a PC, why not a VAX? Was there a difference? Sigh.
But I smiled and said, ``No sweat, I'll train you. The first command you learn is HELP'' and proceeded to type it in on the console terminal. So the data center manager, the shift supervisor and the eight day operators watched the LA100 buzz out the usual introductory text. When it finished they turned to me with expectant faces and I said in an avuncular manner, ``This is your most important command!''
The shift supervisor stepped forward and studied the text for about a minute. He then turned with a very puzzled expression on his face and asked, ``What do you use it
And here everyone thought 64 bit would only be used ONLY in virtual!! Wow! I wonder if it's running on "Big Iron", since T-Rex might be the only monster that could handle the load. (z990 from IBM [ os = z/OS]). They also are using FICON which seems kinda dated for the technology I'd like to see more on the switching capability. I wonder how many CPU's she's running?
Mod +5 Drunk
Yeah, my point wasn't about using Google to search for them, but using Google's technology.
I did a search for "a" on Google, and Results 1 - 10 of about 3,370,000,000. Search took 0.14 seconds . Does US have that many people?
Could this be used with a large enough pre-computed table to crack encryption? Maybe the NSA is hoping to win RSA's next challenge.
</tinfoil>
Want to improve your Karma? Instead of "Post Anonymously", try the "Post Humously" option.
I was seriously looking at this product as a solution for a really bad database performance issue for a major financial institution. Instead of redesigning the whole database (small, around 2-3TB), we were thinking about putting a couple of these in to replace the slow EMC 8830/DMX infrastructure. Unfortunately, the PCI busses of the DB server weren't fast enough to keep up with it (Unisys ES7000 420). When you look at the TCO, the man years required to redesign the DB and applications were much more expensive than dropping in another few million in hardware. The only downside that I saw to the terraramsan solution was that it eats power and generates way too much heat. This thing would be great for horizontally scaled databses, as long as your physical plant could support it. Brute force always wins over recoding. That is, until we got offshored by Indians billing at $30/hour.
Look at the size of that thing!
When I worked for the Inland Revenue in the UK, we hooked into a national database of personal details available to a wide variety of government departments. We used Telnet clients via an intranet, and although I forget the precise specs of the central server, it sure as hell didn't need a RAM disk that big.
We never had any problem waiting for results, either. On a really bad day, you'd get maybe a 1s lag between hitting enter and a results screen coming up.
So if the US.gov needs a RAM disk that big, it's one fricking huge database. I have to wonder what sort of info it carries (part of the size might be due to things like photos, fingerprints, criminal records - stuff our DB didn't use), and how many people are on there (100% of the population?).
The scary thing: what if 2.5TB is a fraction of the database size - say, 25 to 50%? You'd still get reasonable performance, but the idea of a government holding 5-10TB of personal data seems positively Orwellian. "Big Bush is watching you"?
Disclaimer: I am not a techie, a lawyer or a government analyst. And it's only 5 years since I junked my Amiga 500, which did perfectly well with a mere 1MB of RAM, so maybe I'm used to thinking on a different scale. If you feel I'm wrong in any way, please feel free to correct me - I actively appreciate it!
"It is dark. You are likely to be eaten by a grue." -- Zork
Note that name of the buyer has been kept secret. None of the agencies you mentioned need their names to be kept secret... infact Tax Returns dept will scream all over that it can now serve the texpayers better!!
So the issue is who was the buyer who needed this to make his HUGE database reasonably fast AND needs to keep the name secret.
- mritunjai
Where can I buy a *small* ramdisk, on the cheap?
For example, I have about ten 128mb PC100 DIMMs lying around. I'd love to stick them on some kind of PCI card to make a ramdisk, but I have no idea where to go about getting such hardware... google's results are useless, they're all links to *SOFTWARE* ramdisks that use main memory =(
The unofficial
I am uncertain why you suggest the parent read a newspaper or turn on the television news, considering your implication that it would prove you correct. Instead, it proves that your theory is a conspiracy denied countless times. The NY Times, Washington Post and other newspapers you're referring to have all consistently reported that this theory has been denied. Most cable news channels have had limited coverage altogether, reporting on the status of the Marines stationed in Haiti and noting only briefly Aristide's quaint claim that he was kidnapped, followed by a quotation of the Pentagon's denial and subsequent dismissal. I watch daily CNN, FNC and MSNBC, meaning those are the channels I refer to. Located here is an editorial which covers the majority of the material. For the record, both the Seattle Times and San Francisco Chronicle are respected in the industry, but the NY Times is included here as well.
Here (eatme123/eatme123)
Here
Here
Here
There are occasions in which skepticism is warranted, in which the government is lying to you in order to further its agenda. There may even be times in which the White House spokesman can casually lie to you without showing an iota of remorse, although that would be remarkable. This is not one of those times, as is obvious to those of us who are paying attention. Pull your head out of your ass.
$5 / month hosted VPS on linux = awesome!
Rumor has it that Adolf of WW-II infamy managed a lot of his damage with records stored in shoe-boxes.
It is scary even imagine what they could do with that. Do all the posts regarding privacy come to mind?
Once, I had an argument with a buddy of mine that spoke his mind--to much--over Email. I gave him a little grep script to show how email monitoring could select "suspicious material" for further analysis.
Now, all they have to do is tie it in to a profiling system, and there you go. Orwell's 1984^10 all over again:
FROM ORDER BYSELECT TOP 100
A simple search shows the documentation on TMS' customers and the only thing this large that requires this size index is keyword searches of e-mail intercepts.
Perhaps the gov stored every possible key for various encryption codes and its just one big lookup table for fast code cracking.