Slashdot Mirror
US Government Upgrades RAM
Deep Throat writes "Techworld has the scoop on a new super-sized RAM disk that the US government has just bought for a few million dollars in order to speed up searching through huge databases. It's 2.5TB! The VP of the company that made it says it is for Washington DC and searching databases but won't say who. Techworld explains why it reckons it's the Department of Homeland Security searching in the NSA and Pentagon databases for terrorists. And apparently the government is 'very happy' with the purchase and thinking about getting more."
Still not as fast as if they'd make the info public domain and use google :)
I remember when legal used to mean lawful, now it means some kind of loophole. - Leo Kessler
Usually the US Government rams upgrades. Iraq, Haiti, next?
in order to speed up searching through huge database
Have they consulted Google?
Sounds like they're taking heed of Google's success in attaining blazing search speeds by holding all the data in RAM.
See here.
The list price of the system, which is made up from 40 RamSan 320 units, reviewed here, is $4.7million
Purchased from Dell's website that would have been....$12.5 million?
They're upgrading in time for Longhorns release you insensitive clod!
I just hope they rememebr to use the USB2.0 for their new keychain disk; access time might be horrible otherwise...
My guess is that they want to play Doom3 with a fps higher than 5 fps.
Is Unreal Doomed
Wonder if they got hold of an early copy of Doom 3.
However, not that many departments could possibly want to run such vast queries regularly. It would also be extremely difficult to justify a $4.7 million investment unless that work was seen as vital and speed was a main consideration in that work. It is also peculiar that such a large purchase could be approved at a time of tightening belts.
Honestly, I wonder what the author was smoking?
* However, not that many departments could possibly want to run such vast queries regularly.
You don't think so? I think *every* DBA would like to have a few extra TB of RAM. Maybe the Department of Transportation just wants a more efficient way to keep track of US Highway routes?
* It would also be extremely difficult to justify a $4.7 million investment...
What country is this guy living in? If you're high up enough, it's trivially easy to justify $5 million. That's hardly enough to build one Interstate highway intersection.
* It is also peculiar that such a large purchase could be approved at a time of tightening belts.
Oh, now I know the problem. The author has been in a coma for the past 18 months. Wake up, dude, and smell the money!
Stressed? Me? Of course not. Stress is what a rubber band feels before it breaks, silly.
This was bought by the Internal Revenue Service in order to improve the auditing of tax returns. They say that the additional revenue brought in will easily pay for the device many times over.
If it was for the DHS or NSA you would not have heard about the purchase.
Before many users start discussing the privacy laws and what not, it should be noted that the data being stored is probably not new. It's the medium on which it is stored on, which is.
Even without this, the old database could have been searched for some terrorists. Nothing has really changed.
Solid State Storage, .... the data is already saved.
http://www.freebsd.org
They really just needed it to beta test Longhorn.
"2.5 TB ought to be enough for anybody."
Heck, that might even be enough to boot Longhorn!
I want to delete my account but Slashdot doesn't allow it.
No comment needed.
After weeks of seeing overblown headlines on slashdot (IE, Bill Gates runs over a squirel on the way to work, headline reads "Bill Gates Murders Animals as part of his Job!!!"), it's good to see one that is, well, underblown. 2.5TB is a bit more than a 'RAM upgrade'
that's about 8KB for every person in the US, including the 'terrorist' bit. You can put a lot of personal data in 8KB.
try { do() || do_not(); } catch (JediException err) { yoda(err); }
RTFA dude. "It also includes three independent internal UPS systems to ensure that no power loss or power supply failure will stop the RamSan from performing its internal backup procedures."
LOAD "SIG",8,1
LOADING...
READY.
RUN
...right after their purchases of thousands of those little X10 spy cameras that you can mount just about anywhere! Oops, said too much, gotta go before-------------
My god carries a hammer. Your god died nailed to a tree. Any questions?
You have no idea what it's for. The list of known terrorists and their acquaitances is relatively short, I cant imagine more than a few gigabytes being needed.
Perhaps it's to store tax returns so the government can mail you your refund check faster. (Job required, sorry).
Maybe INS (or USCIS or whatever they're called) want to track the tidal wave of benifits being handed to Mexican illegals.
I'm a little tired of all this Big Brother speculation. Get over it.
I don't need no instructions to know how to rock!!!!
I just hope they didn't get it at Fry's. God help them if they've got to return it.
"Prepare for the worst - hope for the best."
oh man, and I thought I was the only one with a collection THAT big.
This is supposed to be a caching system, not a long term archive. They also undoubtedly have both Uninteruptable Power Supplies (think racks full of car batteries) and generators to protect from power failure. The databases that it caches are more than likely mirrored at multiple locations, and backed up daily, if not in realtime to an autmated tape library system.
Sure, it could be used for good. But you know and I know that it will eventually be used for evil.
Sorry, but the best I can imagine is a RAID of those things.
The Tao of math: The numbers you can count are not the real numbers.
A certain three letter agency
NSA, CIA, or the Department of FUD?
Did your compensation include Xena tapes and Hot Pockets?
This cautionary tale is a USENET fable:
VAXen, My Children, Just Don't Belong In Some Places
Usenet Apocrypha
VAXen, my children, just don't belong some places. In my business, I am frequently called by small sites and startups having VAX problems. So when a friend of mine in an Extremely Large Financial Institution (ELFI) called me one day to ask for help, I was intrigued because this outfit is a really major VAX user--they have several large herds of VAXen--and plenty of sharp VAXherds to take care of them.
So I went to see what sort of an ELFI mess they had gotten into. It seems they had shoved a small 750 with two RA60's running a single application, PC style, into a data center with two IBM 3090's and just about all the rest of the disk drives in the world. The computer room was so big it had three street addresses. The operators had only IBM experience and, to quote my friend, they were having ``a little trouble adjusting to the VAX,'' were a bit hostile towards it and probably needed some help with system management. Hmmm, Hostility.... Sigh.
Well, I thought it was pretty ridiculous for an outfit with all that VAX muscle elsewhere to isolate a dinky old 750 in their Big Blue Country, and said so bluntly. But my friend patiently explained that although small, it was an ``extremely sensitive and confidential application.'' It seems that the 750 had originally been properly clustered with the rest of a herd and in the care of one of their best VAXherds. But the trouble started when the Chief User went to visit his computer and its VAXherd.
He came away visibly disturbed and immediately complained to the ELFI's Director of Data Processing that, ``There are some very strange people in there with the computers.'' Now since this user person was the Comptroller of this Extremely Large Financial Institution, the 750 had been promptly hustled over to the IBM data center which the Comptroller said, ``was a more suitable place.'' The people there wore shirts and ties and didn't wear head bands or cowboy hats.
So my friend introduced me to the Comptroller, who turned out to be five feet tall, 85 and a former gnome of Zurich. He had a young apprentice gnome who was about 65. The two gnomes interviewed me in whispers for about an hour before they decided my modes of dress and speech were suitable for managing their system and I got the assignment.
There was some confusion, understandably, when I explained that I would immediately establish a procedure for nightly backups. The senior gnome seemed to think I was going to put the computer in reverse, but the apprentice's son had an IBM PC and he quickly whispered that ``backup'' meant making a copy of a program borrowed from a friend and why was I doing that? Sigh.
I was shortly introduced to the manager of the IBM data center, who greeted me with joy and anything but hostility. And the operators really weren't hostile--it just seemed that way. It's like the driver of a Mack 18 wheeler, with a condo behind the cab, who was doing 75 when he ran over a moped doing it's best to get away at 45. He explained sadly, ``I really warn't mad at mopeds but to keep from runnin' over that'n, I'da had to slow down or change lanes!''
Now the only operation they had figured out how to do on the 750 was reboot it. This was their universal cure for any and all problems. After all it works on a PC, why not a VAX? Was there a difference? Sigh.
But I smiled and said, ``No sweat, I'll train you. The first command you learn is HELP'' and proceeded to type it in on the console terminal. So the data center manager, the shift supervisor and the eight day operators watched the LA100 buzz out the usual introductory text. When it finished they turned to me with expectant faces and I said in an avuncular manner, ``This is your most important command!''
The shift supervisor stepped forward and studied the text for about a minute. He then turned with a very puzzled expression on his face and asked, ``What do you use it
And here everyone thought 64 bit would only be used ONLY in virtual!! Wow! I wonder if it's running on "Big Iron", since T-Rex might be the only monster that could handle the load. (z990 from IBM [ os = z/OS]). They also are using FICON which seems kinda dated for the technology I'd like to see more on the switching capability. I wonder how many CPU's she's running?
Mod +5 Drunk
If you go to disk, just once, you need about 9ms just to get the disk heads in position. If you're reading a file system of complex database, you now have multiple disk seeks and reads. That adds up. Seeking in RAM is orders of magnitude faster. That's why all the good search engines keep *everything* in RAM all the time.
That is why Google has multiple copies of the entire web in memory.
-AS
This is what's going through the mind of someone at redmond right now...
"With that much ram, I won't have to worry about fixing the memory leaks!"
They're using it to store ripped DVDs! It's an outrage! They're stealing the bread from the mouths of Julia Roberts and Steven Seagal!
How are they going to save the data if the power goes out? That's a lot of data to store on hard copies.
Of course, possible power outages were considered before the system was purchased. To prevent the loss of information, they hired 35,000 people to watch console screens and transcribe the data from the screen onto legal pads 24x7.
-- If god wanted me to have a sig, he'd have given me a sense of humor.
Could this be used with a large enough pre-computed table to crack encryption? Maybe the NSA is hoping to win RSA's next challenge.
</tinfoil>
Want to improve your Karma? Instead of "Post Anonymously", try the "Post Humously" option.
Perhaps we should begin to consider the implications of this step in analytical ability. Every techie knows that the Government (in cooperation with major Corporations) has had the methods in place to track individuals... RFID tags, GPS locating (in automobiles as well as hand held units. You can tell your position, but the position is also transmitted back... Five Star anyone?), Cell Phone triangulating, and thanks to the Patriot Act, the Government now has legal access to the records of pretty much any transaction we make with bookstores, libraries, etc (and probably more places as well... and this isn't even taking into account information that they might be recieving in ways that we do not know of).
The thing that (we shall assume) they didn't have before was the ability to instantaneously cross check this information. (I assume this because... well... why would they have bought the drive otherwise?) Now that they can check such information so quickly, will we be brought into an era of "Total Information Awareness" as the government spoke of not too long ago? Does this smell the same as the Thought Police to anyone else?
To me, this presents at least the intent by the government to achieve total information awareness, if it doesn't actually achieve it. And the intent is bad enough. Perhaps they're not reading our minds, but the ability to monitor our actions in such vast varities of levels comes pretty close to doing so.
-Vendal Thornheart
Yeah, I have one of these things too. It really speeds up my builds.
I don't need no stinkin' sig!
Now checking 2,748,779,069.440 bytes of memory, please wait.....
I was seriously looking at this product as a solution for a really bad database performance issue for a major financial institution. Instead of redesigning the whole database (small, around 2-3TB), we were thinking about putting a couple of these in to replace the slow EMC 8830/DMX infrastructure. Unfortunately, the PCI busses of the DB server weren't fast enough to keep up with it (Unisys ES7000 420). When you look at the TCO, the man years required to redesign the DB and applications were much more expensive than dropping in another few million in hardware. The only downside that I saw to the terraramsan solution was that it eats power and generates way too much heat. This thing would be great for horizontally scaled databses, as long as your physical plant could support it. Brute force always wins over recoding. That is, until we got offshored by Indians billing at $30/hour.
Look at the size of that thing!
When I worked for the Inland Revenue in the UK, we hooked into a national database of personal details available to a wide variety of government departments. We used Telnet clients via an intranet, and although I forget the precise specs of the central server, it sure as hell didn't need a RAM disk that big.
We never had any problem waiting for results, either. On a really bad day, you'd get maybe a 1s lag between hitting enter and a results screen coming up.
So if the US.gov needs a RAM disk that big, it's one fricking huge database. I have to wonder what sort of info it carries (part of the size might be due to things like photos, fingerprints, criminal records - stuff our DB didn't use), and how many people are on there (100% of the population?).
The scary thing: what if 2.5TB is a fraction of the database size - say, 25 to 50%? You'd still get reasonable performance, but the idea of a government holding 5-10TB of personal data seems positively Orwellian. "Big Bush is watching you"?
Disclaimer: I am not a techie, a lawyer or a government analyst. And it's only 5 years since I junked my Amiga 500, which did perfectly well with a mere 1MB of RAM, so maybe I'm used to thinking on a different scale. If you feel I'm wrong in any way, please feel free to correct me - I actively appreciate it!
"It is dark. You are likely to be eaten by a grue." -- Zork
Or else the Ministry of Information may just be looking for a Mr. Buttle rather than the Mr. Tuttle they really want to find.
...1-800-MAGNETS reported a very large order from an undisclosed source for very high-powered magnets...
+1 Insightful, -1 Troll. What can I say, I'm an Insightful Troll.
Where can I buy a *small* ramdisk, on the cheap?
For example, I have about ten 128mb PC100 DIMMs lying around. I'd love to stick them on some kind of PCI card to make a ramdisk, but I have no idea where to go about getting such hardware... google's results are useless, they're all links to *SOFTWARE* ramdisks that use main memory =(
The unofficial
In 15 years (or less) people will have this much data on their Palm Pilot.
And somehow, my wife will still be able to fill it up with MP3's.
SELECT * FROM tblNSA WHERE usLivingIn = "true" AND ethnicity = "arab"
SELECT * FROM tblCIA WHERE ss = "xxxxxxxxx" AND surname = "Kerry" AND dirt = "true"
SELECT * FROM tblFBI WHERE student = "true" AND politicalID = "left" ORDER BY antiwar
UPDATE tblTEXASAF SET duty_fulfilled = "true" WHERE ss = "xxxxxxxx" AND surname = "Bush"
errr... "dude", look up what non-volitile means. Volatile memory generally is implemented as a series of capacitors which can be queried if they have a charge or not (1 or 0). The capacitors in the RAM in your computer, however, lose their charge very quickly, and need to be "refreshed" every couple milliseconds or so.
Non-volatile needs little or no refreshing. It's usually implemented by component that do something else than hold a charge. And thus, since it doesn't need freshing it keeps information when the power goes out. The reason all memory isn't like Non-volatile is that it is either much slower (magnetic storage) or much more expensive (ie flash and this RAM disk).
Why, o why must the sky fall when I've learned to fly?
Read the article- it refers to the 2.5 TB space being used as temporary query data storage for a 100TB+ databank.
What we call folk wisdom is often no more than a kind of expedient stupidity.-Edward Abbey
Even 15 years later it's still damn funny.
Just to set the record straight, the original author of this post is Jack Harvey, and it was originally published under the title "The Immortal Murderer" on January 18th, 1989 on DECUServe, the DECUS member bulletin board.
This bulletin board is still active under the name Encompasserve.org after mergers of Digital Equipment Corporation and Compaq with Hewlett Packard.
The original publication can still be found on that bulletin board in the archived Soapbox conference, note number 168.
For those of you who were not born, Monday 19-Oct-1987 was the day the stock market crashed.
Echelon
Look at Chile for a past example. Venezuela today too, although hard to say for certain.
$5 / month hosted VPS on linux = awesome!
2.5 Tb of magnetic disk space is peanuts. I have almost that much attached to my machine at home for video editing purposes. You are talking less than $2000 for fast 7,200 RPM IDE disks. So perhaps $10-15k for 2.5 Tb of really fast disks.
Rumor has it that Adolf of WW-II infamy managed a lot of his damage with records stored in shoe-boxes.
It is scary even imagine what they could do with that. Do all the posts regarding privacy come to mind?
Once, I had an argument with a buddy of mine that spoke his mind--to much--over Email. I gave him a little grep script to show how email monitoring could select "suspicious material" for further analysis.
Now, all they have to do is tie it in to a profiling system, and there you go. Orwell's 1984^10 all over again:
FROM ORDER BYSELECT TOP 100
Given the new cap limits and surcharges alot of american broadband connections have impossed and increased average speed of connections. It would make only perfect sence to have the fastest spamming maching possible. Why - more spam's, more communications infrastructure utiliased the more money changing hands and...more tax's paid. There is the added bonus of not having to find out what people want and elling them. End result is a small database that fits on a PDA were FBA/NSA can instantly know if somebody is a terroist/pervert or simpsons fan. So for there faster database they generate more tax's, more comms infrastructure intilisation and a easier to manage personel details database. Lets not forget the growth in IT stocks thru the growing spam prevention markets. :>
Either that or thats a heck of an MP3 player the presedent has there.
A simple search shows the documentation on TMS' customers and the only thing this large that requires this size index is keyword searches of e-mail intercepts.
"Imagine a RAID5 of these!"
How long will it take until Dell starts shipping their standard PCs with that much RAM? Guesses?
Wow 2.5TB of RAM, they must be running the new Beta of Windows Longhorn!
All those moments will be lost in time, like tears in rain.