Slashdot Mirror
US Government Upgrades RAM
Deep Throat writes "Techworld has the scoop on a new super-sized RAM disk that the US government has just bought for a few million dollars in order to speed up searching through huge databases. It's 2.5TB! The VP of the company that made it says it is for Washington DC and searching databases but won't say who. Techworld explains why it reckons it's the Department of Homeland Security searching in the NSA and Pentagon databases for terrorists. And apparently the government is 'very happy' with the purchase and thinking about getting more."
Imagine a beowulf cluster of those things.
Still not as fast as if they'd make the info public domain and use google :)
I remember when legal used to mean lawful, now it means some kind of loophole. - Leo Kessler
Usually the US Government rams upgrades. Iraq, Haiti, next?
in order to speed up searching through huge database
Have they consulted Google?
How are they going to save the data if the power goes out? That's a lot of data to store on hard copies.
I'd even like HALF of the size of that disk!
There's no place like localhost
Sounds like they're taking heed of Google's success in attaining blazing search speeds by holding all the data in RAM.
See here.
What happens when the power goes out? How long will it take to read all that data back into the RAM disk? Ouch.
Tinfoil hat mode ON! It's for the HAARP antennae, man, and earthquake machines. Or maybe it involves secret Area 51 UFOs! They are probably storing all our thoughts on it. Run for the hills.
The list price of the system, which is made up from 40 RamSan 320 units, reviewed here, is $4.7million
Purchased from Dell's website that would have been....$12.5 million?
They're upgrading in time for Longhorns release you insensitive clod!
I just hope they rememebr to use the USB2.0 for their new keychain disk; access time might be horrible otherwise...
My guess is that they want to play Doom3 with a fps higher than 5 fps.
Is Unreal Doomed
Wonder if they got hold of an early copy of Doom 3.
However, not that many departments could possibly want to run such vast queries regularly. It would also be extremely difficult to justify a $4.7 million investment unless that work was seen as vital and speed was a main consideration in that work. It is also peculiar that such a large purchase could be approved at a time of tightening belts.
Honestly, I wonder what the author was smoking?
* However, not that many departments could possibly want to run such vast queries regularly.
You don't think so? I think *every* DBA would like to have a few extra TB of RAM. Maybe the Department of Transportation just wants a more efficient way to keep track of US Highway routes?
* It would also be extremely difficult to justify a $4.7 million investment...
What country is this guy living in? If you're high up enough, it's trivially easy to justify $5 million. That's hardly enough to build one Interstate highway intersection.
* It is also peculiar that such a large purchase could be approved at a time of tightening belts.
Oh, now I know the problem. The author has been in a coma for the past 18 months. Wake up, dude, and smell the money!
Stressed? Me? Of course not. Stress is what a rubber band feels before it breaks, silly.
This was bought by the Internal Revenue Service in order to improve the auditing of tax returns. They say that the additional revenue brought in will easily pay for the device many times over.
If it was for the DHS or NSA you would not have heard about the purchase.
Before many users start discussing the privacy laws and what not, it should be noted that the data being stored is probably not new. It's the medium on which it is stored on, which is.
Even without this, the old database could have been searched for some terrorists. Nothing has really changed.
Could be memory for pattern matching genomic alignments : Health and Human Services.
Could be memory for modeling Nuclear Blasts : Department of Energy.
Could be everybody's tax returns since the Truman administration : Internal Revenue Service.
WTFK.
Did not see the reference to the actual size of it. I'm guessing a platter the size of Alcatraz (which is in between "as big as a house" and "the size of Pittsburg").
Just wait until AOL starts mailing these out.
Don't blame Durga. I voted for Centauri.
They really just needed it to beta test Longhorn.
"2.5 TB ought to be enough for anybody."
Heck, that might even be enough to boot Longhorn!
I want to delete my account but Slashdot doesn't allow it.
but does it really matter? they'd have spent the money on something anyway because they seem to think that it is theirs to spend and that they are some how above the citizenry that they were elected by and from. They might as well have spent it on something "cool." as for searching for terrorists and stuff, big bloody whoop. As long as they only index and infringe on non-citizens, i really don't care. of course, they are going to do that and it doesn't matter who is president because both major parties are in on the international totalitarian conspiricy.
No comment needed.
After weeks of seeing overblown headlines on slashdot (IE, Bill Gates runs over a squirel on the way to work, headline reads "Bill Gates Murders Animals as part of his Job!!!"), it's good to see one that is, well, underblown. 2.5TB is a bit more than a 'RAM upgrade'
that's about 8KB for every person in the US, including the 'terrorist' bit. You can put a lot of personal data in 8KB.
try { do() || do_not(); } catch (JediException err) { yoda(err); }
...exactly what they're going to use this for. Believe me, the US government isn't doing anything sinister now that there's a department of homeland security and we have the "post 9-11" buzzword that is any worse than what they were doing before. You just hadn't heard about it yet back then.
Let me tell you a little story.
January 2001. There are still two very tall buildings at the southern end of Manhattan. A certain three letter agency paid a *large* sum of money for me to signifigantly alter the virtual memory subsystem of the linux kernel for alpha processors. The validation testing and application of the modifications inolved benchmarks that compared large quantities of text against a large word list. The modification changed the amount of data they could pump through their (very large) cluster of machines by about 15%. Why would their motive today be any different? If this has been going on all this time, how can you say for sure what they're going to use 2.5TB of fast storage for?
Is Techworld a Drudge spinoff?
...right after their purchases of thousands of those little X10 spy cameras that you can mount just about anywhere! Oops, said too much, gotta go before-------------
My god carries a hammer. Your god died nailed to a tree. Any questions?
So I broke down and signed up for the OSDN Personals. I was thinking that it would be kinda cool to hook up with a chick that has a clue about what I am talking about. Boy, what a mistake.
Initially, it didn't seem too bad. There were a few of the expected email solicitations from the frighteningly ugly or grotesquely obese women. Then there were a couple for 40+ year old divorcees with several kids. I could fend these off easily but, I wasn't prepared for what happened next.
I started getting solicitations from gay Slashdotters. At first just a few but, the number kept growing until it finally became a torrent. They won't leave me alone. These homo wackos are stalking me hijacking my PC and my cell phone. I've completely abandoned email because of it but, they won't stop. I've even had a few track down my physical address and now they are stalking me.
These Slashdot homos are scaring the life out of me. I have called the police several times on them but, they keep on coming. Fat, pimply, pasty white, homosexual (or asexual) piles of filth that exude a stench that is offensive from 100 yards away. It is just unbelievable.
What ever you do, beware of OSDN personals and the Slashdot homos.
Washington DC -- Word has it that the United States Government has enlisted the help of numerous male sheep in Afghanistan in the hunt for Osama Bin Laden.
When technology advisors suggested the use of RAM to help with data searches, Atty. Gen. Ashcroft thought a similar technology could be used in the terrorist hunt immediately. Thus while the rams hunt down Bin Laden, RAM would help in bringing to light numerous new scapegoats...er...terrorists to hold blame for the administration's failures.
We'll ride the Terrorism Ticket in '04, adds Ashcroft. A vote for Bush is a vote against Terrorism.
From the article:
you need a monster RAM memory
I can see it now: SUNDAY! SUNDAY! SUNDAY! It's MONSTER RAM DISK vs. THE TERRORIST DATABASE! Get ready for a join table SMACKDOWN! Meet the sysadmin, get your picture taken with the Cisco router pinup girl of the month, and watch all the action. Kids under 12 get in free. SUNDAY SUNDAY SUNDAY!
How much access time are they saving? If they went with a really big cluster, they would also be adding compute resources, not just memory. What is the tradeoff (price/performance)?
You have no idea what it's for. The list of known terrorists and their acquaitances is relatively short, I cant imagine more than a few gigabytes being needed.
Perhaps it's to store tax returns so the government can mail you your refund check faster. (Job required, sorry).
Maybe INS (or USCIS or whatever they're called) want to track the tidal wave of benifits being handed to Mexican illegals.
I'm a little tired of all this Big Brother speculation. Get over it.
I don't need no instructions to know how to rock!!!!
select * from suspect_list where home_page = 'slashdot.org';
430,000 records found.
01000001 01011001 01000010 01000001 01000010 01010100 01010101
Shit, you beat me to it.
maybe I should contact my elected officials and ask them to upgrade my ram!
I just hope they didn't get it at Fry's. God help them if they've got to return it.
"Prepare for the worst - hope for the best."
In 15 years (or less) people will have this much data on their Palm Pilot.
oh man, and I thought I was the only one with a collection THAT big.
Phone police.
Speaking of such a project.
I'm searching for the cheapest way to get a 64-bit processor with 16GB of RAM.
There are some Opteron motherboards but you have to purchase very expensive 2GB DIMM modules with only 8 slots...
Anyone aware of a motherboard that has 16 slots and will take 1GB DIMM modules? Reliable?
For 8GB, I found the Sun Blade 1000's are pretty cheap to get on eBay. I got a 8GB dual 750Mhz system for $4500 used. However, it is maxed out at 8GB...
I figure someone in this thread would have something to share. Thanks.
Sure, it could be used for good. But you know and I know that it will eventually be used for evil.
Folks, I am asking for your full and complete coorperation as we choke and squeeze our way into the 21st century.
Maybe NSA wants to build a searchable database of SHA1 preimages. Maybe FBI wants to crack domestic terrorists' GPG keys (because, thanks to export restrictions, *no* international terrorist could ever have access to strong encryption, right guys?) Maybe NTSB wants to sort everyone's mother's maiden names alphabetically and cross-index with blood type in order to make a pretty picture.
There are a million uses for this kind of lightning-fast random access storage, so it's useless to speculate. Whichever government agency is using it, one thing's certain: they're not using it for our benefit.
I pay more than enough taxes, I think - where's my giant honking RAM disk?? Of course... didn't somebody build a supercomputer out of Xserves for $5 mil? Boy, five million dollars isn't what it used to be... why, I remember when you could make a payphone call for only 35 cents!
[Z?]
The article says thats "10,000 times the RAM in your PC". No, thats only 2,500 times the RAM in my PC....
This cautionary tale is a USENET fable:
VAXen, My Children, Just Don't Belong In Some Places
Usenet Apocrypha
VAXen, my children, just don't belong some places. In my business, I am frequently called by small sites and startups having VAX problems. So when a friend of mine in an Extremely Large Financial Institution (ELFI) called me one day to ask for help, I was intrigued because this outfit is a really major VAX user--they have several large herds of VAXen--and plenty of sharp VAXherds to take care of them.
So I went to see what sort of an ELFI mess they had gotten into. It seems they had shoved a small 750 with two RA60's running a single application, PC style, into a data center with two IBM 3090's and just about all the rest of the disk drives in the world. The computer room was so big it had three street addresses. The operators had only IBM experience and, to quote my friend, they were having ``a little trouble adjusting to the VAX,'' were a bit hostile towards it and probably needed some help with system management. Hmmm, Hostility.... Sigh.
Well, I thought it was pretty ridiculous for an outfit with all that VAX muscle elsewhere to isolate a dinky old 750 in their Big Blue Country, and said so bluntly. But my friend patiently explained that although small, it was an ``extremely sensitive and confidential application.'' It seems that the 750 had originally been properly clustered with the rest of a herd and in the care of one of their best VAXherds. But the trouble started when the Chief User went to visit his computer and its VAXherd.
He came away visibly disturbed and immediately complained to the ELFI's Director of Data Processing that, ``There are some very strange people in there with the computers.'' Now since this user person was the Comptroller of this Extremely Large Financial Institution, the 750 had been promptly hustled over to the IBM data center which the Comptroller said, ``was a more suitable place.'' The people there wore shirts and ties and didn't wear head bands or cowboy hats.
So my friend introduced me to the Comptroller, who turned out to be five feet tall, 85 and a former gnome of Zurich. He had a young apprentice gnome who was about 65. The two gnomes interviewed me in whispers for about an hour before they decided my modes of dress and speech were suitable for managing their system and I got the assignment.
There was some confusion, understandably, when I explained that I would immediately establish a procedure for nightly backups. The senior gnome seemed to think I was going to put the computer in reverse, but the apprentice's son had an IBM PC and he quickly whispered that ``backup'' meant making a copy of a program borrowed from a friend and why was I doing that? Sigh.
I was shortly introduced to the manager of the IBM data center, who greeted me with joy and anything but hostility. And the operators really weren't hostile--it just seemed that way. It's like the driver of a Mack 18 wheeler, with a condo behind the cab, who was doing 75 when he ran over a moped doing it's best to get away at 45. He explained sadly, ``I really warn't mad at mopeds but to keep from runnin' over that'n, I'da had to slow down or change lanes!''
Now the only operation they had figured out how to do on the 750 was reboot it. This was their universal cure for any and all problems. After all it works on a PC, why not a VAX? Was there a difference? Sigh.
But I smiled and said, ``No sweat, I'll train you. The first command you learn is HELP'' and proceeded to type it in on the console terminal. So the data center manager, the shift supervisor and the eight day operators watched the LA100 buzz out the usual introductory text. When it finished they turned to me with expectant faces and I said in an avuncular manner, ``This is your most important command!''
The shift supervisor stepped forward and studied the text for about a minute. He then turned with a very puzzled expression on his face and asked, ``What do you use it
And here everyone thought 64 bit would only be used ONLY in virtual!! Wow! I wonder if it's running on "Big Iron", since T-Rex might be the only monster that could handle the load. (z990 from IBM [ os = z/OS]). They also are using FICON which seems kinda dated for the technology I'd like to see more on the switching capability. I wonder how many CPU's she's running?
Mod +5 Drunk
If you go to disk, just once, you need about 9ms just to get the disk heads in position. If you're reading a file system of complex database, you now have multiple disk seeks and reads. That adds up. Seeking in RAM is orders of magnitude faster. That's why all the good search engines keep *everything* in RAM all the time.
That is why Google has multiple copies of the entire web in memory.
-AS
This is what's going through the mind of someone at redmond right now...
"With that much ram, I won't have to worry about fixing the memory leaks!"
They're using it to store ripped DVDs! It's an outrage! They're stealing the bread from the mouths of Julia Roberts and Steven Seagal!
Could this be used with a large enough pre-computed table to crack encryption? Maybe the NSA is hoping to win RSA's next challenge.
</tinfoil>
Want to improve your Karma? Instead of "Post Anonymously", try the "Post Humously" option.
they just want to have more than 10 files open at once.
http://milkshake.dexy.org
HAHAHAHAHAHAHAHa. Thaat is the greatest thing ive seen all day
Sorry, I didn't realize you revered roblimo that much. If I had, maybe I wouldn't have chosen this username... NOT.
Have a look at my (robslimo) journal
Have a look at my comments as ratometer from back before I lost my password to that account.
Have a look at my comments as robslimo.
Trolling I do not. In fact, I believe I was the first to post to this article with the comparison to Google and included a relevent link.
--
Want to generate your own electricity? Discuss it! [fieldlines.com]
the new cool thing is 'YEAH!' 'OKAY' and 'WHAT?'
It's 2.5TB!
Sweet. This is a sure sign that Doom 3 is almost here...
Where does the school board find them and why do they keep sending them to ME?
Perhaps we should begin to consider the implications of this step in analytical ability. Every techie knows that the Government (in cooperation with major Corporations) has had the methods in place to track individuals... RFID tags, GPS locating (in automobiles as well as hand held units. You can tell your position, but the position is also transmitted back... Five Star anyone?), Cell Phone triangulating, and thanks to the Patriot Act, the Government now has legal access to the records of pretty much any transaction we make with bookstores, libraries, etc (and probably more places as well... and this isn't even taking into account information that they might be recieving in ways that we do not know of).
The thing that (we shall assume) they didn't have before was the ability to instantaneously cross check this information. (I assume this because... well... why would they have bought the drive otherwise?) Now that they can check such information so quickly, will we be brought into an era of "Total Information Awareness" as the government spoke of not too long ago? Does this smell the same as the Thought Police to anyone else?
To me, this presents at least the intent by the government to achieve total information awareness, if it doesn't actually achieve it. And the intent is bad enough. Perhaps they're not reading our minds, but the ability to monitor our actions in such vast varities of levels comes pretty close to doing so.
-Vendal Thornheart
Can you make a beowulf cluster with it?!
You can lead a horse to water, but you can't make it dissolve.
Yeah, I have one of these things too. It really speeds up my builds.
I don't need no stinkin' sig!
who tf is "roblimo" anyway? Why would that get my attention? More importantly why the hell am I even typing this?
I bet they are hooking it to one of their Cray's just to search their own DB.
Can you imagine the amount of signals traffic they intercept in a day? Then try to sort into files of what is coded in cyphers we have broken and what needs work. Then translate the decoded messages and forward them to the agency that needs them.
No wonder they have several Crays.
Professional Politicians are not the solution, they ARE the problem.
to reformat.
Why still haven't we heard any news inside the US about ECHELON?
*DrugCheese rants*
No, I can't say it.
Now checking 2,748,779,069.440 bytes of memory, please wait.....
How much you want to bet they have that ramdrive plugged into a big octopus with no UPS?
<flees with hands over his ears>
"Prepare for the worst - hope for the best."
I was seriously looking at this product as a solution for a really bad database performance issue for a major financial institution. Instead of redesigning the whole database (small, around 2-3TB), we were thinking about putting a couple of these in to replace the slow EMC 8830/DMX infrastructure. Unfortunately, the PCI busses of the DB server weren't fast enough to keep up with it (Unisys ES7000 420). When you look at the TCO, the man years required to redesign the DB and applications were much more expensive than dropping in another few million in hardware. The only downside that I saw to the terraramsan solution was that it eats power and generates way too much heat. This thing would be great for horizontally scaled databses, as long as your physical plant could support it. Brute force always wins over recoding. That is, until we got offshored by Indians billing at $30/hour.
goddammit, what sci-fi movie/story does that come from???? you've got me going crazy over here
Imagine a Beowulf cluster of these things!
There... someone had to say it.
Wouldn't the money be better spent on regular commodity DRAM in servers? I mean, you can buy a lot more DRAM for a lot less money--and just use it as cache or hold an entire 2.5TB database in memory in a cluster.... I dunno, just seems like a waste.
But that's what government is good at - don't find a smarter way to run your database (so it's ultimately faster) - just throw it harder by applying more cash.
Look at the size of that thing!
When I worked for the Inland Revenue in the UK, we hooked into a national database of personal details available to a wide variety of government departments. We used Telnet clients via an intranet, and although I forget the precise specs of the central server, it sure as hell didn't need a RAM disk that big.
We never had any problem waiting for results, either. On a really bad day, you'd get maybe a 1s lag between hitting enter and a results screen coming up.
So if the US.gov needs a RAM disk that big, it's one fricking huge database. I have to wonder what sort of info it carries (part of the size might be due to things like photos, fingerprints, criminal records - stuff our DB didn't use), and how many people are on there (100% of the population?).
The scary thing: what if 2.5TB is a fraction of the database size - say, 25 to 50%? You'd still get reasonable performance, but the idea of a government holding 5-10TB of personal data seems positively Orwellian. "Big Bush is watching you"?
Disclaimer: I am not a techie, a lawyer or a government analyst. And it's only 5 years since I junked my Amiga 500, which did perfectly well with a mere 1MB of RAM, so maybe I'm used to thinking on a different scale. If you feel I'm wrong in any way, please feel free to correct me - I actively appreciate it!
"It is dark. You are likely to be eaten by a grue." -- Zork
Or else the Ministry of Information may just be looking for a Mr. Buttle rather than the Mr. Tuttle they really want to find.
Why did this get modded as 'Insightful'? It should have been modded down as 'Troll'.
Read my keyboard review.
The US installed Duvallier. Then they installed Aristide. Twice! Then when he asked for help again, they forcibly removed him and are now supporting the rebels.
Here
Here
Here
Here
Here
Get a clue. Read a newspaper. Turn on the television news. Pull your head out of your ass.
running memtest on this.
Let's just hope they have a really, really good UPS; saving that thing off to conventional media would take a few hours.
Next on CNN: Terrorists who target the power infrastructure of command-and-control centres! Pictures at eleven!
- undoware.ca
It reminds me of Platypus Technologies Qikdrive. You use it as another hard drive on your system for either database storage or caching or transaction files. Access time is much quicker than normal either way.
Perhaps someone should clue them in on how to create an index in SQL.
Nice, now half of Bush's porn collection will be instantly viewable. Next up you'll see a purchase for 1000 OLED displays to simulcast all his feeds on.
rdev -r /dev/fd0 2684354560
...1-800-MAGNETS reported a very large order from an undisclosed source for very high-powered magnets...
+1 Insightful, -1 Troll. What can I say, I'm an Insightful Troll.
Lets see.. 2500 GB of RAM at $150 a gig (retail).. that $375K ..
Sounds like the $150 hammer to me.
It's not having a collection that big that bothers me, its how fast they want to access it!!
Now, are they trying to search to see if they already have a new item or do the need the speed for serving it?
Where can I buy a *small* ramdisk, on the cheap?
For example, I have about ten 128mb PC100 DIMMs lying around. I'd love to stick them on some kind of PCI card to make a ramdisk, but I have no idea where to go about getting such hardware... google's results are useless, they're all links to *SOFTWARE* ramdisks that use main memory =(
The unofficial
The question is: How do we get someone close to it with a giant magnet?
You know darn well this is going to be used to search for info on everyone in the U.S., let alone the world. Isn't it great when technology and information come together...
(Sponsored by cheeseSource for President 2012)
Texas Memory Systems specifically.
try { do() || do_not(); } catch (JediException err) { yoda(err); }
Oh yeah? How do they know? Maybe I got 2.5TB of RAM in my PC too... Yeah.
Why not put the RAM directly into the database servers, rather than having them access it through a filesystem? Presumably large db systems have all kinds of clever mechanisms in place to make good use of RAM.
I'm not being funny, I'd genuinely like to know since it seems like a roundabout way of doing it.
Maybe they need the database to store all the passenger flight records submitted by the airlines that fly to the US.
Grundgesetz * 23. Mai 1949 - 30. November 2007 - http://www.vorratsdatenspeicherung.de/
It was supposedly used by the USDA at the time, Oracle people wondered why these farmers wanted such a huge DB ;)
2 years and no mod points. Join reddit. Because openness is good.
...and scores!
"Prepare for the worst - hope for the best."
I am uncertain why you suggest the parent read a newspaper or turn on the television news, considering your implication that it would prove you correct. Instead, it proves that your theory is a conspiracy denied countless times. The NY Times, Washington Post and other newspapers you're referring to have all consistently reported that this theory has been denied. Most cable news channels have had limited coverage altogether, reporting on the status of the Marines stationed in Haiti and noting only briefly Aristide's quaint claim that he was kidnapped, followed by a quotation of the Pentagon's denial and subsequent dismissal. I watch daily CNN, FNC and MSNBC, meaning those are the channels I refer to. Located here is an editorial which covers the majority of the material. For the record, both the Seattle Times and San Francisco Chronicle are respected in the industry, but the NY Times is included here as well.
Here (eatme123/eatme123)
Here
Here
Here
There are occasions in which skepticism is warranted, in which the government is lying to you in order to further its agenda. There may even be times in which the White House spokesman can casually lie to you without showing an iota of remorse, although that would be remarkable. This is not one of those times, as is obvious to those of us who are paying attention. Pull your head out of your ass.
Should have been posted under "Your Rights Online". I bet this RAM disk is being used for non-scientific purposes.
Engineering is the art of compromise.
SELECT * FROM tblNSA WHERE usLivingIn = "true" AND ethnicity = "arab"
SELECT * FROM tblCIA WHERE ss = "xxxxxxxxx" AND surname = "Kerry" AND dirt = "true"
SELECT * FROM tblFBI WHERE student = "true" AND politicalID = "left" ORDER BY antiwar
UPDATE tblTEXASAF SET duty_fulfilled = "true" WHERE ss = "xxxxxxxx" AND surname = "Bush"
they have a ram bigger than the hard dick.
ehhhhh.... Pr0n/Europe/Landing + correct title = close enough for slashdot.
15 years from now, people will be buying
these for $250 at CompUSA.
I'm somewhat worried about government snooping--and am therefore labelled as paranoid by my friends who are supporters of big government. But I can't imagine that my concern can be labelled as paranoia--I'm not spending all my money to find out what the government's doing--I'm spending it on taxes.
In 1995 you have roughly 58 million citizens in the UK. In 1995 the US had roughly 263 million citizens. Even small amounts of information would take more space than a UK database. Rough math says a UK database of this type would be 250gigs. Doesn't seem that big to me.
John Susek
According to http://www.siliconpenguin.com/Hardware/Solid_State _Disk/more2.html this ramdisk runs on an embedded Linux.
they could have just bought 6,250 Creative 4GB Muvo MP3 players @ $200 apiece, and removed the 4GB CFdisk. then it would have only cost them 1.25 million dollars.
The Gubment needs to save our money!
They're using their grammar skills there.
Even though Google runs a lot of queries, the key difference is that Google has a web crawler running 24x7. It causes them to update their database constantly that requires very different data structures and application support.
;-).
The government's project looks like a pure Data Warehousing to me. It means that their data set is pretty much static and serves for just retrieving data that will take shitload of time to load into this RAM (they should have some pretty heavy duty UPSs there
Tigers respect lions, elephants and hippos. Maggots respect no one. (C) S. Dovlatov
Woefully underestimating the cost.
Whe the replacement I880 stretch of freeway opened in Oakland California, to replace what collapsed during the '89 earthquake, the local news was reporting that it cost over $1000/inch to construct.
"Politicians are interested in people. Not that this is always a virtue. Fleas are interested in dogs." P.J. O'Rourke
Read the article- it refers to the 2.5 TB space being used as temporary query data storage for a 100TB+ databank.
What we call folk wisdom is often no more than a kind of expedient stupidity.-Edward Abbey
I havent seen any SQL query jokes in a while. Very nicely done! Haven't laughed that hard in a while.
You think we can get them to throw BitTorrent on it?
Was the Government upgrading it's RAM is just gonna hurt, really.
But with April 15 right around the corner, I'm almost used to getting rammed by them.
"Politicians are interested in people. Not that this is always a virtue. Fleas are interested in dogs." P.J. O'Rourke
Even 15 years later it's still damn funny.
Just to set the record straight, the original author of this post is Jack Harvey, and it was originally published under the title "The Immortal Murderer" on January 18th, 1989 on DECUServe, the DECUS member bulletin board.
This bulletin board is still active under the name Encompasserve.org after mergers of Digital Equipment Corporation and Compaq with Hewlett Packard.
The original publication can still be found on that bulletin board in the archived Soapbox conference, note number 168.
For those of you who were not born, Monday 19-Oct-1987 was the day the stock market crashed.
Echelon
Sorry, but your scale is a bit off. 5-10TB isn't large these days. AT&T's Data warehouse is 95TB. Amazon's is 35TB. Heck, there's an 8TB database running on Windows. http://www.wintercorp.com/vldb/2003_TopTen_Survey/ All%20Winners.pdf
This database is huge - 2.5 TB is a LOT of memory. How big is it, you ask?
Well, to put in terms the average Slashdotter can understand: If the database were measured with VW Beetles, it would weigh in at a whopping 128 VW Beetles, assuming each Beetle came with a 20 GB iPod.
Look at Chile for a past example. Venezuela today too, although hard to say for certain.
$5 / month hosted VPS on linux = awesome!
if by terrorists they mean porn, than, yes, this purchase makes sense... it might be able to sift though [insert government official name here]'s porn collection!
what? half of [insert above government official name here]'s porn collection is 3 TB? quick get a few more of these racks!
Don't call me back. Give me a call back. Bye. So yeah. But bye our, well, but alright we are on a shirt this chill.
Ok, the ol' US of A isn't really cutting it in the world anymore after the profits generated rebuilding the world after WW2. We aren't the only show in town anymore, but most of us don't know it yet. Fat, dumb and happy, we are.
Technology is what drives our rise to fame, yet not only do we export the technology freely, we also discourage ( by lack of the promise of any sort of meaningful employment ) our youth from technology fields, choosing to use what others make rather than learning how to innovate on what we made before.
So, we now import damn near everything.
Gone are those days when kids would tear a disposed TV apart and re-use its components to build all sorts of gadgets like I did.
Note the latest wealth-building books all speak of "earned income" as the worst kind of income - taxed to the hilt. What they tout is "passive income", that is making money from appreciation of your investments. Most tout real estate for this. So the price of a home is rising much faster than the wages ( driven by outsource competition ).
In order to "help" the buyers come up with more money, the government drops the federal funds rates. Sure, now the market is flooded with cheap dollars. Lots and lots of borrowed dollars are now competing with earned dollars in the market. Prices go up. Merchants rake in increased dollar sales. The wealthy snap up yet more and more properties. The new incoming generation finds themselves trapped between competing with third world countries labor rates and the price of a home.
So, economic costs force them to delay having kids until they can at least hold down a place to live. Back to the "Rich-Dad, Poor-Dad" syndrome, these kids will soon find they can not work for any sort of corporation who not only will force them to compete internationally, they will also force them to render their SSN, so the corporation can gladly give any earnings info to the gov't so the gov't can tax it back.
I think the writing is on the wall that the next generation - if they are gonna survive - is going to have to work for themselves. That is - find something there is a market for and do it themselves. Things like auto repair come to mind. Its something that one - armed with enough tools, could do at home without involving anyone else. A job paid for in cash. Cash unseen by the monitoring systems currently in place. This way, costs to the consumer can still be held to a low rate so the rich can have their toys maintained. But I think it will become standard knowledge that if you ask for anyone's SSN, you expect to pay about quadruple the rate so your servant can pay the tax on the money you are about to give him.
Its kinda obvious things are heading for a tumble. I think the government knows it. I think they know they better get a system in place to control the people while the going is good. They wanna make sure that any revolt among the wage-slaves is controllable by knowing exactly who is involved, and economically ruining the dissident without taking out the rest of the wage-slaves, whose labor supports the lifestyles of the owning classes.
To me, all this 9-11 stuff did was give our government all the justification it needed to install all sorts of snooping technology aimed at ourselves. I am sure all these fancy databases will be mostly used for taxation enforcement.
I do not like what I see in the future. If we pee-ons don't get busy at the ballot box, and hold politicians accountable, we are gonna be in for a world of hurt.
Example: When Congress is presented with the H1B Visa program showing how many jobs, they should simultaneously consider the equal amount of Americans who won't have jobs, look at their salary, how much taxes they would have paid, and at the same time as they sign the bill, they would be signing immediate budget cuts to reflect the lost tax revenues, and also support costs for the unemployed workforce resulting from their signature.
So, right now, the government is putting the machinery in place for enforce compliance while the getting is good. Can't blame them for that.
( taking off paranoia hat... )
"Prove all things; hold fast that which is good." [KJV: I Thessalonians 5:21]
Where Aristide rigged the last election, and suppressed dissidents, thus setting himself up like most of the leaders in Haiti's history. Bush was kind of engough to get Aristide rear end out of the country before it got shot off.
Haiti should just be turned over the UN to see if they really can save some small country from itself.
Rumor has it that Adolf of WW-II infamy managed a lot of his damage with records stored in shoe-boxes.
It is scary even imagine what they could do with that. Do all the posts regarding privacy come to mind?
Once, I had an argument with a buddy of mine that spoke his mind--to much--over Email. I gave him a little grep script to show how email monitoring could select "suspicious material" for further analysis.
Now, all they have to do is tie it in to a profiling system, and there you go. Orwell's 1984^10 all over again:
FROM ORDER BYSELECT TOP 100
I keep it spread across the entire Internet so no one will know it's mine though.
Given the new cap limits and surcharges alot of american broadband connections have impossed and increased average speed of connections. It would make only perfect sence to have the fastest spamming maching possible. Why - more spam's, more communications infrastructure utiliased the more money changing hands and...more tax's paid. There is the added bonus of not having to find out what people want and elling them. End result is a small database that fits on a PDA were FBA/NSA can instantly know if somebody is a terroist/pervert or simpsons fan. So for there faster database they generate more tax's, more comms infrastructure intilisation and a easier to manage personel details database. Lets not forget the growth in IT stocks thru the growing spam prevention markets. :>
Either that or thats a heck of an MP3 player the presedent has there.
This is to find the Iraqi WMD's, of course. And also, to find the traitor who outed Valerie Plame.
On a different note, OJ has ordered one of these for himself, to help with the search for the real killers.
Mr Shapiro
So let me get this streight... you used a telent client to access a goverment database? lol.
this is my sig, there are many like it, but this one is mine.
1: Insert USB Keydrive into USB port
2: Watch as the drive snaps cleanly in two as all the weight of the drive is centered upon the metal USB drive's neck.
3: Buy a new 2.5 TB USB drive.
4: Goto step 1!
0- Eamonman Proud member of DNRC
That must be really a very large number of these terrorists out there. I hope your government will quicksort them soon.
There you are, staring at me again.
I don't know if you remember there were a lot of folks in the industry that had very mixed feelings about Pointdexters total information awareness initiative. While they didn't like the big brother aspects of it, many people said that the types of software and systems that were proposed had as much potential to change the world of computing as the development of the internet.
I'm pretty sure that they are drawing on the best minds that they can find and I wouldn't be surprised if some of those folks worked at Google.
Even though Poindexter's TIA program was canned most of the projects have lived on under other names. I'm sure that it will be a while but it will be interesting to see what kinds of information technologies get spun off from this.
Anybody want to calculate the error rate on that much RAM?
I hate to think that the only thing seperating me from Honest_Patriotic_Citizen=1 is a stray cosmic ray!
I think IBM announced a box that supports 1 TB of RAM.
A simple search shows the documentation on TMS' customers and the only thing this large that requires this size index is keyword searches of e-mail intercepts.
There's another use for a TB DB. Movies, like what Pixar does. Scientific research is another (Physics). Oil and gas exploration. Large systems modeling (modelling the effects of GW's tax cuts).
The precomputation system you are talking about are called Rainbow Tables (http://lasecpc13.epfl.ch/ntcrack/faq.php). Let's you store password/hash combinations much more effectively than simply storing them in text. Well the thing is, this quickly grows to the point of impossible, both in storage used and time taken to compute.
For example: Let's take Windows LM passwords. It is a shitty hash, which we can calculate very quickly. What's more, the password can never be over 7 characters long (LM uses 2 7 character hases for a max password length of 14 characters). Also, LM converts every alpha character to upper case.
So suppose we want a table for just the alpha space. 26 possibilities per character, 1-7 characters. That'll need 610MB of disk space and take about 2.5 days on a 700mhz computer. That's perfectly feasable for a normal user.
But what if we want alpha numerics? That's not 36 possibilities. Just doing that, we raise the amount of disk space you need to 3GB and the amount of time to 15 days.
And then let's say we want to crack REAL passwords. Well cracking shitty LM hashes isn't going to cut it, most new OSes (Windows included) use something like MD5. MUCH slower to computer, and much more space to store (it's a longer hash). Of course these are also case sensetive, so we have a minimum of 62 characters just for alphanumberics. And many people (espically those we might be interested in) are getting wise and using puncutation. That's about 80 characters.
And now for the REAL fun: Each additonal character in the password make the space to search grow geometricaly. New systems aren't a 7+7 system, they are just a straight hash on the password. So you have to search whatever length you think you might encounter.
As you can see from the gain from going from alpha to alpha numeric with LM, it quickly gets to be an unsolvable (with current hardware) problem. You simply can't build an array big enough to hold it, and even if you could, you can't calculate it in the next 10,000 years.
Oh, and that's not even to mention things like random salts that many UNIXes use.
Maybe someday, but at this point, it's just not happening, even with redicilous amounts of hardware. You could build a table to crack weak passwords, but we've been able to do that in realtime for years.
Check out a used HP dealer for a J6000. It runs HP/UX, will take 16GB ram, should be reasonably priced, and will be about as fast as your Sun.
;))
Another possibility is to look at Suns new Ultrasparc IIIi systems like the V440, or Sun's new opteron systems. You can get a new system with 16GB of ram for a fairly reasonable price. (Reasonable for 64 bit systems with 16GB ram, that is.
Thanks AC for the lead. Looks like Crucial.com sells 1GB modules for $455.99 and it takes them in pairs up to 16GB.
I'm looking to use these with MaxDB and HP is one of the supported platforms.
Get a recording of the congressional inquiries earlier this week. In particular, listen to Charlie Rangel's questions. They make the situation pretty clear.
I've had this sig for three days.
"Imagine a RAID5 of these!"
This little disk would be really convenient for watching pr0n, er, I mean, for efficient statutory law research. Yeah, that's it.
Have they consulted Google?
Does having a former NSA employee with security clearance count? I'm surprised no one else mentioned it.
I just thought of another interesting point too... if it is being used as a cache instead of direct storage, there's a high potential for it being used for thousands to tens of thousands of cross checks at a time... just hold the individual cross-check data in different sections of the Solid State drive. That could be scary, as it could potentially mean the scanning of everyone in, say, an entire city, all at the same time. Perhaps even larger than a city-wide radius worth of people, it depends on how much data each individual cross-check would need. But imagine if that was possible - the Government could be scanning entire cities at a time, looking for people who show signs of... well... whatever they want to look for at the time, and in mere moments. If I may make another popular culture analogy, I'll point at the lidless, flame-wreathed Eye of Mordor.
-Vendal Thornheart
How long will it take until Dell starts shipping their standard PCs with that much RAM? Guesses?
imagine the wicked surcharge the Canada Copyright Board would charge for 2.5TB...
eeek...
my favorite part of the article is...
"It would also be extremely difficult to justify a $4.7 million investment unless that work was seen as vital"
apparently the person who wrote the article doesn't know much about how the government works. they burn money like you wouldn't believe! a few million dollars isn't shit to the gov! trust me, i'm a civil servant!!!
If you're interested in such things, look for contract announcements from the "Maryland Procurement Agency", conveniently located at Fort Meade, Maryland.
Mea navis aericumbens anguillis abundat
"At 2.5TB, it is roughly 10,000 times the size of the RAM in your PC."
...
...
Thanks, steve jobs . honestly ...
not really
for me that's more like 1000 times
You're welcome.
One word of advice re: HP/UX - Make sure you are on HP/UX 11.X and be careful with your patch sets. Last time I had to deal with HP/UX (3 yrs ago) 10.20 was ugly if it didn't have a carefully selected set of patches. I expect that 11.X is better.
Good luck.
Yep, via departmental intranet. ;-)
AFAIK, said DB is not connected at all to the internet, so it's highly unlikely such a thing could be accessed, even if you were the world's best Black Hat. I'm not even close to being a "leet" hacker...
Then again, what do I know about hacking? Last computer I tried to hack called me Dr. Falken and asked if I'd like to play a game
"It is dark. You are likely to be eaten by a grue." -- Zork
Dude why did they flag you as troll. Like some sort of coverup conspiracy goin' on here or what??
Of course it's for Echelon what else would it be for!! No terrorist database needs that kind of processing...
I think $28,000 is cheaper than "a few million dollars" (source: story). In fact, even ten of them would be cheaper. In fact, even forty. Or probably even a hundred.
Wow 2.5TB of RAM, they must be running the new Beta of Windows Longhorn!
All those moments will be lost in time, like tears in rain.
...a Japanese one. The article says:
The list price of the system, which is made up from 40 RamSan 320 units
Soon, Gremlins will crawl out of the units and eat the memory.
Perhaps the gov stored every possible key for various encryption codes and its just one big lookup table for fast code cracking.
Adolf didn't do it all by himself.
In fact, to effect real change, he looked beyond traditional stategic elements. With novel approaches to value-creation and increasingly involved transformation of organizations' non-core activities that focus on strategic capabilities he turned to IBM Holocaust Division for all his ethnic data processing (EDP) needs.
Are you dead? Hit on the head with a blunt object at birth? Who owns & controls those "news"papers you mention?
Sur ehope you don;t claim yourself to be a geek - you're too stupid. . . . "I read it in the papers so it must be true".
Look at some alternative sources. Are you aware that these newspapers all have people who write "reports" of events at locations to which they have never been, from the comfort of their hotel rooms?
"you're" is the word you are looking for; in post, after post, after post, you keep writing it as "your". Quick lesson
..
your : this is YOUR life to live. This is your computer
you're : you're going to visit him. You're right . .
If you're (see, that's how it's done) going to post opinion, do yourself and readers a favor and get basic grammar and spelling correct.
It's because Jeb Bush doesn't want to deal with the Haitian refugees that would appear on Florida's shores.
Debunking the "59 Deceits"
Here is a list of the "irregularities".
* In the last election of the parliment, 8 legislators were allowed to take seats on a plurality, rather than having the required runoff.
* When Aristide last ran, the opposition boycotted the election, largely because they knew that Aristide would get over 60% no matter who they ran.
You have to remember that the Neocons HATE Aristide. They prefer the more palely complected elites.
Add to that the fact that Aristide brought in Cuban doctors and nurses (our foreign aid never got a signature on the check), and he's the next Castro.
In a lot of the country, there are more Cuban than Hatian doctors.
You can figure that the first thing that will happen now that the marines are in is that the Cubans will be sent home, and the clinics closed.
After all, who cares that the "riff raff" (Bob "I'm a traitor" Novak's term) die?
The privilige of the pale elite of Haiti must be preserved.
Debunking the "59 Deceits"