Slashdot Mirror
Internet Archive Gets 4.5PB Data Center Upgrade
Lucas123 writes "The Internet Archive, the non-profit organization that scrapes the Web every two months in order to archive web page images, just cut the ribbon on a new 4.5 petabyte data center housed in a metal shipping container that sits outside. The data center supports the Wayback Machine, the Web site that offers the public a view of the 151 billion Web page images collected since 1997. The new data center houses 63 Sun Fire servers, each with 48 1TB hard drives running in parallel to support both the web crawling application and the 200,000 visitors to the site each day."
I have no idea how much 4.5 PB is until it's given in units of Libraries of Congress.
Does lusting after all their space make me a peta-phile?
Life==Jeopardy. All the answers are right in front us - the hard part is coming up with the correct question.
so all one need to do to "own the internet" is to drive a big rig and ... lift the container off their parking lot?
I can now theoretically steal "the internet" with a flatbed truck and a lift. There's something to be said for conventional data centers: They're rather hard to load onto a truck and drive off with.
#fuckbeta #iamslashdot #dicemustdie
one would assume that something like this does regular off-site back-ups, which must add up to a hell of a-lot, could someone with experiance in such matters shed a little insight into the logistics of backing up such a vast system
floppy disks.
lots of floppy disks.
liqbase
Its like the two USB hard disks I use for backups. Pick up the container and swap it with the container from secure storage,
http://michaelsmith.id.au
It's simple, the backups are compressed -- they simply remove all those useless zeroes from the binary data.
There's no -1 for "I don't get it."
The Internet Archive also works with about 100 physical libraries around the world whose curators help guide deep Internet crawls. The Internet Archive's massive database is mirrored to the Bibliotheca Alexandrina, the new Library of Alexandria in Egypt, for disaster recovery purposes.
Not reliable enough.
I suggest that this important resource be backed up to punched cards.
This would also enable handy comparisons in units that us oldies understand, such as ELOCs
(Equivalent Library of Congress).
I'd calculate it myself, but seem to have mislaid my slide rule...
I would love to revisit the web as it appeared when I first discovered it (1994 at psu.edu).
No, you wouldn't.
The entire internet prior to 1996 is archived on an old PC that I'm currently trying to get the 5GB disk restored on.. why I've kept all that old porn for so long completely escapes me tho. :)
They'd better have it backed-up. Last time the Alexandria library burned-down, we lost about one thousands years of collected information from ancient Greece and Rome. Ooopsie.
"I disapprove of what you say, but I will defend to the death your right to say it." - historian Evelyn Beatrice Hall
They have Charlie Babbitt on their staff. No need to replicate.
If they need much more, I have some AOL disks laying around that they can use.
Don't fight for your country, if your country does not fight for you.
I'd suggest also using stone slabs. Water can do serious damage to paper, and don't get me started on fire hazards. Good old Stone Slabs resist both of those really well. I'm not sure what the write speed is, however, so you'll probably need to hire many stonecutters to work in parallel.
You can ship 4.5 petabytes over a single OC-192 link in about 71 days.
yeah, but just at the 70th day, someone will pick up the phone and the whole thing will have to be resent.
--
"It is now safe to switch off your computer."
It's simple, the backups are compressed -- they simply remove all those useless zeroes from the binary data.
in music today, there is a so-called 'loudness war' and I think I've discovered what it is: they're removing the zeroes, thinking that 'all ones' will make the music even louder!
I wonder if its reversable? where do the zeroes go? can they be unzeroed? we should try to find them.
--
"It is now safe to switch off your computer."
They're keeping the offsite backup distributed around the Internet, using the World-Wide Web to store it in real time.
Part of it may even be on *your* machine! We've really got to stop Brewster from leaching all your storage and make him store his backup himself - this business of using the originals to back up the backup just isn't sustainable!
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Can you say, Parallelism?
Parallelogram.... crap
Parallellellell... dammit
Parapalouza... >
Why did you have to point that out to everyone? :(
It's simple, the backups are compressed -- they simply remove all those useless zeroes from the binary data.
Compressed with XML! Because XML makes everything better... right?
Right?
Homonyms are fun!
You're driving your car, but they're riding their bikes there.
I'd suggest also using stone slabs. Water can do serious damage to paper, and don't get me started on fire hazards. Good old Stone Slabs resist both of those really well. I'm not sure what the write speed is, however, so you'll probably need to hire many stonecutters to work in parallel.
A math problem. My favorite. I don't know much about stone cutters but lets assume they can write one bit every 2 seconds. Thats 1 byte in 16 seconds. The internet archive is (4.5 x 1,125,899,906,842,624) 5,066,549,580,791,808 (5 quadrillion) bytes. That works out to 81,064,793,292,668,928 (81 quadrillion) seconds or about 2,570,547,732 (2.5 billion) years. That is far to long for their stringent 2 month backup cycle. They would need 15,423,286,395 (15.4 billion) stone cutters to keep schedule assuming they had unlimited stone. Last time I checked there were only between 6 and 7 billion people with only a small fraction of them being stone cutters. That leaves but one solution. Force the web developers to become stone cutters. This would not only increase the work force but also reduce the amount needed to backup because fewer people will be making more web pages to backup.
"XML is like violence. If it doesn't solve your problem, you're not using enough of it."