Slashdot Mirror
Internet Archive Gets 4.5PB Data Center Upgrade
Lucas123 writes "The Internet Archive, the non-profit organization that scrapes the Web every two months in order to archive web page images, just cut the ribbon on a new 4.5 petabyte data center housed in a metal shipping container that sits outside. The data center supports the Wayback Machine, the Web site that offers the public a view of the 151 billion Web page images collected since 1997. The new data center houses 63 Sun Fire servers, each with 48 1TB hard drives running in parallel to support both the web crawling application and the 200,000 visitors to the site each day."
TFA indicates that they have a mirror at the library of Alexandria. Unless things have changed since last I read about them, the mirroring is pretty much it. The Internet Archive does very impressive work; but they don't have that much money. No Real Big Serious Enterprise tape silos here.
1 library of congress = 10 terabytes
4.5 petabytes = 4608 terabytes
So, that's 460.8 LOCs.
83 terabyte in the LOC, so 4.5 petabytes == 54 Libraries of Congress
4.5 petabytes == 4500 terabyte hard drives, times $75 each == ~$340,000 == how much taxpayers spend, each hour, to maintain the LOC
"I disapprove of what you say, but I will defend to the death your right to say it." - historian Evelyn Beatrice Hall
Incidentally: FileFront is closing in five days, taking with it any files that aren't hosted elsewhere.
I am told that many of the Half-Life mods hosted there are not available anywhere else, so get while the getting is good...
63 servers * 48 disk of 1 TB = 3024 TB. According to the announcement on the archive.org 3 Petabytes would be right.
The new data center houses 63 Sun Fire servers
That's not very specific. "Sun Fire" is a brand that for a while got applied to all of Sun's rack-mount servers (except for NEBS-compliant servers, which were and are called "Sun Netra"). A little confusing, of course, which is why they've started calling new SPARC boxes "Sun SPARC Enterprise" to differentiate them from those mangy x64 "Sun Fire" systems. Except that there are still SPARC systems called "Sun Fire", so I guess the confusion factor didn't get any better...
Anyway, the specific server being used here is the Sun Firex X4500, a system with no less than 48 1 TB disks in a 4U space. Notice that this model is EOLed; presumably iarchive got a deal on some remaindered machines.
The shipping container is something we've seen before.
TFA says "...eight racks filled with 63 Sun Fire x4500 servers with dual- or quad-core x86 processors running Solaris 10 with ZFS. Each Sun server is combined with an array of 48 1TB hard drives." (emphasis mine)
I would guess this means there's a x4500 with 24TB in local disks, and 48TB in attached storage per machine. (24+48)*63 does give us the quoted number
Blessed are the pessimists, for they have made backups.