Internet Archive Gets 4.5PB Data Center Upgrade

Lucas123 writes "The Internet Archive, the non-profit organization that scrapes the Web every two months in order to archive web page images, just cut the ribbon on a new 4.5 petabyte data center housed in a metal shipping container that sits outside. The data center supports the Wayback Machine, the Web site that offers the public a view of the 151 billion Web page images collected since 1997. The new data center houses 63 Sun Fire servers, each with 48 1TB hard drives running in parallel to support both the web crawling application and the 200,000 visitors to the site each day."

Comments

Where do they store 4.5TB off site

[wjh31]

one would assume that something like this does regular off-site back-ups, which must add up to a hell of a-lot, could someone with experiance in such matters shed a little insight into the logistics of backing up such a vast system

Re:Where do they store 4.5TB off site

[fuzzyfuzzyfungus]

TFA indicates that they have a mirror at the library of Alexandria. Unless things have changed since last I read about them, the mirroring is pretty much it. The Internet Archive does very impressive work; but they don't have that much money. No Real Big Serious Enterprise tape silos here.

Re:Where do they store 4.5TB off site

[LiquidCoooled]

one would assume that something like this does regular off-site back-ups, which must add up to a hell of a-lot, could someone with experiance in such matters shed a little insight into the logistics of backing up such a vast system

floppy disks.
lots of floppy disks.

Re:Where do they store 4.5TB off site

[MichaelSmith]

Its like the two USB hard disks I use for backups. Pick up the container and swap it with the container from secure storage,

Re:Where do they store 4.5TB off site

[MrEricSir]

It's simple, the backups are compressed -- they simply remove all those useless zeroes from the binary data.

Re:Where do they store 4.5TB off site

[DigiShaman]

Umm, how many forklifts and 18 wheelers does it take to swap out 4.5 petabytes worth of data each day?

Re:Where do they store 4.5TB off site

[Bearhouse]

Not reliable enough.
I suggest that this important resource be backed up to punched cards.
This would also enable handy comparisons in units that us oldies understand, such as ELOCs
(Equivalent Library of Congress).
I'd calculate it myself, but seem to have mislaid my slide rule...

Re:Where do they store 4.5TB off site

[commodore64_love]

They'd better have it backed-up. Last time the Alexandria library burned-down, we lost about one thousands years of collected information from ancient Greece and Rome. Ooopsie.

Re:Where do they store 4.5TB off site

[houghi]

If they need much more, I have some AOL disks laying around that they can use.

Re:Where do they store 4.5TB off site

[__aasqbs9791]

I'd suggest also using stone slabs. Water can do serious damage to paper, and don't get me started on fire hazards. Good old Stone Slabs resist both of those really well. I'm not sure what the write speed is, however, so you'll probably need to hire many stonecutters to work in parallel.

Re:Where do they store 4.5TB off site

[TheGratefulNet]

It's simple, the backups are compressed -- they simply remove all those useless zeroes from the binary data.

in music today, there is a so-called 'loudness war' and I think I've discovered what it is: they're removing the zeroes, thinking that 'all ones' will make the music even louder!

I wonder if its reversable? where do the zeroes go? can they be unzeroed? we should try to find them.

Re:Where do they store 4.5TB off site

[pedrop357]

4.5TB isn't that bad. Heck, we have 1TB tapes right now. 5 of them can be carried in a small bag.

It's the 4.5PB that the Internet Archive could use that's hard to store offsite. 4500 1TB tapes can be pretty unruly.

Re:Where do they store 4.5TB off site

[medelliadegray]

i find it impressive they have all that hardware for a mere 200k users a day.

Re:Where do they store 4.5TB off site

[Anonymous Coward]

In Brewster Kahle's December 2007 TED talk he mentions a third mirror in the Netherlands.
http://www.ted.com/index.php/talks/brewster_kahle_builds_a_free_digital_library.html

As he puts it, the Archive is mirrored on 'a fault line, a flood plain, and in the Middle East'.

Funny thing is I can't find another reference to the Netherlands mirror. The Bibliotheca Alexandrina site mentions a plan to eventually have four sites (California, Alexandria, Europe, and Asia), but that's it. Anyone know what happened with the Netherlands site?

Re:Where do they store 4.5TB off site

[Anonymous Coward]

one would assume that something like this does regular off-site back-ups, which must add up to a hell of a-lot, could someone with experiance in such matters shed a little insight into the logistics of backing up such a vast system

Create snapshot of zpool (think LVM VG):
# zfs snapshot mydata@2009-03-24

Send snapshot to remote site:
# zfs send mydata@2009-03-24 | ssh remote "zfs recv mydata@2009-03-24"

Create a new snapshot the next day:
# zfs snapshot mydata@2009-03-25

Send only the incremental changes between the two:
# zfs send -i mydata@2009-03-24 mydata@2009-03-25 | ssh remote "zfs recv mydata@2009-03-25"

Now this looks a lot like rsync, but the difference is that rsync has to traverse the file system tree (directories and files), while ZFS only has to look at the 'birth time' (think ctime) of each block of data (not even the full file metadata) to see if it's newer than the first snap shot. If you're talking about tens (or hundreds) of thousands of directories, and an order of magnitude more files, that's a lot of overhead if nothing has changed. For 48 TB raw (what a Sun X4500 can have), ZFS can see nothing has changed in a few minutes.

Creation of snapshots is instantaneous and there is no overhead in them (except that the space from deleted files isn't reclaimed / reused). There are people who create them every five seconds, and sync with a remote server--so at most you would lose five seconds worth of data if your disk died.

All changes are also ACID, so if you start your send-recv, and the transmission dies part way through, the receiving end won't have a partial copy of the data latest snapshot--it's all or nothing of the last good change.

Re:Where do they store 4.5TB off site

[dziban303]

(The truth about AIG and Congress.) http://www.foxnews.com/video2/video08.html?maven_referralObject=3833532 - Watch now

I can't take anyone seriously who puts "truth" and a link to Fox news in the same signature.

Re:Where do they store 4.5TB off site

[Anonymous Coward]

Can you say, Parallelism?

Parallelogram.... crap
Parallellellell... dammit
Parapalouza... >

Why did you have to point that out to everyone? :(

Re:Where do they store 4.5TB off site

[notthepainter]

Sadly, even modern day archives get wrecked. See http://www.spiegel.de/international/germany/0,1518,611311,00.html

Re:Where do they store 4.5TB off site

[Samah]

It's simple, the backups are compressed -- they simply remove all those useless zeroes from the binary data.

Compressed with XML! Because XML makes everything better... right?
Right?

Re:Where do they store 4.5TB off site

[zach297]

I'd suggest also using stone slabs. Water can do serious damage to paper, and don't get me started on fire hazards. Good old Stone Slabs resist both of those really well. I'm not sure what the write speed is, however, so you'll probably need to hire many stonecutters to work in parallel.

A math problem. My favorite. I don't know much about stone cutters but lets assume they can write one bit every 2 seconds. Thats 1 byte in 16 seconds. The internet archive is (4.5 x 1,125,899,906,842,624) 5,066,549,580,791,808 (5 quadrillion) bytes. That works out to 81,064,793,292,668,928 (81 quadrillion) seconds or about 2,570,547,732 (2.5 billion) years. That is far to long for their stringent 2 month backup cycle. They would need 15,423,286,395 (15.4 billion) stone cutters to keep schedule assuming they had unlimited stone. Last time I checked there were only between 6 and 7 billion people with only a small fraction of them being stone cutters. That leaves but one solution. Force the web developers to become stone cutters. This would not only increase the work force but also reduce the amount needed to backup because fewer people will be making more web pages to backup.

Re:Where do they store 4.5TB off site

[Omniscient Lurker]

Instead of writing in binary you could write the data in a base-36 format and then convert back to binary. The stone cutters could then store more data per glyph increasing their write rate considerably (and decreasing read rate) by amounts I am unwilling to calculate.

Re:Where do they store 4.5TB off site

[Rural]

Their aim is to preserve the content found on the Web. They need the hardware for that. I assume they don't need much for the "serving users" part.

Re:Where do they store 4.5TB off site

[commodore64_love]

>>>I can't take anyone seriously who puts "truth" and a link to Fox news in the same signature.

Neither can I take seriously anyone who believes MSNBC or CNN are unbiased and/or better alternatives. Or is prejudiced (prejudges a report without ever watching it). For example I may think Rachel Maddow is a joke, but at least I listen to what she has to say before I laugh. And sometimes, she says something worthy of hearing... it's good to keep an open mind and listen to the opposition.

Re:Where do they store 4.5TB off site

[CopaceticOpus]

"XML is like violence. If it doesn't solve your problem, you're not using enough of it."

Story is meaningless without LOC measurement

[Dr_Banzai]

I have no idea how much 4.5 PB is until it's given in units of Libraries of Congress.

Re:Story is meaningless without LOC measurement

[Anonymous Coward]

1 library of congress = 10 terabytes
4.5 petabytes = 4608 terabytes
So, that's 460.8 LOCs.

Re:Story is meaningless without LOC measurement

[Wingman 5]

from http://www.lesk.com/mlesk/ksg97/ksg.html The 20-terabyte size of the Library of Congress is widely quoted and as far as I know is derived by assuming that LC has 20 million books and each requires 1 MB. Of course, LC has much other stuff besides printed text, and this other stuff would take much more space.

1. Thirteen million photographs, even if compressed to a 1 MB JPG each, would be 13 terabytes.
2. The 4 million maps in the Geography Division might scan to 200 TB.
3. LC has over five hundred thousand movies; at 1 GB each they would be 500 terabytes (most are not full-length color features).
4. Bulkiest might be the 3.5 million sound recordings, which at one audio CD each, would be almost 2,000 TB.

This makes the total size of the Library perhaps about 3 petabytes (3,000 terabytes).

so 230 libraries by the old standard or 1.5 by the new standard

Re:Story is meaningless without LOC measurement

[commodore64_love]

83 terabyte in the LOC, so 4.5 petabytes == 54 Libraries of Congress

4.5 petabytes == 4500 terabyte hard drives, times $75 each == ~$340,000 == how much taxpayers spend, each hour, to maintain the LOC

Re:Story is meaningless without LOC measurement

[dln385]

from http://www.lesk.com/mlesk/ksg97/ksg.html The 20-terabyte size of the Library of Congress is widely quoted and as far as I know is derived by assuming that LC has 20 million books and each requires 1 MB. Of course, LC has much other stuff besides printed text, and this other stuff would take much more space.

1. Thirteen million photographs, even if compressed to a 1 MB JPG each, would be 13 terabytes. 2. The 4 million maps in the Geography Division might scan to 200 TB. 3. LC has over five hundred thousand movies; at 1 GB each they would be 500 terabytes (most are not full-length color features). 4. Bulkiest might be the 3.5 million sound recordings, which at one audio CD each, would be almost 2,000 TB.

This makes the total size of the Library perhaps about 3 petabytes (3,000 terabytes).

so 230 libraries by the old standard or 1.5 by the new standard

Compress each audio file to a 5 MB MP3. That's 17.5 TB. Total size would be 750 terabytes.

So the data would be 6 LOC.

Re:Story is meaningless without LOC measurement

[merreborn]

Bulkiest might be the 3.5 million sound recordings, which at one audio CD each, would be almost 2,000 TB.

You compressed the video, and the photographs, but not the audio? And why do you need a full CD for every sound recording? Surely many of them are far shorter than a full CD?

Re:Story is meaningless without LOC measurement

[Xtravar]

The CDs are already in digital format, so compressing them is a cardinal sin.

The photos, movies, and maps are in analog format to start with, so we don't feel so bad using lossy compression. Image files are really big. I think the 1GB estimate per movie is pretty good, considering shorts, black and white, and the standard (or lower) definition of most of them. That would allow for a very high detail scan of the movie in something like MPEG4.

And, since they started in analog formats, there's no fair way to determine what resolution to scan them. I mean, even a million by a million pixels could not be a 'lossless' interpretation of a 1x1cm image, so you have to accept that any digital conversion will be lossy regardless of encoding.

At least that would be my rationale. Not that this question needed to be answered...

Re:Story is meaningless without LOC measurement

[Anonymous Coward]

460.8 Lines of Code? What's that supposed to be? Hello World in COBOL?

Storage Envy

[jacksinn]

Does lusting after all their space make me a peta-phile?

Own the internet!

[Anonymous Coward]

so all one need to do to "own the internet" is to drive a big rig and ... lift the container off their parking lot?

Re:Own the internet!

[peragrin]

well if you plug in a laser printer you can print off a hard copy for your boss.

Slight problem?

[girlintraining]

I can now theoretically steal "the internet" with a flatbed truck and a lift. There's something to be said for conventional data centers: They're rather hard to load onto a truck and drive off with.

Re:Slight problem?

[rackserverdeals]

Here's a video tour of one if you need it for reference.

Don't forget to turn off the water and unplug the ethernet cables. Just be very careful with the power cords.

Re:Slight problem?

[fightinfilipino]

so the Internet really is a big truck, hauling all of our lulz and our memes across the globe.

take THAT, Ted Stevens!

What about 1996 and earlier?

[commodore64_love]

Are there any resources the let us see websites from 1996, 95, 94, or 93? I would love to revisit the web as it appeared when I first discovered it (1994 at psu.edu).

Re:What about 1996 and earlier?

[Tumbleweed]

I would love to revisit the web as it appeared when I first discovered it (1994 at psu.edu).

No, you wouldn't.

Re:What about 1996 and earlier?

[Matheus]

The entire internet prior to 1996 is archived on an old PC that I'm currently trying to get the 5GB disk restored on.. why I've kept all that old porn for so long completely escapes me tho. :)

Re:What about 1996 and earlier?

[Profane MuthaFucka]

Because after 1996 women shaved all their hair off due to a mistaken belief that men prefer their women to look like little girls. We don't, we like the big bushes, and that is why you must save that porn for the good of mankind.

They store 4.5PB in Egypt!

[CannonballHead]

The Internet Archive also works with about 100 physical libraries around the world whose curators help guide deep Internet crawls. The Internet Archive's massive database is mirrored to the Bibliotheca Alexandrina, the new Library of Alexandria in Egypt, for disaster recovery purposes.

Re:They store 4.5PB in Egypt!

[Anonymous Coward]

Egypt could be a good choice. The area is fairly famous for reliable persistent storage. From papyrus scrolls to stone engravings, things tend to keep there better than most places. There really aren't many other geographical areas on earth that can claim the same kind of data retention rates over the time periods they've dealt with. Though despite their impeccable track record with avoiding hardware failures, they've done significantly worse when it comes to data loss due to theft and/or hackers/pirates.

The one curious part about that choice is that the library at Alexandria is the one notable case where mass amounts of data were irreparably lost. So it's odd that they'd choose to entrust their data to that specific institution. Perhaps they felt that since it's under new management, the previous problems will have been resolved.

However, had the choice been mine, I would have chosen to store my offsite data in Luxor. It's data retention was quite good, and included one data store that was preserved in its entirety for over 3000 years. As an added benefit, it seems that they've opened a second location that's significantly more convenient for the IA since there's no overseas transmission to worry about.

In Other News

[Erik Fish]

Incidentally: FileFront is closing in five days, taking with it any files that aren't hosted elsewhere.

I am told that many of the Half-Life mods hosted there are not available anywhere else, so get while the getting is good...

Never underestimate the bandwidth ...

[Ungrounded Lightning]

... of a 4.5 petabyte datacenter in a shipping container in transit.

63 x 48 = 3024Tb

[eotwawki]

So wehre does the 4.5PB come in to this?

Re:63 x 48 = 3024Tb

[spinkham]

TFA says "...eight racks filled with 63 Sun Fire x4500 servers with dual- or quad-core x86 processors running Solaris 10 with ZFS. Each Sun server is combined with an array of 48 1TB hard drives." (emphasis mine)

I would guess this means there's a x4500 with 24TB in local disks, and 48TB in attached storage per machine. (24+48)*63 does give us the quoted number

Re:63 x 48 = 3024Tb

[rackserverdeals]

Sun has more information and an Interactive tour of the Internet Archive modular data center on their site.

The total raw capacity of the container is 3 peta bytes. In reality it's going to be less than that. First, 2 disks are likely to be setup in a mirrored pool for the system disks. I believe the root pool only supports mirrors, not raidz. Not sure if this has changed.

That leaves you with 46 disks for data. Maybe they partitioned part of the root pool to include in the data pools, not sure, but zfs works better with whole disks.

In the interactive tour, they weren't clear on how they set up the pools.

Side note. Maybe I'm cynical, but if this was the other way around, with linux servers replacing sun/solaris servers that probably would have been the headline.

Pretty neat to find out that the internet archive is powered by Java too. The wayback machine is java as well as the crawlers.

You can ship it over OC-192...

[Ungrounded Lightning]

... one would assume that something like this does regular off-site back-ups, which must add up to a hell of a-lot,..

As I recall from one of Brewster's talks: Part of the idea was that you can install redundant copies of this data center around the world and keep 'em synced.

You can ship 4.5 petabytes over a single OC-192 link in about 71 days.

Re:You can ship it over OC-192...

[TheGratefulNet]

You can ship 4.5 petabytes over a single OC-192 link in about 71 days.

yeah, but just at the 70th day, someone will pick up the phone and the whole thing will have to be resent.

Re:You can ship it over OC-192...

[aaarrrgggh]

Or, you can ship the 40' containers in just under two weeks!

Math

[PowerKe]

63 servers * 48 disk of 1 TB = 3024 TB. According to the announcement on the archive.org 3 Petabytes would be right.

"Sun Fire"

[fm6]

The new data center houses 63 Sun Fire servers

That's not very specific. "Sun Fire" is a brand that for a while got applied to all of Sun's rack-mount servers (except for NEBS-compliant servers, which were and are called "Sun Netra"). A little confusing, of course, which is why they've started calling new SPARC boxes "Sun SPARC Enterprise" to differentiate them from those mangy x64 "Sun Fire" systems. Except that there are still SPARC systems called "Sun Fire", so I guess the confusion factor didn't get any better...

Anyway, the specific server being used here is the Sun Firex X4500, a system with no less than 48 1 TB disks in a 4U space. Notice that this model is EOLed; presumably iarchive got a deal on some remaindered machines.

The shipping container is something we've seen before.

Re:"Sun Fire"

[ximenes]

Since they're using one of Sun's modular datacenters that is actually on the Sun campus, I would imagine that they got some financial incentives / support from Sun for all of this.

The X4500 is EOL as you mention, although it was still sold a few months back. It lives on as the X4540, which really isn't that different; the main thing is it's moved to a newer Opteron processor type and is a fair bit cheaper. So they didn't really miss out on anything.

It's kind of interesting to me that they went this route, as opposed to a bunch of servers talking to a bunch of storage separately. This seems to be an exact use case for the X4500-type system, which as far as I'm aware is pretty unique.

Re:"Sun Fire"

[fm6]

This seems to be an exact use case for the X4500-type system, which as far as I'm aware is pretty unique.

Indeed. Sun is on a density kick. Check out the X4600, which does for processing power what the X4500 did for storage.

In both cases, there actually are competing products that are sort of the same. The most conspicuous difference is that the Sun versions cram the whole caboodle into 4 rack units per system, about half the space required by their competitors.

More absurdly-dense Sun products:

http://www.sun.com/servers/x64/x4240/
http://www.sun.com/servers/x64/x4140/

The point of these systems is that they take up less expensive rack space than equivalent competitors. They're also "greener": if you broke all that storage and computing power down into less dense systems, you'd need a lot more electricity to run them and keep them cool. That not only saves money, it gives the owner the ability to claim they're working on the carbon footprint.

Re:"Sun Fire"

[Anonymous Coward]

Anyway, the specific server being used here is the Sun Firex X4500 [sun.com], a system with no less than 48 1 TB disks in a 4U space. Notice that this model is EOLed; presumably iarchive got a deal on some remaindered machines.

There are newer X4540s which are mostly the same, but have newer CPUs, and can hold more memory (16 -> 64 GB).

The off-site backup IS the Internet.

[billstewart]

They're keeping the offsite backup distributed around the Internet, using the World-Wide Web to store it in real time.

Part of it may even be on *your* machine! We've really got to stop Brewster from leaching all your storage and make him store his backup himself - this business of using the originals to back up the backup just isn't sustainable!