Slashdot Mirror

← Back to Stories (view on slashdot.org)

Internet Archive Gets 4.5PB Data Center Upgrade

Posted by timothy on from the shipping-containers-are-rad dept.

Lucas123 writes "The Internet Archive, the non-profit organization that scrapes the Web every two months in order to archive web page images, just cut the ribbon on a new 4.5 petabyte data center housed in a metal shipping container that sits outside. The data center supports the Wayback Machine, the Web site that offers the public a view of the 151 billion Web page images collected since 1997. The new data center houses 63 Sun Fire servers, each with 48 1TB hard drives running in parallel to support both the web crawling application and the 200,000 visitors to the site each day."

14 of 235 comments (clear)

  1. Where do they store 4.5TB off site by wjh31 · · Score: 5, Interesting

    one would assume that something like this does regular off-site back-ups, which must add up to a hell of a-lot, could someone with experiance in such matters shed a little insight into the logistics of backing up such a vast system

    1. Re:Where do they store 4.5TB off site by LiquidCoooled · · Score: 5, Funny

      one would assume that something like this does regular off-site back-ups, which must add up to a hell of a-lot, could someone with experiance in such matters shed a little insight into the logistics of backing up such a vast system

      floppy disks.
      lots of floppy disks.

      --
      liqbase :: faster than paper
    2. Re:Where do they store 4.5TB off site by commodore64_love · · Score: 5, Funny

      They'd better have it backed-up. Last time the Alexandria library burned-down, we lost about one thousands years of collected information from ancient Greece and Rome. Ooopsie.

      --
      "I disapprove of what you say, but I will defend to the death your right to say it." - historian Evelyn Beatrice Hall
    3. Re:Where do they store 4.5TB off site by Anonymous Coward · · Score: 5, Funny

      Can you say, Parallelism?

      Parallelogram.... crap
      Parallellellell... dammit
      Parapalouza... >

      Why did you have to point that out to everyone? :(

    4. Re:Where do they store 4.5TB off site by zach297 · · Score: 5, Funny

      I'd suggest also using stone slabs. Water can do serious damage to paper, and don't get me started on fire hazards. Good old Stone Slabs resist both of those really well. I'm not sure what the write speed is, however, so you'll probably need to hire many stonecutters to work in parallel.

      A math problem. My favorite. I don't know much about stone cutters but lets assume they can write one bit every 2 seconds. Thats 1 byte in 16 seconds. The internet archive is (4.5 x 1,125,899,906,842,624) 5,066,549,580,791,808 (5 quadrillion) bytes. That works out to 81,064,793,292,668,928 (81 quadrillion) seconds or about 2,570,547,732 (2.5 billion) years. That is far to long for their stringent 2 month backup cycle. They would need 15,423,286,395 (15.4 billion) stone cutters to keep schedule assuming they had unlimited stone. Last time I checked there were only between 6 and 7 billion people with only a small fraction of them being stone cutters. That leaves but one solution. Force the web developers to become stone cutters. This would not only increase the work force but also reduce the amount needed to backup because fewer people will be making more web pages to backup.

  2. Story is meaningless without LOC measurement by Dr_Banzai · · Score: 5, Funny

    I have no idea how much 4.5 PB is until it's given in units of Libraries of Congress.

    1. Re:Story is meaningless without LOC measurement by Anonymous Coward · · Score: 5, Informative

      1 library of congress = 10 terabytes
      4.5 petabytes = 4608 terabytes
      So, that's 460.8 LOCs.

    2. Re:Story is meaningless without LOC measurement by Wingman+5 · · Score: 5, Interesting

      from http://www.lesk.com/mlesk/ksg97/ksg.html The 20-terabyte size of the Library of Congress is widely quoted and as far as I know is derived by assuming that LC has 20 million books and each requires 1 MB. Of course, LC has much other stuff besides printed text, and this other stuff would take much more space.

      1. Thirteen million photographs, even if compressed to a 1 MB JPG each, would be 13 terabytes.
      2. The 4 million maps in the Geography Division might scan to 200 TB.
      3. LC has over five hundred thousand movies; at 1 GB each they would be 500 terabytes (most are not full-length color features).
      4. Bulkiest might be the 3.5 million sound recordings, which at one audio CD each, would be almost 2,000 TB.

      This makes the total size of the Library perhaps about 3 petabytes (3,000 terabytes).

      so 230 libraries by the old standard or 1.5 by the new standard

  3. Storage Envy by jacksinn · · Score: 5, Funny

    Does lusting after all their space make me a peta-phile?

    --
    Life==Jeopardy. All the answers are right in front us - the hard part is coming up with the correct question.
  4. Own the internet! by Anonymous Coward · · Score: 5, Funny

    so all one need to do to "own the internet" is to drive a big rig and ... lift the container off their parking lot?

    1. Re:Own the internet! by peragrin · · Score: 5, Funny

      well if you plug in a laser printer you can print off a hard copy for your boss.

      --
      i thought once I was found, but it was only a dream.
  5. Slight problem? by girlintraining · · Score: 5, Funny

    I can now theoretically steal "the internet" with a flatbed truck and a lift. There's something to be said for conventional data centers: They're rather hard to load onto a truck and drive off with.

    --
    #fuckbeta #iamslashdot #dicemustdie
  6. In Other News by Erik+Fish · · Score: 5, Informative

    Incidentally: FileFront is closing in five days, taking with it any files that aren't hosted elsewhere.

    I am told that many of the Half-Life mods hosted there are not available anywhere else, so get while the getting is good...

  7. Re:You can ship it over OC-192... by TheGratefulNet · · Score: 5, Funny

    You can ship 4.5 petabytes over a single OC-192 link in about 71 days.

    yeah, but just at the 70th day, someone will pick up the phone and the whole thing will have to be resent.

    --

    --
    "It is now safe to switch off your computer."