Slashdot Mirror

← Back to Stories (view on slashdot.org)

The Internet Archive Has Saved Over 10,000,000,000,000,000 Bytes of the Web

Posted by Soulskill on from the or-one-kilo-library-of-congress dept.

An anonymous reader writes "Last night, the Internet Archive threw a party; hundreds of Internet Archive supporters, volunteers, and staff celebrated that the site had passed the 10,000,000,000,000,000 byte mark for archiving the Internet. As the non-profit digital library, known for its Wayback Machine service, points out, the organization has thus now saved 10 petabytes of cultural material." The announcement coincided with the release of an 80-terabyte dataset for researchers and, for the first time, the complete literature of a people: the Balinese.

32 of 135 comments (clear)

  1. Relevance of byte count by Anonymous Coward · · Score: 5, Funny

    How much of that is porn, I wonder.

    1. Re:Relevance of byte count by martin-boundary · · Score: 4, Funny

      If only one of those files is a MP3, the RIAA is going to have an orgasm.

    2. Re:Relevance of byte count by Xtifr · · Score: 4, Insightful

      They have over 1.5 million unique audio files in the Live Music Archive alone. I know because I helped them count. (That's unique files, not counting the duplicates in different formats.) If the RIAA has anything to say about it, they're serious slacking.

    3. Re:Relevance of byte count by Anonymous Coward · · Score: 2, Insightful

      Because eventually they WILL be accessable when copyright runs out. But if nobody other than the 'rightsholders' have copies, that wouldn't matter, they could trivially remaster them, then have copyright over the remasters for another century after destroying the originals so they could never get out.

    4. Re:Relevance of byte count by GofG · · Score: 5, Funny

      There is a torrent on thepiratebay of every single geocities site. It's an archive, but i've downloaded it. What was your site? I'll rar it up for you.

      --
      GFA/M/S d-- s: a--- C++++ UBL++$ P+ L+++ !E- W++ N+ !o K- w--- !O !M !V PS++ PE Y+ PGP+ t+++ 5- X+ R tv@ b++ DI++++ D+ G
    5. Re:Relevance of byte count by Xtifr · · Score: 2

      Probably. I found a copy of my first-ever homepage, which actually predated Geocities, and was probably even more useless than your average Geocities page. :)

    6. Re:Relevance of byte count by GofG · · Score: 5, Interesting

      No, go ahead and mod me down. Every time i post, I look at my user ID and think "GOD FUCKING DAMNIT IF I HAD WAITED LIKE TEN MINUTES I WOULD HAVE HAD A PALINDROME AUAUUUUUUGGGHHH"

      i deserve all the downmods i get, accidental or otherwise.

      --
      GFA/M/S d-- s: a--- C++++ UBL++$ P+ L+++ !E- W++ N+ !o K- w--- !O !M !V PS++ PE Y+ PGP+ t+++ 5- X+ R tv@ b++ DI++++ D+ G
    7. Re:Relevance of byte count by Raenex · · Score: 5, Funny

      when copyright runs out

      Thanks for the laugh.

    8. Re:Relevance of byte count by Mikkeles · · Score: 2

      They're exaggerating; I know there are only 256 bytes, so I think they're counting duplicates!

      --
      Great minds think alike; fools seldom differ.
  2. Balinese, huh? by Anonymous Coward · · Score: 2, Funny

    Well, I guess they didn't have time to write much, being busy dealing with Orcs and Balrogs.

    What about the Thorinim?

  3. Indeed! by Frosty+Piss · · Score: 3, Funny

    And nothing of value was saved...

    --
    If you want news from today, you have to come back tomorrow.
  4. Yes, but... by Lordfly · · Score: 4, Funny

    I need a car analogy about the Library of Congress before i can understand that number.

    --
    hookers and grits.
    1. Re:Yes, but... by MangoCats · · Score: 2

      It's like the Library of Congress stuffed floor to ceiling with Service Manuals?

    2. Re:Yes, but... by Squeeself · · Score: 4, Interesting

      I know this was in jest, but in this case, unlike so many other times this joke is made, it's slightly relevant. A quick Google turned up the following incomplete info http://www.quora.com/Library-of-Congress/How-much-data-does-the-library-of-congress-actually-represent which states tape storage capacity of the Library of Congress circa 2011 at 4.5 petabytes. The answer, then, is the this is approximately ~2 Library of Congresses of data, which is just a tad bit much to fit in the trunk of your car. It's going to take a few trips to the Library and back to move that data around.

    3. Re:Yes, but... by deblau · · Score: 2

      If you live in Vancouver, it's roughly the number of nanometers you would cover on a round trip drive to the Library of Congress.

      --
      This post expresses my opinion, not that of my employer. And yes, IAAL.
  5. Indispensable reference for slashdotters by guttentag · · Score: 5, Insightful
    For instance, note the archived film "Dating: Do's and Don'ts" (1949) It begins thus:

    How do you choose a date? Whose company would you enjoy?

    Well, one thing you can consider is looks. Woody thought of Janice and how good looking she was. He'd really have to rate to date her. Yes, he'd enjoy that, except... Well, it's too bad Janice always acts so superior. She'd make a fellow feel awkward and bored.

    Well, perhaps someone who doesn't feel so superior. There's Betty. And yet, it just doesn't seem as if she'd be much fun.

    What about Anne? She knows how to have a good time, and how to make the fellow with her relax, too. Yes, that's what a boy likes.

    Yes, the Internet now provides everything you ever needed to know but were afraid to ask.

  6. All of which is rather useless... by pongo000 · · Score: 4, Interesting

    ...since the TOS specifically prohibits copying data from the site:

    "Our terms of use specify that users of the Wayback Machine are not to copy data from the collection. If there are special circumstances that you think the Archive should consider, please contact info at archive dot org. "

    Warrick hasn't been taking new requests for months (and I'm sure it's more of a research tool than an actual service for the public), and the site effectively blocks attempts to backup data using wget. It makes me wonder who (or what) this archive really serves, because it's most certainly not the general public.

    1. Re:All of which is rather useless... by Xtifr · · Score: 2

      A) You can read it just like you can read normal webpages on the main web, most of which also don't allow you to copy them.
      B) The Archive is more than just the Wayback machine. They also have what is almost certainly the worlds largest digital collection of public domain and CC-licensed media files in their media collections.

  7. They Should Copy All Of The Web Site by eugene+ts+wong · · Score: 2

    I have never understood why the few archive sites, that I have been to, never back up the entire web site, instead of just a few important pages and images. I can understand not accessing pages that are supposed to be secure, but all other pages should be fair game. This is most important for product knowledge. Some times a company takes down its site and images. It would be nice to have an archive to go to.

    --
    testing out my trending skills
  8. looks like you forgot to add '-h' switch by Anonymous Coward · · Score: 4, Insightful

    10,000,000,000,000,000 Bytes = 8.88 Petabytes

    1. Re:looks like you forgot to add '-h' switch by Anonymous Coward · · Score: 3, Informative

      looks like you forgot to spell pebibytes correctly

    2. Re:looks like you forgot to add '-h' switch by Anonymous Coward · · Score: 5, Informative

      You have that backwards, kilo, mega, giga, tera and so forth are base ten prefixes and have been for quite a bit longer than people have been misusing them to refer to base 2 numbers. As such it made more sense to leave it consistent with everything else and make a new prefix for the binary numbers.

    3. Re:looks like you forgot to add '-h' switch by swillden · · Score: 2

      As such it made more sense to leave it consistent with everything else and make a new prefix for the binary numbers.

      In the context of storage sizes they were well-established with the binary-based definitions, so changing them to be decimal-based isn't "leaving" anything.

      Not true. In the context of storage sizes they were well-established to be base-10 definitions from the dawn of the computer age up until the 1980s or so. Only in the last 30 years or so have we started using powers of two units, and then only for RAM. Up until then, RAM was measured in powers-of-10 words, and in disk-based storage base 10 was and still is the norm. Network data rates likewise are and always have been in powers-of-10 units.

      This is why it's useful to be careful to use the proper prefix. 10 petabytes is approximately 8.8 pebibytes. See? No confusion.

      --
      Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
  9. Domain parkers deleting archives by linebackn · · Score: 5, Informative

    I don't know if they have done anything about this recently, but there was a problem with domain parking sites putting up a robots.txt that instructs Archive.org to delete or suppress any archives of the site that was there previously. Have run in to a few sites like that. If someone dies and their site goes with them, it isn't right for some squatter to remove their work from history.

    And I wish I could pull up historic copies of the original altavista.digital.com.

  10. Download Link? by mysidia · · Score: 4, Interesting

    How nice of them to do the archiving and release such a large dataset.

    Where can I download the file?

  11. My Poor Infringed Copyright!! by TechyImmigrant · · Score: 4, Funny

    It looks like they've copied my website and are therefore infringing my copyright.

    But I won't be suing them because I don't mind, because I'm not Apple.

    --
    I should use this sig to advertise my book ISBN-13 : 978-1501515132.
  12. Re:The more you know. by 91degrees · · Score: 2

    If you are referring to storage sizes in relation to computers, be it RAM, disk sizes, etc., it is correct to express them in powers of 2.

    No it's not. It's sometimes convenient to do so, especially for RAM, but the prefixes used are defined by the SI and recognised by a large number of international organisations including the IEEE.

    Yes, marketing people find this useful. But it's also recognised as correct by many engineers. It's actually quite useful. Using a certain type of modulation, a 1KHz signal can transfer 1 kilobit in 1 second, and this will take 8 second to transfer 1 kilobyte of data. Why does it make any sense to worry about binary addressing here?

  13. What the hell by nuckfuts · · Score: 5, Interesting

    are they using for backups?

    1. Re:What the hell by rubycodez · · Score: 2

      more disks, and they send a copy to euroarchive and the Library of Alexandria. in 2006, that copy & verify process to remote site took two weeks.

      http://www.enterprisestorageforum.com/technology/features/article.php/3633256/The-Wayback-Machine-From-Petabytes-to-PetaBoxes.htm

  14. Just fucking say Petabytes. by Arancaytar · · Score: 2

    I know the prefix invokes unpleasant connotations, but it also means 10^15.

  15. Were's my page then? by AndyKron · · Score: 3, Informative

    With all those pages stored why does it always tell me that page can't be found?

  16. Private archive by fa2k · · Score: 3, Interesting

    It's great that archive.org is doing this, but it's such an important part of history so I thought I would do a mini-version for the pages I visit, just to be able to refer back to stuff. I've been using the Firefox addon called Shelve to save all pages I visit on my home computer for about 2 months now (at most one version for each day). It's a total of 5.8 GB. It's not useful for browsing though, I'd love it if it was better integrated with Firefox such that I could choose among all versions of each page. There's sometimes some excellent information on university pages or cheap hosting, that could be 10 years old, and you never really know how long it's going to stay up..

    Anyway, this may give some perspective too; 2 months of daily snapshots of slashdot, other news, some tech stuff and a little Facebook takes just 5.8 GB.