Inside the Internet Archives

blackbearnh writes "O'Reilly Media is running an interview with Gordon Mohr, Chief Technologist for the Internet Archive (archive.org). If you've ever wondered how pages are selected for archiving, or just how they manage such a huge quantity of data, the answers are here. The interview also touches on the problems of intellectual property in archives, archiving the Internet in a post Web 2.0 world, and the potential vulnerabilities exposed by archiving web sites that may include security exploits."

Comments

Oblig. Clark/Kubrick

[Anonymous Coward]

My God, it's full of ones and zeros!

mutual exclusivity?

[Itninja]

The Interviewer: And I'm not sure I want to think about what posterity is going to think about a recording of my Twitter feed.

If Twitter becomes so mainstream so as to be more than a 'remember when?' to posterity I will kill myself.

Re:mutual exclusivity?

[GregNorc]

I really like Penny Arcade's comic about twitter.

Twitter seems useless to me. Maybe if my friends used it I might, but for now an away message or facebook status does the job just fine.

Re:mutual exclusivity?

[negRo_slim]

Maybe if my friends used it I might, That's what I always thought, but then I realize how few shits I'd give about what my friends would write... and it all becomes clear, all those sites are pointless!

Re:mutual exclusivity?

[Chyeld]

If Twitter becomes so mainstream so as to be more than a 'remember when?' to posterity I will kill myself.
--
I am ten ninjas.

Lets be realistic here, you are ten ninjas. You will be killing yourself regardless.

but is it indexed by google?

[AliasMarlowe]

and does archive.org record google's cache?

I wished archive.org stored even more stuff

[jacquesm]

I keep running into bookmarks that have gone awol, then find that archive.org also doesn't have the pages anymore.

Combining a bookmarking / chaching service would be really handy.

Re:I wished archive.org stored even more stuff

[blhack]

Combining a bookmarking / chaching service would be really handy. I heard that lexmark makes one, its called a "printer".

Re:I wished archive.org stored even more stuff

[RareButSeriousSideEf]

Yeah, how exactly do pages go AWOL from archive.org? I've encountered that, plus pages suddenly acquiring META refresh tags (maybe through an external script or iframe?) that redirect to some domain squatter's site now. Extremely annoying. I'm going to have to mess around with wget to see what's in the markup, unless someone can suggest an easier way to get at such content.

Combining a bookmarking / chaching service would be really handy. Furl fits that bill, doesn't it?

Re:I wished archive.org stored even more stuff

[jacquesm]

hehe, yes, so true, but then you can't access it electronically any more.

I really think the bookmark + cache would be a nice thing to have without resorting to 'dead tree' format.

But it's a good point, a printer would be an easy way to collect stuff that you really want / need to keep.

Re:I wished archive.org stored even more stuff

[Alphax.au]

Combining a bookmarking / chaching service would be really handy. WebMynd claims to do that; I haven't tried it myself though.

Re:I wished archive.org stored even more stuff

[TTK Ciar]

New material is always being added to The Archive's web archive, and (afaik) unlike the collections archive it is never deliberately deleted. Most of what appears to be "pages going AWOL" is indexing errors. In order for newly archived stuff to become visible to the wayback machine interface, the entire web archive needs to be periodically re-indexed. Unfortunately the indexing process is error-prone, and stuff that might have been accessible before the index might disappear afterwards (and appear again after the next indexing).

Other things that can make data temporarily unavailable are:

* Downed servers. There are over a thousand on the web archive's end of the cluster, and if a server "only" crashes once a year, that's about three servers crashing per day on average, but it doesn't happen at a low constant rate. Traumatic events at the datacenter (like AC failures, power cycles, etc) tend to knock a bunch of hosts onto their asses at a time, and they don't always come back up when rebooted. The Archive runs at a deficit of system administration manpower, so it takes a long time (weeks, sometimes months) to get humpty-dumpty back together again.

* robots.txt. In order to avoid getting their asses sued off, The Archive uses the live site's robots.txt to control which pages are publicly viewable. Every time you hit up a wayback machine URL, it downloads the real site's robots.txt and parses it to see if the owners have rendered the desired content unviewable. So if a site owner changes their robots.txt, archived content that was viewable yesterday might not be viewable today. When a website is abandoned, there isn't a robots.txt to download anymore, so at least entirely "lost" sites are viewable by everyone.

* Genuinely lost data. When I worked at The Archive (a few months ago, now), most of the web archive was on "SOLO" nodes, meaning there was no on-site replication of the content. The data servers lack RAID-level redundancy as well, so if a SOLO loses a disk, and nobody copied the data off it first, and there isn't a copy tucked away on our sister sites (some in Amsterdam, some in Alexandria Egypt), then the data is lost forever. To prevent this from happening, disks are tested hourly for a variety of symptoms (like nonzero sectors reallocated in SMART), and if a disk shows early signs of ill health, its contents gets "shuffled" off onto other machines in the cluster, and the disk itself is replaced.

But the system isn't perfect, not by a long shot, and lossage occurs. It's possible to do a lot better. Numerous people within the archive have tried to put better practices into place over the years, but for various reasons getting those practices into .. well, into practice has proven futile. Fortunately around the time I left, there was a push underway to get more of the web archive onto paired storage (so that all data was stored in duplicate on different physical machines). We can hope that moves forward.

The last time anyone tried seriously measuring the rate of lossage, iirc it ran into the 10-20 Mbits/sec range. That's not even a slow drip against the ~1.1PB in the web archive, but loss is loss.

Anyway, the *vast* majority of missing pages aren't really missing, they're just not indexed at the moment. The content itself is still tucked away in the cluster, and may resurface in the future.

-- TTK

Wayback

[TheRealMindChild]

While I love the wayback machine, a little "problem" creped in a couple of years ago that is still there... and it drives me nuts.

At one point, I forgot to renew my domain name and a squatter snatched it up the second it was available. I have since lost the html/java applets/images/etc that I had originally there. I used to show people what it looked like via the wayback machine. But you can't do it anymore. Example: http://web.archive.org/web/*/http://www.mindchild.net

Apparently, the current squatter put a robots.txt on that domain, and wayback refuses to show any ARCHIVED pages where the domain CURRENTLY has a robots.txt. I emailed them about it, and after a couple of months, I actually got a reply pretty much saying "That is just the way it is. We are underfunded and have no time to fix it. Sorry".

So if for some reason you don't want to have your site viewable via the wayback machine, just put up a robots.txt. It doesn't even need to contain anything.

Re:Wayback

[ibwolf]

This is an unfortunate side effect of their policies but it is very understandable that they would like to err on the side of caution.

Should the robots.txt ever go away or change then your old stuff will become accessible again.

Re:Wayback

[iangoldby]

wayback refuses to show any ARCHIVED pages where the domain CURRENTLY has a robots.txt.

In true Raymond Chen style, think about what the world would be like if this wasn't true: If it wasn't true, then a site owner would have no way to remove his content from the Wayback Machine retrospectively. That raises far more problems that the ability of a new owner to remove a previous owner's content.

Re:Wayback

[corsec67]

User-agent: ia_archiver
Allow: /

in the robots.txt that you mention (http://mindchild.net/robots.txt) is hardly not containing anything.

But, it is interesting how they take the current robots.txt to apply to old content that used to be at that location...

Re:Wayback

[SydShamino]

If it wasn't true, then a site owner would have no way to remove his content from the Wayback Machine retrospectively. I don't necessarily disagree with their policy, but this is the wrong argument for it.

If you publish something, you lose the right to withdraw it from the public archives retrospectively. That's part of the "contract" (term used figuratively) with the public that establishes the foundation of copyright law.

If you don't want it to appear on the Wayback Machine, you have an ability called robots.txt. That's already more than you have if you publish a book and want to keep it out of libraries. In neither case, though, do you have the right to demand or expect the content to be removed from the archive on your request.

I see what the archive does to be a courtesy service, not something that the site owners should expect.

Re:Wayback

[RareButSeriousSideEf]

Ideally they could obey the robots.txt at the time of archiving, and simultaneously grab a snapshot of the whois record. In the future, new robots.txts would by default only take away previously archived content if the domain hadn't changed hands. This would keep squatters from killing the archive, and the original copyright owner could always actively request removal of content if s/he matched the old whois record (though this would take manpower at archive.org, which is a problem).

Re:Wayback

[oodaloop]

I'm sorry you feel that way. I, for one, welcome our robots.txt overlords.

Re:Wayback

[sp332]

If you put it on the internet, it is expected that you want people to see it. I usually prefer opt-in to opt-out, but this is a case where the content is ALREADY PUBLIC. In this case, any opt-out is being generous.

Downside of IP conciderations...

[CFBMoo1]

I had a cheesy site back in college where I played around with HTML and learning the basics. I ended up making a few pages that poked fun at friends.

I went to archive.org years later looking for them cause I remember back in the day they nabbed em and now they're all gone. The images and sounds I used were all gone.

I wanted to recreate a page from that archive for nostalgia reasons with my old friends. Can't do it and I can't find the files anymore in my local archives.

I was kinda disappointed but I guess it was expecting too much. I really wish there was a true and complete archive of the internet that didn't care what was there it just had it.

selection? funding? why not plain Debian?

[bcrowell]

I was left with a several questions that weren't addressed by the article.

The slashdot summary says the article explains how pages are selected for archiving, but I couldn't find anything in the article that explicitly explained that. It does say that the actual crawler is run by alexa, which hands off the data to them, but it didn't say what the criteria were. Alexa computes various stats about web sites, so presumably they could apply some kind of minimum cut. Or do they try to index every single lame personal page, unless the owner opts out? That seems like it would require an unreasonable amount of disk space. The web also has a lot of stuff like, e.g., the kind of spam sites that try to scam google's search/ad system; I wonder if the archive records those.

The article didn't say a darn thing about funding. They have to run thousands of machines, so the electric bills must be formidable. Where the heck do they get their money? Is there a significant chance that their funding will dry up at some point in the future, and the whole archive will disappear?

The article states that they moved from plain Debian to Ubuntu. That surprised me, and I was curious why they'd do that. E.g., if you're shopping for webhosts, it's much more common for them to offer plain Debian than Ubuntu. I love Ubuntu as a desktop distro, but it surprises me that they'd see any big advantage in using Ubuntu for their application.

Re:selection? funding? why not plain Debian?

[gojomo]

I can't comment in more detail about Alexa's bulk crawl strategy because it is only documented to the public (and us at the Internet Archive) in general terms: it is a broad survey crawl of the public web, weighted by Alexa's internal measures of site/page importance and legitimacy (which are at least partially based on the same toolbar data that drives their site rankings). While we expect to continue receiving the Alexa donations indefinitely, a growing proportion of the public archive is likely to come from other sources, including the IA's own crawling and other outside donors, in the future.

The Archive is funded by a combination of private donations from individuals and foundations (sometimes for general operations and sometimes for specific projects), and fees for services provided to our partners, who are public libraries and archives themselves. With 11+ year history, and long partnerships with customers and funding sources, we're pretty stable in the world of technology nonprofits.

I wasn't directly involved in the Ubuntu choice, but it's been nice to have our developer desktops in close sync with cluster servers.

- Gordon @ IA

Remember Slashdot in it's Infancy?

[dbarron]

Check this out....it reads like a free software update blog :)
http://web.archive.org/web/19980113191222/http://slashdot.org/

Re:slashdot uid

[ConceptJunkie]

HAW HAW!

Sorry, I couldn't resist.

Squatters & robots.txt Re:Wayback

[gojomo]

Unfortunately, this "squatters-add-robots-restrictions" problem comes up a lot.

We'd like to address it, and to do so there are two major issues to be tackled: (1) our current Wayback Machine software only excludes sites on a "for all time" basis; (2) short of mechanistically trusting the current domain owner, determining who has the right to exclude or restore material could be a very labor-intensive, error-prone, and liability-compounding process.

The new open-source 'Wayback' software, which will go live for the Worldwide Wayback Machine later this year, enables time-range exclusions. (It's currently only used for many smaller collections we do for partners.) That should give us the capability to address (1). Addressing (2) will require further discussion about the proper and efficient policies -- but it's on our agenda once the technical capability for time-range exclusions is in place.

Specifically regarding the mindchild.net site you mention, it looks like the issue is that our current retroactive-exclude robots.txt-parser doesn't understand the 'Allow' directive. (The mindchild.net/robots.txt tries to enable ia_archiver/WaybackMachine access via an 'Allow'.) That too will be fixed in the new 'Wayback' deploy (if not sooner).

- Gordon @ IA

Re:2008 is the year of Linux on the Archive!

[TTK Ciar]

The transition from Debian to Ubuntu was driven by developers' desire for more and newer features. We originally went with Debian-Stable because it was, well, stable, and did everything we needed the PetaBox to do at the time. But programmers whined and moaned that such-and-such package wasn't supported, or was too old, and claimed that this held back development of features which Brewster wanted to see made into reality.

Brewster was never much for stability anyway, so the transition was made. It bit us several times, as Ubuntu is not as stable as Debian-Stable (which is to be expected when releases happen more often and newer software is deployed without extensive testing), but the developers were a lot happier with it. And, to be fair, while some of the problems have been substantial (like kernel bugs which interacted with the forcedeth device drivers to make servers freeze ~10% of the time when power cycled), afaik it has not contributed directly to data lossage (which is the bottom line at an archive).

-- TTK

Re:Searchable Archive. . .

[gojomo]

'Recall' wasn't exactly Google-like search. IIRC, in some respects it was better, with an advanced idea of related concepts, and with data on frequency of terms over time. In other respects, it was not what people would expect: there was no exact phrase matching, and certain terms that didn't become tracked concepts weren't findable at all, even though you could see the words in other indexed results.

Unfortunately, IA couldn't maintain the deployment when the developer, Anna Patterson, moved to Google. So, Recall turned out to be a short-lived experiment, grand in scale of pages indexed and novel features but not in traffic served.

Patterson did big things at Google and now has another search startup, Cuill, that's likely to do more good things for the web.

At the Internet Archive, we've also been using the open-source projects Nutch and Hadoop to offer search on smaller web collections for our partners. (A pair of such searchable partner collections for the US National Archives and Records Administration lives at webharvest.gov.) Someday we may be able to scale these up to the full 11+ year archive.

- Gordon @ IA