How the Wayback Machine Works

tregoweth writes: "O'Reilly has an interview with Brewster Kahle about how The Internet Archive's Wayback Machine works, with lots of juicy details about how the biggest database ever built works."

Comments

Re:Not very way back!

[tom.allender]

Wayback Slashdot ...only goes back to 2000?

Wayback slashdot.org goes back to 1997...

Not the biggest DB

[costas]

100 TBs do not make the biggest DB ever. I am personally working on an 60-70TB ERP system that's also writeable; I am sure there are bigger systems out there (e.g. Wal-Mart's or GM's ERP systems come to mind).

A read-only DB containing highly-compressible text does not really make for a very challenging datamine. Just because it's on and about the Web and sexier than a stodgy ERP system should not make you overlook the real technology.

Government Removed Site still Available

[Tazzy531]

A number of you have asked whether the websites taken down since 9/11 are available on archive.org. The answer is yes. One example is:

DC Air National Guard on Archive

Same Page - 404

One of the conspiracy websites that I have read was saying that combat airplanes, normally on 24 hour alert, at this base should have and could have prevented the plane from entering the restricted airspace in DC. They were saying that this site was removed because it provided evidence that somebody dropped the ball.

Link to various database sizes

[rkgmd]

http://znet.net/~schester/facts/database_sizes.htm l Apparently, walmart's is 24TB, and the entire www index as of 1999 was only 6TB.

Talk to the US government

[Remus Shepherd]

You're right, the Wayback machine is not the largest collection of data -- not even the largest collection online. I work with the USGS's catalog of satellite data. They have over 300 terabytes of satellite imagery, and the collection is growing at a rate of about 1 terabyte per day.

The USGS collection comprises multiple instruments, but Landsat 7 is a big one, contributing about 100 terabytes that's searchable online.

Perhaps 'Largest TEXT Database' would be a better description of the Wayback Machine?

Re:Copyright infringement

[pjones]

Child! Child! They do not sue you right away -- and they can't. First they send you a cease-and-desist order and you evaluate their claim.

But Brewster answers your question in the interview himself on the second page:

Koman: What about the question of rights? I just wrote about Lawrence Lessig's book on intellectual property. Surely the publishers and the television networks and the record companies aren't willing to let you keep a copy of all of their stuff?

Kahle: All we collect for the Web archive are sites that are publicly accessible for free, and if there's any indication from the site owner that they don't want it in the archive, we take it out. If there's a robot exclusion, it's removed from the Wayback Machine. Over the years, people would notice these things in their logs and would say, what are you doing? And we'd explain what we're doing -- building this archive and donating a copy to the Library of Congress, etc., etc., and 90% of the time they say, "Oh, that's cool, you're crazy, but go ahead." About 10% of the time, they'd say, "I don't want any part of it," and we instruct them on how to use a robot exclusion and they're taken out of history. That seems to work for everybody at this point. People are really excited about this future that we're building together.

Their movie archive has "Hired!"

[for(;;);]

Hot damn! Their movie archive has a downloadable version of the short they showed on MST3K prior to "'Manos:' The Hands of Fate."

"Ma'am, did you realize that Chevrolet has an important plan for your life?"