Slashdot Mirror
How the Wayback Machine Works
tregoweth writes: "O'Reilly has an interview with Brewster Kahle about how The Internet Archive's Wayback Machine works, with lots of juicy details about how the biggest database ever built works."
← Back to Stories (view on slashdot.org)
It's an interesting idea, but the real problem is not storing the 100 TB of data, it's figuring out how to search through it to find what you're looking for. Now, apparently they write a lot of their own software, but it might be better if they could team up with Google and have Google index their sites on a special database. We'd have www.google.com for regular searches, and wayback.google.com for the Wayback Machine's sites.
Something else I found interesting: according to the article, they "use as much open source software as [they] can." That makes sense when they've got between 300 and 400 computers, and with the number growing all the time. Licensing all those with a non-open OS would be quite expensive.
Ok, we have successfully Slashdotted the Wayback Machine. Screw history! :) Let's move on to bigger and better things.
And I thought I'd erased all my old embarrassing HTTP handywork....until I discovered my old website nicely archived - bleargh!
:)
;)
Ah hell, may as well keep it there - it's even got my old web-based Curriculum Vitae on it too - perhaps in some way I've now been "immortalised"??
I've not touched HTML ever since those first abortive attempts I made 5 years ago, cause I realise now that I'm pretty crap at it - I'll stick to Unix admin, what I know best
Wow! You really weren't trying too hard back in the day, were you Taco?
:)
I was glad to see the interviewee was brutally honest about free software -- both its benefits and its drawbacks. Usually discussions among my friends usually degenerate into holy wars, with both of us spouting cliches at one another until we all storm off in huffs.
Free software can save the world, I think. We just need to realize that it needs a lot more work to get there.
They that would sacrifice their
Wayback slashdot.org goes back to 1997...
http://web.archive.org/web/*/http://slashdot.org
$HOME is where the
-- silver_p
They don't seem to think the history of their site would be interesting: http://web.archive.org/web/*/http://web.archive.or g/ lredirects you to their index.html! boring!
Now, that would really be a test for their apps. Same as if Google indexed www.google.com (entirely).
Not only does this sound like a rather far fetched plot from an old StarTrek episode, but it also seems to be an a physical and theoretical impossibility. Even if adequate storage space did exist for such a task (a 10 TB database would be but a small start), I do not foresee any type of technology that could ever adequately capture new data at a sufficient speed to harness that which is human innovation and creativity.
It is a nice thought, however, and I certainly wish him all the best in her pursuits...
Beer is proof that God loves us and wants us to be happy. -- Benjamin Franklin
100 TBs do not make the biggest DB ever. I am personally working on an 60-70TB ERP system that's also writeable; I am sure there are bigger systems out there (e.g. Wal-Mart's or GM's ERP systems come to mind).
A read-only DB containing highly-compressible text does not really make for a very challenging datamine. Just because it's on and about the Web and sexier than a stodgy ERP system should not make you overlook the real technology.
So... would I be right in thinking that the "Wayback Machine" is an (admittedly large scale) exercise in network database programming (of the style popular pre Codd and his relational model?) I am tempted to question if this is indeed the biggest database ever built - I suppose it depends upon definitions - but to my mind a database should be general purpose... whereas it appears to me that this project is basically a large-scale single index.
I also wonder if it would be appropriate to call this the largest project of it's kind - for example - while Google stores less data, I suspect it supports a higher query rate... how exactly do you intend to measure scale... if it is in terms of computing power is it relevant that Google already have thousands of Linux server nodes?
That said - I think it is an exciting project in its own right. I hope and expect this offering to become a significant information resource in years to come.
I thought it was going to tell us how Mr. Peabody's wayback machine worked. You know, like the flux capacitor diagrams that made everything clear . . .
----------
I am an expert in electricity. My father held the chair of applied electricity at the state prision.
Id say is pretty amazing, I actually was able to retreive content I thought lost years ago.
My sites go back to 95, and yep theyre archived starting 96, this is too cool.
I wonder how much of the goverments docs that were pulled off post Sept 11 are still on this ?
A really funny note is it seems like all the p0rn is intact staring in 96, gotta archive the porn.
But seriously , I was unaware of this, Im gonna use this thing like hell as a sales tool if nothing else. Its also great to find certain content thats been pulled.
Sig went tro...aahemmm.....fishing........
I just visited some sites from which I hoped that they dissappeared completely from cyberspace. The only defense I've got now are the old cryptic URLs of these monstrosities... Indexing that database would be a disaster, especially with an unusual name like mine...
(Yes, I was stupid enough to use my real name
Damn you, wayback
Okay... I'll do the stupid things first, then you shy people follow.
[Zappa]
This world needs more people like that... driven to make this world a better place... and having fun doing it, being proud of what they're trying to accomplish... This interview sent shivers down my spine... These are the kind of interviews that inspire people... It makes me think about humanity a little less sceptic... There's still hope.
The article was good with all it's warts and gems. I take it more as a testament to the human spirit. I seriously doubt it's the largest database, though it might be the largest publicly accessible database. I'm sure the NSA could easily dwarf their database considering how much data they collect from around the world every day.
You buy from EMC a terabyte for maybe $300,000. That's just the storage for 1 TB. We can buy 100 TBs with 250 CPUs to work on it, all on a high-speed switch with redundancy built in.
Interesting quote. Mr. Kahle addresses something I've been wondering for a while -- are storage area networks really worth it? Or is he ignoring the costs of maintenance and manpower to keep these things afloat?
Seeing that they cache webpages from other sites, I wonder how long it will take before another company sues them?
Also, I wonder what their criteria will be for "submissions"? 1 month? 1 year?
Me email iz skyewalkerluke at microsoft's free email service.
The interview talked a little about throwing more machines on when the demand deems necessary. I wonder if it is possible to do this over the internet? I mean, I'm seeing something along the lines of SETI, where millions of people worldwide donate their unused processor power. Would it be possible to distribute the searches to remote computers over the internet in real time?
_______________________________
"I'm not Conceited...I'm just a realist..."
A number of you have asked whether the websites taken down since 9/11 are available on archive.org. The answer is yes. One example is:
DC Air National Guard on Archive
Same Page - 404
One of the conspiracy websites that I have read was saying that combat airplanes, normally on 24 hour alert, at this base should have and could have prevented the plane from entering the restricted airspace in DC. They were saying that this site was removed because it provided evidence that somebody dropped the ball.
_______________________________
"I'm not Conceited...I'm just a realist..."
Are you violating copyright laws?
About the Internet Archive
No. Like your local library's collections, our collections consist of publicly available documents. Furthermore, our Web collection (the Wayback Machine) includes only pages that were available at no cost and without passwords or special privileges. And if they wish, the authors of Internet documents can remove their documents from the Wayback Machine at http://www.archive.org/internet/remove.html."
I don't really think that they're neccesarily right about this. I'm glad they've got the archive up, and I think it's dandy, but it seems like the copying and reposting of other's materials is a suspect practice. This will end up in court as soon as something that someone removed from their own webspace re-appears historically accurate here. I'd guess some liable suits will be the first...
--
RumorsDaily
http://znet.net/~schester/facts/database_sizes.htm l
Apparently, walmart's is 24TB, and the entire www index as of 1999 was only 6TB.
The biggest database ever? 100TB? Hardly.
I worked at a large pharmaceutical company for two years (known internally as the Squid), and supported a 380TB protein interaction database (Oracle) and a 260TB SAP-backend database (Informix + custom).
Certainly Wayback's database is large, and certainly it holds far more varied information and appeals to a far larger audience, but by no means is it the biggest. I'm sure there are databases that made the ones I worked on look puny by comparison.
Even though this has been available since Oct, it's the first I've seen of it. I think it's a great resource. Long dead sites that are no longer there now can be found for historical purposes. The interesting thing is that the links on the page are also updated to link to the archived versions. What I found it useful for was building a history page of what my site looked like over the years. Lots of great uses for this so hope it stays up!
liB
So the article says in one place that they wrote their own operating system, and in another that they use linux (or BSD, I forget which).
So which is it?
Sig is taking a break!
But what is the DBMS? Is the database relational? How it was modelled?
Leandro Guimarães Faria Corcete DUTRA
DA, DBA, SysAdmin, Data Modeller
GNU Project, Debian GNU/Lin
I once worked on a site with a 25 year old database that was much larger.
The ancient magnetic storage took up several warehouses. Beat that, for biggest database ever!
Even Slashdot wants to hide some things
Pretty decent read, but one thing they said got me thinking a little bit.
They said that at Thinking Machines they built a super fast computer, but it required a new way of thinking about things in order to program it. And then they called this a mistake, because they couldn't attract any customers.
This seems like a real problem that would lead to technological stagnation. At least from a market place point of view.
It is kind of similar to a company making games off of pre-existing engines, like quake, instead of some new non-quake compatible engine.
Or everybody making x86 compatible CPUs.
It also seems that when a company does come up with some new way of doing things, they get burned, and it is the second generation of companies that pick up the torch that make the money. So nobody wants to be that first company, they are all waiting for someone else to break the ground.
Maybe the only people/companies that come up with new stuff are the ones that are insanely rich, and won't get hurt by doing something new, or the insanely poor who have nothing to lose anyway.
I can't help thinking that this clustering boom going on is just like what 3dfx was trying to do. The difference right now is that clustering actually *does* outperform the super fast single chip. I wonder when technological advances will change this fact.
Slashdot looks the exact same it did 5 years ago!
WHEN is this site going to be updated? Forget the wayback machine, if I want ancient web history I visit slashdot.
--Dood
You're right, the Wayback machine is not the largest collection of data -- not even the largest collection online. I work with the USGS's catalog of satellite data. They have over 300 terabytes of satellite imagery, and the collection is growing at a rate of about 1 terabyte per day.
The USGS collection comprises multiple instruments, but Landsat 7 is a big one, contributing about 100 terabytes that's searchable online.
Perhaps 'Largest TEXT Database' would be a better description of the Wayback Machine?
Genocide Man -- Life is funny. Death is funnier. Mass murder can be hilarious.
From the article:
How the archive works is just with stacks and stacks of computers runnning Solaris on x86, FreeBSD, and Linux, all of which have serious flaws, so we need to use different operating systems for different functions.
The man puts bias aside and uses various OSs in areas in which each performs well. A real, tangible project like this is worth more than any amount of drooling zealotry.
Trolling is a art,
Having so few transactions for a database of this size probably helps them run without needing large expensive machines. Many VLDBs support thousands of transactions per second. I found a list here of top ten winners of a very large database scalability contest. The winner for peak performance was something like 20,000+ TPS.
"So IBM announces a 25 gig hard drive... does the world need this yet? Unless this is in a RAID, would you really want to trust 25 gigs on a single drive? What would you use this for? 400+ hours of MP3s comes to mind... "
mind you, this was only a couple short years ago, and now I'm writing this from a PC with three 80 giggers.
i thought we geeks were supposed to have more foresight than this? *grin*
A year spent in artificial intelligence is enough to make one believe in God.
This brings to mind a serious question. What data, if any, do we throw away? With ever expanding storage capacities it's getting easier and easier to just keep stuff, than to sit down and figure out what you want to throw away.
In the past media degeneration and obscelecense over time have made the decisions for us. But going forward we will have massive distributed, redundant data stores, with geographically remote backups. The data isn't going to go away unless we tell it to.
Freenet addresses this problem by culling the less popular data (not actively, but as a end result of its caching policies) - but this has the unfortunate effect that important data can get lost. Not a desirable behavior for corporate data.
-josh
What do you get when you mulitply nine by six? 42.
It works in base 13.
Fascism starts when the efficiency of the government becomes more important than the rights of the people.
Some things are just plain funny.
dinner: it's what's for beer
I can't seem to get the slashdot pages archived to load.
I think this is a fabulous project, and I hope it does well. However, I think that the notion of such a centralized database will begin to become unrealistic. I think peer to peer projects are the future, and I can see a day far in the future when the database layer comes down and inhabits the filesystem layer and all the databases on the internet can talk to eachother, and in a sense, the net becomes a giant database that anyone can contribute to.
Cheers, Joshua
When in danger or in doubt, run in circles, scream and shout!
"Ma'am, did you realize that Chevrolet has an important plan for your life?"
"Whatever happened to fair use?"
-- Duff-Man
http://www.archive.org/exec/faqsidos/about/faqs.ht ml?index=2 andt ml?index=26
http://www.archive.org/exec/faqsidos/about/faqs.h
The claims made in these faqs are just not consistent with the law. Are they going to repost everything that was available on Napster?
They also have some problems with their algorithm so that some domains that are redirected fool their algorithm into associating content with a site that was never actually associated with the site. To try to find copywritten works would be a nightmare. Archive.org has refused to respond to any of these issues and, in fact, are lying about it if the quotes in the article are factual.
Russ Smith
Don't worry. That's because the base 13 thing is a load of crap made up by people who read that part of "Restaurant at the End of the Universe" and managed to miss the point entirely.
Win dain a lotica, en vai tu ri silota
Has anyone tried this site in Konqueror? There is a floating link to the home page on top of the results that says search in progress, but never goes away?
You'll notice that Slashdot's logo used to be really big...
Win dain a lotica, en vai tu ri silota
index it, split the database horizontally each split on a different machine.
I would have to look at the database to tell you how big each split would ned to be, and how much power the machine housing each split needed to be, as well as other details.
depending on certian variables, I might even consider splitting it accross a cluster od some sort. but its hard to say without a real look at the structure.
The Kruger Dunning explains most post on