Slashdot Mirror
Indexing the Entire Web?
cah1 writes "BBC is carrying a story about another new search engine All The Web. The designers are planning to have the whole shooting match, all billion pages, indexed by the end of the year. " You can also read press from the company as well. I'm skeptical-they claim to be able to catch up within the first year, and keep up thereafter. But they claim to have 200 million already, so who knows?
That was weird, I clicked on the reply to the H-1B article. I then waited half an hour, entered my message and submit it, and it was attached to an article that didn't even exist when I first hit the reply button. Strange...
This is probably obvious, but it's not only computational load that would be more evenly distributed. With some knowledge of the preferred routes of various levels of the net hierarchy, the traffic of the spidering could be more contained to small areas of the network.
Far flung links could be handled at higher levels and passed down to other spidering nodes closer to the link target (from a routing perspective). This would mean a little more computation overhead somewhere but I imagine it wouldn't be too bad. The benefits of distributed spidering seem to me quite attractive...
On the other hand, if it really was that feasible, wouldn't one of the Big Boys take it up, or is it too much hassle to develop a business model for a search engine based on volunteer spiders?
Christopher
Mozilla
Two points: /robots.txt?
(a) Spanning more pages is only half the story. You need to combine huge page indexes with a lookup scheme like google's where the chaff is separated from the wheat. Otherwise you'll just be drowning in 5 times as many useless hits, and you'll need a search engine to search through the 100,000+ hits returned for your query to find what you're actually interested in.
(b) Does anyone have statistics for what %age of the web is excluded in
If you used it a couple of weeks back, you used the old index. A demo-version running an index of about 70 million pages (I believe) have been running for some months. The announcement yesterday is about the new index that claims to be the world's largest.
-- Best regards, Frode Lundgren
It's ok. I would say "mediocre". The only reason that it doesn't get a "bad" score is when I type my name in, it brings up right at the very top some pages from my website. I don't know what kind of search it uses, but if I type in "Sun Microsystems" or "Dana Corporation" .. I kind of expect to get the company's web site right at the top. But mostly what I see are news articles with the companys' names within them. Also, when I type my name in, I get the really obscure pages on my website.
:)
But, if I do the same thing on Yahoo/Infoseek/Lycos/Altavista, I either get nothing that pertains to me with my name or another different, obscure page on the website. When I type in the companys' names I may or may not get the company's website at the top.
What is more important to me than how many pages it returns is how many RELEVANT pages it returns. And yes, it is supposed to read my mind to some extent and know what I want.
The BBC story was pretty good, but at another story on this new search engine (http://www.latimes.com/HOME/BUSINESS/t000068951.h tml), one commentator said,"What does it mean to have another 100,000 or 200,000 links show up in a search? . . . The only thing that matters is the top 10 links you get back . . ."
I think he misses the point. IMHO the ideal search engine (1) covers all of the Web (Yes, I *know* it's impossible! This is an *ideal*.) and (2) allows me to construct a proper Boolean search argument.
Boolean is very important to me. It allows me to pare those results down from 1,276,349 to 280. When I pare down the number of results then the top hits are far more likely to be relevant. So far as I know (and correct me please if I'm wrong) the only search engine that allows the proper construction of Boolean arguments (AND, OR, parentheses and NEAR) is Alta Vista. Other engines such as HotBot and Google allow some ability to refine the argument, but not enough for my taste. This new engine still doesn't satisfy that desire either.
However, it does give some tools (phrases, + and - but no parentheses or OR) so having a bigger database is a Good Thing.
I found it snapped back the results pretty quickly too.
By the way, something that isn't discussed very often, but is pretty relevant in evaluating the effectiveness of search engines is latency. At the WWW8 conference in Toronto, I heard a paper that made the observation that search engines have a bad tendency to "forget" URLs. I.e. the same argument given over time will some times not discover a site that an earlier search found. On occasion, a later search will then "rediscover" the page. (Sorry I don't have the reference to hand. I've really got to do some housekeeping. . . ) The moral of this is: bookmark that interestin' site when you find it or you may never see it again.
I've only considered this as a strictly volunteer project, directed by a university and the top level hosts and database hosted there, with some corporate sponsorship thrown in for good measure.
I don't know if this would work if commercialized, since a lot of the folks who have the knowledge, experience and compute power to participate would probably not feel too warm or fuzzy about helping to build the next Yahoo!, especially when the IPO made the company worth millions overnight. It would certainly be tough to maintain the same level of participation after going commercial, unless some hitherto unforeseen way of rewarding participation per contribution were discovered. Perhaps corporate sponsors could offer premiums to contributors based on sites spidered? Maybe something along the lines of frequent flyer miles?
slashdot broke my sig
>Ok so repeating an effort to find the various purported etymology of the word "strawberry" I
>searched with +etymology +strawberry +origin on both yahoo (my standard) and alltheweb.
>Yahoo found 60 while alltheweb found 117, but a number of allthewebs' finds were xxx sites!?
>How many xxx sites actually use the word etymology and if this is more do we really want more?
I tried "CalTrans Bridge Design Manual" in Google!, Inference Find, and All The Web. Google gave me many links to CalTrans sites and some associated ones. Inference Find found the CalTrans sites and a bunch of tangentially related sites. All the Web found a bunch of CalTrans sites and related sites, but numbers 19 and 21 were porn sites, and putting CalTrans at the end of the string got me more porn sites.
Not a terribly useful site, IMO.
I've just used the search engine,... got two results, wherefrom one non existing.
god! I hate search engines.
(but, he did it fast!)
And then there are all the privacy concerns...
--
Linux user since early January 1992.
> "Actually"? ;-)
> You claim to KNOW this?
> I noticed you live in the same city as FAST headquarters..
> But maybe you cant talk about that
Actually, I think Frode should updates the curriculum vitae on his home page, to include the fact that he's a FAST employee. I claim to know this.
:)
I just created http://www.egroups.com/group/dizz-net/ as a an email discussion list. You can subscribe by sending email to dizz-net-subscribe@egroups.com. There are a lot of interesting issues, many already mentioned here:
-david.
Well, they say they honor ROBOTS.TXT However your post suggests differently. You ought to e-mail them and find out. Their robots policy is stated here: http://www.fast.no/f aq/faqfastwebsearch/faqfastwebcrawler.html
>searched with +etymology +strawberry +origin on both yahoo (my standard) and alltheweb.
>Yahoo found 60 while alltheweb found 117, but a number of allthewebs' finds were xxx sites!?
>How many xxx sites actually use the word etymology and if this is more do we really want more?
I tried "CalTrans Bridge Design Manual" in Google!, Inference Find, and All The Web. Google gave me many links to CalTrans sites and some associated ones. Inference Find found the CalTrans sites and a bunch of tangentially related sites. All the Web found a bunch of CalTrans sites and related sites, but numbers 19 and 21 were porn sites, and putting CalTrans at the end of the string got me more porn sites.
Not a terribly useful site, IMO.
So who says that there are only a billion pages out there, maybe 2 billion exist. So how do they know when they're done?
BTW: Who needs to sort through all that junk after doing a search. Use metacrawler, and get a pretty good compilation of the best search engines out there.
Yep, I got an email from FAST telling me that they had a bug in their ROBOTS.TXT handling when they indexed my site, and they've now fixed it.
Try view source on this page to see the way I would handle it:
Anti-Linking Script
Of course, there are other more sophisticated ways to deal with it, but this can work if the people aren't bound and determined to link to you.
All the creatures will die, And all the things will be broken. That's the law of samurai. (Jubai, 1605)
Uh, Altavista doesn't ignore home pages. I hit them all the time when they happen to have a word I'm searching for.
Whenever I hear about a new search engine, like many other people on /., I have a set of queries that I tend to try. My results were that it seemed somewhere in the ballpark with Google and AltaVista as far as numbers. One of the things that they pride themselves on, is the ability to check more often and catch broken links faster... If this is the case, why am I seeing more broken links off of their engine than any of the other major ones. They are getting the same pages, but the other engines wiped these pages out quite a while ago because they were no longer current.
Would you do it for some scoobie crack?
I judge search engines by the most important criteria of all - how many references to me they have. Alltheweb now has vastly more than runner up Google, making them the biggest ever. I type in "Aaron M. Renn" and I got 1604 on AllTheWeb, ~500 on Google and only ~180 on AltaVista. Even if that number drops as I searched through the pages, it's still impressive. I did look through the plain "Aaron Renn" listings too, where they also crushed the competition (though it's a much smaller number of pages since I virtually always use my middle initial). Believe it or not, there is a page out there with another "Aaron Renn" on it. Pretty weird.
It was good to see that the BBC pointed out that relevance of the search results is probably more important than the number of pages in the database -- and that Google seems somehow to have the most relevant results time after time. Alltheweb didn't do very well on my standard test word, so I'm sticking with Google.
11.0010010000111111011010101000100010000101101000
Retrieving pages to be indexed isn't a problem. Indexing them, and doing fast enough retrieval later is.
Here's what Netcraft has to say about it: www.alltheweb.com is running Apache/1.3.6 (Unix) PHP/3.0.11 on FreeBSD .
Both Apache and FreeBSD are well-proven OpenSource software projects. I imagine this is going to be very stable ;)
Also, based on earlier statements from them, I'd imagine that this will only be used as a proof of concept (they're more interested in selling and licensing their hardware and software than running search engines on their own).
So who says that there are only a billion pages out there, maybe 2 billion exist. So how do they know when they're done?
BTW: Who needs to sort through all that junk after doing a search. Use Metacrawler, and get a pretty good compilation of the best search engines out there.
Alltheweb is distributed (see http://www.fast.no/company/press/twbs02081999.html ), Hotbot is distributed and I guess most of the others are distributed too.
I even read somewhere some of the engines even use multiple Linux machines with applications written in Perl for indexing.
sulka
"Although it is not true that all conservatives are stupid, it is true that most stupid people are conservative."
All the Web is not Lycos and has nothing to do with Lycos.
Eyvind Bernhardsen
Yes but.
"bill clinton", altavista: 251138, fast: 397018
"bill gates", altavista: 131781, fast: 247267
"gates", altavista: 795520, fast: 916898
"slashdot" altavista: 27030, fast: 51102
"george washington", altavista: 178451, fast: 299085
"tour de france", altavista: 35429, fast: 130327
"commander keen", altavista: 2059, fast: 2841
but
"clinton", altavista: 2590260, fast: 1848689
"warez", altavista: 1399830, fast: 73031 !!!! altavista finds 20 times "warez" fast does
"sex", altavista: 35438560, fast: 3907581
altavista finds 10 times "sex" fast does
"senna", altavista: 102450, fast: 44019
Argh! this breaks my head. But my partial conclusion is that altavista has much much much much much more of these spam pages that index every possible keyword and then some more, another
personal experience is that altavista has much
pages that don't exist anymore.
I've been using alltheweb for a couple weeks (yeah, ever since it first showed up on Slashdot). My take on it is that it is fast, as in blazing zippy fast, but mostly useless. It tends to return a lot of pages that are all the same. Dig past the first 10 results or so, and you find that after that it's just one page from the same FAQ, only they list every mirror separately. I haven't found it to be better than any other search engine, and worse than most. Stick with a combination of Google, Altavista, and Yahoo. And Deja.com, for howto questions :-)
----------------------
"This moon-cheese will make me very rich! Very rich indeed!
There is no K5 cabal.
I am not the real rusty.
Exactly. Linux users are much more likely to recompile/rebuild/tweak the kernel, whereas *BSD users just run it. They wait for the kernel writers to distribute and don't bother rebooting. I have a linux boxen up 236 days and it won't get rebooted
Fellowship 9/11
I made two other searches for 'alltheweb' and 'jente' (norweigian word for girl) that gave me an entirely different result
AltaVista gave: 295 and 4744
Fast gave: 918 and 9099
But an analysis like that does not make FAST twice as big as AltaVista
139625351 documents found.
Their crawler (FAST-WebCrawler/0.3) was not very nice when it blasted through my site. The general guideline is that a crawler should grab one page a minute. Their crawler grabbed multiple pages a second even when going through a bunch of pages generated by CGI.
Has anyone else noticed this?
This particular search engine isn't honoring the robots.txt file (at least, not on my site). I checked to see if it knew about my pages, and it had indexed deep into my site DESPITE the "disallow" directives in my robots.txt file.
Shame on them.
I've been using if for a few days now, and it seems impressive. It's certainly fast. Google is still my engine of choice (even though it's visited my page a ton of times, and still won't find it when I search for it).
As for its coverage: it may be "the result of more than a decade of research into optimising search algorithms and architectures", frankly this sounds dubious.
If it covers 30% of the web it'll be twice as good as existing engines, but I suppose thirdoftheweb.com isn't that catchy.
This would be a great application for a distributed computing application, lots of computers indexing the web, and after they finish that, they can revisit sites for broken, moved and changed content sites... First post?
well, according to searchenginewatch.com they aren't the biggest
But another problem, is the amount of dynamically generated content. There simply ISN'T any way for a search engine to safely index everything on the web, because it can't know which CGI's just serve up a finite selections of pages from a database, and which randomly generate content, as long as no decent clues are given.
The amount of dynamically generated content is growing dramatically, so this will be an increasing problem.
According to http://www.netcraft.com/whats
Seems to be the platform of choice for serious stuff like this.
I may be being a bit slow here, but what is the problem which prevents coverage of the entire wbe by search engines ?
Surely if you just hit port 80 of every machine registered in DNS, and search recursively from the pages retrieved by that, you'll get a greater number of pages than the 10-20 percent most search engines have ?
Or is it the case that the problem is in the indexing of the data, and searching it quickly enough, rather than retrieving it ?
I try searching for "HotMedia". In Google, the first(!) result is the HotMedia homepage at IBM. Here, I don't see this page in the first results page.
I'll stick to Google.
--
http://www.wholepop.com/
Whole Pop Magazine Online - Pop Culture
http://www.wholepop.com/
Whole Pop Magazine Online - Pop Culture
I hope it has fun with the 45GB of stuff I have at the end of a 64k line.
BWAHAHAHAHAHH!
Assuming it will be a rather large amount of data, who will index thier index? (and who will index that index... and that one... and that one......)
Sig
Appended to the end of comments you post. 120 chars
Came across it a few weeks back - we were looking to integrate a search engine with our product, and they looked promising (although we didn't go with them in the end). I used it as my default search engine for a week or so to see what it could do, but it seems to have even less coverage than most of the others: pretty much *nothing* outside the US, and an awful lot of holes within it. Still, it's pretty fast and it manages to not return porn sites for searches on techie subjects, which is more than certain other engines do. :)
Not Cowardly, just lazy.
I wondered about those 200M pages already indexed, and I dug into Altavista, which says it has ~140M pages indexed.
I made two searches; one for the word 'Microsoft' and the other for 'Linux'.
Altavista gave : 12,682,370 (M$) and 4,526,430 (LX).
FAST gave : 4689227 (M$) and 2570827 (LX).
So.. If FAST currently is ~40% bigger than Altavista, how come they return numbers that much lower? With such large numbers it can't be pure coincidence, In My Humble Opinion.
-Snotboble
Q: How does a Unix guru have sex? A: unzip;strip;touch;finger;mount;fsck;more;yes;umount;sleep
My rough calculations indicate that it's *very* feasible to spider a billion pages in that length of time.
Check out this
http://www.fast.no/product/fastpmc.html
gaute
-- We plunge for the slipstream the realness to find
-- We plunge for the slipstream the realness to find
The incredible String Band
How is it possible to index the entire web? The entire *static* web should be relatively simple, but dynamic content really throws a monkey wrench in things. And dynamic content is becoming much more commonplace. Not even going into forms, a page referenced by a URL may be different day to day, or even minute to minute (like slashdot).
!. For every site that goes up, one goes down. I don't know how they are going to keep up with dead links.
2. Slow, if they index everything you will notice definite slowness. Even if they find some kind of uber-fast way of searching through stuff their servers will be slowed down by net-troglodytes searching for the "internet" or the letter "a".
Imagine how many pages would pop up if you searched for the word "pictures".
crazy dynamite monkey
Good point. I can also see the xtian right and censorware manufacturers being excited about having a comprehensive list of sites that need shutting down/blocking. If people want to remain unpublicized on search engines, they should be able to. Respecting ROBOTS.TXT is a simple solution already available, and I hope FAST will come around on this.
Is how well it finds my own home page. Google takes me straight to my main page if I enter my e-mail addy, but this engine only shows me one of the files on my page. It has to do with music files, so I think maybe their algorithm decided it was more relevant/interesting to the average person than my main page.
I think I'll stick to Google.
In this article.
I, for one, would like to keep some of the webpages I post on the internet un-indexed because they were ment for a couple of friends, not a couple billion people to rummage through.
If you want to impress me with a search engine, re-run the search machines and get rid of all the
expired and bad URL's. The more URL an engine adds the more it becomes unusable.
Did anyone else notice that these are the same guys that have or had ftpsearch.ntnu.no before lycos got hold of it and there's gratuitous links to lycos all over their site. It's not just lycos in AllTheWeb.com clothing is it ?
The basic idea was that the pages are indexed locally at the server, and indexed data are gatherad and can be queried at "brokers".
This was on /. several months ago
I'm tired of old stories being new.
That story last week about N2H2 and Bess...
Bess is not new, as the subject thingy said, been around for several years, i know, i fought it at my friend's house.
"I want peace on earth and good will toward men." "We're the U.S. government. We don't do that sort of thing!!"
How much do they have in the budget
to pay for porn sites?
Do they have a copy of my son's
Final Fantasy tribute pages?
Questions Questions Questions
http://www.alltheweb.com/cgi-bin/search?type=all&q uery=%22Robert+Ames%22+woodlands+-golf
;^)= (btw, that last claim is completely unsubstantiated, I'm just feeling mean :^)=
Running the above query says: "12 documents found," but it only shows results 1-10, and doesn't have a link to more results.
Now I don't know exactly how many pages that match this criteria are *actually* out there, but it seems as though you should show all the matches that you count, unless you're padding your counts
This must be the bestest search engine ever, because the name says it is. You can't do better than "alltheweb". Everyone else might as well pack up and go home.
Just for fun I decided to search for myself on Alltheweb. To my surprise I found:
1. The plan for an old CS group project from college, where my name was referenced!
2. 2 broken links to ZDNet talkbacks of mine.
3. A CNet page with a dorky little media player I wrote and released as freeware.
4. Some random Italian site hosting Win95 software including my dorky media player with full description extracted!!
Wow...my head is swelling...
Hmm...it didn't find my page though...heh
Aaron
It's 10 PM. Do you know if you're un-American?
- antizeus: 27 hits, and it displayed only the first 20.
- notopia: 17 hits, and it displayed only the first 10.
- "evil farmer": 33 hits, and it displayed only the first 20.
One would think that they'd get this sort of detail worked out early on in the development process. Despite that problem, I was impressed by the thoroughness. There was some stuff there that I'd never see on other search engines.(by the way, "Notopia" was the name of a great radio program on KCSB that disappeared several years ago, and Evil Farmer was a great band in the Santa Barbara Calif area which also disappeared several years ago. I miss both of them. Unfortunately, antizeus is still with us.)
-- $SIGNATURE
no problems with storage at least, opticom(the company which develops 170terabyte polymerbased harddrives) owns 40% of fast search and transfer so they'll probably download the hole internet on one of these hd's
Well, I did an ego search, and they didn't even *have* my page, quite apart from showing only 9 of the 14 matches they said they had. Altavista or Northern Light is much better on this test - as far as I'm concerned, they failed!!
anyone know what they intend to do about the sites that don't want to be indexed? ... is it forced indexing? ... is there any laws (???) anyone can think of that may affect this? ... they are not on any search engine because they don't want to be ... i seriously doubt they want to be indexed either
-
example of site that doesn't want indexed: i know of a pagaen group's site that has info for the group to view quickly without waiting for snail mail
-
if knowledge is power, the internet is god - me again
Ok so repeating an effort to find the various purported etymology of the word "strawberry" I searched with
+etymology +strawberry +origin
on both yahoo (my standard) and alltheweb.
Yahoo found 60 while alltheweb found 117, but a number of allthewebs' finds were xxx sites!?
How many xxx sites actually use the word etymology and if this is more do we really want more?