Slashdot Mirror
Juggernaut GPLd Search Engine
real bio pointed us to Juggernautsearch which actually looks interesting. Its GPLd. It can index 800 million pages every 3 months and deliver 10 million pages a day on a Pentium II. So I guess if you want to run your own Altavista, you can.
Altavista is yesterday's technology- it sucks a fat cock.
Real men dump cores! Read my journal, I am neat.
I'm wondering if this would be THE weak link in this idea. It seems to me that with the speed that the spidering engines works, you'd need a huge amount of processors at the "server" level just to eliminate the doubles. I use a freeware tool which collects data from 10-12 different search engines and attempts to eliminate the duplicates, only to get the same page numerous times from different sourced origin points, not even counting sites which are mirrored.
Any thoughts out there on how to solve this problem?
...Open Source isn't the only answer -- but it's almost always a better value than the alternatives...
Lycos. AltaVista. Yahoo.
These are all big-name, big-money comapnies living on borrowed time.
Just as every ISP provides DNS, mail and usenet services to their clients, the time is rapidly approaching where they will provide search/indexing services based on open industry standards. Products that integrate the search process into the OS, like Copernic or Apple's Sherlock are a clear indication of where the technology will go.
All it takes is a co-operatively networked "juggernaut search" system, the logical successor/complement to DNS, to topple the search/portal companies.
SoupIsGood Food
We use htdig for the internal network for the precise reason that we want it to only return linked directories, trees, and files.
And most crawlers can easily be limited to a particular site, or set of sites. Even wget does that.
-- perl -e'print pack"H*","6e656d6f406d38792e6f7267"'
Moderate this down, please.
I believe Excite was the first company to come up with this idea, when it was Architext Inc. They used a bit of distributed crawling on Stanford owned Sun boxes in off-hours.
The idea of a client reporting crawling is interesting but I have two issues with it. First, it's essential what is being done with services like AllAdvantage or free ISPs. They may monitor the sites you go to in order to build a database for advertisers, instead of for searching. The second issue is, net surfing would be bogged down considerably unless there was high bandwidth for the project.
One way someone could do this though, is to create an open proxy server on a big pipe, which would log all the sites users went to. This would be voluntary of course, and the database of sites could be added to the findings of a crawling bot.
Food for thought...
"In individuals, insanity is rare, but in groups, parties, nations, and epochs it is the rule." -Nietzsche
Female Prison Rape in NY
- the IB project at Basis Systeme netzwerk,
- the former commercial products from PLS that AOL is now giving away,
- the ASF project,
- the Webglimpse pages,
- the pages for the mg system.
For a comprehensive presentation on the subject, see the searchtools site.
I'm sure I'm not the only one who believes that the top-level stories on here should have moderation on them too.
I mean, really! This search engine hardly works at all, only the search part is free (and that's the no-brainer part of any search engine), it certainly doesn't index 800 million pages (I rarely got any results on any queries) and yet they still appear on here like some news item.
Did they pay slashdot? Are they a major stockholder now? What's the deal? Or was once again a story posted that wasn't checked first.
Give me seven million dollars, I'll double check my stories...
Puts me in mind of GPLTrans. "Tests have shown it's more reliable than Babelfish and InterTran," they claimed. Which was actually correct, provided you only want to translate the one sentence they tested it with :-).
Naw. The Whiner's a lurker on the MTK boards. Probably picked it up from me when I ranted about the topic a few months ago.
SoupIsGood Foof
Did you even read the page? It's a demo version; you're searching a minimal subset of their database.
Clearly not obvious to the casual observer, and the entire page just doesn't reflect the claimed quality of the engine itself.
It's a botched launch, and right after GPLTrans too.
Yours Truly,
Dan Kaminsky
DoxPara Research
http://www.doxpara.com
Yea, great it's Op So and all, but it's tough to beat Google. Case in point, I run a small site out of my apartment (for about a month now). I have yet to do any search engine placement or promotion, other than meta tagging. A search on Google with my full site name returns a page full of my pages. Also, if you search Google for "Wah" my page just makes it in there at the bottom.
Google rocks!!
+&x
In mind that they probably filter the pages that they store in their engine, or maybe have it limited for that frontend (though I doubt that). Though it may not be such a great engine for a general-user search-all task, if it was localized to say, a student home page directory, it would be a whole different story. And searching through a hundred thousand pages wouldn't take nearly as long as a few million. But hey, it's a start.
bumppo
give gamers some credit. It's not like it used to be, but you still have to have a pretty good knowledge of your system to get the most out of gaming. Most of what I learned about computers (and perhaps why my Linux is weak..) was learned figuring out how to make various games work. In the win3/dos4gw gaming era, if you didn't know how to reconfigure memory you couldn't play half the games.
Not to mention the hardware expertise that comes with 10 years of upgrades.
There need to be more games on Linux, it helps draw in that next gen. or hackers (ones that will start using *real* OSes at 12). If you really want the tide to turn (faster, I guess) support Linux gaming.
+&x
Thanks for the complement. I'll agree that ht://Dig more suitable for medium-sized collections, but "medium" in this case is starting to stretch to over a million URLs. In many cases, it's simply an issue of system resources--to index 250 million URLs like Google's first set, or 800 million like Juggernaut claims, you need some pretty big iron, at least in terms of RAM and RAID arrays.
We'd obviously love feedback in how well it scales since we rarely get such reports. It's an area that we'd like to improve (since many of the developers don't run "mini-Altavistas" themselves).
I haven't been able to check out the Juggernaut code since it's heavily slashdotted right now. But suffice to say, we'll be checking out whatever code they've made available to see if there are any interesting optimizations.
http://www.dizz.net/
Basically, we need to get down exactly what to do and how to do it. More developers would be nice too...
Here's part of one of my messages on the list:
You can get on the list at http://www.egroups.com/group/dizz-net.
Have they come out with a search engine yet, that, before giving you the results for your keyword searches, TAKES OUT the 404 errors? That would be something nice. Do your keyword search, and then have the search engine check each and every link to see if there's a 404 or whatever, and if there is, take it out of the results before it hands it over, and save the results it for next time in case someone else does the search.
I think many of us would agree that it is not GPL in spirit. But nonetheless, once they've made the code available that doesn't mean folks won't be reverse-engineering their database and projects like ht://Dig won't be examining what code was released.
Ultimately, their method of business may change in unexpected ways. Let's say someone reverse-engineers their database. Suddenly their revenue stream will disappear (unless they have some sort of patent, but that's another story). So they'll have to make money on support and/or hosting the indexing/searching for people w/o the hardware.
Let's not look a gift horse in the mouth. Ultimately the community will derive benefit from this code, either through cross-polinization with projects like ht://Dig, or simply by getting people interested in the concept of an open-source version of large search engines.
I consider the claims of indexing 800 million pages to somewhat exaggerated, since no one else is oing this at this time simply because the hardware required to do so is so intensive. The figure is now doubt based entirely on an "estimate" performed on a much smaller sample and extrapolated - and these are often wrong.
"The first time I got drunk, I got married. The second time I bought a chimpanzee, after that I stayed sober" Arian Seid
Yes sorry, that wasn't the reason. I should have checked before posting. This is the reason.
An extract:
Databases, by definition, mean dealing with huge amounts of data. They also often contain very small computational requirements (although this is not always the case). This means that the bottleneck for database operations usually isn't CPU horsepower, but disk bandwidth. This means that distributed.net would be ill suited to help.
And other posters have pointed out Harvest.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
But to make it do that, you'd want the crawler part, not just the database searcher, and from what everyone else is saying, the crawler is the part that isn't GPL'ed and isn't free. I don't think it's likely worth whatever they're asking for the crawler just to fulfill that simple need. The free part just searches the local database generated by the crawler, or purchased from Juggernaut, when you ask it to. Won't generate browsing-like traffic over your connection.
ProofReading Markup Language - and yes, I find typos.
Of course this idea will raise questions about privacy and the such, and the most popular pages would get evaluated most frequently (which isn't such a bad feature actually, some search engines work this way, they index cnn.com multiple times a day but put mypage.com on a lower priority).
Probably the same number of people who would switch their primary machine from Linux to Windows 2000 just to play Quake 3, if there was no Linux port.
I think you're right -- that would probably work. If I'm understanding you right, the spider would be return two lists, the "intra-site" links and the extra-site links, the server/servers storing the data would only have to check the "outside links".
Would this work for database driven sites as well?
...Open Source isn't the only answer -- but it's almost always a better value than the alternatives...
I pressed a button that said "you are 1 point beneath your current threshold" and the preceding message popped up. Did I do that? If i did, sorry. I'm kinda new here.
Sorry. I pay 19c/mb recvd. No way in hell I'm gonna participate. :-)
Open Source. Closed Minds. We are Slashdot.
Welcome to the world of GRIN :D
In case of Emergency, Curl up in the Fetal position, and lick a Bible for comfort!
Anyone tried it yet. Having a strong, open-sourced search engine would be a tremendous boone to institutions on a tight budget. We have a reasonably large webspace here and we're always watching for effective ways to make the whole thing searchable.
--
Una piccola canzone, un piccolo ballo, poco seltzer giù i vostri pantaloni.
My office has been taken over by iPod people.
"The Juggernaut Search Engine can deliver 10 million unique page views per day using a single Pentium II configured with the Slackware V4.0 distibution of the Linux operating system."
Yeah it's Slackware.
For the love of god, LAUNCH RIGHT!
Don't say you can index 800 million pages in three months when your database gives less results that Lycos circa 1996.
Hyperbole is rife in the computer world in general, and it's one of the genuine strengths of the Open Source community that we're very results oriented--Apache gets *results*. Samba *works*, and actually *does* knock NT out of the park in terms of flexibility and feature sets. And so on.
There are exceptions, granted, but we don't stretch our credibility to the breaking point nearly as much as stock-price-manipu^H^H^H^H^H^Hmaximizing corporations practically have to.
My problem with Juggernaut is that, while their technology might be awesome, their online index *isn't*. When you don't even get enough hits back to compare whether the hits are delivered in an optimum order, you know there's a problem. That, combined with the fact that the site looks decidedly 1996'ish(sorry, I know there's a webmaster out there who doesn't like me right now), tarnishes the otherwise excellent announcement that we now ostensibly(pending testing) have an extremely high quantity and quality search engine system, not to mention the birth of a new business model--the internal search engine of external content.
Honestly, I must admit there's something to be said about companies purchasing internal versions of large search engines, just so no outside source can watch the unencrypted stream of queries coming from a given company to deduce what projects they're working on.
The Juggernaut guys may be on to something, but I'm still a Google addict.
Yours Truly,
Dan Kaminsky
DoxPara Research
http://www.doxpara.com
From their demo, I am not too impressed with the search at all. It seems to be lacking many advanced options. Also, what is up with this??
. shtml)
>>
first fully automated crawler that can reindex all 800 million World Wide Web pages every three months fully available to the public for a nominal two year subscription fee.
Does that mean that they give away the search engine but you have to purchase the database???
I think that there are better options out there right now. One GPL'd search engive out there that I have liked a lot is HTDIG (http://www.htdig.org). It does not have the horse power the the juggernautsearch "claims", but it is great for intranet/corporate/university website search.
If you are looking for a good search engine, you may also want to read the ask slashdot thread from last year on this topic. (http://slashdot.org/askslashdot/98/10/24/1756224
I typed in 'jj thompson' to see would it find my page about the legendary physicist (it's indexed by most engines). It didn't bother returning any matches, or even a 'no matches' message. And it's the most horrible page I've seen in a while. Green on pink text? Yeuch.
I'd like to see a site that really uses the software to see if it is any good. Also I noticed that they seem to be selling a URL list for the search engine... It would seem to me that the engine should be able to find its own URL's...
All we need not is for someone to write a GPL crawler since we obviously have the file format for the database.
Anybody want to contribute one ?
From what I read here and here: the "Juggernaut search Engine" and the "Juggernaut Search Engine Crawler" are two separate pieces of software. The former is GPLed. The latter is not for sale but you can purchase the database it creates (or get a demo/sampler subset of the database for free)
-
<SIG>
"I am not trying to prove that I am right... I am only trying to find out whether." -Bertolt Brecht
<sig>Guvf vf abg n frperg zrffntr
I'll take them for their word on the performance aspect. However, my brief test of the engine disturbed me a bit. It is not obvious to me where the results of a search appear. Cuz it aint on the homepage. Speed and coverage key but it means nothing without results.
I have been wondering for a while now : couldn't building the index for such a search engine be distributed (like SETI@HOME or RC5) ? The server would do the actual page serving, querying etc, but the spidering would be done by the clients. They'd each receive a batch of URL's from the server and start indexing them, collecting lists of URL's and sending those back to the server. The server weeds out the doubles, and assigns those URL's to the clients again. The more people would participate, the bigger the index would grow, as the available bandwidth increased also.
Hmmm... maybe I should patent this...
superblog.org: all your favourite blogs on o
Sounds nice but isn't the bigger concern for something like that the amount of bandwidth it would use to catalog all those pages?
Sure it will run on a PII but does it need a T3 to run efficiently?
Josh
You can run your own altavista. . . and as the open source 'canon' grows, folks will also be able to have an amazon.com, a slashdot, and whatever else you want to do on the Web.
But why just the Web? With enough open-source game engines, applications, and other code to build on . . .
Well, just imagine what happens when the first Open Source 'killer App' is released. (Not that sendmail, apache, and others aren't already -- I'm talking userland, here.) What if the Next Big Computer Game was Open Source? How many zillions would install Linux to play it?
What if Open Source was suddenly the dominant software paradigm?
Can I just say, 'Oh, YEAH!'?
-Omar
I checked out the search engine. I would think that if they are selling a robot that claims to be able to index the entire web every three months they would have an online database to prove it.
:)
try searching on slashdot. You get one link which is at least 2 years old
Dazzle them with bullshit.
Fish! LipHo
If you are very tired of your crappy ISP (like golden triangle) hanging up on you because you didn't load up a webpage for 5 minutes (you were trying to read the one that just loaded, dammit!), run this search engine in the background - It just looks like you are browsing the web! No more random hangups! :-P
The green-on-pink text in the search box reads "Try One or Two Keywords in ANY International Language". So I, being a Typical American, tried English. Single English words (or any single search term, for that matter) work fine. However, using two keywords (be they my name, "Microsoft Windows," "carrot cake," etc., etc., etc.) just returns the home page all over again.
So you can only search on one keyword at a time, it has a butt-ugly page, it doesn't return relevant links, and it has a horrible domain name to boot. What a waste.
Oh wait, it's GPL'ed! Hooray! Down with the software monopolies! We'll take over the world!
Groan...
For more information, click here.
>How many zillions would install Linux to play it?
/. population. But to the mass games playing market it is. Damn tricky stuff. To most minds a game, no matter how great, is not worth learning and installing a whole new OS, especially one like linux.
no zillions. Linux is hard, people. Maybe not to the regular
I thought it sucked didn't find anything i was looking for fuxorit use GOOGLE !
2 questions... Why was that funny and why was that posted? I mean, obviously you're not going to serve 10 million hits/day with a dialup connection. Figure out large the smallest possible page generated by the engine is (probably one with no hits), and multiply that by 10 million.... there's what sort of bandwidth you need.
true - but most people aren't "gamers" they're just people who play games. A big difference. Sure, you and I may have been inside the back of our machines with screwdrivers more often than we'd like...but that would scare most people half to death. fdisk might be your friend, but most people are scared to defrag in case it breaks something.
getting off topic here, alas
Has anyone mirrored the FTP site yet? I'm downloading at 4.2 k/second.... and this is at work where i'm more used to 150+ k/second at this time of day.... I'm very eager to check out the database format, but it seems i need to first download at 50 meg file...
How can it index the entire "800 million webpages" out there and only find 19 hits for "xml"? And the Specification wasn't one of them.
Not one single hit for "wide open beavers"! And the colors are just awful.
But why is it when I search for "ugly webpage" I get a the Juggernaut Technical Support page?
:-)
Oh, I get it, I got EXACTLY what I searched for!
Examination of their ftp distribution site reveals this is an early work in progress...most docs are "under construction," and even their helpers.txt (supposedly giving credit to others) is basically empty.
I'll post more if/when their src tarball ever finishes downloading (54M - whew!...and the site is getting /.'ed right now). My guess is they drew heavily from ht://dig, WAIS, SMART and other public-source search engines and spiders.
For those who can't get through to the site: they hope to sell subscriptions to their database, so that you can run their search engine internally. It's not clear whether they intend to license the spider/crawler or just the database.
Meanwhile, to those who have complained that easy searches turn up with nil results: read the page, dudes! It says clearly that you're searching a minimal test collection, but can search the whole thing (on your local system, seems like) for a subscription fee.
Credibility break: I'm an information science professor and design/evaluate alternate information retrieval systems.
Honestly, the search engine is pretty poor in quality right now. It seems to have only indexed a small proportion of publicly accessable web pages. BUT, to paraphrase on another poster's idea, the open-source nature of the project can be profitably combined the whole Beowulf cluster and distributed computing concepts such that a far greater number of web sites could be indexed than previously possible. Moreover, with distributed spidering, more in-depth analysis can be run on the text contained in the pages. However, I do see 3 major problems with this:
-The bandwidth that most users have is not commiserate with their processing capability
-The index might become stale if a site is not visited repeatedly, but in the distributed spidering case, this risks either duplication of work or gaps where nobody visited the pages in a large web site
-The ability to rank pages based on "relevance" or "linkability" (ala Google) is decreased in this scenario
--
Flames? Think I'm a karma whore?
The Juggernautsearch Engine crawler is the first fully automated crawler that can reindex all 800 million World Wide Web pages every three months fully available to the public for a nominal two year subscription fee.
.02
With the search engine being GPLed it still relies on a subscription service in in order for it to function. It mentions nothing about the crawler needed to create the database, but it also mentions that you are free to create your own database. Is it just me or is this a contradiction.
For the smallest subscription it gives 1.6 million urls at $100 a year. This price goes up to $500 for 10 Million urls.
For such a useful program, it is limiting itself to its own database which costs money to use.
Just my
That's for Managing Gigabytes, and there also is a great book (note that there's a second edition out now) with the same name on the topic from Witten, Moffat and Bell. Very well written. Go to http://www.mds.rmit.edu.au/mg/intro/about_mg.html to learn about the software, including links. It also has a Freshmeat appindex: http://freshmeat.net/appindex/1999/09/09/936885957 .html
BTW, I'm not associated with the university, the book or whatever. I just enjoyed reading it.
Making this a distributed effort would only be useful for a clustering environment ala beowulf where tight syncronization would be needed to prevent machines from revisiting the same websites. Other than that, distributed processing for web crawlers is... dubious.
However, it really does not work when you would like it to find pages that no one points to. Those unique pages are well hidden from crawlers, even those you can e-mail all of your friends about them. Until one of your friends puts a link on his start page, you're immune.
For an organization, it's the wrong avenue of approach. Organizations tend to keep their internet files on a small set of machines, in very specific directory structures. The best search engine for those machines should have permission to look at the directory structures and go through every file in them when it uspdates it's database. This insures that every file in that organization is collected and that no links going outside the organization are followed.
Ken Boucher
No Zen is good zen
Those other sites aren't just search engines/indexes, that's why they're called portals. Have you seen Yahoo! or MSN lately? Shopping, auctions, ticketing, weather, scores, communities, calendaring, gaming, stock quotes, maps, make-your-own web pages, chat, news, e-mail, messaging, etc., etc. etc.? And you think Juggernaut or anything similar is going to make these companies go away? Borrowed time? Good God, man, I've got some beachfront property to sell ya! :)
Cheers,
ZicoKnows@hotmail.com
This is typical GPL software. Very slow and unpolished. Try typing in a search term where it doesn't get any hits. Does it say "No web pages found."? Nope. It just goes back to the search page.
Also, it hardly finds anything and is extremely slow to return results.
Gee you must have read today's DaveNet.
It doesn't have the scalability being boasted about, but it isn't meant to, either.
Well, alright, so it can do 10 mil a day on a PII with what? A t3? How much bandwidth do you need before the processor becomes the limiting factor with this engine? I certainly dont think my 26.4 connection at home can handle 10 mil pages a day. They should make some mention of that on the page.
On a side note, I was very dissapointed when a search for "deez nuts" came up dry.. oh well.
//Phizzy
"Most European technology just isn't worth our stealing," -- Former CIA chief James Woolsey, referring to Echelon
Unlike Juggernaut, it's a complete search engine system (crawler, database & front-end), it was developed over a long time, and has capabilities that even most modern search engines don't (such as relaxed spelling).
IMHO, it would be better for the Open Source community, as a whole, if someone picked up Harvest, modernised it and maintained it. At present, it's the best "openish" source Search Engine out there, and it's going to waste.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)