Slashdot Mirror
Search Engines-Does Obscurity Prevent Exploitation?
GeekLife.com asks: "Search engines refuse to release (and often change) the exact criteria that determines their ranked results, presumably both to prevent competitors from stealing their techniques and to stop (or at least make less successful) attempts at "cheating" - optimizing a site to exploit these criteria, resulting in a higher ranking than it deserves to be. Is this an example where keeping the specifics a secret actually improves the tool? Or would releasing all the rules result in enough feedback ('given enough eyeballs...'), honing the criteria towards unexploitable results?" Interesting though. Can current systems be improved to give better results or have we reached an 'accuracy limit' as far as keyword-based searching is concerned?
err did someone say unexploitable? name me one thing that millions use that has never been exploited that everyone knows how it works and I will show you something not worth being exploited. P.S. If anyone releases a good way to exploit the rankings on google I will kick their ass
there is only a finite number of ways one can use the word 'sex' as a keyword.....
Feed The Need[goatse.cx]
Since there's a strong incentive to get your site listed in the search engines, the search criteria will always be exploited.
A friend of mine left the company I work for and started making porn pages for an australia based porn company.
He is supposed to make 400 pages per month, all somewhat different. He gets a bonus based on how many hits are generated, and a commission based on signups from his banner ads.
He's doing pretty well financially
********* sig: If you don't like the law, get filthy stinking rich, and buy a better one.
The only way to achieve true search engine accuracy is to have an actual person search for pages on request. Why no company has thought of this, I'm not sure, as this could certainly be an explosive business opportunity here. The difficulty of finding trustworthy information on the Internet is legendary, and I'm a sure plenty of clueless newbies would pay a monthly fee to get better search results.
personally i'd like to see critera opened so i can get more hits on my "bee arther naked" pr0n site.
1337
I think the many-eyeball argument doesn't apply here, because the it's not about finding bugs in the rules, but preventing cheating. When a site decides to "cheat", it doesn't exploit a bug. The scoring system is not like a kernel, where you know exactly what should happen, it a (generally) complex AI system. These systems are designed so that they work well enough in 95% (or 80%, this is not the point) of the time. There're going to miss a couple percent, but what else can you do. Now if you have access to the rule, you can make sure your site uses the 5% errors to go on top of the list. Unless someone thinks he can have 100% accuracy (how do you measure accuracy anyway!), the scoring rules shouldn't be released.
Opus: the Swiss army knife of audio codec
...then we would get the most relevant listed first, and not the most exploitative.
"A witty saying proves nothing." -Voltaire
So far I haven't read anything that has "exploited" Google's technique, which is popularity based. Will someday some website create 1000 pages that each have a link, or links, to one site on each one? I don't think Google would be able to seek them out within the sea of pages in its billion-sized index.
I think that we may have reached an "accuracy limit" with search engines until such time that people don't mind search engines leaving cookies on their hard drives, so they can examine a user's past queries and use those to try to present more relevant results for that users current query. I really think that will be the only way for them to grow, because most search terms I've seen (basically, referrer logs for my site and few other sites i've worked on) only consist of 3 or less words. It's a rarity that someone enters more than that, so that doesn't give a search engine much to work with...
However, if say google knew that I'd done searches for "albini" and "shellac" in the past, it could probably surmise that when i did a search for "big black", i'm actually looking for Steve Albini's first band, and not BIG BLACK BOOBS, et al...
I can't figure how else something like that could be accomplished without a sacrafice of our hope for privacy...
Tweaking meta tags to ensure high placement in
search results is dead, dead, dead. AltaVista is doing a rank-by-number-of-links thing, just like
Google. ("Raging Search" and "AltaVista" are the same
information, just different layouts.)
-- Real free software sites don't use GIFs.
I've plenty of useful, informative, and yes trival(but still interesting) sites becuase of the inaccuracy of search engines.
Ahh, A nice legally binding electronic signature...
Straight from the link given in this google article posted just recently.
"Last November, as reported in Google likes directory sites, I discovered that Google had the uncanny ability to sniff out high-quality, but little-known directory sites. As I discussed in that article, Google was able to do this because it ranks sites according to how many people make links to them, and smart people everywhere learn that directories are important, so they make many links to them."
Then one can wonder about that actual article (Google Propping Up Yahoo In Search Results?), which is not quite as good news. But it still rocks. Plain and simple.
Google is the only search engine I use. Well except when Im lazy and I get me a WebBITCH (webhelp) . . . hehe.
/nutt
It's already hard enough to get to the web-page you're looking for without having a bunch of porn site operators or script kiddies skewing the results by embedding background text. If you know the code for the search engine, then it makes it that much easier for them to do so.
The last thing I want is to always, 100% of the time get the 'wrong' web-pages no matter what criteria I use to search for them. I get enough of that as it is.
Fortunately, the search engines keep evolving to try and take care of this. Unfortunately, it also seems to be easier and easier to divert search engines because of holes in the evolution of the browsers or new options allowed in the code or script for web-pages. Hopefully, the search engines will win this fight.
Kierthos
Mr. Hu is not a ninja.
IF I EVER MEET SOMEONE WHO HAS RELEASED A GOOD WAY TO EXPLOIT THE RANKINGS ON GOOGLE, I WILL KICK THEIR ASS.
Note: This is my lame attempt to be funny.. TO answer the question, how about making a search engine that is free, and NOT advertising based? Maybe if the bandwidth was free..
/. is a commercial entity. goto slashdot.com
I like the concept of ranking pages by the number or pages referring to it. That seems to be the best method of keeping it somewhat difficult to cheat. The only problem is that newer material will take longer to reach the top, regardless of the relevance. It would be nice to have a search engine that could record poeples input regarding the results that were found. How many times have you searched for something only to find the best result buried five pages down. A method to go back and vote the ranking results yourself would seem appealing to me.
With enough experimenting someone can find out how the system works. Either through keywords,page text,bribing .. etc whatever. People will find out how it works. Just a matter of time
Well, while there are user-submitted lists of sites (yahoo,whatever) I think it's just about time for a moderated search engine.
The users could submit links in different subjects or categories with different keywords adding them to the harvested ones and, most important, registered users would be able to get x moderator points a week and vote down spam links or links that don't make much sense with the search one conducted.
Add a healthy dose of meta-moderation (maybe three levels) and some obvious anti-cheat prevention techniques and it should work much better than a normal search engine.
God knows many times even on google obviously poisonous sites come up in the search, it would be so nice to have a button to click to moderate down the page or the domain itself...
-- the cake is a lie
Would the search engine source code show how much money sites have paid to be listed in the top 5 results?
It would be hard to cheat at that criteria.
Don't forget that Friday is Hawaiian shirt day.
Though the search engines refusal to give out how they rank a site may disagree with the whole "open source" movement, it does serve a purpose. We can guarentee that every person out there would do their best to exploit the system, and we'd all end up getting porn sites pop up when we searched for "vacuum tubes" (that's actually happened to me) _ The world can't end today because it's tomorrow in Australia
Let's presuppose that their ranking criteria are reasonably accurate. If you search for "girl's soccor" (sans the quotes, since most people don't use them, or know the option is available to them), or something pertaining to female gender in a non sexual sense, why are most of the top results pornography related material? Is it just because there are so many adult oriented sites, or is it because they have some sort of technique that allows search spiders to place them in all sorts of unrelated catagories, say using meta information?
-- From my Best Friend (Written to me over ICQ): "i was gonna go to a party...but i had to reinstall windows"
Like Seti@Home, whose developers chose to keep the source closed to prevent spoofing results. I'm all for open source of pretty much everything, but sometimes there's something that is better off hidden.
-The Tempest
Here's an idea: rotate the criteria between several open-source methods of sorting. Design each to both low-score attempts at exploiting the other methods, while still maintaining an adequate order for people doing searches.
So, at best, a company could be really popular on occasion, but suck the rest of the time.
---
I once did a search, and it contained an acrynom that doesn't really occur anywhere else, I was just looking for the url, and I did a search with it on altavista, and it was on about the third or forth page! They don't have any good search engines really.
Sure, you could pay people to link to your site -- it's done all the time. But only pr0n sites and the Big Portals are gonna have enough coin to make that a factor in a generalized site. If your site is at all focussed or specialized in nature, you're not likely to have the funds or time to pay people to stay linked to your site.
And that brings me to another point: with sophisticated enough keyword ranking algorithms, it'll become more of a pain in the ass to come up with spam that makes it through the filters than to simply put up a good site in the first place. 600 repetions of "sex" are easy to pick out. And if HotGrammar 2.0 can pick through my dangling participles in a reasonable amount of time, then it can't be too much more difficult to point it at a website and say "Does the content match the keywords?"
At that point I think you really would have some reasonably close to a non-hackable search engines: The Google Algorithm to pick out the sites that people have blessed with links, and The Grammar Nazi Algorithm to make sure there's content to match.
Carousel is a lie!
If your algorithm is good, it can't be exploited. Google's algorithm, which is quite well-known, seems to work quite well. For that matter, dmoz.org isn't doing badly.
What I'd suggest, though, is a compromise. The basic popularity rating is based off of number of links, like with Google. However, people would be able "rate" the effectiveness of a given site when it comes up in the list (nothing fancy, just "relevant," "neutral," or "irrelevant"; maybe a five-point scale rather than three-point). No individual vote could do very much to the system, but as votes add up trends start to show, and relevance ratings can be modified based on this. This would require a user registration system to keep track of moderations (though definitely not searches), but so long as registration isn't required to actually use the search engine itself I fail to see the problem in it.
----------
The biggest exploit I could think of for this, would be for a big company like Amazon to pay a lot of people to link to their site (oh wait, they already do). Thus they can use money to skew the results, without having to corrupt Google directly. Hmm.
"Destroy science and religion. Science would re-emerge exactly the same; but not religion." - Penn Jillette, paraphrased
I guess this would defeat the page-rank (page vote) idea that google uses. Although as no one's seen their algorithms I doubt they wouldn't have considered this.
--Giving to trolls for the benefit of us all
Well, also the fact that a huge chunk of the web isn't even indexed at all.
Other than that, though, the interfaces that most search engines use are pretty bad. There is usually no way to filter through a set of results to eliminate things that are obviously not what the searcher wants. Just being able to eliminate a set of domains from the initial results would make a huge difference for me.
Also, most people have no clue how to effectively use search engines - and they're not all that interested in doing so. I've been working in the web industry for quite a long time, and most of my colleagues seem to have no idea that changing the settings can yield better results. The setting 'phrase' for instance, makes a HUGE difference much of the time - yet I've never seen a colleague change any default settings when doing a web search. If you're not willing to do so much as even toggle an individual setting, you deserve the crappy results you get.
Oh, another thing - many of the links I get back are of dubious quality - even on the setting 'phrase', many results don't come back that match what I specified. If you play the the rules and the results STILL don't match, I have little faith in ANY results, even if the web site operators are trying to override accuracy. This is aside the very common result of '404 not found' pages.
Right now, the best search engine I know of is a meta search engine called 'ProFusion' - I've had much better luck with it than with Google. Not enough control over Google...I also like that the results with Profusion ( http://www.profusion.com ) come back with an option next to each result to open in a new browser window - now THAT's a nice idea!
How about a 'vote' icon? Not for each and every click, as no one will do that, but rather a 'demote' button next to a link. You can keep on organizing results by the number of people who click on the link, but if someone clicks through and finds something other than what they're looking for they'll be heading back to the search anyway, so they can give a 'demote' button a quick click as they continue down the list. If you made such a system immediately detract 100 clicks from that site, misleading links would soon be phased to the bottom.
After all, more people like to complain about a bad link than to promote a good one. Let human nature work in our favor, for once.
---
Two points:
Since from when I can first remember seeing banner adds, I can also remember seeing "please click here to support this page" right below them. People often end up clicking adds in order to 'support' a site and generate money, rather than being interested in the content of the adds. If people can exploit something for money, they will.
Secondly, banner adds are what give people incentive to cheat on search engines. If they can get more hits per month, that directly translates to more clicks (or impressions) per month, and more $$$. In today's society with the commercialization of the internet and 'dot-commies', I would bet money that the resulting information would definately be used to make somebody a quick buck.
That being said, I would wager that the final result would end up being more reliable search engines, but the potential problems may or may not be worth it.
Be the Ultimate Ninja! Play Billy Vs. SNAKEMAN today!
Some really good points by previous posters that I want to recap:
If you open up the criteria such that *everyone* exploits the criteria, then there is no discrimination. When the criteria is closed, only those who have found the exploits can get increased exposure, making it inherently unfair.
Another issue is that what a search engine wants you to see is different than what you want the search engine to give you, in some cases.
We want the union of two criteria; the results that give the search engine the most use/reuse(usefulness of the search) and the results that give the search engine the most financial recompense(so that the search engine can grow, get better, get faster, etc)
They may not be correlated, but they are both very important. The most useful pages may not give them the most money, and the pages that pay them the most may not generate enough repeat use for them either.
Perhaps the best search algorithm is two step:
Rank according to links (the more links to a page, the more useful the page)
Count repeat use (the more times a search has to be refined, the less useful the pages returned)
Rank according to links already occurs at Altavista and Google.
I don't know that anyone does the second.
Say you do a search on Google; if you hit the next button, then the pages that were generated get knocked a few points. If you hit Google again a few minutes later with a variant search, then knock a few points to *all* the pages that got listed in the previous search. If a user goes back, and hits 'related' pages, increase the points to that page, and all the related pages. Repeat the above algorithm for every hit to Google.
The nick is a joke! Really!
GPL Deconstructed
Having many-eyeballs thing could improve ranking algorithms. However, the more people know about the algorithm, the more complex the algorithm is going to have to be to defeat cheaters. It's a catch-22. Eventually the algorithms would be so complex that they'd have to render the page to determine the relevance of different elements and parse out sentences to determine if they're gibberish or not.
So what we need is to work on developing an open, complex, and nigh-uncheatable algorithm while search engines continue to use their own closed methods.
-- dR.fuZZo
Perfect example, wouldn't you say? IIRC, Google rates their sites based a good deal on how many other sites link to them. That is going to be non-trivial to hack
Nah, easy. You ever look into reigstering a domain name. Notice how there are all those bulk registration specials, like "only $9.95 each if you register 250 or more?" That's because there are a good many web companies out there with dozens or hundreds of domains. In the case of porn, maybe thousands. It's pretty easy to cross link everything so you look more popular than you are.
Just thought of a variation on my other idea: how a bout tracking users, via their IP address in each session, and determining whether or not they jump back to the search page immediately after following a link?
If a user is still searching after a minute or two, he or she obviously didn't find what they were looking for.
---
. . . "have we reached an 'accuracy limit' as far as keyword-based searching is concerned?" and he didn't really have a conclusive answer, but was able to suggest a places where I could buy foundation and how to apply it.
If you visit the link, add more relevence points to it.
If you hit 'next', decrease the relevence points for all the pages returned to the user.
If you hit 'similar pages' then increase the relevence points even higher for this hit!
If the user refines the search, reduce the rp for all pages that were previously generated.
Of course, I don't see any search engines using this criteria, yet!
The nick is a joke! Really!
GPL Deconstructed
In 1994-96, I ran one of the first-gen search engines (the Open Text Index, r.i.p.), and made my living in a search & retrieval software company for years.
If you look inside the source code for any of the engines, you'll discover that the result rankings is an unholy stew of heuristics layered on linguistics layered on guesses. Among other things, the world isn't in English and there are lots of language-specific techniques. Furthermore, there are people who fine-tune this all the time. Furthermore, the code is shot through with special-case handling, and all sorts of boring tedious stuff to stave off word-spam and <meta>-spam and litigious organizations and so on.
There's no doubt that Google took a serious step forward when they started working the input-link count into the result ratings in a serious way. Works for me, anyhow.
I guess the upshot is that the search engine's source code is the only meaningful specification of how the rankings work. Which probably means that the info won't be in the public domain until they start going Open Source. Which would probably be a good idea, but their management and investors might not see it that way.
-Tim
Why not...:
Increase relevence points if a link is followed
Decrease relevence points if a link is ignored(next is hit, instead)
Decrease relevence points if a new search is defined(none of the prior search were sufficient)
Increase relevence points if 'similar pages' is followed
It should behave something like what you propose, without additional cookies, work, voting, or otherwise. Other than normal behavior!
The nick is a joke! Really!
GPL Deconstructed
Any ideas on how to thwart that?
Carousel is a lie!
In a slap to the face for the record industry, popular alternative rock band "Smashing Pumpkins" has released their supposed last album on old school vinyl and MP3 format via Napster. There will be no CD release of the music through its label Virgin Records according to sources at CNET. Only 25 vinyls were made, and included in the records was a note that stated the ploy was to be a "final f--- you to a record label that didn't give (The Pumpkins) the support they deserved," according to Sonicnet. The new album, entitled "Machina II", can be found all over FTP and download networks.
So fuck yo all, slashdot wont report on this now that its broken here before it.
If your trying to make a search engine that isn't easily exploitable how about identifying how current engines are exploited and design around that....
One way is to include the keyword many many times in a comment tag. A possible solution is to grade the keywords via their entropy with the words surrounding it. Such as testing for repeating patterns in a comment tag. Hell use the HTML tags to help you out by not grading anything in HTML comments. Another way is to do some syntactic analysis on the content of tags like and if they are not unique then they are only counted once. Certainly people with bad grammer skills and languages other than the language intended will suffer but you can add verification critera for each language you want to index. Before the trolls hunt me down and say this method is censorship (americans love to say anything is censorship) then you should just stick to the simple useless search engines we have now. Plus one can always implement this as a searching option to an existing broken wheel.
This isn't really that hard it just requires a bit cleverness and lots of prepratory work.
Or maybe I'm just on glue. :P
"Survival of the fittest Max, and we've got the fucking gun!" - Pi
It seems like the problem to be solved here is, in reduced form, telling the difference between pages that are actually more relevant to the search criteria than others and pages that are trying to fake being especially relevant. That sounds like an AI-complete problem (actually it sounds a whole lot like a web page Turing test). In that case obscurity might be the best model, since the best solutions will probably be exploitable. (Actually, even real intelligence is probably exploitable.)
I was reading these posts and started rolling around ideas in my head for scripting a little meta-search engine that learns from your searches and is customizable, when I realized that such programs already exist (sort of). Search Rocket is one of them (I know there are more). It searches all the big search engines at once, lets you filter and sort results and save your work in a nice xml file for later. I use this tool when I am doing in depth research and need to come back to the links I found later (but don't want to shuffle bookmarks for an hour). Still, it would be even cooler if it did some of the smart auto-filter type stuff mentioned above.
Oh, and just to be completely on topic, if the big guys change their search results, Search Rocket needs a quick patch. So in a way, they could prevent tools like this from working by changing the results a lot, but I would think they want to keep it standard due to cross licensing etc etc
Ok, enough rambling. Time to go home.
The biggest problems with Search Engines, is relevancy. The problem being that when I do a search for a word like "magic" the search engine will return results based upon its algorithm, but trying to produce relevancy from a single search word is just about impossible as a task. With a term like "magic" I could be looking for:
Or any of a large number of subjects that I could have in mind at the time of my search. The results from a search engine such as Google, will rank pages which contain the word magic in the page title, multiple times in the body of the page, in the META tags, in or near HREF links, or which are linked to by many other sites higher than those which do not meat these criteria. It differs from search engine to search engine, depending on criteria.
None of these criteria for ranking take into account the nature of my query - what I had in mind when I did the search. In other words they do not directly address the relevancy of the results. If a search engine offered me the opportunity to pick from results it returned and gradually refine the search to produce better results it would be addressing this situation. Some do with a "search again in this result set" or "more like this" type option on their results pages, but its still kinda mechanical, and not all that reliable.
I think it will take some sort of AI analysis of search requests based on user-feedback of some sort and with a learning capability to surpass the current crop of search engines. Until such time as we have some smart systems working behind the scenes on searching any improvements will no doubt be incremental rather than radical.
Now, as for keeping the specifics of how a page is ranked secret I think its absolutely necessary. There is a constant, quiet, war going on between the search engines and the folks who want to get their websites listed at the top of the page when a result set is produced. The people who regularly submit their sites to the various search engines, with each search engine receiving a specially made page generated just for its benefit to ensure that the website gets the best ranking possible etc, are not interested in how accurate the search engine is, they simply want to come up first. The folks at the search engine generally want the most relevant pages to be returned. There is an essential difference of purpose between the two camps.
On the side of the search engines, they have control over their ranking system, and change it peridically to prevent abuse of the system. The folks who are seriously trying to get to the top of the heap in the search engine results are constantly trying new methods to get ahead.
For instance, at one point some webmasters were creating their webpages with a lot of text at the bottom of the page that was the same font color as the background, so that the search engines would spider the contents of the page but users would never see those contents. This let them list all sorts of words that scored higher in the search engines returns, but had little or no relevancy to the page contents. The search engines got wise to this trick and now most will penalize you for using it.
Opening up the search engines ranking rules would only make the system easier to abuse more precisely. No matter how many eyeballs pour over the code, it will still not change the nature of the guy who will use any method at his disposal to get his porn page returned as Link #1 when you do a search for MP3 because its the hottest term currently being searched for.
Google has altered this battle somewhat by ranking pages higher in their results based on how many other webpages contain links to that page (and also based upon the nature of the linking page. They use a distinction between pages which contain a lot of links - like a web directory such as my own Omphalos - and those which are linked to by a lot of other pages. Both get points for different reasons and in different instances. I don't remember the details), but even this is open to abuse, although with a bit more effort required. I know of a website which has over 200 different URLs registered and operational, all of which contain pages which point back to the main URL they are promoting. When a search engine such as Google goes to anaylize this website, it will rank it higher because it is linked to by so many separate domains and so many separate pages on those domains. Its harder to abuse, but it can be done.
Of course, this is all basically irrelevant, since each of the search engine companies keeps their methodology and their source code highly protected. It is worth millions of dollars in revenue, and I cannot honestly see any of them deciding to release their software in this way.
If you have not noticed, practically every graduate student who devises a new and effective method of indexing and ranking search results ends up creating their own company once they have delivered their thesis and entered the real world. That is certainly how Google started, and I believe is also how Ask Jeeves got going. I am sure that most of the other main search engines have gotten going in the same or similiar manners.
All that said, If you want to play with a true search engine that is GPLed and works quite well, although not on the scale of a Google or an Altavista, try UDMSearch. It runs just fine under Linux or FreeBSD (I have installed it on both in the past) and I am using it on my site under Solaris. It is still in an intense development cycle and new versions are released regularly, but its worth exploring if you are interested in how a search engine works, and want to get your hands dirty.
For more information on the big boys, check out Search Engine Watch, and finally, if you are simply interested in Space, Space Exploration or Space Science, check out SpaceRef.
"The first time I got drunk, I got married. The second time I bought a chimpanzee, after that I stayed sober" Arian Seid
AltaVista has a pretty good set of primitives in its advanced search, including matching against title, the whole URL, the hostname, what sites it links to, and the text in anchors; you can also use "*" to say "search for anything beginning with this". So you could do a serch like:
I find that being able to search by title helps enourmously, and being able to use "*" saves me from having to search on variations of the same term/prefix.
Suppose you were an idiot. And suppose that you were a member of Congress. But I repeat myself.
Give a man a fire, and he'll be warm for a day, but set him on fire, and he'll be warm for the rest of his life.
usually -fuck or -cum works for me. I mean, I may be reading something about "the male sex" or whatever, but I'm not usually reading something about the word fuck.
Also, -"credit card" and -"member"
usually filter out a lot of shit.
willis/
there is no thing
what else could you want?
Yahoo released a method for exploting the rankings on Google. It is known as the "strategic alliance exploit"
It's important to note that most search engines are just like every other internet buisness, they thrive on the number of hits they get, since most of them depend heavily on advertising dollars. How does a search engine get hits? By improving the results that queries produce. Almost all search engines are so bad anyway that when I find a better one, I'm almost instantly converted. (Hence I use google all the time)
The sorting mechanism used by the search engines is their way of creating a great search engine! It's relatively easy (given enough resources) to catalog web pages. The big problem is figuring out which one of those web pages somebody is looking for.
Asking a search engine to release it's sorting mechanism is like asking Big Bob to release his secret barbeque sauce recipe in the name of improving barbeque sauce around the world.
Mad Software: Rantings on Developing So
How does this Windows (I know, it is Windows!) software relate to this story question? Any opinions? :)
Ant(Dude) @ Quality Foraged Links (AQFL.net) & The Ant Farm (antfarm.ma.cx / antfarm.home.dhs.org).
I don't think opening the search criteria will help much, nor will keeping them closed. Either some people cheat, or everybody cheats, or the Web grinds to a halt under all the wasted bandwidth from people trying.
.. so far ..) Why do we have to put up with it? Is it just that people who don't care about anything but ramming their advertisements down as many throats as possible just happen to be good at figuring out how to force us to see them everywhere we go, or is it that the whole process of evolution happens fast enough that any decent search engine has a useful half life of about a year before the indexing algorithm is totally compromised?
..
<META NAME="keyword"> tags were a good deal for a while, but that got corrupted pretty quickly. Notice how long some of the high ranked pages take to load? View source on one of those pages and see how many screenfuls of keywords appear in META tags. There are numerous other ways to crowbar the index, this is only one of the easiest -- if the search engine indexes comments for keywords as well, you can cram the same enormous list of keywords into comment lines. Neither is visible to the end user, so all they see is a long delay.
Sorry, this has been a sore spot of mine for a LONG time, and relates as much to obnoxious unsolicited email as it does to pages and pages of results totally irrelevant to my search. (Google seems to filter out a lot of this c**p out pretty well
OK, OK, done ranting, and I know I could probably force feed my own page to as many eyes as the big admongers do, but the point is I *don't*. Maybe it's because I have ethics, I don't know, but it just p***es me off when so many people don't seem to know any better. Maybe they think they're actually boosting their sales. I don't know. All I can say is they're not getting *my* business
73 de N5VB (ex-KD5BIV) AR SK
If the users could 'rank' sites as well, the the criteria of 'getting the best results' would be fulfilled pretty quickly, regardless of search-sorting.
If all the users get are Coca-cola sites, no matter how they search, and these are not relevent sites, then they will get devalued to the point of non-existence.
So any gross abuse gets moderated by the user population.
Thus any useless site, no matter how hard they try, will not be able to maintain a high rank.
The nick is a joke! Really!
GPL Deconstructed
Come visit the new MegaSearch©®(tm) Engine, the only engine to find exactly what you want, every time, GUARANTEED!
Features Include:
Never coming from a startup
IBM had PL/1, with syntax worse than JOSS,
And everywhere the language went, it was a total loss...
Searching the web for the "right" page is an intrinsically hard problem, made tougher by users typing one-word queries and deception by website authors. Information Retrieval is an active area of research for a good reason. We all know computers are REALLY stupid, yet somehow we expect them to "do the right thing" when we ask for any concept under the sun. Counting links to a page is a useful heuristic, but it says nothing about the actual information content of the page itself. In fact, if you're looking for a really oddball datum, it may hurt. A lot.
Pages I'm looking for are often not designed to exploit search engines anyway. If I'm looking for some technical documentation, it is often just stuck together by someone in a rush, or on a mailing list archive. These pages often don't have optimised META tags, etc, so some engines don't index them well.
Many people are suggesting that making the rules open is good because then everyone will exploit them equally. Perhaps, but that will just make it more difficult to find pages that don't try and get a highly rated search result.
On the other hand, I find Google works pretty well, and how it works is pretty well known.
I got the first post!
---------------- This post is the first post-first post post.
This isn't security as much as it is in the same argument base...
The arguments against "Security by obscurity" apply here.. so just insert those arguments [here] and I'll move on...
It works not by prevention so much as "reduced body count" and I guess thats the best a search engen can hope for.
When someone thwarts security thats it.. your dead...
When someone tricks a search to give them top results it's just a few websites.. it CAN be overlooked.
So say... 1 person hacks AltaVista.. it's down... blah.. 100 persons hack AltaVista.. it's still down... 1 cracker vs 1,000 crackers... makes very little diffrence... it only takes one defect and one joker to ruin your day...
But with searches... a defect becomes known and you don't fix it in time... 1,000 jokers and your screwed...
1 joker however isn't a problem.... your still online and USUALLY you still give good results... just one bad result...
You get bad results by random chance and user mistakes... so big deal...
But your expecting the joker.. once he's discovered this little trick... won't make it public....
Right now this dosn't happen...
But it's a lot to risk...
Recomendation.... sence obscurity is effective... but not perfict... give away the OLD system...
Provide a liccens that basicly says "Any changes may be used by us at any time with out notice... but only we may do this... all else is open source"
I don't actually exist.
A lot of the "preferred" listings on search engines are paid for - that is the company who is at the end of a link is paying a certain amount per click through. GoTo, and I think Yahoo and About provide preferred search results to many sites such as Netscape search and AOL. So essentially, to get yourself to the top pay more $$$. Now this is not a necessarily a horrific practice if you compare it to, say, how the yellow pages works. If you are looking for a plumber, chances are you are going to be drawn to the biggest ad on the page. And how does the plumber afford the biggest ad? - He must have a lot of business - which may infer that he has lots of happy customers (ok I know, I prefer the mom and pop businesses, but we are talking about the masses here). So I suppose relevance is determined by who can afford the most for advertising, and who can afford the most advertising is doing the most business, and they are doing the most business because they are obviously the best at what they do - ahhh, capitalism ladies and gentlemen...
+ insist on a word or phrase or meta term
- reject a word, phrase, or meta term
where the meta terms are
host:
url:
image:
link:
title:
...and many more
So, to find Signal 11 trolling on Slashdot and not on Kuro5hin:
+Signal_11 +troll* +host:slashdot -host:kuro5hin -timothy -Jon_Katz
where the _ will be treated as a non-breaking space and the * will get troll(s)(ed)(ing) as a wildcarded stem. Oh yeah, I've found the -timothy and -Jon_Katz to be useful for increasing relevance :)
Now that search might look complicated but I was trying to illustrate a lot. Just get in the habit of doing all your searches with +plus +prefixes , then add in -terms and +/- host: to clean up the results. easy.
We have not reached the accuracy limit. A search engine should be able to read my mind and infer the best sites to go to.
No, Thursday's out. How about never - is never good for you?
They don't want people figuring out how many "hits" are a result of company buying spots and not actually part of a search criteria.
A solid search criteria that is reselent to abuse can easily be implemented.
The Kruger Dunning explains most post on
had you been using netscape in linux, you'd just hafta center-click to get a new window ;)
I do not think that releasing source means for sure that the search engine could be tricked.
For instance you all know exactly how junkbuster works but NOT my settings so you cant trick me into loading your banner. So they could release (or have leaked) the code and not certainly lose the ability to control things. Just have some "settings" like (yes its too simple here) if ($times_this_word_appears > $how_many_i_allow ) { blah... }
It seems like everyone these days is trying to get their site to appear on all searches. This mystifies me, because if the page isn't titled something that looks like what I'm looking for, I'm not going to look at the page, and I'm probably going to be annoyed when a site keeps coming up on searches for other things, and I wouldn't go there even if I *was* interested.
It makes sense to try to get your site to appear first on a search that actually fits your site. But if the only sites fighting for the top spot are informative pages on the topic I'm actually interested in, I don't think that's really a problem, and it's not really cheating.
It may be that I have a tendancy to look for a particular web page (e.g., the official screen home page), and I don't want other pages on the topic (e.g., the GNU page about screen, which is non-gnu), but cheating never gets my eyes and certainly never gets me to explore a site, and definitely won't get me to advertizers' sites.
It would probably be helpful if search engines would give instructions on how to make your site come up on searches that are looking for it and not on searches which aren't. Of course, having the search engine be better at determining what the site is about without special help and making it easier to tell the engine what you're looking for would be even more helpful.
(incidentally, I've found an official screen ftp site, but no web site; it may very well not have one)
Aren't search engines supposed to remove the bad link when a visitor click on the broken link? Let's say I was using Google, and it gave me a link that I want to click on. I click on it and it gives me a 404 error page/missing page message. I think search engines should remove that particular link when I found this.
:)
Would that help a lot with these broken links? What do you think? Is this possible?
Ant(Dude) @ Quality Foraged Links (AQFL.net) & The Ant Farm (antfarm.ma.cx / antfarm.home.dhs.org).
For example when searching for something, I would like to beable to knock out any of the bulliten board 'hits' i get as there always to messed up. My other option would be to get search engines to put the BBS responses somewhere else where they are nested and threaded, so i can go to the base post instead of digging through 15+ RE: to find the original post.
Also drop out anything with reference to a certain product I.e. Im searching for something on 3d graphic algorithms i would like to drop out all references to 3DFX or Nvidia, i know by doing this i might let something slip through, however it will usually flter out alot of crap.
These things could probably be implimented quite easily with a pull down menu. The other big gripe i have is with Linux/Unix stuff, this is esp true on FTP searches. I wish there was a way to ignore any Linux/Unix/BSD mirrors while searching for something, usually its impossible to search for something general on FTP search engines b/c of this.
so he puts it in past tense and gets modded up go fig
Rather than wondering if search services are going to "Open Source" their search engines, why not start your own? If you're curious about whether many eyes are better, or you're out to prove the ultimate power of Open Source, then go ahead and do it!
Personally, I'm busy right now, but hey, there's gotta be enough people around to try it, right?
This is a manual virus. Copy it to your sig and help me spread!
Perhaps I'm wrong, but doesn't google already essentially tell us their criteria for finding the best page (eachs page is weighted by the sum weights of pages that point to it). I mean, they used to have thesis papers on their site explaining the details of how it works.
-- Superlame http://catpro.dragonfire.net/joshua/
In algorithm design, we often think about the input to our algorithm as being designed by an "adversary" -- a really smart entity which knows our algorithm and aims to defeat it. This is good, because it gives us robust algorithms and data structures which perform well in the worst case.
The solution often involves randomness. If search engines built randomness into their weighting criteria, an adversary with the algorithm would still not be able to influence the random (or "pseudo-random") aspect of the weighting.
Other things beyond the control of the adversary are usable too (some people have mentioned excellent ideas such as external links, voting, etc.).
I still believe that search engines should check for 404s when they return results (and the site hasn't been previously checked in, say, a week) or perhaps right after. Isn't it in their best interest to keep the database clean?
It seems to me, releasing the criteria would, in the long term, result in a more stable system. Not to mention preventing artificial boosting of pages from a certain portal... In the short term, however, there would be chaos. If the technology didn't settle down quickly enough the search engine as a concept might be replaced with something new. On the other hand, with the incredible power of the open-source community, things could gt under control pretty quick.
Even "good" search methods have embedded social values. For example, Google's backlinking methodology tends to reinforce traditional power structures since heavily commercial sites tend to link to each other a lot.
Search engines are in the business of controlling what you become aware of. There are lots of things that become interesting just because lots of other people are also aware of it (e.g., Survivor, Big Brother, etc.).
Search Engines don't really try to maximize relevancy; they try to be relevant enough so that you don't leave. That's why Yahoo uses google search results as a placeholder, but that's just to create more space to promote its own stuff.
Proprietary SE techniques are a bad thing(tm) from the perspective they obscure the embedded social values in their design.
Yes, this is lots of random thoughts but I think this is an important topic.
Hey democracy lovers, add Quorum as a c
You can't view Profusion with the Linux version of Netscape (V4.74). Well, you can VIEW it but you can't do anything after that because it messes up the browser (Linux Netscape users are used to this on sites with amateurish HTML). I'll be intrigued when they take pride in their work and hire authors that can write browser-nonspecific HTML. For now I'll just yawn and use the ever more annoying google.
I like using Oingo. Honestly the hits it turns up are not always great and it is tricky to use but it is not a "stupid" search engine. It attempts some level of ontologizing (ontologization?)... and its about time someone tried that. There are many times when I can't get a decent hit via one of the old fashioned search engines like google or AV but Oingo will find plenty, mostly for searches that require more than just "keywords"... where I'm searching for a related concept or idea and not just a "token".
We know that telling people what the rules are will get them to change their behavior. Game theory tells us that the set of possible equilibria may change. In short, it is not necessarily better to tell everyone what the rules are, and try to iterate to some new steady state, in which we all try to exploit, and "it all balances out".
This really is not related to the security-by-obsecurity issue, I think. It involves security in the sense of "keep your passwords secret", not in the sense of "build a system with no bugs", which seems to be where security-by-obscurity fails. The issue here is that rules which people could know about and not be able to exploit might well be significantly inferior to rules which are exploitable.
I suppose that someone who wanted to find out an answer to this could try to get a grant to set up a search engine with a public rating system, and see what came of it. If we can come up with a reasonable metric for the signal-to-noise ratio which resulted from searches, we could find out what really works. By the way, I suspect that one problem with such a proposal would be that no one will bother gaming the system unless it is REALLY popular, and delivers loads of hits. That is, I don't think that this could be done on a small scale: go Google-size, or you won't get any data that applies to the big engines. But you could find out how alternative rules work, with a site that was below the radar screens of the home shopping network and the porn queens.
I have a question for you CS grad students: is there any academic literature on this issue, looking at how people react to this sort of structure, and how the structure must be designed to get them to react in a way which doesn't screw things up? This seems to relate to the information theory field of economics.
See what I've been reading.
1) Database driven web sites make it difficult to directly index pages on a particular site. This is compounded by the fact that site specific search engines are usually ill-equipped for anything other than the most basic word searches.
2) Apathy on behalf of the user in that they never get around to actually checking out the site's 'How to use this search engine' section and rarely stray from your basic keyword search, but will bemoan the number of results returned anyway. (eg. '+host:' option on altavista)
3) Apathy on behalf of the search engine site in not providing the tips and tricks mentioned in item 2 such that a user can't break the search down any further even if they wanted to.
'sapientia potestas est'
Absolutely! There's a number of techniques already known in the information retrieval research community (relevance feedback in particular) that aren't being exploited in current web search engines, and would make a big difference. What's holding them back is usually some combination of efficiency problems and lack of a good interface metaphor for allowing naive users to effectively use the technique. As others have pointed out, most people don't even use phrases.
I think it's a non-issue whether the criteria the engines used are publicized or not. There's enough index spammers out there that any weaknesses in the criteria get discovered, exploited, and patched fairly quickly.
Dave
case 1 (open source):
search engine is written (1 year)
for (n=1;n<10000000000;n++) {
someone finds a way to spam it (n days)
search engine is fixed (n minutes)
}
case 2 (closed source):
search engine is written (1 year)
for (n=1;n<10000000000;n++) {
someone finds a way to spam it (n minutes)
search engine is fixed (n minutes)
}
ok then your [sic] infringing on my copyright! Could you as [sic] me next time before STEALING my comments for your own?
"honing the criteria towards unexploitable results?" I think this would be, as per the topic, a bad place to apply the open source theory. I can't get too into specifics regarding the theory itself (I only know what I do from context, in all honesty), but having been involved with MUDs, FPS's, and other such systems where the drive to cheat is generally very high, issuing out information on how things work is very rarely a completely effective solution. While methods of cheating-prevention are established, the methods used to cheat become refined, similar to strains of insects and virii, to accomodate. Sometimes it ends in success, but very rarely. The only way I can see this working is if the people behind the engine were constantly working on upgrading the grading process. This would probably require additional manhours on the budget sheet, and I doubt you could convince management that the cost-benefit was worth it. If it ain't broke, don't fix it. If it's only a little broke, don't look at it.
Moo
Here are links to abstracts from two papers that detail the inner workings of Google. The links below lead to abstracts- from those pages you can view cached pdf and postscript copies of the papers. The first paper is a reasonably high-level overview, although it's a bit technical. The second paper is more in-depth and discusses pagerank in more detail than you most of you probably want. In any case this should give you a good idea about what goes on in a real search engine, and should clarify why it's hard to fool google.
GoTo and some other sites already do this.
.02 anyway
Check out the "cost to advertiser" link under the top 'N' results on GoTo.com...
usually it is just a couple of cents, but I've seen some site pay more than a US$ dollar for your click-through.
Makes me wonder if they knew what they were doing when they submitted their bid...
hey--what's the difference between 2 and
---
Interested in the Colorado Lottery?
Interested in the Colorado Lottery or Powerball games?
check out http://colotto.com
Perhaps one place where it might be better to keep things secret it virus scanners... If those of us who write such things could read the algorithms by which scanners look for virii, they could quite easily avoid those algorithms.
There exists a paper on the Google search engine since it is developed in Stanford- read and find out how it works.
I use the Copernic search engine at work,which searches a range of search engines depending
on what criteria you wish to search on
It even has a search engine for programming,and of course slashdot is amongst the computer sites they search for content!
Its not particulary elegant, but it does seem to produce good results.
If they told us *exactly* how it all worked, then they'd lose the ability to sell spots on the front results pages to specific sites ...
--
"I do not speak for my employers, though they are controlled from my Teddy's huge pulsating brain."
The primary reason why it sucks is exactly that it doesn't have moderation... :-) They have a problem that in some categories lots of spammers sign up only for self promotion, and their response is to reject 90% or so of those signing up. Instead, what they should do is to make sure that no individual has too much power, including the meta editors (who are not always awfully clued).
The reason why it never gets corrected is that writing a comment like I do now is considered "illoyal" to the directory.
I have a bunch of ideas for a better web directory. In the meantime, I'm thinking about a classification for skeptical resources.
Employee of Inrupt, Project Release Manager and Community Manager for Solid
Actually some engines (Mamma?) don't give the exact results but something like http://search.tld/redirect?http://sought.site.tld/ page.htm .
From this, they can range the usefulness of the result (or charge the advertiser). They could reach that page as well to check 404s, but with so many searchs, I think it would be hard on their networks.
I don't like this for privacy reasons and because I can't tell whether this redirected URL is one I already visited from another engine or not.
__
__
Men with no respect for life must never be allowed to control the ultimate instruments of death.
GW Bu
Has anyone actually considered writing an open source search engine? Is there one on source forge etc? I can see the problem with some of the more unscrupulous people trying to use the source to cheat on these searches but I tend to agree with the "many eyes" theory. Given that the method for searching does not have to implemented in the code, just input via scripts which call tell the search engine how to do the searches. This means that a truly open source engine could be built and each peice of functionality or method of searching (be it checking out the links, counting words or whatever) could be subjected to many eyes but the way it is all implemented is kept as secret as you want.
Hay, Eye think that making peephole with pore grammar skills and spelling suffer is a great eye dear. It did be "the internet spell checker" and wood force peephole two learn the language that they are attempting two communicate in. Eye fore won think that wood be a God Thing(TM). Grammar not see will back me up on this. Have you ever tried two reed sum internet sites, like sum of slashdot's poster's posts or things like www.techcomedy.com? It's amazing that sum peephole are qualified as literate. I just no this post will be ripped apart bye peephole fore spelling and grammar, heh.
Thank you.
Isn't it a fundamental precept of cryptography that any algorithm dependent on hiding the details of its operation is not secure? That a strong algorithm is strong only if it cannot be broken no matter how much you know about how it works? Why would the same not apply to search algorithms?
Thank you, that was funny.
__
__
Men with no respect for life must never be allowed to control the ultimate instruments of death.
GW Bu
The increasing use of JavaScript, Flash, Acrobat, Images, and other technologies that take human readable text and put it into forms a web search engine can't parse or understand is taking many of the "fancy" sites out of search engines.
The irony is that these tend to be the sites that are trying very hard to get to the top of search engine listings.
-m
Obscurity has its place, some people have to realize that. This is one of those places.
If you're really that interested in making search engines better maybe you should apply to one of the search companies or make a better search engine.
Refrag
I have a website. It's about Macs.
Google's caching system is great for those informative web-sites that dropped off the net, are temporarily down, or just really slow. It's also nice for sites that rotate their content, because even if the current site no longer has the info you were looking for (or they do, but it's archived somehwere deep in the site) the Google cache has it, and with all your keywords highlighted.
Hmm... You know how new users are always saying "I've got the Internet IN my computer"? Well, I guess Google really does.
- Isaac =)
"I like using Oingo. Honestly the hits it turns up are not always great and it is tricky to use but it is not a "stupid" search engine. It attempts some level of ontologizing (ontologization?)... and its about time someone tried that."
Yes, yes, yes! I just tried Oingo (never heard of it before), and it not only turns up relevant web sites, but will return a whole ontological category, with all the major sites in that category as well.
Moreover, you can specify which of several recognized "meanings" of each term the search engine should use. For example, I tried "Dylan incremental compiler," and was able to refine the search by selecting from a pop-up menu that the meaning of "Dylan" I meant was "object oriented programming language" as opposed to a "Celic deity."
Really, it's brilliant. I agree that ontological approaches are the way to go for intelligent people searching the web for meaningful information. Thanks for the link.
Most of the people here have already noted some of the problems with data collection that search engines face. Mainly, pages may have surreptitious content designed to fool search engines about their real agenda. Even Google could be spoofed by a dedicated foe, although it takes a bit more work.
While this is always a pressing concern for those people writing engines, there are other issues that might affect the accuracy of search engines. Mainly, there are certain limitations on the underlying technology, and other technologies are still in early development.
I think every major search engine uses Inverted Indexes to represent the data. The idea behind this is that you can think of every document as being made up of the following tuples: {docid, termid, position, fieldid}, where each doc has a unique id, each word in the lexicon has an id, and a fieldid can be used to indicate special fields like titles or meta tags. All the engines basically take this information and produce inverted indexes for searching which contain the following tuples: {termid, docid, position, fieldid}. Throw in some mapping tables, sorting, and some compression optimizations and you have the basic idea. When a search comes in, pull up the various document lists for each term, scan through them for matches (ones for each term), and return the best results.
This works well for large collections, but it has a few limitations. For one thing, it can only find documents by words that are in them, so relevant documents with related words are ignored (I think this is called polysymy). Also, you can have problems with synonymy (a search for "jaguar" could be a car, a team, etc.). In addition, the lexicon scales in the worst way, causing most indexes to limit the size of their lexicon, causing rarer words to be ignored. Finally, it can perform rather poorly for words with that appear often (try "market share" for example), since the term lists for these are large and require scanning through large amounts of disk. And mainly, all the search engines just tweak this model, but there might be better solutions out there.
Some research has been designed to tackle the scalability problem. k-Nearest-Neighbors works by performing a feature selection on the lexicon and pruning words that aren't really useful for searches (eg, "slashdot" is more significant than "the"). Some approaches can remove up to 98% of the lexicon without a significant loss in quality. Then documents can be represented as vectors of the remaining features and queries can be mapped into this space and the k-nearest neighbors are returned (eg, you calculate a dot-product). This scales in size nicely on disk, but you find yourself doing more vector comparisons as your collection size increases, so it's really only practical for smaller documents. It also requires that your searches contain one of the terms in the feature set, which can be a bit limiting.
Some research has been focused on capturing more of the "latent semantic" information on a document. Indeed, Latent Semantic Information (LSI) is the focus of much recent research. This technique works by feature selection and transforming documents into vectors in a semantic space that represent their semantic information (break out the linear algebra). Researchers claim that this allows you to find documents with related content, even if they don't match your terms. It also claims to solve the "jaguar" synonymy problem, but you really need to enter more than one term for it to work (and they have to be in the feature set too). While early research looks promising against test collections, its performance scales poorly and it doesn't work well against noisy collections like the web yet. But research continues.
Other research has been focused on natural language parsing in an attempt to recognize meaning of user queries and documents. This however is really tentative, and it hasn't been able to show some of the success of the less intelligent and more statistical methods like LSI or kNN.
I hope these rambling notes prove somewhat helpful. Interested slashdotters can probably find some useful primers on the web that explain this better than I can. Also, ACM members should definitely check out the SIGIR conference proceedings (the Digital Library is great).
Actually, not too long ago I remember hearing some guys in a bar going off about the latest & greatest pyramid scheme ^H^H^H^H^H^H^H^H^H^H^H^H^H^H small business.
The core of the scam was that you'd buy a web page (or server space or whatever, they were clueless ex-Amway guys) from them, and agree to link to their web pages. Then fill the pages with banner ads, and give them a cut of the profits. There was something about free internet access rolled into the deal
When I heard the words 'bulk email' I nearly snapped and threw my stool at them...
too bad it was bolted down...
my sig's at the bottom of the page.
Actually, I know a bunch of research librarians who have been snapped up by a new company out on Rt128 which sells their services for doing web hunts....
----------------------------------------------
-*- Any technology indistinguishable from magic is insufficiently advanced -*-
There IS a solution to this problem. It WOULD work, and it would make an AWESOME search engine (once enough people used the rating system). The only weakness is that it requires some sort of log-in to the search site -- which might actually be GOOD for the search site's bottom line.
Consider Amazon. They don't make book recomendations by finding the "most popular books in the world" and suggesting them to everyone... instead they recomend the books that are similar to what YOU like to read. The technology behind it is a technique called "collaborative filtering". Basically (to strip off a whole bunch of marketing designed to make it seem complicated), the idea is to look at the rankings YOU have already made, and then use the pool of people who's rankings are SIMILAR to yours as the pool of people from which to draw on when deciding how "popular" a book (or, for web search engines, a website) is.
So here's how it would work. You start with a basic set of search criteria... maybe start with Google's, for instance. Then, when people sign up for your search engine, you invite them to submit a list of their favorite/most-used websites. This gives a starting place, and from the very beginning, search order can be modified slightly by giving a slight + to the score of those sites which are frequently mentioned in favorites-lists for people whose favorites-lists are very similar to your own. (There should be an option to exclude sites on your own favorites-list from showing up in queries. Some would want it, some wouldn't.)
This gets things started, but favorites-lists won't provide enough data for a really good, web-wide ranking of sites, and it will get stale fairly quickly. The real trick is to develop a "clicked-on" list for each user as well, and use the clicked-on-lists, as well as the favorites-lists, for modifying the ranking of new sites. The clicked-on-lists could be gathered by making the links to the website go to the search site for a redirect, along the way they can be counted.
So, once the system was up and running, new users would have to create an ID when they signed in. Then they could create a favorites list, which would result in immediate "customized results", or they could just use the search site for a while, and their results would gradually become more personalized as they built up a clicked-on-list. Most importantly, a large portion of the ranking of sites would be determined by what users had listed or clicked on, rather than by keyword, linked-to-rating, or other factors more easily manipulated. And it would customize itself to your own searching preferences.
Now, one final thing. I know that this is a GOOD idea. I've been thinking about it for a long time. If anyone else reads this and agrees, please let me know... because I'd be very interested in doing it. (This is a bit long for an elevator speech, but I figure the /. readers I'm interested in have longer attention spans.) In fact, if you even just READ THIS, but AREN'T interested, could you drop me an email to let me know? Thanks.
Michael Chermside
michael.chermside@destiny.com
5715 North Ridge Ave
Chicago IL 60660
If your sites only link to each other, and there are few or no links to them from the outside, then they form an "island". This shouldn't be that hard to detect. Further, if none of the authoritative sites link into the island the credibility of the cross-links isn't high. The Google method isn't quite as easy to spoof as it might sound; I've never been directed to a pr0n site from a Google search.
--
Build a man a fire, and he's warm for a day.
Time is Nature's way of keeping everything from happening at once... the bitch.
Aeiwi don't refuse to release
the exact criteria that determines the results ranking.
It is all on the submit page
The people at Yahoo had once aspired to at some ontological smarts to their search engine. There was an article some years back in wired that talked about how Yahoo had hired some of the people from Cyc Corp. to help them develope an ontology. I remember that one of them was the co-author of BLKBS ("building large knowledge based systems").
This seemed like exciting news to me at the time but Yahoo was then, and remains now, more of a "dewie decimal system" than an ontology.
Hey, when will I get a search engine which listens to me whistling a tune and then pulls up the relevant song(s)?
Oh, and I'm on the DON'T RELEASE CRITERION side of the fence. The spammers and marketeers own enough of the web thank you very much.
BTW - I'm not sure what all the fuss is about Google. It doesn't seem to return as good results as a well formed query on AltaVista advanced search.
How did you do that?
The cure for 1933 is 1917.
Thanks for the link submitcorner.com . You will find other info on searchengineforums.com.
Freeware - Search engines ranker and analyzer - 5 stars Zdnet - http://www.aadsoft.com
Guess you didn't read what I wrote, eh?
;)
Try searching for 'babelfish translator' with PHRASE mode, and it comes up as number one.