Slashdot Mirror
Searching the 'Deep Web'
abysmilliard writes "Salon is running a story on next-generation web crawling technologies, specifically Yahoo's new paid "Content Acquisition Program." The article alleges that current search services like Google manage to access less than 1% of the web, and that the new services will be able to trawl the "deep web," or the 90-odd percent of web databases, forms and content that we don't see. Will access to this new level of specific information change how we deal with companies, governments and private insitutions?"
I bet you this new 'Deep Web' search technology would be something that does not observe robots.txt...
[Something witty and intelligent should have appeared here.]
{Traicovn}
Of course, it's nice to know that the content's there, but how many children are now going to be able to bypass the disclaimer pages on porn sites because of deep linking?
I could care less about Ticketmaster whining out their deep linking, but there's probably some stuff out there that if it isn't taken in context to their intended point of entry may have other problems.
I'm afraid that this is going to give people more reason to go back to using frames, and 'detecting' if their content has been hijacked, and writing more bad code that causes multiple windows to pop up all over the place, and/or crash browsers.
Build it, and they will come^Hplain.
Those of us who place our faith in the Googlebot may be surprised to learn that the big search engines crawl less than 1 percent of the known Web. Beneath the surface layer of company sites, blogs and porn lies another, hidden Web. The "deep Web" is the great lode of databases, flight schedules, library catalogs, classified ads, patent filings, genetic research data and another 90-odd terabytes of data that never find their way onto a typical search results page.
There is a reason for this: a Google search should turn up pointers to the items in the so-called "deep web" (*gag*). To use one of the examples above: if I am looking for information on patents, the search terms I use should point me to the US Patent and Trademark Office. It shouldn't have to point me to all 12 bajillion patent filings.
Besides, what makes anyone think this is going to fly after all the hubbub over "deep-linking"?
I want to drag this out as long as possible. Bring me my protractor.
Oh okay, so your testing directory should be indexed? The best place to test is to actually have your files on a server. The easiest way to do this is to just put it in a "test" directory or something on your server. A simple line in your robots.txt file and that test directory does not get indexed.
It would be a pain in the ass to have a test directory require a login and password all the time (if you don't want people to look at it BUT robots.txt doesn't work anyway).
Generally, google finds the pages that the authors want to be searched. Thats why you submit your site to google. Even if you dont submit your site to google, if it's on a domain that google searches and there is a link to it, it'll be found.
With google storing more than 4 billion web pages, I'd hate to see what kind of crap the other 99% is.
Perhaps they count each iteration of a dynamic page as a seperate page? Even so, google's news page does a great job searching in real time for pages that change dynamicaly.
http://github.com/gbook/nidb
1 percent, and I still don't have a problem feeling lucky almost every time I do a search on google.
zWhat would an EWOULDBLOCK block, if an EWOULDBLOCK could block would? -- me
Judging by the problems with relevancy that often occur in current search engines, (I think of the problem with meta keywords, which for many search engines are now completely useless, and google-bombing) why would a customer pay to add more data to the search engine? The idea of course is 'because they'll be more relevant and because they have more information will come up more often', however, if search engines start searching more and more of this 'deep web' how badly will relevancy be affected? I mean, the more data that is in there, the more chances there are of relevancy being broken, and if the weighting is in favor of this 'featured' searches, then relevancy may be even more broken. Sure, these companies will have more traffic directed to them, but will it merely be useless traffic by frustrated users searching for something else?
I run a search engine for an educational institution, and I will admit, Google misses a significant number of our documents, on the other hand, some of those documents are scripts that when queried will create an (virtually) infinite amount of data (calendar scritpts, etc). How deep do we really need to go though? Do we really need to include calendar entries for the year 2452?
I'm also confused, is this search service 'pay by the searcher' or 'pay by the content provider'. It seems to be content provider to me.
[Something witty and intelligent should have appeared here.]
{Traicovn}
One limitation of Google is that fact that a site that bases its navigation through a drop-down menu or submission form (i.e. choose a section from the list and click Go) cannot be spidered by Google.
Personally, I find this infuriating. A site I once worked on was available in numerous languages, which could be chosen by choosing from a drop down list box. The upshoot of this is that Google has only cached the site in English, meaning users who would use the other languages do not get my site returned when they search in Google.
We need an open-source alternative that can address these problems, as well as get rid of the security concerns and mysterious methods Google uses to rank sites.
Patriotism - the last resort of scoundrels.
There is reams of stuff in there that a search engine can't see. XML could be used to deep search these entire databases, rather than just the stuff that's pulled into the UI by the PHP code.
Sounds to me like that IS the point.
I don't need a search engine to index the interfaces, I need it to index the DATA.
Now, I'll admit, it would be a nice bonus if it can then map that back, and turn its search results into a link to the data via the desired interface - but I'd settle for just getting the data.
I wish you luck using that credit card number without the appropriate expiration date. The FUD spreaders rarely mention the fact that exp dates are almost never stored with the numbers themselves.
occultae nullus est respectus musicae - originally a Greek proverb
So are you implying that you're credit card information is currently availible on web pages, with no password protection, and the only thing stoping hackers is that it isn't listed in a search engine?
I think it means that the crawler will use your credit card.. so you can just search for terms like "What did I buy today", or "If I was going to buy something -- oh? I did buy something?"
This can't be done reliably with current technology (i.e. google)
It's rather stupid, but it has to do with legal practices.
If you have no warnings, then someone can claim that you forced your content on them, and they didn't know what they were getting into, and it was offensive.
By putting up warnings, which inform the user that they shouldn't enter your site if it's illegal for them to do so shifts part of the burden of responsibility to them, and away from you.
So, if you're sued for having distributed offensive material, you can claim that you provided warnings, and that the person chose to disregard them. [Sort of like putting up 'wet floor' signs -- if someone gets hurt, they made an active decision to ignore the sign]
Build it, and they will come^Hplain.
Huh?
It's safe to guess that the exp'y is within 4 years. (otherwise, move onto another card)
That's an amazing 48 possible "passwords" to brute force (assuming that cc subscriptions dates are uniformly distributed. any research on this?). I *THINK* there are >48 web merchants... Hmm.
This, of course, doesn't use the resources mentioned in the other posts.
Hey cheapskate. Maybe if you subscribed to Salon you wouldn't have that problem. Independent news sites like Salon are going to disappear if they get no revenue. Maybe next time you visit salon.com, it'll say "Thanks to our former subscribers for the support. Due to our operating costs going through the roof but only four people subscribing, we've been forced to go out of business. This domain was bought by Fox News in bankruptcy proceedings. Click here to go there now.
If you're too cheap to pay for anything, you have to be satisfied with things like ad-supported internet access (see NetZero) and ad-supported news (like salon's day-pass, and fucking TV, where's the complaining about CNN?). Yes, the ads are more intrusive than they were in 1999. The venture capital investment is gone and advertisers won't pay jack for barely-there banner ads. Now they want your full attention for a moment. So WTF is salon.com supposed to do, just say, "Everything is free! No ads! When the bandwidth bill comes, we'll just mail them some monopoly money"??
If ad-supported websites didn't exist, the only people who could afford to publish on the Internet would be the conglomerated media who make their money from--say it with me--ad revenue from TV (etc.). Get it yet?
Now, Mr. Troll, get back under your bridge.
If you have a public mail server, you deserve any spam you get...