Slashdot Mirror
Google Sheds Light On 'Dark Web' With PDF Search
CWmike writes "Google this week took another step in its effort to shed light on the so-called Dark Web, announcing that its search engine can now search scanned documents in a PDF. In April, Google announced that it was looking for ways for its search engine to index HTML forms such as drop-down boxes or select menus that otherwise couldn't be found or indexed."
An announcement is available at the official Google blog, and it contains some demonstration searches.
Hey Bill, it's great to see that they finally gave you that day pass....
Some people are only alive because it's against the law for me to hunt them down and kill them.
After reading that, I've come to the conclusion that some parts of the internet should definitely remain in the dark.
Increasing the number of items that can be searched is great, but the actual searching algorithms really haven't gotten THAT much better in the past 3 years or so.
Obviously, you can't have breakthroughs every year (or maybe even every 5 years) but search as an algorithm still has much more room to improve. I'd love to see an improvement in that, as opposed to just increasing the number of pages indexed.
Still cool though...
If you can read this... 01110101 01110010 00100000 01100001 00100000 01100111 01100101 01100101 01101011
I think you've got this wrong, to some extent. I don't think its going to "submit" to see what options go where, but more just indexing the options from forms to give a better idea of whats going on in the page - suddenly google can go "Hey, this isn't just a form, but its a form pertaining to X." and thus make their results more relevant by being able to index more of a site as a whole.
If people want their sites to be indexed, they shouldn't use forms for navigation. It's not rocket science.
This isn't about people who want their sites indexed. It's about sites that Google wants to index, but which aren't designed to be indexed. If you prefer not to be indexed, Google says they will abide by robots.txt.
I think I'll stop here.
I think it is just going to look in the contents of the controls. This would be really useful, for instance if you search for "Widget Model XJ123" it will now find a page by a manufacturer where the only place they list it is in a pulldown list that lets you choose the product to buy.
Referenced article is talking about the "deep web", not dark web.
Well yes, but it doesn't mean that no one will want to try and find it.
/b/...
Just look at
"Intelligence has nothing to do with politics!"
-Londo Mollari
It's DEEP web, not dark. This is the internet not astrophyics.
"Scanning is the reverse of printing." -- WTF?! Because of artifacts?
And isn't this what View as HTML has ALWAYS been about?
Points awarded for techtard clarity, but the person at Google who thought writing a press release aimed at techtards should be firmly smacked.
Calm down please. The guy is trying to explain the concept to a broader audience, or 'techtards' as you so pompously refer to them along with your out-of-context quote, and he's doing a fine job of explaining how it is hard for a computer to interpret scanned text. The days are gone when the web was the preserve of nerds with zero social skills. Get over it.
Drill baby drill - on Mars
I'm not sure if you got the point of this - it's about using a form of OCR to translate embedded document images within a PDF, rather than simply sucking the text out of the PDF itself, as you rightly point out is already available in the View as HTML option for PDF search results.
Scanning is the reverse of printing because, well, it's the reverse of printing. When you're scanning something, you're taking a purely human-readable document and translating its contents into a machine interpretable form. This is pretty much the exact opposite of printing from a computer.
I just started reading and it says "powerful search engines such as Google and Yahoo". Yahoo is a search engine? A Powerful one? It's an advertising index, Spam search, Ad finder? I call BS, no one thinks Yahoo is a powerful search engine!
Every time I use image search and see most are not related, I look at Google asking ME to help them label pictures to help. I feel guilty for not helping, and comfort myself knowing Google has a far better shot at image recognition than I ever will.
Never heard of either before. Looks like there's a competition going on to see who comes up with the next buzzword.
The filesystem is the package manager
A "dark web" is a private network, accessible by members over the internet but not accessible to outsiders. (A VPN is one example of a kind of "dark web".)
But as you say, this is something completely different.
for instance if you search for "Widget Model XJ123" it will now find a page by a manufacturer where the only place they list it is in a pulldown list that lets you choose the product to buy.
Shenanigans! And I've been looking everywhere for that elusive XJ123, since the manufacturer stopped producing it. How dare you get my hopes up!
Now that I think about it, I'm pretty sure everything I just said is completely wrong.
The Deep, Deep, Dark, Dark, Deep, Dark Web...coming soon to a web browser near you!
MCSE? No, sir...I don't do Windows. Yes, I am an idealist. What's your point?
Well, if it's a form with a GET request then it should be safe to request it, and it's used merely to display some information. Forms using the POST method, which performs an action, are less safe and I'd hope Google is not trying to spider those.
So the alternative is automatically generating pages and pages of links to every possible item in the database just so that search engines can follow them? If a form is the most natural and convenient interface for a human there's no reason the spider can't use it too.
-- Ed Avis ed@membled.com
Google has long since favoured PDFs - and gives them boosted results, under the guise that anyone who makes a PDF has something serious to say, I guess.
You may have noticed of late that people are wise to this - there are a bunch of sites that are embedding popular search terms / results in PDF files, and clustering their sites with adverts.
It might actually be useful for a search engine to read the product name in a pulldown as a simple indicator that the page should be penalized as content free. I would probably pay to use that kind of search engine.
What I would really like to see is OCR for mathematical formulas, and store those in some standard format. Using a standard input, like LaTeX, the engine would search for mathematical equations. Right now I find it a pain to look for a formula that I know exists, but don't know its name.
This would help bring together a lot of research that is done, but hard to sort through. Then, implement a smart system using a program like Mathematica to find variations of the equations, etc., and see where duplicates exist. Maybe we'll find that we've discovered things that weren't looked at thoroughly enough.
Nice feature, but I think it only works with PDF? I would love to see the same with DjVu as well.
How about adding the word *scanned* into the headline, just as the original headline was.
That way others won't have to read the summary going "Hey, I thought Google was searching PDFs for the last 10 years."
-David
You're mistaken.
"For text boxes, our computers automatically choose words from the site that has the form; for select menus, check boxes and radio buttons on the form, we choose from among the values of the HTML," they noted in a blog post. "Having chosen the values for each input, we generate and then try to crawl URLs that correspond to a possible query a user may have made. If we ascertain that the Web page resulting from our query is valid, interesting and includes content not in our index, we may include it in our index much as we would include any other Web page."
This is not very far removed from a brute force hack on your website. Better make sure you do proper fuzz testing
-1 Uncomfortable Truth
What if I don't want to buy a widget? I'd like to see a Google filter which hides all the product pages from its listing.
Which, incidentally, would probably also boost googles ad business since they would no longer be providing free advertising. Sales in the sponsored links, info in the search results, sounds good to me.
http://marriedmansexlife.com/
dark web.. oh geez. eternal September has only just started.
aparently the world at large loves to shit on standards and practices.
it's been a while since search engines actually returned results I was looking for. google, yahoo, msn, metacrawler,.. they all want my money. "-com" + adblock doesn't really help anymore. I'm so sick and tired of the net. it once was the best thing that ever happened to the world. now it's the hyper-communication tool for fart jokes and perversion.
guess that tells you a lot about humanity.
It's not about fate, it's about character.
there be no shelter here, the frontline is everywhere!
It'll be even cooler when Google are able to automatically detect things like citations and references, and add hyperlinks as appropriate.
It still sort of bugs me that scientific papers are written in LaTeX, and not hypertext, especially considering that the web (in its current form) originated at CERN.
-- If you try to fail and succeed, which have you done? - Uli's moose
There's a module in CPAN for this. It rips out the images and runs them through Tesseract. It's worked well the few times I've tried it. Certainly well enough for search engine indexing.
Also, my understanding of the "dark web" concept was that it refered to sites that had no links going to them, so no spiders are able to access them. I'm not seeing how any of this would fix the "problem".
The only news here is that Google doesn't already index form content in drop down boxes and selection menus. Seems that would have been a fairly obvious extension.
Maybe not
That's under the completely unsafe assumption that forms are being used properly. There have been numerous instances of people putting full SQL queries (with DB connection data) in a GET form - see TheDailyWTF.
Though I suppose that's a bad example, as it would be really damn easy for Google to index THOSE sites. Just swap in a SELECT * and you're all set :)
How are sites slashdotted when nobody reads TFAs?
Not so sure about PDFs as an image format - which is exactly what you have when you use PDF to hold scanned documents. I think the more interesting point is that they feel they have an OCR package good enough to be trustworthy. I wonder if it's based on the Tesseract OCR software that they adopted a while back?
I played with it for a while, and got very poor results from the command line. Even when I made a png or bmp of a full screen single word "HELLO" in 200 pixel font with GIMP (about as perfect as input gets!) I'd often get "HEHO" or "H3H0" or god only knows what else.
Of course, this is when the project relaunch was first announced a year or two ago, I certainly hope it's better now! Looking at their web page, it does appear that there's some significant activity going on. Yay Google!
Maybe I'll try it again, and see if it's worth using yet?
I have no problem with your religion until you decide it's reason to deprive others of the truth.
configure robots.txt please! :)
Very soon they will start evaluating javascript too, that will shed more light on the dark internet.
Some kid's blog will have a new entry "How did I crash Google?"
for instance if you search for "Widget Model XJ123" it will now find a page by a manufacturer where the only place they list it is in a pulldown list that lets you choose the product to buy.
Shenanigans! And I've been looking everywhere for that elusive XJ123, since the manufacturer stopped producing it. How dare you get my hopes up you insensitive clod!
There. Fixed that for you.
Make America grate again!
It looks like they will only use GET requests, not POST requests. You may have trouble if you use GET requests to make changes on your site (which nearly everybody with minimal experience knows you should never do).
Those who would give up liberty to obtain working drivers, deserve neither liberty nor working drivers.
images of text, not images of things. To obtain text from a photograph of a person, or a painting, is beyond even Google at the moment...
BTW: I wish Adobe used this OCR, so search works on a pdf of scanned text.
Deep web is information buried under layers that are not easily penetrable by current indexing tech.
Dark web can either be physically separate from the internet or a virtual network that is hidden through encryption, secrecy, or both.
Women are like electronics: you don't know how damaged they are until you try to turn them on.
But in their Repairing Aluminum Wiring example, the PDF reads:
and the Google HTML reads:
Maybe this IS one of their better examples.
But the difference between web and net is probably not as important as the difference between deep and dark.
You have a good point. If the program could determine which values are undefined, and what the defined portions of the problem are, then I think I have a solution. It would be similar to what happens to your program code as it's being compiled. The compiler doesn't care what the actual variable is, just if that variable is the same as another.
For your solution, the database entry would be something like this:
(arbitrary value 1)^2 = (arbitrary value 2)^2 + (arbitrary value 3)^2, (arbitrary value 1)!=(arbitrary value 2) && (arbitrary value 1)!=(arbitrary value 3) && (arbitrary value 2)!= (arbitrary value 3)
Then any symbol could be transformed into these arbitrary values, and equality would only be based on same symbols within a single equation.
I'm sure there is a much simpler way of stating this, but I'm at a loss for words. Hopefully you can understand what I'm trying to say.
Is it funny or weird or normal that the only hit on Google for "Widget Model XJ123" is this thread?
"Baby shark wisdom cleaner" 2.0 ?
Squirrel!
I suspect they'd only submit a form if the method is "get" rather than "post"... which technically is okay, although in practice it will likely upset some websites!
When in doubt, remove possibly extraneous search terms. I had to dig, but I found an xj123 model...
Tom Swiss | the infamous tms | my blog
You cannot wash away blood with blood