Slashdot Mirror
Google Sheds Light On 'Dark Web' With PDF Search
CWmike writes "Google this week took another step in its effort to shed light on the so-called Dark Web, announcing that its search engine can now search scanned documents in a PDF. In April, Google announced that it was looking for ways for its search engine to index HTML forms such as drop-down boxes or select menus that otherwise couldn't be found or indexed."
An announcement is available at the official Google blog, and it contains some demonstration searches.
Hey Bill, it's great to see that they finally gave you that day pass....
Some people are only alive because it's against the law for me to hunt them down and kill them.
After reading that, I've come to the conclusion that some parts of the internet should definitely remain in the dark.
Google announced that it was looking for ways for its search engine to index HTML forms such as drop-down boxes or select menus that otherwise couldn't be found or indexed."
Great. So basically, it's going to fuss with forms and pretend to be a user clicking "submit". That seems like a BRILLIANT idea, because, naturally, every HTML form out there is used purely for navigation...
If people want their sites to be indexed, they shouldn't use forms for navigation. It's not rocket science.
Please help metamoderate.
Increasing the number of items that can be searched is great, but the actual searching algorithms really haven't gotten THAT much better in the past 3 years or so.
Obviously, you can't have breakthroughs every year (or maybe even every 5 years) but search as an algorithm still has much more room to improve. I'd love to see an improvement in that, as opposed to just increasing the number of pages indexed.
Still cool though...
If you can read this... 01110101 01110010 00100000 01100001 00100000 01100111 01100101 01100101 01101011
Referenced article is talking about the "deep web", not dark web.
Well yes, but it doesn't mean that no one will want to try and find it.
/b/...
Just look at
"Intelligence has nothing to do with politics!"
-Londo Mollari
"Scanning is the reverse of printing." -- WTF?! Because of artifacts? And isn't this what View as HTML has ALWAYS been about? Points awarded for techtard clarity, but the person at Google who thought writing a press release aimed at techtards should be firmly smacked.
-- Dedicated Cthulhu cultist since 1982 A.C.E.
It's DEEP web, not dark. This is the internet not astrophyics.
I always thought the "dark" web was the seedy underside if password-protected forums and such where warez pirates and so on operated, releasing cracks for software and then letting it trickle down into more visible channels. Well, before torrents and TPB, at least.
I just started reading and it says "powerful search engines such as Google and Yahoo". Yahoo is a search engine? A Powerful one? It's an advertising index, Spam search, Ad finder? I call BS, no one thinks Yahoo is a powerful search engine!
Every time I use image search and see most are not related, I look at Google asking ME to help them label pictures to help. I feel guilty for not helping, and comfort myself knowing Google has a far better shot at image recognition than I ever will.
Never heard of either before. Looks like there's a competition going on to see who comes up with the next buzzword.
The filesystem is the package manager
A "dark web" is a private network, accessible by members over the internet but not accessible to outsiders. (A VPN is one example of a kind of "dark web".)
But as you say, this is something completely different.
The Deep, Deep, Dark, Dark, Deep, Dark Web...coming soon to a web browser near you!
MCSE? No, sir...I don't do Windows. Yes, I am an idealist. What's your point?
Google has long since favoured PDFs - and gives them boosted results, under the guise that anyone who makes a PDF has something serious to say, I guess.
You may have noticed of late that people are wise to this - there are a bunch of sites that are embedding popular search terms / results in PDF files, and clustering their sites with adverts.
What I would really like to see is OCR for mathematical formulas, and store those in some standard format. Using a standard input, like LaTeX, the engine would search for mathematical equations. Right now I find it a pain to look for a formula that I know exists, but don't know its name.
This would help bring together a lot of research that is done, but hard to sort through. Then, implement a smart system using a program like Mathematica to find variations of the equations, etc., and see where duplicates exist. Maybe we'll find that we've discovered things that weren't looked at thoroughly enough.
For instance, the goatse guy's innards? Yeah, I'd rather he had kept those where the sun don't shine, too.
Nice feature, but I think it only works with PDF? I would love to see the same with DjVu as well.
How about adding the word *scanned* into the headline, just as the original headline was.
That way others won't have to read the summary going "Hey, I thought Google was searching PDFs for the last 10 years."
-David
dark web.. oh geez. eternal September has only just started.
aparently the world at large loves to shit on standards and practices.
it's been a while since search engines actually returned results I was looking for. google, yahoo, msn, metacrawler,.. they all want my money. "-com" + adblock doesn't really help anymore. I'm so sick and tired of the net. it once was the best thing that ever happened to the world. now it's the hyper-communication tool for fart jokes and perversion.
guess that tells you a lot about humanity.
It's not about fate, it's about character.
there be no shelter here, the frontline is everywhere!
It'll be even cooler when Google are able to automatically detect things like citations and references, and add hyperlinks as appropriate.
It still sort of bugs me that scientific papers are written in LaTeX, and not hypertext, especially considering that the web (in its current form) originated at CERN.
-- If you try to fail and succeed, which have you done? - Uli's moose
There's a module in CPAN for this. It rips out the images and runs them through Tesseract. It's worked well the few times I've tried it. Certainly well enough for search engine indexing.
Also, my understanding of the "dark web" concept was that it refered to sites that had no links going to them, so no spiders are able to access them. I'm not seeing how any of this would fix the "problem".
The only news here is that Google doesn't already index form content in drop down boxes and selection menus. Seems that would have been a fairly obvious extension.
Maybe not
Not so sure about PDFs as an image format - which is exactly what you have when you use PDF to hold scanned documents. I think the more interesting point is that they feel they have an OCR package good enough to be trustworthy. I wonder if it's based on the Tesseract OCR software that they adopted a while back?
I played with it for a while, and got very poor results from the command line. Even when I made a png or bmp of a full screen single word "HELLO" in 200 pixel font with GIMP (about as perfect as input gets!) I'd often get "HEHO" or "H3H0" or god only knows what else.
Of course, this is when the project relaunch was first announced a year or two ago, I certainly hope it's better now! Looking at their web page, it does appear that there's some significant activity going on. Yay Google!
Maybe I'll try it again, and see if it's worth using yet?
I have no problem with your religion until you decide it's reason to deprive others of the truth.
Very soon they will start evaluating javascript too, that will shed more light on the dark internet.
Some kid's blog will have a new entry "How did I crash Google?"
Deep web is information buried under layers that are not easily penetrable by current indexing tech.
Dark web can either be physically separate from the internet or a virtual network that is hidden through encryption, secrecy, or both.
Women are like electronics: you don't know how damaged they are until you try to turn them on.
But in their Repairing Aluminum Wiring example, the PDF reads:
and the Google HTML reads:
Maybe this IS one of their better examples.
But the difference between web and net is probably not as important as the difference between deep and dark.
You have a good point. If the program could determine which values are undefined, and what the defined portions of the problem are, then I think I have a solution. It would be similar to what happens to your program code as it's being compiled. The compiler doesn't care what the actual variable is, just if that variable is the same as another.
For your solution, the database entry would be something like this:
(arbitrary value 1)^2 = (arbitrary value 2)^2 + (arbitrary value 3)^2, (arbitrary value 1)!=(arbitrary value 2) && (arbitrary value 1)!=(arbitrary value 3) && (arbitrary value 2)!= (arbitrary value 3)
Then any symbol could be transformed into these arbitrary values, and equality would only be based on same symbols within a single equation.
I'm sure there is a much simpler way of stating this, but I'm at a loss for words. Hopefully you can understand what I'm trying to say.
"Baby shark wisdom cleaner" 2.0 ?
Squirrel!