New Web Application Attack - Insecure Indexing

An anonymous reader writes "Take a look at 'The Insecure Indexing Vulnerability - Attacks Against Local Search Engines' by Amit Klein. This is a new article about 'insecure indexing.' It's a good read -- shows you how to find 'invisible files' on a web server and moreover, how to see contents of files you'd usually get a 401/403 response for, using a locally installed search engine that indexes files (not URLs)."

Comments

should have been from....

[Anonymous Coward]

the department-of-the-bleedingly-obvious...

and don't forget...

[DrKyle]

to see if you can get the site's robots.txt as the files/directories in that file are sometimes full of goodies.

indexing google

[page275]

Even though here's about internal indexing, it reminded me of the old fashion google indexing: Search google with some sensitive terms such as : 'index of /' *.pdf *.ps

permissions permissions permissions

[Capt'n Hector]

Never give web-executable scripts more permissions than absolutely required. If the search engine has permission to read sensitive documents, and web users have access to this engine... well duh. It's just common sense.

Re:permissions permissions permissions

[WiFiBro]

This document in the first paragraphs describes how to get to files which are not public. So you also need to take the sensitive files out of the public directory, which is easy but hardly ever done. (You can easily make a script to serve the files in non-public directories to those entitled to).

Interesting. Brief summary.

[caryw]

Basically the article says that some site-installed search engines that simply index all the files in /var/www or whatever are insecure because they will index things that httpd would return a 401 or 403 for. Makes sense. A smarter way to do such a thing would be to "crawl" the whole site on localhost:80 instead of just indexing files, that way .htaccess and the such would be preserved throughout.
Does anyone know if the Google search applicance is affected by this?
- Cary
--Fairfax Underground: Where Fairax County comes out to play

Re:Interesting. Brief summary.

[XorNand]

A smarter way to do such a thing would be to "crawl" the whole site on localhost:80 instead of just indexing files, that way .htaccess and the such would be preserved throughout.

Yes, that would be safer. But one of the powers of local search engines is the ability to index content that isn't linked elsewhere on the site, e.g. old press releases, discontinued product documentation, etc. Sometimes you don't want to clutter up your site with irrelavant content, but you want to allow people who know what they're looking for to find it. This article isn't really groundbreaking. It's just another example of how technology can be a double-edged sword.

Re:Interesting. Brief summary.

[tetromino]

Does anyone know if the Google search applicance is affected by this?

No. First of all, the Google Search Appliance crawls over http, and therefore obeys any .htaccess rules your server uses. Second, you can set it up so that users need to authenticate themselves. Third, there are many filters you can set up to prevent it from indexing sensitive content in the first place (except that since any sensitive content the google appliance indexes must already be accessible via an external http connection, one hopes it's not too sensitive).

obvious?

[jnf]

I read the article and it seems to be like a good chunk of todays security papers, 'heres a long drawn out explanation of the obvious', I suppose it wasn't as long as it could be, but really ... using a search engine to find a list of files on a website? I suppose someone has to document it..

I mean, I understand its a little more complex as described in the article- but i would hardly call this a 'new web application attack', at best perhaps one of those humorous advisories where the author overstates things and creates much ado about nothing- or at least thats my take;

-1 not profound

P2P

[Turn-X Alphonse]

goto any P2P network and type @hotmail.com, @Gmail.com or @yahoo.com and see what documents turn up.. I'm willing to put money on them all being e-mails saved on idiots PCs which will contain everything from stuff to sell to spammers (if your so inclined), to sexual stuff and passwords/creditcard info.

Nothing really new here..

Re:Interesting. Brief summary.

[Qzukk]

If you could give the crawler multiple starting points then you could simply have an unlinked page that links to all the old content, and give that page to the crawler as a second starting point.

RTFM

[Tuross]

My company specialises in search engine technology (for almost a decade now). I've worked quite in-depth with all the big boys (Verity, Autonomy, FAST, ...) and many of the smaller players too (Ultraseek, ISYS, Blue Angel, ...)

I can't recall the last time this kind of attack wasn't mentioned in the documentation for the product, along with instructions on how to disable it. If you choose to ignore the product documentation, you get what you deserve.

It's quite simple folks. Don't open the search engine. ACL query connections. Sanitize queries like you (should?) do other CGI applications. Authenticate queries and results. If you can't be bothered, hire someone who can.

Re:Interesting. Brief summary.

[BigGerman]

This is even more important when a search engine (appliance) is capable to crawl the file shares directly (not just over HTTP).
EnterFind appliance (which I participated in developing) has this (still unique) feature and their clients were amazed by what the crawler can dig out. Especially in those "hidden" fields in the Office documents.

Google Hacks Database

[giant_toaster]

I guess a lot of people have seen this site before, but http://johnny.ihackstuff.com/index.php?module=prod reviews has a lot of these google exploits etc, he is posting them up so people can check if their sites are secure. There are some interesting presentations by him on the main site about how search engines can be exploited.

Speaking of firefox

[ad0gg]

Another exploit can out this weekend. The funny thing is that microsoft antispyware beta 1 detects the execution of the payload file and shows a prompt if you want continue or stop the execution.

New option for robots.txt

[michelcultivo]

Please put this new undocumented tag on your robots.txt file: "hackthis=false" "xss=false" "scriptkiddies=log,drop" And all you problems will be solved.

Re:Interesting. Brief summary.

[Grax]

On a site with mixed security levels (i.e. some anonymous and some permission-based access) the "proper" thing to do is to check security on the results the search engine is returning.

That way an anonymous user would see only results for documents that have read permissions for anonymous while a logged-in user would see results for anything they had permissions to.

Of course this idea works fine for a special purpose database-backed web site but takes a bit more work on just your average web site.

Crawling the site via localhost:80 is the most secure method for a normal site. This would index only documents available to the anonymous user already and would ignore any unlinked documents as well.

This is old.

[brennz]

Why is this being labeled as something new? I remember this being a problem back in 1997 when I was still working as a webmaster.

Whoever posted this as a "new" item, is behind the times.

OWASP covers it!

Lets not rehash old things!