Slashdot Mirror
New Web Application Attack - Insecure Indexing
An anonymous reader writes "Take a look at 'The Insecure Indexing Vulnerability - Attacks Against Local Search Engines'
by Amit Klein. This is a new article about 'insecure indexing.' It's a good read -- shows you how to find 'invisible files' on a web server and moreover, how to see contents of files you'd usually get a 401/403 response for, using a locally installed search engine that indexes files (not URLs)."
right?
time is a perception of a being's consciousness
time is your 6th sense, the wierd ones are 7+
the department-of-the-bleedingly-obvious...
to see if you can get the site's robots.txt as the files/directories in that file are sometimes full of goodies.
Even though here's about internal indexing, it reminded me of the old fashion google indexing: Search google with some sensitive terms such as : 'index of /' *.pdf *.ps
Never give web-executable scripts more permissions than absolutely required. If the search engine has permission to read sensitive documents, and web users have access to this engine... well duh. It's just common sense.
Quid festinatio swallonis est aetherfuga inonusti?
Africus aut Europaeus?
Basically the article says that some site-installed search engines that simply index all the files in /var/www or whatever are insecure because they will index things that httpd would return a 401 or 403 for. Makes sense. A smarter way to do such a thing would be to "crawl" the whole site on localhost:80 instead of just indexing files, that way .htaccess and the such would be preserved throughout.
Does anyone know if the Google search applicance is affected by this?
- Cary
--Fairfax Underground: Where Fairax County comes out to play
Search engines let you find stuff! This is precisely why google, yahoo, and all the rest obey robots.txt Personally, I would be amazed if local search engines didn't have their own equivalent of robots.txt that limited the directories they are allowed to crawl.
The attacker first loops through all possible words in English...
I get the idea this might take a while.
The instances mentioned all seem to revolve around the idea of indexing files. Could the same be used for database driven sites? You know, like the old search for "or 1=1" trick?
Then again, it's about being organized, isn't it? A check of what should and shouldn't be allowed to go public, some sort of flag where even if it shows up in the result, it better not make its way onto the HTML being sent back. (I figure that's more DB-centric though)
Last madman rant -- Don't put anything up there that shouldn't be for public consumption to begin with!!! If you're the kind to leave private XLS, DOC, MDB, and other sensitive data on a PUBLIC server thinking it's safe just because nobody can "see" it, to put it delicately, you're an idiot.
Entrepreneur : (noun), French for "unemployed"
Does anyone know if the Google search applicance is affected by this?
.htaccess rules your server uses. Second, you can set it up so that users need to authenticate themselves. Third, there are many filters you can set up to prevent it from indexing sensitive content in the first place (except that since any sensitive content the google appliance indexes must already be accessible via an external http connection, one hopes it's not too sensitive).
No. First of all, the Google Search Appliance crawls over http, and therefore obeys any
I read the article and it seems to be like a good chunk of todays security papers, 'heres a long drawn out explanation of the obvious', I suppose it wasn't as long as it could be, but really ... using a search engine to find a list of files on a website? I suppose someone has to document it..
I mean, I understand its a little more complex as described in the article- but i would hardly call this a 'new web application attack', at best perhaps one of those humorous advisories where the author overstates things and creates much ado about nothing- or at least thats my take;
-1 not profound
bastards always hiding their stash. this'll show 'em
goto any P2P network and type @hotmail.com, @Gmail.com or @yahoo.com and see what documents turn up.. I'm willing to put money on them all being e-mails saved on idiots PCs which will contain everything from stuff to sell to spammers (if your so inclined), to sexual stuff and passwords/creditcard info.
Nothing really new here..
I like muppets.
If you could give the crawler multiple starting points then you could simply have an unlinked page that links to all the old content, and give that page to the crawler as a second starting point.
If I have been able to see further than others, it is because I bought a pair of binoculars.
My company specialises in search engine technology (for almost a decade now). I've worked quite in-depth with all the big boys (Verity, Autonomy, FAST, ...) and many of the smaller players too (Ultraseek, ISYS, Blue Angel, ...)
I can't recall the last time this kind of attack wasn't mentioned in the documentation for the product, along with instructions on how to disable it. If you choose to ignore the product documentation, you get what you deserve.
It's quite simple folks. Don't open the search engine. ACL query connections. Sanitize queries like you (should?) do other CGI applications. Authenticate queries and results. If you can't be bothered, hire someone who can.
Matt
This is even more important when a search engine (appliance) is capable to crawl the file shares directly (not just over HTTP).
EnterFind appliance (which I participated in developing) has this (still unique) feature and their clients were amazed by what the crawler can dig out. Especially in those "hidden" fields in the Office documents.
Oh, we are terribly sorry for taking so long!
Don't worry, we will give you a full refund.
I guess a lot of people have seen this site before, but http://johnny.ihackstuff.com/index.php?module=prod reviews
has a lot of these google exploits etc, he is posting them up so people can check if their sites are secure. There are some interesting presentations by him on the main site about how search engines can be exploited.
Another exploit can out this weekend. The funny thing is that microsoft antispyware beta 1 detects the execution of the payload file and shows a prompt if you want continue or stop the execution.
Have you ever been to a turkish prison?
Please put this new undocumented tag on your robots.txt file: "hackthis=false" "xss=false" "scriptkiddies=log,drop" And all you problems will be solved.
http://www.michel.eti.br
On a site with mixed security levels (i.e. some anonymous and some permission-based access) the "proper" thing to do is to check security on the results the search engine is returning.
That way an anonymous user would see only results for documents that have read permissions for anonymous while a logged-in user would see results for anything they had permissions to.
Of course this idea works fine for a special purpose database-backed web site but takes a bit more work on just your average web site.
Crawling the site via localhost:80 is the most secure method for a normal site. This would index only documents available to the anonymous user already and would ignore any unlinked documents as well.
Coding Blog
Im pretty sure the indexing server on Windows won't return 'search results' for files you dont have permissions to list.
The problem and vulnerablity lies in definition of "you".
The indexing program runs on privledges of a local user with direct access to the harddrive. Listing directory contents, reading user-readable files. "you" are the user, like one behind the console, maybe without access to sensitive system files, but with access to mostly everything in the htroot tree the administrator hasn't blocked using the OS permissions, not the httpd features.
As a webpage visitor "you" are "guest", filtered through httpd, with all httpd restrictions applied. No directory listing, obscure blocking methods (.htaccess, config files, redirects, CGI wrapping) working. Your access is limited to what httpd lets you do, not just what the OS does. Now if you access the search engine database, you can see mostly everything the engine saw, including things it wouldn't see if it was running through httpd, not directly accessing the filesystem.
45 5F E1 04 22 CA 29 C4 93 3F 95 05 2B 79 2A B2
Why is this being labeled as something new? I remember this being a problem back in 1997 when I was still working as a webmaster.
Whoever posted this as a "new" item, is behind the times.
OWASP covers it!
Lets not rehash old things!
I find your ideas intriguing and I wish to subscribe to your newsletter.
Did you RTFA?
Yep. Did you keep reading it? I'm referring to the methods for when no excerpts are given.
here's a solution thats been tried and seems to work: create metadata for each page as an xml/rdf file (or db field). XPATH can be used to scrape content from HTML et al to automate the process, as can capture from CMS or other doc management solutions. create a manifest per site or sub site that is an XML-RDF tree structure containing references to the metadata files and mirroring your site structure. finally, assuming you have an API for your search solution (and don't b*gger around using ones that dont) code the indexing application to only parse the XML-RDF files, beginning with the structural manifest and then down into the metadata files. Your index will then contain relevant data, site structure, and thanks to XPATH, hyperlinks for the web site. No need to directly traverse the HTML. Still standards based. Security perms only need to allow access to the XML-RDF files for the indexer, which means process perms only are needed, user perms are irrelevant.
There are variations and contingencies, but the bottom line is, even if someone cracked into the location for an xml metadata file, its not the data itself and while it may reveal a few things about the page or file it relates to, certainly is bottom line much less of a risk than full access to other file types on the server.
heres another tip for free. because you now have metadata in RDF, with a few more lines of code you can output it as RSS.