Slashdot Mirror
Ask Slashdot: What To Do When Finding a Security Breach On Shared Hosting?
An anonymous reader writes "A few months ago I stumbled across an interesting security hole with my webhost. I was able to access any file on the server, including those of other users. When I called the company, they immediately contacted the server team and said they would fix the problem that day. Since all you need when calling them is your username, and I was able to list out all 500 usernames on the server, this was rather a large security breach. To their credit, they did patch the server. It wasn't a perfect fix, but close enough that moving to a new web host was moved down on my list of priorities. Jump a head to this week: they experienced server issues, and I asked to be moved to a different server. Once it was done, the first thing I did was run my test script, and I was able to list out everyone's files again. The hosting company only applied the patch to old server. I'm now moving off this web host all together. However, I do fear for the thousands of customers that have no clue about this security issue. With about 10 minutes of coding, someone could search for the SQL connection string and grab the username/password required to access their hosting account. What's the best way to handle this type of situation?"
Move to a new host. Don't talk about the old host, don't post the script, don't describe it at all. You don't want the lawsuit/criminal charges that will follow.
Don't reward bad behavior. I recently severed a relationship with a hosting company of more than ten years because there support had gone from great to terrible. We had a problem and they wouldn't or couldn't fix the problem so I switched. The switch didn't come without some pain, but now everything is back to normal. Don't reward bad behavior, period.
I hope this caused some synapses to fire.
Tell the webhost they have XYZ days to fix the problem before you publish the exploit.
https://forms.us-cert.gov/report/ is also a good place to report exploits.
But if you're shy, I'd also consider forwarding the details to a reputable security research company,
so that maybe they can alert others with misconfigured systems and CERT.
[Fuck Beta]
o0t!
You have no idea what idiotic web applications people are running. You should ASSUME that any shared host is compromised. Don't store any unencrypted data there which is at all sensitive. Given the low cost of renting a virtual or physical host machine these days it seems there's little reason to bother with shared hosting (yes, it is cheaper, but honestly the cost of an AWS micro instance is pretty low).
The real problem is bulk shared hosting facilities just can't afford to tinker. There are often 100 or more accounts on a server, sometimes even 1000's. One stupid tweak to fix a security hole can break a LOT of scripts. These places will always prefer to just set up servers and not EVER patch them.
The ultimate observation is just that driving the cost of hosting down to $2.99 a month means doing absolutely nothing beyond what is absolutely needed to make it work. You get what you pay for.
"Malo periculosam, libertatem quam quietam servitutem." -- Jefferson
Ummm, you might want to tell them your plan and give them 10 days to fix & fess up. And make sure your notice to them is sent to the boss, not the sysadmin who screwed up and has no stake is letting anyone know.
and try to find the mail addresses of the users and alert them of the security problems. If many of them leave, maybe the hoster feels it's time to act.
http://en.wikipedia.org/wiki/Responsible_disclosure
Contact them to agree a timeframe to patch.
If you live in the US, or your hosting is in the US, what you have done is technically cyber-crime. While I hate to say this, your best recourse is to move to another host and leave it all behind you. Should the hosting company start losing business because of you warning other users you could face all kinds of civil lawsuits and possibly even criminal penalties.
I wouldn't do that. Original poster has described his history with the company. Effectively, he is no longer anonymous. Lawsuits could follow public statements here.
and attempting to speak with the ISP has not worked (it's not clear if you have tried to inform them that the bug remains on this, and likely other, servers, and given them the chance to fix it (albeit a second chance)), call up your data protection regulator on Monday morning, and explain the nature of the issue and its impact?
Back in the days of dial up, I used a dial-up ISP that offered free scripting (CGI, ASP, you name it) on a Windows server. While teaching myself scripting, I discovered that files I wrote as part of scripts ended up in the c:\windows\system32 directory of the server instead of my user folder. Worse still cgi scripts allowed running executables. Needless to say that is bad as it allowed me to get remote shell access to the box. Finally to complete the incompetence, I found that the ISP was storing the customer records on the server as an access database. When I mean records, I mean everything: names, addresses, credit cards, etc.
I informed the ISP of the problem. They responded, but said it was a "windows" problem and couldn't be fixed so I posted on a message board for customers about the problem (but not the details on how to do it), wiped my own customer records from their database (yes I could read and write) and canceled service. I don't know what ever happened to them, but I'm assuming they went out of business like most other dial up ISPs.
I do the same as I do when I see other illegal stuff. I report it.
I have once reported childporn. I was ordered to go to go to the police station where they tried to put the following on me:
1) Spreading of childporn (Remember that I was the one who reported it)
2) Obstruction of the law (because I called the newspaper, after wich they finaly closed the site)
3) Falsification of my person (because my trow away email address did not have any official address)
I send the report from work. They called there to say they needed to speak to me concerning a childporn case. Luckily I had VERY understand management (who even offered to pay for lawyers if anything would come of it towards me) otherwise I could have been out of a job.
So if I ever see anything illegal again, I would do the right thing and report it.
But somehow I never have seen anything illegal after that. Not even people speeding or pedestrians walking through a red light. Strange, isn't it?
Don't fight for your country, if your country does not fight for you.
Contact the company again with your findings. They patched the hole that you pointed out before but kept the details of the exploit limited to senior programmers and support. When they reloaded the server after a down period, a SNAFU recreated the hole.
So there are two problems. One is the security hole that you found and the other is their back-up and security breach repair process. Point out both problems to them.
Then review the security of your data that you are exchanging with them. How important is it that this data remain secret? And secret to who? To another user who might have stumbled onto the same exploit window? To a Soviet/Russian criminal organization? (a three-way redundancy, yes, I know) To the American feds? To your wife or kid that looks over your shoulder while you type?
Please understand, all this technology is still basically new. It has problems. Tech problems and social problems. The tech issues get discovered and solved faster than the social problems, i.e. crime issues. For example, we (the American government and Interpol) can not go after criminal organizations in the (former) Soviet Union because many of them are in alliance with the corrupt Soviet/Russian/Gangster government that still controls thousands of nuclear bombs. So criminal organizations there can loot American banks and businesses with stolen credit card information with near impunity. It's a defect of the modern computer age. It will get fixed someday, but for now, guard your data and be aware that every data and login password that you type on an internet-linked PC can be stolen.
If the web-server company can't and/or won't fix the issue after you point it out to them several times, document the issue and submit this documentation in writing (not on-line) to both the local Better Business Bureau and your state Attorney General's Office. When they get inquiries from both parties about this issue, they will get the fear of God and fix it right. Until then, be patient and remind people to guard their data.
I've been using Linode for the last 8 months or so, and have been pretty happy with it.
$20 per month gets you 1 static ip address, 512 MB of ram, 20 GB of disk space, 200 GB of upload bandwidth, unlimited download bandwidth, and up to 4 cpu cores.
A good hosting company would have a smart load balancer or somesuch at the gateway that could route internally to whatever based on hostname.
You can't route HTTPS based on hostname until Internet Explorer for Windows XP reaches its end of life in 18 more months. If multiple sites are hosted on port 443 of a given IP address, IE for XP can't see the certificate for any site other than the first because IE for XP doesn't support SNI. To me, at least, getting a dedicated IPv4 address on which to run HTTPS is one of the main reasons to upgrade from shared hosting to a VPS, especially for web site administrators who are concerned about security. Because without HTTPS, anybody can intercept and forge your site's users' session cookies using Firesheep.
Others have made a good case for simply moving on, but another thought would be to move to another provider, then notify them via certified letter why you're moving and informing them that if/when the hole is exploited (and reiterate that you will not exploit it yourself), then the certified letter will be shared with the legal teams of those customers who have suffered damages.
i.e. "Here's your official notice of a potential exploit, don't say you weren't warned."
It won't provide preemptive help for their other customers but may make their damages somewhat recoverable through legal means.
I've been using Linode for the last 8 months or so, and have been pretty happy with it.
$20 per month gets you 1 static ip address, 512 MB of ram, 20 GB of disk space, 200 GB of upload bandwidth, unlimited download bandwidth, and up to 4 cpu cores.
If you don't need much bandwidth or CPU, check out an Amazon Micro instances. If you buy a reserved instance, a Micro instance ends up costing around $7/month plus $0.10/GB for disk and $0.10/GB for outbound bandwidth.
They are cheap enough to run multiple instances - I have my public website on one instance and use the other one for my mail server, and other things I don't want on the public server giving me complete separation between the two. If the webserver ever gets hacked, I can just restore it from an S3 snapshot. I had started looking at chroot'ing Apache or running it in a VM for better isolation, but spinning up a second micro instance was much easier.
If you need to use significant CPU, a micro instance is probably not going to be a good choice, as I've heard that Amazon throttles back CPU to Micro instances that use a lot of sustained CPU. But it runs my PHP based photo gallery software pretty well (shared only to family/friends, so it's not super busy).
The bandwidth costs could get expensive quickly at 10 cents/GB if you have a busy website. I run a script that checks my bandwidth utilization and if I hit more than 10GB in one day it shuts down Apache and notifies me so I don't end up with a huge bandwidth bill if my site ever slashdotted.
Even with multiple S3 snapshots, my total hosting bill is always less than $20/month, less than I was paying for a single VPS server (that was having performance issues due to being oversubscribed so heavily by the ISP)
I have several 512mb vps I run several services on, which cost me a whopping $6/mo. The services are non-critical, and if not for this price-point, would not be running on a vps. I was having a problem with one of them where the vps os would randomly reboot itself. I asked the vendor to check the vps host to make sure there wasn't something amiss. They claimed there wasn't, and I could find nothing amiss on the Debian slice OS. I finally came to the realization that since these vps are OpenVZ, it was likely something one of the other slices was doing, since its well-known that OpenVZ containers are very susceptable to other slices taking more than their share of resources. I therefore began to look for a Xen-based host, as these reboots were hair-pulling annoying. With a bit of searching, I came across a vendor advertising 512MB Xen vps either in LA or Kansas City for .... get this.. $5/mo.. Am in the process of migrating the nasty rebooting OpenVZ services over to a nice Xen instance.. I'd been with the OpenVZ vendor for nearly two years and have had zero issues with them, up to this issue, which I don't believe is their "fault" vs just the nature of the OpenVZ "beast".. This vendor does have Xen vps, but they're quite a bit more than the newly discovered host..
Not gonna provide links to the vendors, but, for the OpenVZ vendor, "ThrustVPS" and for the Xen vendor, "Virpus"..
THANK YOU, Edward Snowden!! Americans owe you a debt of gratitude (whether they know it or not..)
I read a lot of indignant posts and a few moany warning ones on the subject. The authors of either kinds of post have obviously lost touch with the American Way.
When you find a vulnerability, the first thing to do is to disassociate yourself from it. Wipe your data and close down your account (many posts correctly advised this). Then get two sets of some cheap one-off hardware (second-hand paid-in-cash stuff is best). Use one of those to assess the economic potential of your find as best as you can (or you'll get fleeced later on).
Then you Monetize your find. Quickly, before someone else beats you to it. That's the American Way right there.
Use the second piece of old kit you bought to surf the web. There are certain websites, often in Eastern Europe, on which you will find people who'll use a peculiar form of English but who will be prepared to pay smallish but reasonable amounts for such information. Depending on e.g. whether the flaw leads to credit card data (that's why you ascertained the economic potential of your find first) or advanced military technology (in which case you may be able to get better quotes from buyers in the Middle East or the Far East).
Be aware that there is a certain protocol to be followed when conducting this sort of transaction. Contacting them from home, work, or any other place that can easily be traced to you is a beginner's mistake. Secondly, don't *ever* give out information like your real name, physical address, bank account or credit card to them. They won't do that either, and besides, you'll *really* value your privacy when dealing with them.
Use e.g. an old second-hand laptop and work from an Internet cafe or use a prepaid smart phone with Internet browsing facilities. Don't ever use that hardware for *anything* but completing this one transaction. Wipe, disassemble, smash, and ditch said hardware component-wise as soon as the transaction is completed.
The trick is of course to get the money to where you can spend it. Having it wired into your account will show up and may be a bit difficult to explain. Even when done from a US account (you can negotiate for this but it costs extra). They will pay you in bitcoin or E-gold if you insist, but that too is tricky. Asking for cash in the mail is asking to be fleeced, and likewise a bit conspicuous should they actually do it (amateurs).
I'm leaving the question of arranging secure and discreet transfer as homework. Additional points will be awarded (optionally off the record or against a discreet little cash bonus) for really good solutions. Remember: should government officials come calling at your doorstep you'll automatically fail the course and all traces of your enrollment will mysteriously have vanished. No refunds.
So what you're saying is that Linux is LESS secure that Windows? Because it's a piece of cake to make a Windows server run scripts in such a way that they can't read, write, delete, or list directory contents outside their own little sandbox.
But that can't be the case. And in fact... it isn't the case! PHP can indeed be properly secured on Linux in such a way that scripts can't access outside their sandbox, and it certainly doesn't include custom SELinux rules. I've used tons of hosts where this is in fact the configuration.
Please name the hundreds of hosts you've worked with so we can avoid them?
For a site about things like basic rights, Slashdot users sure do like to censor "dissent".
http://status.linode.com/2012/03/manager-security-incident.html
ok.. apparently it was their fault
I work at a website development company and one of our clients websites was hacked/defaced. The web host blamed out of date software on our client's website for the breach and the deface. Our client was on a shared hosting package with the hosting company.
When I was told to be the one to clean up the mess on the website and after getting rid of recently modified files (most of the site hasn't been touched for several months) and other malicious files, I stumbled upon a directly conveniently named "sym". This directory contained a symbolic link to the Root directory on the site which stunned me a little that it could be created in the first place.
I checked some folders and files inside and I had full read/write access to any file on the system. The very first thing I did was make my own employer aware of the situation before then informing the web hosting company that there is a major security risk to the server. I sent the message to them two weeks ago and I have not heard a single thing since.
Since then however, the hosting company has been much harder to deal with not responding to the many messages we have sent to them regarding other issues with this particular client's hosting. The site has been defaced again but this time no matter how many times they say they reset the password to the FTP and cPanel, we still can't login. Without being able to login, we can not make our own backup of the site (database dump and files download) which means we can not move the site to another hosting company
We tried to do a second idea of actually just pointing the domain name to a temporary host with a splash page rather than the defaced page. Unfortunately with this, there were issues with who actually controlled the domain name. The Whois lookup said it was Netregistry however when contacting them, they said it was the web hosting company. Trying to login to the hosting company's domain manager, it said they were not managing that domain name.
We are actually kind of stuck with what to do now. We know we definitely want to transfer them to a new hosting company but like I said above, we can't even make a back up of the site to do a clean move. We did quote them a few months back about redoing their website (the bulk of the website was made several years ago) but they have so far resisted the rebuild.
What would any of the Slashdot crowd do if they were in the same situation?
Still fight with the hosting company to get the data?
Push the client to get a new website built with new data?
But then who would be responsible for the domain name if neither party says they are?
Unless a good friend or business associate is using this insecure host, don't say a word.
Take your business elsewhere. Tell them why you're leaving. Don't tell anyone else.
You'd be exposing yourself to a lot of liability.
LK
"Hi. This is my friend, Jack Shit, and you don't know him." - Lord Kano
I've not been on shared hosting for some time, but things always used to be this way. It is a combination of using default Apache/PHP/other configuration (as provided by the off-the-shelf hosting control panels), default file+directory permissions, and users not being educated to change the permissions on sensitive files (or better: being educated enough to know tweaking those perms is not enough so they should demand a more secure setup from their host).
/home/ is globally readable (which is pretty much standard) which allows you to see what users exist as they all have a directory under /home/. If this is the case then the fix they applied was likely to simply change the read permission flag on /home so that you can not list the contents, which isn't really a fix at all: if you know a username either because of foreknowledge or by finding a list of users from elsewhere (/etc/passwd for instance, which usually globally readable) then you can just list /home/ and blocking reading of /home won't change that. Turning off global execute permission on /home would stop you, but because of the way many shared hosts are configured that would also break Apache. Yoiu can test this if you report the issue and it gets fixed the same way: remember one of the usernames you can find now and after the fix see if you can still read /home//public_html or similar.
If I'm reading between the lines well enough, I suspect the problem is that
If you host runs Apache as a single user then there is no way around this. You can mitigate it somewhat with carefully setting permissions on your own files and some obfuscation of file/directory names, but that isn't really a proper answer to the problem.
Apache can be configured to run scripts (via suexec, phpsuexec, and so forth) as a the owner of the script which allows you to lock down configuration files and others that contain sensitive information so other uses can't read them (only set them to -rw------- and only you can read them, and that includes scripts if Apache runs them as you) - but most hosts don't do this (or they didn't last time I was working in that arena) as it is more hassle to setup and/or because it requires more resources. And by "more hassle to setup" I simply mean that it means more than just the out-of-the-box configuration: the "leading" standard control panel back than was cPanel (it may still be, I've not kept an eye on the market recently) and seeing posts like http://www.linuxgo.net/howto-enable-suphpphpsuexec-on-a-cpanel-server/ indicates that it still does not offer an easy (from the point-and-click PoV most cheap hosts need as they are rarely Linux/Apache/other experts) route to using the more secure arrangement. Most hosts will consider the extra admin time of setting up the more secure options to not be worth keeping (or gaining) your custom - 99%+ of their target market don't care (or don't know any better) and spending time to satisfy the other 1% or less is not worth it to them.
tl;dr: You will probably find this is the standard setup on a great many shared hosts, possibly most, maybe even nearly all. To ensure you are getting a new host that does things more securely when you move, you need to ask some pre-sales questions that are fairly technical (in the sense that sales may not be able to help, unless the company is small enough that the sales and tech support teams are the same people).
I would suggest instead using a VPS provider or self-hosting, that way there are no other direct users of the machine (be it real or virtual) to worry about, but unfortunately both of those options put more administrative load (and cost, unless you are paying far too much for shared hosting) on yourself and can be a minefield of its own (as with shared hosting avoid the cheapest options and ask searching question