Slashdot Mirror
Ask Slashdot: What To Do When Finding a Security Breach On Shared Hosting?
An anonymous reader writes "A few months ago I stumbled across an interesting security hole with my webhost. I was able to access any file on the server, including those of other users. When I called the company, they immediately contacted the server team and said they would fix the problem that day. Since all you need when calling them is your username, and I was able to list out all 500 usernames on the server, this was rather a large security breach. To their credit, they did patch the server. It wasn't a perfect fix, but close enough that moving to a new web host was moved down on my list of priorities. Jump a head to this week: they experienced server issues, and I asked to be moved to a different server. Once it was done, the first thing I did was run my test script, and I was able to list out everyone's files again. The hosting company only applied the patch to old server. I'm now moving off this web host all together. However, I do fear for the thousands of customers that have no clue about this security issue. With about 10 minutes of coding, someone could search for the SQL connection string and grab the username/password required to access their hosting account. What's the best way to handle this type of situation?"
Move to a new host. Don't talk about the old host, don't post the script, don't describe it at all. You don't want the lawsuit/criminal charges that will follow.
If you have provided reasonable time for them to resond to the issue; which you have and the resolution was not satisfactory then the best course of action is to name and shame them so that they will be forced to fix the issue.
Publish the vulnerability and the name of the hoster on slashdot
I assume there was a list of remedies on about page 14 of the license agreement you probably clicked through when you signed up for their service. My advice is same as previous poster, move and forget about it.
Everything I've ever learned the hard way was based on a statistically invalid sample.
Don't reward bad behavior. I recently severed a relationship with a hosting company of more than ten years because there support had gone from great to terrible. We had a problem and they wouldn't or couldn't fix the problem so I switched. The switch didn't come without some pain, but now everything is back to normal. Don't reward bad behavior, period.
I hope this caused some synapses to fire.
Tell the webhost they have XYZ days to fix the problem before you publish the exploit.
https://forms.us-cert.gov/report/ is also a good place to report exploits.
But if you're shy, I'd also consider forwarding the details to a reputable security research company,
so that maybe they can alert others with misconfigured systems and CERT.
[Fuck Beta]
o0t!
You have no idea what idiotic web applications people are running. You should ASSUME that any shared host is compromised. Don't store any unencrypted data there which is at all sensitive. Given the low cost of renting a virtual or physical host machine these days it seems there's little reason to bother with shared hosting (yes, it is cheaper, but honestly the cost of an AWS micro instance is pretty low).
The real problem is bulk shared hosting facilities just can't afford to tinker. There are often 100 or more accounts on a server, sometimes even 1000's. One stupid tweak to fix a security hole can break a LOT of scripts. These places will always prefer to just set up servers and not EVER patch them.
The ultimate observation is just that driving the cost of hosting down to $2.99 a month means doing absolutely nothing beyond what is absolutely needed to make it work. You get what you pay for.
"Malo periculosam, libertatem quam quietam servitutem." -- Jefferson
If you really want to help those other customers, all you have to do is tell us the name of the company, and let the bad publicity take care of the rest.
Does your country have national security incident response teams/authority? If you do i suggest that you contact them and allow them to handle cooperation with company in question and noticing public.
Only placing the company's reputation at risk will provide sufficient motivation for it to assign such a problem the proper priority.
and try to find the mail addresses of the users and alert them of the security problems. If many of them leave, maybe the hoster feels it's time to act.
http://en.wikipedia.org/wiki/Responsible_disclosure
Contact them to agree a timeframe to patch.
If you live in the US, or your hosting is in the US, what you have done is technically cyber-crime. While I hate to say this, your best recourse is to move to another host and leave it all behind you. Should the hosting company start losing business because of you warning other users you could face all kinds of civil lawsuits and possibly even criminal penalties.
Post on a full disclosure list after giving them a week to fix the issue.
Jump ahead.
FTFY
and attempting to speak with the ISP has not worked (it's not clear if you have tried to inform them that the bug remains on this, and likely other, servers, and given them the chance to fix it (albeit a second chance)), call up your data protection regulator on Monday morning, and explain the nature of the issue and its impact?
Also, why is anyone still using shared hosting? Places charge the same for VM's now.
Go Daddy charges more for a VPS than for its bargain basement PHP-only shared hosting package. As for "so just don't use Go Daddy", I thought a VPS was more expensive in general because a VPS needs its own IPv4 address, and we've run out of those. Which VPS provider do you recommend?
Back in the days of dial up, I used a dial-up ISP that offered free scripting (CGI, ASP, you name it) on a Windows server. While teaching myself scripting, I discovered that files I wrote as part of scripts ended up in the c:\windows\system32 directory of the server instead of my user folder. Worse still cgi scripts allowed running executables. Needless to say that is bad as it allowed me to get remote shell access to the box. Finally to complete the incompetence, I found that the ISP was storing the customer records on the server as an access database. When I mean records, I mean everything: names, addresses, credit cards, etc.
I informed the ISP of the problem. They responded, but said it was a "windows" problem and couldn't be fixed so I posted on a message board for customers about the problem (but not the details on how to do it), wiped my own customer records from their database (yes I could read and write) and canceled service. I don't know what ever happened to them, but I'm assuming they went out of business like most other dial up ISPs.
I do the same as I do when I see other illegal stuff. I report it.
I have once reported childporn. I was ordered to go to go to the police station where they tried to put the following on me:
1) Spreading of childporn (Remember that I was the one who reported it)
2) Obstruction of the law (because I called the newspaper, after wich they finaly closed the site)
3) Falsification of my person (because my trow away email address did not have any official address)
I send the report from work. They called there to say they needed to speak to me concerning a childporn case. Luckily I had VERY understand management (who even offered to pay for lawyers if anything would come of it towards me) otherwise I could have been out of a job.
So if I ever see anything illegal again, I would do the right thing and report it.
But somehow I never have seen anything illegal after that. Not even people speeding or pedestrians walking through a red light. Strange, isn't it?
Don't fight for your country, if your country does not fight for you.
First move and get all your data out of their hands.
THEN shame them by naming them publicly.
You already gave them a chance to fix it and they got lazy.
Contact the company again with your findings. They patched the hole that you pointed out before but kept the details of the exploit limited to senior programmers and support. When they reloaded the server after a down period, a SNAFU recreated the hole.
So there are two problems. One is the security hole that you found and the other is their back-up and security breach repair process. Point out both problems to them.
Then review the security of your data that you are exchanging with them. How important is it that this data remain secret? And secret to who? To another user who might have stumbled onto the same exploit window? To a Soviet/Russian criminal organization? (a three-way redundancy, yes, I know) To the American feds? To your wife or kid that looks over your shoulder while you type?
Please understand, all this technology is still basically new. It has problems. Tech problems and social problems. The tech issues get discovered and solved faster than the social problems, i.e. crime issues. For example, we (the American government and Interpol) can not go after criminal organizations in the (former) Soviet Union because many of them are in alliance with the corrupt Soviet/Russian/Gangster government that still controls thousands of nuclear bombs. So criminal organizations there can loot American banks and businesses with stolen credit card information with near impunity. It's a defect of the modern computer age. It will get fixed someday, but for now, guard your data and be aware that every data and login password that you type on an internet-linked PC can be stolen.
If the web-server company can't and/or won't fix the issue after you point it out to them several times, document the issue and submit this documentation in writing (not on-line) to both the local Better Business Bureau and your state Attorney General's Office. When they get inquiries from both parties about this issue, they will get the fear of God and fix it right. Until then, be patient and remind people to guard their data.
you've learned your first lesson as an admin: shared hosting is shit. congrats.
you're concerned about security, but you're on a shared host that could be compromised by any of X hundred people who have access to it (not just your shared server... EVERY shared server is just waiting for a local priv escalation hole)
at least get a VM... yes, you still need a competent hosting company to ensure they apply patches to XenServer/VMWare... but that requires less work by the admins, and is harder to exploit.
a VM at rackspace is $16/mo. If your security isn't worth that, then why are we even talking about it?
You should tell us who the hosting company is so we can switch companies if we are using them.
A good hosting company would have a smart load balancer or somesuch at the gateway that could route internally to whatever based on hostname.
You can't route HTTPS based on hostname until Internet Explorer for Windows XP reaches its end of life in 18 more months. If multiple sites are hosted on port 443 of a given IP address, IE for XP can't see the certificate for any site other than the first because IE for XP doesn't support SNI. To me, at least, getting a dedicated IPv4 address on which to run HTTPS is one of the main reasons to upgrade from shared hosting to a VPS, especially for web site administrators who are concerned about security. Because without HTTPS, anybody can intercept and forge your site's users' session cookies using Firesheep.
Forget this event, but it's a lesson learned.
You have no rights since you aren't rich. The only way to act is from cover and without chance of attribution.
"This post is an artistic work of fiction and falsehood. Only a fool would take anything posted here as fact."
found this on GoDaddy years back - still the case
-------
1. Enjoy your job
2. Make lots of money
3. Work within the law
Choose any two.
Others have made a good case for simply moving on, but another thought would be to move to another provider, then notify them via certified letter why you're moving and informing them that if/when the hole is exploited (and reiterate that you will not exploit it yourself), then the certified letter will be shared with the legal teams of those customers who have suffered damages.
i.e. "Here's your official notice of a potential exploit, don't say you weren't warned."
It won't provide preemptive help for their other customers but may make their damages somewhat recoverable through legal means.
Jump a head to this week: they experienced server issues, and I asked to be moved to a different server. Once it was done, the first thing I did was run my test script, and I was able to list out everyone's files again.
Couldn't you just inform them again that the problem is still present?
The host isn't doing anything wrong. That configuration is actually the most secure of any common configuration. If your script can read other people's files, that probably means it's running as the unprivileged user "nobody" or "apache". All scripts can read all files, but can only WRITE files that are chmod 666. The only commonly used alternative is suexec, where your scripts run as your user. That means they can only read your files, but it also means all scripts can WRITE to any file, delete any file, or create files anywhere. Given that most all PHP scripts have security holes, running them using suexec is super dangerous - FAR more risky than running them as nobody and letting them read files. So the configuration they are using is definitely the safest, in the opinion of poster who has fifteen years of server security experience. It usrd to be, you could run suexec as a different user, bob_scripts, and that was much safer. Recent versions don't allow that due to some poorly thought rules about file ownership. The ultimate would be set up custom.selinux rules such that your scripts could only read your files AND could only write 666 files, but NOBODY does that. I don't think there is a single shared host in the world who offers that, and I've worked with hundreds of hosts.
I complained to my hosting provider when I saw that my username n password were submitted as plain text get data u=john&p=password type thing. When they fixed it it was just sent to the server as post data. I think they use https n r a bit more professional now.
This rule applies to a lot more than just hosting!
What you tolerate, you get more of. Your tolerance is an implicit endorsement of it.
If you reward the good, and punish the bad, you always get more good than bad.
Very few people have the experience/wisdom/gumption to see this however.
Futurist Traditionalism
I read a lot of indignant posts and a few moany warning ones on the subject. The authors of either kinds of post have obviously lost touch with the American Way.
When you find a vulnerability, the first thing to do is to disassociate yourself from it. Wipe your data and close down your account (many posts correctly advised this). Then get two sets of some cheap one-off hardware (second-hand paid-in-cash stuff is best). Use one of those to assess the economic potential of your find as best as you can (or you'll get fleeced later on).
Then you Monetize your find. Quickly, before someone else beats you to it. That's the American Way right there.
Use the second piece of old kit you bought to surf the web. There are certain websites, often in Eastern Europe, on which you will find people who'll use a peculiar form of English but who will be prepared to pay smallish but reasonable amounts for such information. Depending on e.g. whether the flaw leads to credit card data (that's why you ascertained the economic potential of your find first) or advanced military technology (in which case you may be able to get better quotes from buyers in the Middle East or the Far East).
Be aware that there is a certain protocol to be followed when conducting this sort of transaction. Contacting them from home, work, or any other place that can easily be traced to you is a beginner's mistake. Secondly, don't *ever* give out information like your real name, physical address, bank account or credit card to them. They won't do that either, and besides, you'll *really* value your privacy when dealing with them.
Use e.g. an old second-hand laptop and work from an Internet cafe or use a prepaid smart phone with Internet browsing facilities. Don't ever use that hardware for *anything* but completing this one transaction. Wipe, disassemble, smash, and ditch said hardware component-wise as soon as the transaction is completed.
The trick is of course to get the money to where you can spend it. Having it wired into your account will show up and may be a bit difficult to explain. Even when done from a US account (you can negotiate for this but it costs extra). They will pay you in bitcoin or E-gold if you insist, but that too is tricky. Asking for cash in the mail is asking to be fleeced, and likewise a bit conspicuous should they actually do it (amateurs).
I'm leaving the question of arranging secure and discreet transfer as homework. Additional points will be awarded (optionally off the record or against a discreet little cash bonus) for really good solutions. Remember: should government officials come calling at your doorstep you'll automatically fail the course and all traces of your enrollment will mysteriously have vanished. No refunds.
Many shared hosting admins have no clue about security. On one Mac-based admin forum, a member posted asking what PHP was, "but more importantly" how much he could charge. With that now-defunct product, any PHP script essentially owns all hosted sites on the box (all files including database credentials) AND has access to all hosted site configs including each site's login credentials AND read/write access to logs. Imaging what can be done with a tiny 250 character PHP page, that takes a textbox as input and returns its passthru() in a textarea, and can be placed in any site on the box.
As far as the original post, my first 2 web hosts in '96 and '97 had the same problems. You can cd to your site's parent directory and see all other sites, and go where ever you wanted.
In the past I've reported security issues to hosting companies and ISPs, nothing was ever done. In the case of illegal goings on I reported it to the police and was told they weren't interested in such things. Lesson learned. Now I just walk away and take my business with me.
Move hosts, leave it a few weeks, then anonymise the details and stick it on pastebin. Don't leave a trace. Seriously, just do this. Most shared hosting companies don't give a shit about their customers so you're not going to get anywhere by telling them other than a legal case filed against you.
I work at a website development company and one of our clients websites was hacked/defaced. The web host blamed out of date software on our client's website for the breach and the deface. Our client was on a shared hosting package with the hosting company.
When I was told to be the one to clean up the mess on the website and after getting rid of recently modified files (most of the site hasn't been touched for several months) and other malicious files, I stumbled upon a directly conveniently named "sym". This directory contained a symbolic link to the Root directory on the site which stunned me a little that it could be created in the first place.
I checked some folders and files inside and I had full read/write access to any file on the system. The very first thing I did was make my own employer aware of the situation before then informing the web hosting company that there is a major security risk to the server. I sent the message to them two weeks ago and I have not heard a single thing since.
Since then however, the hosting company has been much harder to deal with not responding to the many messages we have sent to them regarding other issues with this particular client's hosting. The site has been defaced again but this time no matter how many times they say they reset the password to the FTP and cPanel, we still can't login. Without being able to login, we can not make our own backup of the site (database dump and files download) which means we can not move the site to another hosting company
We tried to do a second idea of actually just pointing the domain name to a temporary host with a splash page rather than the defaced page. Unfortunately with this, there were issues with who actually controlled the domain name. The Whois lookup said it was Netregistry however when contacting them, they said it was the web hosting company. Trying to login to the hosting company's domain manager, it said they were not managing that domain name.
We are actually kind of stuck with what to do now. We know we definitely want to transfer them to a new hosting company but like I said above, we can't even make a back up of the site to do a clean move. We did quote them a few months back about redoing their website (the bulk of the website was made several years ago) but they have so far resisted the rebuild.
What would any of the Slashdot crowd do if they were in the same situation?
Still fight with the hosting company to get the data?
Push the client to get a new website built with new data?
But then who would be responsible for the domain name if neither party says they are?
I know a lot of people here seems to think that you should do nothing. However, it's your data and they have a major hole in their security. You meed to tell them, and if they leave you unprotected then a simple better business report is enough to embarrass them. Just be sure that everything you say is 100% true, with no false accusations.
I pay 12 EUR/month for my own dedicated server. Real hardware, not just a vm. Why would anybody even consider shared hosting for important things?
If you fail to report who this hoster is, you are covering up THEIR violations, and could be liable if someone who suffers damages as a result finds out you were covering it up.
But the hateful and stupid people in the legal system could bring charges against YOU for "hacking" (even though it can be argued that all you were doing is verifying the security of YOUR OWN data ... and found the security to be defective).
Does this company claim to be secure? If they do, they are COMMITTING FRAUD! Whistle blower time.
Leave and do NOT tell them why. Just leave.
Then at some later date establish anonymous identity and report them as insecure in a public forum. State in that public forum that if they wish to show the public that they are secure, then should make a post in their own blog (surely they have one) that denies the security risks and backs up that denial with a statement that authorizes you to publish your exploit without any risk.
now we need to go OSS in diesel cars
You can send them a warning email (anonymously if you prefer so) telling them you would announce the flaw publicly unless they fix urgently. Give them a decent deadline as well.
Those are all good security practices, and all irrelevant to being able to write a script that reads files. exec() and passthru() certainly can be considered dangerous, but they not relevant yo the original post's "flaw". As to storing credit card numbers (unencrypted?) on a public web server, THAT would be the security error. You start with the assumption that a public web site is, well, publicly accessible, so you don't store credit card numbers there. You don't expect a shared host, or anyone else, to turn a web server into a secure vault for sensitive financial data. To even store CC details unencrypted violates PCI and therefore the merchant agreement.
Complete and utter release!
Let those bastards learn a lesson or two.
Unless a good friend or business associate is using this insecure host, don't say a word.
Take your business elsewhere. Tell them why you're leaving. Don't tell anyone else.
You'd be exposing yourself to a lot of liability.
LK
"Hi. This is my friend, Jack Shit, and you don't know him." - Lord Kano
I know a Nigerian prince who can help you out...
Table-ized A.I.
If it was me, I'd anonymise it so it didn't refer to the particular web host and then post it somewhere, and link to it on some mailing list, with my real email. There is so much to lose by restricting information about security flaws. It makes it much easier for criminals and governments to have illegitimate access to many systems. Like many people said above, if you post it, it could possibly fuck up your life. I'd put up a hell of a fight (hopefully with the help of EFF et al) if they tried to convict me, but I don't have that much to lose by going to jail (no family,etc. and even if they didn't allow me to use a computer I could read and learn about things).
I've not been on shared hosting for some time, but things always used to be this way. It is a combination of using default Apache/PHP/other configuration (as provided by the off-the-shelf hosting control panels), default file+directory permissions, and users not being educated to change the permissions on sensitive files (or better: being educated enough to know tweaking those perms is not enough so they should demand a more secure setup from their host).
/home/ is globally readable (which is pretty much standard) which allows you to see what users exist as they all have a directory under /home/. If this is the case then the fix they applied was likely to simply change the read permission flag on /home so that you can not list the contents, which isn't really a fix at all: if you know a username either because of foreknowledge or by finding a list of users from elsewhere (/etc/passwd for instance, which usually globally readable) then you can just list /home/ and blocking reading of /home won't change that. Turning off global execute permission on /home would stop you, but because of the way many shared hosts are configured that would also break Apache. Yoiu can test this if you report the issue and it gets fixed the same way: remember one of the usernames you can find now and after the fix see if you can still read /home//public_html or similar.
If I'm reading between the lines well enough, I suspect the problem is that
If you host runs Apache as a single user then there is no way around this. You can mitigate it somewhat with carefully setting permissions on your own files and some obfuscation of file/directory names, but that isn't really a proper answer to the problem.
Apache can be configured to run scripts (via suexec, phpsuexec, and so forth) as a the owner of the script which allows you to lock down configuration files and others that contain sensitive information so other uses can't read them (only set them to -rw------- and only you can read them, and that includes scripts if Apache runs them as you) - but most hosts don't do this (or they didn't last time I was working in that arena) as it is more hassle to setup and/or because it requires more resources. And by "more hassle to setup" I simply mean that it means more than just the out-of-the-box configuration: the "leading" standard control panel back than was cPanel (it may still be, I've not kept an eye on the market recently) and seeing posts like http://www.linuxgo.net/howto-enable-suphpphpsuexec-on-a-cpanel-server/ indicates that it still does not offer an easy (from the point-and-click PoV most cheap hosts need as they are rarely Linux/Apache/other experts) route to using the more secure arrangement. Most hosts will consider the extra admin time of setting up the more secure options to not be worth keeping (or gaining) your custom - 99%+ of their target market don't care (or don't know any better) and spending time to satisfy the other 1% or less is not worth it to them.
tl;dr: You will probably find this is the standard setup on a great many shared hosts, possibly most, maybe even nearly all. To ensure you are getting a new host that does things more securely when you move, you need to ask some pre-sales questions that are fairly technical (in the sense that sales may not be able to help, unless the company is small enough that the sales and tech support teams are the same people).
I would suggest instead using a VPS provider or self-hosting, that way there are no other direct users of the machine (be it real or virtual) to worry about, but unfortunately both of those options put more administrative load (and cost, unless you are paying far too much for shared hosting) on yourself and can be a minefield of its own (as with shared hosting avoid the cheapest options and ask searching question
chroot
Cd to the directory where all websites are. Then type:
chmod -R 000
That will learn them, and you deleted nothing.
I currently have shared hosting and I'd like to know if my host is affected. I'd rather not sit with this in the back of my mind.
Wasn't the point of IPV6 to replace IPV4? Give the VPS an IPV6 address. (visibility problems are its problems)
If you're expecting to host a web site on this sort of VPS, you won't be able to reach viewers behind IPv4-only home ISPs or using IPv4-only customer premise equipment. Or do you expect home IPv6 to be widely deployed before April 2014, when Windows XP reaches its announced end of life?
Why do you think ISPs are not co-operating? Mine certainly appears to be.
It's not that one ISP needs to cooperate as much as that all relevant ISPs need to cooperate. If even one major home ISP doesn't offer IPv6, then I'm going to get complaints from frustrated viewers if I advertise an IPv6-only web site to the public.
I had a similar problem with a site that rented servers. They imaged a new server with Centos for me. My investigation found that it had the same SSL private key pair...and other key pairs that other imaged machines on their system have! I fixed it myself, but I suspect that there are hundreds if not thousands that do not know this.
Great. He tells it how it is and you asswipes down mod him for it.
So you'd rather downgrade your security or pay for additional IP addersses to support XP users who can't bother to download firefox?
Yes. Operators of hobby sites would rather downgrade their security, and operators of commercial sites would rather pay for additional IP addresses because customers who run Windows XP and can't bother to download Firefox or Chrome (or lack permission to install it) might spread the word to other people that the site is broken for them.
Roughly 11 percent are using Internet Explorer 8* (source: caniuse.com), which is the latest version of IE for Windows XP, and about 1 percent are using IE versions. This far exceeds the usage share of Safari for Mac OS X and Safari for iOS combined. So if you were to block IE 6 through 8, you'd be telling one out of every eight viewers to patronize your competitor.
* I'll admit that using IE < 9 as a proxy for IE on XP is imperfect. But I imagine that the usage share of IE 7 on Windows Vista, IE 8 on Windows Vista, and IE 8 on Windows 7 is minimal compared to IE on XP.