Slashdot Mirror
How To Prevent Being Hacked Via Backups?
Popsikle writes "A few days ago one of the Web's largest hosting discussion forums was supposedly hacked via their backup servers. From the story: 'We've since learned that this very deliberate, sophisticated and calculated hack against Web Hosting Talk was carried out by gaining access to our offsite backup servers. From our backup servers, the hacker gained access to the WHT db server. The malicious attacker deleted all backups from the backup servers within the infrastructure before deleting tables from our db server. We were alerted of the db exploitation and quickly shut down the site to prevent further damage.' What sort of security do you put on your backup infrastructure? Looking at your backup solution could you be completely taken down by either someone obtaining a backup or accessing your backup servers? What sort of recommendations does everyone have for this not to happen?"
Offline and offsite storage (i.e. iron mountain) is a simple (though sometimes costly) way of doing things.
it'll solve this problem quite easily.
"Omnis tuus capsa sunt inesse nos"
Take a lesson from Ghost in the Shell, hire digital Tachikoma to protect you :) Problem solved!
"I Don't Have Enough Faith to be an Atheist"
>>What sort of security do you put on your backup infrastructure?
It depends.
I guess it depends how valuable the data is, how current it needs to be. Does it need to be kept secret or simply kept uncorrupted? How fast do you need to access these backups. The harder for you, potentially the more secure.
For starters let's have a copy off-line in a location where the servers are not. Heck, choose another location for more security. I am thinking: A tape and a DVD in a safe with a lock on it. And another somewhere else.
If you really need to keep it safe, commit it all to memory and then shoot yourself in the temple.
Encrypt your backups.
Don't let your backup system have access to your main system.
Allow your main system write-only access to your backup system, for the sole purpose of delivering new backups.
1) divide your eggs in at least two baskets, thoughtfully designed to protect their integrity
2) keep your baskets in physically isolated locations
3) take steps to protect your eggs from theft
4) after retrieving your eggs, inspect them for tampering before using them in your souffle
5) purchase insurance for the off chance you get yolk on your face
Never ascribe to malice what can be adequately attributed to ignorance. -Napoleon
accessible in the first place? Somebody in IT was not doing their job.
Why were your backup servers accessible from the outside network? Your backup servers are arguably even more valuable than your production servers. They should be behind yet another firewall. You can even have the production servers connect to them through a separate network interface. (Network interfaces and Cat5 cables are cheap.) If you are really paranoid, you can create folders where the server can upload data, but can't erase or overwrite what it has uploaded.
Easy, don't do backups! 99% of the world is already way ahead of you on this. Hard drive failure is a myth, anyway.
You see? You see? Your stupid minds! Stupid! Stupid!
If you really need to keep it safe, commit it all to memory and then shoot yourself in the temple.
hey, the guy might NOT be jewish.
did you consider that?
--
"It is now safe to switch off your computer."
There was a very blatant oversight and an unfortunate assumption on the part of WHT and iNET Interactive.
They quite obviously overlooked the fact that the WHT servers (and ONLY the WHT servers) would ever need routine access to the backup servers. Therefore it was an obvious security hole that could have been plugged by restricting traffic through iptables to only iNET-affiliated IPs. Any teleworkers who needed access should simply use a VPN to iNet's offices if they really need access to the backup systems. If under some extreme circumstance (such as the loss of a database) an outside party needs access to the backup servers, the system admin can then add an exception under iptables.
And on that note, the other incredibly thoughtless assumption was that any traffic coming from the backup servers would be approved traffic. So once the attacker gained access to the backup servers, the database servers were one insecure hop away.
I think this proves the following very important points to the entire IT industry:
1) Internal infrastructure should remain just that - internal! Restrictions should always be put in place as to who can (or can't) access a system.
2) No traffic can be guaranteed authorized or authentic. It's one thing to add an SSH keyfile to your home servers, but in an enterprise environment everything must be highly scrutinized. It's no longer a matter of protecting systems from users - it's now a matter of protecting systems from other systems as well.
I was personally affected by the loss of information at WHT and while it's annoying, it's a fact of life and can't be undone. All that's left now is to pick up the pieces, secure the site as best they can and move on with lessons learned.
Well done sir. Either way, it'd be a horrible way to parish.
If your system is compromised for an extended period of time, and you don't know about it, your backups can still get tainted.
i.e. the hacker might modify your backup process in a subtle way, like getting random bits corrupted on the tape, sufficient to make it useless for restoring.
Hardly anyone actually completely tests their backups, to make sure they'll work.
After 6-12 months, the hacker hoses the real DB, not expecting their trap, you try to load from backup from physical media, only to make the situation even worse.
It could take you quite a long time to find a backup that's not hosed (assuming you keep physical media backups older than 12 months)
I think you need to altar your attitude.
Ahh - My eye!
The doctor said I'm not supposed to get Slashdot in it!
I understand why the most recent backup would be online, but I assumed it was standard procedure to have the remaining archive offline.
"Thanks for all the money you paid to us. We've used it to buy off ISO among other things" -Microsoft
I don't understand. The attackers gained access to the database...through the backup servers? That leaves me with two questions:
1. Why were the backup servers accessible to the attacker?
2. Why was the database accessible from the backup servers?
It seems to me that the only way you need to access the backup servers is through some mechanism that allows you to transfer data to and from them. A single open port, which you need a password (or key) to use seems all that should be exposed. That shouldn't be too hard to secure.
It also seems to me that the backup server has no business accessing the database, and therefore shouldn't be able to access the database.
Unless, of course, the way the system works is that the backup server accesses the production server to retrieve the data from it. That doesn't seem the most obvious design to me, but it would at least explain why the backup server could access the database. Maybe that is a good reason not to design the system that way (on the other hand, it saves complexity on the production server, which is good). At any rate, it doesn't answer the first question, which is why the attackers were able to access the backup server.
My sympathy goes out to the WHT administrators. Good luck on recovering from this and figuring out what went wrong. I hope you will keep us posted, so that we can all learn from this incident.
Please correct me if I got my facts wrong.
It's a special, copyrighted, variant spelling of soufflé. Apparently it also requires equally special baskets and eggs.
Probably was a former employee who did the job. At work we always joke to make a backdoor, ready to delete all backups in case we get fired. Unfortunately we have fireproof safes and off-site backups...
Too soon?
This is not the funny you're looking for.
Dunno, but our backups are so secure that even the DBAs are usually unable to restore them. We might need this evil hacker to teach them a trick or two.
Tape drive (or even other external device) with encrypted data. There, problem solved. Do you see why such devices/features are standard on anything that has the word "backup" on it?
However, if you insist on having servers running all day long that you want to backup to, then at least make sure they are a million times more secure than your production servers - as the name suggests, they are your BACKUP if anything goes wrong. This means - encrypted data, locked-down networking (i.e. absolute minimum necessary - one port, one user, one ultra-secure key), proper permissioning (it might not be a bad idea to set the permissions so that you can append to a file but can't read it, thus forcing you to have physical access in order to restore any data from backup, or certainly that you can't overwrite existing files) and physical security (i.e. properly hosted in a decent hosting outfit and/or securely placed in an offsite location where they can't be got at - even if this is in a secure cage in a branch office).
And letting the backup server have ANY permissions on your local network is just stupid. Push, don't pull. Tell the backup server what to backup, don't let it pick and choose and require access back to your network - it's a backup device not "just another machine". A simple "nothing outbound" firewall rule on that machine would have stopped them coming back to you, providing the backup was encrypted (I'm assuming they actually piggy-backed onto your network than stole the db passwords from your backups). And the backup should ALWAYS be encrypted because it might well contain information such as your passwords in it, especially so if you are storing other people's data.
Yeah, it costs money to do this properly, but that's the price you pay for not losing thousands of people's data. I imagine the kick-back from that data loss will run to more than the price of a tape drive or two in the long run. What they had was NOT a backup. It was a rapid-restore machine. That's fine to have *as well as* a backup, but no better than hanging a 250Gb USB drive off the database server (in fact, worse, because that machine was able to be remotely-compromised).
The one thing about encryption -- key management is just as important as the algorithm used. A business has to figure out how they are going to manage keys. Are they going to use a passphrase that only the backup admins will know, or are they going to use some type of RSA key functionality? If private keys are used, how are they backed up?
There are a lot of factors involved, and one important thing is not depending on one single person as one doesn't depend on one single server. There should be some type of mechanism put in place to deal with an employee gone missing (or even worse, rogue.) For a smaller business, the risks may be low, but its something that should be thought about when planning a backup infrastructure and implementing it.
First off, lets focus on one thing, if your company is already at a size, complexity, or business need of having backup data electronically replicated offsite (whether this is a true hotsite you're replicating to, or just a method for not rotating tapes through your front door every day), then we're not talking about USB drive data protection schemes here...
OK now, the offsite system should NOT be an administrator access node of your existing backup solution. It should receive replicated data sets and have the ability to operate as a DR server to recover them, but it should be controlled remotely from YOUR site, and should not be able to initiate backups, restores, or make operations changes to another site's DR servers. This is handled by wither firewall rules or user structures.
Second, Physical access at the DR site needs to be as or more strict than at your primary site. If you don;t own the building (or host services through someone like SunGuard) they should have your units in locked racks. Insist that anyone with access has passed Government C2 or higher clearance, and DO NOT give ANYONE outside your organization passwords to those systems.
Next, If the system allows, the user accounts at the hot site should not be the same at any of your other sites.
Next, You STILL NEED TO ARCHIVE DATA, preferably monthly at worst, weekly may be needed depending on your data set size. Replicated systems can corrupt data, and replicated date sets typically only contain your current data set (no rollback copies). Generally, if your replication system is handling moving data automatically, you can leave your local DR copies in your building in a secure area. Daily backups should be archived to local disks so if you have a DR server failue you still have something to recover from (recovery over the WAN is usually not an option). If you have a fire or disasterous event, your offsite systems is your #1 backup, last month/week's archive is your #2.
I've spent a lot of time working with multiple DR companies, and I've supported medium and enterprise class businesses alike (my current employer has 12 mainframes, a couple hundred AIX servers, and over 2000 x86/64 boxes. DR is a BIG thing around here!) Once you enter enterprise class, we're not typically talking "backups" as you know them, more SAN replication and disk write journaling. Recovering 400TB from backups simply isn't an option in a 24-48 hour recovery model! However, if you've got less than 50TB, you can VERY REASONABLY build a great solution out of DPUs (Data Protection Units) made by Unitrends (unitrends.com). They're totally tapeless systems, can share jobs between multiple units at multiple sites, have secure packet level replication technology, and can use 256bit real time encryption as 8GB/minute (per chassis!) with up to 7 concurrent jobs running (per chassis!). a 50TB environment would use 4-8 DPUs (usually departmentalized) conencted to a SAN, and another 1-2 units offsite to handle replication and DR recovery. They also support P2V and V2P recovery and dissimilar recovery. Fully integrated BareMetal backup (they actually invented and patented that one!), and support for 40 operating systems in one multi-site seamless DR solution. You should REALLY look into them. For small sites (less than 1TB data) you can get into one of these units for around $15K with all the bells and whistles (another 10K for replication). There are NO PER CLIENT OR PER AGENT LICENSING FEES, it's ALL inclusive, the only thing you license is storage. Oh, their Data Vault unit can handle up to 40 sites, and each site can be run by a different company with different accounts, and their data can be logically seperated on the vault making it impossible for one customer to interfere with another's data, you can even lock the vault operator out of the solution. This has been validated by the DOD as a secure solution...
There is no contest in life for which the unprepared have the advantage.
A lot of people are suggestion that backups be encrypted and I assume they mean the actual files/volumes, but I *really* don't see the usefulness of this case. Encryption backups may protect the data from being stolen, but it isn't going to protect it from being *wiped* in most cases. If you have root access then assumedly there's also raw access to device the backups reside on, in which case you can still nuke it with something as simple as "dd if=/dev/zero of=/dev/backup"
Now maybe if the backup server further connects onward to a mass-storage device that isn't at-all accessible until logged-in, perhaps, but it doesn't seem a likely scenario for most cases.
Christ, you guys are merciless. Next you'll be wanting to nail him to a tree or something.
Jesus, I come here for the intellectual discussion. I didn't expect this kind of Spanish Inquisition.
09F91102 no, 455FE104 nope, F190A1E8 uh-uh, 7A5F8A09 that's not it, C87294CE no. Ah! 452F6E403CDF10714E41DFAA257D313F.
HDs are NOT backup media.
It appears you've taken the oft-repeated mantra that RAID is not a backup solution and gone a step further by suggesting that hard drives themselves aren't backup media.
Which, by the way, is ludicrous. Hard drives use tried-and-true technology, they're cheap as hell, and transfer speeds are outrageously faster than any other media in contention. Suggesting that they don't make good backup media is well, I'll give you the benefit of the doubt and say misguided.
yes, because people regularly break into my NOC, pull the drives out of my backup servers and practice juggling. Tapes are rubbish. Tape is expensive and unreliable. Anyone that tells you otherwise is selling the stuff.
Hard drives have replaced tapes around here. For decades we juggled tapes in a vain attempt to maintain useful backups only to find that EVERY SINGLE TIME we needed critical data from backup, the data was unrecoverable.
For the past few years now, we have been backing up all of our critical data to a low powered server with TB drives in mirrored arrays. Security on this server is exteremly high. The only service it runs is SSH, backups are all done as pulls from the servers, has no untrusted local users and sits on an extremely restriced network segment.
With the thousands we have saved not replacing tape media and drives (not to mention the amount of overtime not wasted screwing around in the middle of the night trying to find a working tape to restore from) we are adding a mirror for this archive offsite.
the above is my personal opinion and does not necessarily reflect that of the little voices in my head
i dont want thresholds. i want to be able to see valid comments even if some fanboi mods someone's valid, insightful comment -1 due to fanboi spirit.
Read radical news here
Keep a 3 day backup and cross your fingers.