Slashdot Mirror
How To Prevent Being Hacked Via Backups?
Popsikle writes "A few days ago one of the Web's largest hosting discussion forums was supposedly hacked via their backup servers. From the story: 'We've since learned that this very deliberate, sophisticated and calculated hack against Web Hosting Talk was carried out by gaining access to our offsite backup servers. From our backup servers, the hacker gained access to the WHT db server. The malicious attacker deleted all backups from the backup servers within the infrastructure before deleting tables from our db server. We were alerted of the db exploitation and quickly shut down the site to prevent further damage.' What sort of security do you put on your backup infrastructure? Looking at your backup solution could you be completely taken down by either someone obtaining a backup or accessing your backup servers? What sort of recommendations does everyone have for this not to happen?"
Offline and offsite storage (i.e. iron mountain) is a simple (though sometimes costly) way of doing things.
it'll solve this problem quite easily.
"Omnis tuus capsa sunt inesse nos"
Take a lesson from Ghost in the Shell, hire digital Tachikoma to protect you :) Problem solved!
"I Don't Have Enough Faith to be an Atheist"
>>What sort of security do you put on your backup infrastructure?
It depends.
I guess it depends how valuable the data is, how current it needs to be. Does it need to be kept secret or simply kept uncorrupted? How fast do you need to access these backups. The harder for you, potentially the more secure.
For starters let's have a copy off-line in a location where the servers are not. Heck, choose another location for more security. I am thinking: A tape and a DVD in a safe with a lock on it. And another somewhere else.
If you really need to keep it safe, commit it all to memory and then shoot yourself in the temple.
Encrypt your backups.
Don't let your backup system have access to your main system.
Allow your main system write-only access to your backup system, for the sole purpose of delivering new backups.
1) divide your eggs in at least two baskets, thoughtfully designed to protect their integrity
2) keep your baskets in physically isolated locations
3) take steps to protect your eggs from theft
4) after retrieving your eggs, inspect them for tampering before using them in your souffle
5) purchase insurance for the off chance you get yolk on your face
Never ascribe to malice what can be adequately attributed to ignorance. -Napoleon
accessible in the first place? Somebody in IT was not doing their job.
Why were your backup servers accessible from the outside network? Your backup servers are arguably even more valuable than your production servers. They should be behind yet another firewall. You can even have the production servers connect to them through a separate network interface. (Network interfaces and Cat5 cables are cheap.) If you are really paranoid, you can create folders where the server can upload data, but can't erase or overwrite what it has uploaded.
Easy, don't do backups! 99% of the world is already way ahead of you on this. Hard drive failure is a myth, anyway.
You see? You see? Your stupid minds! Stupid! Stupid!
I'm no expert, but if it were me, my off site backups wouldn't be accessible via the net at all, they'd be completely offline. I'd back up locally to a system not connected to the Internet at all then move my backups off site manually.
If you really need to keep it safe, commit it all to memory and then shoot yourself in the temple.
hey, the guy might NOT be jewish.
did you consider that?
--
"It is now safe to switch off your computer."
There was a very blatant oversight and an unfortunate assumption on the part of WHT and iNET Interactive.
They quite obviously overlooked the fact that the WHT servers (and ONLY the WHT servers) would ever need routine access to the backup servers. Therefore it was an obvious security hole that could have been plugged by restricting traffic through iptables to only iNET-affiliated IPs. Any teleworkers who needed access should simply use a VPN to iNet's offices if they really need access to the backup systems. If under some extreme circumstance (such as the loss of a database) an outside party needs access to the backup servers, the system admin can then add an exception under iptables.
And on that note, the other incredibly thoughtless assumption was that any traffic coming from the backup servers would be approved traffic. So once the attacker gained access to the backup servers, the database servers were one insecure hop away.
I think this proves the following very important points to the entire IT industry:
1) Internal infrastructure should remain just that - internal! Restrictions should always be put in place as to who can (or can't) access a system.
2) No traffic can be guaranteed authorized or authentic. It's one thing to add an SSH keyfile to your home servers, but in an enterprise environment everything must be highly scrutinized. It's no longer a matter of protecting systems from users - it's now a matter of protecting systems from other systems as well.
I was personally affected by the loss of information at WHT and while it's annoying, it's a fact of life and can't be undone. All that's left now is to pick up the pieces, secure the site as best they can and move on with lessons learned.
Well done sir. Either way, it'd be a horrible way to parish.
have backups delivered to backup site physically, with backup site not actually connected to any network.
so basically unless they nab your delivery guy or show up at the site with guns, it should be safe(tm).
in my mind this is half-funny/half-hmmmm-maybe
>>What sort of security do you put on your backup infrastructure?
It depends.
I guess it depends how valuable the data is, how current it needs to be. Does it need to be kept secret or simply kept uncorrupted? How fast do you need to access these backups. The harder for you, potentially the more secure.
For starters let's have a copy off-line in a location where the servers are not. Heck, choose another location for more security. I am thinking: A tape and a DVD in a safe with a lock on it. And another somewhere else.
If you really need to keep it safe, commit it all to memory and then shoot yourself in the temple.
I think you are missing an even bigger problem. How many of your servers does your backup infrastructure have admin access to - or if not admin elevated access? Are your backups a push or pull? If they are a pull you now have INBOUND firewall rules from your backup segment into your other network segments allowing the backup server to talk and start the pull. Or if you have a backup segment, there is not firewall protection, except host based firewalls, which have rules to allow the backup server to talk.
Yes having multiple copies is a good idea, but backups are a very dangerous thing security wise if not done right and secured properly.
The simple solution is to have backups that go to physical media which is offline and not accessible from any computer system without human physical intervention. The only way to tamper with this backup would be to get physical access to it, which is much harder and much more dangerous/risky than cracking into a system remotely. A periodic rotation of tapes offline will prevent this kind of attack from succeeding in bringing your systems down and loosing all your data.
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
Your backup server should have no access whatsoever to your production server. Your production server should access your backup, not the other way around.
Was the best movie of all time.
Even if you need to use rsync where read access to the main server is needed...Use NFS to export the RO file system from the main server to the backup. No write access from backup whatsoever.
Also, firewall the backup heavily. Backup should really only needed to make outgoing NFS connection, and incoming SSH perhaps.
Turn off the backup when not needed. Wake up on ACPI Alarm (or less ideal, WOL by others). This should make it more secure, and saving some powers too. *Potentially* this makes it stronger against electrical strike too.
If you have adequate physical security in meatspace, then it's reasonable to assume your backups are not significantly more vulnerable over and above your main cubical farm.
It is also an reasonable initial assumption when setting up shop, that backups of any kind should be situated in a more bomb/fire/quake/flood/theft proof place than your main operation. That goes for external intrusions to your hot failover systems too.
After logging in slashdot still does not take you back to the page you were on. It's been that way for 20 years.
OK, I think we've done this one to death.
Why would there be a different security policy for the backup server or media?
I think you need to altar your attitude.
Ahh - My eye!
The doctor said I'm not supposed to get Slashdot in it!
I understand why the most recent backup would be online, but I assumed it was standard procedure to have the remaining archive offline.
"Thanks for all the money you paid to us. We've used it to buy off ISO among other things" -Microsoft
Use SSH to connect to your backup server, do not use port 22, but something higher than 10,000. Generate a large key 4000-6000 bit range, to connect up...no passwords allowed. Do not use Root but rather generate a random user name..not "backup". Also, you can set your IP's in the hosts.allow file on the backup servers to futher restrict who can connect. Unless I'm missing something, this is quite foolproof..even to a talented fool.
I couldn't care less if any of those machines got compromised. Ok, the onsite nfs backup servers would be kind of weird, since they are behind my firewall. So if they compromise my backup servers they could as well compromise any other machine on that lan. Offsite backups are stored via ftp using Duplicity. Even if they compromise the backup servers they would still have to break my n-digit random passprase in order to look into it. And even if they took a look inside my backups it still wouldn't compromise my network, since none of the passwords are stored in clear text.
I don't understand. The attackers gained access to the database...through the backup servers? That leaves me with two questions:
1. Why were the backup servers accessible to the attacker?
2. Why was the database accessible from the backup servers?
It seems to me that the only way you need to access the backup servers is through some mechanism that allows you to transfer data to and from them. A single open port, which you need a password (or key) to use seems all that should be exposed. That shouldn't be too hard to secure.
It also seems to me that the backup server has no business accessing the database, and therefore shouldn't be able to access the database.
Unless, of course, the way the system works is that the backup server accesses the production server to retrieve the data from it. That doesn't seem the most obvious design to me, but it would at least explain why the backup server could access the database. Maybe that is a good reason not to design the system that way (on the other hand, it saves complexity on the production server, which is good). At any rate, it doesn't answer the first question, which is why the attackers were able to access the backup server.
My sympathy goes out to the WHT administrators. Good luck on recovering from this and figuring out what went wrong. I hope you will keep us posted, so that we can all learn from this incident.
Please correct me if I got my facts wrong.
If he was Jewish mocking him about the events of 70 AD this way would be pretty damn insensitive!
It's a special, copyrighted, variant spelling of soufflé. Apparently it also requires equally special baskets and eggs.
If you really need to keep it safe, commit it all to memory and then shoot yourself in the temple.
... was here!
We have a back up server the hosting company provides, and that is on a seperate subnet, no external facing ip. In addtion, we have an FTP server at another hosting co, and finally the emergancy back up in the office, which gets backed up to an external hdd and DVD.
"The problem with socialism is eventually you run out of other people's money" - Thatcher.
What they describes is DR, not a backup. Backup is offline media not online servers.
"Passwords are hashed with salt. It would be an unprecedented event to reverse engineer our passwords."
Yea, sure, like most users will choose good passwords when registering for such forums. Having the salt will slow down an attack against all passwords at the same time, but that's all. A normal brute force attack even against a "light" dictionary will probably reveal enough passwords for any evil purpose (like spam).
And really for a forum that's called "WebHostingTalk" they should be better at, well, WebHosting. And really the security of the backup server is trivial compared to the production server. Of course it needs to be considered seriously but it is a much easier problem: you don't have to deal with tons of services, demons, mysql, tangled forum/gallery/etc software and most importantly you need to keep it online only a fraction of the time and it doesn't need such a fast connection if you do the backups with any intelligence.
What's more you can encrypt everything towards a public key (it can be done either by the production machine or by the backup server). The secret key can be safely backup up in any fashion you like offline (it's a tiny, tiny file).
There's a cool way you can have backups that you can bring only very quickly with FreeBSD. You build a jailed environment, then just tarball, then encrypt the whole thing. Then host it in another country.) If your server fails, you simply bring another one online and load the same jail (userspace vm). You can automate database backups, and just make sure that they are encrypted, and sshtunnel>>ftpd to a secure firewalled location. Backup your VM whenever you make a software change, and backup the database and scripting files daily (or as needed). Encrypt whatever you need, and store the files in a location where they cannot be executed or accessed without secured access. The backups can be automated from the host system crontab, (ex: tar -czf $date-jailbackup.tar.gz /path/to/jail). Since the Jail has no access to the host system, and the secured backup server has no access to the host either, then it's relative secure.
Probably was a former employee who did the job. At work we always joke to make a backdoor, ready to delete all backups in case we get fired. Unfortunately we have fireproof safes and off-site backups...
You don't do backups, you do redundancy. If your IT model depends on one server doing its thing right, you have already gone down the path the doom.
Too soon?
This is not the funny you're looking for.
Dunno, but our backups are so secure that even the DBAs are usually unable to restore them. We might need this evil hacker to teach them a trick or two.
It is goes off site encrypt it. That's what I do.
You have to compress the data first as the tape drive can't do its own compression on encrypted data.
The only exception is content that's meant to be public in the first place.
set up this with your isp.
Slashdot doesn't support Unicode, so the e-acture came out mojibake-style. (As for why Slashdot doesn't support Unicode in 2009, see if you can think up a decent explanation, I can't...) It's not the grandparent's fault.
(1)DOCOMEFROM!2~.2'~#1WHILE:1<-"'?.1$.2'~'"':1/.1$.2'~#0"$#65535'"$"'"'&.1$.2'~'#0$#65535'"$#0'~#32767$#1"
Quite often the service providers being discussed are organized criminal rings (e.g. RBN) or are simply fraudulent. I am guessing that WHT accidentally upset one of these criminal web hosters which would explain the attack.
I really do hope that they catch the people responsible.
Tape drive (or even other external device) with encrypted data. There, problem solved. Do you see why such devices/features are standard on anything that has the word "backup" on it?
However, if you insist on having servers running all day long that you want to backup to, then at least make sure they are a million times more secure than your production servers - as the name suggests, they are your BACKUP if anything goes wrong. This means - encrypted data, locked-down networking (i.e. absolute minimum necessary - one port, one user, one ultra-secure key), proper permissioning (it might not be a bad idea to set the permissions so that you can append to a file but can't read it, thus forcing you to have physical access in order to restore any data from backup, or certainly that you can't overwrite existing files) and physical security (i.e. properly hosted in a decent hosting outfit and/or securely placed in an offsite location where they can't be got at - even if this is in a secure cage in a branch office).
And letting the backup server have ANY permissions on your local network is just stupid. Push, don't pull. Tell the backup server what to backup, don't let it pick and choose and require access back to your network - it's a backup device not "just another machine". A simple "nothing outbound" firewall rule on that machine would have stopped them coming back to you, providing the backup was encrypted (I'm assuming they actually piggy-backed onto your network than stole the db passwords from your backups). And the backup should ALWAYS be encrypted because it might well contain information such as your passwords in it, especially so if you are storing other people's data.
Yeah, it costs money to do this properly, but that's the price you pay for not losing thousands of people's data. I imagine the kick-back from that data loss will run to more than the price of a tape drive or two in the long run. What they had was NOT a backup. It was a rapid-restore machine. That's fine to have *as well as* a backup, but no better than hanging a 250Gb USB drive off the database server (in fact, worse, because that machine was able to be remotely-compromised).
As a design engineer (self employed, work from home) my stuff goes into a Firewire drive via Time Machine. It and the MacPro are both chained to a metal pole, so that covers theft I would think. About every 3 months or so I make an incremental backup to yet another DVD and give it to my sister who keeps it at her house, and that covers fire. I'm comfy with that, and none of my stuff is in the hands of some vulnerable cloud computer.
If I didn't have absolutely NOTHING to do, I wouldn't be here.
Freakin' nice...
Did the hacker attack the backup servers through the internet? Or did he gain physical access to the servers?
Makes a big difference - you can secure the machine from the internet, it's hard to secure it from someone sat at the keyboard...
"Some days even my lucky rocketship underpants don't help."
Bacula - In comes in the night and sucks the essence out of your servers
open source enterprise backup software, supports SSL transport encryption, and AES storage encryption. Supports most any type of media, optical, tape or files on an hdd.
seems like a much better solution.
(As for why Slashdot doesn't support Unicode in 2009, see if you can think up a decent explanation, I can't...)
Chinese spam
(As for why Slashdot doesn't support Unicode in 2009, see if you can think up a decent explanation, I can't...)
Slashdot is written in English, whose native words do not use characters outside US-ASCII. Characters outside US-ASCII have been used to attack the site in the past. Therefore, Slashdot staff put no effort into making sure that characters outside US-ASCII can be used in comments.
it might not be a bad idea to set the permissions so that you can append to a file but can't read it, thus forcing you to have physical access in order to restore any data from backup
Even for a restore to a sandbox system for testing the backup media?
Excuse me being lame, but can you please explain the reference?
Do you or your partner snore? - Visit www.snoring.com.au
i had to scroll 5 pages through troll spam until i was able to get to any real comment in 2 of the stories i read today. this is getting annoying. we should either truncate the immediately visible length of posts to some low number enough for a paragraph so spamming wont necessitate scrolling. or ban their ips.
Read radical news here
If you think jail is secure, you really need to not be working in the security field.
First off, lets focus on one thing, if your company is already at a size, complexity, or business need of having backup data electronically replicated offsite (whether this is a true hotsite you're replicating to, or just a method for not rotating tapes through your front door every day), then we're not talking about USB drive data protection schemes here...
OK now, the offsite system should NOT be an administrator access node of your existing backup solution. It should receive replicated data sets and have the ability to operate as a DR server to recover them, but it should be controlled remotely from YOUR site, and should not be able to initiate backups, restores, or make operations changes to another site's DR servers. This is handled by wither firewall rules or user structures.
Second, Physical access at the DR site needs to be as or more strict than at your primary site. If you don;t own the building (or host services through someone like SunGuard) they should have your units in locked racks. Insist that anyone with access has passed Government C2 or higher clearance, and DO NOT give ANYONE outside your organization passwords to those systems.
Next, If the system allows, the user accounts at the hot site should not be the same at any of your other sites.
Next, You STILL NEED TO ARCHIVE DATA, preferably monthly at worst, weekly may be needed depending on your data set size. Replicated systems can corrupt data, and replicated date sets typically only contain your current data set (no rollback copies). Generally, if your replication system is handling moving data automatically, you can leave your local DR copies in your building in a secure area. Daily backups should be archived to local disks so if you have a DR server failue you still have something to recover from (recovery over the WAN is usually not an option). If you have a fire or disasterous event, your offsite systems is your #1 backup, last month/week's archive is your #2.
I've spent a lot of time working with multiple DR companies, and I've supported medium and enterprise class businesses alike (my current employer has 12 mainframes, a couple hundred AIX servers, and over 2000 x86/64 boxes. DR is a BIG thing around here!) Once you enter enterprise class, we're not typically talking "backups" as you know them, more SAN replication and disk write journaling. Recovering 400TB from backups simply isn't an option in a 24-48 hour recovery model! However, if you've got less than 50TB, you can VERY REASONABLY build a great solution out of DPUs (Data Protection Units) made by Unitrends (unitrends.com). They're totally tapeless systems, can share jobs between multiple units at multiple sites, have secure packet level replication technology, and can use 256bit real time encryption as 8GB/minute (per chassis!) with up to 7 concurrent jobs running (per chassis!). a 50TB environment would use 4-8 DPUs (usually departmentalized) conencted to a SAN, and another 1-2 units offsite to handle replication and DR recovery. They also support P2V and V2P recovery and dissimilar recovery. Fully integrated BareMetal backup (they actually invented and patented that one!), and support for 40 operating systems in one multi-site seamless DR solution. You should REALLY look into them. For small sites (less than 1TB data) you can get into one of these units for around $15K with all the bells and whistles (another 10K for replication). There are NO PER CLIENT OR PER AGENT LICENSING FEES, it's ALL inclusive, the only thing you license is storage. Oh, their Data Vault unit can handle up to 40 sites, and each site can be run by a different company with different accounts, and their data can be logically seperated on the vault making it impossible for one customer to interfere with another's data, you can even lock the vault operator out of the solution. This has been validated by the DOD as a secure solution...
There is no contest in life for which the unprepared have the advantage.
If you have critical data in systems, hire a reputable technical architect to work with you on: /.
- Importance of your backups; the criticality of the data will determine your budget
- RTO/RPO - google it if your don't know these terms
- Disaster Recovery - if you had a workable DR plan, your systems would be running now - not posting questions to
- Encrypt at the source
- Near/Online backups; Far/Offline backups - both have there place
- Physical separation **is important** and for some systems, 500+ miles is important
- Limit access to the back store(s) on top of encryption. Don't use a trivial password for the data either
- Use Physical access controls and/or firewall rules to limit access.
- Use ssh tunnels with key exchange (no passwords) to control login access
- A tape in a non-networked vault can't be hacked remotely.
- Design your systems such that a single failure anywhere doesn't prevent them from working.
- Never under estimate the bandwidth of a station wagon full of tapes.
- TEST THE RESULTS - if you don't test your DR or backup efforts, you just wasted a bunch of time for ZERO return. After going through all this for a few years, then the day comes and you can't recover from your backups, how cool is that?
If you have just a few TB to store, this is a trivial problem. There are trade offs between ease, security, viability, and recovery time.
You can google on each of these things to build a backup solution. It can be done with free tools, but it is much easier and usually more reliable to pay someone, an expert, to set this up. I consider myself an expert, so we're doing our enterprise DR/backups with free tools and validate the solution every time we upgrade a system. I take the last full backup and perform the upgrade on that image on a different system as a test before touching the real production system.
I shopped around for a few months for my offsite, online backups, and most providers were adjuncts of larger ISPs, and the backups were generally stored on larger, general purpose servers.
Usually this was in conjunction with all sorts of extra "services" tied to the backup. But the bottom line was, I was storing files on a server that was running imap and pop and PHP and all manner of other services and ports open, etc.
That's a mistake. The backup provider I use now (rsync.net) has three services running (I nmap my target regularly):
- ftp (I don't use it)
- ssh
- https
No php, no app servers, no mail servers, etc., and when I asked them, they confirmed that their ftpd is just plain old FreeBSD built-in.
Oh, and I encrypt the backups with duplicity, which is absolutely fantastic.
For backups where I work, we've got a few different systems in place. We've done away with tape backups long ago, and our primary backup system is a disk-based system. Twelve drives in a RAID 6+0 for maximum availability and ability to recover from disk issues. This system keeps a daily snap shot from the past week, a weekly snap shot for the past month, and a monthly snap shot for the past six months. We then place the most recent daily snap shot on a USB external disk drive, which we carry off-site each day. This disk is encrypted and not connected to a network or otherwise available except physically.
This has worked well, and if the proverbial crap does hit the fan, we've got our off-site disk for last-chance recovery if the world comes to an end.
Look up Jewish temple.
I'm glad at least you add an interesting comment... Ohwait. No, you're adding to the spam by posting offtopic rants.
Besides that, you might want to have a look at the thresholds, that way you can define exactly what content you want to see.
But no, whining is easier uh.
Okay, let's sort something out here. They didn't get hacked by "backups," which would imply that somehow the backups themselves got trojaned or modified to cause compromise. They got hacked normally; the backup servers were just the means by which this took place. This is nothing new or exotic, since backup software is software like any other, and prone to vulnerability. In fact, there have been a lot of issues found with software from many vendors. Unfortunately, companies rarely patch their backup systems, and so these vulnerablities tend to stick around a while.
What makes it all worse is the fact that backup servers have access to read just about everything as a matter of necessity. After all, if they can't access the whole filesystem, they can't back it up. So you have the combination of durable vulnerabilities with a system that pretty much has the keys to the kingdom.
Oh, and earlier suggestions related to offsite backup storage? Irrelevant.
For your security, this post has been encrypted with ROT-13, twice.
Please elaborate, you've caught my eye.
I think using an ecrypted jail is a very effective way to rapidly restore a lost server (a true backup) and is not inherantly insecure in anyway.
However, I am curious how it is that a FreeBSD jail is insecure? Certainly the userland isolation can't hurt security?
A lot of people are suggestion that backups be encrypted and I assume they mean the actual files/volumes, but I *really* don't see the usefulness of this case. Encryption backups may protect the data from being stolen, but it isn't going to protect it from being *wiped* in most cases. If you have root access then assumedly there's also raw access to device the backups reside on, in which case you can still nuke it with something as simple as "dd if=/dev/zero of=/dev/backup"
Now maybe if the backup server further connects onward to a mass-storage device that isn't at-all accessible until logged-in, perhaps, but it doesn't seem a likely scenario for most cases.
Borrowing from the military mindset this is my suggestion:
"Any online backup locations shall have a physical break in the networking infrastructure that must be phsyically connected as needed. Once operations are complete the physical connection is again broke until it is needed."
(IMPLEMENTATION A)
From an implementation standpoint what you have at the core switching rack is a RED cable in the switch that is the connection to the backup server (onsite, i.e. disk to disk) that is unplugged at all times except when the backups are to run.
(IMPLEMENTATION B)
Tapes, removeable disks, are stored off site, offline. In the event that they are needed they are either order and shipped, or a temporary connection is made to the dat warehouse and the data delivered via VPN\SSH. Once the restore is completed the data is taken offline and stored.
(IMPLEMENTATION C)
Using a pair of hardware VPN routers, at each endpoint (server site, backup site) they sit, unconnected to the networking fabric until a request is made or the scheduled backup\restore windows are near.
YOu cannot remote hack (normally) devices that are stand alone.
-=[ Who Is John Galt? ]=-
Christ, you guys are merciless. Next you'll be wanting to nail him to a tree or something.
Jesus, I come here for the intellectual discussion. I didn't expect this kind of Spanish Inquisition.
09F91102 no, 455FE104 nope, F190A1E8 uh-uh, 7A5F8A09 that's not it, C87294CE no. Ah! 452F6E403CDF10714E41DFAA257D313F.
I lost all respect for WHT years ago. It went down the tubes after Chicken left. Note they're well in with Rackspace who capitulated to SCO. Couldn't happen to nicer people.
what sort of security do you put on your backup infrastructure? the same security you put on your primary infrastructure, of course. it's still your data isn't it? your customers' private information? you're still charged with the duty of securing and maintaining that data due to hipaa and sox requirements aren't you? how is backup data any less valuable to an attacker?
Nobody expects the Spanish Inquisition!!
I think you need to altar your attitude.
Well played sir.
HDs are NOT backup media.
It appears you've taken the oft-repeated mantra that RAID is not a backup solution and gone a step further by suggesting that hard drives themselves aren't backup media.
Which, by the way, is ludicrous. Hard drives use tried-and-true technology, they're cheap as hell, and transfer speeds are outrageously faster than any other media in contention. Suggesting that they don't make good backup media is well, I'll give you the benefit of the doubt and say misguided.
My offsite backup server is currently running Ubuntu Server with full disk encryption. The only access it has is with key based SSH on a non-standard port, restricted to a specific subnet. There are no ports forwarded to this machine; instead, it initiates a VPN connection from it's location directly to my local network. It gets a static IP on the VPN subnet, and that IP is the only subnet SSH will listen to.
I use a combination of Rsync over SSH and rsnapshot server side.
I'm not storing any hyper-critical data here, just backups for a webmail/website combo I run for a dozen friends or so. I just like it to be fairly secure. Anything I'm missing here?
Yeah, jails are totally insecure; they're full of criminals.
Anyone that lets other third parties do stuff for them, is essentially trusting someone else to do a job properly. This is most likely a bad idea if you are dealing with mission critical data. Also the question should be asked...what happens when it goes wrong? When would you know of a problem? It can be difficult for people to own up to things when they make a bo bo... especially if they don't even directly work for the company in question.
A critical task such as backup of customer DB should be done locally and the backup taken off site. For a 99.9% peace of mind you would check the data integrity before taking off site. Of course any data should be encrypted using a reasonable number of bits for the key. The number of bits would be relative to the value of the data.
Treat your backups, and your backup servers, with every bit of seriousness that you treat your main systems. Use the same level of monitoring, management, and security. Investing good practice in your system while slacking on your backup only narrows the attack window, it doesn't tighten the security - your security is only as strong as your weakest link.
And someone needs to add a filter to slashdot to ban IPs that submit >10kbyte posts with "obama" in them in every single new article on slashdot. I'm getting sick of seeing the first 10 pages of every new thread by some retards going off on obama. AGAIN.
I work for the Department of Redundancy Department.
You guys are morons.
"Therefore it was an obvious security hole that could have been plugged by restricting traffic through iptables to only iNET-affiliated IPs. "
Do some google searches, wht was rooted back in august, it can be done internally.
Really? Who actually backs things up? *Ducks for cover*
I am pretty sure these people using Gmail's huge storage as their backup server.
I don't see any other reason why their backup server was accessible to the outside world.
And if you are thinking about they got access to the production boxes by hacking the backup server(gmail)
- it's pretty simple. They have emailed their production server passwords to their GMail ID, just so that
they don't forget it.
yes, because people regularly break into my NOC, pull the drives out of my backup servers and practice juggling. Tapes are rubbish. Tape is expensive and unreliable. Anyone that tells you otherwise is selling the stuff.
Hard drives have replaced tapes around here. For decades we juggled tapes in a vain attempt to maintain useful backups only to find that EVERY SINGLE TIME we needed critical data from backup, the data was unrecoverable.
For the past few years now, we have been backing up all of our critical data to a low powered server with TB drives in mirrored arrays. Security on this server is exteremly high. The only service it runs is SSH, backups are all done as pulls from the servers, has no untrusted local users and sits on an extremely restriced network segment.
With the thousands we have saved not replacing tape media and drives (not to mention the amount of overtime not wasted screwing around in the middle of the night trying to find a working tape to restore from) we are adding a mirror for this archive offsite.
the above is my personal opinion and does not necessarily reflect that of the little voices in my head
The server itself probably wasn't a backup so much as a hot spare.
This is extremely common with web servers currently. You get 1 server with The Planet and one somewhere else and use the faster/better one as your main one and mirror your changes to the spare.
The article states the backups were deleted off the backup server and then the person gained access to the main server.
There was probably a mysql master/slave scenario using the same login/pass for easy sharing.
There was also probably a shared drive between the two servers to perform automatic, scheduled VSS backups over the Internet.
I have seen hundreds of these setups from 'companies' or small hosting places that cannot afford a real cluster or multiple servers in multiple locations.
If you really need to keep it safe, commit it all to memory and then shoot yourself in the temple.
hey, the guy might NOT be jewish.
did you consider that?
Works for Shikhs, Hindus, Buddhists and Mormons too!
We need a "+1 -- nice sig" moderation.
i dont want thresholds. i want to be able to see valid comments even if some fanboi mods someone's valid, insightful comment -1 due to fanboi spirit.
Read radical news here
Don't treat your backup/disaster recovery systems as any less critical than your production ones. I didn't find out anything about how their "backup" system worked from the forum post, can't think of a good reason to not lock things in both places or technical hurdle that would make it overly difficult to do.
Keep a 3 day backup and cross your fingers.
All of the feedback and suggestions here are great. Thanks for the support and suggestions. You can read the latest on what's going on over at WHT here - http://www.webhostingtalk.com/showthread.php?t=729727 Rob Farrell iNET Interactive
get thee to a punnery.
I suspect this problem was the result of outsourcing of some kind.
Companies figure that there is no way they could possibly get backups right, and don't want to pay to have somebody run them or set them up with an automated solution.
So, they outsource. Of course, they pick the CHEAPEST vendor out there. That vendor probably helps them punch some holes in their firewall so that they can remotely connect, log into servers using administrative passwords, and copy off data. That cheap vendor probably doesn't have robust internal security, and they have access of some kind into every one of their clients (who are too cheap to have decent security of their own). This sort of situation is just a disaster waiting to happen.
Bottom line is that you get what you pay for. Hire the admin willing to work for the lowest salary and you don't get a very good admin. Pay the cheapest rate available for outsourcing and you get cheap service. Cheap stuff can be a way to save money - but usually only when you have somebody competant to figure out when you can and can't get away with it. Hire cheap staff and cheap services and mix that with cheap stuff and you're going to get what you paid for.
No one expects the Spanish Inquisition
Nobody expects the Spanish Inquisition!
If you can gain access to the onsite backups of an organization, pollute them with your hacks and then cause some disruption to their system (Slashdot effect HD meltdown?) they will most likely put blind faith in that onsite backup as being pure and untainted until events transpire which lead them to question their security. By that time a well crafted hack will have served its purpose. This is why we should all be nice to the friendly IT folks in our organizations.
A lot of companies tend to treat backup or development environments as less important than their production environments, and have less protection for them.
Huge mistake.
You need to look not at the environment, but the data. If the data in your backup environment is the same as production, the protection needs to be the same. Same controls. Same technology. Same standards. Same restrictions. Same monitoring. Because the data in backup is just as valuable as the data in production.
"Don't need to worry about that stuff, it's just the backup." If someone steals it, do they still get credit card numbers, SSNs, account credentials? Can they still hurt the business if they delete it? Then you need to protect it like it's production.
Don't think about where the data is. Think about WHAT the data is. It's not about the environment, or the use. It's about the data.
Yeah - I'm visiting /. less these days other than to read the 'headlines'. Too much garbage in threads.
I could tell you but then I would have to kill you... Okay, you still want to know... Just don't say I didn't warn you! Each successive backup is made to alternating magnetic media and then periodically to optical media on a daily, weekly, monthly, quarterly, annual schedule. Each backup sequence is protected at separate, isolated mountain underground cave locations by 10 ninja guardian dogs and 100 ferocious, vicious, man eating attack pigs. If you get through those two rings of defenses then the marauding chickens will peck out your eyes while pooping down your throat. Survivors will be composted, along with the failures. So far nobody has stolen any of our backups so it must be working. The DVD optical backup does kind of make sure that nobody can erase the backups unless they manage to get through the rings seven rings of defenses. ("Seven you ask? Seven? He only described three rings...?" Yes, well, I'm not going to forewarn you about all the secret defenses! :) )
Seriously though, optical is much more permanent.
Seriously if your data is that important to the daily operations of your business then off site with third party vendor that specializes in archiving is the only way to go it removes some of the liability from you as a whole if they are the ones to let your system be compromised Generally the places they keep the medium in a place thats more secure than a bank(our vendor has an old silo with armed guards in an undisclosed location and yea I've been there). If this is to pricey then you should probably rent a safe deposit box and as part of the routine rotate the tapes
no matter how good it is, it is human nature always wants to make things better
Are you nuts? You got exactly what you deserve. There is NO security if you don't have physical security. Bring your backup severs in house and keep your backup tapes in a security site (such as Iron Mountain).
NOBODY expects the Spanish Inquisition!
Hey, when it comes to backups you're preaching to the choir (see sig).
512 MB RAM, 20 GB disk, 200 GB transfer, five datacenters. $19.95/month.