How To Prevent Being Hacked Via Backups?

Popsikle writes "A few days ago one of the Web's largest hosting discussion forums was supposedly hacked via their backup servers. From the story: 'We've since learned that this very deliberate, sophisticated and calculated hack against Web Hosting Talk was carried out by gaining access to our offsite backup servers. From our backup servers, the hacker gained access to the WHT db server. The malicious attacker deleted all backups from the backup servers within the infrastructure before deleting tables from our db server. We were alerted of the db exploitation and quickly shut down the site to prevent further damage.' What sort of security do you put on your backup infrastructure? Looking at your backup solution could you be completely taken down by either someone obtaining a backup or accessing your backup servers? What sort of recommendations does everyone have for this not to happen?"

Comments

Easy fix

[bingbong]

Offline and offsite storage (i.e. iron mountain) is a simple (though sometimes costly) way of doing things.

it'll solve this problem quite easily.

Re:Easy fix

[tepples]

HDs are NOT backup media.

Please provide a citation that hard disks are noticeably worse than tape, which you appear to otherwise recommend.

Re:Easy fix

[QuantumRiff]

No, they are not, you are correct. In my post, I was assuming that this was a small website or business, not a mission critical company product. I didn't mention software, or tape libraries, or hot backups. I think sometimes its better to have some backups, then none at all. An external drive can be bought for next to nothing.. really, I can get a 500GB HD for about the cost of a couple of tapes, but then I have to buy 2 tape drives (in case one has hardware failure).

Believe me, I could go on about backup windows, media, and techniques, but was hoping by keeping it simple, they would not be overwhelmed. By not being overwhelmed, they might start the process.

Also, by not using a computer based backup, they would not have the same problem as the site mentioned in the linked article, where someone first cracked their backup servers, and deleted their only backups.

Tachikoma

[Anenome]

Take a lesson from Ghost in the Shell, hire digital Tachikoma to protect you :) Problem solved!

Encrypt it

[micksam7]

Encrypt your backups.

Don't let your backup system have access to your main system.

Allow your main system write-only access to your backup system, for the sole purpose of delivering new backups.

Why were your backup servers

[Jane Q. Public]

accessible in the first place? Somebody in IT was not doing their job.

Why was this possible?

[RAMMS EIN]

I don't understand. The attackers gained access to the database...through the backup servers? That leaves me with two questions:

1. Why were the backup servers accessible to the attacker?
2. Why was the database accessible from the backup servers?

It seems to me that the only way you need to access the backup servers is through some mechanism that allows you to transfer data to and from them. A single open port, which you need a password (or key) to use seems all that should be exposed. That shouldn't be too hard to secure.

It also seems to me that the backup server has no business accessing the database, and therefore shouldn't be able to access the database.

Unless, of course, the way the system works is that the backup server accesses the production server to retrieve the data from it. That doesn't seem the most obvious design to me, but it would at least explain why the backup server could access the database. Maybe that is a good reason not to design the system that way (on the other hand, it saves complexity on the production server, which is good). At any rate, it doesn't answer the first question, which is why the attackers were able to access the backup server.

My sympathy goes out to the WHT administrators. Good luck on recovering from this and figuring out what went wrong. I hope you will keep us posted, so that we can all learn from this incident.