Slashdot Mirror
Hacker Destroys Avsim.com, Along With Its Backups
el americano writes "Flight Simulator community website Avsim has experienced a total data loss after both of their online servers were hacked. The site's founder, Tom Allensworth, explained why 13 years of community developed terrains, skins, and mods will not be restored from backups: 'Some have asked whether or not we had back ups. Yes, we dutifully backed up our servers every day. Unfortunately, we backed up the servers between our two servers. The hacker took out both servers, destroying our ability to use one or the other back up to remedy the situation.'"
To any sysadmins and DBAs...
Make sure you have offsite backups
I realize that from quite a few people's perspectives, storing their backups in a separate building constitutes off site storage. I'd almost buy that strategy. Not in the same environment, network, city etc.
These guys were stupid.
The day after 9/11 I was in an elevator, and caught a snippet of conversation between 2 people that had business interests with a firm that was in the WTC. The comment I heard was 'their backups were in the other building'. Another company lost.
You can never totally plan for every contingency, but you can insure yourself. I know many developers that take hard copies of their code (meaning on removable media) home just for this reason. I have seen sys admins do the same because they didn't trust their DR stratagy.
This was avoidable. This isn't even about disaster recovery. It is about business continuity.
You can't afford not to protect your data.
I worked for a computer bureaux in the 80's. We upgraded the operating system - very cool, the new release allowed larger files. We didn't, unfortunately, upgrade the backup utility to handle these larger files. Months go by - then there's a problem - whoops backups are useless - Luckily there's a physical audit trail so we we can pay for very large data entry exercise to get our client's data back.
A couple of years later, I am in the pub with some mates and John turns up. I ask him how he's managed to finish work and get to the pub so early. "I did a fast backup" he said. I was interested so I asked him to explain. "Oh, it's easy, get the target tapes from the rack, rub out the old date, write the new date, put them back into rack and go to the pub"
Worked for a large software shop in the 90's. I am part of a decent sized Oracle development (circa 50 devs). Ops decides that Oracles backup routines are too slow and 'optimize' them. Some weeks later - guess what - there's a problem and the backups are useless - No physical audit trail this time - the team has to redo all of there work - it was not good for the project budget, the team moral or the client
Wait, we have to care? I thought we were supposed to point and laugh...
Blank until
"Only wimps use tape backup: _real_ men just upload their important stuff ;)"
on ftp, and let the rest of the world mirror it
Linus Torvalds Jul 20 1996, 3:00 am
Anyone who hacks a flight-sim sight has no life and really needs to get laid.
Coming from a slashdotter that is pretty rough.
"I don't necessarily agree with everything I say." - Marshall McLuhan
This is a lesson every system administrator worth his or her salt learns over the long haul. You might back up dutifully, test restore, and have a well done system of ensuring backups are rotated correctly. Then you find out the tape drive you use is miscalibrated so only it can read your backup tapes, or you find the backup software you use on a daily basis is not in production, or the latest version has no support for the backlevel formats.
I have found that in a production environment, you really need multiple methods for backup if at all possible:
The first level is a dedicated backup server. This machine is locked down to the best of your abilities, and firewalled from the network, only allowing critical ports such as what the backup software uses, and perhaps ssh or RDP (if a Windows box). This machine copies everything from the other servers onto a large disk array, then to tape. The tapes are then cycled offsite via a service like Iron Mountain. Of course, the tapes are encrypted, and corporate officers get a copy of the master keys.
Why tapes? Because they can be set read only after they are dismounted, and no computer, no matter how infected can modify or delete the tape contents once this is done, outside of a reflash of the tape drive's BIOS. This is important because its not unheard of for someone to write a program that trashes backups over a time interval. Higher end tapes can be used as WORM media like DLT-ICE.
I can't emphasize enough about securing the backup server, both physically and network-wise. If this box gets compromised, all your data is available. On Windows machines, I recommend using some form of disk encryption (Bitlocker if the machine has a TPM, TrueCrypt, etc) so if the backup server or an array gets physically stolen, the data is of no use to a thief. This is in addition to the backup program's encryption.
After you have a central backup server installed, secured (security is paramount on this machine unless the backup program client can do encryption), and backups running, you focus on the other levels of backup.
The next level of backup is on the local servers. Most operating systems have a method of backing up the computer. If you can do this with a server, fire off a snapshot backup every month or so. Most OS backup methods don't have encryption, so this backup should go directly to a tape safe or secured container in the data center. Optionally, you can install backup software locally that can encrypt. I like using the backup/restore utility the OS gives for an image every quarter, then using more secure software more often, so the OS backups can be stored in a tape safe or physically secure container. This way, if the third party backup software ends up inoperable, there is still a method of getting a machine up somehow, or putting it in a virtual machine for recovery purposes.
Finally, after you have backup servers and a rotation, companies might consider offsite cloud backup services like Mozy. Mozy offers use of keyfiles so all data is stored encrypted (encrypted on the client end). Of course, making sure the encryption key is stored safely is paramount, and the cost of storing a large backup in Mozy's cloud may be prohibitive. However, if worse comes to worst and your site is completely knocked out, as well as the offsite backup site, it may be thing that keeps your business up.
Of course, scale this up or down as per your company's needs. A smaller business can get by using Mozy and a Windows Server 2008 box running Bitlocker, a network backup program with encryption such as Retrospect or Backup Exec, and using external drives every month to copy backup sets from the main ones to store offsite.
A larger business might see about a true backup fabric system sold by IBM (TSM), EMC (Networker), or Microsoft's solution.
The key is to not just have some built in redundancy so if one backup method is not usable, you have another, even if the backups are older, but to be able to do this in a manner that doesn't add too much time and equipment expense.
I kept them in my other pocket.
A public viewing will be available at:
http://web.archive.org/web/20080116064652/http://www.avsim.com/
No date has been set for the funeral.
It's beyond me how the blame is always shifted on the victim of an attack. There's a line between equitible share of responsibility and blame, and it's nowhere as fine as you think it is.
Remember kids if it isn't backed up to an off-line copy then it isn't backed up.
The [a href="http://16systems.com/zero.php"]Great Zero Challenge[/url] says otherwise. They're simply asking for the filename of one of the files on a drive that has been wiped once with zeros. Despite offering the challenge for over a year and actively speaking to data recovery companies, no one has taken them up on the offer.
Bleh!
I am a computer forensics expert. I search for deleted data for a living, and I testify in court as to what can be done.
Unfortunately you are wrong about recovering data that has been overwritten by using magnetic magic.
That is an urban legend that has been disproven. Maybe 20 years ago using low density MFM drives it was theoretically possible, but now it is not. Maybe the NSA has some tech they reversed engineered from an Area-51 UFO to do this, but I've never seen or heard of it.
Even Gutmann has recanted his 38 wipes recommendation.
Now don't mistake overwritten data for deleted data. When data is deleted it is NOT overwritten. When a hard drive is re-formatted almost nothing is over-written. When a file is overwritten with zeros or random bytes there are probably 10 more copies of that file and previous versions of that file floating around in unallocated sectors, swap space, file slack, hibernation files, etc.
But what IS overwritten is gone.
- For the complete works of Shakespeare: cat
And I'm going to respectfully disagree with you too.
For most small businesses cash flow is critical. If you don't have a record of who paid you in the last month then you can't invoice the rest and you are dead. Your repeat customers will spot duplicate invoices and probably just block payments until it's all sorted out. The attack that you are defending against is either a fire which destroys your office or a burglary which steals all your computers in the night, including the backup box, taking the backups just because they happen to be there.
You need off site backups on a different, non internet-connected medium no less often than once a week. That is the maximum time for which it is acceptable (we are talking about disaster recovery here; "acceptable" has a different meaning from normal) to re-invoice people who have already paid you. Even so, most such incidents destroy small businesses completely just because they don't manage to get people back working in time. This just gives you a fighting chance if you have a nice and understanding bank manager and do a little more disaster planning. It is astounding how much difference spending four hours just thinking about it can make (e.g. you know the number of the temporary office providers, you know which people in your office can work from home and you realise everybody in your company should have a mobile phone, especially the receptionist).
And finally; if you haven't tried restoring from it, it isn't a backup.
=~ s,(.*),<sarcasm>$1</sarcasm>,g if any_point_you_wish();
I've never seen *any* evidence or heard of *any* occasion that such a recovery, even from a only-once-zeroed drive was done.
Now the point is, one could say "of cooourse not, guys that can do this won't do it for peanuts, besides they're secres service" etc etc. But the point is: even if it's secret service and really expensive, at leas *some* news about it should have hit the public -- after all, this myth has been around for several years (a decade?) now.
I'd still even like to hear from a success story. Or even find a company that advertizes "We can (partly?) recover your zero'ed data -- it's going to cost a fortune, an arm and a leg, but we can." Haven't seen that one either yet. Not a commercial, not an offer, nothing... besides legends.
So they had no real backup strategy....but what happened to them REALLY REALLY sucks. It really irks me seeing so many comments saying these "retards" had it coming to them.
Listen folks....we're talking about a couple of guys who spent their free time creating a website. They're not making any real money out of this (in fact, they all have regular day jobs).
They've been advertising for a Tech Manager (non-paid) for quite a quite so time now. They did get one recently...but it turns out the guy harvested the emails from the systems and sent out a bunch of spam. He has since been fired.Even though the avsim folks aren't saying it was him who hacked and destroyed their site, it's quite hard not to think it was him.
It's been quite a blow to the flightsim community and I have noticed a lot of IT folks are offering help.....I just haven't seen a single one on this thread.
A dedicated backup box can be much more hardened
What you've described is only marginally better than what these people did. A second server playing backup device, even if it's "much more hardened", whatever that means, is still an extremely lousy and ineffective backup. If lightening hits your building or arson or theft, your "it's hardened"! backup server is just as toasted as the primary. Backups MUST be to removable media that's kept off site and inactive.
Otherwise you've done practically the same thing for data "backup" as the RAID does via disks, except with two servers.
Tedious and expensive, but several people made a good living out of doing it (one guy I knew did it as a hobby and made over UKP100K one year.) However, as bits get smaller, servos get more accurate, and tracks get denser, the modus operandi just ceases to exist any more.
Mind you, for security reasons I always dismantle old drives and bend the disks in half using a lump hammer. That, and the fact that hard drive magnets are just incredibly useful if you have a steel hulled boat and want convenient attachments for e.g. cable ties. They are powerful and very short range, and usually nickel plated. To buy a pair of equally useful magnets from hardware stores costs nearly as much as a drive.
From scarped cliff or quarried stone she cries "A thousand types are gone, I care for nothing, no not one."
"And finally; if you haven't tried restoring from it, it isn't a backup."
That, my friend, need to be carved on a marble plate and hung over the door to every datacenter.
To any sysadmins and DBAs...
Make sure you have offsite backups
Any person in the IT community who was alive to remember the events of 9/11 should have learned a valuable IT lesson from that event.
Repeat after me. I will not store my "offsite" backups in the other tower.
Why would you need to take that risk? It's standard business practice to just make a tape and ship it off site. The cost of shipping the tapes isn't worth the risk of leaving the backups on an internet connected box in my opinion.
If it's on the internet, then it is exposed.
That's not an archive, pal, that's evidence.
You wanna destroy that stuff, the sooner the better.
You are welcome on my lawn.
Note to self: never, ever ask you about your hobbies.
"It's too bad that stupidity isn't painful." - Anton LaVey