Slashdot Mirror
Hacker Destroys Avsim.com, Along With Its Backups
el americano writes "Flight Simulator community website Avsim has experienced a total data loss after both of their online servers were hacked. The site's founder, Tom Allensworth, explained why 13 years of community developed terrains, skins, and mods will not be restored from backups: 'Some have asked whether or not we had back ups. Yes, we dutifully backed up our servers every day. Unfortunately, we backed up the servers between our two servers. The hacker took out both servers, destroying our ability to use one or the other back up to remedy the situation.'"
I realize that from quite a few people's perspectives, storing their backups in a separate building constitutes off site storage. I'd almost buy that strategy. Not in the same environment, network, city etc.
These guys were stupid.
The day after 9/11 I was in an elevator, and caught a snippet of conversation between 2 people that had business interests with a firm that was in the WTC. The comment I heard was 'their backups were in the other building'. Another company lost.
You can never totally plan for every contingency, but you can insure yourself. I know many developers that take hard copies of their code (meaning on removable media) home just for this reason. I have seen sys admins do the same because they didn't trust their DR stratagy.
This was avoidable. This isn't even about disaster recovery. It is about business continuity.
You can't afford not to protect your data.
I worked for a computer bureaux in the 80's. We upgraded the operating system - very cool, the new release allowed larger files. We didn't, unfortunately, upgrade the backup utility to handle these larger files. Months go by - then there's a problem - whoops backups are useless - Luckily there's a physical audit trail so we we can pay for very large data entry exercise to get our client's data back.
A couple of years later, I am in the pub with some mates and John turns up. I ask him how he's managed to finish work and get to the pub so early. "I did a fast backup" he said. I was interested so I asked him to explain. "Oh, it's easy, get the target tapes from the rack, rub out the old date, write the new date, put them back into rack and go to the pub"
Worked for a large software shop in the 90's. I am part of a decent sized Oracle development (circa 50 devs). Ops decides that Oracles backup routines are too slow and 'optimize' them. Some weeks later - guess what - there's a problem and the backups are useless - No physical audit trail this time - the team has to redo all of there work - it was not good for the project budget, the team moral or the client
Rootkits nowadays come with disk wiping utility.
An Education is the Font of All Liberty
The publicity value of being the one successfully recovering that data is much higher than $500. People who say they could do it but don't because the money's not enough are full of shit.
didn't work for me, that where I uploaded my mods and paintjobs for my flightsim aircraft... I'm now going to see if I've still got my own copies of the stuff I uploaded and put it back up.
Donald 'Duck' Dunn: We had a band powerful enough to turn goat piss into gasoline.
Unless you have overwritten the area on the physical disk that contained the data, multiple times, the data can still be recovered.
The DoD spec is written as it is for a reason. Given a drive with confidential data on it, an unauthorized person attempting to access the drive does not need to get everything back to pristine condition. Even recovering a small part of the total data set can cause incalculable damage if it's the right small part. The value of sites like Avsim are in the whole rather than the sum of the parts.
I've recovered data off of formatted HDD's, off of corrupted file systems, off of compact flash cards and other media (Really useful if you want to keep those photo's that someone thought was deleted, be aware of this people).
There's a large dependency on what you're trying to recover off of. DOS/NTFS are fairly easy to do recovers from. The first character of the filename is zilched out and the rest of the data to find the file is left intact. UNIX/Linux filesystems are a bear. Once you hit "rm", you've lost the ref to your inode. Putting Humpty Dumpty together again at that point becomes nearly impossible because the record which shows where all the pieces are is lost to you. If you have known text from the file, and a good knowledge of how the filesystem works, you should be able to backtrack. Otherwise? God help you.
There's also an issue of how the data is stored. A single-drive system is fairly straightforward. 2 drives are harder. Once you get into a SAN/NAS where data is spread over multiple drives, recovery of even a single file with known text becomes tricky. Multiple files? Unknown data? The only hope I would see at that point is to put a large segment of the Slashdot community on the problem and tell them a large trove of high-res pictures of Natalie Portman completely nekkid are stored within.
Assuming we are talking about a disk that has been entirely written with zero's or random data, eg a deliberate attempt to render the disk unreadable - citation needed (or are you just repeating something you heard from someone who heard it from someone else?)
The 'black boxes' are designed to ensure the survival of the internal medium, so it's no surprise that the data is recoverable (don't they use analogue tape on a loop? or is that just for the voice recorder? or is my knowledge way out of data :)
When inter track spacings were wider and density in other dimensions was lower (20 years ago?) it was possible to recover data after a complete write with zero's, but not now.
While Wikipedia isn't the definitive answer on anything, it clearly states in several places that a single pass of the entire disk is enough to erase the disk with no chance of recovery.
Sounds awfully like an urban legend. Are the illuminati involved somehow? :p
Oh, he gets my sympathies. I've had cases where I was specifically told "that is a scratch server: do not back it up, no one is supposed to keep real data on it". And when it crashed, my employers were very fortunate indeed that I'd completely ignored this and quietly been backing it up with my reserve, emergency tape drive, partly to make sure it kept working, partly to test out new backup tools, and partly because I knew staff would ignore this and use the big lump of spare storage for convenient archival space. My employer was actually angry at me for doing so, but the QA department was very, very, very grateful indeed.
The lesson is more subtle than some of us might realize, though. Never rely on a _single_ method of backup or data storage, because any factor that ruins that backup can ruin all copies of it. This is true for backup systems that use proprietary format, or a failed tape drive that's been screwing up backups for the last year (which I've seen happen with old mag-tape media). And I _love_ online backups: you can make the data accessible via NFS or CIFS or other file-sharing to people to recover the files they just accidentally deleted, without having to invest in a very, very expensive NetApp or similar file server. But oh, dear, I've also seen what happens when someone screws up the backup tools and deletes all the copies at the same time.
At least one. I've been doing this ever since one of my colleagues, who had been working on a book for a year and a half, had his house burgled and the thief took his computer AND his backup system AND the lockbox in which he kept backup tapes (it was a long time ago). He thought he was doing everything right.
I had to help him scan several hundred pages from several sets of loose pages and chapters from a marked-up stale manuscript, and this was back in a day when scanners and OCR weren't all that great.
I saw him suffer so badly, and was so shaken in his faith in technology (he's an English teacher, not really a technical adept) that we devised a simple system of rotating tapes (later external hard drives, now 8gig flash drives) that we used to keep for each other (he's moved away now).
If I'm working on something that I absolutely cannot lose, I'll occasionally put it on a flash drive and give it to my wife. That's my safest keeping. I don't ask her where she keeps it, but sometimes when I get it back it has a faint aroma of the Pacific Ocean.
You are welcome on my lawn.
Surely all the people who've downloaded the downloadable content over the years can all band together and restore a large proportion of it?
I also thought: developers and user must have downloaded/synced great deal of source (binaries for sure :-).
So why not scramble it back together? As for the web site, perhaps pages are cached somewhere on the web, as some people already pointed out.
I've visited their site about a month or so ago, and they did really impressive work, they should try to restore it.
Obviously you've never had to back up about 8TB of data, with about 200GB of incremental changes weekly. We manufacture products with custom artwork, so we go through a lot of larger art files every week. You have any idea how much fun it would be to split up that much data across LTO-3 tapes? And then do incrementals? And then test it regularly? Our stateside IT staff that would be responsible for the backups consists of one other person aside from myself. We don't have the time to maintain a tape-based system. We do a combination of offsite, online backups...some to our facilities in the Philippines in China for both backup and operating purposes, and some to our 'local' IPSs for purely backup purposes. We do keep an additional copy our 40GB SQL database for our ERP system backed up weekly to a portable hard drive that we switch with one in our safety deposit box each week.
While tapes have been 'standard business practice' for years, the technology is lagging behind and is becoming inadequeate, especially for large businesses (you think Google, Microsoft, Amazon, etc. back everything up on tapes?), but also smaller businesses, too.
As our way of thanking you for your positive contributions to Slashdot, you are eligible to disable Slashdot 2.0.