Slashdot Mirror
Hacker Destroys Avsim.com, Along With Its Backups
el americano writes "Flight Simulator community website Avsim has experienced a total data loss after both of their online servers were hacked. The site's founder, Tom Allensworth, explained why 13 years of community developed terrains, skins, and mods will not be restored from backups: 'Some have asked whether or not we had back ups. Yes, we dutifully backed up our servers every day. Unfortunately, we backed up the servers between our two servers. The hacker took out both servers, destroying our ability to use one or the other back up to remedy the situation.'"
To any sysadmins and DBAs...
Make sure you have offsite backups
more than one backup. always! especially if two servers are running the same software, who says they won't both fail at the same time?
Reserved for people who don't do archival backups, don't secure their systems, and then try to blame their ineptitude on hackers.
Do backups.
Do security.
Do restore from your backups to test them.
Do not blame others when it's shown you failed steps 1-3.
"Only wimps use tape backup: _real_ men just upload their important stuff ;)"
on ftp, and let the rest of the world mirror it
Linus Torvalds Jul 20 1996, 3:00 am
And for those who don't like to pay $10000 for backup software, there's Bacula. Couple that with an LTO-4 drive (~1000) and LTO-4 tapes (800GB uncompressed, ~60/piece) and you're set. Rsync.net is a decent, cheap online provider for those gaps when you haven't rotated tapes.
Bacula is pretty sweet because it lets you backup to disk volumes and then you can schedule a roll to tape. So you can just back everything up incrementally to a disk volume and then copy those backups to tape, and then run rsync on the disk volumes to have an offsite, online backup. When recovering, you ask to recover from whatever's available. If you keep enough disk storage around (and there's really no reason not to) you can recover to any date in the past. In the event of a disaster your tapes come into play.
Now with drives so cheap the temptation is to buy a external hard drive and use that. But tapes have a long history, guaranteed backwards compatibility (planned anyway, LTO drives have to R/W the previous generation and Read 2 generations back), last longer than moving drives, are simpler, lighter, more robust and more portable. Not that I wouldn't keep a external around to dump desktops but tape is the DR standard.
Cool! Amazing Toys.
The admins' claim that they were backed up is nothing short of an outright lie. A dependency on rsync or any other mirroring technique alone is just plain negligent, when both servers are exposed to the world at large. As a bad analogy, it's like allowing someone to light two fuses with the same match.
The only way to do backups properly is to have a complete set, offline, in a separate location.
Sheesh. When will people learn?
People always dis tapes. However, enterprise grade tapes are designed from the ground up, chemically, physically, electrically, and mechanically for long term data storage. I say enterprise grade because there is a difference between a tape format like DLT and LTO which was designed from the ground up as a high end data storage medium versus a tape format like the ones which were adapted from video or audiotapes where longetivity takes a back seat to economy.
I drop a tape, check its spindle, dust it off, its fine. I drop a hard disk, and there is a good chance that all the data on it is history.
As for Bacula, I am always wary of it. Does it just back up files, or does it back up vital components that are not file related, such as the Registry, ACLs, ADFs, and other things?
> I'd like to see you recover something that has been overwritten once.
You can't do it at home, but professional data recovery service can. Usually you can guess the previous data by precisely measuring the magnetic levels. The old values will influence the resulting intensity. Roughly (I'm not expert!) works like this:
was -- now -- result
0 -- 1 -- 0.9
1 -- 0 -- 0.1
1 -- 1 -- 1.1
0 -- 0 -- 0
That is why you should have MULTIPLE overwrites with RANDOM data.
- tested
- offline
- off-site
- several times
anything else is "high-availability", not "backup".
The Cloud - because you don't care if your apps and data are up in the air.
So they had no real backup strategy....but what happened to them REALLY REALLY sucks. It really irks me seeing so many comments saying these "retards" had it coming to them.
Listen folks....we're talking about a couple of guys who spent their free time creating a website. They're not making any real money out of this (in fact, they all have regular day jobs).
They've been advertising for a Tech Manager (non-paid) for quite a quite so time now. They did get one recently...but it turns out the guy harvested the emails from the systems and sent out a bunch of spam. He has since been fired.Even though the avsim folks aren't saying it was him who hacked and destroyed their site, it's quite hard not to think it was him.
It's been quite a blow to the flightsim community and I have noticed a lot of IT folks are offering help.....I just haven't seen a single one on this thread.
Tedious and expensive, but several people made a good living out of doing it (one guy I knew did it as a hobby and made over UKP100K one year.) However, as bits get smaller, servos get more accurate, and tracks get denser, the modus operandi just ceases to exist any more.
Mind you, for security reasons I always dismantle old drives and bend the disks in half using a lump hammer. That, and the fact that hard drive magnets are just incredibly useful if you have a steel hulled boat and want convenient attachments for e.g. cable ties. They are powerful and very short range, and usually nickel plated. To buy a pair of equally useful magnets from hardware stores costs nearly as much as a drive.
From scarped cliff or quarried stone she cries "A thousand types are gone, I care for nothing, no not one."
They should be kept on a different part of the electricity grid, preferably in a differnt postcode.
It all depends on what kind of disasters you want your data to survive. If you want it to survive nuclear war, you need off-shore backup. Preferably in a neutral country that won't get involved in the war.
If you want your data to survive a Vogon constructor fleet, use off-planet backup. Recovering it from the brain of a single surviving human (if any) is going to be costly and painful.
It's too late. That battle is over and the word is lost.
Just like Kleenex (the company) had its trademark stolen from it by falling into common usage, so did the word "hacker" lose its original meaning.
No, its not. Login/Password required. And Lame explanations why this should be necessary:
http://web.archive.org/web/20080116064652/http://www.avsim.com/
So the content not only got lost because of a stupid backup-strategy, but because of an even dumber login-required-strategy.
Linus said it: "Only wimps use tape backup: _real_ men just upload their important stuff on ftp, and let the rest of the world mirror it ;)" And thats precisly what avsim should have done.
"The more prohibitions there are, The poorer the people will be" -- Lao Tse
I didn't say you had to ship off hourly tapes. What hat did you pull that out of? You can use a mirror for minor recovery. We're talking about DR here, not a simple restore of an hourly type data request. The entire site for these folks is gone, not some data set for a transaction 3 hours ago, but everything.
As to tapes getting lost in transit, that happens very rarely given the tracking techniques in use by folks like FedEx and UPS. Even so, you wouldn't have only a single set of tapes with all of your data on it, you would have an established rotation of data. Every company I have worked for uses this method. Some used daily, some weekly, some monthly, etc, but all shipped tapes off site at regular routines and cycled them out yearly, or every 7 years depending on the type of data and retention requirements.
"The main fault here was that they had fail-over and called it backups."
Right.
"There is no one dogmatic way to look at backups. If you think there is, good luck finding a job in 10 years when conditions have changed."
Wrong. Conditions have not changed in the last 35 years and I don't see them changing on the foreseable future. Technical conditions and abilities will change, true, but the essence of the work that has to be achieved won't change the same a mathematical theorem doesn't change.
What a backup strategy is (short version):
* A means to recover from a failure.
It's obvious Tom Allensworth's strategy is a failure and it was obvious it was a failure from the very beginnig (it has been a hacker, but what if it were a virus or a worm, or a human failure deleting some critical files and then the deletion being replicated? Same result).
What makes a minimal backup strategy (any less than this and your "solution" is not entitled to be called "backup strategy"):
* There has to be no less than two complete data sets non connected with the systems being protected.
* There has to be no less than one complete data set off-sited from were the systems being protected "live in".
* There has to be no less than one current copy of the documentation needed to redeploy from barebones the protected systems off-sited from the facilities were the systems being protected "live in".
* At the very least two people -the backup responsible and her direct superior, have to know where the above mentioned documentation lives and they must have the ability to recover it.
Some side notes:
* The last two points are not needed on a lone star-driven system, only on company-style ones. If there's only one person which will benefit from the data (i.e.: your personal data or a single-person bussiness') is good enough if only you know how to recover the data -it can even be only anotated "on you head" and not in paper, although you still would be better if in paper: memory fails with time.
* The above point-set is not absolutly "failure-proof" and some common sense should be applied (if your system is likely to be attacked, you'd better have more datasets splitted over longer time ranges; if the backup admin and her superior tend to go together there's the risk you lose them both at a time, and so your ability to recover out of -now unknown to exist, documentation, etc.) but they are the bare minimum.
* Last but not least, backups have exactly ZERO value. Recovering from backups when need arises is the valuable part, so test your recovery procedures, once and again and again. And let it do the test your less knowledgeable/capable/valuable people: maybe when the need arises that will be all you have.