Slashdot Mirror
Hacker Destroys Avsim.com, Along With Its Backups
el americano writes "Flight Simulator community website Avsim has experienced a total data loss after both of their online servers were hacked. The site's founder, Tom Allensworth, explained why 13 years of community developed terrains, skins, and mods will not be restored from backups: 'Some have asked whether or not we had back ups. Yes, we dutifully backed up our servers every day. Unfortunately, we backed up the servers between our two servers. The hacker took out both servers, destroying our ability to use one or the other back up to remedy the situation.'"
Owned.
To any sysadmins and DBAs...
Make sure you have offsite backups
more than one backup. always! especially if two servers are running the same software, who says they won't both fail at the same time?
Off. Site. Backups. Textbook example of why you need to secure your backup data in a secure, non-networked location.
Reserved for people who don't do archival backups, don't secure their systems, and then try to blame their ineptitude on hackers.
Do backups.
Do security.
Do restore from your backups to test them.
Do not blame others when it's shown you failed steps 1-3.
They say they had backups, and put them on the Internet where any hacker could get to them, under the same security the originals were stored under. If that's all they cared about their data, I don't see why the Slashdot community should care any more than they did.
I'm an American. I love this country and the freedoms that we used to have.
You now will be escorted off-site.
I realize that from quite a few people's perspectives, storing their backups in a separate building constitutes off site storage. I'd almost buy that strategy. Not in the same environment, network, city etc.
These guys were stupid.
The day after 9/11 I was in an elevator, and caught a snippet of conversation between 2 people that had business interests with a firm that was in the WTC. The comment I heard was 'their backups were in the other building'. Another company lost.
You can never totally plan for every contingency, but you can insure yourself. I know many developers that take hard copies of their code (meaning on removable media) home just for this reason. I have seen sys admins do the same because they didn't trust their DR stratagy.
This was avoidable. This isn't even about disaster recovery. It is about business continuity.
You can't afford not to protect your data.
Repeat after me: mirroring is not a backup. Backups are physically removed from the machine and stored where they can't be altered until they're needed for a restore. If they aren't removed from the machine, well, as we've just seen that only ends in tears. Observe their pain and learn from it!
'Backed up between two servers'... that's not what a backup is.
I'm... astonished at the level of incompetence here. A site with 13 years of work like this, and they didn't bother to backup anything at all?
And now they're trying to handwave it away with 'oh uh, uh really folks, seriously, were really did have backups haha, between servers olol'.
I don't think 'olol' is going to impress anyone whos work was just wiped out by their incompetence.
"I hope the same administrator will never again make the same mistake with backups."
He won't for this company, that is for sure.
It could be worse, it could be Monday.
As the subject says. "Online" backups and replication are simply tools to try and minimize downtime. They are NOT a backup solution. They never were and never should be touted as one, just as this example shows. The only good backup is one that occurs frequently, is verified that it worked, and is stored in a secure location such as a fire-proof safe, and even better in two different fire-proof safes in two different locations, preferably more than 100 miles apart.
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
Whoever did this must have willfully wanted to destroy the website and its content. Deleting data in this manner is far beyond vandalism or criminal mischief.
I hope the perps get served by a judge who recognizes just how severely malicious this was, and that enough of the people who used the site can provide the files back to the owners and the community.
Maybe future historians will consider this a dark age, whose intellectual production was lost.
Please don't say our treasured facebook, twitter, slashdot posts, wikipedia revision wars and v1agra spam may not be preserved for posterity.
I'm not yet convinced that information that today exists only on the internet is really meant for eternity :)
I worked for a computer bureaux in the 80's. We upgraded the operating system - very cool, the new release allowed larger files. We didn't, unfortunately, upgrade the backup utility to handle these larger files. Months go by - then there's a problem - whoops backups are useless - Luckily there's a physical audit trail so we we can pay for very large data entry exercise to get our client's data back.
A couple of years later, I am in the pub with some mates and John turns up. I ask him how he's managed to finish work and get to the pub so early. "I did a fast backup" he said. I was interested so I asked him to explain. "Oh, it's easy, get the target tapes from the rack, rub out the old date, write the new date, put them back into rack and go to the pub"
Worked for a large software shop in the 90's. I am part of a decent sized Oracle development (circa 50 devs). Ops decides that Oracles backup routines are too slow and 'optimize' them. Some weeks later - guess what - there's a problem and the backups are useless - No physical audit trail this time - the team has to redo all of there work - it was not good for the project budget, the team moral or the client
...the thieves and vandals who steal data and wreck servers.
THIEVES AND VANDALS.
Not "hackers".
What was done was not hacking. It was vandalism. Plain and simple.
Hackers create. Vandals destroy. Thieves steal.
I'm surprised that this needs to be explained to the Slashdot community.
Guaranteed! This comment 100% Anthrax free!
"Only wimps use tape backup: _real_ men just upload their important stuff ;)"
on ftp, and let the rest of the world mirror it
Linus Torvalds Jul 20 1996, 3:00 am
Anyone who hacks a flight-sim sight has no life and really needs to get laid.
Coming from a slashdotter that is pretty rough.
"I don't necessarily agree with everything I say." - Marshall McLuhan
This is a lesson every system administrator worth his or her salt learns over the long haul. You might back up dutifully, test restore, and have a well done system of ensuring backups are rotated correctly. Then you find out the tape drive you use is miscalibrated so only it can read your backup tapes, or you find the backup software you use on a daily basis is not in production, or the latest version has no support for the backlevel formats.
I have found that in a production environment, you really need multiple methods for backup if at all possible:
The first level is a dedicated backup server. This machine is locked down to the best of your abilities, and firewalled from the network, only allowing critical ports such as what the backup software uses, and perhaps ssh or RDP (if a Windows box). This machine copies everything from the other servers onto a large disk array, then to tape. The tapes are then cycled offsite via a service like Iron Mountain. Of course, the tapes are encrypted, and corporate officers get a copy of the master keys.
Why tapes? Because they can be set read only after they are dismounted, and no computer, no matter how infected can modify or delete the tape contents once this is done, outside of a reflash of the tape drive's BIOS. This is important because its not unheard of for someone to write a program that trashes backups over a time interval. Higher end tapes can be used as WORM media like DLT-ICE.
I can't emphasize enough about securing the backup server, both physically and network-wise. If this box gets compromised, all your data is available. On Windows machines, I recommend using some form of disk encryption (Bitlocker if the machine has a TPM, TrueCrypt, etc) so if the backup server or an array gets physically stolen, the data is of no use to a thief. This is in addition to the backup program's encryption.
After you have a central backup server installed, secured (security is paramount on this machine unless the backup program client can do encryption), and backups running, you focus on the other levels of backup.
The next level of backup is on the local servers. Most operating systems have a method of backing up the computer. If you can do this with a server, fire off a snapshot backup every month or so. Most OS backup methods don't have encryption, so this backup should go directly to a tape safe or secured container in the data center. Optionally, you can install backup software locally that can encrypt. I like using the backup/restore utility the OS gives for an image every quarter, then using more secure software more often, so the OS backups can be stored in a tape safe or physically secure container. This way, if the third party backup software ends up inoperable, there is still a method of getting a machine up somehow, or putting it in a virtual machine for recovery purposes.
Finally, after you have backup servers and a rotation, companies might consider offsite cloud backup services like Mozy. Mozy offers use of keyfiles so all data is stored encrypted (encrypted on the client end). Of course, making sure the encryption key is stored safely is paramount, and the cost of storing a large backup in Mozy's cloud may be prohibitive. However, if worse comes to worst and your site is completely knocked out, as well as the offsite backup site, it may be thing that keeps your business up.
Of course, scale this up or down as per your company's needs. A smaller business can get by using Mozy and a Windows Server 2008 box running Bitlocker, a network backup program with encryption such as Retrospect or Backup Exec, and using external drives every month to copy backup sets from the main ones to store offsite.
A larger business might see about a true backup fabric system sold by IBM (TSM), EMC (Networker), or Microsoft's solution.
The key is to not just have some built in redundancy so if one backup method is not usable, you have another, even if the backups are older, but to be able to do this in a manner that doesn't add too much time and equipment expense.
Only goatse is eternal. The rest is being used to seed a randomness generator somewhere.
Futurist Traditionalism
I kept them in my other pocket.
A public viewing will be available at:
http://web.archive.org/web/20080116064652/http://www.avsim.com/
No date has been set for the funeral.
The admins' claim that they were backed up is nothing short of an outright lie. A dependency on rsync or any other mirroring technique alone is just plain negligent, when both servers are exposed to the world at large. As a bad analogy, it's like allowing someone to light two fuses with the same match.
The only way to do backups properly is to have a complete set, offline, in a separate location.
Sheesh. When will people learn?
Wikipedia revision wars will be a GOLDMINE for future archeologist.
Think about just how much they reveal about a certain topic.
HI O WISE PRINCE. WHT TOOK U SO DAM LONG?
The publicity value of being the one successfully recovering that data is much higher than $500. People who say they could do it but don't because the money's not enough are full of shit.
nonsense...
completely inaccurate guestimation, but probably only about 1% of anything carved in stone, is still decipherable or even exists, same with scrolls, otherwise we'd be littered with 2000 year old shopping lists, love letters, etc, how many notebooks (the paper kind) have you gone through during school, as journals, boredom... still have them all?
Hell, we probably only have about 1% of the stuff that was written down 100 years ago, probably only about 3% of the buildings, 0.3% of the cars, 2% of the paintings...etc...etc... most of the ancient books we have, are copies of copies of copies, and we can do that with magnitudes of efficiency now, not to mention recovery, hard drive gets erased, it's easier to get the data back than a scroll that's been erased, or a stone.
If even 0.1% of what we have on the internet right now exists in 500 years, it'll still probably be more than everything we have in stone, scrolls, and print right now...
With the various sorts of "Library of Congress" out there, if you had the chance to peruse and take/read whatever you wanted, you'd probably only find 0.5% of it interesting anyways, much like what's on the internet.
- tested
- offline
- off-site
- several times
anything else is "high-availability", not "backup".
The Cloud - because you don't care if your apps and data are up in the air.
The [a href="http://16systems.com/zero.php"]Great Zero Challenge[/url] says otherwise. They're simply asking for the filename of one of the files on a drive that has been wiped once with zeros. Despite offering the challenge for over a year and actively speaking to data recovery companies, no one has taken them up on the offer.
Bleh!
Markup Fail! Great Zero Challenge
Bleh!
I am a computer forensics expert. I search for deleted data for a living, and I testify in court as to what can be done.
Unfortunately you are wrong about recovering data that has been overwritten by using magnetic magic.
That is an urban legend that has been disproven. Maybe 20 years ago using low density MFM drives it was theoretically possible, but now it is not. Maybe the NSA has some tech they reversed engineered from an Area-51 UFO to do this, but I've never seen or heard of it.
Even Gutmann has recanted his 38 wipes recommendation.
Now don't mistake overwritten data for deleted data. When data is deleted it is NOT overwritten. When a hard drive is re-formatted almost nothing is over-written. When a file is overwritten with zeros or random bytes there are probably 10 more copies of that file and previous versions of that file floating around in unallocated sectors, swap space, file slack, hibernation files, etc.
But what IS overwritten is gone.
- For the complete works of Shakespeare: cat
Wikipedia revision wars will be a GOLDMINE for future archeologist.
Think about just how much they reveal about a certain topic.
Such as the difference of opinion about the color variations of the carrot !
I've never seen *any* evidence or heard of *any* occasion that such a recovery, even from a only-once-zeroed drive was done.
Now the point is, one could say "of cooourse not, guys that can do this won't do it for peanuts, besides they're secres service" etc etc. But the point is: even if it's secret service and really expensive, at leas *some* news about it should have hit the public -- after all, this myth has been around for several years (a decade?) now.
I'd still even like to hear from a success story. Or even find a company that advertizes "We can (partly?) recover your zero'ed data -- it's going to cost a fortune, an arm and a leg, but we can." Haven't seen that one either yet. Not a commercial, not an offer, nothing... besides legends.
So they had no real backup strategy....but what happened to them REALLY REALLY sucks. It really irks me seeing so many comments saying these "retards" had it coming to them.
Listen folks....we're talking about a couple of guys who spent their free time creating a website. They're not making any real money out of this (in fact, they all have regular day jobs).
They've been advertising for a Tech Manager (non-paid) for quite a quite so time now. They did get one recently...but it turns out the guy harvested the emails from the systems and sent out a bunch of spam. He has since been fired.Even though the avsim folks aren't saying it was him who hacked and destroyed their site, it's quite hard not to think it was him.
It's been quite a blow to the flightsim community and I have noticed a lot of IT folks are offering help.....I just haven't seen a single one on this thread.
Tedious and expensive, but several people made a good living out of doing it (one guy I knew did it as a hobby and made over UKP100K one year.) However, as bits get smaller, servos get more accurate, and tracks get denser, the modus operandi just ceases to exist any more.
Mind you, for security reasons I always dismantle old drives and bend the disks in half using a lump hammer. That, and the fact that hard drive magnets are just incredibly useful if you have a steel hulled boat and want convenient attachments for e.g. cable ties. They are powerful and very short range, and usually nickel plated. To buy a pair of equally useful magnets from hardware stores costs nearly as much as a drive.
From scarped cliff or quarried stone she cries "A thousand types are gone, I care for nothing, no not one."
To any sysadmins and DBAs...
Make sure you have offsite backups
Any person in the IT community who was alive to remember the events of 9/11 should have learned a valuable IT lesson from that event.
Repeat after me. I will not store my "offsite" backups in the other tower.
Why would you need to take that risk? It's standard business practice to just make a tape and ship it off site. The cost of shipping the tapes isn't worth the risk of leaving the backups on an internet connected box in my opinion.
If it's on the internet, then it is exposed.