Slashdot Mirror
Hacker Destroys Avsim.com, Along With Its Backups
el americano writes "Flight Simulator community website Avsim has experienced a total data loss after both of their online servers were hacked. The site's founder, Tom Allensworth, explained why 13 years of community developed terrains, skins, and mods will not be restored from backups: 'Some have asked whether or not we had back ups. Yes, we dutifully backed up our servers every day. Unfortunately, we backed up the servers between our two servers. The hacker took out both servers, destroying our ability to use one or the other back up to remedy the situation.'"
Owned.
To any sysadmins and DBAs...
Make sure you have offsite backups
more than one backup. always! especially if two servers are running the same software, who says they won't both fail at the same time?
if it isn't verified
Off. Site. Backups. Textbook example of why you need to secure your backup data in a secure, non-networked location.
Reserved for people who don't do archival backups, don't secure their systems, and then try to blame their ineptitude on hackers.
Do backups.
Do security.
Do restore from your backups to test them.
Do not blame others when it's shown you failed steps 1-3.
They say they had backups, and put them on the Internet where any hacker could get to them, under the same security the originals were stored under. If that's all they cared about their data, I don't see why the Slashdot community should care any more than they did.
I'm an American. I love this country and the freedoms that we used to have.
You now will be escorted off-site.
I realize that from quite a few people's perspectives, storing their backups in a separate building constitutes off site storage. I'd almost buy that strategy. Not in the same environment, network, city etc.
These guys were stupid.
The day after 9/11 I was in an elevator, and caught a snippet of conversation between 2 people that had business interests with a firm that was in the WTC. The comment I heard was 'their backups were in the other building'. Another company lost.
You can never totally plan for every contingency, but you can insure yourself. I know many developers that take hard copies of their code (meaning on removable media) home just for this reason. I have seen sys admins do the same because they didn't trust their DR stratagy.
This was avoidable. This isn't even about disaster recovery. It is about business continuity.
You can't afford not to protect your data.
Repeat after me: mirroring is not a backup. Backups are physically removed from the machine and stored where they can't be altered until they're needed for a restore. If they aren't removed from the machine, well, as we've just seen that only ends in tears. Observe their pain and learn from it!
'Backed up between two servers'... that's not what a backup is.
I'm... astonished at the level of incompetence here. A site with 13 years of work like this, and they didn't bother to backup anything at all?
And now they're trying to handwave it away with 'oh uh, uh really folks, seriously, were really did have backups haha, between servers olol'.
I don't think 'olol' is going to impress anyone whos work was just wiped out by their incompetence.
"I hope the same administrator will never again make the same mistake with backups."
He won't for this company, that is for sure.
It could be worse, it could be Monday.
When invaded their identities system was lost too.
All they had was a back up copy that made it out.
After the war they could go in and find what was tampered with. ie who got a false identity.
Take your data home with you every night.
Domestic spying is now "Benign Information Gathering"
As the subject says. "Online" backups and replication are simply tools to try and minimize downtime. They are NOT a backup solution. They never were and never should be touted as one, just as this example shows. The only good backup is one that occurs frequently, is verified that it worked, and is stored in a secure location such as a fire-proof safe, and even better in two different fire-proof safes in two different locations, preferably more than 100 miles apart.
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
Whoever did this must have willfully wanted to destroy the website and its content. Deleting data in this manner is far beyond vandalism or criminal mischief.
I hope the perps get served by a judge who recognizes just how severely malicious this was, and that enough of the people who used the site can provide the files back to the owners and the community.
Maybe future historians will consider this a dark age, whose intellectual production was lost.
Please don't say our treasured facebook, twitter, slashdot posts, wikipedia revision wars and v1agra spam may not be preserved for posterity.
I'm not yet convinced that information that today exists only on the internet is really meant for eternity :)
I'm assuming he wasn't backed up, either.
I worked for a computer bureaux in the 80's. We upgraded the operating system - very cool, the new release allowed larger files. We didn't, unfortunately, upgrade the backup utility to handle these larger files. Months go by - then there's a problem - whoops backups are useless - Luckily there's a physical audit trail so we we can pay for very large data entry exercise to get our client's data back.
A couple of years later, I am in the pub with some mates and John turns up. I ask him how he's managed to finish work and get to the pub so early. "I did a fast backup" he said. I was interested so I asked him to explain. "Oh, it's easy, get the target tapes from the rack, rub out the old date, write the new date, put them back into rack and go to the pub"
Worked for a large software shop in the 90's. I am part of a decent sized Oracle development (circa 50 devs). Ops decides that Oracles backup routines are too slow and 'optimize' them. Some weeks later - guess what - there's a problem and the backups are useless - No physical audit trail this time - the team has to redo all of there work - it was not good for the project budget, the team moral or the client
...the thieves and vandals who steal data and wreck servers.
THIEVES AND VANDALS.
Not "hackers".
What was done was not hacking. It was vandalism. Plain and simple.
Hackers create. Vandals destroy. Thieves steal.
I'm surprised that this needs to be explained to the Slashdot community.
Guaranteed! This comment 100% Anthrax free!
"Only wimps use tape backup: _real_ men just upload their important stuff ;)"
on ftp, and let the rest of the world mirror it
Linus Torvalds Jul 20 1996, 3:00 am
Anyone who hacks a flight-sim sight has no life and really needs to get laid.
Coming from a slashdotter that is pretty rough.
"I don't necessarily agree with everything I say." - Marshall McLuhan
This is a lesson every system administrator worth his or her salt learns over the long haul. You might back up dutifully, test restore, and have a well done system of ensuring backups are rotated correctly. Then you find out the tape drive you use is miscalibrated so only it can read your backup tapes, or you find the backup software you use on a daily basis is not in production, or the latest version has no support for the backlevel formats.
I have found that in a production environment, you really need multiple methods for backup if at all possible:
The first level is a dedicated backup server. This machine is locked down to the best of your abilities, and firewalled from the network, only allowing critical ports such as what the backup software uses, and perhaps ssh or RDP (if a Windows box). This machine copies everything from the other servers onto a large disk array, then to tape. The tapes are then cycled offsite via a service like Iron Mountain. Of course, the tapes are encrypted, and corporate officers get a copy of the master keys.
Why tapes? Because they can be set read only after they are dismounted, and no computer, no matter how infected can modify or delete the tape contents once this is done, outside of a reflash of the tape drive's BIOS. This is important because its not unheard of for someone to write a program that trashes backups over a time interval. Higher end tapes can be used as WORM media like DLT-ICE.
I can't emphasize enough about securing the backup server, both physically and network-wise. If this box gets compromised, all your data is available. On Windows machines, I recommend using some form of disk encryption (Bitlocker if the machine has a TPM, TrueCrypt, etc) so if the backup server or an array gets physically stolen, the data is of no use to a thief. This is in addition to the backup program's encryption.
After you have a central backup server installed, secured (security is paramount on this machine unless the backup program client can do encryption), and backups running, you focus on the other levels of backup.
The next level of backup is on the local servers. Most operating systems have a method of backing up the computer. If you can do this with a server, fire off a snapshot backup every month or so. Most OS backup methods don't have encryption, so this backup should go directly to a tape safe or secured container in the data center. Optionally, you can install backup software locally that can encrypt. I like using the backup/restore utility the OS gives for an image every quarter, then using more secure software more often, so the OS backups can be stored in a tape safe or physically secure container. This way, if the third party backup software ends up inoperable, there is still a method of getting a machine up somehow, or putting it in a virtual machine for recovery purposes.
Finally, after you have backup servers and a rotation, companies might consider offsite cloud backup services like Mozy. Mozy offers use of keyfiles so all data is stored encrypted (encrypted on the client end). Of course, making sure the encryption key is stored safely is paramount, and the cost of storing a large backup in Mozy's cloud may be prohibitive. However, if worse comes to worst and your site is completely knocked out, as well as the offsite backup site, it may be thing that keeps your business up.
Of course, scale this up or down as per your company's needs. A smaller business can get by using Mozy and a Windows Server 2008 box running Bitlocker, a network backup program with encryption such as Retrospect or Backup Exec, and using external drives every month to copy backup sets from the main ones to store offsite.
A larger business might see about a true backup fabric system sold by IBM (TSM), EMC (Networker), or Microsoft's solution.
The key is to not just have some built in redundancy so if one backup method is not usable, you have another, even if the backups are older, but to be able to do this in a manner that doesn't add too much time and equipment expense.
Honestly, how many man-hours and equipment do you really want to commit to backup? Do you really think it's worthwhile to get a tape system and regularly move tapes off-site for some community mods? Anyone can envision a system that is far more secure than this, but paying for it is another thing.
If the mods were good quality and downloaded often, the community should be able to act as a backup of sorts.
Only goatse is eternal. The rest is being used to seed a randomness generator somewhere.
Futurist Traditionalism
Actually, he got regular backups at the Dollhouse. I'm not sure how he'll respond to being in Eliza Dushku's body...
Unless you have overwritten the area on the physical disk that contained the data, multiple times, the data can still be recovered.
How about once? With zeros.
http://16systems.com/zero.php
If you can retrieve you data from a drive after it has been dd'd with /dev/zero, you might be able to win this prize.
If you happen to be in the situation described, chances are you're fucked.
How we know is more important than what we know.
I kept them in my other pocket.
A public viewing will be available at:
http://web.archive.org/web/20080116064652/http://www.avsim.com/
No date has been set for the funeral.
The admins' claim that they were backed up is nothing short of an outright lie. A dependency on rsync or any other mirroring technique alone is just plain negligent, when both servers are exposed to the world at large. As a bad analogy, it's like allowing someone to light two fuses with the same match.
The only way to do backups properly is to have a complete set, offline, in a separate location.
Sheesh. When will people learn?
Wikipedia revision wars will be a GOLDMINE for future archeologist.
Think about just how much they reveal about a certain topic.
HI O WISE PRINCE. WHT TOOK U SO DAM LONG?
nonsense...
completely inaccurate guestimation, but probably only about 1% of anything carved in stone, is still decipherable or even exists, same with scrolls, otherwise we'd be littered with 2000 year old shopping lists, love letters, etc, how many notebooks (the paper kind) have you gone through during school, as journals, boredom... still have them all?
Hell, we probably only have about 1% of the stuff that was written down 100 years ago, probably only about 3% of the buildings, 0.3% of the cars, 2% of the paintings...etc...etc... most of the ancient books we have, are copies of copies of copies, and we can do that with magnitudes of efficiency now, not to mention recovery, hard drive gets erased, it's easier to get the data back than a scroll that's been erased, or a stone.
If even 0.1% of what we have on the internet right now exists in 500 years, it'll still probably be more than everything we have in stone, scrolls, and print right now...
With the various sorts of "Library of Congress" out there, if you had the chance to peruse and take/read whatever you wanted, you'd probably only find 0.5% of it interesting anyways, much like what's on the internet.
They had redundancy. Another online copy of data isn't a back, it is redundancy. A backup is a separate, offline copy.
For example if you have a RAID-10, you do NOT have a backup of your data. What you've got is redundancy. In the event you have a disk failure, you don't lose data and you also don't lose system functionality. That's actually the main reason for RAID (at least RAID other than 0). You don't want your system to have downtime. If you drop a disk you can use the system while the replacement comes in, rather than being SOL.
A backup is separate. It can be another harddrive, it can be DVDs, it can be tape, whatever. It is something you use to take data from the system, and move it offline.
Now why is the offline thing so important? Well this demonstrates one reason. A bigger one would be catastrophic hardware failure. What happens if your PSU goes nuts and pumps out 120 volts on the 12v lines? That kind of thing can burn out all your hardware, and thus anything you have internally. An external backup isn't affected, of course. Then there's things like fire, or flood and so on.
However the biggest would be your own screwup. What happens if you accidentally overwrite the data with garbage? What if you then trigger a backup sync, or it happens automatically before you realize your mistake? Well you are screwed now. You backup is now of useless data.
Ideally the backup is offsite, of course, since that protects against anything that might happen to one site. As a practical matter for non critical data, like your home PC, an external harddrive in a good fire/water/security safe will do the trick. It takes a lot to destroy one of those and your data is probably safe from just about anything, including you screwing shit up.
So having multiple online systems for better availability is fine. You don't want downtime, you have more redundancy so that if a given unit fails, the operation keeps going. However it's NOT a backup, especially if they are all on the same site. You need backups in addition to redundancy.
How much redundancy and how many backups depends on the importance of the data you are storing. At home, I do an external drive in a safe with some very important files copied to the server at work. At work, we have a NetApp storage unit (which is quite redundant itself) and back that up to tape, which gets rotated out to a vault in a different building. At a higher level at work, for things like financial records, that same kind of thing happens but there's a backup system in a different city as well.
Get yourself a good backup system BEFORE you need it.
> I'd like to see you recover something that has been overwritten once.
You can't do it at home, but professional data recovery service can. Usually you can guess the previous data by precisely measuring the magnetic levels. The old values will influence the resulting intensity. Roughly (I'm not expert!) works like this:
was -- now -- result
0 -- 1 -- 0.9
1 -- 0 -- 0.1
1 -- 1 -- 1.1
0 -- 0 -- 0
That is why you should have MULTIPLE overwrites with RANDOM data.
- tested
- offline
- off-site
- several times
anything else is "high-availability", not "backup".
The Cloud - because you don't care if your apps and data are up in the air.
Well, maybe, but it won't be cheap. I doubt that the guy running some amateur mod site is willing to fork over some thousands out of his own pocket to have someone take the drive apart and use an electron microscope or whatever on it.
A polar bear is a cartesian bear after a coordinate transform.
Wikipedia revision wars will be a GOLDMINE for future archeologist.
Think about just how much they reveal about a certain topic.
Such as the difference of opinion about the color variations of the carrot !
So they had no real backup strategy....but what happened to them REALLY REALLY sucks. It really irks me seeing so many comments saying these "retards" had it coming to them.
Listen folks....we're talking about a couple of guys who spent their free time creating a website. They're not making any real money out of this (in fact, they all have regular day jobs).
They've been advertising for a Tech Manager (non-paid) for quite a quite so time now. They did get one recently...but it turns out the guy harvested the emails from the systems and sent out a bunch of spam. He has since been fired.Even though the avsim folks aren't saying it was him who hacked and destroyed their site, it's quite hard not to think it was him.
It's been quite a blow to the flightsim community and I have noticed a lot of IT folks are offering help.....I just haven't seen a single one on this thread.
Tedious and expensive, but several people made a good living out of doing it (one guy I knew did it as a hobby and made over UKP100K one year.) However, as bits get smaller, servos get more accurate, and tracks get denser, the modus operandi just ceases to exist any more.
Mind you, for security reasons I always dismantle old drives and bend the disks in half using a lump hammer. That, and the fact that hard drive magnets are just incredibly useful if you have a steel hulled boat and want convenient attachments for e.g. cable ties. They are powerful and very short range, and usually nickel plated. To buy a pair of equally useful magnets from hardware stores costs nearly as much as a drive.
From scarped cliff or quarried stone she cries "A thousand types are gone, I care for nothing, no not one."
A few years ago, hackers would try to remain undetected in a system while they tried to infiltrate more systems, with the goal being to see how many they can get into... They wouldn't destroy data because that's a great way to get detected.
Even website defacers would move the old site to oldindex.html or similar when they performed a defacement...
Doing something so blatant and aggressive as to delete everything from a compromised server will lose you access to the system, as well as provoke the owners of it to try and hunt you down. Just what is the point?
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
To any sysadmins and DBAs...
Make sure you have offsite backups
Any person in the IT community who was alive to remember the events of 9/11 should have learned a valuable IT lesson from that event.
Repeat after me. I will not store my "offsite" backups in the other tower.
Why would you need to take that risk? It's standard business practice to just make a tape and ship it off site. The cost of shipping the tapes isn't worth the risk of leaving the backups on an internet connected box in my opinion.
If it's on the internet, then it is exposed.