Slashdot Mirror


Hacker Destroys Avsim.com, Along With Its Backups

el americano writes "Flight Simulator community website Avsim has experienced a total data loss after both of their online servers were hacked. The site's founder, Tom Allensworth, explained why 13 years of community developed terrains, skins, and mods will not be restored from backups: 'Some have asked whether or not we had back ups. Yes, we dutifully backed up our servers every day. Unfortunately, we backed up the servers between our two servers. The hacker took out both servers, destroying our ability to use one or the other back up to remedy the situation.'"

24 of 780 comments (clear)

  1. This should be a lesson... by Anonymous Coward · · Score: 5, Informative

    To any sysadmins and DBAs...

    Make sure you have offsite backups

    1. Re:This should be a lesson... by nemesisrocks · · Score: 5, Informative

      Make sure you have offsite backups

      In this case, even offline (as opposed to offsite) backups would have sufficed.

      Removable hard disks, DVDs -- hell, even tapes. These are all forms of backups that can't be compromised (well, easily) over the internets.

    2. Re:This should be a lesson... by coryboehne · · Score: 5, Informative

      It's actually very difficult to truly destroy data, especially remotely. There is actually a reason the DoD spec. requires physical destruction of the media.

      Unless you have overwritten the area on the physical disk that contained the data, multiple times, the data can still be recovered.

      The article doesn't lead me to believe that he's tried very hard to get this data back.. Maybe somebody (not me) who cares about this resource, should offer an attempt at data recovery.. Just be sure to hurry, before they do something that will ensure you cannot recover the data.

      I've recovered data off of formatted HDD's, off of corrupted file systems, off of compact flash cards and other media (Really useful if you want to keep those photo's that someone thought was deleted, be aware of this people).

      It's amazing how most people seem to think deleted means gone.

    3. Re:This should be a lesson... by unlametheweak · · Score: 3, Informative

      From the article

      ... we backed up the servers between our two servers.

      Nope, backing up a server to another online server is not a backup, it's merely another online copy.

    4. Re:This should be a lesson... by unlametheweak · · Score: 5, Informative

      Which reminds me. They could always use the WayBack Machine to (help in) retrieving their archives:
      http://web.archive.org/web/*/http://www.avsim.com/

      Google Cache seems to archive only the most recent pages:
      http://74.125.95.132/search?q=cache%3Ahttp%3A%2F%2Fwww.avsim.com%2F&submit2=Google

    5. Re:This should be a lesson... by QuoteMstr · · Score: 5, Informative

      pv < /dev/zero > /dev/device is pretty nifty too.

    6. Re:This should be a lesson... by jamesh · · Score: 3, Informative

      There are no reports anyone would be even able to restore data after rewriting them with simple /dev/zero. OTOH rewriting by /dev/urandom and /dev/zero costs mostly the same so why to care if /dev/zero is enough.

      Well, yes. And in fact due to the way data is encoded (MFM, RLL, whatever they use these days) a zero bit of data in a sector does not necessarily correspond to a physical zero bit in a magnetic sense.

      And given that one of the theories about how to recover data is "subtract the 'perfect' waveform of the track from the actual waveform of the track, and the difference will be some indication of the data that was there previously", it doesn't matter if a single pass is random, all 1's, or all 0's. If you were doing multiple passes then random data would be better, but psuedorandom would probably suffice as long as it was different with each rewrite because the objective is to push the variations well under the noise floor.

      cat /dev/something >/dev/sda is enough/easier on any Linux kernel, dd had to be used on some old commercial Unices nobody has seen for 30 years now.

      When I was writing floppies under AIX about 10 years ago, 'dd' with a suitable block size was many times faster than 'cat'. Maybe it wouldn't have made a difference for a harddisk though.

    7. Re:This should be a lesson... by batkiwi · · Score: 3, Informative

      Police forces do not recover data from overwritten disks.

      "Formatted" (quick format, destroying partitoin table) yes. Overwritten, no.

  2. lesson is by PhrostyMcByte · · Score: 3, Informative

    more than one backup. always! especially if two servers are running the same software, who says they won't both fail at the same time?

  3. There's a special place in hell... by Anonymous Coward · · Score: 3, Informative

    Reserved for people who don't do archival backups, don't secure their systems, and then try to blame their ineptitude on hackers.

    Do backups.
    Do security.
    Do restore from your backups to test them.
    Do not blame others when it's shown you failed steps 1-3.

  4. Real men... by hugetoon · · Score: 5, Informative

    "Only wimps use tape backup: _real_ men just upload their important stuff
    on ftp, and let the rest of the world mirror it ;)"
                                                        Linus Torvalds Jul 20 1996, 3:00 am

  5. Re:Hindsight is always 20/20 by inKubus · · Score: 4, Informative

    And for those who don't like to pay $10000 for backup software, there's Bacula. Couple that with an LTO-4 drive (~1000) and LTO-4 tapes (800GB uncompressed, ~60/piece) and you're set. Rsync.net is a decent, cheap online provider for those gaps when you haven't rotated tapes.

    Bacula is pretty sweet because it lets you backup to disk volumes and then you can schedule a roll to tape. So you can just back everything up incrementally to a disk volume and then copy those backups to tape, and then run rsync on the disk volumes to have an offsite, online backup. When recovering, you ask to recover from whatever's available. If you keep enough disk storage around (and there's really no reason not to) you can recover to any date in the past. In the event of a disaster your tapes come into play.

    Now with drives so cheap the temptation is to buy a external hard drive and use that. But tapes have a long history, guaranteed backwards compatibility (planned anyway, LTO drives have to R/W the previous generation and Read 2 generations back), last longer than moving drives, are simpler, lighter, more robust and more portable. Not that I wouldn't keep a external around to dump desktops but tape is the DR standard.

    --
    Cool! Amazing Toys.
  6. Lies, damn lies. by BrokenHalo · · Score: 4, Informative

    The admins' claim that they were backed up is nothing short of an outright lie. A dependency on rsync or any other mirroring technique alone is just plain negligent, when both servers are exposed to the world at large. As a bad analogy, it's like allowing someone to light two fuses with the same match.

    The only way to do backups properly is to have a complete set, offline, in a separate location.

    Sheesh. When will people learn?

    1. Re:Lies, damn lies. by Gerzel · · Score: 5, Informative

      Remember kids if it isn't backed up to an off-line copy then it isn't backed up.

    2. Re:Lies, damn lies. by SanityInAnarchy · · Score: 3, Informative

      I'm going to respectfully disagree, there.

      A dedicated backup box can be much more hardened than a general-purpose webserver, as the backup box pretty much has a job of storing and retrieving files.

      A solid system of incremental backups helps, too.

      Yes, taking it offline is great. Do that... maybe monthly, if that.

      This scenario sounds much more like someone confused "RAID" with "Backup". RAID (and other high-availability schemes) protects you from hardware failure. Backup protects you from more software failure and human error.

      --
      Don't thank God, thank a doctor!
    3. Re:Lies, damn lies. by mustafap · · Score: 3, Informative

      >but how many people actually keep off-site backups for home use?

      er, I do. I have a 4GB memory stick that I sync with my back drive on my home PC and a PC at work.

      Once a month I burn a DVD.

      --
      Open Source Drum Kit, LPLC deve board - mjhdesigns.com
    4. Re:Lies, damn lies. by magarity · · Score: 5, Informative

      A dedicated backup box can be much more hardened
       
      What you've described is only marginally better than what these people did. A second server playing backup device, even if it's "much more hardened", whatever that means, is still an extremely lousy and ineffective backup. If lightening hits your building or arson or theft, your "it's hardened"! backup server is just as toasted as the primary. Backups MUST be to removable media that's kept off site and inactive.
       
      Otherwise you've done practically the same thing for data "backup" as the RAID does via disks, except with two servers.

    5. Re:Lies, damn lies. by Ephemeriis · · Score: 3, Informative

      I'm going to respectfully disagree, there.

      A dedicated backup box can be much more hardened than a general-purpose webserver, as the backup box pretty much has a job of storing and retrieving files.

      A solid system of incremental backups helps, too.

      Yes, taking it offline is great. Do that... maybe monthly, if that.

      This scenario sounds much more like someone confused "RAID" with "Backup". RAID (and other high-availability schemes) protects you from hardware failure. Backup protects you from more software failure and human error.

      Wrong.

      What if your building burns down? What if some minor fire triggers the sprinklers? What if you get struck by lightning? What if an employee goes postal and takes a sledgehammer to all the electronics? What if a tree falls on the power lines and sends a giant surge through your wiring? What if someone breaks in and steals all the computers?

      It isn't a backup unless it leaves the site.

      Of course you could put your live backup box on the other end of some fiber in another state... That's physically off-site... But as long as it is up and running you have to worry about it as well. Hardened or not, it could get hacked. Or it could get a virus. Or some random glitch could corrupt the data on disk. Or its motherboard/HDD/CPU/whatever could die.

      It isn't a backup unless it is offline.

      And then there's the question of whether the thing actually works... You can have all the backups in the world, but if they're all corrupt it won't do you any good. You'll be restoring broken garbage to your replacement server.

      It isn't a backup unless it has been verified.

      What all of this comes down to is some kind of relatively portable media. Tapes, removable HDDs, CDs, DVDs, whatever. You want something that can leave the building on a daily basis. You want pretty much all your media to be out of the building. Bring in just what you need to run today's backup, and then take it out of the building as soon as that is done. Preferably to someplace relatively remote and safe... A safety deposit box is great. Or if someone has a safe at home. Or if you've got a branch-office or something.

      --
      "Work is the curse of the drinking classes." -Oscar Wilde
  7. Data is NOT backed up until it is by obarthelemy · · Score: 3, Informative

    - tested
    - offline
    - off-site
    - several times

    anything else is "high-availability", not "backup".

    --
    The Cloud - because you don't care if your apps and data are up in the air.
  8. Re:overwritten once CAN be recovered by crisco · · Score: 5, Informative

    The [a href="http://16systems.com/zero.php"]Great Zero Challenge[/url] says otherwise. They're simply asking for the filename of one of the files on a drive that has been wiped once with zeros. Despite offering the challenge for over a year and actively speaking to data recovery companies, no one has taken them up on the offer.

    --

    Bleh!

  9. Re:overwritten once CAN be recovered by crisco · · Score: 3, Informative

    Markup Fail! Great Zero Challenge

    --

    Bleh!

  10. The people running the site ARE NOT IT Admins by IvanTheNotSoBad · · Score: 5, Informative

    So they had no real backup strategy....but what happened to them REALLY REALLY sucks. It really irks me seeing so many comments saying these "retards" had it coming to them.

    Listen folks....we're talking about a couple of guys who spent their free time creating a website. They're not making any real money out of this (in fact, they all have regular day jobs).

    They've been advertising for a Tech Manager (non-paid) for quite a quite so time now. They did get one recently...but it turns out the guy harvested the emails from the systems and sent out a bunch of spam. He has since been fired.Even though the avsim folks aren't saying it was him who hacked and destroyed their site, it's quite hard not to think it was him.

    It's been quite a blow to the flightsim community and I have noticed a lot of IT folks are offering help.....I just haven't seen a single one on this thread.

  11. You are correct about drive age by Kupfernigk · · Score: 5, Informative
    Data recovery was possible, and was not actually that hard, on older drives. The reason was the size of the bits, and the inaccuracy of the tracking servos. As a result, an overwrite would rarely be on exactly the same path as the original data. Mounting the disc in a special drive with precision tracking and more than one head meant that the overwritten data could be read by the leading head, and then used to generate a correction signal which was added (with the correct delay) to the signal coming from the trailing head which was on a different alignment and so was picking up more of the previous signal. We're talking raw signal here, not ones and zeroes.

    Tedious and expensive, but several people made a good living out of doing it (one guy I knew did it as a hobby and made over UKP100K one year.) However, as bits get smaller, servos get more accurate, and tracks get denser, the modus operandi just ceases to exist any more.

    Mind you, for security reasons I always dismantle old drives and bend the disks in half using a lump hammer. That, and the fact that hard drive magnets are just incredibly useful if you have a steel hulled boat and want convenient attachments for e.g. cable ties. They are powerful and very short range, and usually nickel plated. To buy a pair of equally useful magnets from hardware stores costs nearly as much as a drive.

    --
    From scarped cliff or quarried stone she cries "A thousand types are gone, I care for nothing, no not one."
  12. Re:Offsite backups? by mcvos · · Score: 3, Informative

    They should be kept on a different part of the electricity grid, preferably in a differnt postcode.

    It all depends on what kind of disasters you want your data to survive. If you want it to survive nuclear war, you need off-shore backup. Preferably in a neutral country that won't get involved in the war.

    If you want your data to survive a Vogon constructor fleet, use off-planet backup. Recovering it from the brain of a single surviving human (if any) is going to be costly and painful.