Slashdot Mirror


Hacker Destroys Avsim.com, Along With Its Backups

el americano writes "Flight Simulator community website Avsim has experienced a total data loss after both of their online servers were hacked. The site's founder, Tom Allensworth, explained why 13 years of community developed terrains, skins, and mods will not be restored from backups: 'Some have asked whether or not we had back ups. Yes, we dutifully backed up our servers every day. Unfortunately, we backed up the servers between our two servers. The hacker took out both servers, destroying our ability to use one or the other back up to remedy the situation.'"

62 of 780 comments (clear)

  1. It isn't a backup... by IntentionalStance · · Score: 2, Insightful

    if it isn't verified

  2. Three words: by Girtych · · Score: 4, Insightful

    Off. Site. Backups. Textbook example of why you need to secure your backup data in a secure, non-networked location.

  3. yes we had backups by frovingslosh · · Score: 4, Insightful

    They say they had backups, and put them on the Internet where any hacker could get to them, under the same security the originals were stored under. If that's all they cared about their data, I don't see why the Slashdot community should care any more than they did.

    --
    I'm an American. I love this country and the freedoms that we used to have.
    1. Re:yes we had backups by BenBoy · · Score: 4, Insightful

      Absolutely, I mean, so what if those guys broke into your house and killed you and raped your mom *right in your own basement bedroom* ... y'know, you should have had better locks, and used them more consistently; y'know, if you'd really cared.

      Sure, there are *much* better backup strategies; that having been said, somebody broke in and did a bunch of damage for shits and grins. They suck.

    2. Re:yes we had backups by maxwell+demon · · Score: 3, Insightful

      That's a really bad analogy. The backup is not there in order to prevent hackers from breaking in, the backup is there in order to prevent loss if they break in, or if data gets lost for some other reason. That is, backups are not a security measure, they are a measure to limit damage.

      --
      The Tao of math: The numbers you can count are not the real numbers.
  4. Sigh. Mirror != backup by Todd+Knarr · · Score: 4, Insightful

    Repeat after me: mirroring is not a backup. Backups are physically removed from the machine and stored where they can't be altered until they're needed for a restore. If they aren't removed from the machine, well, as we've just seen that only ends in tears. Observe their pain and learn from it!

  5. So, they had NO backups? by MrMista_B · · Score: 3, Insightful

    'Backed up between two servers'... that's not what a backup is.

    I'm... astonished at the level of incompetence here. A site with 13 years of work like this, and they didn't bother to backup anything at all?

    And now they're trying to handwave it away with 'oh uh, uh really folks, seriously, were really did have backups haha, between servers olol'.

    I don't think 'olol' is going to impress anyone whos work was just wiped out by their incompetence.

    1. Re:So, they had NO backups? by SpeZek · · Score: 0, Insightful

      I don't think 'olol' is going to impress anyone whos work was just wiped out by their incompetence.

      Anyone who relied on online storage to store their work without having backups of their own is even dumber than these guys.

    2. Re:So, they had NO backups? by borizz · · Score: 2, Insightful

      And that's stupid. Fact of the Intertubes: Shit is going to get probed/hacked. Designing your backup policy in a way that doesn't cover malicious attackers when you're securing an internet facing website is just asking for trouble.

  6. Re:Copying between servers is NOT backing up by lecithin · · Score: 3, Insightful

    "I hope the same administrator will never again make the same mistake with backups."

    He won't for this company, that is for sure.

    --
    It could be worse, it could be Monday.
  7. And yet another example why you need real backups by Fallen+Kell · · Score: 3, Insightful

    As the subject says. "Online" backups and replication are simply tools to try and minimize downtime. They are NOT a backup solution. They never were and never should be touted as one, just as this example shows. The only good backup is one that occurs frequently, is verified that it worked, and is stored in a secure location such as a fire-proof safe, and even better in two different fire-proof safes in two different locations, preferably more than 100 miles apart.

    --
    We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
  8. These aren't hackers by fishnuts · · Score: 4, Insightful

    Whoever did this must have willfully wanted to destroy the website and its content. Deleting data in this manner is far beyond vandalism or criminal mischief.

    I hope the perps get served by a judge who recognizes just how severely malicious this was, and that enough of the people who used the site can provide the files back to the owners and the community.

    1. Re:These aren't hackers by Anonymous Coward · · Score: 1, Insightful

      I'd look for any members who recently had a disagreement with the community. Wouldn't surprise me if it was the result of a petty flame war.

  9. Re:This should be a lesson... by _xeno_ · · Score: 5, Insightful

    How about we just shoot all hackers?

    I'm not sure how that will protect against data loss from equipment failure, natural disaster, fire, software failure, solar flares, Secret Service, or really anything other than hackers.

    Offsite, offline backups aren't a good idea solely to protect against hackers. They're a good idea to protect against data loss in general.

    --
    You are in a maze of twisty little relative jumps, all alike.
  10. There's a perfectly good set of words for... by Chris+Tucker · · Score: 4, Insightful

    ...the thieves and vandals who steal data and wreck servers.

    THIEVES AND VANDALS.

    Not "hackers".

    What was done was not hacking. It was vandalism. Plain and simple.

    Hackers create. Vandals destroy. Thieves steal.

    I'm surprised that this needs to be explained to the Slashdot community.

    --
    Guaranteed! This comment 100% Anthrax free!
  11. Re:This should be a lesson... by adolf · · Score: 5, Insightful

    What, you mean like this guy? You probably wouldn't even have the browser you're using right now if it weren't for that particular, uh. hacker.

  12. Re:This should be a lesson... by Anonymous Coward · · Score: 5, Insightful

    this really is a pathetic situation. Everybody is hammering these guys for just mirroring their data and saying that they should have had off site backup.........true, they should have. What really is the issue here is that ASSHOLES feel the need to attack for the sake of attacking a site. It would be like me going out and punching random people in the face just because I can.

    We have to stand up for those that cannot stand up for themselves.

    People that destroy just because they can are completely USELESS...............and should be SHOT.

  13. Re:This should be a lesson... by Khashishi · · Score: 4, Insightful

    multiple times? I'd like to see you recover something that has been overwritten once.

  14. Re:This should be a lesson... by jamesh · · Score: 5, Insightful

    Unless you have overwritten the area on the physical disk that contained the data, multiple times, the data can still be recovered.

    People keep repeating that mantra to each other, but is it really true? Getting data off a 'formatted' disk is pretty easy as a format rarely does more than write a few sectors at the start of the disk. Getting data off of a disk that has had 'dd if=/dev/random of=/dev/sda' done to it is a different matter altogether.

    There have been papers written about getting some data out of the inter-track space, and scraping it off the noise floor etc with electron microscopes, but as far as I have researched, nobody has actually done it.

    I put it to you that more people have had their kidney's stolen after meeting a pretty girl at a party than there have been disks recovered after being completely overwritten with random data.

  15. Hindsight is always 20/20 by mlts · · Score: 5, Insightful

    This is a lesson every system administrator worth his or her salt learns over the long haul. You might back up dutifully, test restore, and have a well done system of ensuring backups are rotated correctly. Then you find out the tape drive you use is miscalibrated so only it can read your backup tapes, or you find the backup software you use on a daily basis is not in production, or the latest version has no support for the backlevel formats.

    I have found that in a production environment, you really need multiple methods for backup if at all possible:

    The first level is a dedicated backup server. This machine is locked down to the best of your abilities, and firewalled from the network, only allowing critical ports such as what the backup software uses, and perhaps ssh or RDP (if a Windows box). This machine copies everything from the other servers onto a large disk array, then to tape. The tapes are then cycled offsite via a service like Iron Mountain. Of course, the tapes are encrypted, and corporate officers get a copy of the master keys.

    Why tapes? Because they can be set read only after they are dismounted, and no computer, no matter how infected can modify or delete the tape contents once this is done, outside of a reflash of the tape drive's BIOS. This is important because its not unheard of for someone to write a program that trashes backups over a time interval. Higher end tapes can be used as WORM media like DLT-ICE.

    I can't emphasize enough about securing the backup server, both physically and network-wise. If this box gets compromised, all your data is available. On Windows machines, I recommend using some form of disk encryption (Bitlocker if the machine has a TPM, TrueCrypt, etc) so if the backup server or an array gets physically stolen, the data is of no use to a thief. This is in addition to the backup program's encryption.

    After you have a central backup server installed, secured (security is paramount on this machine unless the backup program client can do encryption), and backups running, you focus on the other levels of backup.

    The next level of backup is on the local servers. Most operating systems have a method of backing up the computer. If you can do this with a server, fire off a snapshot backup every month or so. Most OS backup methods don't have encryption, so this backup should go directly to a tape safe or secured container in the data center. Optionally, you can install backup software locally that can encrypt. I like using the backup/restore utility the OS gives for an image every quarter, then using more secure software more often, so the OS backups can be stored in a tape safe or physically secure container. This way, if the third party backup software ends up inoperable, there is still a method of getting a machine up somehow, or putting it in a virtual machine for recovery purposes.

    Finally, after you have backup servers and a rotation, companies might consider offsite cloud backup services like Mozy. Mozy offers use of keyfiles so all data is stored encrypted (encrypted on the client end). Of course, making sure the encryption key is stored safely is paramount, and the cost of storing a large backup in Mozy's cloud may be prohibitive. However, if worse comes to worst and your site is completely knocked out, as well as the offsite backup site, it may be thing that keeps your business up.

    Of course, scale this up or down as per your company's needs. A smaller business can get by using Mozy and a Windows Server 2008 box running Bitlocker, a network backup program with encryption such as Retrospect or Backup Exec, and using external drives every month to copy backup sets from the main ones to store offsite.

    A larger business might see about a true backup fabric system sold by IBM (TSM), EMC (Networker), or Microsoft's solution.

    The key is to not just have some built in redundancy so if one backup method is not usable, you have another, even if the backups are older, but to be able to do this in a manner that doesn't add too much time and equipment expense.

  16. Re:Copying between servers is NOT backing up by Khashishi · · Score: 2, Insightful

    Honestly, how many man-hours and equipment do you really want to commit to backup? Do you really think it's worthwhile to get a tape system and regularly move tapes off-site for some community mods? Anyone can envision a system that is far more secure than this, but paying for it is another thing.

    If the mods were good quality and downloaded often, the community should be able to act as a backup of sorts.

  17. Re:This should be a lesson... by Steffan · · Score: 2, Insightful

    ... we backed up the servers between our two servers.

    Nope, backing up a server to another online server is not a backup, it's merely another online copy.

    It's the difference between HA [High Availability] and DR [Disaster Recovery].

    Unfortunately, they suffered a disaster, not a 'mere' server failure.

    All that said, my condolences to the server admin / founder, and especially, to all of the contributors. Thirteen years is a lot of data.

  18. Re:This should be a lesson... by bill_kress · · Score: 2, Insightful

    Or pay them to find shit like this before someone does this.

    The logic behind "Destroy your only resource that can work to actually help you fix the holes that will be exploited by foreign hackers or terrorists" is completely beyond me.

    In fact, it seems so utterly stupid that I get furious every time I hear some thoughtless moron spout "Punish the hackers". Suggesting they should be killed? I'd personally sooner keep those intelligent if misguided people--being the only ones that are really going to be useful at preventing external penetration of our systems--and kill assholes who can't think of a solution beyond a statement like "Kill the hackers".

    Not that I'd really condone either, but if I had to choose...

  19. Re:This should be a lesson... by Anonymous Coward · · Score: 1, Insightful

    I work in IM and Digital Asset Management, and my mantra "mirrors" many others in the field:
    If your data doesn't exist in three places, it doesn't exist at all.

    It's a shame in this day and age, people feel secure with having two online backups. The most reliable backup is off-line and off-site.
    If you can afford 2 servers, you can't NOT afford 1 USB hard drive.

  20. Re:This should be a lesson... by addsalt · · Score: 5, Insightful

    In fact, it seems so utterly stupid that I get furious every time I hear some thoughtless moron spout "Punish the hackers".

    A little blame needs to come from all areas. Not every website or messageboard is run by someone with a CS degree with a minor in website security. A break-in of a government site or large corporate site is one thing, a family website another. This site is probably somewhere in between.

    Saying it isn't the hackers fault that improper mehtods were used to secure a site is like saying it isn't the muggers fault that the lady's handbag was so easy to steal.

  21. Re:the web is ephemeral by imsabbel · · Score: 3, Insightful

    Wikipedia revision wars will be a GOLDMINE for future archeologist.

    Think about just how much they reveal about a certain topic.

    --
    HI O WISE PRINCE. WHT TOOK U SO DAM LONG?
  22. Re:This should be a lesson... by adamchou · · Score: 2, Insightful

    Well, besides that, this site sounds like a community where people share UGC. This means that although they might not have it centrally backed up, they still have all the UGC out there, somewhere. I'm sure they can recover a good portion of their original content. The forums will be a bit harder to replace. But all that knowledge is in someone's head.

  23. Re:the web is ephemeral by Vectronic · · Score: 4, Insightful

    nonsense...

    completely inaccurate guestimation, but probably only about 1% of anything carved in stone, is still decipherable or even exists, same with scrolls, otherwise we'd be littered with 2000 year old shopping lists, love letters, etc, how many notebooks (the paper kind) have you gone through during school, as journals, boredom... still have them all?

    Hell, we probably only have about 1% of the stuff that was written down 100 years ago, probably only about 3% of the buildings, 0.3% of the cars, 2% of the paintings...etc...etc... most of the ancient books we have, are copies of copies of copies, and we can do that with magnitudes of efficiency now, not to mention recovery, hard drive gets erased, it's easier to get the data back than a scroll that's been erased, or a stone.

    If even 0.1% of what we have on the internet right now exists in 500 years, it'll still probably be more than everything we have in stone, scrolls, and print right now...

    With the various sorts of "Library of Congress" out there, if you had the chance to peruse and take/read whatever you wanted, you'd probably only find 0.5% of it interesting anyways, much like what's on the internet.

  24. Re:There's a special place in hell... by Anonymous Coward · · Score: 5, Insightful

    It's beyond me how the blame is always shifted on the victim of an attack. There's a line between equitible share of responsibility and blame, and it's nowhere as fine as you think it is.

  25. They didn't have backups by Sycraft-fu · · Score: 2, Insightful

    They had redundancy. Another online copy of data isn't a back, it is redundancy. A backup is a separate, offline copy.

    For example if you have a RAID-10, you do NOT have a backup of your data. What you've got is redundancy. In the event you have a disk failure, you don't lose data and you also don't lose system functionality. That's actually the main reason for RAID (at least RAID other than 0). You don't want your system to have downtime. If you drop a disk you can use the system while the replacement comes in, rather than being SOL.

    A backup is separate. It can be another harddrive, it can be DVDs, it can be tape, whatever. It is something you use to take data from the system, and move it offline.

    Now why is the offline thing so important? Well this demonstrates one reason. A bigger one would be catastrophic hardware failure. What happens if your PSU goes nuts and pumps out 120 volts on the 12v lines? That kind of thing can burn out all your hardware, and thus anything you have internally. An external backup isn't affected, of course. Then there's things like fire, or flood and so on.

    However the biggest would be your own screwup. What happens if you accidentally overwrite the data with garbage? What if you then trigger a backup sync, or it happens automatically before you realize your mistake? Well you are screwed now. You backup is now of useless data.

    Ideally the backup is offsite, of course, since that protects against anything that might happen to one site. As a practical matter for non critical data, like your home PC, an external harddrive in a good fire/water/security safe will do the trick. It takes a lot to destroy one of those and your data is probably safe from just about anything, including you screwing shit up.

    So having multiple online systems for better availability is fine. You don't want downtime, you have more redundancy so that if a given unit fails, the operation keeps going. However it's NOT a backup, especially if they are all on the same site. You need backups in addition to redundancy.

    How much redundancy and how many backups depends on the importance of the data you are storing. At home, I do an external drive in a safe with some very important files copied to the server at work. At work, we have a NetApp storage unit (which is quite redundant itself) and back that up to tape, which gets rotated out to a vault in a different building. At a higher level at work, for things like financial records, that same kind of thing happens but there's a backup system in a different city as well.

    Get yourself a good backup system BEFORE you need it.

  26. Re:This should be a lesson... by EvanED · · Score: 2, Insightful

    Because after all, we know that words only have one meaning, so if someone uses the word "hacker" one way, it must mean the same thing as when everyone uses the word hacker.

  27. Yes, but it's not cheap by Moraelin · · Score: 2, Insightful

    Well, maybe, but it won't be cheap. I doubt that the guy running some amateur mod site is willing to fork over some thousands out of his own pocket to have someone take the drive apart and use an electron microscope or whatever on it.

    --
    A polar bear is a cartesian bear after a coordinate transform.
  28. Re:overwritten once CAN be recovered by DerekLyons · · Score: 2, Insightful

    Nobody has taken them up on the offer because they (16 Systems) are meaningless nobodies seeking to use the data recovery companies for their own PR ends.

  29. Re:This should be a lesson... by EvanED · · Score: 3, Insightful

    I think everybody in the Linux and MS-DOS-prompt community knows what a hacker is. However, I will supply you with a formal definition:

    Why's that the definition we should be using? Are we in the Linux and MS-DOS community? Hell, even /. doesn't fall into that camp; last I heard (which was admittedly a good while ago) the majority of visitors here were using IE.

    And I can also supply the definition for a hacker, from a bit more authoritative sources than uncyclopedia. One of Random House's definitions is "a microcomputer user who attempts to gain unauthorized access to proprietary computer systems." Or the American Heritage Dictionary: "One who uses programming skills to gain illegal access to a computer network or file."

    Sure, both of these have the "computer enthusiast" definition preferred by ESR too, but that's my point -- words have more than one meaning. And unless you're not very familiar with English, stupid, or deliberately being obtuse, it's pretty clear which one is intended here.

    And unless there's something big that Jamie Zawinski's wikipedia page leaves out, one of those applies to adolph (the poster I was responding to originally).

  30. Re:overwritten once CAN be recovered by hoggoth · · Score: 5, Insightful

    I am a computer forensics expert. I search for deleted data for a living, and I testify in court as to what can be done.

    Unfortunately you are wrong about recovering data that has been overwritten by using magnetic magic.
    That is an urban legend that has been disproven. Maybe 20 years ago using low density MFM drives it was theoretically possible, but now it is not. Maybe the NSA has some tech they reversed engineered from an Area-51 UFO to do this, but I've never seen or heard of it.

    Even Gutmann has recanted his 38 wipes recommendation.

    Now don't mistake overwritten data for deleted data. When data is deleted it is NOT overwritten. When a hard drive is re-formatted almost nothing is over-written. When a file is overwritten with zeros or random bytes there are probably 10 more copies of that file and previous versions of that file floating around in unallocated sectors, swap space, file slack, hibernation files, etc.

    But what IS overwritten is gone.

    --
    - For the complete works of Shakespeare: cat /dev/random (may take some time)
  31. Re:Lies, damn lies. by rtfa-troll · · Score: 5, Insightful

    And I'm going to respectfully disagree with you too.

    For most small businesses cash flow is critical. If you don't have a record of who paid you in the last month then you can't invoice the rest and you are dead. Your repeat customers will spot duplicate invoices and probably just block payments until it's all sorted out. The attack that you are defending against is either a fire which destroys your office or a burglary which steals all your computers in the night, including the backup box, taking the backups just because they happen to be there.

    You need off site backups on a different, non internet-connected medium no less often than once a week. That is the maximum time for which it is acceptable (we are talking about disaster recovery here; "acceptable" has a different meaning from normal) to re-invoice people who have already paid you. Even so, most such incidents destroy small businesses completely just because they don't manage to get people back working in time. This just gives you a fighting chance if you have a nice and understanding bank manager and do a little more disaster planning. It is astounding how much difference spending four hours just thinking about it can make (e.g. you know the number of the temporary office providers, you know which people in your office can work from home and you realise everybody in your company should have a mobile phone, especially the receptionist).

    And finally; if you haven't tried restoring from it, it isn't a backup.

    --
    =~ s,(.*),<sarcasm>$1</sarcasm>,g if any_point_you_wish();
  32. Re:overwritten once CAN be recovered by BinaryOpty · · Score: 2, Insightful

    In addition, the reward is far below the cost of the processes needed to retrieve that data, so no one's going to bother for that reason as well.

  33. Re:Total Pwnage by Anonymous Coward · · Score: 1, Insightful

    Still. Thirteen years worth of data, and they didn't have a single tape, a single external drive, a single... anything, not even a fucking burned CD that might help them. No, I respectfully disagree, SECURITY can only do so much. (However, for the avsim.com admin I seriously have zero respect.) Security might have prevented this attack, but what if there was a fire, or a burglary, or some careless jackass with a cup of coffee? Stranger things have happened that cause the exact same outcome, total data loss. This isn't just about disaster prevention, this is about common goddamn sense, which the admin of this site apparently have none of at all.

    One offline (and preferably off-site) backup, even if it wasn't complete to that day or even that month, would have been the difference between losing everything and losing almost nothing.

  34. Re:overwritten once CAN be recovered by getuid() · · Score: 5, Insightful

    I've never seen *any* evidence or heard of *any* occasion that such a recovery, even from a only-once-zeroed drive was done.

    Now the point is, one could say "of cooourse not, guys that can do this won't do it for peanuts, besides they're secres service" etc etc. But the point is: even if it's secret service and really expensive, at leas *some* news about it should have hit the public -- after all, this myth has been around for several years (a decade?) now.

    I'd still even like to hear from a success story. Or even find a company that advertizes "We can (partly?) recover your zero'ed data -- it's going to cost a fortune, an arm and a leg, but we can." Haven't seen that one either yet. Not a commercial, not an offer, nothing... besides legends.

  35. Re:Lies, damn lies. by SausageOfDoom · · Score: 3, Insightful

    I don't think anyone would disagree that the backup machine has to be at a separate location, but you and the gp poster are saying it's somehow risky if it's internet connected. You should be fine provided:

    * the backup box only runs an up-to-date SSH server with key-based access
    * it's hidden behind a firewall and/or port knocking
    * it connects out to the primary server to initiate the backup and pull the data (rather than the other way around)
    * you make incremental backups

    That way when your primary machine is compromised, all they can do is corrupt your live data, and your backups from that date.

    Certainly keep weekly/monthly off-site offline backups as well, just in case, but I think it's wrong to say you can't have a reasonable expectation for the reliability of an online backup box.

    After all, plenty of things can go wrong with offline backups, but there's a reasonable expectation that they will be fine.

  36. Re:Lies, damn lies. by trawg · · Score: 3, Insightful

    I wouldn't call it lies - I'd call it ignorance

  37. Re:Lies, damn lies. by NoobixCube · · Score: 3, Insightful

    Off topic, the internet would be a much nicer place if all disagreements were presumed to be respectful until obviously indicated otherwise...

    --
    Admit it. You post strawman arguments as AC so you get modded Insightful for refuting them, rather than Troll
  38. Re:Lies, damn lies. by Alex+Belits · · Score: 3, Insightful

    They buy USB hard drives (at least six times the amount of data they have, split among at least three drives), rent a safety deposit box in a bank, and install rdiff-backup. Then they rotate the drives weekly -- at any point one drive is backing up their systems daily, two are stored at the bank. Complete incremental backup solution with offsite storage.

    --
    Contrary to the popular belief, there indeed is no God.
  39. Re:Lies, damn lies. by fuzzywig · · Score: 2, Insightful

    And if you can't restore from it then it's not backed up either. Test those backups people, test 'em!

  40. Re:Lies, damn lies. by sigxcpu · · Score: 2, Insightful

    If we are making a list of backup rules, I should also add that if you have not tested to see that you can actually recover from the offline copy, it is not backed up.

    It is very common for the first few restore attempts to fail because of a miss-configured backup solution.

    One really colossal failure I have witnessed was when several years of offline backups were found to be useless, following a server failure.
    It appears that the backup agent did not have the right permission to read some of the files.
    (Yes, it generated errors that should have not been ignored.)

    Another really painful one I witnessed was loosing the only 10 year old tape drive, this side of the ocean, that can read the media to a fire, along with the backed up server.

    The only way to know that your data is probably safe is after you have seen a successful restore, on another machine.

    --
    As of Postgres v6.2, time travel is no longer supported.
  41. Re:Lies, damn lies. by RichardJenkins · · Score: 4, Insightful

    Backups: Not hard to get right, just very easy to get wrong.

  42. Re:bullshit by QuantumG · · Score: 2, Insightful

    What services? Where? What is their name? Fucking urban legend bullshit.

    --
    How we know is more important than what we know.
  43. Re:Lies, damn lies. by RichardJenkins · · Score: 4, Insightful

    If by 'dedicated backup box' you mean two offsite machines both of which are themselves in highly secure and robust sites I could be convinced that it's possible to build an effective backup strategy around them.

    Our backup strategy for the office (files/databases) is to have a single, off-site 'consolidation server' which we dump transaction logs to real time (with full database dump overnight), and make incremental backups of files every hour throughout the day rsyncing the full current file overnight. Then, this machine is itself backed up using a full weekly/daily differential tape backups.

    I get shivers how everyone talks about backup strategies but not restore strategies as if the data fairy will wave a wand to restore your backups when it all goes tits up. We have a regularly rehearsed backup strategy. If we're in the same office, we attempt a pull down from the remote consolidation server. If that is down, or we are in another office etc. we put a copy of the encrypted tape backups on USB drives, courier them over and restore them here.

    My biggest bugbear is that the remote consolidation server is not encrypted - we have to trust the hosting partner. We could not find an acceptable method that didn't involve remote plaintext data existing.

    We spend about £12k (or about half a junior IT FTE) a year on backups and there is not a single day where I do not worry and personally check that they're working correctly.

    Does anyone see any holes or room for improvement? Would be very happy for suggestions to improve.

  44. Re:The people running the site ARE NOT IT Admins by An+dochasac · · Score: 4, Insightful

    Mod parent up. These guys made mistakes, but well paid admins for enormous organizations make these same mistakes. (Bush's email anyone ;-) We should be more interested in informing and helping than in criticizing and 'persecuting(sic)'. When I first started in IT, I brought a hard drive back which contained important data for an Aids research clinic. I suggested that they make sure to do a backup now. I felt for them because the state of the art PC tape backup technology in 1988 was so slow, expensive and prone to eat tapes that I'd have almost suggested swapping out a 2nd MFM drive every day. A few weeks later I got a call, they'd lost their data again and this time there wasn't much I could do. Real men backup their data to slashdot. I hope you don't mind if I use this thread. beegin 665 mydailybackup.uue M27-N)W0@=&AIR!A(&=R96%T(&)A8VMU"$*27-N)W0@=&AIR!A(&=R96%T )(&)A8VMU"$* end

  45. Re:Offsite backups? by Kjella · · Score: 2, Insightful

    The day after 9/11 I was in an elevator, and caught a snippet of conversation between 2 people that had business interests with a firm that was in the WTC. The comment I heard was 'their backups were in the other building'. Another company lost.

    If you start going down that path, you end up at what'd I'd call the company doomsday scenario. If you first try to imagine a DR situation of such magnitude that both WTC locations are destroyed, it might as well be someone blowing up the foundations in which case they'd all be dead. We sometimes go on company trips, often a fully chartered plane. If that plane had crashed and 100+ employees were lost, the company would be G-O-N-E. DR is supposed to save you from recoverable situation, if all that's left as is a smoldering crater companies like people sometimes are beyond rescue.

    --
    Live today, because you never know what tomorrow brings
  46. Downhill... by Bert64 · · Score: 2, Insightful

    A few years ago, hackers would try to remain undetected in a system while they tried to infiltrate more systems, with the goal being to see how many they can get into... They wouldn't destroy data because that's a great way to get detected.
    Even website defacers would move the old site to oldindex.html or similar when they performed a defacement...

    Doing something so blatant and aggressive as to delete everything from a compromised server will lose you access to the system, as well as provoke the owners of it to try and hunt you down. Just what is the point?

    --
    http://spamdecoy.net - free throwaway anonymous email - avoid spam!
  47. Re:Lies, damn lies. by digitig · · Score: 3, Insightful

    It's not just accounts received that matters. At a company I used to work for we once got a letter from a supplier saying that they'd lost all accounts in a crash and could we please tell them how much we owed them. It's one thing not knowing whether an invoice has been paid: not knowing who to invoice or for how much is more serious. In that case it did turn out to be a death sentence.

    --
    Quidnam Latine loqui modo coepi?
  48. Re:Lies, damn lies. by Anonymous Coward · · Score: 5, Insightful

    "And finally; if you haven't tried restoring from it, it isn't a backup."

    That, my friend, need to be carved on a marble plate and hung over the door to every datacenter.

  49. Re:overwritten once CAN be recovered by commodore64_love · · Score: 2, Insightful

    By that logic Santa Claus might exist - he just hasn't revealed himself yet.

    For myself I prefer the scientific method, where if a thing or technique has never proved itself to exist, then it does not. Not seen == not believed. Therefore I don't believe an erased and zero'd hard drive can be recovered.

    --
    "I disapprove of what you say, but I will defend to the death your right to say it." - historian Evelyn Beatrice Hall
  50. Too Risky by DJRumpy · · Score: 5, Insightful

    Why would you need to take that risk? It's standard business practice to just make a tape and ship it off site. The cost of shipping the tapes isn't worth the risk of leaving the backups on an internet connected box in my opinion.

    If it's on the internet, then it is exposed.

    1. Re:Too Risky by SausageOfDoom · · Score: 2, Insightful

      Sorry, I think we might be talking at cross purposes. You said "why take the risk", and my point was that there was a reason to have an online backup box, namely that by automating it you can avoid any issues such as human holidays or disasters making the data centre inaccessible etc. It's also likely to be faster and easier to restore from an online backup, especially if you don't have little or no physical access to the machines (ie co-located or rented dedicated in a DC in another county or country).

      I certainly didn't suggest that you should use online without any offline backup. Like I said, there's a reasonable expectation that online can be secured, and a reasonable expectation that offline can be relied upon, but you have nothing to lose by running both together.

  51. Re:This should be a lesson... by Dan541 · · Score: 5, Insightful

    13 years of work lost!

    Suddenly those external hard drives and safe deposit box don't look so expensive.

    --
    An SQL query goes to a bar, walks up to a table and asks, "Mind if I join you?"
  52. Re:Lies, damn lies. by icannotthinkofaname · · Score: 2, Insightful

    I'm going to respectfully agree. :)

    --
    Let q be a radix > 1. I am in ur base-q, killing 10 d00ds.
  53. Re:This should be a lesson... by camperdave · · Score: 3, Insightful

    well, rocket science isn't really "rocket science". You put fuel in it, it burns, and it ejects pressure from the back.

    How much fuel? What type? How toxic is it? Does it require special handling? Does it require special tanking? What are the safety procedures? Are there boil-off rates, or evaporation rates? What type of oxidizer will the fuel require? How much will it require? How big of an engine? What type of nozzle? Will it withstand the heat of the exhaust? What materials will it be built from? How do you ignite the fuel? Is the rocket strong enough to withstand the launch stresses? What happens when it breaks the sound barrier? What size of payload will that much of that type of fuel running through that engine lift? Will it reach orbit? Will it reach the size and shape of orbit needed to put that payload where we want it? Will it be able to do that in a single stage or multiple stages? Will it need multiple engines? How many? What layout? Will the upper stage engines start in a vacuum? When does the staging event occur? Where will the spent stages land? Are they expendable, or do they need to be recovered? What sort of accelerations will the payload experience? Will they be gentle enough to put humans on top of the stack? Can the engines be throttled? What sort of failure modes does this rocket experience? Is there a way of detecting an imminent failure? Will there be enough time to trigger the Launch Abort System? What sort of guidance system will there be? How will you steer the rocket? Is it even dynamically stable? What happens to that stability if an engine fails? What happens to the center of mass as the fuel is expended? Does that affect the stability?

    Yeah, rocket science is real easy.

    --
    When our name is on the back of your car, we're behind you all the way!
  54. Re:This should be a lesson... by Just+Some+Guy · · Score: 1, Insightful

    Those aren't mutually exclusive positions. Yes, fry the hacker for destroying someone else's work for giggles. However, this is a known danger for Internet-facing servers, and not taking that into account when designing a backup plan deserves ridicule.

    In real life, muggers are scum who deserve whatever punishment they get. However, walking through the hood with your wallet dragging along on a string a block behind you doesn't get you a lot of sympathy when it gets stolen.

    --
    Dewey, what part of this looks like authorities should be involved?
  55. Re:The people running the site ARE NOT IT Admins by pandrijeczko · · Score: 2, Insightful

    Just like you can give a smartass answer because it's not you it happened to, you'd probably be able to give a smartass answer as to why it wasn't your fault if it had been you.

    I've never used the site (I don't even play flight sims) but I feel sorry for the guys because they've actually done something that is in the spirit of what the Internet should be - namely useful (at least to some people) and even better, FREE!

    Yes, I'll have myself a good chuckle if Microsoft, Sony or [INSERT FACELESS CORPORATION HERE] get hacked but not these guys who are just hobbyists.

    I'm a well-paid security consultant and five years ago my home server got hacked because I rather stupidly forgot to turn an FTP server off - it happens to the *BEST* of us and the only thing to do is learn from the experience. But it doesn't help when a patronisingly smug individual like you makes retarded comments.

    --
    Gentoo Linux - another day, another USE flag.
  56. Why I dis tapes by Anonymous Coward · · Score: 1, Insightful

    People always dis tapes.

    I dis tapes because of sour grapes. There was a period where enterpri-- well, ok -- low-end enterprise tape systems were affordable by small business and home users. For $600 and $15 per tape I could buy the same stuff that my 200-desktop clients were using, and it was big enough to back up my home computer.

    But hard disks got bigger and affordable tapes didn't. :( Now I fucking hate tape, because I can't afford a tape drive that can back up a $100 drive.