Slashdot Mirror

← Back to Stories (view on slashdot.org)

Code Spaces Hosting Shutting Down After Attacker Deletes All Data

Posted by Unknown on from the hackers-without-manners dept.

An anonymous reader writes Code Spaces [a code hosting service] has been under DDOS attacks since the beginning of the week, but a few hours ago, the attacker managed to delete all their hosted customer data and most of the backups. They have announced that they are shutting down business. From the announcement: An unauthorized person who at this point who is still unknown (All we can say is that we have no reason to think its anyone who is or was employed with Code Spaces) had gained access to our Amazon EC2 control panel and had left a number of messages for us to contact them using a Hotmail address. Reaching out to the address started a chain of events that revolved around the person trying to extort a large fee in order to resolve the DDOS.

At this point we took action to take control back of our panel by changing passwords, however the intruder had prepared for this and had already created a number of backup logins to the panel and upon seeing us make the attempted recovery of the account he proceeded to randomly delete artifacts from the panel.

15 of 387 comments (clear)

  1. The cloud by Anonymous Coward · · Score: 5, Insightful

    Good thing people hosted their stuff on the cloud...

    1. Re:The cloud by i+kan+reed · · Score: 5, Insightful

      But that would have cost the company a little more money.

    2. Re:The cloud by Penguinisto · · Score: 5, Interesting

      Good thing people hosted their stuff on the cloud...

      I don't think their problem is necessarily because it was "on the cloud" - the same thing could have happened if someone penetrated a corporate network and got hold of a VM farm. A bigger obstacle to be sure, but if your corporation has partner/vendor access and a not-so-sharp security guy...

      One question I have though - instead of changing a password, why couldn't they have called Amazon, had the thing universally locked out for that company, replaced all root-level access with a new account, and sent the new username and p/w by phone back to the company?

      Also, why didn't they have an offline (think: off-cloud) backup of the stuff? Sure it costs time/money/skull-sweat to do that, but it's worth the time and trouble in the end. After all, if your family jewels are hanging out there, it always pays to have a DR plan for 'em...

      If nothing else, they could have set up a separate and distinct AWS account/rigging as a "DR" of sorts, with DB replication and the works feeding it as a warm DR site. That way if some jackass compromises the first, you only need to stop DB replication, turn on the rest of the DR servers, do a quick test, and shift your DNS to the backup site - 15 mintues later, you can delete the objects yourself in the original site if you want (while you set up yet a different site and build a new backup site to replace the one you just put into production.)

      We have a sizable AWS setup where I work, and first/foremost we back that shit up (the DB contents) to machinery that we control. We also have a means of re-deploying/rebuilding if necessary; sure it takes time, but it's better to have it and not need it...

      --
      Quo usque tandem abutere, Nimbus, patientia nostra?
    3. Re:The cloud by vux984 · · Score: 5, Insightful

      I don't think their problem is necessarily because it was "on the cloud"

      No. The cloud was a key part of the problem. They had as much access and control over the system as the hacker did with no physical fall back.

      A VM farm on an onsite rack or even a colo rack? You knock out the hacker by unplugging it from the router to the internet, and then audit and reset security to your hearts content.

    4. Re:The cloud by Anonymous Coward · · Score: 5, Informative

      With Amazon's service you can contact them and have all access blocked until there is time to sort things out, and authenticate the real admin with billing information or the root SSH key you're given, etc.

    5. Re:The cloud by Munchr · · Score: 5, Insightful

      Exactly this. They state in the article that they had off-site backups. What use are off-site backups if the "on-site" control panel has direct online access to them? "In summary, most of our data, backups, machine configurations and offsite backups were either partially or completely deleted."

    6. Re: The cloud by seebs · · Score: 5, Insightful

      Good job shifting the goalposts, but that's pretty much totally unrelated. See, the lions are generally not considered to be moral actors. Humans usually are.

      --
      My blog: http://www.seebs.net/log/ --- My iPhone/iPad app: http://www.seebs.net/seebsfrac/
  2. Just unplug your server from the internet... by Anonymous Coward · · Score: 5, Funny

    So you just unplug your server's network connection from the internet while you fix the damage... oh. cloud stuff needs constant internet connection? hm. well I guess that's it then. It was an honor to serve with you. BOOM!

    1. Re: Just unplug your server from the internet... by Penguinisto · · Score: 5, Informative

      Who do you "call" with most cloud vendors? After all, sounds like whoever was doing the DDOS to extort Code Spaces could have also "called" Amazon to do any number of things, as whoever it was had the passwords, other accounts, etc..

      I've actually worked with them once - sure someone could impersonate them, but you could just as easily call up, explain the situation, and then prove you're the rightful owner of the account (using info that most script kiddies aren't going to think of gathering in the first place, let alone spoof the original contact phone #.)

      To their credit, Amazon is actually fairly intelligent and responsive, even to small accounts.

      BTW - if you use/handle it right, each instance comes pre-made with a specific SSH auth keyset for root, and you're the only one with the private key (even Amazon doesn't have it) - store/use that as your proof by logging into an instance with one (it's something the script kiddie definitely won't have).

      --
      Quo usque tandem abutere, Nimbus, patientia nostra?
  3. I can't think of a better argument... by Lab+Rat+Jason · · Score: 5, Insightful

    for air gapped backups.

    --
    Which has more power: the hammer, or the anvil?
    1. Re:I can't think of a better argument... by Russ1642 · · Score: 5, Insightful

      If your backups are sitting right next to your active files they aren't backups. They're just copies sitting there.

  4. Re:Backing up your cloud in your cloud... by gstoddart · · Score: 5, Funny

    Yo dawg, I hear you like clouds.

    --
    Lost at C:>. Found at C.
  5. Re:No offsite backups? by gstoddart · · Score: 5, Insightful

    No, because it was all in Amazon. Who needs tape when you have the cloud, right?

    So the stuff they had backed up from Amazon to Amazon, was still controlled by the same logins (or the ones the hacker had created).

    So when he/she/they started deleting stuff, the backups also got deleted.

    Sounds like a brilliant strategy, and an epic demonstration of what can go wrong with the cloud.

    If you host your own stuff, you do your own backups. If you backup your cloud data to the cloud using the same stuff as the rest of it ... well, your backups are hardly secure, are they.

    So unless Amazon has offsite tape backups (which I highly doubt) ... they're pretty much screwed.

    I think this is about the same as backing up your hard drive to itself so you have a spare copy.

    --
    Lost at C:>. Found at C.
  6. Not a Great Response by Edrick · · Score: 5, Insightful

    If you're a hosted site with important data and your site is compromised, the first & best move is to cut the cord immediately. Contact Amazon (or whomever is hosting the data) and get all access shut down instantly and immediately, thereby ending the attacker's ability to do anything further. This will cause an outage, but at least everything is safe.

    Working with Amazon, they can create a new account, give it a strong password, and begin cleaning up the mess with the new account (which the hacker will be unaware of). Now they can, at their own leisure, change passwords, administer accounts, delete crap created by the hacker, etc...Trying to outpace a professional hacker at their own game is a gamble that isn't worth it---especially if no offsite backups exist!!!

    Lastly, they should be forwarding all of the email/attacker info to Amazon, Microsoft (Hotmail), and to the authorities. Whether they can be caught or not is up in the air, but odds are almost certain that this attacker has hit other sites and would eventually have different cases correlated to each other.

    Safety & security of data is #1, fixing damage caused is #2, and accountability is #3. Securing the site against future attacks is part of #3---there's no reason to put the site up (or leave it up) and risk further attacks, thereby risking data loss or a security breach.

  7. Git by blackiner · · Score: 5, Interesting

    This is why git is such an effective code hosting solution. Everyone who has cloned the repository is a potential backup copy.