Slashdot Mirror
Code Spaces Hosting Shutting Down After Attacker Deletes All Data
An anonymous reader writes Code Spaces [a code hosting service] has been under DDOS attacks since the beginning of the week, but a few hours ago, the attacker managed to delete all their hosted customer data and most of the backups. They have announced that they are shutting down business.
From the announcement: An unauthorized person who at this point who is still unknown (All we can say is that we have no reason to think its anyone who is or was employed with Code Spaces) had gained access to our Amazon EC2 control panel and had left a number of messages for us to contact them using a Hotmail address. Reaching out to the address started a chain of events that revolved around the person trying to extort a large fee in order to resolve the DDOS.
At this point we took action to take control back of our panel by changing passwords, however the intruder had prepared for this and had already created a number of backup logins to the panel and upon seeing us make the attempted recovery of the account he proceeded to randomly delete artifacts from the panel.
At this point we took action to take control back of our panel by changing passwords, however the intruder had prepared for this and had already created a number of backup logins to the panel and upon seeing us make the attempted recovery of the account he proceeded to randomly delete artifacts from the panel.
Good thing people hosted their stuff on the cloud...
...doesn't seem to work so well.
Koans and fables for the software engineer
So you just unplug your server's network connection from the internet while you fix the damage... oh. cloud stuff needs constant internet connection? hm. well I guess that's it then. It was an honor to serve with you. BOOM!
for air gapped backups.
Which has more power: the hammer, or the anvil?
would you mind going into ebay.com & deleting my account?
Ebay refuses to close it.
At least they had backups of their cloud data in a safe place where no random asshat could just go in and waste the data. That is a code hosting company you can trust with your stuff that is for sure!
ACK
They didn't have offline backups? tapes? I'm not familiar with codespaces service, but how come the backups could be deleted remotely?
"Luck is my middle name," said Rincewind, indistinctly. "Mind you, my first name is Bad." -- Terry Pratchett
That jail must be very crowded with all the nigerian scammers and fake craigslist landlords who use hotmail to scam people.
lucm, indeed.
The guys behind Code Spaces should be issued a citation for Operating While Pwned. If you know admin access is compromised, shut it down out-of-band.
Any self respecting bassmint dweller would not have used their home network to do this.
Presumably when they realized that the attacker had access to their control panel they shoulda coulda (yes I know I hate that too) called Amazon and had them shut everything down until order could be restored.
Nullius in verba
I must be a cynic but my first reaction is to think:
1 - Create cloud based system.
2 - Sell subscriptions for hundreds of $.
3 - Announce hacker attack!
4 - Profit.
If you're a hosted site with important data and your site is compromised, the first & best move is to cut the cord immediately. Contact Amazon (or whomever is hosting the data) and get all access shut down instantly and immediately, thereby ending the attacker's ability to do anything further. This will cause an outage, but at least everything is safe.
Working with Amazon, they can create a new account, give it a strong password, and begin cleaning up the mess with the new account (which the hacker will be unaware of). Now they can, at their own leisure, change passwords, administer accounts, delete crap created by the hacker, etc...Trying to outpace a professional hacker at their own game is a gamble that isn't worth it---especially if no offsite backups exist!!!
Lastly, they should be forwarding all of the email/attacker info to Amazon, Microsoft (Hotmail), and to the authorities. Whether they can be caught or not is up in the air, but odds are almost certain that this attacker has hit other sites and would eventually have different cases correlated to each other.
Safety & security of data is #1, fixing damage caused is #2, and accountability is #3. Securing the site against future attacks is part of #3---there's no reason to put the site up (or leave it up) and risk further attacks, thereby risking data loss or a security breach.
If someone has penetrated your system so that they have root or admin privileges over all your machine, you shut down immediately. In the physical world, you pull the plug. On Amazon, you immediately tell Amazon to lock things down, disable all passwords and administrative control, and then work back up to fixing things.
The trade-offs can be really good even for a large company. It has to be done right though and many companies don't even do their local IT properly.
Lord Kril at Rylos.
the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff
Normally things form clouds AFTER going up in smoke. With the 'new technology' it is the opposite.
Not providing for your own OFFLINE BACKUPS is a reckless oversight of such magnitude that I am entirely incapable of having sympathy for these asshats. We need a few examples such as these to serve as cautionary tales for those who think the Cloud is the answer to everything.
and our admin password is "letmein"
if this is supposed to be a new economy, how come they still want my old fashioned money?
This is why git is such an effective code hosting solution. Everyone who has cloned the repository is a potential backup copy.
Any self respecting bassmint dweller would not have used their home network to do this.
Basement dwellers don't leave the basement.
Basement dwellers are self-loathing, not self-respecting.
Basement dwellers use their own network to connect to proxies, which just makes it more of a pain in the ass to trace back.
Extreme basement dwellers will use other means of accessing a separate network - a cantenna pointed at a neighbors house, a spliced line, whatever. This just means the cops track down the victim, figure out they're not computer literate, and ask "Any people who could have done this?" and learn about the freak in the neighbor's basement.
Hackers don't get caught because law enforcement doesn't care.
When the cops, the government, or a corporation cares, hackers get caught or disappeared.
Someone else mentioned having offline backups, so I won't belabor that. But once they knew they were compromised, perhaps a smarter thing to do would have been to contact the service provider and take countermeasures (ask for a snapshot of the site as it was, examine and disable accounts, change admin passwords, perhaps contact authorities) before reaching out to the perp. I'm not sure reaching out to the perp was a good idea in any case.
For awhile I hosted a number of websites from a rental space, and I did get compromised once. (security hole in a popular web admin tool) As soon as I detected it, I drove to the physical site, unplugged the server from the internet, and worked from the console. It occurs to me that this might be a difficult strategy to implement with cloud services.
Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
So these guys apparently had no training on proper backup policies and procedures.
This is definitely a training issue. Clearly no one taught them how to do proper backups or even what a proper backup policy should look like.
I feel bad for them, but at the point that they have done nothing to protect themselves I cannot bring myself to feel too bad.
Why does no one take their backups offsite anymore or backup to a NAS device that backs itself up to something that can be taken offsite?
Backups Backups BACKUPS!!!
Seriously.... no offline backups? Not a real business in that case.
Instead of trying to take back control themselves, shouldn't they have contacted Amazon and let them handle it? Perhaps they could have frozen the entire account, locking out both the rightful owner and the attacker, until things were sorted.
This is a bummer, man.
lose != loose
This is why distributed version control is important (git/mercurial), even if you think SVN is easier. Sometimes your remote server will disappear, whether its hackers, fires, or someone forgot to pay the bill.
This must be where the IRS stored backups of emails.
My password is "invalid" so when I type it wrong I get a message: "Your password is invalid."
Our offsite backups are put in a metal box and taken offsite. Unless you plan on hijacking a truck, it's a lot harder deleting our data than using a nice control panel on the web.
Support microSD: in a post 9/11 world, it is unwise to carry your data on media that you cannot comfortably swallow.
So, after getting blackmail email, first course of action was to take matters into your own hands? The Cloud is just a tool to allow us to be tracked online. I'm going back to dial up and UUCP to select individuals in similar configuration. He he he
IMHO:
1) Backups that don't get done automatically often don't get done regularly, so they should be automatically performed via scripts.
2) Offline isn't as important as offsite. Buildings catch fire, get flooded, disappear into sink holes, get hit by falling jet airplanes.
3) Security matters. Paranoia should be the order of the day.
I have no problem with your religion until you decide it's reason to deprive others of the truth.
....oh never mind.
Nothing copied elsewhere or onto tape? - Guess not. The cloud is SOOOO secure...
"For every complex problem, there is a solution that is simple, neat, and wrong." -- H.L. Mencken (1880-1956) --
this changes everything, well atleast my default password
Would the correct response not have been to contact Amazon and have them immediate suspend all access and reset the passwords for them?
You forgot to put Apple in the centralised category too.
HTTP, Internet and WWW didn't exist in the 80s either.
Out of curiosity, lets say you find yourself in the same position where you have a hacker/hackers with multiple accounts, and you want to change passwords, etc. How would you lock the system down so they could not do damage in this case? Is there a way to quickly purge all unknown users ? Could they have spoofed known good users? ..Is it possible to blow everyone else away except for the administrator, and reference an older archive of users? I'm very curious about how you could safely contain such of contagion.
Also, lets say you do have an off-line back-up, but you have a situation where a hacker has access to the usernames and passwords because they somehow got root access. How do you protect all their data once you decide to turn back on-line? Do you send out notice to all your users over their email accounts?
I'm curious about how admins deal with this in the real world.
If you never heard of the many cases for tape backup, here it is.
Sorry, but a backup is something you keep *off line* for a good reason. This is a near-line copy, possibly at another site we're talking about here.
I was promised a flying car. Where is my flying car?
The Internet existed in the 80s, and was descended from a 70s project. HTTP and the WWW came later.
"When you have eliminated the unacceptable, whatever is left, however improbable, must be the truthiness" - Holmes
Nobody is saying they were legally responsible to prevent crime.
People *are* saying that they were poor businessmen who didn't plan for disasters. (What if the cloud provider failed catastrophically, or they lost all the passwords, or any number of other catastrophic events?)
With managed hosting, the provider handles support, backup/restore, etc. Typically with "the cloud" the resources are unmanaged. The end-user is responsible for all of that stuff.
I don't believe Amazon themselves offer managed services, but there are lots of other companies that will sell you managed services built on AWS.
Dammit. Time to update my password to hunter3.
-=This sig has nothing to do with my comment. Move along now=-
It wasn't anything like what it is today until 1989 when BGP was introduced and the last of the centralised routing was removed when ARPANET was decommissioned in the early 90's
The term "internet" was short for "inter-networking" and described any situation where two or more networks were connected.
I believe the owner of the EC2 had 1 single account (root account) when he should have setup 2 factor authentication for such an account and then created separate accounts, this would have prevented his issue using the security policies AWS has in place.... :)
AWS is always targeted and being reliant on a single account for security is negligent.
So you people out there that use AWS, PLEASE don't use the default account, secure it with 2 factor and then create individual accounts for the services, using security policies to allow communication between each other. - from an AWS certified engineer
It's not a typo if you understood the meaning!
Bit of a shame code spaces weren't geared up with som AWS HA configuration hoping companies take note of this attack and how to limit the risk to their organisations. A simple AWS cli command could have been implemented here to force all IAM accounts with only read only access until AWS could be involved. Also master and all IAM accounts should have 2FA enabled to stop this happening.
Also, lets say you do have an off-line back-up, but you have a situation where a hacker has access to the usernames and passwords because they somehow got root access. How do you protect all their data once you decide to turn back on-line? Do you send out notice to all your users over their email accounts?
I'm curious about how admins deal with this in the real world.
If a hacker can recover plaintext passwords by compromising your admin account you have failed as an admin. The most they should be able to recover is a (hopefully salted) password hash.
Enigma
"Never attribute to malice that which is adequately explained by stupidity."
This reminds me the cloud service of Microsoft, called Danger. It died the same way - they simply lost all customer information, with no backups made (and, actually, the size of full backup could be less than 1Tb).
In my humble opinion, these people are too lame to stay in business. Having offline backups couldn't be the only problem of their service. As I can conclude from the article, they also had problems with security and lack of common sense and strategic thinking.
https://en.wikipedia.org/wiki/...
That makes one wonder, why these gits did not call AWS support to have their account completely locked down first?
Everything is monitored. EVERYTHING. The only way that he will not get caught is if "they" (you know, them!) don't want him to get caught.
I don't know what kind of magic sauce would allow one to have "IT in the cloud" setup. Windows clients with roaming profiles quickly get to be a drain even on a gigabit network. Even without a roaming profile, anything that isn't the boring old secretarial style work will require a decent bandwidth. Most media work or CAD work can't really be done over your typical cable internet. Those who would most benefit from an "IT in the cloud" type of a service - small businesses - really can't afford having gigabit links to their premises. Neither do I think that the bandwidth from any particular Amazon instance is where it needs to be. Does Amazon run their instances on machines/blades with 10Gbit links?
A successful API design takes a mixture of software design and pedagogy.
How? If they were not off-line, they really were not backups.
---- Booth was a patriot ----