Slashdot Mirror
Code Spaces Hosting Shutting Down After Attacker Deletes All Data
An anonymous reader writes Code Spaces [a code hosting service] has been under DDOS attacks since the beginning of the week, but a few hours ago, the attacker managed to delete all their hosted customer data and most of the backups. They have announced that they are shutting down business.
From the announcement: An unauthorized person who at this point who is still unknown (All we can say is that we have no reason to think its anyone who is or was employed with Code Spaces) had gained access to our Amazon EC2 control panel and had left a number of messages for us to contact them using a Hotmail address. Reaching out to the address started a chain of events that revolved around the person trying to extort a large fee in order to resolve the DDOS.
At this point we took action to take control back of our panel by changing passwords, however the intruder had prepared for this and had already created a number of backup logins to the panel and upon seeing us make the attempted recovery of the account he proceeded to randomly delete artifacts from the panel.
At this point we took action to take control back of our panel by changing passwords, however the intruder had prepared for this and had already created a number of backup logins to the panel and upon seeing us make the attempted recovery of the account he proceeded to randomly delete artifacts from the panel.
Good thing people hosted their stuff on the cloud...
...doesn't seem to work so well.
Koans and fables for the software engineer
So you just unplug your server's network connection from the internet while you fix the damage... oh. cloud stuff needs constant internet connection? hm. well I guess that's it then. It was an honor to serve with you. BOOM!
for air gapped backups.
Which has more power: the hammer, or the anvil?
would you mind going into ebay.com & deleting my account?
Ebay refuses to close it.
At least they had backups of their cloud data in a safe place where no random asshat could just go in and waste the data. That is a code hosting company you can trust with your stuff that is for sure!
ACK
A back-up that can be deleted so easily is no back-up at all.
Definitely a strong reminder to have at least some off cloud presence, unreal to think a hacker could ruin your business by stealing your thunder (cloud).
They didn't have offline backups? tapes? I'm not familiar with codespaces service, but how come the backups could be deleted remotely?
"Luck is my middle name," said Rincewind, indistinctly. "Mind you, my first name is Bad." -- Terry Pratchett
ah a hotmail account
Like MS will not give up the IPs that accessed that account...
Someone is going to jail...
I still don't get the logic of running your business in the cloud as a company. Sure it makes sense when your small and it gives you global presence with no investment, but people don't realize how much security / freedom they give up doing this. and EC2 ensures it will cost a lot to move out of their space. Good luck with the founders of code space. Glad I didn't put my repo there.
yes, because using a proxy to access hotmail is the most difficult thing ever...
The guys behind Code Spaces should be issued a citation for Operating While Pwned. If you know admin access is compromised, shut it down out-of-band.
I assume that this is probably becoming a relatively common practice, but, to me, if it is not reliably written on offline physical media of which I have control, it is not a backup.
Presumably when they realized that the attacker had access to their control panel they shoulda coulda (yes I know I hate that too) called Amazon and had them shut everything down until order could be restored.
Nullius in verba
Yeah, that is what I was wondering. Did the hosting company have offline backups? Too expensive to implement? Too time consuming to copy gigabytes of data to an off-line storage disk? Just asking.
I must be a cynic but my first reaction is to think:
1 - Create cloud based system.
2 - Sell subscriptions for hundreds of $.
3 - Announce hacker attack!
4 - Profit.
If you're a hosted site with important data and your site is compromised, the first & best move is to cut the cord immediately. Contact Amazon (or whomever is hosting the data) and get all access shut down instantly and immediately, thereby ending the attacker's ability to do anything further. This will cause an outage, but at least everything is safe.
Working with Amazon, they can create a new account, give it a strong password, and begin cleaning up the mess with the new account (which the hacker will be unaware of). Now they can, at their own leisure, change passwords, administer accounts, delete crap created by the hacker, etc...Trying to outpace a professional hacker at their own game is a gamble that isn't worth it---especially if no offsite backups exist!!!
Lastly, they should be forwarding all of the email/attacker info to Amazon, Microsoft (Hotmail), and to the authorities. Whether they can be caught or not is up in the air, but odds are almost certain that this attacker has hit other sites and would eventually have different cases correlated to each other.
Safety & security of data is #1, fixing damage caused is #2, and accountability is #3. Securing the site against future attacks is part of #3---there's no reason to put the site up (or leave it up) and risk further attacks, thereby risking data loss or a security breach.
If someone has penetrated your system so that they have root or admin privileges over all your machine, you shut down immediately. In the physical world, you pull the plug. On Amazon, you immediately tell Amazon to lock things down, disable all passwords and administrative control, and then work back up to fixing things.
Lord Kril at Rylos.
the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff
I arrived on the Internet in 1994 and took a part in developing a couple well-known web sites. I loved the idea of a network empowering individuals to both control and share their data rather than relying on some big mainframe company. "Peer-to-peer" wasn't just a technical achievement, but a social achievement.
As "the cloud" emerged, I decided to pack my shit up and move to another career. I don't even begin to understand how billions of dollars are invested into this retrograde leap.
Normally things form clouds AFTER going up in smoke. With the 'new technology' it is the opposite.
Not providing for your own OFFLINE BACKUPS is a reckless oversight of such magnitude that I am entirely incapable of having sympathy for these asshats. We need a few examples such as these to serve as cautionary tales for those who think the Cloud is the answer to everything.
and our admin password is "letmein"
if this is supposed to be a new economy, how come they still want my old fashioned money?
This is why git is such an effective code hosting solution. Everyone who has cloned the repository is a potential backup copy.
Someone else mentioned having offline backups, so I won't belabor that. But once they knew they were compromised, perhaps a smarter thing to do would have been to contact the service provider and take countermeasures (ask for a snapshot of the site as it was, examine and disable accounts, change admin passwords, perhaps contact authorities) before reaching out to the perp. I'm not sure reaching out to the perp was a good idea in any case.
For awhile I hosted a number of websites from a rental space, and I did get compromised once. (security hole in a popular web admin tool) As soon as I detected it, I drove to the physical site, unplugged the server from the internet, and worked from the console. It occurs to me that this might be a difficult strategy to implement with cloud services.
Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
So these guys apparently had no training on proper backup policies and procedures.
This is definitely a training issue. Clearly no one taught them how to do proper backups or even what a proper backup policy should look like.
I feel bad for them, but at the point that they have done nothing to protect themselves I cannot bring myself to feel too bad.
Why does no one take their backups offsite anymore or backup to a NAS device that backs itself up to something that can be taken offsite?
Backups Backups BACKUPS!!!
looks like someone got fired and was pissed!
i bet they didn't read http://docs.aws.amazon.com/IAM/latest/UserGuide/IAMBestPractices.html when creating their accounts, so pissed ppl still had access ....
Now things have "gone away" .... go figure ....
ofc, this may or may not have been the problem.
http://docs.aws.amazon.com/IAM/latest/UserGuide/IAMBestPractices.html#lock-away-credentials
Why didn't they first contact Amazon to lock everything down and reset all the passwords?
Also, STOP USING THE SAME PASSWORD FOR EVERY WEBSITE
Seriously.... no offline backups? Not a real business in that case.
Instead of trying to take back control themselves, shouldn't they have contacted Amazon and let them handle it? Perhaps they could have frozen the entire account, locking out both the rightful owner and the attacker, until things were sorted.
This is a bummer, man.
lose != loose
Is there additional evidence that this is what happened or is the only narrative the one on Code Space's homepage?
This is why distributed version control is important (git/mercurial), even if you think SVN is easier. Sometimes your remote server will disappear, whether its hackers, fires, or someone forgot to pay the bill.
What other services can former customers go to for SVN and issue tracking services?
Any business infrastructure that has data of any value of the cloud is fundamentally flawed.
1) data can be tampered with (NSA, blackhats, competitors)
2) data can be lost
3) data can be stolen and/or misused.
4) access can be denied.
5) cloud company may deny access to data for either their own purposes or governmental purposes.
The first rule of security is physical security, if you don't control physical security you don't *have* security.
Period.
This must be where the IRS stored backups of emails.
The only sensible thing to do would of course be to pull the plug and have Amazon reset admin controls for them. I think they messed up and are trying to cover up with the good old hacker excuse.
What are better services which former CodeSpaces users can go to?
My password is "invalid" so when I type it wrong I get a message: "Your password is invalid."
Our offsite backups are put in a metal box and taken offsite. Unless you plan on hijacking a truck, it's a lot harder deleting our data than using a nice control panel on the web.
Support microSD: in a post 9/11 world, it is unwise to carry your data on media that you cannot comfortably swallow.
So, after getting blackmail email, first course of action was to take matters into your own hands? The Cloud is just a tool to allow us to be tracked online. I'm going back to dial up and UUCP to select individuals in similar configuration. He he he
IMHO:
1) Backups that don't get done automatically often don't get done regularly, so they should be automatically performed via scripts.
2) Offline isn't as important as offsite. Buildings catch fire, get flooded, disappear into sink holes, get hit by falling jet airplanes.
3) Security matters. Paranoia should be the order of the day.
I have no problem with your religion until you decide it's reason to deprive others of the truth.
....oh never mind.
Nothing copied elsewhere or onto tape? - Guess not. The cloud is SOOOO secure...
"For every complex problem, there is a solution that is simple, neat, and wrong." -- H.L. Mencken (1880-1956) --
this changes everything, well atleast my default password
Lord Kril at Rylos. [imdb.com]
Amazon, when the firewalls fell.
My password is "invalid" so when I type it wrong I get a message: "Your password is invalid."
And it shows you your password in plain text? Well, thats not very secure. I'd suggest at least changing your password to ******* so that its obscured.
They and their customers got exactly what they deserved. Backups connected to the internet? How stupid can you be?? What a bunch of dumbasses!
1950s - IBM, Bell, Big gov't = Centralized, government and institutional.
1980s - MIT, Apple, HTTP, Internet, WWW = Decentralized, knowledge everywhere. Hacker = one who creates
2010s - Google, Amazon = Centralized "cloud". Hacker = one who destroys
Is it me, or are we reverting?
Would the correct response not have been to contact Amazon and have them immediate suspend all access and reset the passwords for them?
Out of curiosity, lets say you find yourself in the same position where you have a hacker/hackers with multiple accounts, and you want to change passwords, etc. How would you lock the system down so they could not do damage in this case? Is there a way to quickly purge all unknown users ? Could they have spoofed known good users? ..Is it possible to blow everyone else away except for the administrator, and reference an older archive of users? I'm very curious about how you could safely contain such of contagion.
Also, lets say you do have an off-line back-up, but you have a situation where a hacker has access to the usernames and passwords because they somehow got root access. How do you protect all their data once you decide to turn back on-line? Do you send out notice to all your users over their email accounts?
I'm curious about how admins deal with this in the real world.
This article and it's comments give me an idea for a story for my Shadowrun game. Thanks.
Trolls attack the City! The police blame the population for living in a city in the Trolls' war path.
They just changed the root AWS account password. They apparently didn't bother to check the IAM section where other user accounts can be created. They would have been clearly displayed there and could have been deleted.
The whole operation was probably ran by one guy. They could have purchased a DDoS protection service and changed their DNS record(s). Done. They were just in over their head.
If you never heard of the many cases for tape backup, here it is.
Sorry, but a backup is something you keep *off line* for a good reason. This is a near-line copy, possibly at another site we're talking about here.
I was promised a flying car. Where is my flying car?
Nobody is saying they were legally responsible to prevent crime.
People *are* saying that they were poor businessmen who didn't plan for disasters. (What if the cloud provider failed catastrophically, or they lost all the passwords, or any number of other catastrophic events?)
With managed hosting, the provider handles support, backup/restore, etc. Typically with "the cloud" the resources are unmanaged. The end-user is responsible for all of that stuff.
I don't believe Amazon themselves offer managed services, but there are lots of other companies that will sell you managed services built on AWS.
Dammit. Time to update my password to hunter3.
-=This sig has nothing to do with my comment. Move along now=-
Hahahahahaha--[gasp, gasp]--Bahahahahaha!
Enjoy it, dumb asses.
Serves them right for being such silly dicks.
I believe the owner of the EC2 had 1 single account (root account) when he should have setup 2 factor authentication for such an account and then created separate accounts, this would have prevented his issue using the security policies AWS has in place.... :)
AWS is always targeted and being reliant on a single account for security is negligent.
So you people out there that use AWS, PLEASE don't use the default account, secure it with 2 factor and then create individual accounts for the services, using security policies to allow communication between each other. - from an AWS certified engineer
It's not a typo if you understood the meaning!
Bit of a shame code spaces weren't geared up with som AWS HA configuration hoping companies take note of this attack and how to limit the risk to their organisations. A simple AWS cli command could have been implemented here to force all IAM accounts with only read only access until AWS could be involved. Also master and all IAM accounts should have 2FA enabled to stop this happening.
Also, lets say you do have an off-line back-up, but you have a situation where a hacker has access to the usernames and passwords because they somehow got root access. How do you protect all their data once you decide to turn back on-line? Do you send out notice to all your users over their email accounts?
I'm curious about how admins deal with this in the real world.
If a hacker can recover plaintext passwords by compromising your admin account you have failed as an admin. The most they should be able to recover is a (hopefully salted) password hash.
Enigma
Why didn't they call amazon so amazon could shutdown their servers and help them gain access and change passwords? They should have realized they could not win the war and realize the data was in serious danger. They gave the attackers too much time while they were trying to get it back...
"Never attribute to malice that which is adequately explained by stupidity."
If you want secure AWS clusters, you DO NOT rely upon passwords - you require the use of multi-factor authentication as well. Without the appropriately keyed MFA device (physical or virtual), your password alone will not get someone into the system. I managed multiple EC2 clouds for several years and this is something I am very familiar with. An MFA device used in this way is very hard to duplicate. Yes, a virtual MFA device (usually a cell-phone application) can be pwnd if the user is careless and gets a serious infection, but it isn't simple to do.
Code Spaces ended their explanatory comment with, "We hope that one day we will be able to and reinstate the service and credibility that Code Spaces once had!" Don't these people understand what they "once had" was an illusion? They had best NOT get back to the disastrous what they "once had". What they had better do is create something better and more secure. But given their stated goal is to get back to where they were, it seems to me they are utterly and totally lost. Who will trust the again ... ever?
This reminds me the cloud service of Microsoft, called Danger. It died the same way - they simply lost all customer information, with no backups made (and, actually, the size of full backup could be less than 1Tb).
In my humble opinion, these people are too lame to stay in business. Having offline backups couldn't be the only problem of their service. As I can conclude from the article, they also had problems with security and lack of common sense and strategic thinking.
https://en.wikipedia.org/wiki/...
That makes one wonder, why these gits did not call AWS support to have their account completely locked down first?
The summary fails to mention the initial breach was due to the attacker being able to login with an administrator password. That, and apparently their hosting service does not allow for restoring from an image.
Yes, you do have to account for yourself, but the exception is honest naivety.
It is everyone's responsibility, including the victim's, to ensure their own safety in this dangerous world, because ultimately we're all alone here. If you haven't yet realized this very real truth then now would be a great time. If you have a power in this world you also have a responsibility to use it rightly. If you have the power to predict a bad scenario then it is your responsibility to do your best to ensure it never comes about.
This is not a social law. This has nothing to do with society. This is selective pressure coming from the deformed animal within humanity that does indeed make a strong appearance in today's sick society. There is a great separation of those who have too much and those who have too little. Look into Maslow's Hierarchy of Needs if you want some sort of psychological understanding. We act out when we lack!
We are independent individuals: depending on others is a weakness as it places you at their mercy.Not actively protecting yourself when you are aware of being in a dangerous environment is called neglect, and in the end it doesn't matter who's fault it is - you were exposed, attacked and defeated because you live in a fairy tale idealistic illusion that left you a target.
If you were never taught this you can blame your parents for your naievity for they are without a doubt ultimately responsible for your future.
There is so much discussion about who's to blame and none about solving the problem!
Today's world is supported by the 80's generation. What happened to you, 80's generation?
Where is your wisdom and correctness? You justness with righteous reality?
Are you so socialized and drugged that you've forgotten the truth?
I think there's little point in addressing the Slashdot crowd with this question - they seem to be a better, more finely informed group than the greater populace.
Software as a Disservice
Everything is monitored. EVERYTHING. The only way that he will not get caught is if "they" (you know, them!) don't want him to get caught.
How? If they were not off-line, they really were not backups.
---- Booth was a patriot ----