Slashdot Mirror
Bots Scanning GitHub To Steal Amazon EC2 Keys
New submitter juniq writes: As one developer found out, posting your Amazon keys to GitHub on accident can be a costly mistake if they are not revoked immediately.
"When I woke up the next morning, I had four emails from Amazon AWS and a missed phone call from Amazon AWS. Something about 140 servers running on my AWS account. What? How? I only had S3 keys on my GitHub and they where gone within 5 minutes! Turns out through the S3 API you can actually spin up EC2 instances, and my key had been spotted by a bot that continually searches GitHub for API keys. Amazon AWS customer support informed me this happens a lot recently; hackers have created an algorithm that searches GitHub 24 hours per day for API keys. Once it finds one it spins up max instances of EC2 servers to farm itself bitcoins."
"When I woke up the next morning, I had four emails from Amazon AWS and a missed phone call from Amazon AWS. Something about 140 servers running on my AWS account. What? How? I only had S3 keys on my GitHub and they where gone within 5 minutes! Turns out through the S3 API you can actually spin up EC2 instances, and my key had been spotted by a bot that continually searches GitHub for API keys. Amazon AWS customer support informed me this happens a lot recently; hackers have created an algorithm that searches GitHub 24 hours per day for API keys. Once it finds one it spins up max instances of EC2 servers to farm itself bitcoins."
Oh right: http://developers.slashdot.org...
They could have achieved so much more by being less greedy.
People....
Guy stores his password online, is surprised when he gets got.
I recall running across someone who had discovered several thousands of PHP files that contained commands like exec($_POST["randomgibberish"]) that was clearly a backdoor exploit that had gotten into the source and then pushed to git for whoever to share.
AWS strongly discourages the uses of root API keys, as they give bad guys who find them the "keys to the kingdom". Why should the credentials for one's S3 account also work for creating EC2 instances?
Amazon provides extensive control over access credentials through IAM, so one can create (for example) an S3-specific user with limited privileges and generate API keys for that user. If they get compromised, the bad guy has limited access: they might be able to add new files to S3, which is bad, but it's less bad than them spinning up hundreds of servers for nefarious purposes, deleting all your files, etc.
Judicious user of IAM can also reduce user errors: I use Amazon Glacier for backing up certain critical files (e.g. wedding photos, baby photos, copies of wills, passports, etc.). I created an "upload, view, and restore/download" user for Glacier that explicitly does not have the "delete" permission enabled. I have a second IAM user with "view and delete" permissions. API keys for both users are stored in FastGlacier, with the "delete" user credentials stored encrypted so I need to enter a password to switch to that user. The user without delete permissions is the default user and the credentials are not stored with a password. This way I can do the standard backup/restore functions needed while working with backups but significantly reduce the possibility of my accidentally deleting backed-up files if I fat-finger the wrong key.
HACKERS, I TELL YOU
But the user is still mostly to blame. Okay, so you might not find it intuitive that S3 keys can lead to new EC2 instances - I wouldn't have guessed that either, even though I've used both repeatedly. Maybe that shouldn't even be a possibility. But what howling insanity persuaded you to put those keys on github in the first fucking place? And if it was a mistake, why didn't you change them after? This isn't amateur hour, guys - there's real money at stake here.
...nor are algorithms usually created. Algorithms are discovered, devised, or designed. Software is created. Bots are created. Algorithms exist independent of their discovery.
Anyway, I'm getting sick of hearing the word "algorithms" used as it seems to be in the movies a lot lately.
I guess i am too old to understand how loose people treat the internet these days. 'I posted my credentials openly on the internet and am now shocked that I have been taken advatage of'... no way! You shared the keys to your kingdom and someone abused it?? Shocking.
As a complete side note: I hate when people like the author don't know the difference between 'where' and 'were'....fuck, no wonder he was easy fodder
Dumbass.
Their algorithm will crash on March 8, a 23-hour day.
Amazon shouldn't have refunded him any money. We really need idiot taxes.
Why the f*ck would you post your S3 keys on github anyway? I sincerely hope you are not looking for sympathy because you won't find any here (nor do you deserve any).
God-damn "social media" generation that feels they need to share everything. Good riddance.
When you sign up for a developer account, you should be asked how much you plan to spend per month. $2375/day would not be a common option for an individual. Given proliferation of free 15GB storage accounts, a very low end developer account with no credit card is not a crazy option. People will learn the API and use it in future, but neither them nor hackers will have enough quota to run a production site. This is just like limited data cell plans where a single buggy app can run up crazy charges. Good that they refunded money, but fundamental structural problem must be fixed.
Really, leaving your money on a park-bench somewhere exposes it to be stolen...
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
You almost spelled Rivest right:
https://en.wikipedia.org/wiki/...
Keep on the good work!
Everything I write is lies, read between the lines.
People mess up. An API key will occasionally get into a file that's in a source tree. It is an API - an Application Programming Interface, so programs will have that key. Program source gets posted to github. Shit happens.
My bank knows shit happens and I could get my card stolen, so when I make a large purchase which appears unusual my cell phone rings - the bank's security company calling to verify the purchase. That's the bank doing a good job.
People should watch where they're walking. That doesn't mean you leave an open manhole in the middle of the sidewalk unmarked - someone is likely to mess up and fall in.
dey be haxxin amazon2
Oh, wait... you're American...
WTF?
"posting your Amazon keys to GitHub ON accident"
Why do Americans have such trouble with two letter words? Idiots.
Surely it is not that unreasonable to (1) realize that those keys will be scraped within 5 minutes after uploading to an obscure project, and (2) not realize that an S3 key in a free trial subscription wouldn't allow racking up $2375 in EC charges within 10 hours?
Avantslash: low-bandwidth mobile slashdot.
Amazon issues several fabricated keys, posts them to GitHub. Once they are used, they ban the IP and inform the police.
put the real key in a single file and include that file in any file that needs access to said key, and then include the file with the real key in .gitignore so it doesn't accidentally get uploaded?
It's not rocket science.
And my bank is going to get pretty pissed at me if they find out I published my bank details on a public access website and then ask them to fix it.
I'm wondering if Amazon couldn't post fake keys and pretend to spin up fake EC2 instances? Delay, delay, delay ... increase the cost of having a "hit"?
Would it be perhaps useful to now have a bunch of bots to commit bogus AWS keys to github repos, so the harvesting bots have chaff to weed through as well as accidental wheat?
Can somebody answer please? I am serious.
He did exactly that and .gitignore was accidentally lost.
Service providers are responsible for 90% of work by designing solutions secure by default and in depth. It should take multiple explicit steps and dismissed warnings for an inexperienced developer to incur a catastrophic financial liability.
That was too funny not to search for; it obviously got taken down from github, but someone posted the (very NSFW) .zsh_history on a paste site when the story broke in 2012. If you search for lines starting with "mplayer", you can see how they're clustered into several obvious jerkoff sessions... ROTFL! The file names were so sick they made my eyes pop open.. I had to push them back into their sockets!
This wins the prize for the worst commit I've ever seen. There's a lesson for pedophiles here: echo ".zsh_history" >> .gitignore
This type of problem has been reported many times before, with much more knowledgable writeups.
The original poster is so naive that they didn't even bother to read enough Amazon documentation to realize there is no such thing as an "S3 key" - API access by AWS keys is limited only by the IAM profile of the key (and my guess is that the OPs keys were unrestricted). They also apparently didn't realize how version control systems work, otherwise they would have known that deleting the key from a revision doesn't actually remove it from the history of the repository.
This article isn't doing anyone any favors - if you want to actually help the community then maybe source an original article reminding people that they should read the docs and understand the services they're using, with pointers to the relevant warnings for commonly used services (both github and amazon have prominent notices with service-relevant notes about how to protect your sensitive data).
The mistake he made was not understanding the tools he was using. Apparently neither do you.
(1) The key could have been scraped at any time once it was pushed, because you can't actually "delete[d] all traces from GitHub" (some ways are more thorough than others, but nothing is foolproof with Google wandering the earth). He needed to revoke his keys immediately.
(2) There is no such thing as an "S3 key". There are only AWS API keys, which potentially have access to every service that you have enabled (plus the default ones). You need to use IAM profiles to restrict what services they can access, and what rights they have.
Signing up for a service and then using it without reading the documentation is foolish.
In addition to the various other oversights already mentioned, OP doesn't seem to understand Git (or perhaps SCMs in general) given that those (now revoked) keys are still on GitHub -- there was no need for a bot to be all that quick.
Although I wouldn't blame OP for any single one of these oversights -- nobody's perfect -- it's fair to say that it took a number of different oversights / misunderstandings on OP's part for this to become a real problem.
WFT would anyone want to post keys to anything anywhere public?