Slashdot Mirror
Over 100,000 GitHub Repos Have Leaked API or Cryptographic Keys (zdnet.com)
A scan of billions of files from 13 percent of all GitHub public repositories over a period of six months has revealed that over 100,000 repos have leaked API tokens and cryptographic keys, with thousands of new repositories leaking new secrets on a daily basis. From a report: The scan was the object of academic research carried out by a team from the North Carolina State University (NCSU), and the study's results have been shared with GitHub, which acted on the findings to accelerate its work on a new security feature called Token Scanning, currently in beta. The NCSU study is the most comprehensive and in-depth GitHub scan to date and exceeds any previous research of its kind. NCSU academics scanned GitHub accounts for a period of nearly six months, between October 31, 2017, and April 20, 2018, and looked for text strings formatted like API tokens and cryptographic keys.
I wonder if GitHub could offer a service where if an API key, be it PGP, SSH, or others, it would automatically disable that item on the relevant repository. This wouldn't stop the best of the best, but it would at least be some remedial security... far better than none.
Say a desktop or mobile application distributed as free software in source code form acts as a client for some Internet service. How is the application's developer supposed to distribute the required API key to the user's machine without exposing it in the source code? Or is each user of the application supposed to apply for API keys for his or her own copy of the application?
(See also my previous thoughts on the API key matter)
Gee, if only there was some quick and easy way to migrate from GitHub to Sourceforge... Oh wait!
It could be the researchers made a mistake in their regular expression that is picking up something that looks like keys but aren't keys.
If this does happen to you it's because you aren't doing code review. If you are solo, then give yourself a quick review by doing a "git add" on each individual file before committing. That gives you a chance to double-check, and you can even do a "git diff" on each file before committing to be extra sure. There are lots of processes you can use to avoid this kind of mistake.
"First they came for the slanderers and i said nothing."
How do they differentiate bogus keys from real keys? In my projects I deliberately include keys that are valid, but won't get you into anything but 'local' applications running with no sensitive data. There are plenty of valid reasons (integration tests, clone-and-run dev applications, etc) to have 'valid' but practically useless keys in github.
that every time microsoft gets ownership of something a few weeks or months later some bad shit like this happens, makes me wonder if a lot of this sort of thing is an inside job,
Politics is Treachery, Religion is Brainwashing
From what I gather from the article is that people put the key in their code. It is not that Github did anything unsafe.
I assume what it could do is block sites that put the key in their code somehow. Obviously if I post the key somewhere else (e.g. here) there would not be a lot that could be done. It also means that code from Github can not be called safe to use, as you never know if somebody has not added malicious code to something that was safe before.
If I am somewhere mistaken, please do not hesitate to correct me.
Don't fight for your country, if your country does not fight for you.
So, it looks like they more-or-less did a regex for things that looked like keys.
How did they know they were "real" keys? If I check in some integration tests, they're going to need a key.....and no one should use that key in anything other than a local integration test. Nor would they expect to since it's in "test" folder only used to build and run tests.
Or you might check in a key to provide an "example" mode with all sorts of warnings about "change this key before production", similar to how many web services will out-of-the-box use http instead of https. Not ideal, but not necessarily an issue.
The problem then comes when a service requires that users be 13 to use the service but 18 to register as a developer in order to obtain an API key. It means 13 to 17 year olds are required to either use proprietary software or not use the service.
I guess my question is that are these keys in question in source modules or just configuration files. If they are in configuration files, how do they not know these are just test keys that will then get changed to production values.
If you're talking about something like ssh, you distribute the public half of the key and not the private half. If you're talking about something like https, you get a cert from one of the official places, and don't distribute it at all (you could make your own cert and distribute the public half, but it's more painful). If you're using a key for user authentication, each user is going to need to generate their own key and you aren't distributing anything.
There are valid reasons to check in a private key (integration tests, "dev mode"), and it can be made clear that those keys are not for production use (documentation, it's in the "test" folder and only used by the testing engine, only listen on localhost if it's the checked-in key, etc)
NCSU academics scanned GitHub accounts for a period of nearly six months... and looked for text strings formatted like API tokens and cryptographic keys.
Or maybe they were just misidentifying old Perl scripts
What kind of weirdo edge case are you making up?
An edge case that has occurred in my own circle of friends. I have relatives who joined Twitter before age 18.
Are you a pedophile?
No.
If you're using a key for user authentication, each user is going to need to generate their own key and you aren't distributing anything.
I'm talking about OAuth, version 1 or 2. The client ID and client secret in OAuth authenticate an application to a service so that the application can receive a session ID representing the user.
Say a million users install an application. Do you think Twitter would appreciate a million requests to register a nearly identical application, differing only by internal timestamps and compiler optimization flags?
And those are going to need to be generated per-app. Otherwise you aren't authenticating anything.
But what's an "app"? Is it the executable program built from a particular repository, or a particular installation thereof?
The featured article is light on details on the patterns used to determine whether a string is "in the format of particular API tokens or cryptographic keys." GitHub's page about "token scanning" likewise doesn't say much. A link deeper in the article to "git secrets" by Amazon gives regular expressions for Amazon API credentials but not those of other well-known providers. The actual regular expressions used are buried in Table III of a PDF linked near the end of the article.
Fortunately, ZDNet is not paywalled.
Take a second to think about this.
If every single install of a program, anywhere on the planet, uses exactly the same identity, how do you know who to let in and who to keep out?
You wouldn't. Which is why you don't give everyone the same identity just because they're running the same executable.
Suppose I write an Android app that uses Google Maps & requires an API key. I dutifully follow Google's instructions, build my app with the key in Strings.xml, compile it, sign it, and publish it to Google Play.
What, exactly, is there to stop someone from obtaining my app through Google Play, ripping it from their phone, deodexing the binary, extracting my API key, then writing THEIR OWN Maps-using app that uses my API key and distributing it to a million users in China (or anywhere else in the world) so the API calls of ITS users end up getting charged against MY account?
I mean, I certainly hope that Google at least does binary-analysis and would reject an app uploaded to Play that attempted to re-use MY API key... but that still does nothing to address API key security in apps NOT distributed via Google Play.
Likewise, Google could theoretically do some form of sanity-checking, and pass along BOTH my API key AND an ID tied to that specific user via Google Play Services, and reject the key's use if the user had never actually installed my app... but that wouldn't help with something like Amazon API keys.
I could see it being secure if API keys offloaded responsibility for registration and payment directly to end users, so that apps would be distributed with dummy keys that were replaced by "real" ones after end users directly authenticated THEMSELVES to Google Maps, Amazon AWS, etc., and used MY app with THEIR key at THEIR OWN expense... but I just don't see how the current way API keys are used with things like Android apps is in any way even REMOTELY secure against key-hijacking.
This is a MAJOR reason why I've always been afraid to publish any Android app that depends upon having an AWS key. At least Google will allow you to distribute a Maps-API-using app with a key that simply gets disabled if you go over the free limit. Amazon won't. They'll give you a certain free allowance, and it might even be fairly high... but Amazon WILL NOT give you the courtesy of setting any kind of absolute circuit breaker and tell them, "if my app exceeds {free-allowance | some-absolute-limit}, just kill it dead and stop any additional charges from accruing". Even worse, Amazon's billing isn't realtime... you can set up all the safeguards you want to notify you if your usage exceeds some amount and kill your account, and even to try and kill it via scripted means... then see the amount you own CONTINUE to skyrocket AFTER you've pulled the plug due to use that happened before you suspended the service, but AFTER the most recent balance update.
Frankly, Amazon's AWS billing policies (above and beyond any concerns about the security of distributing an app with my API key compiled into it) scare me shitless, and apparently I'm far from being the only one who feels that way. I, for one, will NEVER willingly give someone a literal blank check to bill me into bankruptcy.