Slashdot Mirror
Cache On Delivery — Memcached Opens an Accidental Security Hole
jamie spotted this eye-opening presentation (here's a longer explanation) about how easy it is to access sensitive data on many sites using memcached, writing "If you already know what memcached is, skim to slide #17. The jaw-drop will happen around slide #33. Turns out many websites expose their totally-non-protected memcached interface to the Internet, including gowalla, bit.ly, and PBS."
I run my memcacheds behind firewall. I thought that the basic server security rule was that you firewall everything opening ports very cautiously as necessary.
Much less 'memcached' being at fault. They say it themselves:
All this is is stupid admins doing stupid things story and those are dime a dozen.
Your hair look like poop, Bob! - Wanker.
Memcache's one purpose in life is to be as fast as possible. It makes perfect sense for it to drop the overhead of authentication and leave it on the server operator's head to not make it publicly accessible. It's not rare to strip out MySQL's authentication layer (and presumably the same for other DBs) for a speedup when your DB server is sitting behind a firewall.
How are sites slashdotted when nobody reads TFAs?
Memcache allows anyone to overwrite a cache instance. Seriously? It does not authenticate a write to the cache? And they didn't see this as a problem when desgining memcache? Really?
Anyone can write on your underwear too, if you are stupid enough to wear it outside your pants.
Is that an underwear design flaw?
"There is more worth loving than we have strength to love." - Brian Jay Stanley
In terms of the vendors identified, Bit.ly, GoWalla and Pbs were notified. Bit.ly and GoWalla repaired the flaws within minutes. I am not aware of Pbs repairing the issue. This talk seems to have struck a chord which I can't really explain (suggestions welcome). Yes, exposing your memcached's is bad (the talk shows just how bad), but it's not a clever find to discover them. [fd: that's my name on the slides]
Debian's default config says:
# Specify which IP address to listen on. The default is to listen on all IP addresses
# This parameter is one of the only security measures that memcached has, so make sure
# it's listening on a firewalled interface.
-l 127.0.0.1
Are there any distros that don't have it locked down by default? I would hope not, but if something has it insecure out of the box with no warning that might explain it... (though a good sysadmin would firewall all internal services, whether the documentation tells them to or not)
I mod down anyone who says "I will be modded down for this", regardless of the rest of their comment
I've dealt with a lot of large production databases, Oracle, SQL Server, MySQL, PostgreSQL, and similar. I've never seen the authentication layer striped out. These were pretty performance hungry applications as well, though perhaps not that many quick connections, but still the overhead is so small, that when you're communicating internally, it's generally not a problem. For companies like Google I could see this being a problem, but for MOST instances of these servers, I wouldn't suggest people do it.
At the very least, for that extra micro-second of overhead, it means if some cracker gets into your network, you don't have to worry about that as much (not that you don't have to worry, hell, some crackers in your network!).
Also, wouldn't you at least want to restrict it from employees that shouldn't be fucking with it. Granted companies whose purpose it is to develop this application, likely want almost everyone having access, but most instances you wouldn't want this.
None the less, this is off topic, and the article is retarded.
You don't leave the keys in the Batmobile when it's outside of the cave. That's just asking for trouble.
Yeah like... all these open a socket to listen to, and man, if somebody logs into it, they're like, owning your server! Geeze, what where mysql/ssh/postgres/ftp/etc. thinking?!!!
Experiments and other stuff
Jesus Tapdancing Christ, they explicitly say that in the doc(s) where they discuss design decisions. They can't use stunnel or blah or blah2 or blah3?
"It does not authenticate a write to the cache? And they didn't see this as a problem when desgining memcache? Really?"
Yeah, they saw it and they saw their site and their systems and saw that they did not require that feature for themselves - they weren't creating memcache for charity to donate to the world at large - ffs. Say what you want about livejournal but they created a bunch of high performance distributed system tools - gearman, memcache, mogileFS, etc that allowed anyone to build massive social website prior to them, the tools were not there. Now I am sure some smug historical revisionists was come and explain how these things were all built in SNOBOL in the 1950s then reinvented in Java as apache foundation project with simple 12,000 line xml config files that future generations will claim are alien code for time travel devices but those historical revisionists would be dead wrong.
It's open source, if you want to add a layer of protection, go away. For the rest of us, we are using memcached for maximum speed, no such thing in.
Lets say you add a layer of protection to your memcache, it now requires a password to get in. The hacker is in your network, so he can sniff those packets. Where's your protection ? Ok, lets say you add in some crypto to prevent sniffing attack. He's in your network, on your computers, reading your files. He then parse your code and grab the password.
So, what have you been able to achieve exactly ? Nothing beside slow downs everyone downs. People should just learn to use the tools at their disposal. And yes, you can kill yourself with a car. Should we lock them all down to 10mph ?
For the parent, here's memcached help. Notice that even if it's ADRANY, you have it in your face should you be on an unsecure network. And besides, locking it to your ip won't prevent someone inside your network from getting the content.
memcached -help
memcached 1.2.6
-p TCP port number to listen on (default: 11211)
-U UDP port number to listen on (default: 0, off)
-s unix socket path to listen on (disables network support)
-a access mask for unix socket, in octal (default 0700)
-l interface to listen on, default is INDRR_ANY
-d run as a daemon
-r maximize core file limit
-u assume identity of (only when run as root)
-m max memory to use for items in megabytes, default is 64 MB
-M return error on memory exhausted (rather than removing items)
-c max simultaneous connections, default is 1024
-k lock down all paged memory. Note that there is a
limit on how much memory you may lock. Trying to
allocate more than that would fail, so be sure you
set the limit correctly for the user you started
the daemon with (not for -u user;
under sh this is done with 'ulimit -S -l NUM_KB').
-v verbose (print errors/warnings while in event loop)
-vv very verbose (also print client commands/reponses)
-h print this help and exit
-i print memcached and libevent license
-b run a managed instanced (mnemonic: buckets)
-P save PID in , only used with -d option
-f chunk size growth factor, default 1.25
-n minimum space allocated for key+value+flags, default 48
Think of it like this:
System that is never intended to be secure: plastic apple with a warning label stating "THIS IS NOT FOOD"
System that should be secure, but isn't: apple full of worms
You're not going to have a good experience biting into either apple, but there's definitely a difference in the expectations that someone would have when looking at them.
No, you're just splitting hairs. My point is that software can be configured incorrectly and that it isn't the developers' fault if that happens. I believe the parent's point of view (and the general view espoused further up the comment chain) was that software should be distributed in such a manner that it cannot be configured insecurely. I'm hoping I just misunderstood what you wrote and that you were simply disagreeing for the sake of, well, disagreement. Or trolling. Regardless, I'd encourage you to read up the chain so you may better understand my point (the parent was just convenient fodder since his view was mentioned numerous times, and I got fed up with this notion that it's the responsibility of the developers to ensure no configuration can be abused).
N.B. that if I were to split hairs, I'd point out that your comment that implies (I think? You may have missed a word or two while typing it out) that sysadmins shouldn't be forced to do anything to make a particular software install work. That's a fairly short order--almost all software works out of the box as expected, but whether it works securely across all "default" installs is another matter entirely and one that, again, is the responsibility of the user to verify. But the point goes far beyond that, and I'm somewhat disappointed that you seem to agree with the general sentiment that post-install configuration should almost never happen except in rare outlying circumstances (hint: this isn't likely to happen, particularly with the vast array of requirements, disparate hardware, and so forth). Note, also, that if this notion of "secure by default" is taken to the extreme, you wind up with OpenBSD. OpenBSD is secure by default. You also cannot use it without installing and configuring additional software (thus no longer having a "default" install), because it comes with only OpenSSH installed by default! In other words, we're just encountering a difference of philosophies, and I ascribe to the one that makes the burden of configuration your responsibility as a user, not mine as a developer.
And really, this point it doesn't matter at all, because someone will still find a way to configure it incorrectly, regardless of how well it was written (don't believe me? you'd be surprised what lurks out there in the wild) or how strict the default configuration is. There are people out there who will tinker with things, including things they may not fully understand, and configure it in a manner that is suddenly insecure. That is my fundamental point; the point of the OP and those subsequently up the chain is that the responsibility should rest upon the shoulders of the developers to ensure their software cannot be configured incorrectly--that's the whole point many of the posts here are arguing against Memcached for! There seems to be so much heartache that Memcached (which doesn't require authentication) can be configured in a manner that it exposes sensitive data to the Interwebs. Guess what? If it does, that is the fault of the administrator and not the fault of Memcached or its developers.
I'm not sure how else to state that more clearly. But I'll try:
Responsibility rests on the person who installs the software to ensure it is configured securely for their needs, not the developers nor the package maintainers. The only point in time, IMO, where such responsibility is the developers' is when it becomes an issue of out-of-band behavior; e.g. something that is unexpected and unintended (a bug or security flaw) and not fundamentally something that is the result of a misconfiguration. In short:
Misconfigurations = user's responsibility (even if t
He who has no