Slashdot Mirror
Cache On Delivery — Memcached Opens an Accidental Security Hole
jamie spotted this eye-opening presentation (here's a longer explanation) about how easy it is to access sensitive data on many sites using memcached, writing "If you already know what memcached is, skim to slide #17. The jaw-drop will happen around slide #33. Turns out many websites expose their totally-non-protected memcached interface to the Internet, including gowalla, bit.ly, and PBS."
I run my memcacheds behind firewall. I thought that the basic server security rule was that you firewall everything opening ports very cautiously as necessary.
DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
Version 3, August 2010
Copyright (C) 2010 Anonymous Coward
Everyone is permitted to copy and distribute verbatim or modified copies of this license document, and changing it is allowed as long as the name is changed.
DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. You just DO WHAT THE FUCK YOU WANT TO.
Much less 'memcached' being at fault. They say it themselves:
All this is is stupid admins doing stupid things story and those are dime a dozen.
Your hair look like poop, Bob! - Wanker.
Memcache's one purpose in life is to be as fast as possible. It makes perfect sense for it to drop the overhead of authentication and leave it on the server operator's head to not make it publicly accessible. It's not rare to strip out MySQL's authentication layer (and presumably the same for other DBs) for a speedup when your DB server is sitting behind a firewall.
How are sites slashdotted when nobody reads TFAs?
Memcache allows anyone to overwrite a cache instance. Seriously? It does not authenticate a write to the cache? And they didn't see this as a problem when desgining memcache? Really?
Anyone can write on your underwear too, if you are stupid enough to wear it outside your pants.
Is that an underwear design flaw?
"There is more worth loving than we have strength to love." - Brian Jay Stanley
In terms of the vendors identified, Bit.ly, GoWalla and Pbs were notified. Bit.ly and GoWalla repaired the flaws within minutes. I am not aware of Pbs repairing the issue. This talk seems to have struck a chord which I can't really explain (suggestions welcome). Yes, exposing your memcached's is bad (the talk shows just how bad), but it's not a clever find to discover them. [fd: that's my name on the slides]
Debian's default config says:
# Specify which IP address to listen on. The default is to listen on all IP addresses
# This parameter is one of the only security measures that memcached has, so make sure
# it's listening on a firewalled interface.
-l 127.0.0.1
Are there any distros that don't have it locked down by default? I would hope not, but if something has it insecure out of the box with no warning that might explain it... (though a good sysadmin would firewall all internal services, whether the documentation tells them to or not)
I mod down anyone who says "I will be modded down for this", regardless of the rest of their comment
I've dealt with a lot of large production databases, Oracle, SQL Server, MySQL, PostgreSQL, and similar. I've never seen the authentication layer striped out. These were pretty performance hungry applications as well, though perhaps not that many quick connections, but still the overhead is so small, that when you're communicating internally, it's generally not a problem. For companies like Google I could see this being a problem, but for MOST instances of these servers, I wouldn't suggest people do it.
At the very least, for that extra micro-second of overhead, it means if some cracker gets into your network, you don't have to worry about that as much (not that you don't have to worry, hell, some crackers in your network!).
Also, wouldn't you at least want to restrict it from employees that shouldn't be fucking with it. Granted companies whose purpose it is to develop this application, likely want almost everyone having access, but most instances you wouldn't want this.
None the less, this is off topic, and the article is retarded.
You don't leave the keys in the Batmobile when it's outside of the cave. That's just asking for trouble.
Yeah like... all these open a socket to listen to, and man, if somebody logs into it, they're like, owning your server! Geeze, what where mysql/ssh/postgres/ftp/etc. thinking?!!!
Experiments and other stuff
I'm guessing you've never used it and have no idea why it would be useful.
Yes, totally true. But it would still have been better to have authentication enabled by default. The user who actually knows what he is doing can then disable it.
If Pandora's box is destined to be opened, *I* want to be the one to open it.
Because of MySQL injection attacks and other problems, it is nice to have different access rights for different parts of your site. So I would not recommend stripping out the auth layer, especially if you have more than 1 DB running.
Even the best programmers and practices can create bugs.
Can we have a Flash warning like we have on PDFs? At least I can open PDFs but not Flash.
Jesus Tapdancing Christ, they explicitly say that in the doc(s) where they discuss design decisions. They can't use stunnel or blah or blah2 or blah3?
"It does not authenticate a write to the cache? And they didn't see this as a problem when desgining memcache? Really?"
Yeah, they saw it and they saw their site and their systems and saw that they did not require that feature for themselves - they weren't creating memcache for charity to donate to the world at large - ffs. Say what you want about livejournal but they created a bunch of high performance distributed system tools - gearman, memcache, mogileFS, etc that allowed anyone to build massive social website prior to them, the tools were not there. Now I am sure some smug historical revisionists was come and explain how these things were all built in SNOBOL in the 1950s then reinvented in Java as apache foundation project with simple 12,000 line xml config files that future generations will claim are alien code for time travel devices but those historical revisionists would be dead wrong.
It's open source, if you want to add a layer of protection, go away. For the rest of us, we are using memcached for maximum speed, no such thing in.
Lets say you add a layer of protection to your memcache, it now requires a password to get in. The hacker is in your network, so he can sniff those packets. Where's your protection ? Ok, lets say you add in some crypto to prevent sniffing attack. He's in your network, on your computers, reading your files. He then parse your code and grab the password.
So, what have you been able to achieve exactly ? Nothing beside slow downs everyone downs. People should just learn to use the tools at their disposal. And yes, you can kill yourself with a car. Should we lock them all down to 10mph ?
For the parent, here's memcached help. Notice that even if it's ADRANY, you have it in your face should you be on an unsecure network. And besides, locking it to your ip won't prevent someone inside your network from getting the content.
memcached -help
memcached 1.2.6
-p TCP port number to listen on (default: 11211)
-U UDP port number to listen on (default: 0, off)
-s unix socket path to listen on (disables network support)
-a access mask for unix socket, in octal (default 0700)
-l interface to listen on, default is INDRR_ANY
-d run as a daemon
-r maximize core file limit
-u assume identity of (only when run as root)
-m max memory to use for items in megabytes, default is 64 MB
-M return error on memory exhausted (rather than removing items)
-c max simultaneous connections, default is 1024
-k lock down all paged memory. Note that there is a
limit on how much memory you may lock. Trying to
allocate more than that would fail, so be sure you
set the limit correctly for the user you started
the daemon with (not for -u user;
under sh this is done with 'ulimit -S -l NUM_KB').
-v verbose (print errors/warnings while in event loop)
-vv very verbose (also print client commands/reponses)
-h print this help and exit
-i print memcached and libevent license
-b run a managed instanced (mnemonic: buckets)
-P save PID in , only used with -d option
-f chunk size growth factor, default 1.25
-n minimum space allocated for key+value+flags, default 48
I don't allow random systems to communicate with my database. It is only accessed directly from localhost. Any other system has to go through a predefined RPC function that sanitizes inputs and outputs and does a very specific task. Even that access is only granted to a few systems and never over the public network. I usually still use the built-in security too but I would consider removing it as I consider it almost useless.
At what price learning? At what cost wisdom? The price is a man's peace of mind, and the cost is his life.
Besides, one could easily secure this by using IPsec and machine certificates.
I'm not a Linux admin, but that's how i would do it on Windows.
I always use MySQL permissions to separate databases so that one application can't access another database so easily, I also never allow access to it from the internet directly, same policy applies to all server software, unless it's made to be directly connected to and needs to then it stays on the internal network or loopback.
As for all programmers having bugs, even Rivest et all's MD6 reference implentation code sent for the SHA3 NIST competition had a buffer overflow in it: slashdot.org, and while we all should say security experts should know better, these mistakes happen from time to time.
Web 2.0 is an interesting phenomenon. In many ways, it's nothing but embracing ignorance, bad ideas, the worst possible way of doing things, and the stupidest people (both users and developers!) around.
First, let's ignore how Web 2.0 sites cater to the stupidest users, and because of this are willing to abuse private data given to them by their users. That's a whole different story. Let's just focus on the technology here.
The first thing we see is that the use of PHP is prevalent among Web 2.0 sites. If there's one thing that PHP is "good" for, it is quickly developing poorly-performing and insecure web applications, written by programmers with a minimum of technical knowledge. That's why it has blossomed, so to speak, among the Web 2.0 development community. It caters well to their dislike of quality and their lack of technical knowledge. In many cases, the developers are too ignorant to know that there are better options out there, let alone identify the flaws and problems with their own code.
The second thing we see is a lack of knowledge about data storage and scalability. This has manifest itself as the rise of the NoSQL movement. NoSQL is all about throwing away decades of accumulated knowledge concerning relational databases, and replacing it with hash tables. It's about embracing ignorance.
Many of the most popular Web 2.0 sites moved to NoSQL "databases" claiming it was necessary to do so for "scalability reasons", but that's obviously bunk after reading the many articles they publish describing their architectures. We find out that they're clueless about such basic things as indexes, and in some cases even perform joins by pulling ALL of the data back to their PHP code, and manually joining it there.
Worse yet, these Web 2.0 sites are prone to failure. Twitter's "fail whale" has become widely known, and reddit suffers from frequent downtime, including a lengthy incident just yesterday. What's funniest, however, is that the traffic they face is minuscule compared to the activity that many POS and financial systems, or even "Web 1.0" sites, handle with ease because they're developed by knowledgeable developers who know how to properly use relational databases.
Given that the Web 2.0 developers thrive on ignorance and doing things as incorrectly as possible, it's no wonder some of their flagship technologies and sites would have such glaring security holes. That's just what's to be expected from people who don't have a fucking clue about what they're doing.
Sounds like a great reason to use something like SSL between memcached and the webservers. Forward the incoming SSL connections to memcached back to the memcached port over a loopback address. Now you can have authentication and encryption, if you want it, without adding any complexity to memcached. That way if someone wants to run it locally on a webserver you haven't really added any complexity.
What the fuck is a "MySQL injection attack"?
Think of it like this:
System that is never intended to be secure: plastic apple with a warning label stating "THIS IS NOT FOOD"
System that should be secure, but isn't: apple full of worms
You're not going to have a good experience biting into either apple, but there's definitely a difference in the expectations that someone would have when looking at them.
Perhaps utilities like rm should be rewritten to use -i (or similar) by default then? Or how about we ought to remove them outright since they might be used incorrectly?
Honestly, it's not up to the developer to make any such assumptions. Any piece of software can be used "insecurely" if configured incorrectly (yes, even IIS). Ultimately, you as the user bare the responsibility for ensuring that your system and your data are correctly secured.
As far as beaten over the head, the default configuration for Memcached that ships with Gentoo has this note immediately over the configuration for what interfaces it should listen on:
I'm sure other distros have similar notes.
He who has no
Read my blog about it here:
http://blog.codesherpas.com/on_the_path/2010/08/securing-memcache-in-2-minutes.html
This isn't a security problem - this is operating by design. If you are a memcache user and this is news to you, you need to read more about the tools you are using. I bet you have security problems beyond this one.
You might be surprised to learn that the problems are slightly different than your ERP system, in that pretty much every field is either very large and required to grow to infinity.
You want to support picture uploads to your site. You can either:
1. Store them raw on disk and watch the file system collapse under the weight.
2. Make a table to store all the pictures in an RDBMS and then spend a lot of $$$$ making sure you have enough disk space on your SAN to store them all as well as create a method to sync that data to all of your data centers.
3. Store a pointer in your RDBMS to the hash keys in your DHT, which runs on significantly cheaper hardware and requires a lot less man power to keep running and is generally much easier to keep in sync.
At some point, the data gets so sharded that there is no "relational" involved anymore. So why store it in a database tuned for it?
I'd say that "injecting" MySQL on to a server is a pretty malicious thing to do.
I wouldn't recommend stripping out the authentication layer either, I merely said it's been done. That being said, if your app can suffer from SQL injection, you've got bigger problems. If your app is to the point of needing to strip out DB authentication for some extra performance (and if it is, you should probably be taking a different approach... such as memcache, the point of this whole discussion), you damn well better have all sorts of tests in place to make sure that you don't introduce problems that severe.
How are sites slashdotted when nobody reads TFAs?
This whole conversation is stupid.
I have mod points, but the +1 troll modifier seems to be missing.
How many FreeBSD fans attend Indiana University-Purdue University Fort Wayne?