Slashdot Mirror

← Back to Stories (view on slashdot.org)

Cache On Delivery — Memcached Opens an Accidental Security Hole

Posted by timothy on from the everything-has-a-downside dept.

jamie spotted this eye-opening presentation (here's a longer explanation) about how easy it is to access sensitive data on many sites using memcached, writing "If you already know what memcached is, skim to slide #17. The jaw-drop will happen around slide #33. Turns out many websites expose their totally-non-protected memcached interface to the Internet, including gowalla, bit.ly, and PBS."

23 of 149 comments (clear)

  1. Firewall? by chx1975 · · Score: 4, Insightful

    I run my memcacheds behind firewall. I thought that the basic server security rule was that you firewall everything opening ports very cautiously as necessary.

    1. Re:Firewall? by MikeFM · · Score: 4, Insightful

      My memcached server is on the private network only accessible to other servers and is firewalled to everything but the servers that need access. Not exactly rocket science.

      --
      At what price learning? At what cost wisdom? The price is a man's peace of mind, and the cost is his life.
    2. Re:Firewall? by IICV · · Score: 4, Interesting

      Yeah, slide 52 (paraphrased) is as follows:

      Fixes?

      1. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW....
      2. .....
      3. Also, FW

      I assume he means "firewalls" by "FW". Seriously, you can't even bother to spell out "firewall" in a presentation?

    3. Re:Firewall? by PIBM · · Score: 4, Insightful

      Ive been running memcached since it's out, even sent some patches in.

      The thing is, why aren't they running this on a private network ?? Memcached is designed to be fast AND non-secure, to be run on your local network. Running it on a server farm with thousands of people having access to your computers and ips is not a private network.

      I had heard about people running it on the local interface and still getting problems before (somebody else with the same computer ran it too and forgot to pick the good port and finally used the same key ...) but that's because IT'S NOT BUILT TO BE USED ON AN UNSECURED NETWORK.

      Nothing new, bad admins get bad things done to them, move along.

  2. I fail to see why this is news by OverlordQ · · Score: 5, Insightful

    Much less 'memcached' being at fault. They say it themselves:

    Memcached does not spend much, if any, effort in ensuring its defensibility from random internet connections. So you must not expose memcached directly to the internet, or otherwise any untrusted users.

    All this is is stupid admins doing stupid things story and those are dime a dozen.

    --
    Your hair look like poop, Bob! - Wanker.
    1. Re:I fail to see why this is news by TheRaven64 · · Score: 3, Insightful

      A good tool is designed in such a way that the correct use is the easiest. Does memcached, for example, default to only accepting connections from the local host and require other IPs to be explicitly added? That would be trivial to implement and, if done, would require the admin to implement something like the correct security policy. In contrast, defaulting to accepting connections from anywhere means that the admin can use it incorrectly without needing to do any thinking, but needs to think before using it correctly.

      --
      I am TheRaven on Soylent News
    2. Re:I fail to see why this is news by MikeFM · · Score: 4, Insightful

      The difference is that in this case a non-retarded admin can secure things. With Microsoft products it often takes an act of God to secure them (the best security feature of a Windows system is a blue screen of death). And memcached isn't meant to be a public service. It's very plainly described as not being secure. Completely different than a service that is meant to be public such as web or email not being secure.

      --
      At what price learning? At what cost wisdom? The price is a man's peace of mind, and the cost is his life.
    3. Re:I fail to see why this is news by MikeFM · · Score: 4, Insightful

      It defaults to not being installed and running. Memcached is meant to be ran from one or more caching servers (not really on the web server itself). It isn't really meant to be ran on localhost under ideal usage.

      --
      At what price learning? At what cost wisdom? The price is a man's peace of mind, and the cost is his life.
    4. Re:I fail to see why this is news by nebular · · Score: 3, Insightful

      I believe the point here is that software designers should assume that in terms of security their users are complete idiots and WILL install and setup the program in an unsecure manner unless they are specifically beat over the head with the notion that what they are doing is BAD!

    5. Re:I fail to see why this is news by TheRaven64 · · Score: 5, Insightful

      Which is exactly the point. The default install should never be working-and-insecure. It should be secure, and ideally it should be working. If it is not possible for the default install to be both useful and secure, as appears to be the case with memcached, then it should install only listening on localhost and require explicit intervention by the user to accept connections from other hosts.

      If you can install it and have it work by default, then there is no reason for the user to bother reading the manual, so they won't learn that it needs to be specially configured to be secure. If the default is secure but not particularly useful, then the user needs to explicitly adjust the setting that makes it insecure, and in so doing needs to read the documentation explaining that this will make it insecure and how to mitigate it.

      --
      I am TheRaven on Soylent News
  3. Re:Let me see if I understand this by Firehed · · Score: 5, Informative

    Memcache's one purpose in life is to be as fast as possible. It makes perfect sense for it to drop the overhead of authentication and leave it on the server operator's head to not make it publicly accessible. It's not rare to strip out MySQL's authentication layer (and presumably the same for other DBs) for a speedup when your DB server is sitting behind a firewall.

    --
    How are sites slashdotted when nobody reads TFAs?
  4. More Boiled and Distilled. by SuperKendall · · Score: 5, Insightful

    Memcache allows anyone to overwrite a cache instance. Seriously? It does not authenticate a write to the cache? And they didn't see this as a problem when desgining memcache? Really?

    Anyone can write on your underwear too, if you are stupid enough to wear it outside your pants.

    Is that an underwear design flaw?

    --
    "There is more worth loving than we have strength to love." - Brian Jay Stanley
    1. Re:More Boiled and Distilled. by Farmer+Tim · · Score: 5, Insightful

      Best. Analogy. Ever.

      --
      Blank until /. makes another boneheaded UI decision.
    2. Re:More Boiled and Distilled. by pushing-robot · · Score: 4, Funny

      As spokesman for the Justice League, I say yes.

      --
      How can I believe you when you tell me what I don't want to hear?
    3. Re:More Boiled and Distilled. by davester666 · · Score: 4, Funny

      No. Car. Was. Involved.

      --
      Sleep your way to a whiter smile...date a dentist!
    4. Re:More Boiled and Distilled. by outsider007 · · Score: 5, Funny

      That's actually more of a feature.

      --
      If you mod me down the terrorists will have won
    5. Re:More Boiled and Distilled. by Farmer+Tim · · Score: 5, Funny

      That's. Why.

      --
      Blank until /. makes another boneheaded UI decision.
  5. A few clarifications by marcoslaviero · · Score: 5, Informative

    In terms of the vendors identified, Bit.ly, GoWalla and Pbs were notified. Bit.ly and GoWalla repaired the flaws within minutes. I am not aware of Pbs repairing the issue. This talk seems to have struck a chord which I can't really explain (suggestions welcome). Yes, exposing your memcached's is bad (the talk shows just how bad), but it's not a clever find to discover them. [fd: that's my name on the slides]

    1. Re:A few clarifications by marcoslaviero · · Score: 5, Interesting

      There's a deeper issue at play here as it relates to shifting apps and platforms away from your own hardware/networks. Developers are now often responsible for deploying apps onto cloud systems where they don't have experience with network-security or the tools for protecting network-based services, and this is an obvious difference from the traditional network/app split that occurs in most corporates. It doesn't help that memcached (by default) binds to * but they do make this pretty clear (also, remote enumeration of the cache is genuinely a debug feature).

      Man pages help, but when the defaults don't aid developers we need to a rethink both of the software (memcached) and the systems were it's not running securely (cloud platforms).

    2. Re:A few clarifications by IAmGarethAdams · · Score: 5, Funny

      Mostly through rouge employees

      Luckily, they often get caught red-handed.

  6. Admin or distro? by shish · · Score: 5, Interesting

    Debian's default config says:

    # Specify which IP address to listen on. The default is to listen on all IP addresses
    # This parameter is one of the only security measures that memcached has, so make sure
    # it's listening on a firewalled interface.
    -l 127.0.0.1

    Are there any distros that don't have it locked down by default? I would hope not, but if something has it insecure out of the box with no warning that might explain it... (though a good sysadmin would firewall all internal services, whether the documentation tells them to or not)

    --
    I mod down anyone who says "I will be modded down for this", regardless of the rest of their comment
    1. Re:Admin or distro? by TheRaven64 · · Score: 4, Insightful

      default to INDRR_ANY

      And this is why they're to blame. Default should be the loopback, and enabling external access should require explicit configuration.

      --
      I am TheRaven on Soylent News
    2. Re:Admin or distro? by bill_mcgonigle · · Score: 3, Insightful

      Memcached is not meant for single-server configurations

      That's silly, it's a generic object store. There's no reason not to use it to cache expensive local operations. Of course it shines across a farm of caches, but the server mapping hash will work just fine with one machine.

      If you're a startup with just one webserver and starting to hit performance problems, memcached will likely buy you a few more months.

      Going from one server to two is hard, three is a bit more work, and after three it's roughly all the same until you start adding more data centers and then it's all the same until you're Facebook. Taking on that 'hard' expense too early would be a poor allocation of resources.

      --
      My God, it's Full of Source!
      OUTSIDE_IP=$(dig +short my.ip @outsideip.net)