Cache On Delivery — Memcached Opens an Accidental Security Hole

jamie spotted this eye-opening presentation (here's a longer explanation) about how easy it is to access sensitive data on many sites using memcached, writing "If you already know what memcached is, skim to slide #17. The jaw-drop will happen around slide #33. Turns out many websites expose their totally-non-protected memcached interface to the Internet, including gowalla, bit.ly, and PBS."

Comments

Firewall?

[chx1975]

I run my memcacheds behind firewall. I thought that the basic server security rule was that you firewall everything opening ports very cautiously as necessary.

Re:Firewall?

[MikeFM]

My memcached server is on the private network only accessible to other servers and is firewalled to everything but the servers that need access. Not exactly rocket science.

Re:Firewall?

[IICV]

Yeah, slide 52 (paraphrased) is as follows:

Fixes?

1. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW....
2. .....
3. Also, FW

I assume he means "firewalls" by "FW". Seriously, you can't even bother to spell out "firewall" in a presentation?

Re:Firewall?

[Penguinisto]

I assume he means "firewalls" by "FW". Seriously, you can't even bother to spell out "firewall" in a presentation?

...depends if he's a FreeBSD fan or not. :)

Re:Firewall?

[tsm_sf]

My memcached server[...]

Since you run memcached maybe you can answer this. From what I've read it seems like an alerted (or just savvy) admin could easily detect this kind of futzing and (if you're lucky just) inject honey directly onto your machine. Attempting to maliciously write to memcached seems equally problematic. You'd never know if you were successful or just seeing what someone wanted you to see. How off base am I?

Re:Firewall?

[Anonymous Coward]

You forgot step 4 and 5
4: ???
5: profit

Re:Firewall?

[marcoslaviero]

eh... this presentation was more an "in-person" thing than meant for later perusing.

that said, i'll stick to "firewall" next time.

Re:Firewall?

[neerolyte]

Just run two. Only one of them has to have real world data, the other one siphens IPs to your firewall software :)

Re:Firewall?

[WrongSizeGlass]

Tech people aren't exactly known for being overly verbose (except maybe on /.):
* ar
* cp
* df
* ls
* mv
* ps
* rm
* sh
* tr
M'kay?

Re:Firewall?

[marcoslaviero]

I prefer pf actually :)

Re:Firewall?

[PIBM]

Ive been running memcached since it's out, even sent some patches in.

The thing is, why aren't they running this on a private network ?? Memcached is designed to be fast AND non-secure, to be run on your local network. Running it on a server farm with thousands of people having access to your computers and ips is not a private network.

I had heard about people running it on the local interface and still getting problems before (somebody else with the same computer ran it too and forgot to pick the good port and finally used the same key ...) but that's because IT'S NOT BUILT TO BE USED ON AN UNSECURED NETWORK.

Nothing new, bad admins get bad things done to them, move along.

Re:Firewall?

[MikeFM]

I doubt most admins look and if memcached was open it's virtually a feast of access. You wouldn't necessarily find it easy to jump from memcached to the rest of the system but you could gather a lot of information and alter data going to users and internal systems. Depending how memcache is being used it could be a pretty big exploit. The hardest part would be finding the right data keys. All mine are md5 hashes of a per site secret token combined with the given key so pretty difficult but I'm sure the kind of developer that leaves the service open to the public might be bright enough to name something 'password'.

I'm hardly an expert but it seems pretty obvious. Unless the developer is pretty idiotic it's probably not a problem but if they are then maybe it's a big problem.

Re:Firewall?

[bsDaemon]

All the cool kids are using 'pf' these days: https://cipitunk.wordpress.com/2007/07/07/ipfw-vs-pf/

Re:Firewall?

[Call Me Black Cloud]

Ah, I was wondering what FW meant. Thanks.

Re:Firewall?

[newdsfornerds]

If he's a FreeBSD guy he should be using ipfilter. HEH!

Re:Firewall?

[Anonymous Coward]

A slow and hopelessly outdated version. It surprises me that a community who wanks to benchmarks would run that kind of stuff. Update to pf 4.8 now!

Re:Firewall?

[bill_mcgonigle]

Memcached is designed to be fast AND non-secure, to be run on your local network

My entire training on memcached is a 45-minute LUG presentation I went to a couple years back, and even then this was readily apparent.

The affected websites need to review their security personnel and procedures.

Re:Firewall?

[allenw]

They likely don't have security personnel. I wouldn't be surprised if there were some "just get in the way" or "security can be added later" or "who cares" or any number of other statements made that clueless developers later on regret.

Re:Firewall?

[Anonymous Coward]

A better sequence:

        unzip, strip, touch, finger, grep, mount, fsck, more, yes, fsck, fsck, fsck, umount, sleep

Re:Firewall?

[cynyr]

that was due to the slow rate of terminals at ~8 chars/second back in the day. copy would have taken you 2 times as long to type, and don't forget each char had to be sent to the computer and then sent back to your display. meaning typing copy took at minimum 1 second at 8 baud.

DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE

[Anonymous Coward]

DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
Version 3, August 2010

  Copyright (C) 2010 Anonymous Coward

Everyone is permitted to copy and distribute verbatim or modified copies of this license document, and changing it is allowed as long as the name is changed.

DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

    0. You just DO WHAT THE FUCK YOU WANT TO.

Re:DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE

[blue trane]

http://tunes.org/legalese/bugroff.html [tunes.org]

Richard Stallman of the Free Software Foundation devised, in addition to some marvelous software, the GNU General Public License (GPL for short). Or the CopyLeft it is sometimes called.

It is quite a revolutionary document, using the "copyright" tool to to protect your right to use free software.

Unfortunately using copyright to protect free software is a lot like using a Jackal to guard the hens.

In fact, various inconveniences relating to this have resulted in modifications such as the LGPL (Library General Public License) and more recently the NPL (Netscape Public License)

I call these matters mere inconveniences, the real damage will occur when the Jackal's, (sorry, I mean lawyers), actually get to test the GPL in court for the first time.

Thus enter my version.

Its very simple.

Entirely consistent.

Completely unrestrictive.

Easy to apply.

The "No problem Bugroff" license is as follows...

The answer to any and every question relating to the copyright, patents, legal issues of Bugroff licensed software is....

Sure, No problem. Don't worry, be happy. Now bugger off.

All portions of this license are important..

"Sure, no problem." Gives you complete freedom. I mean it. Utterly complete. A bit of a joke really. You have complete freedom anyway.
"Don't worry, be happy." Apart from being good advice and a good song, it also says :- No matter what anyone else says or does, you still have complete freedom.
Now bugger off. The only way to get rid of pushy Jackals is to ignore them and not feed them. The GPL is just begging somebody to take it to court. Can't you just see it. Exactly the same thing that happened when some twit (not Linus) registered Linux as his own personal trademark. People got upset, started a fund, and hired, off all ruddy things, a Jackal to try and defend the chicken! Who really benefits from this trademark / patent / copyright thing anyway? The lawyers. Who made it up in the first place? The lawyers.
OK so the last part of the license sounds a bit harsh, but seriously folks, if you are a :-

Lawyer asking these legalese questions... You should go off and learn an honest trade that will actually contribute to life instead of draining it.
Programmer asking these legalese questions... You have amazingly powerful tools in your hands and mind, use them to ask and answer the worthwhile questions of life, the universe and everything. Stop mucking about with such legal nonsense and get back to programming.
User/reader asking these question... Don't worry. Go off and be happy. Have fun. Enjoy what has been created for you.

Re:DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE

[flyingfsck]

Sorry to pop your bubble, but there already is a Do What The Fuck You Want To license.

Re:DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE

[blue trane]

I want to reinvent it!

Re:DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE

[BradMajors]

Sorry to pop your bubble, but there already is a Do What The Fuck You Want To license.

But, the "Do What The Fuck You Want To License" allows users to copy the license and claim they invented and own this license.

Re:DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE

[Anonymous Coward]

Anonymous Coward here. Property is theft, therefore theft is property, therefore the license is mine.

Re:DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE

[Anonymous Coward]

You have complete freedom anyway.

No, you don't. Hence the problem.

I fail to see why this is news

[OverlordQ]

Much less 'memcached' being at fault. They say it themselves:

Memcached does not spend much, if any, effort in ensuring its defensibility from random internet connections. So you must not expose memcached directly to the internet, or otherwise any untrusted users.

All this is is stupid admins doing stupid things story and those are dime a dozen.

Re:I fail to see why this is news

[Etylowy]

This.

That's what you get for being cheap when hiring security team. Setting up memcached on a public IP or without a firewall is as bad as having your session directory fully writable on a public ftp (and quite similar too).
Memcached is good software and does exactly what it has been designed to do - provides fast key cache - but as with every tool if you are dumb you will hurt yourself. If those admins were carpenters they'd have an average of 4 fingers ;-)

Re:I fail to see why this is news

[Anonymous Coward]

Are there double standards on this site, depending on whether Microsoft are involved or not? You blame the admins here, but if it were Windows, I bet you'd blame Microsoft and not stupid users.

Re:I fail to see why this is news

[TheRaven64]

A good tool is designed in such a way that the correct use is the easiest. Does memcached, for example, default to only accepting connections from the local host and require other IPs to be explicitly added? That would be trivial to implement and, if done, would require the admin to implement something like the correct security policy. In contrast, defaulting to accepting connections from anywhere means that the admin can use it incorrectly without needing to do any thinking, but needs to think before using it correctly.

Re:I fail to see why this is news

[Anonymous Coward]

You don't need a security team, just a clue.

Re:I fail to see why this is news

[flimflammer]

I really wish I had mod points right now. +1

Re:I fail to see why this is news

[MikeFM]

The difference is that in this case a non-retarded admin can secure things. With Microsoft products it often takes an act of God to secure them (the best security feature of a Windows system is a blue screen of death). And memcached isn't meant to be a public service. It's very plainly described as not being secure. Completely different than a service that is meant to be public such as web or email not being secure.

Re:I fail to see why this is news

[MikeFM]

It defaults to not being installed and running. Memcached is meant to be ran from one or more caching servers (not really on the web server itself). It isn't really meant to be ran on localhost under ideal usage.

Re:I fail to see why this is news

[Anonymous Coward]

http://code.google.com/p/memcached/wiki/NewConfiguringServer

Networking
By default memcached listens on TCP and UDP ports, both 11211. -l allows you to bind to specific interfaces or IP addresses. Memcached does not spend much, if any, effort in ensuring its defensibility from random internet connections. So you must not expose memcached directly to the internet, or otherwise any untrusted users. Using SASL authentication here helps, but should not be totally trusted.

From their wiki page detailing how to configure a new server. Surely the part they highlight in bold should have raised a flag to even the dumbest administrator.

Re:I fail to see why this is news

[grumbel]

The "blame the user" is pretty much standard under Unix/Linux, after all it makes you:

1) Feel clever
2) Stops you from thinking of how to improve the situation

Stuff like this happens simply because currently day computer systems are extremely crappy in communicating what they export to the world, thus it is very easy to overlook cases like this where an app exposes more then you intended.

Re:I fail to see why this is news

[Enleth]

netstat -lpn seems simple enough. I tend to run it every time I change something in a configuration file of a network-enabled service, just to be sure. It would be irresponsible to do otherwise.

Re:I fail to see why this is news

[bickerdyke]

I wouldn't call it double standard, as this is exactly about what MS is doing: Pretending that it's easy.

It's ok to design a cache-db thats only secure behind a firewall. As long as you're informing your users about it. I'd take any bet that it is in the memcachd manuals in bold.

The MS way usually is to pretend that it's easy. The docs consist of a leaflet that explains how to open your cd drive and place the cd the right side up, and how to handle the mouse to make the cursor move to the "Ok" button.

At first glance, making things easier sounds like a good idea. "Hey, installing WinXP is easy! Just insert the CD and hit OK three times. Anyone can do it." Yes, sounds like a way to make everyones life easier. But it's a really bad idea if anyone CAN install an OS if only very few people know about the additional steps needed to finish setting up a reasonybly secure machine. (Antivirus, Firewall/NAT, AdBlocker, Browser, YMMV)

Re:I fail to see why this is news

[vrmlguy]

http://code.google.com/p/memcached/wiki/NewConfiguringServer

Networking
By default memcached listens on TCP and UDP ports, both 11211. -l allows you to bind to specific interfaces or IP addresses. Memcached does not spend much, if any, effort in ensuring its defensibility from random internet connections. So you must not expose memcached directly to the internet, or otherwise any untrusted users. Using SASL authentication here helps, but should not be totally trusted.

From their wiki page detailing how to configure a new server. Surely the part they highlight in bold should have raised a flag to even the dumbest administrator.

Here's an idea that won't impact performance: At startup, issue a big multi-line warning if the IP addresses that are getting bound aren't on a Private Internet:

The Internet Assigned Numbers Authority (IANA) has reserved the
      following three blocks of the IP address space for private internets:

          10.0.0.0 - 10.255.255.255 (10/8 prefix)
          172.16.0.0 - 172.31.255.255 (172.16/12 prefix)
          192.168.0.0 - 192.168.255.255 (192.168/16 prefix)

Re:I fail to see why this is news

[nebular]

I believe the point here is that software designers should assume that in terms of security their users are complete idiots and WILL install and setup the program in an unsecure manner unless they are specifically beat over the head with the notion that what they are doing is BAD!

Re:I fail to see why this is news

[TheRaven64]

Which is exactly the point. The default install should never be working-and-insecure. It should be secure, and ideally it should be working. If it is not possible for the default install to be both useful and secure, as appears to be the case with memcached, then it should install only listening on localhost and require explicit intervention by the user to accept connections from other hosts.

If you can install it and have it work by default, then there is no reason for the user to bother reading the manual, so they won't learn that it needs to be specially configured to be secure. If the default is secure but not particularly useful, then the user needs to explicitly adjust the setting that makes it insecure, and in so doing needs to read the documentation explaining that this will make it insecure and how to mitigate it.

Re:I fail to see why this is news

[grumbel]

"netstat -lpn" lists a lot of stuff that is not exported, you can get closer with "netstat -tulpen" or whatever, but even then you only have a very very rough overview about what you are exporting. For example you could have samba (or apache or whatever) running, which you want, but have it export all of '/' instead of just the directory you want. There is no easy way to find that out without digging through a lot of config files.

Re:I fail to see why this is news

[Anonymous Coward]

Disclaimer: I don't run a public site or run memcache.

From the description (and the memcache "NewConfigurinServer" instruction excerpts) that have been posted here, it looks like memcache is a knife in a box that says "do not handle by blade" on the outside. It comes sharp, but there's a warning. If you're smart, you either know not to wrap your fingers around the blade, or you read the warning and you learn that you shouldn't hold it that way. If you're dumb, you ignore the warning and reach into the box, and wrap your (remaining) fingers around what you find most convenient. I suppose they could also blunt the knife and force you to sharpen it first, but I can only have so much sympathy for those who ignore warnings and cry later when it is harder to play piano.

In the case of a big public site, if you're a professional that gets paid to set these things up, knowing what should be on the inside of the firewall and what should be on the outside of the firewall has to be job 1, and Rule #1 that goes with that should be "it goes on the inside." Rule #2 should be "see Rule #1." Rule #3 is, "are you absolutely, positively, stake your reputation, your job, you life on it sure it needs to be exposed to the outside? Still, unless the site fails to operate, see Rule #1." Rules 1 & 2 are also a good for setting up a home network, too.

Re:I fail to see why this is news

[jon3k]

Because in a lot of scenarios you might have a memcached server that uses public address space. You'd just use iptables to only allow ssh for remote management and then only allow your webservers to connect on your memcached port. No need for a big expensive hardware firewall. But I guess a warning wouldn't _hurt_ anything? Why not just "***WARNING - Only your webservers should be able to connect to memcache!"

Re:I fail to see why this is news

[apoc.famine]

That's like saying knives should be designed to be dull, because users are idiots, and will cut themselves.

It's not in any way the programmer's responsibility to hold the hands of idiots so they don't hurt themselves. The people who wrote this wrote a solid programs, and IN THE DOCS noted that if you set this up like this, you were an idiot.

Come on... we have far too much nanny-state as it is. My coffee has a warning on it that it's hot, my keyboard has a warning on it about RSI, my girlfriends hair straightener has a warning on it that it can burn you because it's hot....do we really need to extend this to software?

I forget who said it, but some comedian said we should remove the warnings from everything, and let nature sort it out. Same for software. If you're not smart enough to run a website, perhaps you need a painful pointing out of that.

Re:I fail to see why this is news

[SanityInAnarchy]

extremely crappy in communicating what they export to the world,

If you're an admin -- particularly one for, say, PBS -- and you're setting up a server, don't you think it'd be worth asking how that server does its communication? And it's not like this is undocumented -- from the wiki page on setting up a server:

By default memcached listens on TCP and UDP ports, both 11211. -l allows you to bind to specific interfaces or IP addresses. Memcached does not spend much, if any, effort in ensuring its defensibility from random internet connections. So you must not expose memcached directly to the internet, or otherwise any untrusted users. Using SASL authentication here helps, but should not be totally trusted.

And maybe I am just "clever", but this was pretty much obvious to me within about two minutes of learning of memcached -- I'm amazed anyone actually ran it this way. Sorry, but this isn't a case of poor UI design -- if you're employed as a system administrator, it's your job to know this shit.

Re:I fail to see why this is news

[SanityInAnarchy]

More than that, once installed, it defaults to not being used at all.

You have to actively develop and deploy software which uses and trusts memcached before this is a security hazard.

In other words, while it might be nice if it was secure by default, you now need two groups of people to fuck up at the same time -- system administrators and software developers -- both of whom, as part of their job, are supposed to think about security.

Re:I fail to see why this is news

[Bigjeff5]

I'm sorry, but I don't see how a system with absolutely no security can somehow be intrinsically more secure than a system without security.

The same security rules apply no matter what systems you use. If you're too stupid to understand basic security practices, then even Microsoft can't save you (though they do at least try).

Re:I fail to see why this is news

[Bigjeff5]

You secure memcached the same way you secure any cache: by restricting direct access. All you have to do is put a proxy between the memcached server/instance and the internet and the problem is solved. If you don't have command-level access to the memcached instance you can't mine it, period.

Many very large websites don't bother to do this, which is very bad security practice, as the slideshow clearly demonstrates.

The weakness isn't really with memcached. To say memcache is insecure is like saying your documents are insecure because they aren't encrypted. Yeah, you could look at it that way if you wanted to, but the real problem is the access to the documents. Restrict the access, and the problem goes away. Same with memcached. It's designed to be very fast, and to facilitate that it has no security at all. So you need to secure it externally - i.e. firewalls and proxies, among other things.

Re:I fail to see why this is news

[bjourne]

Yes memcached defaults to only accepting connections from the local address. From memcached.conf:

# Specify which IP address to listen on. The default is to listen on all IP addresses # This parameter is one of the only security measures that memcached has, so make sure # it's listening on a firewalled interface. -l 127.0.0.1

Re:I fail to see why this is news

[DiegoBravo]

With your philosophy, MySQL wouldn't be where it is.

Re:I fail to see why this is news

[Rudolf]

Yes memcached defaults to only accepting connections from the local address. From memcached.conf

What you posted does not say the default is localhost. It says the opposite:
"# Specify which IP address to listen on. The default is to listen on all IP addresses "

Re:I fail to see why this is news

[beanluc]

What you posted does not say the default is localhost.

That describes the default behavior. The GP showed us the default config, which does indeed specify localhost.

Re:I fail to see why this is news

[Anonymous Coward]

Yes, and that would be a good thing. The best place for MySQL is at the bottom of a lake attached to a concrete block.

Re:I fail to see why this is news

[bjourne]

If you have not specified otherwise (e.g. if /etc/memcached.conf is missing), memcached listens on all interfaces yes. If you install memcached from a Debian-derived distros package repository, then it will create a default /etc/memcached.conf file with the above, default, configuration specified.

Re:I fail to see why this is news

[MikeFM]

It's pointless to design for the biggest idiots you can find because they keep designing better idiots. Memcached isn't an app for average users. It's a programmers tool. If people that should know better are still idiots then let the damage be on their heads. Why waste resources trying to save idiots from themselves.

I say we let people self destruct early so that we can stop wasting resources. Let bad developers get themselves fired, let drug users overdose, etc. Eliminate the chaff instead of coddling everyone.

Re:I fail to see why this is news

[MikeFM]

Maybe the problem is that Linux has gotten to easy to use. We should go back to requiring everyone download source files and compile everything themselves. Call it an intelligence test.

Re:I fail to see why this is news

[turbidostato]

"You secure memcached the same way you secure any cache: by restricting direct access."

Which is exactly the parent's point. That's why he says it should default to listening only on 127.0.0.1: in order for the default install to restrict direct access.

Re:I fail to see why this is news

[drsmithy]

The MS way usually is to pretend that it's easy. The docs consist of a leaflet that explains how to open your cd drive and place the cd the right side up, and how to handle the mouse to make the cursor move to the "Ok" button.

Can you link to an example of this ?

Re:I fail to see why this is news

[Zancarius]

That describes the default behavior. The GP showed us the default config, which does indeed specify localhost.

For that distribution. Some distributions default to all available interfaces, thus the Memcached default.

This is also why it is important to verify software configurations before running them.

Re:I fail to see why this is news

[enoz]

I forget who said it, but some comedian said we should remove the warnings from everything, and let nature sort it out.

I have a hunch it was George Carlin, maybe someone can find the video evidence.

Re:I fail to see why this is news

[Unoti]

You and the some other ancestors are not understanding the distributed/shared nature of memcached, and how it needs to be accessed from more than one host except in the most trivial use cases.

Re:I fail to see why this is news

[Ash Vince]

All this is is stupid admins doing stupid things story and those are dime a dozen.

I'll say. I have been looking at using memcached in conjunction with the mod_auth memcache module. I know it exists and have done very cursory investigation. I do not have a working prototype yet but I do know my predecessor built one a year or so ago. My very cursory glance already made it startlingly clear that this would be an issue, any competent sysadmin really should have spotted the same thing.

Let me see if I understand this

[Anonymous Coward]

Let me see if I understand this and can boil it down:

Memcache allows anyone to overwrite a cache instance. Seriously? It does not authenticate a write to the cache? And they didn't see this as a problem when desgining memcache? Really?

Re:Let me see if I understand this

[Firehed]

Memcache's one purpose in life is to be as fast as possible. It makes perfect sense for it to drop the overhead of authentication and leave it on the server operator's head to not make it publicly accessible. It's not rare to strip out MySQL's authentication layer (and presumably the same for other DBs) for a speedup when your DB server is sitting behind a firewall.

Re:Let me see if I understand this

[Anonymous Coward]

I've dealt with a lot of large production databases, Oracle, SQL Server, MySQL, PostgreSQL, and similar. I've never seen the authentication layer striped out. These were pretty performance hungry applications as well, though perhaps not that many quick connections, but still the overhead is so small, that when you're communicating internally, it's generally not a problem. For companies like Google I could see this being a problem, but for MOST instances of these servers, I wouldn't suggest people do it.

At the very least, for that extra micro-second of overhead, it means if some cracker gets into your network, you don't have to worry about that as much (not that you don't have to worry, hell, some crackers in your network!).

Also, wouldn't you at least want to restrict it from employees that shouldn't be fucking with it. Granted companies whose purpose it is to develop this application, likely want almost everyone having access, but most instances you wouldn't want this.

None the less, this is off topic, and the article is retarded.

Re:Let me see if I understand this

[riskeetee]

You don't leave the keys in the Batmobile when it's outside of the cave. That's just asking for trouble.

Re:Let me see if I understand this

[Anonymous Coward]

I'm guessing you've never used it and have no idea why it would be useful.

Re:Let me see if I understand this

[StripedCow]

Yes, totally true. But it would still have been better to have authentication enabled by default. The user who actually knows what he is doing can then disable it.

Re:Let me see if I understand this

[j_sp_r]

Because of MySQL injection attacks and other problems, it is nice to have different access rights for different parts of your site. So I would not recommend stripping out the auth layer, especially if you have more than 1 DB running.

Even the best programmers and practices can create bugs.

Re:Let me see if I understand this

[Anonymous Coward]

Jesus Tapdancing Christ, they explicitly say that in the doc(s) where they discuss design decisions. They can't use stunnel or blah or blah2 or blah3?

"It does not authenticate a write to the cache? And they didn't see this as a problem when desgining memcache? Really?"

Yeah, they saw it and they saw their site and their systems and saw that they did not require that feature for themselves - they weren't creating memcache for charity to donate to the world at large - ffs. Say what you want about livejournal but they created a bunch of high performance distributed system tools - gearman, memcache, mogileFS, etc that allowed anyone to build massive social website prior to them, the tools were not there. Now I am sure some smug historical revisionists was come and explain how these things were all built in SNOBOL in the 1950s then reinvented in Java as apache foundation project with simple 12,000 line xml config files that future generations will claim are alien code for time travel devices but those historical revisionists would be dead wrong.

Re:Let me see if I understand this

[PIBM]

It's open source, if you want to add a layer of protection, go away. For the rest of us, we are using memcached for maximum speed, no such thing in.

Lets say you add a layer of protection to your memcache, it now requires a password to get in. The hacker is in your network, so he can sniff those packets. Where's your protection ? Ok, lets say you add in some crypto to prevent sniffing attack. He's in your network, on your computers, reading your files. He then parse your code and grab the password.

So, what have you been able to achieve exactly ? Nothing beside slow downs everyone downs. People should just learn to use the tools at their disposal. And yes, you can kill yourself with a car. Should we lock them all down to 10mph ?

For the parent, here's memcached help. Notice that even if it's ADRANY, you have it in your face should you be on an unsecure network. And besides, locking it to your ip won't prevent someone inside your network from getting the content.

  memcached -help
memcached 1.2.6
-p TCP port number to listen on (default: 11211)
-U UDP port number to listen on (default: 0, off)
-s unix socket path to listen on (disables network support)
-a access mask for unix socket, in octal (default 0700)
-l interface to listen on, default is INDRR_ANY
-d run as a daemon
-r maximize core file limit
-u assume identity of (only when run as root)
-m max memory to use for items in megabytes, default is 64 MB
-M return error on memory exhausted (rather than removing items)
-c max simultaneous connections, default is 1024
-k lock down all paged memory. Note that there is a
                            limit on how much memory you may lock. Trying to
                            allocate more than that would fail, so be sure you
                            set the limit correctly for the user you started
                            the daemon with (not for -u user;
                            under sh this is done with 'ulimit -S -l NUM_KB').
-v verbose (print errors/warnings while in event loop)
-vv very verbose (also print client commands/reponses)
-h print this help and exit
-i print memcached and libevent license
-b run a managed instanced (mnemonic: buckets)
-P save PID in , only used with -d option
-f chunk size growth factor, default 1.25
-n minimum space allocated for key+value+flags, default 48

Re:Let me see if I understand this

[MikeFM]

I don't allow random systems to communicate with my database. It is only accessed directly from localhost. Any other system has to go through a predefined RPC function that sanitizes inputs and outputs and does a very specific task. Even that access is only granted to a few systems and never over the public network. I usually still use the built-in security too but I would consider removing it as I consider it almost useless.

Re:Let me see if I understand this

[lukas84]

Besides, one could easily secure this by using IPsec and machine certificates.

I'm not a Linux admin, but that's how i would do it on Windows.

Re:Let me see if I understand this

[ewanm89]

I always use MySQL permissions to separate databases so that one application can't access another database so easily, I also never allow access to it from the internet directly, same policy applies to all server software, unless it's made to be directly connected to and needs to then it stays on the internal network or loopback.

As for all programmers having bugs, even Rivest et all's MD6 reference implentation code sent for the SHA3 NIST competition had a buffer overflow in it: slashdot.org, and while we all should say security experts should know better, these mistakes happen from time to time.

Re:Let me see if I understand this

[jon3k]

Sounds like a great reason to use something like SSL between memcached and the webservers. Forward the incoming SSL connections to memcached back to the memcached port over a loopback address. Now you can have authentication and encryption, if you want it, without adding any complexity to memcached. That way if someone wants to run it locally on a webserver you haven't really added any complexity.

Re:Let me see if I understand this

[Anonymous Coward]

What the fuck is a "MySQL injection attack"?

Re:Let me see if I understand this

[allo]

injecting my SQL instead of the site SQL. see SQL injection.

Re:Let me see if I understand this

[Anonymous Coward]

I'd say that "injecting" MySQL on to a server is a pretty malicious thing to do.

Re:Let me see if I understand this

[Firehed]

I wouldn't recommend stripping out the authentication layer either, I merely said it's been done. That being said, if your app can suffer from SQL injection, you've got bigger problems. If your app is to the point of needing to strip out DB authentication for some extra performance (and if it is, you should probably be taking a different approach... such as memcache, the point of this whole discussion), you damn well better have all sorts of tests in place to make sure that you don't introduce problems that severe.

More Boiled and Distilled.

[SuperKendall]

Memcache allows anyone to overwrite a cache instance. Seriously? It does not authenticate a write to the cache? And they didn't see this as a problem when desgining memcache? Really?

Anyone can write on your underwear too, if you are stupid enough to wear it outside your pants.

Is that an underwear design flaw?

Re:More Boiled and Distilled.

[Farmer Tim]

Best. Analogy. Ever.

Re:More Boiled and Distilled.

[pushing-robot]

As spokesman for the Justice League, I say yes.

Re:More Boiled and Distilled.

[davester666]

No. Car. Was. Involved.

Re:More Boiled and Distilled.

[outsider007]

That's actually more of a feature.

Re:More Boiled and Distilled.

[Farmer Tim]

That's. Why.

Re:More Boiled and Distilled.

[Lord Kano]

Is that an underwear design flaw?

Well, kinda. That's why I don't use them.

LK

Re:More Boiled and Distilled.

[sjames]

Boiled and distilled underwear....Ewwwww.

Re:More Boiled and Distilled.

[imsabbel]

Tell that superman face to face and lets see how strong your point is :)

Re:More Boiled and Distilled.

[Anonymous Coward]

Who said the underwear did not have cars on them?

Re:More Boiled and Distilled.

[icebraining]

No, they obviously have funny cartoon animals.

Re:More Boiled and Distilled.

[human-cyborg]

Hear, hear!

Re:More Boiled and Distilled.

[debruce]

Do you like my whiteboard underwear? In comes with a dry erase marker in there, just feel around.

Re:More Boiled and Distilled.

[Anonymous Coward]

my pants keep falling down, you insenstive clod!

A few clarifications

[marcoslaviero]

In terms of the vendors identified, Bit.ly, GoWalla and Pbs were notified. Bit.ly and GoWalla repaired the flaws within minutes. I am not aware of Pbs repairing the issue. This talk seems to have struck a chord which I can't really explain (suggestions welcome). Yes, exposing your memcached's is bad (the talk shows just how bad), but it's not a clever find to discover them. [fd: that's my name on the slides]

Re:A few clarifications

[gabba_gabba_hey]

It's just odd to me that people don't read the manual. Perhaps the memcached authors could make the fact that it doesn't do any sort of authentication more obvious. Still, it was quite plain to me when setting it up that it needs to be firewalled - just like everything else that you don't want to be accessed from outside.

Re:A few clarifications

[marcoslaviero]

There's a deeper issue at play here as it relates to shifting apps and platforms away from your own hardware/networks. Developers are now often responsible for deploying apps onto cloud systems where they don't have experience with network-security or the tools for protecting network-based services, and this is an obvious difference from the traditional network/app split that occurs in most corporates. It doesn't help that memcached (by default) binds to * but they do make this pretty clear (also, remote enumeration of the cache is genuinely a debug feature).

Man pages help, but when the defaults don't aid developers we need to a rethink both of the software (memcached) and the systems were it's not running securely (cloud platforms).

Re:A few clarifications

[gabba_gabba_hey]

Excellent points and I don't disagree with you at all.

I come from a different background where I would never think of exposing this sort of thing and I think that also needs to be drilled into anyone dealing with networked apps. Security 101 never really changes.

Re:A few clarifications

[DrSkwid]

The deeper issue that's exposed is that your mythical developers are ignoring the fact that most attacks come from inside. Mostly through rouge employees or compromised machines. They are operating on the fallacy that you can have one big happy subnet and everybody attached it is a team player.

I don't use memcached but it sounds like one should have your memcached server machines on their own LAN and multiple network cards in the client machines.

It's really not that difficult, it's just :

1 developers are not security minded
2 people are lazy
3 people have a false sense of security
4 people have an overblown confidence in their own ability
5 the company hasn't been burned yet

Most things happen once 5 has changed - when was the last time you ran your disaster recovery plan?

Re:A few clarifications

[marcoslaviero]

Indeed, i've had confirmation from a number of other security people about encountering memcacheds internal to organisations, and your points are all valid for internal installations. The external-facing instances we found are not explained by assuming the internal network is trusted, hence the point about developers taking an increasing network-security role.

Re:A few clarifications

[IAmGarethAdams]

Mostly through rouge employees

Luckily, they often get caught red-handed.

Re:A few clarifications

[Anonymous Coward]

Mostly through rouge employees or compromised machines.

Bet that leaves them red in the face with embarrassment.

Re:A few clarifications

[MikeFM]

I refuse to give even the boss access. I've even designed systems that wouldn't give me access except under very specific conditions such as a second admin also entering their credentials from a known system during business hours. Some data needs to be treated with extreme care.

Re:A few clarifications

[IMightB]

Terry? Is that you?

Re:A few clarifications

[Anonymous Coward]

+5 Informative? Whoosh...

Re:A few clarifications

[Anonymous Coward]

Mostly through rouge employees

Luckily, they often get caught red-handed.

I think the mods are color-blind to your jokes.

Re:A few clarifications

[Anonymous Coward]

The cloud systems are adapting well.

Heroku (a Ruby cloud platform) offers a Memcache server that is wrapped up in SASL. There's no way to use it insecurely and there's no way to access another users's keys. The Memcache server and the Ruby gem is offered by Northscale directly, the company formed by the leaders of the Memcache project.

Admin or distro?

[shish]

Debian's default config says:

# Specify which IP address to listen on. The default is to listen on all IP addresses
# This parameter is one of the only security measures that memcached has, so make sure
# it's listening on a firewalled interface.
-l 127.0.0.1

Are there any distros that don't have it locked down by default? I would hope not, but if something has it insecure out of the box with no warning that might explain it... (though a good sysadmin would firewall all internal services, whether the documentation tells them to or not)

Re:Admin or distro?

[Anonymous Coward]

Admin. The memcached man page for 1.4.5 is 126 lines long in a standard terminal.

You have no business administering a machine, if you can't be bothered to read 126 lines of text, don't know what the fuck you're doing, and are relying on the defaults of your distro to keep you safe.

Especially if you're getting paid to run large public services like the ones described in the presentation.

Re:Admin or distro?

[buchner.johannes]

From http://linux.die.net/man/1/memcached :

-l
        Listen on ; default to INDRR_ANY. This is an important option to consider as there is no other way to secure the installation. Binding to an internal or firewalled network interface is suggested.

yeah ... it's hard to blame the memcached guys

Re:Admin or distro?

[TheRaven64]

default to INDRR_ANY

And this is why they're to blame. Default should be the loopback, and enabling external access should require explicit configuration.

Re:Admin or distro?

[akanouras]

I've met smartpants web developers that felt they were getting a more tailored-to-their-system installation by downloading tarballs and compiling the stack all the way down.

Nevermind updates or security though...

Re:Admin or distro?

[Paul Jakma]

So what if you want to run memcached on a multi-user machine?

It's slightly mad that software like this, which is designed without security, would use TCP per default, instead of local Unix sockets (access to which can be controlled with standard Unix filesystem permissions on the containing directory (careful about relying on permissions on the socket itself have any effect - not portable)). Indeed, it doesn't even seem to support Unix sockets (would be a trivial patch though).

Re:Admin or distro?

[MikeFM]

That's why you create your own packages and have a system to add your mods to the public versions of the packages. I don't custom compile everything but the components most key to what I'm doing are often custom compiled because the public packaged lack the functionality I need.

Re:Admin or distro?

[MikeFM]

Don't run it on a multi-user machine. That's why we invented server farms and virtualization. Nothing should be on a multi-user machine.

Re:Admin or distro?

[akanouras]

Your way is the proper one to go about it, provided you stay on top of security updates by yourself.

In the aforementioned developers' case, they just followed some shitty "How to install LAMP" "howto" and never bothered to check if their distro provided the relevant packages. On top of that, they had absolutely no clue on how to provide a secure or even performance-optimised configuration for the custom compiled packages. If all that wasn't enough, they also never applied security updates after the initial installation.

Re:Admin or distro?

[fearlezz]

Sorry, I don't agree with that.
Memcached is not meant for single-server configurations. It's meant to combine the memory of all connected servers, so that if you connect 10 servers with 4GB each, you get 40GB of cache. This won't work if you only allow loopback connections.
The admin should block memcached connections on the outer interface. And set the default firewall policy to DENY to start with.

Re:Admin or distro?

[Culture20]

Nothing should be on a multi-user machine.

Then Unix shouldn't be multi-user. Somehow I think there are practical issues with that sentiment.

Re:Admin or distro?

[bill_mcgonigle]

Memcached is not meant for single-server configurations

That's silly, it's a generic object store. There's no reason not to use it to cache expensive local operations. Of course it shines across a farm of caches, but the server mapping hash will work just fine with one machine.

If you're a startup with just one webserver and starting to hit performance problems, memcached will likely buy you a few more months.

Going from one server to two is hard, three is a bit more work, and after three it's roughly all the same until you start adding more data centers and then it's all the same until you're Facebook. Taking on that 'hard' expense too early would be a poor allocation of resources.

Re:Admin or distro?

[beanluc]

Default should be the loopback

Default config IS the loopback. You have to deliberately change it.

Default behavior is another thing.

Re:Admin or distro?

[Anonymous Coward]

Indeed, it doesn't even seem to support Unix sockets (would be a trivial patch though).

If you are running memcached on a machine that's serving multiple unrelated users, YOU ARE DOING IT WRONG.

If you are running memcached on the same machine as the service that uses it (needed for Unix sockets), YOU ARE DOING IT WRONG.

The whole point of memcached is to provide a fast, scalable interface to metric fucktons of RAM. If you're running your whole infrastructure from a single box (or worse, a shared box) YAGNI.

mysql/ssh/postgres/ftp/etc. opens security hole

[pyalot]

Yeah like... all these open a socket to listen to, and man, if somebody logs into it, they're like, owning your server! Geeze, what where mysql/ssh/postgres/ftp/etc. thinking?!!!

Re:mysql/ssh/postgres/ftp/etc. opens security hole

[Anonymous Coward]

They all have authentication and authorisation mechanisms. Memcached, by design, does not. This is clearly stated on the memcached site and in its config file.

Re:mysql/ssh/postgres/ftp/etc. opens security hole

[Lord Bitman]

..and presumably a developer would notice when at some point they write connect_to_memcached() and don't need to pass in a username or password..

Re:mysql/ssh/postgres/ftp/etc. opens security hole

[Anonymous Coward]

Yeah, if it was a web developer, they'd notice and would then think, "Shit, I wish PostgreSQL was like that. It's such a pain in the ass having to give it a username and password."

Re:mysql/ssh/postgres/ftp/etc. opens security hole

[pyalot]

If you somehow think that having user/password authentication would prevent you from being owned, then you're even more incompetent then the admins of the exploited memcached servers.

Re:mysql/ssh/postgres/ftp/etc. opens security hole

[nacturation]

Yeah, if it was a web developer, they'd notice and would then think, "Shit, I wish PostgreSQL was like that. It's such a pain in the ass having to give it a username and password."

Right... because if someone exploits a security bug in your app (I know... your apps have no bugs) and is able to 'drop table users;' because the DB had no security mechanism, that would really rock. Instead, we have to suffer the tedium of only granting the privileges that the app needs and locking down all other functionality.

Flash required

[Anonymous Coward]

Can we have a Flash warning like we have on PDFs? At least I can open PDFs but not Flash.

Re:Flash required

[Skapare]

Flash for a freaking slide show? This is too easily doable without Flash. Even Javascript is overkill. And it requires Flash version 9 as if no earlier Flash version could do a slide show.

Re:Flash required

[Bigjeff5]

I don't understand, how is writing a JavaScript slideshow easier than hitting "Upload" and then "Embed" at SlideShare?

Seriously, you've got way too much time on your hands.

Just upgrade your fucking flash already, nobody else cares.

Re:Flash required

[Anonymous Coward]

You have to be trolling me. SlideShare should implement their slideshow in Javascript, duh! Upgrade your fucking brain already.

Re:Flash required

[mzs]

Sure, I used bugmenot:

http://s3.amazonaws.com/ppt-download/slaviero2010memcache-100803232447-phpapp02.pdf?Signature=CatPvvPgJL3nxkECUrqgQ%2ByH4OE%3D&Expires=1281288118&AWSAccessKeyId=AKIAJLJT267DEGKZDHEQ

PS: I also have no Flash installed and felt your pain as well.

Re:Flash required

[Anonymous Coward]

I care enough that I couldn't read TFA. I had to read the long version instead.

I suppose client-side scripting is needed for animated transitions that I don't care about.

Never underestimate the ignorance of Web 2.0 devs.

[Anonymous Coward]

Web 2.0 is an interesting phenomenon. In many ways, it's nothing but embracing ignorance, bad ideas, the worst possible way of doing things, and the stupidest people (both users and developers!) around.

First, let's ignore how Web 2.0 sites cater to the stupidest users, and because of this are willing to abuse private data given to them by their users. That's a whole different story. Let's just focus on the technology here.

The first thing we see is that the use of PHP is prevalent among Web 2.0 sites. If there's one thing that PHP is "good" for, it is quickly developing poorly-performing and insecure web applications, written by programmers with a minimum of technical knowledge. That's why it has blossomed, so to speak, among the Web 2.0 development community. It caters well to their dislike of quality and their lack of technical knowledge. In many cases, the developers are too ignorant to know that there are better options out there, let alone identify the flaws and problems with their own code.

The second thing we see is a lack of knowledge about data storage and scalability. This has manifest itself as the rise of the NoSQL movement. NoSQL is all about throwing away decades of accumulated knowledge concerning relational databases, and replacing it with hash tables. It's about embracing ignorance.

Many of the most popular Web 2.0 sites moved to NoSQL "databases" claiming it was necessary to do so for "scalability reasons", but that's obviously bunk after reading the many articles they publish describing their architectures. We find out that they're clueless about such basic things as indexes, and in some cases even perform joins by pulling ALL of the data back to their PHP code, and manually joining it there.

Worse yet, these Web 2.0 sites are prone to failure. Twitter's "fail whale" has become widely known, and reddit suffers from frequent downtime, including a lengthy incident just yesterday. What's funniest, however, is that the traffic they face is minuscule compared to the activity that many POS and financial systems, or even "Web 1.0" sites, handle with ease because they're developed by knowledgeable developers who know how to properly use relational databases.

Given that the Web 2.0 developers thrive on ignorance and doing things as incorrectly as possible, it's no wonder some of their flagship technologies and sites would have such glaring security holes. That's just what's to be expected from people who don't have a fucking clue about what they're doing.

Food Analogy

[TimTucker]

Think of it like this:

System that is never intended to be secure: plastic apple with a warning label stating "THIS IS NOT FOOD"

System that should be secure, but isn't: apple full of worms

You're not going to have a good experience biting into either apple, but there's definitely a difference in the expectations that someone would have when looking at them.

In that case...

[Zancarius]

I believe the point here is that software designers should assume that in terms of security their users are complete idiots and WILL install and setup the program in an unsecure manner unless they are specifically beat over the head with the notion that what they are doing is BAD!

Perhaps utilities like rm should be rewritten to use -i (or similar) by default then? Or how about we ought to remove them outright since they might be used incorrectly?

Honestly, it's not up to the developer to make any such assumptions. Any piece of software can be used "insecurely" if configured incorrectly (yes, even IIS). Ultimately, you as the user bare the responsibility for ensuring that your system and your data are correctly secured.

As far as beaten over the head, the default configuration for Memcached that ships with Gentoo has this note immediately over the configuration for what interfaces it should listen on:

#Listen for connections on what address?
# If this is empty, memcached will listen on 0.0.0.0
# be sure you have a firewall in place!

I'm sure other distros have similar notes.

Re:In that case...

[turbidostato]

"Any piece of software can be used "insecurely" if configured incorrectly"

Not that you don't have a point, but you are doing nothing but supporting the argument you tried to debunk.

Note that your point stands on the "if configured incorrectly" while the parent's is "it should be distrubuted on such a state that the sysadmin would be forced to do *any* kind of configuration prior for it to work" (thus giving the option to even "configure" it, either correctly or incorrectly).

I think the parent's point of view is better.

"As far as beaten over the head, the default configuration for Memcached that ships with Gentoo has this note immediately over the configuration for what interfaces it should listen on:
#Listen for connections on what address?
# If this is empty, memcached will listen on 0.0.0.0
# be sure you have a firewall in place!
I'm sure other distros have similar notes."

Ever tried an automated install?

Just to probe a point I installed memcached as packaged on Debian:

aptitude install memcached

And just as I expected, Debian's default configuration is to only listen on the loopback. Bet what I do think is the safest default config between that from Gentoo and the one from Debian.

Re:In that case...

[Zancarius]

Note that your point stands on the "if configured incorrectly" while the parent's is "it should be distrubuted on such a state that the sysadmin would be forced to do *any* kind of configuration prior for it to work" (thus giving the option to even "configure" it, either correctly or incorrectly).

No, you're just splitting hairs. My point is that software can be configured incorrectly and that it isn't the developers' fault if that happens. I believe the parent's point of view (and the general view espoused further up the comment chain) was that software should be distributed in such a manner that it cannot be configured insecurely. I'm hoping I just misunderstood what you wrote and that you were simply disagreeing for the sake of, well, disagreement. Or trolling. Regardless, I'd encourage you to read up the chain so you may better understand my point (the parent was just convenient fodder since his view was mentioned numerous times, and I got fed up with this notion that it's the responsibility of the developers to ensure no configuration can be abused).

N.B. that if I were to split hairs, I'd point out that your comment that implies (I think? You may have missed a word or two while typing it out) that sysadmins shouldn't be forced to do anything to make a particular software install work. That's a fairly short order--almost all software works out of the box as expected, but whether it works securely across all "default" installs is another matter entirely and one that, again, is the responsibility of the user to verify. But the point goes far beyond that, and I'm somewhat disappointed that you seem to agree with the general sentiment that post-install configuration should almost never happen except in rare outlying circumstances (hint: this isn't likely to happen, particularly with the vast array of requirements, disparate hardware, and so forth). Note, also, that if this notion of "secure by default" is taken to the extreme, you wind up with OpenBSD. OpenBSD is secure by default. You also cannot use it without installing and configuring additional software (thus no longer having a "default" install), because it comes with only OpenSSH installed by default! In other words, we're just encountering a difference of philosophies, and I ascribe to the one that makes the burden of configuration your responsibility as a user, not mine as a developer.

And really, this point it doesn't matter at all, because someone will still find a way to configure it incorrectly, regardless of how well it was written (don't believe me? you'd be surprised what lurks out there in the wild) or how strict the default configuration is. There are people out there who will tinker with things, including things they may not fully understand, and configure it in a manner that is suddenly insecure. That is my fundamental point; the point of the OP and those subsequently up the chain is that the responsibility should rest upon the shoulders of the developers to ensure their software cannot be configured incorrectly--that's the whole point many of the posts here are arguing against Memcached for! There seems to be so much heartache that Memcached (which doesn't require authentication) can be configured in a manner that it exposes sensitive data to the Interwebs. Guess what? If it does, that is the fault of the administrator and not the fault of Memcached or its developers.

I'm not sure how else to state that more clearly. But I'll try:

Responsibility rests on the person who installs the software to ensure it is configured securely for their needs, not the developers nor the package maintainers. The only point in time, IMO, where such responsibility is the developers' is when it becomes an issue of out-of-band behavior; e.g. something that is unexpected and unintended (a bug or security flaw) and not fundamentally something that is the result of a misconfiguration. In short:

Misconfigurations = user's responsibility (even if t

Re:In that case...

[totally bogus dude]

I think that in your rush to rebuke the parent with scathing sarcasm, you have misinterpreted their point. I don't see anyone suggesting the software should install in a secure, perfectly functional manner, with no configuration whatsoever. The only quote I saw was saying that it's preferable for the software to be installed without a working configuration, so the administrator must explicitly configure it for their purposes. If they fail to configure it securely in the process, that's their own failing. What software shouldn't do is install a configuration which is "insecure by default" but actually functional, as this encourages (or at least, permits) clueless admins from installing it, testing it to see if it works for their own purposes, and then forgetting about it. This appears to gel perfectly well with your philosophy, i.e. it's the system administrator's responsibility to ensure their system is configurable correctly and securely.

The Gentoo example appears to install a functional but insecure configuration by default. If the user simply starts the service (or possibly reboots the server without doing further configuration and it auto-starts at the next boot), then tests their application, they will be overjoyed that it's working and move on to their next task. The admin may not be aware that memcached is not designed to be exposed directly to the internet. Or, they may doing a small install with the web server being on the same machine, and it may never even occur to them that the software might be listening on all network interfaces (given they've configured their web application to talk it to via loopback). Or, they may have simply had a momentary lapse in concentration and forgotten about it. People are fallible, after all.

The Debian default install, on the other hand, is reasonably safe against human forgetfulness. If they're only configuring a local instance, then the configuration will work and they'll move on like in the Gentoo example - but without having the service unintentionally exposed to all network interfaces. If they're configuring it to be network-accessible, then they have to modify the configuration, and therefore received a reminder that they need to take steps to protect the service.

Re:In that case...

[azmodean 1]

Seriously? Your point is that Gentoo doesn't do enough hand-holding? This is a distribution that has built its reputation around NOT doing excessive hand-holding.

That having been said, Gentoo does leave services that have security implications unconfigured, but I'm having trouble understanding how memcached is such a service. This isn't a system configuration issue, but a network layout issue.

Finally, there is this:

The admin may not be aware that memcached is not designed to be exposed directly to the internet.

Huh? You're administering an internet-facing server, and installing an internet-facing service, and you haven't checked that it is in fact intended to be internet-facing? If you're that oblivious, how the hell is making you fill out a config file going to help you? Captain McBlivious is still going to just blindly add configuration to the system until it starts doing what he expected, which is providing caching services.

Re:In that case...

[totally bogus dude]

True, it's an unusual network configuration to be "exposed by default"; however I still haven't seen any rational explanation why "listen on all interfaces" is a better default than "listen only on loopback".

If you're that oblivious, how the hell is making you fill out a config file going to help you?

Because the config file section that needs to be modified has comments explaining the risk? Not everyone is going to go and read external documentation on a piece of software before installing it. If you're going to do that, why even use a distribution in the first place? Any documentation you read is likely to be helpful, but may refer to paths that are different because the maintainer for your distro's version of the software has changed it; or may be for a different version; or may not have certain patches applied; and may very likely have a different default configuration.

It's very common for people's first real exposure to the configuration of a piece of a software to come after having installed it from their distribution's repository. It makes sense for that initial configuration to be as conservative as it can reasonably be made, and to guide the administrator through the correct configuration of it to meet their needs.

It could well be a cultural thing though; perhaps it's normal for Gentoo to install services so that they're configured to listen on all interfaces by default. If you install PostgreSQL or MySQL, can you then immediately connect to them from an external host? If you install squid does it default to proxying traffic for your local network? If so, then perhaps the behaviour of memcached is entirely expected; but I still think it's a sub-optimal default that carries some risk for little or no benefit.

Re:In that case...

[Zancarius]

The Gentoo example appears to install a functional but insecure configuration by default. If the user simply starts the service (or possibly reboots the server without doing further configuration and it auto-starts at the next boot), then tests their application, they will be overjoyed that it's working and move on to their next task. The admin may not be aware that memcached is not designed to be exposed directly to the internet.

Insecure is in the eyes of the beholder. If the machine is on a firewalled network (e.g. inaccessible to the outside world) it is secure.

This whole point is moot, however, because Gentoo doesn't start services by default. You have to explicitly start them (that is, manually). Presumably, you understand the security implications of relying on preinstalled configurations to work precisely as expected. If you don't, shame on you and you have no business being a sysadmin in the first place. (No, really, you don't.)

I vehemently disagree on the premise that if someone knows enough to be running Memcached, he or she probably knows and understands enough to configure Memcached appropriately. By default, Memcached on most distributions I've seen is configured to use 64 megs of RAM for caching. If someone needs--really NEEDS--Memcached to be running, 64 megs may or may not suit their needs, so they will likely be reading the configuration file anyway.

FreeBSD goes a step further and reminds users that a particular package has installed a network service that may invoke security implications if it's enabled.

Or, they may doing a small install with the web server being on the same machine, and it may never even occur to them that the software might be listening on all network interfaces (given they've configured their web application to talk it to via loopback). Or, they may have simply had a momentary lapse in concentration and forgotten about it. People are fallible, after all.

Fallible? Certainly. Stupid? Possibly.

If someone is running a web server that is neither firewalled nor correctly configured, they probably shouldn't be running it in the first place. Better yet, they should probably switch to another vendor that takes sufficient precautions to assume the end user is too inept to verify their configurations in the first place. After all, if they're unwilling or unable to bother running something as simple as a firewall, they probably haven't bothered to configure SSH to accept key-based authentication, moved it to another port, and they're very likely to be one user away from having a successful SSH scan hit and possibly root their box. ("But Uncle Joe is really computer literate and wanted a shell! I had no idea he used "asdf" for his login "joe" on every box he connects to!")

As a side note, FreeBSD does roughly the same thing as Gentoo in that it runs Memcached entirely without configuration (meaning attaching to all interfaces) if you launch it from /usr/local/rc.d/memcached. Thusfar, that's two vendors that appear to ascribe to the same notion that correct configuration is the user's responsibility.

I must say that I agree, but I think your disappointment with my "scathing sarcasm" may have colored your opinion of my beliefs.

Re:In that case...

[Zancarius]

True, it's an unusual network configuration to be "exposed by default"; however I still haven't seen any rational explanation why "listen on all interfaces" is a better default than "listen only on loopback".

I can think of a number of network-facing software packages in the *nix world that are usually configured, by default, to listen on all interfaces, because that's usually what the end user wants. There are some particular issues I'd like to point out, but if you're feeling exceptionally annoyed you may find it more rewarding to skip to the last paragraph, as I answer more directly (and seriously) your point related to culture. Hopefully, you'll read this and take away some education related to other distributions outside the somewhat walled gardens of Debian. (As an aside, it does bug me that most Debian users tend to feel that Debian is the One True Distro and that differences of opinion to the contrary as expressed via other distributions are therefore wrong; that is the primary reason for my "scathing sarcasm" in an earlier post.)

But be warned! There is sarcasm ahead. You really should skip to the last paragraph.

Because the config file section that needs to be modified has comments explaining the risk? Not everyone is going to go and read external documentation on a piece of software before installing it.

They don't need to read external documentation if the configuration file explains the risks of a particular configuration. Furthermore, that same configuration file is installed with the application; therefore, if they were going to "read external documentation," chances are they'd be doing it after the installation and presumably after they've encountered the warning in the config. Such external documentation may or may not be read anyway since it would appear that opening something as simple as a textual configuration file in one's favorite editor is such a tremendous burden. I cannot imagine what great pains it would place upon the shoulders of such an individual to be forced to read manpages or other informative resources provided by the software developer(s).

Any documentation you read is likely to be helpful, but may refer to paths that are different because the maintainer for your distro's version of the software has changed it; or may be for a different version; or may not have certain patches applied; and may very likely have a different default configuration.

I'm not sure what distros you're using, but most of them tend to ship the manpages that the developer provided with the original distribution whenever you install the package, particularly for most major software. FreeBSD and Gentoo have identical manpages for the current version of Memcached, for instance.

On the other hand, your last point related to "different default configuration[s]" makes me chuckle. Since all distros are different, the only thing you're arguing is in favor of checking the configuration to ensure it does what is expected--which is what the end user should be doing in the first place, particularly in a mixed environment. After all, it might be different!

It's very common for people's first real exposure to the configuration of a piece of a software to come after having installed it from their distribution's repository. It makes sense for that initial configuration to be as conservative as it can reasonably be made, and to guide the administrator through the correct configuration of it to meet their needs.

This point bugs me, because it smacks of the hand-holding the poster you're replying to was stating Gentoo explicitly does NOT do.

First, some education about Memcached: It has no configuration file; those distributions that do provide one provide something that is effectively a wrapper around Memcached's command line arguments. Gentoo's configuration file comes largely without configuration (hence

Secure mamcached with the '-l' option on startup

[bokmann]

Read my blog about it here:

http://blog.codesherpas.com/on_the_path/2010/08/securing-memcache-in-2-minutes.html

This isn't a security problem - this is operating by design. If you are a memcache user and this is news to you, you need to read more about the tools you are using. I bet you have security problems beyond this one.

Re:Never underestimate the ignorance of Web 2.0 de

[Anonymous Coward]

You might be surprised to learn that the problems are slightly different than your ERP system, in that pretty much every field is either very large and required to grow to infinity.

You want to support picture uploads to your site. You can either:

1. Store them raw on disk and watch the file system collapse under the weight.
2. Make a table to store all the pictures in an RDBMS and then spend a lot of $$$$ making sure you have enough disk space on your SAN to store them all as well as create a method to sync that data to all of your data centers.
3. Store a pointer in your RDBMS to the hash keys in your DHT, which runs on significantly cheaper hardware and requires a lot less man power to keep running and is generally much easier to keep in sync.

At some point, the data gets so sharded that there is no "relational" involved anymore. So why store it in a database tuned for it?

Let me be the first to say

[Evets]

This whole conversation is stupid.

Re:Never underestimate the ignorance of Web 2.0 de

[totally bogus dude]

I have mod points, but the +1 troll modifier seems to be missing.

Mastodoniasis

[tepples]

How many FreeBSD fans attend Indiana University-Purdue University Fort Wayne?