Scaling Facebook To 140 Million Users

1sockchuck writes "Facebook now has 140 million users, and in recent weeks has been adding 600,000 new users a day. To keep pace with that growth, the Facebook engineering team has been tweaking its use of memcached, and says it can now handle 200,000 UDP requests per second. Facebook has detailed its refinements to memcached, which it hopes will be included in the official memcached repository. For now, their changes have been released to github."

Comments

Blaming Linux...

[TypoNAM]

Is it just me or does the entire first part of the article scream "Linux is to blame!" when they were discussing about dealing with UDP network overhead issues in their software? For example:

We discovered that under load on Linux, UDP performance was downright horrible. This is caused by considerable lock contention on the UDP socket lock when transmitting through a single socket from multiple threads. Fixing the kernel by breaking up the lock is not easy. Instead, we used separate UDP sockets for transmitting replies (with one of these reply sockets per thread). With this change, we were able to deploy UDP without compromising performance on the backend.

I bolded the quote to show what their real problem was. They had a shit load of threads trying to use a single socket and of course there was huge overhead involved due to the mutex lock (Semaphore on kernel side) on a shared resource (the socket). So they blame Linux instead of them selves for such a half-ass implementation of sending out packets from multiple threads with a single socket. They would have gotten the same exact result if they tried it with a single TCP connection socket and attempted to have multiple threads firing off packets with that. If you want multiple threads sending out packets use multiple sockets... Wow what a concept!

Sorry for my ranting, but it just pisses me off when moron programmers blame the operating system for their own stupidity.

Anyway, haven't nearly all MMOs gone with using UDP internally of the game cluster network and TCP externally to reduce latency and network overhead? So this is nothing new to me.

Re:Blaming Linux...

[imboboage0]

No... I don't think they were really blaming Linux. If anything, I'd say they were praising it for having the functionality to be modified to fit their needs. They admitted that the previous configuration they had wasn't ideal, and they fixed it. I think the important part here is that they used Linux to fix it, they continue to use Linux, they documented the fix, and now they are giving back to the OSS community with information on how they did it.

Re:Blaming Linux...

[blitzkrieg3]

They said that "on Linux, UDP performance was downright horrible."

This statement is just downright disingenuous and wrong. UDP performance in general on Linux is comparable or better than other Operating Systems. What he found out is that accessing a single UDP socket on Linux requires a lock, and that when trying to share that lock over multiple threads you have a performance issue. Welcome to intro level operating systems.

This has nothing to do with UDP performance, which I define as either throughput or in some cases packets per second. He then goes on to imply that he worked around some issues in Linux, when in actuality he attacked the problem from the wrong angle and through trial and error found the obvious solution. Why would you even think to use the same socket in a connectionless protocol like UDP in the first place?

I do agree that in general the article was written in more or less praise of Linux, but reading that sentence makes my blood boil.

Re:Blaming Linux...

[hesaigo999ca]

Too often the people that are left to explain the problem in detail to the press are not the engineers that worked on the solution for that problem. If we had a discussion with one of them, we would hear a totally different story!

Re:Blaming Linux...

[ranulf]

[...] So they blame Linux instead of them selves for such a half-ass implementation of sending out packets from multiple threads with a single socket.[...]

Sorry for my ranting, but it just pisses me off when moron programmers blame the operating system for their own stupidity.

The point is that it wasn't their own stupidity. They took someone's open source project and improved it so it could better handle high loads. I don't see them blaming Linux, I see them recognising the limitations of the system they are using and coming up with a solution and then sharing it. Normally, this is cause to say "Yay! Open source!" rather than calling them "moron programmers".

That is *not* a Facebook problem

[RMH101]

User is sent link, directed to website with malware payload, such as a 0-day IE exploit. User is running unpatched Windows, user is 0wned, PC is 0wned. Hilarities ensue.
It's just a standard trojan with an unusual delivery method of using fake Facebook profiles run by trojan bots. I can't see how this is Facebook's problem any more than it's your email program's fault that you clicked on a dodgy link without checking it.

Re:... And Yet Very Lacking From a Security Angle

[bigstrat2003]

It can't be addressed... because it's not a security issue with the site. It's an issue that the user needs to be trained on how to spot, and good luck getting that to happen.

I mean, come on, banks have the "problem" you described, and most banks aren't what we'd call insecure.

Yes,

[internerdj]

if by validation you mean:
Being able to find old friends you haven't been able to contact in years.
Having a central pull information spot rather than the push model of spaming every email address you have with pics of the new baby, house, car, toaster.
A central and standardized organization spot for arranging informal gatherings with friends, like parties.

Re:Wow

[Baton Rogue]

What they know about you can fill a warehouse.

What they know about you is only what you tell them.