Slashdot Mirror
How Facebook Keeps Messenger From Crashing On New Year's Eve (ieee.org)
Wave723 quotes IEEE Spectrum: On New Year's Eve, millions of people will use Facebook's Messenger app to wish friends and family a 'Happy New Year!' If everything goes smoothly, those messages will reach recipients in fewer than 100 milliseconds, and life will go on. But if the service stalls or fails, a small team of software engineers based in the company's New York City office will have to answer for it.
The article says the team "tested and tweaked the app throughout the year and will soon face their biggest annual performance exam," since Messenger's 1.3 billion monthly active users send more messages on New Year's Eve than any other day of the year. Many of them hit "send" at the exact moment when their clock strikes midnight, "and people often try to resend messages that don't appear to make it through right away, which piles on more requests."
The solution appears to be load testing, re-directing traffic, message batching, and discarding "read receipts" and temporarily disabling other minor Facebook functions -- or, more generally, what their engineering manager describes as "graceful degradation."
The article says the team "tested and tweaked the app throughout the year and will soon face their biggest annual performance exam," since Messenger's 1.3 billion monthly active users send more messages on New Year's Eve than any other day of the year. Many of them hit "send" at the exact moment when their clock strikes midnight, "and people often try to resend messages that don't appear to make it through right away, which piles on more requests."
The solution appears to be load testing, re-directing traffic, message batching, and discarding "read receipts" and temporarily disabling other minor Facebook functions -- or, more generally, what their engineering manager describes as "graceful degradation."
"The solution appears to be ..." Stuff we've known since 1999?
Simple, do awful things that will make people avoid using any of your services.
AC comments get piped to
"The solution appears to be ..." Stuff we've known since 1999?
It's one thing to say you know how to do it...
Quite another when literally BILLIONS of people are using your services all at once - especially around NYE where it's not even spread through the day, it's a huge DDOS equivalent with billions of messages at midnight exactly...
Planning for that kind of load and super-extreme bursting is not easy, at all. No matter how much you "know".
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Want to keep it from crashing on New Year's Eve? Just to load the damn thing. There, simple. Problem solved.
I read at +2. If your post doesn't reach that level I will not see or respond to it.
... couldn't they simply split their users up into, say, 24 groups, and reduce the load that way?
http://harridanic.com
The vast majority of messages avoid that peak: hardly anyone waits for the exact midnight to send a message. So the load gets smeared onto quite a chunk of time.
Look around you at the next NYE party and you will see just how wrong you are. Most people queue them up ahead of time and lots of people are hitting Send as the ball drops... (hint to devs, if someone has typed a partial message transmit that to the server in case they come back and hit send later - course Facebook was just screwed by that recently when it was found they had cached images on the server from never sent messages...).
At least it is spread across time zones but that is still a LOT of people, especially from the U.S. coasts.
The engineering problem boils down to: send short messages between pairs of arbitrary sources and destinations (although usually the source and destination are close to each other), with message size usually within 50-100 bytes. Let's be generous and say that with metadata they fit within 1500 bytes
Come on man, you know that modern web API's are not that compact, and we are talking Facebook here. You are off by an order of magnitude at least, way more when you stop to think that on NYE way more people are sending images also... One single response to a post on Facebook I just did with 14 words had a 9.5kb body going out, and a 21.2 k response.
Let's estimate the flow: after everyone raises the toast, exchanges hugs and kisses, says greetings, then sits down with the phone -- sending, let's say, 10 messages. This should take around half an hour. You get 300K messages per second. Not so impressive...
Think MILLIONS, possibly BILLIONS and you might be closer to the mark. On a *normal* day, Messenger and Whats App process over 60 billion messages a day... so that is 2.5billion messages every hour *normally*.
And that was from 2016. Do you think people send more, or fewer messages now than then!
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Millions of people.. sending a small TCP packet... containing a couple of hundred characters...
Wow. Gosh. The infrastructure that must take to handle...
Like... a couple of servers in a rack and a few gigabits of uplink at worst.
Honestly, has modern technology come to this?
One single YouTube video probably has more bandwidth, more data transferred, more CPU usage and less latency.
"Graceful degradation" is the unsung hero of properly engineered systems.