Slashdot Mirror

← Back to Stories (view on slashdot.org)

Facebook's Corona: When Hadoop MapReduce Wasn't Enough

Posted by Soulskill on from the named-after-a-desire-to-launch-hadoop-into-the-sun dept.

Nerval's Lobster writes "Facebook's engineers face a considerable challenge when it comes to managing the tidal wave of data flowing through the company's infrastructure. Its data warehouse, which handles over half a petabyte of information each day, has expanded some 2500x in the past four years — and that growth isn't going to end anytime soon. Until early 2011, those engineers relied on a MapReduce implementation from Apache Hadoop as the foundation of Facebook's data infrastructure. Still, despite Hadoop MapReduce's ability to handle large datasets, Facebook's scheduling framework (in which a large number of task trackers that handle duties assigned by a job tracker) began to reach its limits. So Facebook's engineers went to the whiteboard and designed a new scheduling framework named Corona." Facebook is continuing development on Corona, but they've also open-sourced the version they currently use.

42 comments

  1. Misleading headline by Anonymous Coward · · Score: 0

    What do you mean "Hadoop MapReduce isn't enough"? It's the same fucking framework with a better scheduler.

    1. Re:Misleading headline by Anonymous Coward · · Score: 0

      It needed a better scheduler, what was there wasn't enough.

      Do you just knee-jerk rage for internet karma, or do you legitimately have reading comprehension problems?

    2. Re:Misleading headline by ArcadeMan · · Score: 4, Funny

      And why the fuck should I care about Windows 8 tablets? You are not making any sense!

      --
      Get free satoshi (Bitcoin) and Dogecoins
    3. Re:Misleading headline by ArcadeMan · · Score: 0

      Oh, shut the fuck up you idiot!

      Look, ponies!

      --
      Get free satoshi (Bitcoin) and Dogecoins
    4. Re:Misleading headline by ArcadeMan · · Score: 1

      No, I'M Spartacus!

      --
      Get free satoshi (Bitcoin) and Dogecoins
    5. Re:Misleading headline by Anonymous Coward · · Score: 0

      What was there was plenty enough, they were having problems because they were using it all wrong. So they decided to do spend a lot of time and effort to build things differently instead of working out how to do them properly with what was there.

    6. Re:Misleading headline by Anonymous Coward · · Score: 1

      Much as those who were holding their iPhone wrong were at fault?

      Seriously, the Job Tracker just didn't scale well and applications had to worry about it - that's a broken architecture, not a broken application or deployment. Blaming the application or deployment for serious fundamental architectural flaws of the platform is much like blaming an application programmer in 1980 for using a=a+1 which a compiler happened to implement less efficiently than a++ or even a+=1 (or, for you old timers, a=+1 not to be confused with a= +1).

      I don't know how Corona relates to YARN, but both should be a big improvement over the Job Tracker based architecture - even if they still may not technically scale linearly to n (for VERY large n) nodes.

    7. Re:Misleading headline by Anonymous Coward · · Score: 0

      "Hadoop MapReduce is not enough" is a misleading headline when they are still using "Hadoop Mapreduce".

      I'm not enraged, I don't care about internet karma, I'm not even logged in. You seem to be severely retarded though.

  2. Junk. by Anonymous Coward · · Score: 0

    "Its data warehouse, which handles over half a petabyte of information each day, has expanded some 2500x in the past four years"
    Too bad that's 99.9% junk I don't care about.

    1. Re:Junk. by isopropanol · · Score: 2

      But between you and 1000 other people who care about slightly different sets, much of it is stuff that someone cares about.

    2. Re:Junk. by John+Bokma · · Score: 1

      And yet you just made an effort to post similar junk to Slashdot....

      --

      Perl Programmer for hire
    3. Re:Junk. by Daniel+Dvorkin · · Score: 4, Insightful

      Too bad that's 99.9% junk I don't care about.

      But between you and 1000 other people who care about slightly different sets, much of it is stuff that someone cares about.

      This. 99.9% (at least) of the entire internet is junk that any one person doesn't care about. But every bit has someone who cares about it (or did at one time) or it wouldn't be there.

      Well. I opened the story expected some reflexive Facebook-bashing, and I wasn't disappointed. When are people going to realize that FB's just another internet company with a reasonably successful business model, and worthy of neither adulation nor hatred?

      --
      The correlation between ignorance of statistics and using "correlation is not causation" as an argument is close to 1.
    4. Re:Junk. by Revotron · · Score: 4, Funny
      Yes, Facebook sure would be a lot more successful if 99.9% of people's posts got deleted and replaced with an on-screen notification that reads,

      This post has been removed because it is of no interest to Anonymous Coward. Please try posting things more in line with the following categories:

      1. Linux
      2. Open-source software
      3. Richard M Stallman
      4. OMG!!! PONIES!!!

    5. Re:Junk. by Daniel+Dvorkin · · Score: 1

      s/expected/expecting/

      [sigh] I do so wish Slashdot would allow editing posts, at least for a limited time (say, until they've been moderated or replied to). C'mon, even Facebook can manage that. ;)

      --
      The correlation between ignorance of statistics and using "correlation is not causation" as an argument is close to 1.
    6. Re:Junk. by Anonymous Coward · · Score: 0

      Luckily there's that 0.001% that you do care about, which measures to about half a terabyte each day. Plenty of interesting information for you there.

    7. Re:Junk. by rtaylor · · Score: 1

      This. 99.9% (at least) of the entire internet is junk that any one person doesn't care about.

      I've done a crawl of a few billion pages.

      No person at all cares about 99% of the content available on the interent. In fact, nearly that much is completely unreadable and was machine generated gibberish (real words, not sentences) in an attempt to fool Google and other search engines.

      There are a few servers which host millions of subdomains with millions of manufactured pages under each subdomain.

      In short, it's far worst than 99.9% of the entire internet being of little use.

      --
      Rod Taylor
    8. Re:Junk. by SolitaryMan · · Score: 2

      So, you care about (1 - 0.999) * 500 TB = 500 GB of Facebook information every day??? Dude, where do you get the time?

      --
      May Peace Prevail On Earth
    9. Re:Junk. by Anonymous Coward · · Score: 0

      4. I, for one, look forward to an April first in which /. properly pay homage to Pinkie Pie.
      3. Maybe someone could draw an epic enough beard on her and we can all pretend it's RMS?
      2. Speaking of Pinkie, did you know Pinkie Pie has pwned chrome twice? Maybe if I twist this enough, it can be a "many-eyes" post about why OSS is better? I mean, if you've got _ponies_ breaking your browser, you've obviously done something wrong.
      1. got nothin'

    10. Re:Junk. by nurb432 · · Score: 1

      You may not care, but the people doing datamining to find new ways to push ads at us or find the next serial killer care greatly. You know, the ones that actually pay the bills.

      --
      ---- Booth was a patriot ----
    11. Re:Junk. by martin-boundary · · Score: 2

      When are people going to realize that FB's just another internet company with a reasonably successful business model, and worthy of neither adulation nor hatred?

      Wrong. FB is worthy of hatred because what they do is inherently evil. They spy on people, and sell off that information.

      The "it's just a job/business" excuse doesn't work when the job/business is evil. For example, when the local Mafia goons come to collect protection money, it's "just a job" for them right? Nothing personal. They're just regular people who are trying to make ends meet, like eveybody else. Don't hate them. Wrong, it's evil, and the goons display a singularly bad sense of judgement in accepting to do this kind of work.

      Similarly, spies are evil, James Bond notwithstanding. They steal secrets, and betray people in the process. And FB are a spying organization. They treat users' rights as a joke, and due to their size and ubiquity, are substantially responsible for the state of privacy on the internet today.

    12. Re:Junk. by ilsaloving · · Score: 1

      What you say is sort of true, but I disagree that it is inherently evil. Evil implies a malicious intent. At worst, it's simply sociopathic. Facebook is doing what it's doing so that it can make money, and it's methods arn't even remotely secret. They would have no power at all if it wasn't handed to them gleefully by people.

      Further it's disingenuous to compare them to the mafia and similar, for one simple reason. The mafia does what it does against people who are unwilling participants. Facebook on the other hand, is being fed almost limitless information by people giving it up willingly.

      I don't see a problem with Facebook for the same reason I don't see a problem with 419 scammers. Only incredibly stupid and greedy people fall for 419 scammers. People who post their most intimate details on Facebook are either similarly stupid, or just don't think what they're posting is of value to anyone. Which would be true if you're posting pictures of kittens. But as many have already found out, that picture of you drunk at a party can get you fired. In either case, it's the people themselves that orchestrated their own demise.

      So calling Facebook evil is like calling a crocodile evil because you were stupid enough to walk your dog right on the edge of a swamp known to home crocodiles, and a crocodile came out and ate your dog. The crocodile didn't do anything any other crocodile wouldn't do, and Facebook isn't doing anything that bajillions of other companies arn't already doing (case in point, google for Target and pregnant daughter).

      What they do can't be stopped because not enough people care enough to actually try. All you can do is walk with your eyes open and avoid problems where possible.

  3. First beer! by Anonymous Coward · · Score: 0

    With lime. Had to do it.

  4. What? by Anonymous Coward · · Score: 0

    In layman's terms:

    What is Hadoop?
    What is MapReduce?

    From the article, I derive that it is a scheduling framework. What the hell is a scheduling framework?

    1. Re:What? by Em+Adespoton · · Score: 3, Informative

      Hadoop: massive data storage system framework... "Apache Hadoop is an open-source software framework that supports data-intensive distributed applications"
      MapReduce: a way of managing distributed clusters of data sets... "MapReduce is a programming model for processing large data sets, and the name of an implementation of the model by Google. MapReduce is typically used to do distributed computing on clusters of computers"

      Scheduling framework: a framework for providing optimal scheduling of something such that events are handled in an optimal manner.

      Or, to put it another way:
      http://lmgtfy.com/?q=hadoop
      http://lmgtfy.com/?q=mapreduce
      http://lmgtfy.com/?q=scheduling+framework

    2. Re:What? by Anonymous Coward · · Score: 0

      Well shit, if we have to google half the article terminology, why not just google the article itself?

  5. java by Anonymous Coward · · Score: 1

    after paging through the code a bit, i found it interesting that they use java in their implementation (not just corona, but hadoop as well). i was wondering why, and after some googling found this link which helped explain the situation a bit clearer.

    pretty interesting stuff. but id be willing to bet googles map reduce is written in c/c++

    1. Re:java by Anonymous Coward · · Score: 0

      Because life is more fun if you have to worry about nodes taking naps while garbage collecting every so often and developers wouldn't know what to do with their spare time if they didn't have to spend time on those issues?

      (Seriously, the link above provides several much more rational explanations!)

    2. Re:java by Rakishi · · Score: 1

      Hadoop is not real time, it's a batch processing system, no one gives a damn if a node spend 50ms garbage collecting or not every so often.

    3. Re:java by rioki · · Score: 1

      Until you process petabytes of data and suddenly you it makes a difference of a couple of hours per run. All the coll dynamic web technology is really nice and empowering, but once you start hitting real traffic, it makes sense to invest into more efficient core systems. See G-WAN for how to do it right.

  6. Facebook by gman003 · · Score: 3, Interesting

    I have to admit, while I hate using Facebook, and hate most of their business practices, I like how they're not just writing new infrastructure software, but are open-sourcing it all. I don't think it quite makes up for everything else, but it helps.

  7. Wow! 2500X by Anonymous Coward · · Score: 0

    They went from a couple GB to 5 TB. Impressive!!!

    Half a PB of data flows through the infrastructure, but how much of that actually has to be stored? That's the real question.

  8. Song to celebrate major version after 8 by Anonymous Coward · · Score: 0

    9 Coronas

  9. How many IT projects... by Synerg1y · · Score: 1

    Have been code-named corona these last few years?? Seems like every org's got a project named corona nowadays.

    1. Re:How many IT projects... by VortexCortex · · Score: 1

      Have been code-named corona these last few years?

      The only one I can think of involves me remotely managing a server from the beach with only a lime wedge and cold beer.

  10. Thanks For Nothing by Anonymous Coward · · Score: 0

    Mmm. Snarky sarcasm, industry lingo, and obtuse responses. Thanks ofr nothing.

    Hadoop: massive data storage system framework... "Apache Hadoop is an open-source software framework that supports data-intensive distributed applications"

    So it's some sort of file system or database? Or is it simply yet another programming language abstraction layer upon other programming language abstraction layers?

    MapReduce: a way of managing distributed clusters of data sets... "MapReduce is a programming model for processing large data sets, and the name of an implementation of the model by Google. MapReduce is typically used to do distributed computing on clusters of computers"

    Then Facebook's map reduce is what, a programming method, as described above? It's made out by the article to sound like something more tangible than a methodology. Even reading the Wikipedia article doesn't make anything clear in layman's terms.

    Thanks again.

    1. Re:Thanks For Nothing by Em+Adespoton · · Score: 1

      No snark intended... no sarcasm given. The terms describe things that are technical. If you want something more generic, I could go as far as "Database management architecture" and "database communication architecture" but that dumbs things down to the point where it ads nothing to the discussion. If you don't understand what a database is and how it works (and that we're talking about database management here), you're going to find this entire article over your head, not just the industry buzzwords.

      Kind of like if we were discussing an article dealing with some new algorithm for sequencing DNA or some new tool being used by biochemists -- background reading is required to gain anything useful from the article -- explaining the buzzwords won't mean much.

      Slashdot. Stuff that mutters.

  11. Hard to believe by Anonymous Coward · · Score: 0

    Hard to believe how much technology goes into such a shitty website. Even right now over 90% of profile images aren't loading for me.

  12. I have little sympathy for them by stickyboot · · Score: 1

    They could start by actually deleting deleted content. Seems simple to me. Lets hope their shortsightedness continues when everyone jumps ship for the next social fad, and continuing this rat race becomes far to costly.

    1. Re:I have little sympathy for them by Lazy+Jones · · Score: 1

      They could start by actually deleting deleted content

      They could, but why should they put themselves at a disadvantage over Google, every other corporation that buys such data and the NSA, who all most certainly do not delete stuff in the way you'd like them to?

      --
      "I love my job, but I hate talking to people like you" (Freddie Mercury)
  13. Oh! I See. by Anonymous Coward · · Score: 0

    So, my inability to derive that a massive data storage system framework is a "Database management architecture" in this case is a demonstration of my inability to comprehend technical stuff.

    Sort of like your inability to understand that Let Me Google That For You with obtuse results is pure snark when someone asks for a layman's description. You didn't have to respond if you thought that the response was pointless or that you lacked sufficient understanding to explain it in a coherent fashion. Yet, you felt compelled to provide a smart-ass(not smart) response.

    So, by your example, your inability to understand the technicalities of language and your absence of social skills makes it nearly impossible for you to understand the textbook definition of snarky. Allow me to try to explain it to you, by example.
    http://www.lmgtfy.com/?q=define+snarky

    1. Re:Oh! I See. by Anonymous Coward · · Score: 0

      BURN!