Facebook Details the Software Engineering Behind Graph Search

Nerval's Lobster writes "Facebook's Graph Search, its new and powerful way of searching the social network for all manner of information, has drawn a lot of attention since its January unveiling. Some have praised its innovation; others have wondered openly whether its search abilities will end up threatening Google and LinkedIn. Still more have questioned what it all means for users' privacy—always a touchy subject in conjunction with Facebook. The social network previously revealed how it's adjusting its hardware infrastructure to deal with the spike in traffic that will come from interactions with Graph Search (short answer: the Disaggregated Rack, which will break up hardware resources and scale them independently of one another). Now, in a new blog posting, it's offering a bit more with regard to the software side of things, and how the company repurposed an existing system to solve Graph Search's enormous engineering challenge. Bottom line: Facebook's engineers and executives finally decided on Unicorn, an inverted-index system they'd had in development for quite some time."

Comments

Let's rewrite that

[Anonymous Coward]

"Facebook's Stalker Search, its new and powerful way of searching the social network for all manner of information about you, has drawn a lot of negative attention since its January unveiling. Few have praised its innovation; fewer have wondered openly whether its search abilities will end up threatening Google and LinkedIn. Most have questioned what it all means for users' privacyâ"always a touchy subject in conjunction with Facebook. The social network previously revealed how it's adjusting its hardware infrastructure to deal with the spike in traffic that will come from interactions with Stalker Search (short answer: the Disorganized Rack, which will break up hardware resources and scale them independently of one another). Now, in a new blog posting, it's offering a bit more with regard to the software side of things, and how the company repurposed an existing system to solve Stalker Search's enormous engineering challenge. Bottom line: Facebook's engineers and executives finally decided on Unicorn, a mythical flying horned horse they'd had in the basement for quite some time."

Privacy

[Anonymous Coward]

I love knowing how exactly Zuckerberg will strip-mine our connections for $profit.

It had to be unicorns.

[DougOtto]

Mermaids wouldn't have worked.

Re:It had to be unicorns.

[oztiks]

No no, it was a highly complex set of sql queries devised by a highly complex set of mathematical algorithms which took a team of washed up college drop outs to devise. I was able to speak to Mark Zuckkerberg on this matter and he divulged to key areas of the system's source to me.

$searchquery = 'people who are hipster losers';

$sqlquery = 'select * from pages p inner join friends f on p.userid=f.friendid where 1'

mysql_connect("localhost", "leet", "hax0r");
mysql_select_db("facebook_db");

$page = mysql_fetch_array($sqlquery);

if($page['like']>0) {
        if(strpos($page['content'], $searchquery) === true) {
                echo $page['content'];
        }
}

Re:It had to be unicorns.

[c0lo]

But lolcats would have done miracles. Add in a pinch of hot grits and the rack disaggregation would have been at light speed (what am I talking?... would have been as instantaneous as the loss of entanglement in a quantum setup)

Zuckerberg

[Dunbal]

I think it's really funny how they call a drop in traffic a "spike in traffic". Facebook was a fad and people have moved on.

Re:Zuckerberg

[Anonymous Coward]

Yeah, right. Keep dreaming.

Re:Zuckerberg

[HaZardman27]

Haven't you ever played volleyball?

Re:Zuckerberg

[erp_consultant]

There is some evidence of this. The younger crowd that was initially attracted to FB are starting to move on to other things like Tumblr. This is not just me talking...I've read a few articles that have pointed out this trend. Why are they moving on? Same thing that kills every other trend...their parents start getting into it so it's no longer cool. But...the corporations are still heavily invested in FB.

It seems to me that FB is becoming more of an advertising medium than a social network. The corporations and celebrities are using FB to try and sell you stuff. So who knows...maybe that's the direction it takes as regular folks move on to other 'social' time wasters.

Just know this...the moment that the corporations move on from FB...poof...it's dead.

What it means for user's privacy.

[QilessQi]

You get as much privacy from Facebook/Gmail/Hotmail/etc as you pay for. Sometimes, you get less.
If you're unhappy with those terms, you probably shouldn't use the service.

Re:What it means for user's privacy.

[PopeRatzo]

It's still worthwhile making sure as many people as possible know just how shitty Facebook is about respecting privacy.

They've made every effort to obfuscate what they're doing, and not everybody has the time and energy to search out the details of the privacy risks of using Facebook.

I salute the public-spirit minded people who make an effort to inform. As often as Facebook changes their privacy policy and acts dishonestly, I don't think you can say "People who use Facebook know what they're in for."

Re:What it means for user's privacy.

[QilessQi]

I agree. But I think the simple axiom is this: if you use a free service, assume that any information can be sold to anyone for any purpose at any time.
If people could be made to understand that, the specifics would almost be uninteresting.

Re:What it means for user's privacy.

[tlhIngan]

I agree. But I think the simple axiom is this: if you use a free service, assume that any information can be sold to anyone for any purpose at any time.
If people could be made to understand that, the specifics would almost be uninteresting.

An even simpler one dates way back to when we were connecting computers together by modems. "Don't post online what you don't want the world to know." Because anything posted online IS pretty much available for the world.

Yes, even with privacy settings. Privacy settings are a way to extract even more personal information from people than they otherwise would give. They're the ultimate in marketing - people mistakenly believe they're one thing but they're really something else.

The only real way to have privacy is to not put that information online in the first place.

Re:What it means for user's privacy.

[Anonymous Coward]

Every time there is at least one apologist posting a reply like this. And it's so stupid it hurts.

Believe me dude, I find the terms VERY offensive and certainly DON'T use any of those services. However millions of others do and that's the problem.

And why do people make bad decisions? Because they don't understand the IMPLICATIONS of their actions. The level of knowledge in the market is not perfect, as hard as it may be for you to believe.

Re:What it means for user's privacy.

[PopeRatzo]

The only real way to have privacy is to not put that information online in the first place.

Yes, except how much information comes not from what you PUT on the internet, but what you LOOK AT on the internet?

Re:What it means for user's privacy.

[tlhIngan]

Yes, except how much information comes not from what you PUT on the internet, but what you LOOK AT on the internet?

In which case Google should be the bigger offender than Facebook because Google is literally everywhere. Facebook's like buttons are on popular sites and such. Google's tentacles are everywhere, and gather data through practically everyone using Google Analytics, or a Google-owned ad service (not just AdWords, but DoubleClick, AdMob, and other advertising agencies). Or YouTube videos which are embedded everywhere as well. And soon, Google Glass will be there to capture your every movement in public by everyone else.

Facebook's base of information comes from what people put on it mostly. And Like buttons are in a lot of places, but generally speaking, not on the seedier side of the 'net (hell, even TPB had a hard time getting on Facebook). But who knows where those free pr0n companies are getting their ads from (probably Google-owned in the end).

Re:What it means for user's privacy.

[PopeRatzo]

In which case Google should be the bigger offender than Facebook because Google is literally everywhere. Facebook's like buttons are on popular sites and such.

That's my point.

But I think you may be understating Facebook's use of the like buttons. They are much more widespread than just "popular sites and such".

I've been seeing them everywhere. It's become unusual for me to view a page that does NOT have a Facebook button. And that means Facebook is tracking you. And lately, I've noticed that even blocking those little widgets or scripts or whatever they are often makes the entire page non-functional. I can either turn Ghostery off or forget about viewing the page.

Re:What it means for user's privacy.

[Anonymous Coward]

My personal solution to that is to run Ghostery, NoScript, etc. on the browser I actually log into stuff on and if I hit a page that I can't view but want to read, I open it up in another browser with no persistent cookies (either use private browsing mode or clear the cookies regularly).

But the point of this thread stands: just because people reading /. may have a good idea of what to expect from Facebook with respect to privacy, the vast majority of internet users are not even aware of the concept of web bugs or the long-term implications of giving Facebook information.

too soon

[aahpandasrun]

Tech sites love to post over and over again about these new Facebook features. But by the time they actually roll out to users everyone's forgotten about them.

Cool stuff

[futhermocker]

Would really like to see Unicorn become open source.

Where I work we use datamarts spread across several data warehouses, which is quite similar to the FB way.
Since we use a bottom-up design model, creating so called solutions using this indexer would be very straightforward.

What could be more social?

[Eugriped3z]

It's differentiation based on values, interuser traffic going down is less consequential than metauser traffic going up. The value system in place is obviously corporate. Has there ever been any doubt in your mind? If so, then consider yourself slow or idealistic. At least you can decide that for yourself.

Still others commented about it on facebook....

[genghisjahn]

...and still others replied to those comments.

Make benefit glorious engineering teams

[Drunkulus]

"Hard part of startup is make money from wheel after reinvent it." -Devops_Borat

Software Eng. or Comp. Sci. ?

[H0p313ss]

And does anyone on staff at Slashdot know the difference?

Re:Software Eng. or Comp. Sci. ?

[Anonymous Coward]

They're running the site, not in academia. Therefore, they just "know" that IT is a synonym for both, SE/SA and CS.

This is what I think of when I read this...

[fldsofglry]

“What are the differences between Mark Zuckerberg and me? I give private information on corporations to you for free, and I’m a villain. Zuckerberg gives your private information to corporations for money and he’s Man of the Year.” – Julian Assange I can't confirm if, where, and when he said this, but regardless the idea rings true for me.

Nope

[Anonymous Coward]

Nobody actually cares about graph search, it's a total non-event. Unless you work at facebook or read /. you probably didn't particularly notice. Google are not sweating. See also Bing.

Brilliant!

[Anonymous Coward]

SELECT 1 AS is_deleted, `id`, `from`, `time`, `text` FROM `wall_deleted` WHERE `to` = %d AND `media_type` = %d AND `media_id` = %d UNION SELECT 0 AS is_deleted, `id`, `from`, `time`, `text` FROM `wall` WHERE `to` = %d AND `media_type` = %d AND `media_id` = %d ORDER BY `time`'

Needs more complex SNA components.

[elucido]

Right now it's just a toy. They need to expand on it greatly so that we can do legitimate social network analysis research with it and even use it to make decisions.

Search isn't enough. Social network analysis is it

[elucido]

We need the ability to run any algorithm we want, we need an API to build apps to take advantage of it, we need the full range and capabilities of social network analysis, we need to be able to use intelligent agents to regularly collect opinion and other analytic information about our friends to help us make better decisions.

How many of my friends like X is important, but that's not going deep enough in my opinion. How many of my friends use certain phases, now we are getting somewhere useful. How popular are certain phrases in my social network? Useful. What are the common attitudes and trends of my social network? Useful. What about location data? How many hours do my friends spend on the road or are they mostly at home? What about even deeper? Specific topics like say how many of my friends like violence, and then it can show that certain friends like violent movies, video games, books, are members of the NRA, etc.

There is more, such as associations the ability to find correlations, do regression analysis, and even make predictions on what my friends might like or what topics they might dislike.

If it can do SNA, we could learn and predict a lot

[elucido]

Such as the political positions of our friends, how useful is that?
What products our friends might want in the future, how useful is that?
How our friends feel about certain things and who in the social network have feelings in common, how useful is that?
The job prospects and career prospects of our friends, how useful is that?

When all the data points are connected and social network analysis properly conducted you can learn a lot of the mysteries about people that wouldn't ordinarily be easily known. This is incredibly important and can make Facebook actually useful to me.

Re:Search isn't enough. Social network analysis is

[History's Coming To]

We need to allow app makers to do the things and offer the services we can't, the really intrusive stuff that we need plausible deniability over, and by monetizing our data via licensed app services which perform tasks which we find morally ambiguous we can keep our new and desperate shareholders happy in both ways.

Re:Search isn't enough. Social network analysis is

[elucido]

We need to allow app makers to do the things and offer the services we can't, the really intrusive stuff that we need plausible deniability over, and by monetizing our data via licensed app services which perform tasks which we find morally ambiguous we can keep our new and desperate shareholders happy in both ways.

But it's not really intrusive. People can change their privacy settings. Also big corporations are allowed to do it, so why not let everyone else in on it?

Even on SlashDot...no one cares about Graph Search

[xxxJonBoyxxx]

>> Some have praised its innovation

Er...what? 28 comments in 8 hours tells me no one cares about Graph Search - not even on SlashDot.

Re:Search isn't enough. Social network analysis is

[Anonymous Coward]

Even Zuckerberg's sister didn't know how to properly use the privacy settings, and she used to work there.

Graph search on linked open data

[meteormarc]

I would like to see their platform work on the combined datastores of linked open data (http://linkeddata.org/). Get fast answers because much of the processing is pre-indexed.

ERROR: syntax error at or near "from"

[Anonymous Coward]

Surely this isn't a leak?

a WHERE clause?

[Anonymous Coward]

wow, that's a toughie.

Next week they'll knock our socks off by introducing "GraphSearch 2.0- Now with ORDERY BY!"

RAM sled?

[Kishin]

Re: "run it on a RAM sled with between 128 GB and 512 GB of memory" Google gave me absolutely nothing on RAM sleds. I've used RAM disks for years and even know of hard disk's that are flash-backed RAM for performance. 128GB-512GB of RAM? If I needed that in a server, SGI (rip) and others have it. I doubt that's what they mean, though, as it's expensive custom stuff. So, what is a RAM sled? And where are they bought or how are they set up? Thanks ahead of time for any answers.