How Facebook Ships Code

Hugh Pickens writes "The two largest teams at Facebook are Engineering and Ops, with roughly 400-500 team members each, together making up about 50% of the company. All engineers go through 4 to 6 week 'Boot Camp' training where they learn the Facebook system by fixing bugs. After boot camp, all engineers get access to the live DB and any engineer can modify any part of Facebook's code base and check-in at-will so that engineers can modify specs mid-process, re-order work projects, and inject new feature ideas anytime. Then arguments about whether or not a feature idea is worth doing or not generally get resolved by spending a week implementing it and then testing it on a sample of users, e.g., 1% of Nevada users. 'All changes are reviewed by at least one person, and the system is easy for anyone else to look at and review your code even if you don't invite them to,' writes yeegay. 'It would take intentionally malicious behavior to get un-reviewed code in.' What is interesting for a company this size is that there is no official QA group at Facebook but almost every employee is dogfooding the product every day."

Comments

no process

[mortonda]

any engineer can modify any part of Facebook's code base and check-in at-will

That explains a LOT...

Re:no process

[iluvcapra]

the culture of the company seems to be set so that *everyone* feels responsibility for the product

It's astonishing that they can keep such a process rolling with 500 engineers, let alone 200

product managers have a lot of independence and freedom. The key to being influential is to have really good relationships with engineering managers. Need to be technical enough not to suggest stupid ideas.

So basically, Facebook is run by an aggressive engineering culture based not on consensus or managerial decision making but by cliques and lobbying, where the worst thing someone can possibly do is suggest an idea that an engineer claims is "stupid" and doesn't give them opportunity for nerd glory. How much you want to bet that Zuckerberg sets the tone and decides that any modifications to the way the privacy settings are run is "stupid" and "boring."

Re:no process

[FuckingNickName]

Like Google, unlike Apple.

I don't mind a professional engineer for a boss, but any which displays the traits celebrated by xkcd would be absolutely unbearable.

Re:no process

[Anonymous Coward]

Facebook is run by an aggressive engineering culture based not on consensus or managerial decision making but by cliques and lobbying

At any other company, that could be counter-productive. But considering that the tool they're building is basically the ideal tool for building your clique and growing your influence, it probably just adds to their "dogfooding."

Re:no process

[ScentCone]

It's astonishing that they can keep such a process rolling with 500 engineers, let alone 200

I'm always curious about this expression, which - just like "I could care less" in place of "I could not care less" - is the opposite of what's meant. Surely, 200 would be astonishing, let alone 500.

Re:no process

[Thelasko]

Actually, it sounds like a form of quality circle. Presumably, every piece of code interacts with at least one other piece of code. Each programmer is responsible for checking that the code he/she interfaces with behaves as expected. The same thing happens on assembly lines. If a component arrives and it doesn't look as expected, the assembler is expected to flag that component for review.

It also sounds like all changes are controlled by comparing them against some sort of metric when distributed to the sample audience. This is a classic continuous improvement method. If the new system does not outperform the old one, it doesn't get implemented.

Sincerely,
The Devil's Advocate

Re:no process

[somersault]

The engineers in Dilbert are fine, it's the managers you don't want.

Re:no process

[Hoi Polloi]

Facebook does have a formal QA group, it is called the user base.

Re:no process

[syousef]

any engineer can modify any part of Facebook's code base and check-in at-will

That explains a LOT...

Personal journal: This company trusts me with the code base. I don't know why. Dumb fucks!

Re:no process

[iluvcapra]

Something TFA doesn't talk about is how the engineers are organized -- surely there's a team that does memcache, a team that does database work (like those Cassandra guys), a team that does i18n, a team that does rendering, a team that specializes in Flash doodads, etc. It's hard to see how the engineers check each other without knowing where demarcations lie.

My issue wouldn't be with the quality of the code, it's how they decide what code to write in the first place. All of the Kaizen processes work great on an assembly line, but a feature request and a coworker evaluation aren't fender panels. They require someone in authority to tell you what you're trying to accomplish in the first place. The description of the way people decide what to add (basically shutting out the marketing people if they speak out too much) and how they discipline each other (too many blames in the SVN log and too many "public shamings" result in termination) speaks to a culture where new ideas and customer focus are almost stigmatized, and where the engineers don't try to evaluate each others ideas as much as they worry about getting voted off the island.

Re:no process

[CppDeveloper]

I would pass on Wally...

Re:no process

[tsm_sf]

Except that tasks become increasingly difficult the more nerds you throw at them, not less. 500 engineers makes a project harder than 200.

Re:no process

[timeOday]

It's astonishing that they can keep such a process rolling with 500 engineers, let alone 200

It is astonishing. I guess that's why nobody was doing it before, and they were able to get rich from a mere website in what seemed to be an already established niche.

Most of the comments here are looking at the process and predicting a result. That's backwards. We can see the result; the company is extremely successful. If that doesn't match your mental model of processes, maybe the mental model needs updating.

Re:no process

[ScentCone]

Except that tasks become increasingly difficult the more nerds you throw at them, not less. 500 engineers makes a project harder than 200.

Which is exactly my point. It's amazing that the casual way they handle code could work with 200 programmers, let alone 500. See? "Let alone" means "never mind." As in, it's hard enough for 200, and 500 isn't even worth mentioning ... so, let it alone. And that's why I pointed out that the phrase was being used exactly backwards.

Re:no process

[catmistake]

based not on consensus or managerial decision making but by cliques and lobbying

I have worked, via contract, in at least a dozen medium to large sizes companies where this is the underlying mechanism. If you stick out in that you focus solely on the work and do not engage with the de facto clique leaders, who as often discuss droll subjects such as retiling their bathroom as often as they collectively decide how they want you to do their jobs for them, then absurd personal complaints will begin to stack up against you and you will be forced out of a job.

I used to flaunt my individuity, but since 9/11 I have been consciously conforming to old school business formality... never arrive late, never leave early, never get sick or take off, always dress nice, keep short hair, no superfluous non-work related conversations, never mentioning anything from my personal life, always ready with a polite smile and an enthusiasm for the work no matter how mundane it is... I've tried to embody what the ideal is for a person in my field. This doesn't work. One needs to become *good* friends with whomever that defacto leader is, who is as often as not NOT the management (but, indeed, through strength of personality has the manager under their thumb, and completely snowjobs the executives with an almost supernatural confidence).

If they like you, you win. And when you win, they like you. Otherwise... you are the first of any approaching cutbacks... you become the sacrifice that saves everyone elses jobs, the scapegoat for any complaints that come down from the executive levels.

The last 2 teams I worked with were more like a gang than a corporate division, whose self-preservation far outweights the work they are responsible for (My last contract ended abruptly when 3 of my counterparts in another building took heavy criticism for their laziness and distict lack of any work ethic... but because they were well liked, somehow I got the boot and the blame for their incompetance... without even remotely having anything to do with whatever incident occurred (never quite clear on exactly what it was... something to do with an assigned task that they kept passing between them and was never completed).

I hope someday my perserverance will pay off and I will be able to work not only with smart people (intelligence was rarely an issue), but educated and honest, and hopefully enlightened individuals that do not synthesize drama to manipulate perceptions such that those they like, towing the gang's loyalty, remain employed, while those that are effective, and thus throwing the curve, are terminated.

Re:no process

[NoSig]

It doesn't just sound silly, it's a crime against intelligence is what it is. I doubt it's a contraction like you say since "as if I could care less" is not something I've ever heard, while "I couldn't care less" is. It's just a bunch of people who don't know what they are saying that cut out the "not" because they don't understand how the sentence is supposed to work. That's like contracting "I don't like Hitler" to "I like Hitler". I guess by your reckoning it would be like contracting "as if I like Hitler" to "I like Hitler" which is just as stupid, but I still think it's the former.

Re:no process

[multipartmixed]

I AM Wally, you insensitive clod!

Re:no process

[LateArthurDent]

I would pass on Wally...

There are consistent references that Wally was a great engineer in his youth. Years of bad management has disillusioned him to the point where he realizes nothing he does really matters, so he might as well do nothing.

Basically, Wally is what Dilbert would end up being with 30 years of experience.

Re:no process

[jgagnon]

Oh the huge manatee?

Re:no process

[Fjandr]

Not a contraction, since the omission is not internal. More likely it was first used as a sarcastic play on the original phrase, and then the tone was dropped by imitators who understood the form but not the function.

Re:no process

[Fjandr]

That's a 'wholly-owned subsidiary' so that there's no conflict of interest. *wink*

Oh, wait, we're past the point where that sort of thing is hidden. Yup, forgot the data mining team...

Re:no process

[NoSig]

I could care less about your question. In fact, I think it's very interesting. Thank you.

Re:no process

[tehcyder]

Most of the comments here are looking at the process and predicting a result. That's backwards. We can see the result; the company is extremely successful. If that doesn't match your mental model of processes, maybe the mental model needs updating.

Even if the company is (arguably) successful, this still may be despite, rather than thanks to, their processes.

Facebook ships code?

[El_Muerte_TDS]

Then where can I get that shipped code?

Although, after reading that story, I don't think I want that code. It sounds like a textbook example of feature and focus creep.

Why would Facebook need 500 engineers?

[Anonymous Coward]

It's not like anything really changes on it.

Re:Why would Facebook need 500 engineers?

[anyGould]

Heck, do you need to invent new ways to harvest data, when after your 4-6 weeks you get unfettered access to the entire database? Seems like a security/privacy nightmare (for us, anyway - FB doesn't seem to care.)

Re:Why would Facebook need 500 engineers?

[Panaflex]

Have you handled 300 million users lately? Just the backend storage scaling itself probably eats a hundred engineers... then there's the code performance engineering, php compiler and memcached wranglers... there's another 100 engineers. And of course there's the network engineers! And the Apache engineers(50) and the cable engineers, the customer support engineers... OH and the social engineers!!

That leaves about... (click click click...) 4 engineers to work on the interface code.

No access controls?

[choongiri]

After boot camp, all engineers get access to live DB

So anyone who's ever worked at FB as an engineer will have likely downloaded copies of all their friends' / family's / ex-girlfriends' inboxes, chat history, etc.

Not surprising really.

Re:No access controls?

[gorzek]

I could've sworn there was an article on /. some time ago that indicated Facebook employees had to justify accessing "personally interesting" information they didn't have legitimate access to through their FB profile, namely, the sorts of things you mentioned: inboxes of exes, family members, etc. There was a master password and all uses of it were recorded and audited so you had to explain yourself whenever you used it. My memory might be a little fuzzy but it seemed that FB employees didn't have unmonitored access to all data on the site.

Re:No access controls?

[travdaddy]

Here it is: http://slashdot.org/story/10/01/21/179242/Facebook-Master-Password-Was-Chuck-Norris. How much they enforced that policy seems sketchy.

Programmers != Engineers

[Anonymous Coward]

Once and for all, programmers are not Engineers.

Programmers can not be held liable, have no professional license, no government accountable body, no code of ethics, no liability insurance, or other measures in place to directly protect the public.

Think about it.

If facebook was staffed by real engineers, a privacy breach would cost the engineer his/her license, and they could be personally sued, as well.

But then again, anyone in the good old US of A is an engineer. From the sanitation engineer who picks up my trash, to the beverage refreshment engineer who makes my coffee at Starbucks.

Engineer is just another cheap title, like CEO/CFO/CIO/CTO, etc, free to be used by anyone.

Re:Programmers != Engineers

[morgan_greywolf]

Not all engineers are licensed. Civil engineers are usually licensed. Mechanical engineers and electrical engineers are usually not licensed. Similarly, there is no licensure for system engineers. There are "certifications" but these are essentially meaningless.

Re:Programmers != Engineers

[Anonymous Coward]

But then again, anyone in the good old US of A is an engineer. From the sanitation engineer who picks up my trash, to the beverage refreshment engineer who makes my coffee at Starbucks.

Engineer is just another cheap title, like CEO/CFO/CIO/CTO, etc, free to be used by anyone.

As one of slashdot's many Comment Moderation Engineers, I believe you bring up a good point - henceforth Anonymous Coward shall be known as Anonymous Engineer.

Re:Programmers != Engineers

[Mentally_Overclocked]

I have a degree from a university in electrical engineering. I work as an electrical engineer and I consider myself one. I am not licensed as a professional engineer (PE) and have little interest in obtaining that license at this point as the type of work simply doesn't appeal to me.

As you suggest, those PEs do put their license on the line when they sign a document. From my understanding, companies that have resident PEs will only have a few and have other non-licensed engineers do the less expensive work.

If it is a product it will usually need to meet expectations set by a different regulatory body (ETL, UL, FCC, whatever). If it is a building, power related, whatever, then it requires the review of a PE.

http://en.wikipedia.org/wiki/Professional_Engineer

I'm not really sure how they would regulate Facebook with their data ... I've never dealt with something like that.

Re:Programmers != Engineers

[ScentCone]

Engineer is just another cheap title, like CEO/CFO/CIO/CTO, etc, free to be used by anyone.

No, this is not true. There is a big difference between being the employee of a company and being an officer of the company. Those "O" titles actually mean something. Doesn't mean that the people who are officers of the company are the right people for those roles, but there's real baggage that comes with those titles, including a higher standard for the consequences of entering into contracts, obligating the company to act or pay bills, etc.Being an "O" also makes you more of a law suit magnet.

Re:Programmers != Engineers

[mini me]

Engineer is defined as someone who designs, builds, and maintains.

It has nothing to do with licensing, government accountability, code of ethics, accreditation, or anything else.

It is not a cheap title, it is a description of the job. When they say Facebook engineer, we get a sense of what those employees are required to do; which extends beyond the role of just programming.

Re:Programmers != Engineers

[russotto]

Once and for all, programmers are not Engineers.

In the narrow sense of holding a P.E. (Professional Engineering) license or equivalent, no. But the P.E. does not define the engineer, no matter how many holders of such licenses think it does.

Re:Programmers != Engineers

[kevinNCSU]

Once and for all, programmers are not Engineers.

Programmers can not be held liable, have no professional license, no government accountable body, no code of ethics, no liability insurance, or other measures in place to directly protect the public.

Think about it.

Professional Engineer != Engineer

Unless of course you wish to make the argument that making say, a space shuttle, doesn't require any "real" engineers or didn't you know that aerospace engineering is largely unregulated with no professional license, no government accountable body, no code of ethics, no liability insurance or other measures in place to directly protect the public from the individual engineer rather than the company or organization? The same thing with the majority of mechanical engineers, as cars don't require professional engineers because they are under interstate commerce and it's the State's that license professional engineers. Really it seems your only view of engineering here is people that make buildings and bridges. Exciting, glad you loved your statics and materials science course so much. The rest of us "engineers" in the later half of the 20th century are going to go over here and keep building spaceships, cars, missiles and airplanes and you know what? We're even going to put SOFTWARE in them and we're going to do it ALL without a professional license from the great state of *insert your local jurisdiction here which doesn't govern space, the sky, or even interstate roads*

Believe it or not, the profession of engineering existed long before jurisdictions realized they could make money by charging people fees for professional licensing to "protect the public" while at the same time protecting local jobs by making it harder for out-of-state engineers to do work there unless they go through the local licensing process.

Fake

[MBCook]

It's too bad much of the article is wrong.

Re:Fake

[FuckingNickName]

Great, a series of statements about Facebook by some people not speaking on behalf of Facebook corrected by one person claiming to be from Facebook but not speaking on behalf of Facebook. Who knew Facebook engineers and fanboys were as capricious and unwilling to understand boundaries as the users?

Re:Fake

[trollertron3000]

Sir the post you cite actually confirms a lot of this is true. Such as the lack of QA. They are all encouraged to test and then report any bug in a new internal version? Is that supposed to be sufficient?

Just more proof that Facebook really doesn't care about anything except driving eyeballs to ads. I bet _that_ system is NASA engineered.

Intentionally malicious code, eh?

[noidentity]

It would take intentionally malicious behavior to get un-reviewed code in.

But will they catch unintentionally malicious behavior that gets un-reviewed code in?

It's a component system

[Animats]

Bear in mind how Facebook works internally. It's a large number of programs intercommunicating through a remote procedure call system. There's no one big "build". The interfaces between programs are well defined, and changing out programs individually is normal.

"dogfooding"?

[Tetsujin]

Yeah, I had to look it up. Never heard of the term "dogfooding" before.

It seems a silly word choice, since in context it carries no more meaning than the rather more conventional word "using".

Re:"dogfooding"?

[Urban Garlic]

The choosing is weirding, you're right, but as long as the meaning is clearing, it's not really probleming.

Re:"dogfooding"?

[Locke2005]

The usual expression is "eating their own dog food". I've never heard it referred to as "dogfooding" either.

Engineers?

[Anonymous Coward]

I'm just laughing (sadly) that web developers are considered engineers.

But hey, they call garbage collectors "Sanitation engineers" so why not Javascript and PHP coders...

So how then......

[Lumpy]

Does idiotic ideas like sharing my private info with any random app that is easily setup to be malicious get through?

Why dont they have a checks and balances with the executives that should be smacked hared in the head for their bad ideas?

Same old tune

[trollertron3000]

We get to read this story every few years. Some company makes a popular product and all of a sudden they are the authority on development. News flash guys, enterprise software wasn't just invented and we've been rolling out features to millions of people for years. In environments where any discrepancy can lead to serious legal repercussions. So just for a second, hear us out. We know what we're doing and we don't need to reinvent it every few years because the guys from Facebook, Basecamp, Hacker news, or any other place said we're doing it wrong. Our bottom line and years of quality deliveries say we're doing it right. What do these guys have besides a history of rolling out mistakes? Their value is based on hype, not their code quality. Some of us don't have the luxury to make huge mistakes and have our customers be okay with it.

Re:Same old tune

[hsmith]

Yawn. Some of us find it fascinating how one of the largest websites on the internet does its business day in and day out. It escapes me how people can't find it interesting. They are dealing with a shit ton of data and do it pretty well.

This explains bizarre feature regressions

[Y-Crate]

I know they're always looking to "streamline" Facebook, but sometimes really obviously good features vanish without explanation and leave users scratching their heads. These decisions would make more sense if you could see some way that FB was making life easier for advertisers or something, but often, the feature regressions are just nonsensical to the extreme.

A few:

- Facebook got rid of statuses. The one e-crack feature they're best known for. Gone with the new profile. Now they're just wall posts to yourself that quickly fall down the page. *facepalm*

- Photos are now uploaded in descending order. It used to be that a group of images would be uploaded like this: beachtrip-1.jpg, beachtrip-2.jpg... and so on. Now it reverses the order of an uploaded album. The last photos taken are displayed first. Posting vacation pics? Well your friends get to see the day you left and work their way back to the day you arrived.

- The "Reverse Photo Order" option was removed when the above "feature" was introduced. You can drag around photos to manually reorder them, but every photo that you mouse-over jumps out of the way and moves to different rows. It's a UI disaster.

- A few months ago I started seeing tagged photos of friends despite the fact they were in private albums of people I'm not friends with who also happen to have locked-down profiles where you can't see anything at all unless you add them.

Re:This explains bizarre feature regressions

[Y-Crate]

Status updates are effectively gone. They are no longer stuck to the top of your profile and get buried in your own news feed. Sure, they still show up in the main news feed where it may or may not be seen thanks to the amount of info being dumped on the page, but it seems pretty nonsensical to have someone go to your profile and not be able to see your status at a glance.

As for photos, let me break this down.

When...

...I go to the profile of a person I'm not friends with on FB and discover the entire thing is locked down. I cannot see anything at all beyond their profile photo.

...I click on "Photos" and get a blank page with the "LOL, you don't have permission to see any of the photos this user has uploaded" message.

...I cannot see anything in the album my friend is tagged in except for the photos they're tagged in.

Then I'm going to assume that yes, Facebook has allowed me limited access to private data that I should not have access to. If someone I'm not friends with sets an album privacy to "Friends of Friends" then I can cycle through the entire album. This is a security breach.

The Cheetah and the Elephant

[eldavojohn]

This is still a learning process as they're one of the biggest guys out there usage-wise. It's hard to tell what's real and what's fake but I reviewed Beautiful Data for Slashdot a while ago and am reminded of a chapter by Jeff Hammerbacher about Facebook's database in the early days. Maybe the culture is still a little bit like those early days with the database where they spend more time constantly addressing data issues and trying to step up to the next <prefix>abyte? You can go ahead and presume all you want but I'm going to guess the developers are pretty busy dealing with some serious scaling issues. Past performance of the website reinforces this and, let's face it, the "chat" feature was and always will be a nightmare to use.

At a staff of 200, I would also wager that new features take a back seat and would propose that this is why we see Facebook's "upgrades" as being almost purely cosmetic (i.e. layout, markup, etc.).

Of course they have no QA...

[RJHelms]

It's easy not to have QA when your users aren't paying you.

Google's in the same boat - the websites aren't the product, the end users are. When your website is your product this crap won't fly.

Licensed engineers != Engineers

[Chemisor]

An engineer is a professional practitioner of engineering, concerned with applying scientific knowledge, mathematics, economics and ingenuity to develop solutions to meet economic and societal needs. Engineers design structures, machines and systems while considering the limitations imposed by practicality, safety and cost. The word engineer is derived from the Latin root ingenium, meaning "cleverness" -Wikipedia

Nowhere does this mention licensing of any kind. Licenses and liability to lawsuits are a recent invention and are little more than unnecessary government intervention in the free market for the purpose of restricting the supply of engineers. This sort of guild mentality has always been detrimental to the economy by forcing people to pay for something they don't necessarily want. It would have been much better if all this licensing nonsense disappeared and we could rely on the traditional reputation system that the free market uses to maintain quality.

Re:Licensed engineers != Engineers

[Americano]

Okay, thought exercise:

You're walking down the street, and somebody walks up with a knife and demands your wallet. In the ensuing scuffle, you end up stabbed 3 times in the chest. Nothing instantly fatal, but your lungs are now filling with blood.

A passer-by calls an ambulance for you, and once you're loaded in and ready to go, the EMT pulls out a list of all the people who practice medicine in the area and says, "which doctor do you want to have care for you?"

You weakly point at one of the names on the list, J. R. McAllister III, and say "That guy has a reputable-sounding name, and he has a 4 star rating."

The ambulance delivers you to Mr. McAllister's door. Mr. McAllister looks at your wounds, shrugs, and says, "I just have no idea what to do here. I just deal in herbal remedies and acupuncture most of the time."

You die.

Still think that professional licensing is completely stupid, and a "traditional reputation system" is better? Or is there some room for professional licensing that guarantees a minimum standard of competency and training before someone can call themselves a "doctor" or an "engineer"?

Insider Attacks

[gweihir]

Urgh. That means every competent insider can easily put disguised backdoors into the code to be used later. Ordinarily you want to severely limit who can do that. In fact one fundamental principle when operating high security applications is that development personnel must not have access to the production environment, exactly to not hand all eggs to everybody.

Obviously a young and highly risk-taking company. Not good at all.

Re:Amazing

[neoform]

Scaling a site to be the 2nd most popular site on the web is no small feat. I would imagine a good number of people on their tech team are designated to managing the load.

Very common sentiment

[Anonymous Coward]

> I'd have thought that a team of 3-4 engineers could achieve the same effect.

This is very typical sentiment in people with little to no experience in running large sites. They believe a) What they see is all the exist, b) Scale does not matter, and c) The site never changes.

This lead them to the faulty conclusion that any high-availability, high-traffic site could be run by two people (of which they no doubt are one) live-updating PHP scripts on the fly. See Also: Dunning-Kruger effect.

Re:down the road

[guyfawkes-11-5]

Its for a different reason; GS created a vehicle with exactly 499 investors to specifically skirt a rules that requires any company with more than 500 investors to be publicly traded. The SEC woke up for once in their lives and started poking around, rather than taking the heat, GS created the vehicle overseas, and only offered it to non US clients. Shady and circumspect to say the least. Heres more info: http://dealbook.nytimes.com/2011/01/17/goldman-limits-facebook-investment-to-foreign-clients/

Took the Dive

[twotimes]

I registered for /. after years of lurking AND deleted my fb in the same day.

Re:Amazing

[MooseMuffin]

Back in 09, Blizzard said their programming team for WoW's engine, gameplay, tools, servers and UI was 32 people total.

http://wow.joystiq.com/photos/adgc09-the-universe-behind-world-of-warcraft/#2296009