How Facebook Ships Code

Hugh Pickens writes "The two largest teams at Facebook are Engineering and Ops, with roughly 400-500 team members each, together making up about 50% of the company. All engineers go through 4 to 6 week 'Boot Camp' training where they learn the Facebook system by fixing bugs. After boot camp, all engineers get access to the live DB and any engineer can modify any part of Facebook's code base and check-in at-will so that engineers can modify specs mid-process, re-order work projects, and inject new feature ideas anytime. Then arguments about whether or not a feature idea is worth doing or not generally get resolved by spending a week implementing it and then testing it on a sample of users, e.g., 1% of Nevada users. 'All changes are reviewed by at least one person, and the system is easy for anyone else to look at and review your code even if you don't invite them to,' writes yeegay. 'It would take intentionally malicious behavior to get un-reviewed code in.' What is interesting for a company this size is that there is no official QA group at Facebook but almost every employee is dogfooding the product every day."

Comments

no process

[mortonda]

any engineer can modify any part of Facebook's code base and check-in at-will

That explains a LOT...

Re:no process

[iluvcapra]

the culture of the company seems to be set so that *everyone* feels responsibility for the product

It's astonishing that they can keep such a process rolling with 500 engineers, let alone 200

product managers have a lot of independence and freedom. The key to being influential is to have really good relationships with engineering managers. Need to be technical enough not to suggest stupid ideas.

So basically, Facebook is run by an aggressive engineering culture based not on consensus or managerial decision making but by cliques and lobbying, where the worst thing someone can possibly do is suggest an idea that an engineer claims is "stupid" and doesn't give them opportunity for nerd glory. How much you want to bet that Zuckerberg sets the tone and decides that any modifications to the way the privacy settings are run is "stupid" and "boring."

Re:no process

[Anonymous Coward]

Facebook is run by an aggressive engineering culture based not on consensus or managerial decision making but by cliques and lobbying

At any other company, that could be counter-productive. But considering that the tool they're building is basically the ideal tool for building your clique and growing your influence, it probably just adds to their "dogfooding."

Re:no process

[ScentCone]

It's astonishing that they can keep such a process rolling with 500 engineers, let alone 200

I'm always curious about this expression, which - just like "I could care less" in place of "I could not care less" - is the opposite of what's meant. Surely, 200 would be astonishing, let alone 500.

Re:no process

[Hoi Polloi]

Facebook does have a formal QA group, it is called the user base.

Re:no process

[NoSig]

It doesn't just sound silly, it's a crime against intelligence is what it is. I doubt it's a contraction like you say since "as if I could care less" is not something I've ever heard, while "I couldn't care less" is. It's just a bunch of people who don't know what they are saying that cut out the "not" because they don't understand how the sentence is supposed to work. That's like contracting "I don't like Hitler" to "I like Hitler". I guess by your reckoning it would be like contracting "as if I like Hitler" to "I like Hitler" which is just as stupid, but I still think it's the former.

Re:no process

[NoSig]

I could care less about your question. In fact, I think it's very interesting. Thank you.

No access controls?

[choongiri]

After boot camp, all engineers get access to live DB

So anyone who's ever worked at FB as an engineer will have likely downloaded copies of all their friends' / family's / ex-girlfriends' inboxes, chat history, etc.

Not surprising really.

Fake

[MBCook]

It's too bad much of the article is wrong.

Re:Programmers != Engineers

[morgan_greywolf]

Not all engineers are licensed. Civil engineers are usually licensed. Mechanical engineers and electrical engineers are usually not licensed. Similarly, there is no licensure for system engineers. There are "certifications" but these are essentially meaningless.

Same old tune

[trollertron3000]

We get to read this story every few years. Some company makes a popular product and all of a sudden they are the authority on development. News flash guys, enterprise software wasn't just invented and we've been rolling out features to millions of people for years. In environments where any discrepancy can lead to serious legal repercussions. So just for a second, hear us out. We know what we're doing and we don't need to reinvent it every few years because the guys from Facebook, Basecamp, Hacker news, or any other place said we're doing it wrong. Our bottom line and years of quality deliveries say we're doing it right. What do these guys have besides a history of rolling out mistakes? Their value is based on hype, not their code quality. Some of us don't have the luxury to make huge mistakes and have our customers be okay with it.

Re:"dogfooding"?

[Urban Garlic]

The choosing is weirding, you're right, but as long as the meaning is clearing, it's not really probleming.

This explains bizarre feature regressions

[Y-Crate]

I know they're always looking to "streamline" Facebook, but sometimes really obviously good features vanish without explanation and leave users scratching their heads. These decisions would make more sense if you could see some way that FB was making life easier for advertisers or something, but often, the feature regressions are just nonsensical to the extreme.

A few:

- Facebook got rid of statuses. The one e-crack feature they're best known for. Gone with the new profile. Now they're just wall posts to yourself that quickly fall down the page. *facepalm*

- Photos are now uploaded in descending order. It used to be that a group of images would be uploaded like this: beachtrip-1.jpg, beachtrip-2.jpg... and so on. Now it reverses the order of an uploaded album. The last photos taken are displayed first. Posting vacation pics? Well your friends get to see the day you left and work their way back to the day you arrived.

- The "Reverse Photo Order" option was removed when the above "feature" was introduced. You can drag around photos to manually reorder them, but every photo that you mouse-over jumps out of the way and moves to different rows. It's a UI disaster.

- A few months ago I started seeing tagged photos of friends despite the fact they were in private albums of people I'm not friends with who also happen to have locked-down profiles where you can't see anything at all unless you add them.

Re:Programmers != Engineers

[ScentCone]

Engineer is just another cheap title, like CEO/CFO/CIO/CTO, etc, free to be used by anyone.

No, this is not true. There is a big difference between being the employee of a company and being an officer of the company. Those "O" titles actually mean something. Doesn't mean that the people who are officers of the company are the right people for those roles, but there's real baggage that comes with those titles, including a higher standard for the consequences of entering into contracts, obligating the company to act or pay bills, etc.Being an "O" also makes you more of a law suit magnet.

Re:Programmers != Engineers

[mini me]

Engineer is defined as someone who designs, builds, and maintains.

It has nothing to do with licensing, government accountability, code of ethics, accreditation, or anything else.

It is not a cheap title, it is a description of the job. When they say Facebook engineer, we get a sense of what those employees are required to do; which extends beyond the role of just programming.

Very common sentiment

[Anonymous Coward]

> I'd have thought that a team of 3-4 engineers could achieve the same effect.

This is very typical sentiment in people with little to no experience in running large sites. They believe a) What they see is all the exist, b) Scale does not matter, and c) The site never changes.

This lead them to the faulty conclusion that any high-availability, high-traffic site could be run by two people (of which they no doubt are one) live-updating PHP scripts on the fly. See Also: Dunning-Kruger effect.

Re:"dogfooding"?

[Locke2005]

The usual expression is "eating their own dog food". I've never heard it referred to as "dogfooding" either.