How Facebook Ships Code

Hugh Pickens writes "The two largest teams at Facebook are Engineering and Ops, with roughly 400-500 team members each, together making up about 50% of the company. All engineers go through 4 to 6 week 'Boot Camp' training where they learn the Facebook system by fixing bugs. After boot camp, all engineers get access to the live DB and any engineer can modify any part of Facebook's code base and check-in at-will so that engineers can modify specs mid-process, re-order work projects, and inject new feature ideas anytime. Then arguments about whether or not a feature idea is worth doing or not generally get resolved by spending a week implementing it and then testing it on a sample of users, e.g., 1% of Nevada users. 'All changes are reviewed by at least one person, and the system is easy for anyone else to look at and review your code even if you don't invite them to,' writes yeegay. 'It would take intentionally malicious behavior to get un-reviewed code in.' What is interesting for a company this size is that there is no official QA group at Facebook but almost every employee is dogfooding the product every day."

Comments

no process

[mortonda]

any engineer can modify any part of Facebook's code base and check-in at-will

That explains a LOT...

Re:no process

[iluvcapra]

the culture of the company seems to be set so that *everyone* feels responsibility for the product

It's astonishing that they can keep such a process rolling with 500 engineers, let alone 200

product managers have a lot of independence and freedom. The key to being influential is to have really good relationships with engineering managers. Need to be technical enough not to suggest stupid ideas.

So basically, Facebook is run by an aggressive engineering culture based not on consensus or managerial decision making but by cliques and lobbying, where the worst thing someone can possibly do is suggest an idea that an engineer claims is "stupid" and doesn't give them opportunity for nerd glory. How much you want to bet that Zuckerberg sets the tone and decides that any modifications to the way the privacy settings are run is "stupid" and "boring."

Re:no process

[Anonymous Coward]

Facebook is run by an aggressive engineering culture based not on consensus or managerial decision making but by cliques and lobbying

At any other company, that could be counter-productive. But considering that the tool they're building is basically the ideal tool for building your clique and growing your influence, it probably just adds to their "dogfooding."

Re:no process

[ScentCone]

It's astonishing that they can keep such a process rolling with 500 engineers, let alone 200

I'm always curious about this expression, which - just like "I could care less" in place of "I could not care less" - is the opposite of what's meant. Surely, 200 would be astonishing, let alone 500.

Re:no process

[somersault]

The engineers in Dilbert are fine, it's the managers you don't want.

Re:no process

[Hoi Polloi]

Facebook does have a formal QA group, it is called the user base.

Re:no process

[syousef]

any engineer can modify any part of Facebook's code base and check-in at-will

That explains a LOT...

Personal journal: This company trusts me with the code base. I don't know why. Dumb fucks!

Re:no process

[iluvcapra]

Something TFA doesn't talk about is how the engineers are organized -- surely there's a team that does memcache, a team that does database work (like those Cassandra guys), a team that does i18n, a team that does rendering, a team that specializes in Flash doodads, etc. It's hard to see how the engineers check each other without knowing where demarcations lie.

My issue wouldn't be with the quality of the code, it's how they decide what code to write in the first place. All of the Kaizen processes work great on an assembly line, but a feature request and a coworker evaluation aren't fender panels. They require someone in authority to tell you what you're trying to accomplish in the first place. The description of the way people decide what to add (basically shutting out the marketing people if they speak out too much) and how they discipline each other (too many blames in the SVN log and too many "public shamings" result in termination) speaks to a culture where new ideas and customer focus are almost stigmatized, and where the engineers don't try to evaluate each others ideas as much as they worry about getting voted off the island.

Re:no process

[tsm_sf]

Except that tasks become increasingly difficult the more nerds you throw at them, not less. 500 engineers makes a project harder than 200.

Re:no process

[ScentCone]

Except that tasks become increasingly difficult the more nerds you throw at them, not less. 500 engineers makes a project harder than 200.

Which is exactly my point. It's amazing that the casual way they handle code could work with 200 programmers, let alone 500. See? "Let alone" means "never mind." As in, it's hard enough for 200, and 500 isn't even worth mentioning ... so, let it alone. And that's why I pointed out that the phrase was being used exactly backwards.

Re:no process

[catmistake]

based not on consensus or managerial decision making but by cliques and lobbying

I have worked, via contract, in at least a dozen medium to large sizes companies where this is the underlying mechanism. If you stick out in that you focus solely on the work and do not engage with the de facto clique leaders, who as often discuss droll subjects such as retiling their bathroom as often as they collectively decide how they want you to do their jobs for them, then absurd personal complaints will begin to stack up against you and you will be forced out of a job.

I used to flaunt my individuity, but since 9/11 I have been consciously conforming to old school business formality... never arrive late, never leave early, never get sick or take off, always dress nice, keep short hair, no superfluous non-work related conversations, never mentioning anything from my personal life, always ready with a polite smile and an enthusiasm for the work no matter how mundane it is... I've tried to embody what the ideal is for a person in my field. This doesn't work. One needs to become *good* friends with whomever that defacto leader is, who is as often as not NOT the management (but, indeed, through strength of personality has the manager under their thumb, and completely snowjobs the executives with an almost supernatural confidence).

If they like you, you win. And when you win, they like you. Otherwise... you are the first of any approaching cutbacks... you become the sacrifice that saves everyone elses jobs, the scapegoat for any complaints that come down from the executive levels.

The last 2 teams I worked with were more like a gang than a corporate division, whose self-preservation far outweights the work they are responsible for (My last contract ended abruptly when 3 of my counterparts in another building took heavy criticism for their laziness and distict lack of any work ethic... but because they were well liked, somehow I got the boot and the blame for their incompetance... without even remotely having anything to do with whatever incident occurred (never quite clear on exactly what it was... something to do with an assigned task that they kept passing between them and was never completed).

I hope someday my perserverance will pay off and I will be able to work not only with smart people (intelligence was rarely an issue), but educated and honest, and hopefully enlightened individuals that do not synthesize drama to manipulate perceptions such that those they like, towing the gang's loyalty, remain employed, while those that are effective, and thus throwing the curve, are terminated.

Re:no process

[NoSig]

It doesn't just sound silly, it's a crime against intelligence is what it is. I doubt it's a contraction like you say since "as if I could care less" is not something I've ever heard, while "I couldn't care less" is. It's just a bunch of people who don't know what they are saying that cut out the "not" because they don't understand how the sentence is supposed to work. That's like contracting "I don't like Hitler" to "I like Hitler". I guess by your reckoning it would be like contracting "as if I like Hitler" to "I like Hitler" which is just as stupid, but I still think it's the former.

Re:no process

[NoSig]

I could care less about your question. In fact, I think it's very interesting. Thank you.

No access controls?

[choongiri]

After boot camp, all engineers get access to live DB

So anyone who's ever worked at FB as an engineer will have likely downloaded copies of all their friends' / family's / ex-girlfriends' inboxes, chat history, etc.

Not surprising really.

Fake

[MBCook]

It's too bad much of the article is wrong.

Re:Fake

[FuckingNickName]

Great, a series of statements about Facebook by some people not speaking on behalf of Facebook corrected by one person claiming to be from Facebook but not speaking on behalf of Facebook. Who knew Facebook engineers and fanboys were as capricious and unwilling to understand boundaries as the users?

Re:Fake

[trollertron3000]

Sir the post you cite actually confirms a lot of this is true. Such as the lack of QA. They are all encouraged to test and then report any bug in a new internal version? Is that supposed to be sufficient?

Just more proof that Facebook really doesn't care about anything except driving eyeballs to ads. I bet _that_ system is NASA engineered.

Intentionally malicious code, eh?

[noidentity]

It would take intentionally malicious behavior to get un-reviewed code in.

But will they catch unintentionally malicious behavior that gets un-reviewed code in?

Re:Programmers != Engineers

[morgan_greywolf]

Not all engineers are licensed. Civil engineers are usually licensed. Mechanical engineers and electrical engineers are usually not licensed. Similarly, there is no licensure for system engineers. There are "certifications" but these are essentially meaningless.

It's a component system

[Animats]

Bear in mind how Facebook works internally. It's a large number of programs intercommunicating through a remote procedure call system. There's no one big "build". The interfaces between programs are well defined, and changing out programs individually is normal.

"dogfooding"?

[Tetsujin]

Yeah, I had to look it up. Never heard of the term "dogfooding" before.

It seems a silly word choice, since in context it carries no more meaning than the rather more conventional word "using".

Re:"dogfooding"?

[Urban Garlic]

The choosing is weirding, you're right, but as long as the meaning is clearing, it's not really probleming.

Re:"dogfooding"?

[Locke2005]

The usual expression is "eating their own dog food". I've never heard it referred to as "dogfooding" either.

Same old tune

[trollertron3000]

We get to read this story every few years. Some company makes a popular product and all of a sudden they are the authority on development. News flash guys, enterprise software wasn't just invented and we've been rolling out features to millions of people for years. In environments where any discrepancy can lead to serious legal repercussions. So just for a second, hear us out. We know what we're doing and we don't need to reinvent it every few years because the guys from Facebook, Basecamp, Hacker news, or any other place said we're doing it wrong. Our bottom line and years of quality deliveries say we're doing it right. What do these guys have besides a history of rolling out mistakes? Their value is based on hype, not their code quality. Some of us don't have the luxury to make huge mistakes and have our customers be okay with it.

This explains bizarre feature regressions

[Y-Crate]

I know they're always looking to "streamline" Facebook, but sometimes really obviously good features vanish without explanation and leave users scratching their heads. These decisions would make more sense if you could see some way that FB was making life easier for advertisers or something, but often, the feature regressions are just nonsensical to the extreme.

A few:

- Facebook got rid of statuses. The one e-crack feature they're best known for. Gone with the new profile. Now they're just wall posts to yourself that quickly fall down the page. *facepalm*

- Photos are now uploaded in descending order. It used to be that a group of images would be uploaded like this: beachtrip-1.jpg, beachtrip-2.jpg... and so on. Now it reverses the order of an uploaded album. The last photos taken are displayed first. Posting vacation pics? Well your friends get to see the day you left and work their way back to the day you arrived.

- The "Reverse Photo Order" option was removed when the above "feature" was introduced. You can drag around photos to manually reorder them, but every photo that you mouse-over jumps out of the way and moves to different rows. It's a UI disaster.

- A few months ago I started seeing tagged photos of friends despite the fact they were in private albums of people I'm not friends with who also happen to have locked-down profiles where you can't see anything at all unless you add them.

The Cheetah and the Elephant

[eldavojohn]

This is still a learning process as they're one of the biggest guys out there usage-wise. It's hard to tell what's real and what's fake but I reviewed Beautiful Data for Slashdot a while ago and am reminded of a chapter by Jeff Hammerbacher about Facebook's database in the early days. Maybe the culture is still a little bit like those early days with the database where they spend more time constantly addressing data issues and trying to step up to the next <prefix>abyte? You can go ahead and presume all you want but I'm going to guess the developers are pretty busy dealing with some serious scaling issues. Past performance of the website reinforces this and, let's face it, the "chat" feature was and always will be a nightmare to use.

At a staff of 200, I would also wager that new features take a back seat and would propose that this is why we see Facebook's "upgrades" as being almost purely cosmetic (i.e. layout, markup, etc.).

Of course they have no QA...

[RJHelms]

It's easy not to have QA when your users aren't paying you.

Google's in the same boat - the websites aren't the product, the end users are. When your website is your product this crap won't fly.

Re:Programmers != Engineers

[Mentally_Overclocked]

I have a degree from a university in electrical engineering. I work as an electrical engineer and I consider myself one. I am not licensed as a professional engineer (PE) and have little interest in obtaining that license at this point as the type of work simply doesn't appeal to me.

As you suggest, those PEs do put their license on the line when they sign a document. From my understanding, companies that have resident PEs will only have a few and have other non-licensed engineers do the less expensive work.

If it is a product it will usually need to meet expectations set by a different regulatory body (ETL, UL, FCC, whatever). If it is a building, power related, whatever, then it requires the review of a PE.

http://en.wikipedia.org/wiki/Professional_Engineer

I'm not really sure how they would regulate Facebook with their data ... I've never dealt with something like that.

Re:Programmers != Engineers

[ScentCone]

Engineer is just another cheap title, like CEO/CFO/CIO/CTO, etc, free to be used by anyone.

No, this is not true. There is a big difference between being the employee of a company and being an officer of the company. Those "O" titles actually mean something. Doesn't mean that the people who are officers of the company are the right people for those roles, but there's real baggage that comes with those titles, including a higher standard for the consequences of entering into contracts, obligating the company to act or pay bills, etc.Being an "O" also makes you more of a law suit magnet.

Re:Programmers != Engineers

[mini me]

Engineer is defined as someone who designs, builds, and maintains.

It has nothing to do with licensing, government accountability, code of ethics, accreditation, or anything else.

It is not a cheap title, it is a description of the job. When they say Facebook engineer, we get a sense of what those employees are required to do; which extends beyond the role of just programming.

Very common sentiment

[Anonymous Coward]

> I'd have thought that a team of 3-4 engineers could achieve the same effect.

This is very typical sentiment in people with little to no experience in running large sites. They believe a) What they see is all the exist, b) Scale does not matter, and c) The site never changes.

This lead them to the faulty conclusion that any high-availability, high-traffic site could be run by two people (of which they no doubt are one) live-updating PHP scripts on the fly. See Also: Dunning-Kruger effect.