How Facebook Ships Code
Hugh Pickens writes "The two largest teams at Facebook are Engineering and Ops, with roughly 400-500 team members each, together making up about 50% of the company. All engineers go through 4 to 6 week 'Boot Camp' training where they learn the Facebook system by fixing bugs. After boot camp, all engineers get access to the live DB and any engineer can modify any part of Facebook's code base and check-in at-will so that engineers can modify specs mid-process, re-order work projects, and inject new feature ideas anytime. Then arguments about whether or not a feature idea is worth doing or not generally get resolved by spending a week implementing it and then testing it on a sample of users, e.g., 1% of Nevada users. 'All changes are reviewed by at least one person, and the system is easy for anyone else to look at and review your code even if you don't invite them to,' writes yeegay. 'It would take intentionally malicious behavior to get un-reviewed code in.' What is interesting for a company this size is that there is no official QA group at Facebook but almost every employee is dogfooding the product every day."
That explains a LOT...
Then where can I get that shipped code?
Although, after reading that story, I don't think I want that code. It sounds like a textbook example of feature and focus creep.
It's not like anything really changes on it.
i share the worry facebook represents in terms of privacy and the future of the internet with everyone else here
but at the same time, i get a warm feeling in my pants thinking about the programming and workings of an enterprise this big and powerful
and if you laugh at me for getting that warm feeling in my pants, you're clearly on the wrong website
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
After boot camp, all engineers get access to live DB
So anyone who's ever worked at FB as an engineer will have likely downloaded copies of all their friends' / family's / ex-girlfriends' inboxes, chat history, etc.
Not surprising really.
Once and for all, programmers are not Engineers.
Programmers can not be held liable, have no professional license, no government accountable body, no code of ethics, no liability insurance, or other measures in place to directly protect the public.
Think about it.
If facebook was staffed by real engineers, a privacy breach would cost the engineer his/her license, and they could be personally sued, as well.
But then again, anyone in the good old US of A is an engineer. From the sanitation engineer who picks up my trash, to the beverage refreshment engineer who makes my coffee at Starbucks.
Engineer is just another cheap title, like CEO/CFO/CIO/CTO, etc, free to be used by anyone.
Now we know how/why there have been such privacy issues if there is no formal process to determine what features should be included or not!
It's too bad much of the article is wrong.
Comment forecast: Bits of genius surrounded by a sea of mediocrity.
...that they have very poor/disorganised engineering practice, and they're trying to sell a bug in their methodology as a feature. It's like when Google suddenly fluffs something up with your search results momentarily, and you think "huh, that's weird," but actually they're just testing (or think they're testing... in reality, they're just annoying you).
Since there's nothing technically interesting or requiring resilience about anything Facebook has to offer, they can get away with it.
Way too many Facebook stories today. I wish that website would disappear. Just walk anywhere in public for 15 minutes and count how many times you hear a reference to this website. Sometimes the count reaches double digits. It is ridiculous. You would think there was nothing else in the world that mattered. People are boring *yawn*
I can see the social network equivalent to SOX or HIPAA coming down the road :/
-- if you mod me down, I will become more powerful than you can possibly imagine
But will they catch unintentionally malicious behavior that gets un-reviewed code in?
This one time, at Facebook camp ...
You just have to play with their annoying product for a little while and you see all the constant and apparently not thought out changes. I stopped using facebook years ago because the interface kept changing. I wrote some scripts to do some basic stuff and they kept breaking because of the constantly changing interface. Changes that just made no sense. One day a button would be there the next day gone. A week later it would come back. I don't see how people tolerate that. I guess they are so self absorbed that they don't even notice the changes.
Because fortunately there is no history of humanity ever acting with intentional malice. #phew
I went to the city because I wished to live without deliberation.
Bear in mind how Facebook works internally. It's a large number of programs intercommunicating through a remote procedure call system. There's no one big "build". The interfaces between programs are well defined, and changing out programs individually is normal.
Yeah, I had to look it up. Never heard of the term "dogfooding" before.
It seems a silly word choice, since in context it carries no more meaning than the rather more conventional word "using".
Bow-ties are cool.
I'm just laughing (sadly) that web developers are considered engineers.
But hey, they call garbage collectors "Sanitation engineers" so why not Javascript and PHP coders...
Does idiotic ideas like sharing my private info with any random app that is easily setup to be malicious get through?
Why dont they have a checks and balances with the executives that should be smacked hared in the head for their bad ideas?
Do not look at laser with remaining good eye.
We get to read this story every few years. Some company makes a popular product and all of a sudden they are the authority on development. News flash guys, enterprise software wasn't just invented and we've been rolling out features to millions of people for years. In environments where any discrepancy can lead to serious legal repercussions. So just for a second, hear us out. We know what we're doing and we don't need to reinvent it every few years because the guys from Facebook, Basecamp, Hacker news, or any other place said we're doing it wrong. Our bottom line and years of quality deliveries say we're doing it right. What do these guys have besides a history of rolling out mistakes? Their value is based on hype, not their code quality. Some of us don't have the luxury to make huge mistakes and have our customers be okay with it.
Tiger Blooded Bi-Winning Machine
I know they're always looking to "streamline" Facebook, but sometimes really obviously good features vanish without explanation and leave users scratching their heads. These decisions would make more sense if you could see some way that FB was making life easier for advertisers or something, but often, the feature regressions are just nonsensical to the extreme.
A few:
- Facebook got rid of statuses. The one e-crack feature they're best known for. Gone with the new profile. Now they're just wall posts to yourself that quickly fall down the page. *facepalm*
- Photos are now uploaded in descending order. It used to be that a group of images would be uploaded like this: beachtrip-1.jpg, beachtrip-2.jpg... and so on. Now it reverses the order of an uploaded album. The last photos taken are displayed first. Posting vacation pics? Well your friends get to see the day you left and work their way back to the day you arrived.
- The "Reverse Photo Order" option was removed when the above "feature" was introduced. You can drag around photos to manually reorder them, but every photo that you mouse-over jumps out of the way and moves to different rows. It's a UI disaster.
- A few months ago I started seeing tagged photos of friends despite the fact they were in private albums of people I'm not friends with who also happen to have locked-down profiles where you can't see anything at all unless you add them.
This is still a learning process as they're one of the biggest guys out there usage-wise. It's hard to tell what's real and what's fake but I reviewed Beautiful Data for Slashdot a while ago and am reminded of a chapter by Jeff Hammerbacher about Facebook's database in the early days. Maybe the culture is still a little bit like those early days with the database where they spend more time constantly addressing data issues and trying to step up to the next <prefix>abyte? You can go ahead and presume all you want but I'm going to guess the developers are pretty busy dealing with some serious scaling issues. Past performance of the website reinforces this and, let's face it, the "chat" feature was and always will be a nightmare to use.
At a staff of 200, I would also wager that new features take a back seat and would propose that this is why we see Facebook's "upgrades" as being almost purely cosmetic (i.e. layout, markup, etc.).
My work here is dung.
I'm amazed that "Facebook" needs about 1000 engineers for their site. I'd have thought that a team of 3-4 engineers could achieve the same effect. Honestly, I'm not joking. Then again, I haven't looked at their site for more than a year and no longer have an account, so perhaps its by now powered by rocket science.
First, this certainly explains Facebook's "glitch-of-the-hour" syndrome. Second, let's all pray that these folks never get to work on anything related to avionics, life support or nuclear power plants.
It's easy not to have QA when your users aren't paying you.
Google's in the same boat - the websites aren't the product, the end users are. When your website is your product this crap won't fly.
Ken Thompson got annoyed that Unix users would call him up or email him complaining about bugs, but would refuse to give him root access to their systems to snoop around. So he added a seemingly innocuous chunk of code to the Unix source, that would be transformed by the C compiler into a backdoor for him. Then when folks called up to complain, he would instruct them to recompile the kernel, and presto, he could get in to debug.
When describing this years later, he quipped something like, "Never trust or hire a guy like me."
So the summary says, "All changes are reviewed by at least one person, and the system is easy for anyone else to look at and review your code even if you don't invite them to"
So folks can review the code . . . can they understand what it really does?
Schroedinger's Brexit: The UK is both in and out of the EU at the same time!
I guess slashdot is also missing a bit of QA :)
This isn't that surprising if you're trying to figure out how Facebook works by looking at it from the outside. Facebook has always looked like a mangled mish-mash of features. Things get implemented pretty much a random with no apparent regard for an overarching design or vision. Facebook looks like one gigantic hack. Turns out, it is.
Nowhere does this mention licensing of any kind. Licenses and liability to lawsuits are a recent invention and are little more than unnecessary government intervention in the free market for the purpose of restricting the supply of engineers. This sort of guild mentality has always been detrimental to the economy by forcing people to pay for something they don't necessarily want. It would have been much better if all this licensing nonsense disappeared and we could rely on the traditional reputation system that the free market uses to maintain quality.
How would you know? Are you one of their paying customers?
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
and Facebook is dog food.
Funny coincidence, no?
While I'm not wild about the term, dogfooding your own product is not a bad way to find bugs, and also to see if the product sucks or not. Things people want are likely things customers want. Fix your niche wish (wouldn't it be great if we had...) and you are likely (with Facebook) fixing 10,000,000 users wish. I occasionally go to an upscale Chinese restaurant. Sure, they flip the dirty tablecloths over, and if the other side is dirty too, they pull it off quick, and put on a clean one. But the thing I remember most is that before they open (at noon), all the waiters, the chefs, and the boss sit down to lunch. If there is any problem with the food (or it isn't quite right), everyone will know right away, before they serve it to customers, so customers get the version thats been past quality control. The chefs sit across from the boss. I always wonder about companies that prefer other companies products over their own. There is something insincere about it.
Urgh. That means every competent insider can easily put disguised backdoors into the code to be used later. Ordinarily you want to severely limit who can do that. In fact one fundamental principle when operating high security applications is that development personnel must not have access to the production environment, exactly to not hand all eggs to everybody.
Obviously a young and highly risk-taking company. Not good at all.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Programmers can not be held liable, have no professional license, no government accountable body, no code of ethics, no liability insurance
So government is what makes a good engineer? Pfft. I think we all know what really makes a good engineer:
- Knowledge
- Experience
- Precision
- Imagination
- Genuine interest in the subject
At best, government simply verifies some of the above.
(Let me guess -- you're one of those people who spent gobs of time and money petitioning government to call yourself an engineer, and now you're complaining because you're out of work and "god damn it, I earned the right to an engineering job".)
> I'd have thought that a team of 3-4 engineers could achieve the same effect.
This is very typical sentiment in people with little to no experience in running large sites. They believe a) What they see is all the exist, b) Scale does not matter, and c) The site never changes.
This lead them to the faulty conclusion that any high-availability, high-traffic site could be run by two people (of which they no doubt are one) live-updating PHP scripts on the fly. See Also: Dunning-Kruger effect.
That does seem high. 1000 engineers is indeed ridiculous. Perhaps 1000 are needed to keep scotch taping the site together. It might very well be a fragile mess of servers and services just barely hanging on by a thread...
This guy's obviously fresh out of college. It would be interesting to hear from someone with a little more real-world experience.
I registered for /. after years of lurking AND deleted my fb in the same day.
a 27-year-old former Google engineer repeatedly took advantage of his position as a member of an elite technical group at the company to access users' accounts
Although SREs are probably a privileged subgroup and ordinary feature engineers do not have access to all data. Elementary measure, but when did Facebook ever care about your privacy?
you had me at #!
That is a narrow definition of engineer. Surely almost everyone here understands engineer to mean a PE; like a civil, chemical, mechanical, electrical, &c, engineer, but maybe I'm assuming too much.
Maybe people just call themselves engineers because the label makes them feel a bit like actual licensed professional engineers who pass certification exams and have liability insurance.
FWIW, I'm not an engineer, and I can't ethically or legally sell my services by calling myself one, being in Canada.
"What is interesting for a company this size is that there is no official QA group at Facebook..."
No kidding!!
Keep it up, hacks.
Kriston
Aha, this is definitely not SOX compliance!
I wonder how are they planning to manage the SOX heavy auditions before going public if every single developer has access to the live DB,etc.
Anyone have a guesstimate as to the size of the backend, frontend, and synchronization/distribution codebase sizes?
It sounds to me like FB is rolling in so much money, that they may be throwing 150% more development staff at the project than they really need. If money were object, my bet is that they would find that their process model is wildly skewed from optimal for their business.
I guess by your reckoning it would be like contracting "as if I like Hitler"
Yes, using strict grammatical logic, it is contradictory. You get a cookie and a gold star for spotting something illogical in the English language.
While isn't literally what the speaker is intending, everybody in the western world understands the intent. The English language isn't a static construct, and it's richness is a result of it's tendency to beg borrow or steal everything under the sun, and change it around at the speaker's whim. Yeah, it mangles the king's English, but who really cares?
HA! I just wasted some of your bandwidth with a frivolous sig!
'400-500 team members each' and they still can't fix the horribly buggy instant messaging system...
almost every employee is dogfooding the product every day.
Problem, is, like a dog, they tend to eat their own shit too.
Obviously stolen from Python and no doubt misspelling in frog.
John McAfee 'It was like that time I hired that Bangkok prostitute; to do my taxes, while I fucked my accountant'
it's part of stupid-English, the anti-language spoken by stupid people. It's actually very nice of them to self-identify like that.
Spend less time whining about the font that a book is written in, and more time paying attention to the content.
HA! I just wasted some of your bandwidth with a frivolous sig!
WTF is with Facebook and 500+ engineers?
Meetup.com has 63 people, not all engineers, and they deal with a client base which uses their product in a much more demanding way than Facebook's users. I look at Meetup.com as more of a performance-critical application, at least in terms of their users. If Meetup gets a bug, it affects whole groups of people - a lot of them engaged in technical, educational, aid and political activities - trying to organise themselves. I'd like to say it again - *63 employees*.
Facebook, on it's face (so to speak), is not rocket science. Neither is Meetup. This indicates to me that there's a LOT going on behind the scenes and most of it has to do with implementing the API and other technology to support all the marketing, advertising, strategic partnerships, etc. Either that or the UI changes are deceptively complicated.
Yeah, I'd love to know what those 500 people are mostly coding. Perhaps this is an indication of the difference between an advertising-run business vs subscription. One of them allows you to get on with what matters to your users. If anything shows where Facebooks's primary focus is, this does.