Facebook Seeks Devs To Make Linux Network Stack As Good As FreeBSD's

← Back to Stories (view on slashdot.org)

Facebook Seeks Devs To Make Linux Network Stack As Good As FreeBSD's

Posted by timothy on Wednesday August 6, 2014 @05:30AM from the high-praise-all-around dept.

An anonymous reader writes Facebook posted a career application which, in their own words is 'seeking a Linux Kernel Software Engineer to join our Kernel team, with a primary focus on the networking subsystem. Our goal over the next few years is for the Linux kernel network stack to rival or exceed that of FreeBSD.' Two interesting bullet points listing "responsibilities": Improve IPv6 support in the kernel, and eliminate perf and stability issues. FB is one of the worlds largest IPv6 deployments; Investigate and participate in emerging protocols (MPTCP, QUIC, etc) discussions,implementation, experimentation, tooling, etc.

107 of 195 comments (clear)

Min score:

Reason:

Sort:

This does pose the question: by Anonymous Coward · 2014-08-06 05:35 · Score: 1, Insightful

Why not use FreeBSD? It's already there and at least as good as linux. Or have they perhaps hung themselves on systemd?
1. Re:This does pose the question: by HuguesT · 2014-08-06 06:00 · Score: 4, Informative
  
  I love FreeBSD, I support them financially every year, and I use it daily but it is not uniformly better than Linux. Hardware support, in particular, is very far behind. Two random examples:
  1- My NAS system does not recognise any USB storage when they are plugged in after boot (no hotplug). It does not support USB superspeed (USB 3.0) either (I have to boot in compatibility mode by disabling xHCI in the BIOS). This is a known issue with some Asus motherboards, still unfixed in 10.0
  2- FreeBSD does not install on some of my HP G6 servers. The kernel simply segfaults. I really wanted FreeBSD on this hardware, so I run it in a VM under Linux (using KVM). Has been running brilliantly for about 2 years now.
  Also security update in FreeBSD are really difficult. I haven't finished dealing with updating my ports since I moved from 9.2 to 9.3 last week.
  I have to say this though: when it runs, it runs really well.
2. Re:This does pose the question: by ci4 · 2014-08-06 06:35 · Score: 4, Interesting
  
  USB3 support in FreeBSD 10 is OK (bunch of external disks used for PC backup - speed was essential). No problem with hot-plug either. Ports upgrade is trivial (although I have switched to pkg-ng now). I really can't find why do you think that security updates are difficult either. I've got only one 9.2 system around which I at the moment am not bothered to upgrade.
3. Re:This does pose the question: by jdew · 2014-08-06 06:39 · Score: 2
  
  What asus board? I've got an asus crosshair iv formula and the usb3 has been working fine in both 9.2 and 9.3. The aibs driver doesn't support this board yet though.
  as for security updates:
  freebsd-update fetch; freebsd-update install
  and port updates:
  portsnap fetch update; portmaster -gda
  So not sure what the issues are here.
4. Re:This does pose the question: by TheRaven64 · 2014-08-06 06:45 · Score: 4, Interesting
  
  I'm pretty sure that Facebook buys enough hardware that they can afford to write drivers for anything they're missing and demand FreeBSD support from vendors for their next round of purchases. Netflix already does this (they won't buy any hardware that doesn't have vendor support for FreeBSD), as do a few other companies, and so a number of NIC vendors (particularly in the 10G/40G space) are now putting quite a bit of effort into their FreeBSD drivers.
  
  --
  I am TheRaven on Soylent News
5. Re:This does pose the question: by Fweeky · 2014-08-06 06:47 · Score: 4, Interesting
  
  pkgng's made port upgrading much less burdensome - even fairly complex dependency changes can be handled automatically as of 1.3, and the official package repositories are a lot more useful now. They even have stable security-fix-only branches.
  I still make my own customised builds, but I make binary packages in an isolated jail using poudriere. 99% of upgrades are a matter of updating its ports tree, running rebuild-packages, and running pkg upgrade on all my machines.
  You couldn't pay me to go back to portupgrade/portmaster/portmanager.
6. Re:This does pose the question: by lgw · 2014-08-06 07:10 · Score: 4, Informative
  
  If you have 1 million Linux machines deployed, with full Linux-specific software stacks on each, it's cheaper and easier (and most likely faster) to fix the problems you see in Linux than to move the fleet to a new OS.
  Facebook's dev shop culture is all about banging out code as fast as possible for the problem in front of them, then moving on. Forward planning isn't really the thing there, from what I hear (though I think you're no longer discouraged from testing your code before it goes live, that's a recent change). Moving to BSD might well be a better long term plan, but it would take years to get there and they don't really think on that timescale, from the rumors I hear anyhow.
  
  --
  Socialism: a lie told by totalitarians and believed by fools.
7. Re:This does pose the question: by Pieroxy · 2014-08-06 07:10 · Score: 4, Insightful
  
  Don't you think it's easier and cheaper to optimize the network stack of Linux rather than writing tons of hardware drivers for FreeBSD? Hardware which, most of the time, will be undocumented. Furthermore, when you change your servers, yay, more drivers to write...
  
  --
  Write boring code, not shiny code!
8. Re:This does pose the question: by Anonymous Coward · 2014-08-06 07:54 · Score: 1
  
  > I think you're no longer discouraged from testing your code before it goes live, that's a recent change
  You think you are making facebook look like idiots, but all you are doing is casting doubt on your own credibility.
  Extraordinary claims like that demand extraordinary proof.
9. Re:This does pose the question: by Score+Whore · 2014-08-06 08:04 · Score: 2
  
  Facebook buys custom servers, so will be 100% documented. Also they are of the vanity free variety lacking any bolted on bits added strictly to make the numbered list of features on the side of the box longer. I suspect that the only thing they are going to care about are disks and nics. Sounds cards, video cards, random USB hardware, bluetooth, none of that matters to them at all. These are datacenter housed pieces of equipment.
10. Re:This does pose the question: by lgw · 2014-08-06 08:28 · Score: 2
  
  Well, I only know this stuff second hand, but Facebook's dev motto used to be "go fast and break stuff". They had a very hard time hiring senior devs for some reason. Recently, they've had a bit of a (recruiting world) push to let people know that their new motto is "go fast with stable infra". I hear there used to be a banner on the wall proclaiming "don't test, just ship", but to be fair I haven't seen that myself.
  
  --
  Socialism: a lie told by totalitarians and believed by fools.
11. Re:This does pose the question: by ShanghaiBill · 2014-08-06 10:25 · Score: 1
  
  You think you are making facebook look like idiots
  No, he's making them look smart. 30 years in the software business has taught me that being first is a lot more important than being best. Successful companies whip up a quick fix and push it out the door, while the losers are busy doing another round of code review and testing. More rigorous engineering may be better in the long run, but there is no long run if you are bankrupt because you missed your opportunity window.
12. Re:This does pose the question: by gangien · 2014-08-06 13:53 · Score: 2
  
  I interviewed there, and there and had a dude with under a year of experience out of college interview me. and of course, give a horrible interview. I'm pretty sure that's the reason I didn't get the offer (of course who knows, maybe other reasons). they have this idea that an engineer is an engineer, which I think is good. to an extent, but in the end it's a bullshit philosophy.
13. Re:This does pose the question: by TheGratefulNet · 2014-08-06 14:20 · Score: 1
  
  I used to run a lot of freebsd at home - until I tried linux for samba and never went back to bsd again. bsd samba simply sucks. its dog slow and it hammers on disks, unlike linux which is very cache-oriented (from just looking at the drive lights).
  that was my main complaint. maybe linux has better integration with smb and the network stack but bsd is just not nearly as fast.
  stability wise, this isn't 10 yrs ago and linux stays up as long as you need it (years, even).
  bsd has better mgmt, though; one distro and one style of doing things. THAT is much nicer than the linux fragmentation.
  
  --
  
  --
  "It is now safe to switch off your computer."
14. Re:This does pose the question: by CrankyFool · 2014-08-06 15:46 · Score: 2
  
  Two comments:
  1. Their internal motto was, in fact, "go fast and break stuff"; I know this first-hand because I talked with them about that at my interview back in February, where they mentioned that they've changed to "go fast and be bold" because, in fact, they were trying to lower incidents of availability hits;
  2. 20+ years of tech industry experience here, and I was totally ready to be interviewed by some snot-nosed kid. What I got instead was an interview panel whose average tech industry tenure was around 17 years. I was, uniformly, impressed with the caliber of the people I met with there -- they verged from "pretty decent" in one case, to "pretty great" in all but two other cases, to "I'd take a $10K pay cut to work with this person" for the last two people. I was pretty surprised, and delighted.
15. Re:This does pose the question: by TheRaven64 · 2014-08-06 20:45 · Score: 1
  
  The network stack is a complex bit of code that interacts in complex ways with the scheduler, the generic locking frameworks, the driver infrastructure, the bus and DMA, infrastructure and everything up to the system call layer. The device drivers are self-contained and (individually) small pieces of code. Which do you think is cheaper and easier to modify?
  
  --
  I am TheRaven on Soylent News
16. Re:This does pose the question: by TheRaven64 · 2014-08-06 20:53 · Score: 1
  
  The binary package manager is a pain in the ass in FreeBSD/quote. Have you used FreeBSD since the package infrastructure was replaced? If so, we'd be interested in bug reports that are more detailed than 'it's a pain in the ass'.
  
  --
  I am TheRaven on Soylent News
17. Re:This does pose the question: by Stuarticus · 2014-08-06 23:39 · Score: 1
  
  If Facebook went down for a few days it would be the bigger news than the war in Gaza, my bank has more down time than facebook.
  
  --
  If you think someone isn't free to have a different definition of "freedom" you may be a tyrant.
18. Re:This does pose the question: by Pieroxy · 2014-08-07 00:40 · Score: 1
  
  Well, they may be self contained (which is not true because NIC drivers interact directly with the network stack for example) and small, but they are also highly untestable and undocumented.
  What I see is a stack to fix and a continuous stream of drivers to write. The one shot operation is bound to be cheaper on the long term.
  
  --
  Write boring code, not shiny code!
19. Re:This does pose the question: by TheRaven64 · 2014-08-07 02:25 · Score: 2
  
  NIC drivers interact directly with the network stack only via a set of well-defined interfaces (in FreeBSD at least), and writing the drivers is a one-shot operation because for new purchases they can demand FreeBSD drivers and support from their vendors (as Netflix, Verisign, and others do already). In the networking space, enough big companies are already demanding FreeBSD support for the high-end cards that the drivers are already likely to exist, so the missing ones are likely to be other things.
  
  --
  I am TheRaven on Soylent News
20. Re:This does pose the question: by Ben+Hutchings · 2014-08-07 03:12 · Score: 1
  
  Right. They want FreeBSD drivers, they can put that on the requirements list for network and storage controller vendors. But that does leave the issue of where the vendors are going to find good FreeBSD hackers to write these drivers.
21. Re:This does pose the question: by Mdk754 · 2014-08-07 03:49 · Score: 2
  
  Do you know why Linux seems to do so much better on VM NIC performance?
  
  Honestly curious, not trying to troll. In my ESXi tests I can manage more than double the throughput on a Linux VM compared to a BSD VM. Is there just crappy paravirtual NIC support in BSD?
  
  Tests were done with IPFIre and pfSense on two different ESXi boxes, and going to virtual client/server on etiher end of the router to avoid gigabit wire speed limits. On my slower box, i got 6xxMB/s with pfsense and 2.xxGB/s with IPFire. Minimal speed difference in either OS when using vmnet3 vs e1000. e1000 offered minimal throughput increase, vmxnet3 offered minimal host CPU usage decrease.
22. Re:This does pose the question: by evilviper · 2014-08-07 04:30 · Score: 1
  
  2- FreeBSD does not install on some of my HP G6 servers. The kernel simply segfaults.
  Did you try disabling anything and everything (drivers) in the kernel? Or perhaps just blacklisting most/all the modules?
  OpenBSD makes this easier with UKC and "config -e".
  
  --
  Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
23. Re:This does pose the question: by RockDoctor · 2014-08-12 22:24 · Score: 1
  
  they can put that on the requirements list for network and storage controller vendors. But that does leave the issue of where the vendors are going to find good FreeBSD hackers to write these drivers.
  Isn't there a cart and horse misalignment here. If you're a hardware vendor looking to get a big (and FB are big buyers of hardware) contract, and the tendering terms include "must have drivers for [Open Source system]", then won't you look at what you've already got on the shelf in terms of capabilities, and then burn custom firmware - or even an entire new product - that fits that driver at the hardware/ register level.
  Or, if you know that your NewShiny line of product is likely to be punted to FB by Sales, then you damned well make sure that it works with $FB-favoured-OS$ while you're still designing the damned thing.
  
  --
  Birds are not dinosaur descendants;birds are dinosaurs, for all useful meanings of "birds", "are" and "dinosaurs"
24. Re:This does pose the question: by Ben+Hutchings · 2014-08-13 06:02 · Score: 1
  
  ASICs generally aren't flexible enough that you could simply emulate another controller in firmware, while FPGAs suck too much power to use on commodity network adapters. Writing a new driver (or bringing an existing neglected driver up to scratch) is going to be quicker than trying to make hardware that's compatible enough to work with a driver written for another vendor's controller.
  (Besides which, as that other driver is probably maintained by your competitor, do you really think they're going to make an effort to ensure that their later updates are compatible with your clone controller? You'll still have to maintain your own fork.)
  I have often wondered why there isn't a vendor-neutral register-level standard for Ethernet controllers, along the lines of AHCI and xHCI. There is the virtio networking standard, but as it's designed for VMs I assume it does not cover Ethernet link management. I seem to remember that VMware tried to promote a common interface for SR-IOV virtual functions at one time, but that didn't get very far. Again that would not have included link management.
Why not just use FreeBSD then? by johanwanderer · 2014-08-06 05:38 · Score: 1, Insightful

It might be a silly question, but why don't they just use FreeBSD in that case?
1. Re:Why not just use FreeBSD then? by halivar · 2014-08-06 05:47 · Score: 2
  
  If they've done a lot of work already on a custom kernel, it may not make sense to try porting all that work to completely different kernel architecture. They may have done a cost benefit analysis and decided that the cost of improving their current architecture is less than retrofitting FreeBSD. Not saying this is the case, just posing a possible scenario where this would be the better option.
2. Re:Why not just use FreeBSD then? by discord5 · 2014-08-06 05:47 · Score: 4, Funny
  
  It might be a silly question, but why don't they just use FreeBSD in that case?
  You haven't heard? I'm sorry... My condolences, but the writing had been on the wall for a while though. Netcraft even confirmed it years ago...
  *BSD is dying.
3. Re:Why not just use FreeBSD then? by HuguesT · 2014-08-06 06:02 · Score: 2
  
  May be multiple issues. Perhaps better OpenMP support ? maybe NUMA ? Maybe Linux has a better virtual machine infrastructure ? maybe hardware support.
4. Re:Why not just use FreeBSD then? by gstoddart · 2014-08-06 06:02 · Score: 1
  
  Isn't Netcraft dead?
  Because people have been saying the BSDs are dying for night on 20 years now, possibly longer.
  
  --
  Lost at C:>. Found at C.
5. Re:Why not just use FreeBSD then? by devphaeton · 2014-08-06 06:02 · Score: 1
  
  Ye gods I'm so tired of this 'joke' cropping up every so often. It wasn't really that funny 12 years ago, either.
  
  --
  
  do() || do_not(); // try();
6. Re:Why not just use FreeBSD then? by dr.Flake · 2014-08-06 06:42 · Score: 2
  
  Ye gods I'm so tired of this 'joke' cropping up every so often. It wasn't really that funny 12 years ago, either.
  Actually,
  i still find the dying remark funny.
  And the more time has passed, the funnier it gets. As still being around and still being a relevant OS proves how silly the study was at the time.
  So i hope we can still laugh about it in 30 years or so.
  
  --
  Why are other peoples sig's always more witty ???
7. Re:Why not just use FreeBSD then? by gstoddart · 2014-08-06 07:02 · Score: 1
  
  So i hope we can still laugh about it in 30 years or so.
  You know, anybody who knows that joke now ... in 30 years or so will be sitting in the old geeks home laughing into their beard for reasons nobody else understands.
  And, yes, I consider myself among them ... now, if you'll excuse me, I should get started on my beard.
  
  --
  Lost at C:>. Found at C.
8. Re:Why not just use FreeBSD then? by K.+S.+Kyosuke · 2014-08-06 07:37 · Score: 1
  
  BSD doesn't support jokefs.
  
  --
  Ezekiel 23:20
9. Re:Why not just use FreeBSD then? by hawk · 2014-08-06 11:58 · Score: 1
  
  Gee, Apple has been dying *far* longer than that . . . :)
  hawk
LOL, so why not use theirs? by gstoddart · 2014-08-06 05:39 · Score: 5, Insightful

Look, this is FreeBSD ... why not just take their damned code?
It's not like you're not allowed to do that. That's what is great about the BSD license.
If FreeBSD's network stack is what you aspire to, why reinvent the wheel?

--
Lost at C:>. Found at C.
1. Re:LOL, so why not use theirs? by Bengie · 2014-08-06 05:44 · Score: 4, Insightful
  
  You can't just copy/paste code and expect it to work, it must be refactored, and may not even be compatible until completely new features are added in the current code to allow the new code to function. It's like saying "that fusion reactor is an open design, why not just place that in our coal power plant?". You may need to make some changes to your current power plant before you change its core.
2. Re:LOL, so why not use theirs? by mccoma · 2014-08-06 05:53 · Score: 1
  
  IANALBIHHTSF It can be included in a GPL project, but there is no provision in the BSD license to change the BSD licensed code to GPL. I would bet there is already some BSD code in the Linux kernel, so it shouldn't be a big deal.
3. Re:LOL, so why not use theirs? by gstoddart · 2014-08-06 05:59 · Score: 3, Insightful
  
  Licensing? Can code released under the BSD license be re-released under the GPL?
  Been years since I read the license, but the BSD licenses are pretty permissive, to the extent you can take BSD stuff and use it as a basis for commercial products.
  BSD has always been about writing awesome code, and letting people do what they want with it, as opposed to imposing ideology on people.
  I'd be surprised if Linux doesn't already have code from FreeBSD in it.
  
  --
  Lost at C:>. Found at C.
4. Re:LOL, so why not use theirs? by devphaeton · 2014-08-06 06:03 · Score: 1
  
  It does. Lots.
  
  --
  
  do() || do_not(); // try();
5. Re:LOL, so why not use theirs? by maliqua · 2014-08-06 06:09 · Score: 1
  
  there is a lot of bsd code in the linux kernel bsd is probably the least restrictive license so basically bsd code can be used by anyone for any reason
6. Re:LOL, so why not use theirs? by Hadlock · 2014-08-06 06:32 · Score: 1
  
  Like Apple... meet OSX
  
  --
  moox. for a new generation.
7. Re:LOL, so why not use theirs? by Anonymous Coward · 2014-08-06 06:33 · Score: 2, Informative
  
  Licensing? Can code released under the BSD license be re-released under the GPL?
  IANAL.
  Sort of. Technically, as you are (presumably) not the copyright holder of the work, you would have no permission to grant any sort of copyright license to third parties.
  But the license used by FreeBSD is very permissive. Anyone is permitted to copy, modify, redistribute, or redistribute modified versions of the software and essentially the only requirement is that you also include the three parts of the license text, verbatim, somewhere in all copies.
  So what this allows you to do is to create a modified version of the software. If your modifications are non-trivial, they may be a copyrightable work themselves. You may license your modifications to others under the terms of the GPL. The result is a combined work, with portions of it under the GPL license, and portions of it under the terms of the FreeBSD license. This works because someone can distribute the combination and simultaneously satisfy the conditions of both licenses: in addition to the GPL requirements, one must also include the text as required by the FreeBSD license.
8. Re:LOL, so why not use theirs? by mccoma · 2014-08-06 06:37 · Score: 1
  
  That's about what I said, except you just cannot change the license on the code itself. Some folks think you can strip the copyright and license from BSD code and that just isn't so.
9. Re:LOL, so why not use theirs? by TheRaven64 · 2014-08-06 06:50 · Score: 1
  
  Quite a few consumer router manufacturers actually ship Linux with the FreeBSD network stack grafted onto it. I don't know why.
  
  --
  I am TheRaven on Soylent News
10. Re:LOL, so why not use theirs? by Anonymous Coward · 2014-08-06 06:50 · Score: 1
  
  Still produces forward thrust, I'd say it's a drop-in replacement - nothing to it :P
11. Re:LOL, so why not use theirs? by petermgreen · 2014-08-06 07:02 · Score: 1
  
  Coal to gas conversion is pretty easy (though the result won't be as efficient as a combined cycle plant desgined to run on GAS). You just have to add some gas jets and gas piping to the combustion chamber. The design of the boiler and it's operating pressure can stay much the same and any extra physical space requirements are minimal.
  That doesn't mean coal to other things will be anywhere near as easy even if the fundamental "boiling water drives steam tubine" bit remains. You have to think of the physical location requirements of the new steam source, the steam pressures it can provide without overheating (AIUI nuclear power plants tend to run at lower temperature/pressure than coal fired ones).
  In the case of fusion it is expected that to achive breakeven fusion plants will have to be BIG.
  
  --
  note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
12. Re:LOL, so why not use theirs? by mi · 2014-08-06 08:07 · Score: 3
  
  Check your /usr/include/X11/extensions/fontcachstr.h for example. Or, to put us back on topic of FreeBSD networking, the <netinet/igmp.h>, <netinet/if_ether.h>, and <net/ethernet.h>...
  
  --
  In Soviet Washington the swamp drains you.
13. Re:LOL, so why not use theirs? by wonkey_monkey · 2014-08-06 09:20 · Score: 1
  
  You can't just copy/paste code and expect it to work
  Yes you can. You'll be disappointed, but you can.
  
  --
  systemd is Roko's Basilisk.
14. Re:LOL, so why not use theirs? by countach · 2014-08-06 17:16 · Score: 1
  
  It's the almost bit that will get you unstuck. No doubt the BSD network code could be dropped into Linux, but making it all work still wouldn't be trivial.
15. Re:LOL, so why not use theirs? by TheRaven64 · 2014-08-06 20:56 · Score: 1
  
  It's a common misconception that any code that you link with Linux has to be GPL'd. The GPL only requires that linked code has no restrictions not present in the GPL and that the combined work is covered by the GPL. Individual components can be in one of several more permissive licenses. The Linux kernel already contains fairly large amounts of BSDL and similar code.
  
  --
  I am TheRaven on Soylent News
16. Re:LOL, so why not use theirs? by david_thornley · 2014-08-07 05:40 · Score: 1
  
  You don't have to modify the code. You can take a BSD-licensed piece of code and just slap it into a GPLed program. It is then subject to the requirements of both licenses, and (as long as we're not dealing with the BSD advertising clause) since they're compatible there's no problem. Of course, the original is still available under the BSD license, and you do have to leave that license in even if you change it.
  Non-trivial modifications to the source (and "trivial" really does mean trivial here - if they require any creativity, they're not trivial) are copyrighted by whoever does them, and require a license in order to be useful. In a GPLed program, the modifications can be used under the GPL, and if the modifications would be useful to the original it's nice to BSD them as well.
  
  --
  "When you have eliminated the unacceptable, whatever is left, however improbable, must be the truthiness" - Holmes
Biggest troll on Slashdot ever by CAPSLOCK2000 · 2014-08-06 05:41 · Score: 2

Insulting their work might not be the right way to get the best Linux kernel network engineers to join your company.
1. Re:Biggest troll on Slashdot ever by Anonymous Coward · 2014-08-06 05:49 · Score: 3, Funny
  
  Insulting their work might not be the right way to get the best Linux kernel network engineers to join your company.
  Or it might be the best way ever. Linux people and their egos....
2. Re:Biggest troll on Slashdot ever by Zapotek · 2014-08-06 05:56 · Score: 1
  
  How is that an insult? As a coder (software engineer/developer/whatever) myself I'm always glad when people point out things in my systems that can be improved. The possibility of someone hiring someone else to improve them for me, that'd make me ecstatic. This isn't about egos, this is about getting stuff done right.
3. Re:Biggest troll on Slashdot ever by rogoshen1 · 2014-08-06 06:48 · Score: 1, Insightful
  
  Yes, but Facebook. Somewhere between working for the SS and Big Tobacco.
4. Re:Biggest troll on Slashdot ever by fuzzyfuzzyfungus · 2014-08-06 09:09 · Score: 1
  
  Yes, but Facebook. Somewhere between working for the SS and Big Tobacco.
  So snappy uniforms and very, very, deep pockets?
5. Re:Biggest troll on Slashdot ever by rogoshen1 · 2014-08-06 09:17 · Score: 1
  
  Yar, but you don't see the immediate effects of guilt and shame rotting their souls away; slowly turning them into twisted husks of matted neckbeard.
  I'd imagine being at Facebook would increase the rapidity of that process a thousand fold.
6. Re:Biggest troll on Slashdot ever by donscarletti · 2014-08-06 15:25 · Score: 1
  
  As a programmer, project manager and substantial shareholder in my own company, I'm always grateful when people point out things I do that could be done better.
  If I was just the former and not the latter two, I think the relative importance of my ego and the quality of the product would be shifted somewhat.
  Most, Open Source programmers tend to value recognition pretty highly relative to absolute merit of the product. To your average guy, if you say "your project sucks, we'll pay you to fix it" means that you will get the credit when it is fixed. If you say "your project is good but it could be better" means you will give them the credit either way.
  
  --
  When Argumentum ad Hominem falls short, try Argumentum ad Matrem
FreeBSD network stack by CadentOrange · 2014-08-06 05:48 · Score: 4, Insightful

What makes the FreeBSD network stack superior?
1. Re:FreeBSD network stack by gstoddart · 2014-08-06 05:53 · Score: 4, Funny
  
  Way more cowbell, and a much cooler logo.
  
  --
  Lost at C:>. Found at C.
2. Re:FreeBSD network stack by Zarjazz · 2014-08-06 06:05 · Score: 5, Informative
  
  As someone who has used various BSD's and Linux in large scale environments, and is a fan of both, I've configured servers with multi-10Gb interfaces and handling 100k+ requests a second I honestly can't think of any example of where Linux has been inferior. The often repeated line that FreeBSD has a better networking stack was probably true over 10 years ago with Linux 2.2 and earlier, but since then I'd say that myth is just bullcrap.
  Maybe Facebook are talking about some specific IPv6 or cutting edge features like MPTCP they need on their network, but as a general statement it's utterly misleading.
3. Re:FreeBSD network stack by Bengie · 2014-08-06 06:10 · Score: 4, Informative
  
  A lot of sysadmins from companies that push a lot of data over lots of connections have blogs about tweaking your OS to handle stuff like 10gb+ of traffic and millions of connections. A lot of these people complain about Linux having strange problems under these loads, and FreeBSD just seems to work. Linux may be faster in some cases, but it still has stability issues that are hard to debug.
  
  Then there's the whole thing about most network stack research happening primarily on FreeBSD because of licensing. There's a new zero-copy network API that was developed in FreeBSD that allows line rate 64byte 10gb traffic on a 450mhz quadcore cpu. Linux and old-api FreeBSD were about 1/10th the packets-per-second.
  
  A new thread friendly socket API has just been pushed to FreeBSD 11. One of Netflix' engineers had a pet project that now allows near zero lock-contention thread scaling. He was able to done line speed 40gb/s with 150k TCP sessions. Instead of having one file descriptor with a single listening thread, you instead have one file-descriptor and listening thread per MSS queue from the NIC and you can lock your thread to the same CPU as the MSS queue, so the packet is already in L2 cache. No shared network state. This also means no share locks with nearly perfect linear scaling and virtually no cache trashing or bouncing.
  
  They're starting work for extend the API to also allow the OS to better handle NUMA and to attach the MSS queues to the CPU to which the NIC is attached. This will virtually remove all cross-talk among the CPU cores trying to handle the network state.
  
  They're looking into expanding this same concept to the Storage IO system.
4. Re:FreeBSD network stack by dkman · 2014-08-06 06:12 · Score: 1
  
  If I knew what made it superior I'd be sending a resume, not writing this post.
  
  --
  I refuse to sign
5. Re:FreeBSD network stack by mvdwege · 2014-08-06 06:26 · Score: 1, Troll
  
  [citation needed]
  I really want to believe you are sincere and everything you say is true. Unfortunately, you don't seem to have provided any references.
  This is the Web. Hyperlinks exist for a reason. Use them.
  
  --
  "I know I will be modded down for this": where's the option '-1, Asking for it'?
6. Re:FreeBSD network stack by phoenix_rizzen · 2014-08-06 06:37 · Score: 3, Informative
  
  Google searches for "netmap" and "FreeBSD" will give you lots of information on pushing millions of pps through 900 MHz single-core machines. Netmap is also available on Linux. There's even a netmap-enabled version of IPFW that allows you to do packet filtering and routing completely in userspace, again will millions of pps. IPFW is also available on Linux, although I don't know if the netmap-enabled version is.
  Google searches for "openconnect" and "FreeBSD" will give you lots of information and blog posts from the Netflix guys about why they picked FreeBSD, and how it all works, including details on the networking.
  Google searches for "Adrian Chadd", or "RSS scaling", or similar terms will show you threads and posts on various FreeBSD mailing lists with information detailing a lot of the MSS/RSS work that's going into FreeBSD 11, and several projects that build off that. Those also have links to other information around sockets and similar.
  Google searches for "NUMA" and "FreeBSD" will bring up mailing list threads that cover the different projects being undertaken to improve the CPU affinity and thread locality and all that jazz.
  Sure, it would be nice if the OP had posted links to the info, but it's not like the information is secret or hard to find.
7. Re:FreeBSD network stack by Anonymous Coward · 2014-08-06 06:41 · Score: 1
  
  I'm really getting tired of people like you: This is not a scientific publication, and this is not for sure wikipedia, this is a forum where people expose opinions. Want citations? Go back to the fucking Wikipedia and please DO verify some of the already present sources on many articles there (I grew tired of pointing out the ones that didn't even mention what they're supposed to support). Otherwise use a search engine, the beauty of the 2000's is that you no longer need to spend hours searching for information on crappy information bases and "search" engines. Asking people to do your research for you just tells me you're a lazy bastard.
8. Re:FreeBSD network stack by Bengie · 2014-08-06 06:54 · Score: 1
  
  They're random blogs that I tend to come across when googling how to tweak network stacks for handling lots of connections. Take it as you will. It was an entirely anecdotal and opinionated blob of text that I probably just should have left out.
9. Re:FreeBSD network stack by Bengie · 2014-08-06 07:02 · Score: 1
  
  Re-reading when I posted, this was meant for just the first sentence talking about the blogs. The rest of the information is "true", but based off of memory. The part about the threaded api code for FreeBSD 11 was from a youtube from BSDCON(or similar convention)(https://www.youtube.com/watch?v=7CvIztTz-RQ) that was just published a week or two ago. The zero copy part is about "netmap" I think. Another /. post in this thread also mentioned it.
10. Re:FreeBSD network stack by ray-auch · 2014-08-06 07:49 · Score: 2
  
  Red demon costumes vs. penguin costumes. No contest really.
  http://freebsd-image-gallery.n...
  http://freebsd-image-gallery.n...
  http://www.flickr.com/photos/1...
  http://www.flickr.com/photos/1...
11. Re:FreeBSD network stack by mi · 2014-08-06 08:18 · Score: 1, Redundant
  
  Google searches for ...
  Sorry, but no. The onus is on you — one making the claim — to offer links supporting it.
  You Google it, you pick the links you deem most suitable, you embed them in your posting.
  Making the reader do it is not only impolite, it also makes it easier to attack your argument (that FreeBSD is superior) — the Google search could very well offer a link to some blog saying "FreeBSD networking sucks"...
  
  --
  In Soviet Washington the swamp drains you.
12. Re:FreeBSD network stack by Lennie · 2014-08-06 09:18 · Score: 3, Informative
  
  The MPTCP stack for Linux isn't in mainline, but is much further ahead then the FreeBSD version (not sure if it that in mainline).
  
  --
  New things are always on the horizon
13. Re:FreeBSD network stack by mvdwege · 2014-08-06 09:42 · Score: 1
  
  Ok, that's too bad. I was genuinely curious about the work being done.
  Sorry if I sounded a bit gruff, but when I see categorical statements, I like to have references to check myself.
  
  --
  "I know I will be modded down for this": where's the option '-1, Asking for it'?
14. Re:FreeBSD network stack by mvdwege · 2014-08-06 09:51 · Score: 1
  
  Ok, who did I hurt by contradicting his faith?
  
  --
  "I know I will be modded down for this": where's the option '-1, Asking for it'?
15. Re:FreeBSD network stack by shutdown+-p+now · 2014-08-06 12:31 · Score: 1
  
  What makes you believe OP was "making an argument"? He was responding to a comment that asked for more details, and he has provided them. If you want to argue with him, go ahead, but then you'll be the one making the claim that he's wrong (or whatever it is you want to argue about).
16. Re:FreeBSD network stack by Anonymous Coward · 2014-08-06 17:23 · Score: 2, Informative
  
  Hi! adri here. I'm Adrian (adrian@freebsd.org) the ex-Netflix engineer who was doing this as his project whilst I was working there.
  I'm continuing to do it in my spare time now, as time and hardware permit.
  https://wiki.freebsd.org/NetworkRSS
  I'm working out the kinks in how IP fragments are correctly handled. It'll be more useful for real world deployments after that - unfortunately in the real world you have to deal with fragmented TCP and UDP; you can't just pretend it's not there.
  I'll continue chipping away at it after the Defcon weekend. It's nice to see no lock contention between CPUs when doing TCP and UDP workloads. The main thing that's left is NUMA awareness.
  I'll likely do up a memcached style example once this is done so we can really stress it. I'm pretty sure I can get a few million transactions a second out of it on 8-core single-socket hardware - without RDMA, without crazy vendor-specific products.
  The thing to keep in mind is that I'm doing this in my spare time now and it's not being funded. I've had 10G and soon 40G NICs donated to me and my employer will likely loan me some 8, 16 and 32 core machines to expand upon this.
17. Re:FreeBSD network stack by CadentOrange · 2014-08-06 19:52 · Score: 1
  
  That's a useful list, especially if you have no clue as to what to actually search for. Like me.
  Cheers.
18. Re:FreeBSD network stack by TheRaven64 · 2014-08-06 21:01 · Score: 1
  
  Netmap is also available on Linux
  While this is technically true, we recently had a PhD student try this. First, the Linux version is only available as patches so it took him a while to find a version of Linux that they'd apply against. Once he'd done this, it turns out that the driver support is basically only there in Linux for Intel NICs, which are modified as part of the patch. In FreeBSD, because it's merged into trunk, most NIC drivers support it.
  Oh, and Adrian isn't at Netflix anymore, but he's still working on networking stuff (now at one of the CDN companies, but I forget which one).
  
  --
  I am TheRaven on Soylent News
19. Re:FreeBSD network stack by Anonymous Coward · 2014-08-06 23:47 · Score: 1
  
  > FBSD essentially always wins who has the fastest network stack contests, sets internet speed records etc.
  Which ones? Back when the "Internet 2 land speed records" ran, Linux was almost always being used for record attempts toward the end, although NetBSD was occasionally used too.
  I'm not aware of other "fastest network stack constests" or internet speed records, but I'll be really interested in them if you would please point me to them.
20. Re:FreeBSD network stack by saleenS281 · 2014-08-07 10:34 · Score: 1
  
  Out of curiosity, are they giving the code back, or is it all tied up in their own distro?
21. Re:FreeBSD network stack by adri · 2014-08-08 08:01 · Score: 1
  
  Hi,
  That'd be me (adrian@freebsd.org)
  https://wiki.freebsd.org/Netwo...
22. Re:FreeBSD network stack by adri · 2014-08-08 08:06 · Score: 1
  
  Me - adrian@freebsd.org. Also, https://wiki.freebsd.org/Netwo... .
23. Re:FreeBSD network stack by adri · 2014-08-08 08:40 · Score: 1
  
  I don't work at a CDN company. :-)
  I work at norse-corp.com. RSS is a non-paid thing I do at home.
24. Re:FreeBSD network stack by TheRaven64 · 2014-08-09 01:23 · Score: 1
  
  Oh, sorry. I was thinking of Sean. Both of you switched jobs at almost the same time.
  
  --
  I am TheRaven on Soylent News
Re:Won't Happen by fuzzyfuzzyfungus · 2014-08-06 06:05 · Score: 2

Why is the organizational separation between the kernel and the userland going to affect the quality of the TCP/IP stack?
Remember Microsoft Windows? by rs79 · 2014-08-06 06:20 · Score: 1

It used the FreeBSD networking code. This doesn't mean windows is fast and it's sort of specious. BSD has tricks in the Kernel to make I/O faster that pretty much anything else.

--
Need Mercedes parts ?
1. Re:Remember Microsoft Windows? by Dahan · 2014-08-06 07:31 · Score: 2
  
  It used the FreeBSD networking code. This doesn't mean windows is fast and it's sort of specious. BSD has tricks in the Kernel to make I/O faster that pretty much anything else.
  No it didn't. A few utilities that nobody used (e.g., the commandline ftp.exe, which doesn't even support PASV mode) were ported from BSD (not even FreeBSD), but the TCP/IP stack in Windows was not from BSD.
2. Re:Remember Microsoft Windows? by phoenix_rizzen · 2014-08-06 08:43 · Score: 2
  
  Windows NT used a STREAMS-based networking stack, culled from some other UNIX (not directly, but using the concepts and frameworks), not a BSD-derived networking stack.
  I have no idea how the DOS-based Windows networking stack developed. But it wasn't pulled from any BSD.
  A few command-line utilities (ftp.exe is the most common cited one) were pulled from BSD sources, though.
3. Re:Remember Microsoft Windows? by Anonymous Coward · 2014-08-06 09:10 · Score: 1
  
  Not true at all. Microsoft stole the TCP/IP stack from either FreeBSD or OpenBSD when they couldn't write their own. This is confirmed by TCP/IP packet fingerprinting. The update occurred sometime between Windows NT4 and Windows 2000.
  "In their 'completely rewritten' TCP stack for NT5, Microsoft uses 0x402E. Interestingly, that is exactly the number used by OpenBSD and FreeBSD."
  http://nmap.org/nmap-fingerprinting-old.html
Corporate Contributions to OSS by phizi0n · 2014-08-06 06:33 · Score: 5, Insightful

I don't understand why there's all these comments saying they should just use FreeBSD. There are many reasons to despise Facebook but their desire to improve the Linux networking stack is admirable. We should be encouraging corporations to contribute to OSS, not telling them to just use that other thing that is better in some ways but not others. Kudos to them for contributing back to the projects they use.
Huh? by Millennium · 2014-08-06 06:34 · Score: 1

What effect does the userland have on the TCP/IP stack?
High perf SMP coding is in a category of its own by m.dillon · 2014-08-06 06:38 · Score: 5, Informative

Designing algorithms that play well in a SMP environment under heavy loads is not easy. It isn't just a matter of locking within the protocol stack... contention between cpus can get completely out of control even from small 6-instruction locking windows. And it isn't just the TCP stack which needs be contention-free. The *entire* packet path from the hardware all the way through to the system calls made by userland have to be contention-free. Plus the scheduler has to be able to optimize the data flow to reduce unnecessary cache mastership changes.
It's fun, but so many kernel subsystems are involved that it takes a very long time to get it right. And there are only a handful of kernel programmers in the entire world capable of doing it.
-Matt
Re:Will Happen by phoenix_rizzen · 2014-08-06 06:43 · Score: 2

Except when you start talking about netmap. :) That's a userspace network stack that can push millions of pps, on sub-GHz systems.
There's even a netmap-enabled version of the IPFW packet filter that runs in userspace, filtering millions of pps on sub-GHz systems.
And there's an applications ecosystem starting to grow around netmap that keeps all network-related packet processing in userspace.
As a twist, netmap and IPFW are also available on Linux, and provide better performance than the in-kernel network stack and iptables. :)
Re:Won't Happen by TheRaven64 · 2014-08-06 06:48 · Score: 4, Interesting

Actually, that's not always true. FreeBSD ships with netmap, which allows you to talk to the network hardware directly from userspace. A significant chunk of the DNS root zone is served by FreeBSD boxes using a completely custom TCP/IP stack on top of netmap. There's a paper at this year's SIGCOMM about building specialised network stacks in this infrastructure.
If you're talking about the FreeBSD TCP/IP stack, then libuinet allows running it entirely in userspace.
These might not be the ones that Facebook is interested in, but a significant amount of their workload could be sped up by using the work described in the SIGCOMM paper...

--
I am TheRaven on Soylent News
Why not rebuild your ports? by mi · 2014-08-06 07:52 · Score: 1

People, you do not have to rebuild your ports just because you updated.
And why would it be such a bad thing, even if you had to? I maintain a handful of ports myself and people's complaining about rebuilding them always irks me — what am I missing?
There are, of course, precompiled binary packages for almost all the ports (where licensing allows redistribution of such binaries). But I don't use them myself — building everything from source. Why not?

--
In Soviet Washington the swamp drains you.
1. Re:Why not rebuild your ports? by Anonymous Coward · 2014-08-06 08:31 · Score: 5, Informative
  
  If you are managing a bunch (more than 2) FreeBSD machines building ports on the live system is wrong and counterproductive. In this kind of scenario you really should setup a package building machine and distribute your own binary packages to them. With pkgng this really works well.
  This applies if you don't want to use the official binary packages (for example because you need different options from the defaults, or you tweak some ports)
Re:Why do you think they will contribute back? by phizi0n · 2014-08-06 08:36 · Score: 2

The job posting specifically says they prefer to have the code in mainline.
Hardware support by phorm · 2014-08-06 09:46 · Score: 1

Hardware support, in particular, is very far behind
Amusingly, I had the opposite problem years back. The wireless drivers and stack were better on BSD, while on Linux they were hard to find, and often you ended up having to use ndiswrapper which was a nightmare (often resulting a decision between "do I upgrade my kernel and fix X, or keep my kernel and have working wifi").
Just taking a quick peek, it looks like USB3.0 works, but it depends on your host-controller whether it's supported or not.
It's not just NetMap that brings performance ... by Anonymous Coward · 2014-08-06 11:33 · Score: 3, Informative

FreeBSD also includes an alternative to select/poll called kqueue that allows it to scale client connections massively with minimal performance degradation. Linux introduced epoll as a work-alike, but it has some drawbacks ...
http://www.eecs.berkeley.edu/~sangjin/2012/12/21/epoll-vs-kqueue.html
What's a massive scale? WhatsApp, recently acquired by Facebook, uses FreeBSD and Erlang to power it's service offerings. They sustain over 2 million simultaneous client connections per FreeBSD server ...
http://blog.whatsapp.com/196/1-million-is-so-2011
I wouldn't be surprised if the internal comparison between Linux and FreeBSD network features/performance was fueled by feedback from their new subsidiary.
FreeBSD also works very closely with the Nginx community. If you look at the dev mailing list, you will see a fair amount of kernel level dev work sponsored by companies that use nginx on top of FreeBSD. This constant tuning keeps nginx consumers loyal to FreeBSD for obvious reasons. There is no wonder why this combination was selected by NetFlix to power their new content delivery network.
Re:Won't Happen by Cramer · 2014-08-06 15:28 · Score: 1

There are a few "hacks" that do that in linux as well. Basically the network driver is in userland, with a kernel shim to handle DMA and IRQ which isn't available to userland. (in fact, I use that same mode to deal with broadcom SoC's -- not for network traffic, just to configure and monitor) There are advantages to pulling packets direct to userland.
Re:Hire Me! I'll do it. by TheRaven64 · 2014-08-06 21:03 · Score: 1

How much of Facebook's code is not platform-independent already? Most of their web infrastructure is PHP with a custom compiler. The compiler may need some tweaks, but the PHP code that it compiles should be completely independent. Drivers are a different issue, but if they're employing a load of kernel devs it's a lot easier to write a few missing drivers than make large and invasive changes to the network stack.

--
I am TheRaven on Soylent News
Re:Switch to Solaris... seriously... by TheRaven64 · 2014-08-06 21:06 · Score: 1

Out of interest, how many cores are you using? I've not used Solaris for... a long time, but their network stack was redesigned a little while ago to do each layer in a different thread with lockless ring buffers between them all. I'd imagine latency might suffer a bit, but throughput should be insane with enough hardware contexts. On something like the UltraSPARC Tx series, I'd expect it to outperform FreeBSD, but I've never seen numbers.

--
I am TheRaven on Soylent News
I hate FB... by vandamme · 2014-08-07 06:06 · Score: 1

...slightly less, now.
1. Re:I hate FB... by iggymanz · 2014-08-12 03:07 · Score: 1
  
  wrong response, you should be hating Linux slightly more. Great for desktops, but for servers for the common stacks BSD is usually better
Won't Happen by saleenS281 · 2014-08-07 08:47 · Score: 1

Which works great if you're writing all your own software, or are able to rely on nothing but open source. VERY few companies have that luxury, and FreeBSD/OpenBSD are a non-starter for anything you want vendor support on. RHEL and SLES on the other hand are almost universally supported by hardware and software vendors alike.
Re:Switch to Solaris... seriously... by d3vi1 · 2014-08-11 20:12 · Score: 1

I remember seeing at some point numbers. It didn't impress in a single thread, but could easily saturate a 10Gb link in multi-threaded tests. They tested an FTP server on a T2plus. Regarding cores, we have anything from dual UltraSPARC IIIi to T4 based systems including some M-class. I believe the T3-4 has the highest number of cores. It should be 64 cores and 512 threads, but a single Solaris instance can only see 256. I believe that the M9000 and M9000-64 should have the same problem, but the biggest M series I've worked with is M8000.

--
UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever ones.
Linking to evidence (Re:FreeBSD network stack) by mi · 2014-08-12 05:37 · Score: 1

My objection was not to merits or lack thereof of a particular OS, but to the practice of placing the burden of research on the audience (and opponents).
Whatever it is you are stating, should be backed by evidence. It is best to include the links with the statement being supported, but it can be tedious. So, links should be provided upon request — without any lip like "just google it yourself"...

--
In Soviet Washington the swamp drains you.