Facebook Seeks Devs To Make Linux Network Stack As Good As FreeBSD's

An anonymous reader writes Facebook posted a career application which, in their own words is 'seeking a Linux Kernel Software Engineer to join our Kernel team, with a primary focus on the networking subsystem. Our goal over the next few years is for the Linux kernel network stack to rival or exceed that of FreeBSD.' Two interesting bullet points listing "responsibilities": Improve IPv6 support in the kernel, and eliminate perf and stability issues. FB is one of the worlds largest IPv6 deployments; Investigate and participate in emerging protocols (MPTCP, QUIC, etc) discussions,implementation, experimentation, tooling, etc.

Comments

This does pose the question:

[Anonymous Coward]

Why not use FreeBSD? It's already there and at least as good as linux. Or have they perhaps hung themselves on systemd?

Re:This does pose the question:

[HuguesT]

I love FreeBSD, I support them financially every year, and I use it daily but it is not uniformly better than Linux. Hardware support, in particular, is very far behind. Two random examples:

1- My NAS system does not recognise any USB storage when they are plugged in after boot (no hotplug). It does not support USB superspeed (USB 3.0) either (I have to boot in compatibility mode by disabling xHCI in the BIOS). This is a known issue with some Asus motherboards, still unfixed in 10.0
2- FreeBSD does not install on some of my HP G6 servers. The kernel simply segfaults. I really wanted FreeBSD on this hardware, so I run it in a VM under Linux (using KVM). Has been running brilliantly for about 2 years now.

Also security update in FreeBSD are really difficult. I haven't finished dealing with updating my ports since I moved from 9.2 to 9.3 last week.

I have to say this though: when it runs, it runs really well.

Re:This does pose the question:

[ci4]

USB3 support in FreeBSD 10 is OK (bunch of external disks used for PC backup - speed was essential). No problem with hot-plug either. Ports upgrade is trivial (although I have switched to pkg-ng now). I really can't find why do you think that security updates are difficult either. I've got only one 9.2 system around which I at the moment am not bothered to upgrade.

Re:This does pose the question:

[jdew]

What asus board? I've got an asus crosshair iv formula and the usb3 has been working fine in both 9.2 and 9.3. The aibs driver doesn't support this board yet though.

as for security updates:
freebsd-update fetch; freebsd-update install

and port updates:
portsnap fetch update; portmaster -gda

So not sure what the issues are here.

Re:This does pose the question:

[TheRaven64]

I'm pretty sure that Facebook buys enough hardware that they can afford to write drivers for anything they're missing and demand FreeBSD support from vendors for their next round of purchases. Netflix already does this (they won't buy any hardware that doesn't have vendor support for FreeBSD), as do a few other companies, and so a number of NIC vendors (particularly in the 10G/40G space) are now putting quite a bit of effort into their FreeBSD drivers.

Re:This does pose the question:

[Fweeky]

pkgng's made port upgrading much less burdensome - even fairly complex dependency changes can be handled automatically as of 1.3, and the official package repositories are a lot more useful now. They even have stable security-fix-only branches.

I still make my own customised builds, but I make binary packages in an isolated jail using poudriere. 99% of upgrades are a matter of updating its ports tree, running rebuild-packages, and running pkg upgrade on all my machines.

You couldn't pay me to go back to portupgrade/portmaster/portmanager.

Re:This does pose the question:

[lgw]

If you have 1 million Linux machines deployed, with full Linux-specific software stacks on each, it's cheaper and easier (and most likely faster) to fix the problems you see in Linux than to move the fleet to a new OS.

Facebook's dev shop culture is all about banging out code as fast as possible for the problem in front of them, then moving on. Forward planning isn't really the thing there, from what I hear (though I think you're no longer discouraged from testing your code before it goes live, that's a recent change). Moving to BSD might well be a better long term plan, but it would take years to get there and they don't really think on that timescale, from the rumors I hear anyhow.

Re:This does pose the question:

[Anonymous Coward]

I see this a lot on IRC... People, you do not have to rebuild your ports just because you updated. With the exception of something like, a third party driver that links against the kernel, or maybe one or two random small things, everything else still runs.. Just make sure you installed the compat package for the previous version.

Re:This does pose the question:

[Pieroxy]

Don't you think it's easier and cheaper to optimize the network stack of Linux rather than writing tons of hardware drivers for FreeBSD? Hardware which, most of the time, will be undocumented. Furthermore, when you change your servers, yay, more drivers to write...

Re:This does pose the question:

[Anonymous Coward]

The binary package manager is a pain in the ass in FreeBSD. Linux and FreeBSD are a royal pain in the ass to recompile from source and mix binary packages, as there's entire chains of dependency hell you have to go through if you need X but not Y.

FreeBSD's hardware support isn't quite as diverse as Linux, but that's mostly in the "desktop" department. I'd say the hardware support is superior on FreeBSD for server platforms, as there's less "cooks in the kitchen"

Where Linux is terribly weak though is on networking. There is no support for network card features designed to improve performance or reduce buffer bloats. Linux insists on doing absolutely everything on the CPU, and as much in user-space as possible, so from a security point of view it has some advantages in that design, but from a performance point of view, this is what makes it awful as a virtual machine OS, because there's multiple layers of virtual CPU's doing things in software that it should be doing in hardware.

Re:This does pose the question:

[Anonymous Coward]

> I think you're no longer discouraged from testing your code before it goes live, that's a recent change

You think you are making facebook look like idiots, but all you are doing is casting doubt on your own credibility.
Extraordinary claims like that demand extraordinary proof.

Re:This does pose the question:

[Score Whore]

Facebook buys custom servers, so will be 100% documented. Also they are of the vanity free variety lacking any bolted on bits added strictly to make the numbered list of features on the side of the box longer. I suspect that the only thing they are going to care about are disks and nics. Sounds cards, video cards, random USB hardware, bluetooth, none of that matters to them at all. These are datacenter housed pieces of equipment.

Re:This does pose the question:

[Anonymous Coward]

Poudriere on a fast machine as a repository for all your packages, couldn't be easier.

Re:This does pose the question:

[lgw]

Well, I only know this stuff second hand, but Facebook's dev motto used to be "go fast and break stuff". They had a very hard time hiring senior devs for some reason. Recently, they've had a bit of a (recruiting world) push to let people know that their new motto is "go fast with stable infra". I hear there used to be a banner on the wall proclaiming "don't test, just ship", but to be fair I haven't seen that myself.

Re: This does pose the question:

[Anonymous Coward]

99% sounds really made up...

Re: This does pose the question:

[Anonymous Coward]

Watch the movie. The basis for this is there. Yes it is a movie. Still based on real life. Google it. Zuch is renowned for this attitude.

Re:This does pose the question:

[ShanghaiBill]

You think you are making facebook look like idiots

No, he's making them look smart. 30 years in the software business has taught me that being first is a lot more important than being best. Successful companies whip up a quick fix and push it out the door, while the losers are busy doing another round of code review and testing. More rigorous engineering may be better in the long run, but there is no long run if you are bankrupt because you missed your opportunity window.

Re:This does pose the question:

[Anonymous Coward]

Would that not be highly dependent on the industry? In this case it is just facebook. No big harm if something doesn't work for a few hours or even days. So somebody doesn't get to play Farmville or update their status. Nobody dies, nobody gets investigated by the SEC/EPA, etc. Other application domains have different rules.

Re:This does pose the question:

[gangien]

I interviewed there, and there and had a dude with under a year of experience out of college interview me. and of course, give a horrible interview. I'm pretty sure that's the reason I didn't get the offer (of course who knows, maybe other reasons). they have this idea that an engineer is an engineer, which I think is good. to an extent, but in the end it's a bullshit philosophy.

Re:This does pose the question:

[TheGratefulNet]

I used to run a lot of freebsd at home - until I tried linux for samba and never went back to bsd again. bsd samba simply sucks. its dog slow and it hammers on disks, unlike linux which is very cache-oriented (from just looking at the drive lights).

that was my main complaint. maybe linux has better integration with smb and the network stack but bsd is just not nearly as fast.

stability wise, this isn't 10 yrs ago and linux stays up as long as you need it (years, even).

bsd has better mgmt, though; one distro and one style of doing things. THAT is much nicer than the linux fragmentation.

Re:This does pose the question:

[Anonymous Coward]

In some dev circles, you are absolutely correct. But when you start working with financial or medical related code (including data) you have to make sure all your i's are dotted and t's are crossed, or you lose customers fast. Missing a decimal point in one of your numeric displays on a personal blog page is nothing like doing this on an online banking app. And if you accidentally leave the door open on Aunt Jane's gardening tips, its nothing like leaving the page detailing Aunt Jane's bout last year with a raging case of Chlamydia open for the world to see.

Re:This does pose the question:

[CrankyFool]

Two comments:

1. Their internal motto was, in fact, "go fast and break stuff"; I know this first-hand because I talked with them about that at my interview back in February, where they mentioned that they've changed to "go fast and be bold" because, in fact, they were trying to lower incidents of availability hits;

2. 20+ years of tech industry experience here, and I was totally ready to be interviewed by some snot-nosed kid. What I got instead was an interview panel whose average tech industry tenure was around 17 years. I was, uniformly, impressed with the caliber of the people I met with there -- they verged from "pretty decent" in one case, to "pretty great" in all but two other cases, to "I'd take a $10K pay cut to work with this person" for the last two people. I was pretty surprised, and delighted.

Re:This does pose the question:

[TheRaven64]

The network stack is a complex bit of code that interacts in complex ways with the scheduler, the generic locking frameworks, the driver infrastructure, the bus and DMA, infrastructure and everything up to the system call layer. The device drivers are self-contained and (individually) small pieces of code. Which do you think is cheaper and easier to modify?

Re:This does pose the question:

[TheRaven64]

The binary package manager is a pain in the ass in FreeBSD/quote. Have you used FreeBSD since the package infrastructure was replaced? If so, we'd be interested in bug reports that are more detailed than 'it's a pain in the ass'.

Re:This does pose the question:

[Anonymous Coward]

Facebook was not the first.

Re:This does pose the question:

[Stuarticus]

If Facebook went down for a few days it would be the bigger news than the war in Gaza, my bank has more down time than facebook.

Re:This does pose the question:

[Pieroxy]

Well, they may be self contained (which is not true because NIC drivers interact directly with the network stack for example) and small, but they are also highly untestable and undocumented.

What I see is a stack to fix and a continuous stream of drivers to write. The one shot operation is bound to be cheaper on the long term.

Re:This does pose the question:

[TheRaven64]

NIC drivers interact directly with the network stack only via a set of well-defined interfaces (in FreeBSD at least), and writing the drivers is a one-shot operation because for new purchases they can demand FreeBSD drivers and support from their vendors (as Netflix, Verisign, and others do already). In the networking space, enough big companies are already demanding FreeBSD support for the high-end cards that the drivers are already likely to exist, so the missing ones are likely to be other things.

Re:This does pose the question:

[Ben Hutchings]

Right. They want FreeBSD drivers, they can put that on the requirements list for network and storage controller vendors. But that does leave the issue of where the vendors are going to find good FreeBSD hackers to write these drivers.

Re:This does pose the question:

[Mdk754]

Do you know why Linux seems to do so much better on VM NIC performance?

Honestly curious, not trying to troll. In my ESXi tests I can manage more than double the throughput on a Linux VM compared to a BSD VM. Is there just crappy paravirtual NIC support in BSD?

Tests were done with IPFIre and pfSense on two different ESXi boxes, and going to virtual client/server on etiher end of the router to avoid gigabit wire speed limits. On my slower box, i got 6xxMB/s with pfsense and 2.xxGB/s with IPFire. Minimal speed difference in either OS when using vmnet3 vs e1000. e1000 offered minimal throughput increase, vmxnet3 offered minimal host CPU usage decrease.

Re:This does pose the question:

[evilviper]

2- FreeBSD does not install on some of my HP G6 servers. The kernel simply segfaults.

Did you try disabling anything and everything (drivers) in the kernel? Or perhaps just blacklisting most/all the modules?

OpenBSD makes this easier with UKC and "config -e".

Re: This does pose the question:

[Anonymous Coward]

I get 90mb/s in samba 36, freebsd 10, under esxi vmxnet3

Re:This does pose the question:

[Anonymous Coward]

its kind of sad that they would have to rewrite the network stack IMO, is it really that bad?

Re:This does pose the question:

[Bengie]

That's a driver issue, not a network stack issue.

Re:This does pose the question:

[RockDoctor]

they can put that on the requirements list for network and storage controller vendors. But that does leave the issue of where the vendors are going to find good FreeBSD hackers to write these drivers.

Isn't there a cart and horse misalignment here. If you're a hardware vendor looking to get a big (and FB are big buyers of hardware) contract, and the tendering terms include "must have drivers for [Open Source system]", then won't you look at what you've already got on the shelf in terms of capabilities, and then burn custom firmware - or even an entire new product - that fits that driver at the hardware/ register level.

Or, if you know that your NewShiny line of product is likely to be punted to FB by Sales, then you damned well make sure that it works with $FB-favoured-OS$ while you're still designing the damned thing.

Re:This does pose the question:

[Ben Hutchings]

ASICs generally aren't flexible enough that you could simply emulate another controller in firmware, while FPGAs suck too much power to use on commodity network adapters. Writing a new driver (or bringing an existing neglected driver up to scratch) is going to be quicker than trying to make hardware that's compatible enough to work with a driver written for another vendor's controller.

(Besides which, as that other driver is probably maintained by your competitor, do you really think they're going to make an effort to ensure that their later updates are compatible with your clone controller? You'll still have to maintain your own fork.)

I have often wondered why there isn't a vendor-neutral register-level standard for Ethernet controllers, along the lines of AHCI and xHCI. There is the virtio networking standard, but as it's designed for VMs I assume it does not cover Ethernet link management. I seem to remember that VMware tried to promote a common interface for SR-IOV virtual functions at one time, but that didn't get very far. Again that would not have included link management.

Won't Happen

[Anonymous Coward]

I like Linux as much as the next guy, but FreeBSD has the advantage of being a complete OS, not simply a kernel with a userland. No matter how tight the code, how well implemented, FreeBSD will likely always have the better TCP/IP stack. As I'm fond of telling people, if I ran my own company, it would be Linux on the desktops and FreeBSD/OpenBSD on the servers.

Re:Won't Happen

[fuzzyfuzzyfungus]

Why is the organizational separation between the kernel and the userland going to affect the quality of the TCP/IP stack?

Re:Won't Happen

[Anonymous Coward]

The TCP/IP stack is inside the kernel. :)

Re:Won't Happen

[TheRaven64]

Actually, that's not always true. FreeBSD ships with netmap, which allows you to talk to the network hardware directly from userspace. A significant chunk of the DNS root zone is served by FreeBSD boxes using a completely custom TCP/IP stack on top of netmap. There's a paper at this year's SIGCOMM about building specialised network stacks in this infrastructure.

If you're talking about the FreeBSD TCP/IP stack, then libuinet allows running it entirely in userspace.

These might not be the ones that Facebook is interested in, but a significant amount of their workload could be sped up by using the work described in the SIGCOMM paper...

Re:Won't Happen

[MrHanky]

Wow, "2, interesting" for this childish nonsense. Slashdot really is failing hard these days.

Re:Won't Happen

[Cramer]

There are a few "hacks" that do that in linux as well. Basically the network driver is in userland, with a kernel shim to handle DMA and IRQ which isn't available to userland. (in fact, I use that same mode to deal with broadcom SoC's -- not for network traffic, just to configure and monitor) There are advantages to pulling packets direct to userland.

Re:Won't Happen

[Anonymous Coward]

Netmap is zero copy and is not a "hack". It still allows for proper OS and usermode separation. It's more akin to shared memory where the NIC writes directly to the memory location where the usermode code can access without a context switch. This also allows multiple applications to view the incoming packets without making copies. Snort can be monitoring the packets without the kernel making extra copies.

Netmap is so good that it is several times faster than Linux in-kernel packet processing.

If I remember correctly, it did have some trade offs. Like another process could potentially corrupt packets from another process, but not read them. This was being looked into. But this can be gotten around by spinning up a new jail, which has separate ring buffers. Ohh, sorry, Linux doesn't separate the network stack across "jails" like FreeBSD does.

It's still cool to think that you can have a usemode firewall running inside a jail, and it'll be about 4x faster than an in-kernel firewall in Linux. The performance difference grows as the network speed increases. With a 40gb NIC, Netmap will be much more than 4x faster. Linux is getting Netmap also, but I'm not sure how it'll work with the Linux version of "jails".

Why not just use FreeBSD then?

[johanwanderer]

It might be a silly question, but why don't they just use FreeBSD in that case?

Re:Why not just use FreeBSD then?

[halivar]

If they've done a lot of work already on a custom kernel, it may not make sense to try porting all that work to completely different kernel architecture. They may have done a cost benefit analysis and decided that the cost of improving their current architecture is less than retrofitting FreeBSD. Not saying this is the case, just posing a possible scenario where this would be the better option.

Re:Why not just use FreeBSD then?

[Anonymous Coward]

So our solution to them having one great platform that could be improved in one area is to throw out the entire platform and adopt another one which would need to be improved in several areas? Are you a PHB?

Re:Why not just use FreeBSD then?

[discord5]

It might be a silly question, but why don't they just use FreeBSD in that case?

You haven't heard? I'm sorry... My condolences, but the writing had been on the wall for a while though. Netcraft even confirmed it years ago...

*BSD is dying.

Re:Why not just use FreeBSD then?

[Anonymous Coward]

They were making this thing called a joke. It's a pretty new invention so I'm not surprised you've never heard of it before.

Re:Why not just use FreeBSD then?

[HuguesT]

May be multiple issues. Perhaps better OpenMP support ? maybe NUMA ? Maybe Linux has a better virtual machine infrastructure ? maybe hardware support.

Re:Why not just use FreeBSD then?

[gstoddart]

Isn't Netcraft dead?

Because people have been saying the BSDs are dying for night on 20 years now, possibly longer.

Re:Why not just use FreeBSD then?

[devphaeton]

Ye gods I'm so tired of this 'joke' cropping up every so often. It wasn't really that funny 12 years ago, either.

Re:Why not just use FreeBSD then?

[Anonymous Coward]

I would bet it's because hadoop/hbase and other bigdata stuff are better supported on linux

Re:Why not just use FreeBSD then?

[Anonymous Coward]

Yes! But please be careful with that beak.

Re:Why not just use FreeBSD then?

[dr.Flake]

Ye gods I'm so tired of this 'joke' cropping up every so often. It wasn't really that funny 12 years ago, either.

Actually,

i still find the dying remark funny.
And the more time has passed, the funnier it gets. As still being around and still being a relevant OS proves how silly the study was at the time.

So i hope we can still laugh about it in 30 years or so.

Re:Why not just use FreeBSD then?

[gstoddart]

So i hope we can still laugh about it in 30 years or so.

You know, anybody who knows that joke now ... in 30 years or so will be sitting in the old geeks home laughing into their beard for reasons nobody else understands.

And, yes, I consider myself among them ... now, if you'll excuse me, I should get started on my beard.

Re:Why not just use FreeBSD then?

[K. S. Kyosuke]

BSD doesn't support jokefs.

Re: Why not just use FreeBSD then?

[Anonymous Coward]

Missing drivers for their cache system, file systems, basically open rack hardware support is a linux project.

Re:Why not just use FreeBSD then?

[Anonymous Coward]

There was no study. Netcraft didn't claim that BSD was dead. Netcraft showed webserver statistics and the people who said "Netcraft confirms it" were alluding to the stats showing that a smaller and smaller percentage of webservers were BSD with Linux quickly climbing up the percentages.

Nobody of note ever said BSD was dead.

Re:Why not just use FreeBSD then?

[hawk]

Gee, Apple has been dying *far* longer than that . . . :)

hawk

Re:Why not just use FreeBSD then?

[Anonymous Coward]

In Soviet Russia, Netcraft confirms YOU!

da, tovarisch

LOL, so why not use theirs?

[gstoddart]

Look, this is FreeBSD ... why not just take their damned code?

It's not like you're not allowed to do that. That's what is great about the BSD license.

If FreeBSD's network stack is what you aspire to, why reinvent the wheel?

Re:LOL, so why not use theirs?

[Bengie]

You can't just copy/paste code and expect it to work, it must be refactored, and may not even be compatible until completely new features are added in the current code to allow the new code to function. It's like saying "that fusion reactor is an open design, why not just place that in our coal power plant?". You may need to make some changes to your current power plant before you change its core.

Re:LOL, so why not use theirs?

[Anonymous Coward]

Licensing? Can code released under the BSD license be re-released under the GPL?

Re:LOL, so why not use theirs?

[mccoma]

IANALBIHHTSF It can be included in a GPL project, but there is no provision in the BSD license to change the BSD licensed code to GPL. I would bet there is already some BSD code in the Linux kernel, so it shouldn't be a big deal.

Re:LOL, so why not use theirs?

[gstoddart]

Licensing? Can code released under the BSD license be re-released under the GPL?

Been years since I read the license, but the BSD licenses are pretty permissive, to the extent you can take BSD stuff and use it as a basis for commercial products.

BSD has always been about writing awesome code, and letting people do what they want with it, as opposed to imposing ideology on people.

I'd be surprised if Linux doesn't already have code from FreeBSD in it.

Re:LOL, so why not use theirs?

[devphaeton]

It does. Lots.

Re:LOL, so why not use theirs?

[maliqua]

there is a lot of bsd code in the linux kernel bsd is probably the least restrictive license so basically bsd code can be used by anyone for any reason

Re:LOL, so why not use theirs?

[Anonymous Coward]

technically, if the reactor produces heat to turn water into steam, it's almost a drop-in replacement for coal. it's why so many power producers were able to switch from coal to gas so quickly.

Re:LOL, so why not use theirs?

[Anonymous Coward]

How about converting your prop plane into a jet?

Re:LOL, so why not use theirs?

[Hadlock]

Like Apple... meet OSX

Re:LOL, so why not use theirs?

[Anonymous Coward]

Licensing? Can code released under the BSD license be re-released under the GPL?

IANAL.

Sort of. Technically, as you are (presumably) not the copyright holder of the work, you would have no permission to grant any sort of copyright license to third parties.

But the license used by FreeBSD is very permissive. Anyone is permitted to copy, modify, redistribute, or redistribute modified versions of the software and essentially the only requirement is that you also include the three parts of the license text, verbatim, somewhere in all copies.

So what this allows you to do is to create a modified version of the software. If your modifications are non-trivial, they may be a copyrightable work themselves. You may license your modifications to others under the terms of the GPL. The result is a combined work, with portions of it under the GPL license, and portions of it under the terms of the FreeBSD license. This works because someone can distribute the combination and simultaneously satisfy the conditions of both licenses: in addition to the GPL requirements, one must also include the text as required by the FreeBSD license.

Re:LOL, so why not use theirs?

[mccoma]

That's about what I said, except you just cannot change the license on the code itself. Some folks think you can strip the copyright and license from BSD code and that just isn't so.

Re:LOL, so why not use theirs?

[TheRaven64]

Quite a few consumer router manufacturers actually ship Linux with the FreeBSD network stack grafted onto it. I don't know why.

Re:LOL, so why not use theirs?

[Anonymous Coward]

Still produces forward thrust, I'd say it's a drop-in replacement - nothing to it :P

Re:LOL, so why not use theirs?

[Anonymous Coward]

Did you miss the part where they aid they want the Linux networking stack to be better than FreeBSD's? You can make a better solution if you simply copy someone else's.

Re:LOL, so why not use theirs?

[petermgreen]

Coal to gas conversion is pretty easy (though the result won't be as efficient as a combined cycle plant desgined to run on GAS). You just have to add some gas jets and gas piping to the combustion chamber. The design of the boiler and it's operating pressure can stay much the same and any extra physical space requirements are minimal.

That doesn't mean coal to other things will be anywhere near as easy even if the fundamental "boiling water drives steam tubine" bit remains. You have to think of the physical location requirements of the new steam source, the steam pressures it can provide without overheating (AIUI nuclear power plants tend to run at lower temperature/pressure than coal fired ones).

In the case of fusion it is expected that to achive breakeven fusion plants will have to be BIG.

Re:LOL, so why not use theirs?

[Anonymous Coward]

Please provide examples.

Re:LOL, so why not use theirs?

[Anonymous Coward]

Look, this is FreeBSD ... why not just take their damned code?

It's not like you're not allowed to do that. That's what is great about the BSD license.

If FreeBSD's network stack is what you aspire to, why reinvent the wheel?

FreeBSD people would be fine with that. It is the Linux side that forbids the import. The license is too free to include in the kernel. They don't like zfs there either.

Re:LOL, so why not use theirs?

[mi]

Check your /usr/include/X11/extensions/fontcachstr.h for example. Or, to put us back on topic of FreeBSD networking, the <netinet/igmp.h>, <netinet/if_ether.h>, and <net/ethernet.h>...

Re:LOL, so why not use theirs?

[wonkey_monkey]

You can't just copy/paste code and expect it to work

Yes you can. You'll be disappointed, but you can.

Re:LOL, so why not use theirs?

[countach]

It's the almost bit that will get you unstuck. No doubt the BSD network code could be dropped into Linux, but making it all work still wouldn't be trivial.

Re:LOL, so why not use theirs?

[TheRaven64]

It's a common misconception that any code that you link with Linux has to be GPL'd. The GPL only requires that linked code has no restrictions not present in the GPL and that the combined work is covered by the GPL. Individual components can be in one of several more permissive licenses. The Linux kernel already contains fairly large amounts of BSDL and similar code.

Re:LOL, so why not use theirs?

[Anonymous Coward]

No they don't.

Re:LOL, so why not use theirs?

[Anonymous Coward]

And yet you couldn't even name one of these mythical manufacturers or the model number of router? Real convincing.

Re:LOL, so why not use theirs?

[david_thornley]

You don't have to modify the code. You can take a BSD-licensed piece of code and just slap it into a GPLed program. It is then subject to the requirements of both licenses, and (as long as we're not dealing with the BSD advertising clause) since they're compatible there's no problem. Of course, the original is still available under the BSD license, and you do have to leave that license in even if you change it.

Non-trivial modifications to the source (and "trivial" really does mean trivial here - if they require any creativity, they're not trivial) are copyrighted by whoever does them, and require a license in order to be useful. In a GPLed program, the modifications can be used under the GPL, and if the modifications would be useful to the original it's nice to BSD them as well.

Re:LOL, so why not use theirs?

[Anonymous Coward]

It's never actually been clear to me that it really is legally possible to "upgrade" a derived work from a BSD license to the GPL. The BSD license says you can redistribute modified or unmodified source or binary code as long as you include the copyright statement, the disclaimer, and the list of conditions (which includes the terms allowing redistribution and modification). It doesn't say you can't impose additional terms, but it doesn't explicitly say you can, either.* And since you only own 1) the complete derived work as such, and 2) your individual changes, if someone subtracts out your changes, summarizes them to a third party who then clean-rooms their own implementation, I'm not sure you'd have any standing to make a GPL claim there.

* In any event, the GPL does forbid you from imposing conditions outside its scope upon redistribution, which I think makes it impossible for anyone receiving a GPL/BSD mixed license product from redistributing it - "You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License." By including the BSD terms, you are violating the GPL, but by omitting them you are violating the BSD license.

Re:LOL, so why not use theirs?

[Anonymous Coward]

In any event, the GPL does forbid you from imposing conditions outside its scope upon redistribution, which I think makes it impossible for anyone receiving a GPL/BSD mixed license product from redistributing it - "You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License." By including the BSD terms, you are violating the GPL, but by omitting them you are violating the BSD license.

The GPL-3 is much clearer on this point, it explicitly permits addition of restrictive terms that fall into one of several narrow categories. One of those categories is additional text preservation requirements, such as those found in the BSD-style licenses.

I think the argument for GPL-2 goes that the BSD-style requirements are in practice not any different than what the GPL already requires, so it's not really an extra restriction.

Requirements

[Anonymous Coward]

Backwards compatibility with In-Q-Tel required.

Biggest troll on Slashdot ever

[CAPSLOCK2000]

Insulting their work might not be the right way to get the best Linux kernel network engineers to join your company.

Re:Biggest troll on Slashdot ever

[Anonymous Coward]

Insulting their work might not be the right way to get the best Linux kernel network engineers to join your company.

Or it might be the best way ever. Linux people and their egos....

Re:Biggest troll on Slashdot ever

[Zapotek]

How is that an insult? As a coder (software engineer/developer/whatever) myself I'm always glad when people point out things in my systems that can be improved. The possibility of someone hiring someone else to improve them for me, that'd make me ecstatic. This isn't about egos, this is about getting stuff done right.

Re:Biggest troll on Slashdot ever

[Anonymous Coward]

Oh horseshit. This would be a dream job for a kernel developer; "we'll pay you well to make this thing not suck."

How many job descriptions in software development has, as a premise, improving an important code base? 99.9% of software work is piling one new feature or workaround on top of another, stacking steaming shit piles ever higher. Here, Facebook is offering to pay people to improve a code base to compete with an excellent rival in a highly visible way.

If you don't think that will stroke the ego of the sort of people that contribute to Linux then you don't understand who the fuck you're talking about.

Re:Biggest troll on Slashdot ever

[rogoshen1]

Yes, but Facebook. Somewhere between working for the SS and Big Tobacco.

Re:Biggest troll on Slashdot ever

[Anonymous Coward]

Revulsion at tracking people and selling eyeballs to advertisers hasn't stopped kernel developers working for Google.

Re:Biggest troll on Slashdot ever

[fuzzyfuzzyfungus]

Yes, but Facebook. Somewhere between working for the SS and Big Tobacco.

So snappy uniforms and very, very, deep pockets?

Re:Biggest troll on Slashdot ever

[rogoshen1]

Yar, but you don't see the immediate effects of guilt and shame rotting their souls away; slowly turning them into twisted husks of matted neckbeard.

I'd imagine being at Facebook would increase the rapidity of that process a thousand fold.

Re:Biggest troll on Slashdot ever

[donscarletti]

As a programmer, project manager and substantial shareholder in my own company, I'm always grateful when people point out things I do that could be done better.

If I was just the former and not the latter two, I think the relative importance of my ego and the quality of the product would be shifted somewhat.

Most, Open Source programmers tend to value recognition pretty highly relative to absolute merit of the product. To your average guy, if you say "your project sucks, we'll pay you to fix it" means that you will get the credit when it is fixed. If you say "your project is good but it could be better" means you will give them the credit either way.

FreeBSD network stack

[CadentOrange]

What makes the FreeBSD network stack superior?

Re:FreeBSD network stack

[gstoddart]

Way more cowbell, and a much cooler logo.

Re:FreeBSD network stack

[maliqua]

FBSD essentially always wins who has the fastest network stack contests, sets internet speed records etc.

it just is always has been slightly a head.

There is a reason why juniper uses it to base junos on

Re:FreeBSD network stack

[Zarjazz]

As someone who has used various BSD's and Linux in large scale environments, and is a fan of both, I've configured servers with multi-10Gb interfaces and handling 100k+ requests a second I honestly can't think of any example of where Linux has been inferior. The often repeated line that FreeBSD has a better networking stack was probably true over 10 years ago with Linux 2.2 and earlier, but since then I'd say that myth is just bullcrap.

Maybe Facebook are talking about some specific IPv6 or cutting edge features like MPTCP they need on their network, but as a general statement it's utterly misleading.

Re:FreeBSD network stack

[Anonymous Coward]

I've always wondered too.

Could someone please provide a explanation that does not demand me to read the entire stack code?

Re:FreeBSD network stack

[Bengie]

A lot of sysadmins from companies that push a lot of data over lots of connections have blogs about tweaking your OS to handle stuff like 10gb+ of traffic and millions of connections. A lot of these people complain about Linux having strange problems under these loads, and FreeBSD just seems to work. Linux may be faster in some cases, but it still has stability issues that are hard to debug.

Then there's the whole thing about most network stack research happening primarily on FreeBSD because of licensing. There's a new zero-copy network API that was developed in FreeBSD that allows line rate 64byte 10gb traffic on a 450mhz quadcore cpu. Linux and old-api FreeBSD were about 1/10th the packets-per-second.

A new thread friendly socket API has just been pushed to FreeBSD 11. One of Netflix' engineers had a pet project that now allows near zero lock-contention thread scaling. He was able to done line speed 40gb/s with 150k TCP sessions. Instead of having one file descriptor with a single listening thread, you instead have one file-descriptor and listening thread per MSS queue from the NIC and you can lock your thread to the same CPU as the MSS queue, so the packet is already in L2 cache. No shared network state. This also means no share locks with nearly perfect linear scaling and virtually no cache trashing or bouncing.

They're starting work for extend the API to also allow the OS to better handle NUMA and to attach the MSS queues to the CPU to which the NIC is attached. This will virtually remove all cross-talk among the CPU cores trying to handle the network state.

They're looking into expanding this same concept to the Storage IO system.

Re:FreeBSD network stack

[dkman]

If I knew what made it superior I'd be sending a resume, not writing this post.

Re:FreeBSD network stack

[Anonymous Coward]

Very informative, thanks.

Re:FreeBSD network stack

[Anonymous Coward]

Thank you.

Re:FreeBSD network stack

[mvdwege]

[citation needed]

I really want to believe you are sincere and everything you say is true. Unfortunately, you don't seem to have provided any references.

This is the Web. Hyperlinks exist for a reason. Use them.

Re:FreeBSD network stack

[Anonymous Coward]

IPTables/netfilter sucks ass....

BSD firewalls alone make its networking better.

Long live PF && IPFW

Re:FreeBSD network stack

[phoenix_rizzen]

Google searches for "netmap" and "FreeBSD" will give you lots of information on pushing millions of pps through 900 MHz single-core machines. Netmap is also available on Linux. There's even a netmap-enabled version of IPFW that allows you to do packet filtering and routing completely in userspace, again will millions of pps. IPFW is also available on Linux, although I don't know if the netmap-enabled version is.

Google searches for "openconnect" and "FreeBSD" will give you lots of information and blog posts from the Netflix guys about why they picked FreeBSD, and how it all works, including details on the networking.

Google searches for "Adrian Chadd", or "RSS scaling", or similar terms will show you threads and posts on various FreeBSD mailing lists with information detailing a lot of the MSS/RSS work that's going into FreeBSD 11, and several projects that build off that. Those also have links to other information around sockets and similar.

Google searches for "NUMA" and "FreeBSD" will bring up mailing list threads that cover the different projects being undertaken to improve the CPU affinity and thread locality and all that jazz.

Sure, it would be nice if the OP had posted links to the info, but it's not like the information is secret or hard to find.

Re:FreeBSD network stack

[Anonymous Coward]

I'm really getting tired of people like you: This is not a scientific publication, and this is not for sure wikipedia, this is a forum where people expose opinions. Want citations? Go back to the fucking Wikipedia and please DO verify some of the already present sources on many articles there (I grew tired of pointing out the ones that didn't even mention what they're supposed to support). Otherwise use a search engine, the beauty of the 2000's is that you no longer need to spend hours searching for information on crappy information bases and "search" engines. Asking people to do your research for you just tells me you're a lazy bastard.

Re:FreeBSD network stack

[Anonymous Coward]

For F's sake!!! This is /. Not References needed. Just Google it.
Go drink some Sake. :-)

Re:FreeBSD network stack

[Bengie]

They're random blogs that I tend to come across when googling how to tweak network stacks for handling lots of connections. Take it as you will. It was an entirely anecdotal and opinionated blob of text that I probably just should have left out.

Re:FreeBSD network stack

[Bengie]

Re-reading when I posted, this was meant for just the first sentence talking about the blogs. The rest of the information is "true", but based off of memory. The part about the threaded api code for FreeBSD 11 was from a youtube from BSDCON(or similar convention)(https://www.youtube.com/watch?v=7CvIztTz-RQ) that was just published a week or two ago. The zero copy part is about "netmap" I think. Another /. post in this thread also mentioned it.

Re:FreeBSD network stack

[Anonymous Coward]

I see you make a lot of comments with a lot of "facts" but you don't cite anything. Why don't you step up your game first?

Re:FreeBSD network stack

[Anonymous Coward]

+1, Masochistic delusion

Re:FreeBSD network stack

[ray-auch]

Red demon costumes vs. penguin costumes. No contest really.

http://freebsd-image-gallery.n...
http://freebsd-image-gallery.n...

http://www.flickr.com/photos/1...
http://www.flickr.com/photos/1...

Re:FreeBSD network stack

[mi]

Google searches for ...

Sorry, but no. The onus is on you — one making the claim — to offer links supporting it.

You Google it, you pick the links you deem most suitable, you embed them in your posting.

Making the reader do it is not only impolite, it also makes it easier to attack your argument (that FreeBSD is superior) — the Google search could very well offer a link to some blog saying "FreeBSD networking sucks"...

Re:FreeBSD network stack

[Anonymous Coward]

Would be interesting Calomel's FreeBSD/OpenBSD 10Gbe tests replicated on Linux to get an idea of how they differ - https://calomel.org/freebsd_network_tuning.html

Re:FreeBSD network stack

[Anonymous Coward]

Touche!

Re:FreeBSD network stack

[Lennie]

The MPTCP stack for Linux isn't in mainline, but is much further ahead then the FreeBSD version (not sure if it that in mainline).

Re:FreeBSD network stack

[mvdwege]

Ok, that's too bad. I was genuinely curious about the work being done.

Sorry if I sounded a bit gruff, but when I see categorical statements, I like to have references to check myself.

Re:FreeBSD network stack

[mvdwege]

Ok, who did I hurt by contradicting his faith?

Re:FreeBSD network stack

[shutdown -p now]

What makes you believe OP was "making an argument"? He was responding to a comment that asked for more details, and he has provided them. If you want to argue with him, go ahead, but then you'll be the one making the claim that he's wrong (or whatever it is you want to argue about).

Re:FreeBSD network stack

[Anonymous Coward]

...before you wreck yourself?

Re:FreeBSD network stack

[Cramer]

And the absence of any pesky gotta-show-you-the-code GPL cruft has nothing to do with it.

Re:FreeBSD network stack

[Anonymous Coward]

Hi! adri here. I'm Adrian (adrian@freebsd.org) the ex-Netflix engineer who was doing this as his project whilst I was working there.

I'm continuing to do it in my spare time now, as time and hardware permit.

https://wiki.freebsd.org/NetworkRSS

I'm working out the kinks in how IP fragments are correctly handled. It'll be more useful for real world deployments after that - unfortunately in the real world you have to deal with fragmented TCP and UDP; you can't just pretend it's not there.

I'll continue chipping away at it after the Defcon weekend. It's nice to see no lock contention between CPUs when doing TCP and UDP workloads. The main thing that's left is NUMA awareness.

I'll likely do up a memcached style example once this is done so we can really stress it. I'm pretty sure I can get a few million transactions a second out of it on 8-core single-socket hardware - without RDMA, without crazy vendor-specific products.

The thing to keep in mind is that I'm doing this in my spare time now and it's not being funded. I've had 10G and soon 40G NICs donated to me and my employer will likely loan me some 8, 16 and 32 core machines to expand upon this.

Re:FreeBSD network stack

[Anonymous Coward]

I never knew ... http://freebsd-image-gallery.netcode.pl/_daemonette/freebsd_t_shirt.jpg

Re:FreeBSD network stack

[Anonymous Coward]

That public spanking you got in the GCC thread really hurts, doesn't it, perpenso?

Re:FreeBSD network stack

[CadentOrange]

That's a useful list, especially if you have no clue as to what to actually search for. Like me.

Cheers.

Re:FreeBSD network stack

[TheRaven64]

Netmap is also available on Linux

While this is technically true, we recently had a PhD student try this. First, the Linux version is only available as patches so it took him a while to find a version of Linux that they'd apply against. Once he'd done this, it turns out that the driver support is basically only there in Linux for Intel NICs, which are modified as part of the patch. In FreeBSD, because it's merged into trunk, most NIC drivers support it.

Oh, and Adrian isn't at Netflix anymore, but he's still working on networking stuff (now at one of the CDN companies, but I forget which one).

Re:FreeBSD network stack

[Anonymous Coward]

> FBSD essentially always wins who has the fastest network stack contests, sets internet speed records etc.

Which ones? Back when the "Internet 2 land speed records" ran, Linux was almost always being used for record attempts toward the end, although NetBSD was occasionally used too.

I'm not aware of other "fastest network stack constests" or internet speed records, but I'll be really interested in them if you would please point me to them.

Re:FreeBSD network stack

[Anonymous Coward]

new zero-copy network API that was developed in FreeBSD that allows line rate 64byte 10gb traffic on a 450mhz quadcore cpu.

http://stackoverflow.com/questions/13059070/tcp-and-pf-ring

Instead of having one file descriptor with a single listening thread, you instead have one file-descriptor and listening thread per MSS queue from the NIC and you can lock your thread to the same CPU as the MSS queue

http://lwn.net/Articles/370153/
https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD tl;dr Linux does all that shit

They're starting work for extend the API to also allow the OS to better handle NUMA and to attach the MSS queues to the CPU to which the NIC is attached.

cool story bro, but "benchmarks or it didn't happen."

I don't deny that FreeBSD can do better sometimes. Their clean code is an advantage. Just trying again with a fresh set of eyes and slightly different culture can produce better results, just from genetic drift, so it's good to have more than one go at a problem.

but characterizing Linux as having to "catch up" with BSD is basically a troll, and the examples you cite are mostly stuff that Linux did first, or at about the same time.

Solaris 10 made a big deal about their lock-free CPU-affinity TCP stack, but Linux was already doing it at the time they were bragging. Granted, Linux work is spread over a bunch of patches and modules and config flags and /proc flags and slapdash userspace knob-twiddler tools, to the point it's just retarded, but for a shop like Facebook that should not be an obstacle.

Re:FreeBSD network stack

[saleenS281]

Out of curiosity, are they giving the code back, or is it all tied up in their own distro?

Re:FreeBSD network stack

[adri]

Hi,

That'd be me (adrian@freebsd.org)

https://wiki.freebsd.org/Netwo...

Re:FreeBSD network stack

[adri]

Me - adrian@freebsd.org. Also, https://wiki.freebsd.org/Netwo... .

Re:FreeBSD network stack

[adri]

I don't work at a CDN company. :-)

I work at norse-corp.com. RSS is a non-paid thing I do at home.

Re:FreeBSD network stack

[TheRaven64]

Oh, sorry. I was thinking of Sean. Both of you switched jobs at almost the same time.

Re:FreeBSD network stack

[iggymanz]

you can follow provided links elsewhere in thread, meanwhile we can also express opinions. For example, I make my living with Linux server, but for my own domains and appliances I use two BSD, it is superior. FreeBSD has the superior performance of any open source OS, superiority in other realms (correctness to standards, security) goes to OpenBSD.

That's simple

[Anonymous Coward]

Hire Linus

Well played, Zuckerberg, well played

[Anonymous Coward]

what a beautiful job of trolling Torvalds et al

Hire Me! I'll do it.

[Anonymous Coward]

Step 1: Use FreeBSD

Step 2: There is no step 2.

Re:Hire Me! I'll do it.

[Anonymous Coward]

Good, port all the stuff to freebsd, there's 100 gigs of source code. Gotta be ready by next month. Also, you need to port all the drivers to our servers. Aahh, now, are you going to go ahead and have those TPS reports for us this afternoon?

Re:Hire Me! I'll do it.

[TheRaven64]

How much of Facebook's code is not platform-independent already? Most of their web infrastructure is PHP with a custom compiler. The compiler may need some tweaks, but the PHP code that it compiles should be completely independent. Drivers are a different issue, but if they're employing a load of kernel devs it's a lot easier to write a few missing drivers than make large and invasive changes to the network stack.

Why not just use FreeBSD then?

[Anonymous Coward]

Because that would make sense.

Why not create more Lennartware?

[Anonymous Coward]

Why not just merge the TCP/IP stack into systemd like everything else? :D

I was only half joking. This is something where I have anxieties about a younger developer (who didn't grow up with UNIX culture and philosophy) building a net/firewall/security/IIS type monolithic binary (with binary logs) in response to this.

I try to stay optimistic, but you know.

Re:Why not create more Lennartware?

[Anonymous Coward]

[hushed tone to everyone else on site] No one tell him about firewalld.

Re:Will Happen

[Jizzbug]

Userland has little to do with in-kernel TCP/IP stack... Your argument is invalid.

Why target second best, go for the best.

[Anonymous Coward]

They are still missing the mark. The Solaris has the best performing stack. They should go hire up some of the Sun Micrsystem programmer that are looking to leave Oracle.

True that one can not just copy the code. But if you get the engineers that made the best, you can at least get the best improvement.

Remember Microsoft Windows?

[rs79]

It used the FreeBSD networking code. This doesn't mean windows is fast and it's sort of specious. BSD has tricks in the Kernel to make I/O faster that pretty much anything else.

Re: Remember Microsoft Windows?

[Anonymous Coward]

BSD doesn't use tricks, at least not any that Linux doesn't use.

The problem with Linux is two-fold. 1) feature creep. Look at the Linux code and then at the FreeBSD code. Linux is totally convoluted, partly because of the million random patches that have touched the code. BSDs are much more systematic when adding or changing features. It's a tighter knit community.

2) Linux has had a ton of real-time patches applied. Generally speaking, real-time algorithms and tweaks will run slower overall. There's often a tradeoff between efficiency and low-latency.

Re:Remember Microsoft Windows?

[Dahan]

It used the FreeBSD networking code. This doesn't mean windows is fast and it's sort of specious. BSD has tricks in the Kernel to make I/O faster that pretty much anything else.

No it didn't. A few utilities that nobody used (e.g., the commandline ftp.exe, which doesn't even support PASV mode) were ported from BSD (not even FreeBSD), but the TCP/IP stack in Windows was not from BSD.

Re:Remember Microsoft Windows?

[Anonymous Coward]

> It used the FreeBSD networking code. This doesn't mean windows is fast and it's sort of specious.

Early versions of NT shipped with a couple of bsd-derived userland utilities like telnet. But the network stack in the kernel had no significant amount of bsd code. So yeah, 'sort' of specious is an understatement.

Re:Remember Microsoft Windows?

[phoenix_rizzen]

Windows NT used a STREAMS-based networking stack, culled from some other UNIX (not directly, but using the concepts and frameworks), not a BSD-derived networking stack.

I have no idea how the DOS-based Windows networking stack developed. But it wasn't pulled from any BSD.

A few command-line utilities (ftp.exe is the most common cited one) were pulled from BSD sources, though.

Re:Remember Microsoft Windows?

[Anonymous Coward]

Not true at all. Microsoft stole the TCP/IP stack from either FreeBSD or OpenBSD when they couldn't write their own. This is confirmed by TCP/IP packet fingerprinting. The update occurred sometime between Windows NT4 and Windows 2000.

"In their 'completely rewritten' TCP stack for NT5, Microsoft uses 0x402E. Interestingly, that is exactly the number used by OpenBSD and FreeBSD."

http://nmap.org/nmap-fingerprinting-old.html

Corporate Contributions to OSS

[phizi0n]

I don't understand why there's all these comments saying they should just use FreeBSD. There are many reasons to despise Facebook but their desire to improve the Linux networking stack is admirable. We should be encouraging corporations to contribute to OSS, not telling them to just use that other thing that is better in some ways but not others. Kudos to them for contributing back to the projects they use.

Re:Corporate Contributions to OSS

[Anonymous Coward]

You don't walk into a hardware store and ask for a hammer, then ask them to turn it into a shovel so you can dig a hole. They will look at you like you are stupid and offer you a real shovel.

Look, I can understand that your a GNUt, but this isn't about "contributing back" to Open Sores. What this is really about is whiners. Many (many!) large scale tech shops use FreeBSD because it's the right solution. However, they hire Linux people because they are abundant and "close enough." These new hires then start quipping that it should be built on Linux because Linuxism is the only thing they know. When one of those guys makes a director title, they immediately set about making Linux better than FreeBSD so they can swap them out.

Facebook is not the first company to go through this. They wont be the last.

Huh?

[Millennium]

What effect does the userland have on the TCP/IP stack?

High perf SMP coding is in a category of its own

[m.dillon]

Designing algorithms that play well in a SMP environment under heavy loads is not easy. It isn't just a matter of locking within the protocol stack... contention between cpus can get completely out of control even from small 6-instruction locking windows. And it isn't just the TCP stack which needs be contention-free. The *entire* packet path from the hardware all the way through to the system calls made by userland have to be contention-free. Plus the scheduler has to be able to optimize the data flow to reduce unnecessary cache mastership changes.

It's fun, but so many kernel subsystems are involved that it takes a very long time to get it right. And there are only a handful of kernel programmers in the entire world capable of doing it.

-Matt

Re:Will Happen

[phoenix_rizzen]

Except when you start talking about netmap. :) That's a userspace network stack that can push millions of pps, on sub-GHz systems.

There's even a netmap-enabled version of the IPFW packet filter that runs in userspace, filtering millions of pps on sub-GHz systems.

And there's an applications ecosystem starting to grow around netmap that keeps all network-related packet processing in userspace.

As a twist, netmap and IPFW are also available on Linux, and provide better performance than the in-kernel network stack and iptables. :)

Switch to Solaris... seriously...

[Anonymous Coward]

Best TCP/IP stack on the market today (Solaris 11.2)

Re:Switch to Solaris... seriously...

[d3vi1]

While I'm a Solaris admin for some time, I can tell you that it's not the best TCP/IP stack. It does have all the bells and whistles, but it's not even close to the speed of FreeBSD. It's actually not even in the same ballpark as FreeBSD. It's probably Linux fast if you tune it properly. It does have cool configuration, virtual switches, link aggregations, hardware crypto that can be usable by OpenSSL, OpenSSH, and ipsec but it's not even close speed-wise. But the cost of all those features basically means that it has mediocre performance for simple, yet performance-hungry scenarios.

Re:Switch to Solaris... seriously...

[TheRaven64]

Out of interest, how many cores are you using? I've not used Solaris for... a long time, but their network stack was redesigned a little while ago to do each layer in a different thread with lockless ring buffers between them all. I'd imagine latency might suffer a bit, but throughput should be insane with enough hardware contexts. On something like the UltraSPARC Tx series, I'd expect it to outperform FreeBSD, but I've never seen numbers.

Re:Switch to Solaris... seriously...

[d3vi1]

I remember seeing at some point numbers. It didn't impress in a single thread, but could easily saturate a 10Gb link in multi-threaded tests. They tested an FTP server on a T2plus. Regarding cores, we have anything from dual UltraSPARC IIIi to T4 based systems including some M-class. I believe the T3-4 has the highest number of cores. It should be 64 cores and 512 threads, but a single Solaris instance can only see 256. I believe that the M9000 and M9000-64 should have the same problem, but the biggest M series I've worked with is M8000.

Holy shit

[Anonymous Coward]

Facebook actually did something I approve of?

I'm not sure I want to live in this world anymore.

Linux's stack is still 10000x better than windows'

[Anonymous Coward]

Windows is simply out of the league. Cheap toy for pleb.

Why not rebuild your ports?

[mi]

People, you do not have to rebuild your ports just because you updated.

And why would it be such a bad thing, even if you had to? I maintain a handful of ports myself and people's complaining about rebuilding them always irks me — what am I missing?

There are, of course, precompiled binary packages for almost all the ports (where licensing allows redistribution of such binaries). But I don't use them myself — building everything from source. Why not?

Re:Why not rebuild your ports?

[Anonymous Coward]

If you are managing a bunch (more than 2) FreeBSD machines building ports on the live system is wrong and counterproductive. In this kind of scenario you really should setup a package building machine and distribute your own binary packages to them. With pkgng this really works well.

This applies if you don't want to use the official binary packages (for example because you need different options from the defaults, or you tweak some ports)

Why do you think they will contribute back?

[Anonymous Coward]

Who says they will give back anything?

Re:Why do you think they will contribute back?

[phizi0n]

The job posting specifically says they prefer to have the code in mainline.

Huh?

[Anonymous Coward]

Anytime you add the word "user" to anything, it only has negative effect

FreeBSD developers get better jobs.

[Anonymous Coward]

As a FreeBSD developer I now understand why it's becoming so easy to get jobs in big Linux camps.
Guess what ... no matter how big feet a developer may have, just kicking the pig won't make it fly.

But it DOES support....

[Anonymous Coward]

ports/games/fortunes-data-jokes :)

Why not rebuild your ports?

[Anonymous Coward]

I don't know why it's a bad thing, I don't think it's a bad thing, but people are always complaining about it and no only is it not a big deal.. it's not even required.. I was pointing it out :-)

2 clause and maybe 3 clause BSD...

[Anonymous Coward]

is fine, ZFS however is licensed under the CDDL which is incompatible with GPLv2 (and probably a bunch of other stuff).
It is also a decidely *LESS* free license than BSD, although depending on what you're concerned with it may be more free than the GPL, but regardless it's not compatible with GPL and is only compatible with BSD insofar as the released work could be licensed under the CDDL without running afoul of conflicting license provisions unlike CDDL/GPLv2.

Hardware support

[phorm]

Hardware support, in particular, is very far behind

Amusingly, I had the opposite problem years back. The wireless drivers and stack were better on BSD, while on Linux they were hard to find, and often you ended up having to use ndiswrapper which was a nightmare (often resulting a decision between "do I upgrade my kernel and fix X, or keep my kernel and have working wifi").

Just taking a quick peek, it looks like USB3.0 works, but it depends on your host-controller whether it's supported or not.

Why get rid of perf?

[Anonymous Coward]

Oh I see, get rid of performance issues.
Why not just write it out to avoid confusion?

Re:Why not create more Retardware?

[Anonymous Coward]

they figured a firewall belongs into the kerneld after all

HA!

[Anonymous Coward]

They will probably optimize it the way they optimize their news feeds: drop 80% and randomize the rest.

BSD is Dying...

[Anonymous Coward]

https://www.youtube.com/watch?v=g7tvI6JCXD0

It's not just NetMap that brings performance ...

[Anonymous Coward]

FreeBSD also includes an alternative to select/poll called kqueue that allows it to scale client connections massively with minimal performance degradation. Linux introduced epoll as a work-alike, but it has some drawbacks ...

http://www.eecs.berkeley.edu/~sangjin/2012/12/21/epoll-vs-kqueue.html

What's a massive scale? WhatsApp, recently acquired by Facebook, uses FreeBSD and Erlang to power it's service offerings. They sustain over 2 million simultaneous client connections per FreeBSD server ...

http://blog.whatsapp.com/196/1-million-is-so-2011

I wouldn't be surprised if the internal comparison between Linux and FreeBSD network features/performance was fueled by feedback from their new subsidiary.

FreeBSD also works very closely with the Nginx community. If you look at the dev mailing list, you will see a fair amount of kernel level dev work sponsored by companies that use nginx on top of FreeBSD. This constant tuning keeps nginx consumers loyal to FreeBSD for obvious reasons. There is no wonder why this combination was selected by NetFlix to power their new content delivery network.

Re:It's not just NetMap that brings performance ..

[Anonymous Coward]

FreeBSD also works very closely with the Nginx community

Both nginx and memcached were written by active FreeBSD devs and were developed for FreeBSD in mind. The *BSD community in general loves portability and clean code, so these programs also run great on Linux.

I hate FB...

[vandamme]

...slightly less, now.

Re:I hate FB...

[iggymanz]

wrong response, you should be hating Linux slightly more. Great for desktops, but for servers for the common stacks BSD is usually better

Re:It's not just NetMap that brings performance ..

[Anonymous Coward]

FreeBSD also includes an alternative to select/poll called kqueue that allows it to [snip]

Once again, you're bragging about something everyone supports.

Linux calls it epoll
Solaris calls it /dev/poll or something (I forget)

Linux introduced epoll as a work-alike, but it has some drawbacks ...

http://www.eecs.berkeley.edu/~sangjin/2012/12/21/epoll-vs-kqueue.html

All of the interfaces work fine in nginx and http://libevent.org/

I'm sure you can find some strong argument that the BSD interface is more elegant because it almost always is, but unfortunately, no one cares. kqueue might have a performance advantage, but if so it does not have the kind decisive performance advantage the job posting suggests it might.

and there is also kevent in Linux. As usual Linux is a mess in this area, and there are lots of ways to optimize programs with lots of file descriptors (http://bulk.fefe.de/scalability/ ), and this is only one of them. tl;dr linux works fine with lots of file descriptors.

Won't Happen

[saleenS281]

Which works great if you're writing all your own software, or are able to rely on nothing but open source. VERY few companies have that luxury, and FreeBSD/OpenBSD are a non-starter for anything you want vendor support on. RHEL and SLES on the other hand are almost universally supported by hardware and software vendors alike.

Linking to evidence (Re:FreeBSD network stack)

[mi]

My objection was not to merits or lack thereof of a particular OS, but to the practice of placing the burden of research on the audience (and opponents).

Whatever it is you are stating, should be backed by evidence. It is best to include the links with the statement being supported, but it can be tedious. So, links should be provided upon request — without any lip like "just google it yourself"...