Paid Developers Power the Linux Kernel

Hugh Pickens writes "Believe it or not, there is still this illusion that Linux and open-source software is written by counter-culture, C++ programming cultists living in their parents' basements or huddled together in Cambridge, Mass. group-houses. Now CNet reports that the Linux Foundation has found that 'over 70% of all [Linux] kernel development is demonstrably done by developers who are being paid for their work.' That Linux is primarily developed by paid developers should come as no surprise considering that Linux enables many companies — hardware, software, and online services — to be more competitive in their markets and to find new ways to generate revenue. 'What's important about how Linux and open-source software is created isn't the side issues of politics or how its developers are perceived; it's that its fundamental methodology produces better software,' writes Stephen Vaughan-Nichols."

Comments

C++ programming cultists?

[razvan784]

It's written in C, not C++.

Re:C++ programming cultists?

[bcmm]

Mod parent funny. I think that bit really sets the tone for the rest of the summary.

Re:C++ programming cultists?

[Anonymous Coward]

There is a very small part in C++, how ever Linus got alot of critique for that but none the less committed it.

Re:C++ programming cultists?

[mswhippingboy]

C++ programming cultists often write in C.

Re:C++ programming cultists?

[Zarhan]

"make xconfig" as far as I know uses Qt.

Re:C++ programming cultists?

[bogolisk]

make xconfig!!!!

Then you probably can claim that Linux is written partly in bourne shell.

Re:C++ programming cultists?

[SigmundFloyd]

C++ programming cultists often write in C.

Those are C++ programming occultists.

Re:C++ programming cultists?

[marcosdumay]

That would be a reason to not use templates... But the Linux kernel is already so dependent of GNU tools that it may not even be a problem.

Really, classes are a very good thing to have. You don't need to throw it away with the rest of the C++ language.

Re:C++ programming cultists?

[Tacvek]

Because Bjarne Stroustrup's C with Classes language is merely an early version of C++. I mean modern C has evolved quite a bit since that time, and it would be a shame to have to limit yourself to the C language constructs of 1983.

Re:C++ programming cultists?

[TheRaven64]

The Linux kernel currently compiles with GCC, Clang/LLVM, PCC, Path64 (development branch only - some of the inline asm isn't handled correctly by the released version), ICC and (I think) XLC. An earlier version also compiled with TCC. It's not really that dependent on GNUisms. Or, rather, the GNUisms that it depends on are pretty well supported by other compilers.

Re:C++ programming cultists?

[EnsilZah]

Actually, little known fact, if you run it through a C compiler you get the Linux kernel, if you run it through a C++ compiler you get SCO Unix.

Re:C++ programming cultists?

[Daniel Phillips]

Yes, thankfully.

Re:C++ programming cultists?

[Zero__Kelvin]

"I'm aware of that; just taking the opportunity to point out that C++ files on UNIX systems can use extensions other than those you mentioned, and in particular that the one bit of C++ in the Linux kernel repository uses the .cc extension. "

Actually, you are only just now taking that opportunity. I appreciate the correction and additional information. In fact, that is exactly why I said I wanted to challenge the assertion, rather than saying there definitely isn't any.

What?

[QuaveringGrape]

The Linux kernel is written in C, not C++. And haven't there been a number of articles on how IBM, RedHat, Sun etc all have employees who develop Linux? One post in particular... and I know there have been others.

I'm not surprised

[jimicus]

My employer uses F/OSS extensively - and as the sysadmin, I've started to notice a pattern.

F/OSS products which scratch an individual or a small group of peoples' itch generally get developed to a certain point and then stagnate. If you're lucky, that point is acceptable to you.

The products that do really well - the "best of F/OSS", if you like - are almost invariably the sort which scratches a very common itch. They're usually bankrolled by a number of companies (the Linux kernel falls under this category) or become self-funding when the project leader sets up a company to sell a commercial version with support and possibly extra features.

Re:I'm not surprised

[DarkKnightRadick]

Are you saying that a meme has come true?!

1.) Develop F/OSS
2.) Market it to a broad range of users
3.) ???
4.) Profit!

Re:I'm not surprised

[Bert64]

Well the same is true anywhere, the more potential users a given piece of software has the more interest there is in developing it...

Commercial software works the same way, something with mass market appeal is highly likely to be developed and either result in multiple competitors or serious effort to stifle competition...
But something small with a very limited market either won't exist at all, or will be extremely expensive if that niche market has the money to pay for it. Niche products also tend to be rather buggy.

OSS serves certain niches very well, ie those niches occupied by technically minded people who are capable of writing what they need for themselves... Other niches are served somewhat less well because those who would be capable of producing software have no need for it, and those who need it aren't capable of writing it.

You also get a lot of businesses and individuals using software which is extremely poorly suited to their needs because they aren't able to customise it for themselves, and the only people who would be capable of doing so aren't willing to.

I'm sure there are many things that could be improved by being computerized, but where the people capable of producing such software either don't realise or don't care.

Re:I'm not surprised

[Kjella]

Well, more people = more developers. But when it comes to paid developers, I'm not sure the same is true. If it becomes too common an itch that you're trying to collect microfunding from millions of people then it's not doing that great. Even though millions of people need a little photo editing they aren't funding GIMP and the professionals rather end up paying for Photoshop, same goes for OpenOffice vs MS Office. Firefox is a bit different in that Google is paying indirectly rather than the users directly, but in general OSS fans are very opposed to any kind of applications that "sells" your eyeballs or your data or direct you to specific services. It seems easier to make one company pay $1000 for RHEL than to make 100 people pay $10 or 1000 people pay $1.

Re:I'm not surprised

[jc42]

One real benefit is that if you are a company developing kernel code and contribute it back you will get goodwill and you will enhance the competence of your employees.

Actually, I've long thought it strange that the business/industrial world has an objection to supporting things like an OS kernel, runtime libraries, etc. The obvious parallel comes to mind: Lost of companies farm out part of their operations to subcontractors. They routinely subcontract for cleaning, delivery, electrical services, for example, not to mention their phone, water and sewage systems. They don't seem to be taken aback by the fact that the companies that supply these services also subcontract to "the competition".

The idea of paying a separate company for software development and supports services is also hardly new. That is how IBM has made much of its money, after all. Paying a company like Red Hat doesn't strike me as very different from any of the above. It doesn't take much management genius to understand that paying a contracting firm for software support at the "system" level is a fairly good idea. That way, you can share the cost with all the other companies that hire the same software firm, and everyone can get the benefits from having the software organized by people who (hopefully ;-) know what they're doing.

So why is this even a story? You'd think there would have been enough sensible businessmen all along for lots of Red Hats to prosper.

A related question is all the propaganda against "open" software. Systems such as water, sewage, electrical, etc. all have "open" designs, with everything published and the detailed specs easily available to anyone. Companies don't often buy fleet vehicles without shop manuals, which give the detailed specs for the innards of the vehicles. Why would people classify open software as "hippie" or "communist", when they don't say the same about shop manuals or electrical diagrams? You'd think that sensible managers who approve of open standards for these other things would also want software that follows published standards (e.g., POSIX), and whose specs (i.e., the source code) is easily available to everyone.

But for some unexplained reason, business people keep buying software systems with hidden, "proprietary" innards. They wouldn't do this with delivery vans or electrical wiring; why would they accept it with software? Exactly the same reasoning says that software should be open, standardized, and accessible to anyone with the technical training. And the same reasoning that supports specialized firms to do common tasks should also support specialized firms for software needs.

It may be yet another example of a theory that keeps popping up: Whenever a computer is introduced anywhere, all precedent is forgotten, and people have to relearn from scratch all the things that they knew from before there were computers. I wonder what it is about computers that causes this social amnesia and inability to see parallel situations?

Old News

[PiAndWhippedCream]

Do even the editors read anymore?

non-story

[Hognoxious]

Believe it or not, there is still this illusion that Linux and open-source software is written by counter-culture, C++ programming cultists living in their parents' basements or huddled together in Cambridge, Mass. group-houses. [...]
That Linux is primarily developed by paid developers should come as no surprise

Hang on a minute. If there's an illusion that it's written by smelly vegan hippies, then surely it should come as a surprise that it's written by paid employees of €eevu£ corpra$hunz.

Though the question is actually irrelevant - everybody has known it for years. It's a non-story.

Re:non-story

[Yvanhoe]

Actually the real surprise would be that these top-coders would be unemployed.

The real question is...

[mswhippingboy]

Is Windows written by a bunch of C# programming neophytes living in Steve Balmer's basement or huddled together in Delhi, India dormitories?

Not a total non-story

[Sycraft-fu]

This perception of OSS still remains rather strong. You will notice that a common advocacy for OSS is the "many eyes" thing. The idea seems to be that there are just tons and tons of developers out there with amazing amounts of time that will jump on a project and help, if only it was opened up. The advocates then point to things like Linux or Firefox or MYSQL and how great they are. What they miss or don't understand is that these high profile, top-flight OSS projects have heavy financial backing. They have developers who's full time jobs it is to work on the software. That's wonderful and all, but don't then try to claim that you'll get that kind of development on a project just because you open up the code.

Re:Not a total non-story

[Attila Dimedici]

Actually, most of the people who miss that are not listening closely enough. The point of the "many eyes" thing with Open Source software is that if an OSS project is of interest to a company (in particular a company that hires programmers anyway) that company can have its people look at the software from the perspective of the way that company uses the software and they can make changes (either bug fixes or otherwise) to make the software better for their company. If they were using proprietary software, they would be at the mercy of the company that owned the rights to that software for any fixes or improvements.

Re:Not a total non-story

[Antique Geekmeister]

Of course opening the code doesn't make it automatic. But closing it often precludes such change or sophistication: when I have professional access to the software base for commercial packages, I'm often _amazed_ at the boneheaded practices I'd pull a release candidate for on the spot, and make the author go back and rewrite it during our code review meetings.

Re:Not a total non-story

[Attila Dimedici]

If they aren't interested in that, how do you explain all of the companies that pay programmers to work on Open Source?
Most (if not all) of those companies already had programmers on staff to write custom software for the company. They discovered that it was easier and cheaper (and often better) to modify Open Source Software than it was to write their own application from scratch (or to buy such an application from some proprietary vendor).

Re:Not a total non-story

[cduffy]

Many companies have learned the hard way the true cost of custom software, which in many ways can be worse than proprietary software.

If you're making custom software out of your OSS software, you're doing it wrong.

If you're doing it right, you're submitting your changes back upstream -- so the software doesn't "stand dead-still", as you put it, even on those times when you aren't shoveling man-hours into improvements. If you happen to be curious for some examples, google around for patches under my name submitted to open source projects over the last decade. Just about all of those were paid for by my employers -- from the OS X VNC plugin bugfix to the feature enhancements to libvirt to improved cover page generation for HylaFAX.

For the work I did at Dell, we worked together with Red Hat to get as many of the libvirt and qemu improvements we wanted as possible into the RHEL6 release schedule, enabling some of Dell's internal QA tools to work out-of-the-box with RHEL6 (whereas those same tools required heavy tweaking on RHEL5). Sure, we could have gotten the same thing done as a professional services engagement rather than a friendly collaboration between engineering groups... but this way was far easier, cheaper and lower-paperwork (and by building the patches in-house, we made sure that we got exactly what we wanted).

No surprises in this article for me.

Re:Not a total non-story

[jc42]

The idea seems to be that there are just tons and tons of developers out there ...

It doesn't usually take very many software developers to make a ton.

Not sure about the conclusion

[Registered Coward v2]

No, what’s important about how Linux and open-source software is created isn’t the side issues of politics or how its developers are perceived; it’s that its fundamental methodology produces better software. That’s why businesses invest in Linux’s development. Linux works. If it didn’t, big business wouldn’t bother with it.

I am not sure the methodology produces better software; it does however produce software companies can use and modify to their needs without paying ongoing license fees. That's why they use it - it allows them to develop other, higher-value, products and maintain control over the source and not be beholden to some third party company. The methodology does work to produce high quality software since many people are looking for bugs, with that part I agree - but companies don't invest in software because others are QC'ing it; they invest because it enables them to make money. While what the author says is, IMHO, true about the Linux kernal, I don't think it is true for many other FOSS projects; unless they are used as part of a larger product, such as a server. There simply isn't the same incentive to spend time and money on an application, such as an office suite, that you can't use to sell something more profitable.

"Believe it or not" ... I'm a not

[petes_PoV]

I doubt this has been the case for 10,15 years or more. The fact is that most of these "counter culture" people are inventions of the media or hollywood and have never, really, existed in the real world. The few people who would describe themselves as such may still exist in some parts of the world, and are usually referred to as criminal hacking gangs, but they've not contributed a dam' thing to the Linux kernel.

That's not to say some individuals with long hair and others with low personal hygiene standards haven't done their bit, but those attributes don't make you counter-culture.

not really the point

[ninja59]

who cares? The point is that it is open. Today paid programmers are doing, before nerdy basement dwellers did it, maybe tomorrow homosexual vampires will do it. Being open allows the "who" to change.

Re:not really the point

[theNAM666]

It was always people from major companies and universities. Read the names and affiliations of the people listed in the man pages :) (-- there's a history of development there).

This is how OSS is SUPPOSED to work

[Anonymous Coward]

The GPL facilitates (forces) the work to be shared among stakeholders. My company buys maintenance and feature development on Postgresql just the same and it's a really great deal for all of us who contribute to eachother not having to buy Oracle licenses.

Re:So it works the way Stallman envisioned?

[jimicus]

The thing is, those other people have to make any of their own improvements available. Compare and contrast that with the BSDs, where there is no obligation to re-release improvements. Lots of expensive, specialist kit is based around a BSD Unix (eg. F5, Juniper). But BSD doesn't have anything like the mindshare in the generic server market.

Re:So it works the way Stallman envisioned?

[cduffy]

I don't see how money is being made with "several companies bankrolling" a piece of software that scratches a common itch. Is it that a particular company sees that it can make money by using, though not owning, a particular piece of software and they don't mind their money making improvements available to other people?

Typically it's "bankrolling" by assigning some of your people to spend some of their time on making improvements you happen to need and contributing them upstream.

(Contributing improvements upstream means that you won't need to continually maintain your patches, that they'll eventually be included in vendor packages/kernels and thus that you'll later need to do less packaging yourself, and is otherwise a money-saving action. As a somewhat-related aside -- once upon a time I worked in embedded Linux, and you had companies who structured their default contracts for kernel work such that everything was submitted upstream, making each contract effectively a one-time engagement when everything was done right, and others who didn't... ehh... encourage their clients to pursue submitting their code, such that said clients would keep paying in to keep their patch current with newer upstream kernels).

Re:ugh

[Kijori]

This being Slashdot, I suspect "better" means more open.

Not necessarily an untenable point of view, but a little tautological when you say that open source software is better.

Featured on Slashdot in August of 09

[ojintoad]

The Myth of the Isolated Kernel Hacker

Ant writes...

"The Linux Foundation's report (PDF) on who writes Linux — "... Linux isn't written by lonely nerds hiding out in their parents' basements. It's written by people working for major companies — many of them businesses that you probably don't associate with Linux. To be exact, while 18.2% of Linux is written by people who aren't working for a company, and 7.6% is created by programmers who don't give a company affiliation, everything else is written by someone who's getting paid to create Linux. From top to bottom, of the companies that have contributed more than 1% of the current Linux kernel, the list looks like this: ..."

Sorry CNET, you're just a tad late on Computer World on this one.

How "many eyes" work

[mangu]

The idea seems to be that there are just tons and tons of developers out there with amazing amounts of time that will jump on a project and help, if only it was opened up

This is a common misconception about free software.

"Many eyes" does not mean every user is a developer, as a matter of fact the vast majority is not.

What "many eyes" means is that IF a user is bothered enough about a bug and that user has the ability to develop software, then he CAN fix that bug. There might be a million users, but if only 0.1% of them are interested developers then there will be a thousand people fixing that bug.

And every user will profit by that fix.

I'm one of those paid kernel developers

[Izaak]

I've been working as an embedded Linux developer for almost a decade now, and yes, most of us kernel hackers are paid for our work. For example, right now I'm working for a major microchip manufacturer that wants to make sure their products are fully supported by Linux. Consequently, they fund teams of open source developers (often hired through big name consulting firms) to port the kernel to their latest CPU's, develop drivers for integrated peripherals, etc. Just look at the email addresses in the submit logs for major open source projects. You will see ibm.com, intel.com, ti.com, redhat.com, windriver.com ..., and many, many more big commercial technology companies. Its been this way for a while, which is why I would always laugh whenever some MS fanboy would try to denigrate Linux programmers as a bunch of basement dwellers. I make a better than average living from Linux coding, with multiple job offers right now, even in this horrid economy.

This is also why I have no worries about Microsoft ever killing off Linux. There are far too many companies making far too much money from Linux based products in market niches that MS has no traction in. The embedded and mobile markets are pretty much owned by Linux, and those are pretty much the only tech sectors seeing strong growth right now. If you haven't yet added Linux skills to your resume, do it.

If anyone wants to ask me about the Linux / embedded / open source consulting world, go ahead and post your questions. I'll check back and answer if I can.

Re:Not the way forward.

[Izaak]

As one of those 'sell outs', I'm curious why you think that? At the heart of the open source ethos is the license under which the software is distributed. As long as the code that is developed is submitted back to the open source community, that ethos is satisfied. How or even if the developer is paid is not really relevant. Indeed, a world in which most commercial software is open source and the developers are paid for their efforts is very much in keeping with our dreams of an open source utopia.