Torvalds Says It's No Picnic To Become Major Linux Coder

Jack Spine writes "Linus Torvalds has given an interview to ZDNet.co.uk about the trials and tribulations of becoming a Linux kernel developer. 'Torvalds said that, while it is relatively easy for coders and organisations to contribute small patches, the contribution of large patches, developed in isolation, could lead to both new and established contributors becoming frustrated. "It's definitely not easy to become a 'big contributor'," wrote Torvalds. "For one thing, the kernel is quite complex and big, and it inevitably simply takes time to learn all the rules — not just for the code, but for how the whole development environment works. Similarly, for a new developer, it will take time before people start recognising the name and start trusting the developer to do the right things.""

Comments

huh

[nomadic]

Similarly, for a new developer, it will take time before people start recognising the name and start trusting the developer to do the right things.

In other words, it's hard because I make it hard.

Re:huh

[Carlos Matesanz]

I Agree. Coding a kernel is not trivial, we all know that. But sometimes it seems that core devs in Linux kernel are so self-sufficient they don't want anyone else really involved in the project. Or at least that's the opinion i've extracted along the years from mailing lists, interviews and whatnot. It's a luck they have been doing a good job until now.

Re:huh

[LWATCDR]

Nope.
It is hard because.
1. Programming is hard.
2. System's programming is even harder.
3. Kernel code is mission critical code which is really hard.
4. When you are the new person it takes time before people trust that you know what you are doing.

In other words it is just like everything else. The difference is that if you want to make changes to your Kernel you can. If you want to put up a site with your patches you can.
If you want your code adopted in the "official" kernel you have to play by the rules and write good code.
So it works exactly as it should and really can not work any other way.

Re:huh

[Seakip18]

Would you let a doctor fresh out of med school slice and dice in heart surgeries?

Would you let pilots start out flying fully loaded commercial jets?

In my limited experience, you don't let a n00b make permanent or long-lasting changes till they have gotten their training wheels off and you know that the n00b knows what the fuck they are doing or at least understand what is going on.

Not to say they couldn't be more welcoming to new developers, but a single bad line (sound familiar debian devs?) and a whole bunch of stuff could stop working.

Re:huh

[dutchd00d]

Bullshit. You can download the source code, make any changes you want and publish your own version without restrictions. That's the definition of open source, so the Linux kernel most definitely qualifies. The fact that it's hard for you to get your changes into Linus' kernel tree has nothing to do with it.

Re:huh

[Skreems]

While I don't agree with your current flamebait mod (you bring up a very common criticism of open source) I do think you're wrong.

Nothing in open source has ever guaranteed that you get to contribute directly to a specific project. When a specific group of developers is maintaining a release (Linus et al, in this case) it's absolutely up to them what code gets in and what doesn't, and who they will accept contributions from. What open source guarantees is that when they make that release, you're free to take the source code they've created and modify it in any way you choose. You're free to fork the project and maintain releases just as strictly as they did, or open it up to all newcomers.

I think you'll find, though, that opening it up to any unknown person right off the bat will trash your project pretty quickly. The problem is, there are hugely varying ideas of what constitutes "correct" code and architecture, and it's just a fact that it takes time to prove that you understand what that means in terms of a large project such as this.

Re:huh

[mea37]

I disagree. I think you have the wrong idea about what "open source" means.

Open source pertains to the code base, not to any particular repository of the code. You are quite free to read, modify, and redistribute (with modifications) the code. You cannot compell Linus to incorporate your changes into his version, any more than he can compell you to revert a change from your version.

That Linus has a widely-respected "official" version is a moot point. It really just means he has an audience for his product (i.e. the version of the code he/his team host), and you may not have one for yours (a modified version you create but which isn't accepted as part of his product).

Much like free (as in speech) speech, open source doesn't guarantee you that anyone will listen (where "listen" in this context means "run your version of the code").

As for MS -- well, getting a job at MS isn't the same as getting a job that lets you make major modifications to the Windows kernel. I'd say the sitaution with regard to the high-profile products (i.e. Windows on oen hand, the "official" Linux kernel on the other) is about the same. The difference is that you can't modify Windows without being part of the official team. Not to distribute (even if you can find an audience for your work). Not even for your own personal use. That's the difference between open and closed source.

Re:huh

[moderatorrater]

I just started a new job 2 weeks ago. I haven't touched any code other than two trivial patches to some HTML. I expect it'll be another 2 weeks before I touch any actual code, and it'll be a few months before I'll touch anything that customers rely on. This is the same process that happens everywhere, the difference being that in the Linux kernel it's more open and ability based.

Re:huh

[b4dc0d3r]

Mod parent down. I don't recognize him, therefore he can't know what he's talking about.

Re:huh

[mr_mischief]

One additional point: don't try to take over a whole subsystem in one rewrite. Contribute small patches that are easily reviewed to get your feet wet and get noticed. Then, as you're better known and respected within the community, start scaling up your contributions.

It works the same way in any open source community. The new guy who rewrites half the code all at once isn't going to get a review of his code. Show that you can do the small changes right first.

Actually, it works this way in almost any cooperative group. You don't show up to your first meeting at Kiwanis, the Jaycees, or the Lions and start making resolutions. You don't sit in on drums once for a band and start telling them how to write songs. The US Senate even has a rotating term cycle so that there are always Senators with more experience to get the junior Senators acquainted with how things work.

People who think they should suddenly be in charge of a large portion of an established organization they've just joined are showing signs of detachment from society or megalomania. If you've never contributed anything worthwhile, you're nobody special compared to the people who have been doing the work. Don't expect to be a big part of a group without being a small part first. Some people move up the ranks faster than others through skill and hard work, but everyone pays some dues.

Re:huh

[LWATCDR]

And the honest truth is that.
Unless you are some kind of super prodigy or have years of experience writing systems code odds are your complete subsystem rewrite is total junk.
It takes time to be good at anything.

Re:huh

[fishbowl]

>Would you let a doctor fresh out of med school slice and dice in heart surgeries?

Yes, if that was their residency specialty. Do you know much about med school matriculation?

Re:huh

[v(*_*)vvvv]

I am no kernel developer, but I think Linus is getting at why "it works exactly as it should and really can not work any other way" has some demerits, and that it not being able to work any other way is why we are in a fix.

In other words, it works as it should, but it is very slow, so how can we improve the process and make better patches faster?

If the answer is that there is no better way, then that is a sad awakening for a lot of us, because it means precisely that Linux isn't going anywhere sooner than it has since the current state has been established.

But there has to be a better way, and I think Linus is trying to find it, as are many others.

Re:huh

[BPPG]

http://janitor.kernelnewbies.org/

Maybe this will answer some of your questions. But yes, and it's good that the linux kernel doesn't operate like wikipedia, for obvious reasons.

Re:huh

[Mastodon]

all surgeons have a first time they perform any surgery.

IAAMD. When you start doing surgery there is a more senior surgeon standing right there, watching your every move, and ready to take over if necessary.

Re:huh

[billcopc]

I think the problem with Linux / Linus isn't so much the patch resistance, but the attitude. Countless times there have been well-written and widely-used patches that were stubbornly rejected by Linus, with little or not explanation as to why. This paints him as a fussy dictator, which may or may not be true - I don't know him personally so I can't say.

It's perfectly valid to reject bad code, and he should continue to do so, to ensure the quality and reliability of the kernel. What would be important, at least in my opinion, is to give some sort of constructive criticism to help that developer improve their code, or maybe point to a similar patch where the developers could join heads.

It's the whole "this sucks, you suck, fuck off" attitude that has built up Linus' reputation as an ego-tripper - so much that his programming ability has taken a back seat to the drama.

Haven't heard of him.

[Neuroelectronic]

Who is this No Picnic fellow? I don't believe there is a new major Linux contributer.

Re:Haven't heard of him.

[Barsteward]

Me neither but maybe he's one sandwich short of a picnic to want to become a major kernel developer, i heard it helps

Re:Haven't heard of him.

[jonaskoelker]

It has been said that:

TFA is a wholly remarkable book. It's already supplanted Operating Systems: Design and Implementation as the standard repository of all knowledge and wisdom, for two important reasons. First, it's slightly cheaper; and secondly it has the words DON'T PICNIC printed in large friendly letters on its cover.

;)

I gave up a few times

[EmbeddedJanitor]

I considered, and once tried, pushing a file system into Linux. Unfortunately the fs does not have the right coding style and a few other things which make it hard to put into mainstream. Instead it just sits independently as a big patch which is pretty easy to apply by running a simple script.

This suites everyone that uses it pretty fine, except for the purist "it's got to be in the mainline" folks. Realists just pull it from a public cvs and apply it with minimal effort.

Although I might consider mainlining it again, for the moment the effort just is not worth it. The current model is workable for those that use it.

Re:I gave up a few times

[Vellmont]

Although I might consider mainlining it again, for the moment the effort just is not worth it. The current model is workable for those that use it.

It sounds like your FS serves mostly a niche that isn't served by the mainline FSs. Call me a "purist", but I just don't have the time or inclination to re-compile my kernel. I did it many years ago to save a few kilobytes of memory when it was at a premium, but these days, why? If you don't care about keeping up on kernel patches and have some specific needs that aren't supplied by the mainstream kernel, then re-compiling is fine. But if not, then the mainstream kernels (vendor provided) wind up being a lot easier to work with.

Re:I gave up a few times

[x2A]

Worse? ... or... better? :-D

I dunno, all I've tried in the past is an SQL injection.

In other news...

[Dorkmaster Flek]

Kernel development is hard! Film at 11.

This goes for every OS/FOSS project out there.

[scenestar]

And considering the widespread usage plus amount of people that rely on the Linux kernel to be stable and not explode in a horrible firestorm I can certainly understand that Linux kernel development requires a Stalinist approach.

Re:This goes for every OS/FOSS project out there.

[houghi]

I am sure this goes for closed source as well. If the rookie comes in, he will also have to prove himself. The same goes for every other aspect in life.

If are new into a group, it is to be expected that you have to prove yourself. Whether this is with your local gang, a family you are married into or a bunch of coders is irrelevant.

Even Neo had to prove himself first. Each group will have its own rules and speed of how fast you are accepted. A group of drunk people in a bar might have a lower standard, but if you do not fit in, they will not accept you and will not listen to your sugestions, no matter how wise they are. (Stop drinking? Why? This is FREE BEER)

Learning the rules

[Fnord666]

"...and it inevitably simply takes time to learn all the rules..."

1. Linus is always right
2. If in doubt, see rule #1

Re:Learning the rules

[Chris Mattern]

Yes, but trick is that since you can't constantly be bugging Linus for all the answers, you have to know what his opinion is without asking him. That's the tough part.

Re:Learning the rules

[SlipperHat]

Yes, but trick is that since you can't constantly be bugging Linus for all the answers, you have to know what his opinion is without asking him. That's the tough part.

Skills required in Linux kernel development:
...
Mindreading
...

Allegiances

[Anonymous Coward]

What happens if:

1. Batman is more or less responsible for a big chunk of the kernel, e.g. the scheduler.

2. Torvalds knows Batman, and knows that Batman is employed by Redhat to work on the scheduler.

3. The Joker writes a new improved scheduler which has the potential to replace the old one.

Now, how does Torvalds react? It would be hard to tell Batman that he's no longer in charge of the scheduler. Batman's job might be on the line - why would Redhat keep paying Batman if he suddenly had a lot less work to do? Maybe Torvalds met Batman a few times and had a beer with him, making it even harder to replace his work because it becomes personal. Torvalds could harm Batman's career.

Surely this makes it hard to become a big new contributor? All the existing contributors already know eachother and they won't want to dump eachother's work.

Am I right or am I right?

Re:Allegiances

[fictionpuss]

Am I right or am I right?

You're a tautology.

But let's take your unproven hypothetical as given for a second. If these sorts of decisions are being made, which provide technically inferior solutions for the Linux kernel.. then over time it will become obsolete.

But way before then we'll all be using the nuLinux kernel which has all of "The Joker"s fancy code.

In other words, F/OSS can take care of itself; we're just the dumb monkeys hitting random keys.

Re:Allegiances

[KasperMeerts]

You forgetting these are full-time kernel developers. They would offer their firstborn son in exchange for a 0.049% better scheduler if they could ever have partners.

Re:Allegiances

[bigstrat2003]

3. The Joker writes a new improved scheduler which has the potential to replace the old one.

when (process.wantsToRun) {
retort(process, "Why so serious!?");
kill(process);
}

Very efficient. I like it.

Still easier than coding the Windows Kernel

[fictionpuss]

Qualify "luck". It seems to me that any large distributed development effort is going to require some sort of process - the anarchic development model isn't terribly successful.

With that in mind, developing the Windows kernel requires you to be employed by Microsoft etc, whereas developing for the Linux kernel just requires you to follow some established open processes.

What's the problem with that?

Re:Still easier than coding the Windows Kernel

[Anonymous Coward]

At Microsoft, working on the kernel pays as a fulltime job

They pay monkeys to bash keys nowadays?

Re:Still easier than coding the Windows Kernel

[cervo]

As further evidence of the egos, remember when Linus attempted to contribute the patches to Gnome as part of the Linus versus Gnome war?

http://www.desktoplinux.com/news/NS8745257437.html

Anyway he didn't start small at all. Go Ego :)

Re:Still easier than coding the Windows Kernel

[GooberToo]

His experiences are far from unique. The problem is the gnome guys have huge egos and anyone offering suggestions are often meet with disdain.

In the link provided above, it's not like Linus' comments are off base in the least. That's hardly egotistical. From the article it's obvious he already did the footwork. He already made an effort. The developers even confirmed it not only does not do what he wanted but they would not do it. He then went off to put his money where his mouth was. To summarize, this means Linus did the right thing and the Gnome developer are shamed and proved impotent, because of their own huge egos.

While I much prefer Gnome to KDE, it has long been screwed over by ignorance and huge egos from none other than Gnome's own Miguel Icaza.

Andy's revenge

[Stormwatch]

For one thing, the kernel is quite complex and big

If that's the problem, wouldn't it be easier to work on it if it was a microkernel?

Re:Andy's revenge

[gardyloo]

Long answer: Fuck no.

    (From Stephen Fry)

Re:Andy's revenge

[0xABADC0DA]

Seriously though, the problem with microkernels is that they (in theory) help the system cope with mistakes but they don't help prevent mistakes in the first place. Each component in a microkernel is isolated from others using memory protection, but they can still corrupt themselves or crash themselves.

There's very few parts of the kernel that actually need pointer arithmetic, unsafe casts, or for that matter need to operate particularly quickly. You don't have to believe me just look at the code. Open up some random source files from the kernel and look for pointer math, unsafe casts. Figuring out what locks are held when is harder, but can be done (performance being more important when locks are held).

Microkernels are solving the wrong problem. They should be focussed on preventing the errors in the first place not on recovering from them. So, a 'safe kernel' that is mostly written in a language that prevents errors, such as Limbo/Dis or for that matter Java or C#. That would be much easier to work on and an improvement over Linux style kernels.

Re:Andy's revenge

[serviscope_minor]

If that's the problem, wouldn't it be easier to work on it if it was a microkernel?

Yes. Fortunately linux is half way to being a microkernel now. I can personally attest that writing a user-space file system (eg in FUSE) is vastly easier and quicker than hacking (let alone writing) a filesystem in the kernel.

The evidence is on my side. Look at how many filesystems Linux supports, and count how many are in FUSE, versus how many are in the kernel.

It can't be THAT hard

[Rik Sweeney]

Surely you only need to know a bunch of C keywords and you should be set. Here's the bunch I know

malloc
free
<<
>>
++
--
That star thingy I see every now and again.

I might have a look at this so called complicated kernel later :)

Re:It can't be THAT hard

[howlingmadhowie]

he's heard of 'free'! quick! someone offer him a job!

Re:It can't be THAT hard

[joss]

at mozilla

Wisdumb

[stonecypher]

It's no picnic to become a major anything. Major people are people who have differentiated themselves from minor people. The means by which they've done that is to do something that's more difficult, which the other people cannot do. This is a tautology masquerading as wisdom.

Batman sometimes has an ego

[Anonymous Coward]

Actually, the parent was slightly overgenerous, because he mentioned only the case of Batman being a wholly nice guy (implied).

The bigger problem occurs when Batman is a regular human with a large ego and a dislike for NIH code (or worse, Not Invented By Me code), rather than an objective engineer fully willing to accept that another developer has come up with something better.

Although we heard about one high-profile case recently where this happened, with an outcome that was more political than based on engineering merits, in a project that receives patches from thousands of developers on every release this must be happening *A LOT*. Developers with egos are fairly common after all.

It's a sad indictment of people, and while "F/OSS can take care of itself" is a common response, it doesn't really address the fact that some good ideas are being lost or marginalized by lead developers' occasional small-mindedness, and that as a result, overall kernel progress is less good than it might be. "Stability" is a very worthwhile goal and can be a good reason for rejecting a contribution, but sometimes the same word is used to hide a very ugly personal failing of the assessor.

I don't get it

[papabob]

Can you explain your point of view with a car analogy, please?

No problemo.

[fahrbot-bot]

Similarly, for a new developer, it will take time before people start recognising the name and start trusting the developer to do the right things.

Simply Photoshop yourself into a few choice picts with Linus and start blathering on about "spin locks" or some such stuff...

and remember

[Fallen Andy]

to pick up both the knife and the fork...

Gee wiz

[Abcd1234]

So the Linux kernel is developed like every major software project that's ever existed.

How completely and utterly unremarkable.