Slashdot Mirror
Linus Torvalds No Longer Knows the Whole Linux Kernel and That's OK (eweek.com)
darthcamaro writes: In a wide-ranging conversation at the Open Source Summit, Linus Torvalds admitted that he no longer knows everything that's in LInux. "Nobody knows the whole kernel anymore," Torvalds said. "Having looked at patches for many years, I know the big picture of all the areas in the kernel and I can look at a patch and know if it's right or wrong." Overall, he emphasized that being open source has enabled Linux to attract new developers that can pick up code and maintain all the various systems in Linux. In his view, the only way to deal with complexity is to be open. "When you have complexity you can't manage it in a closed environment, you need to have the people that actually find problems and give them the ability to get involved and help you to fix them," Torvalds said. "It's a complicated world and the only way to deal with complexity is the open exchange of ideas."
and for that sin, every Linux nerd everywhere deserves all the swirlies they get
He probably just has too many search tools, so now he's more stupider.
Winning
Winning
I'll bite the flamebait regarding monolithic vs micro kernels.
The assertion that Torvalds not knowing all of what in Linux simply means that those subsystems were logically delegated out. In a microkernel, those same subsystem functions would still be done by a different group, except they wouldn't be in kernel.
Simply because a kernel is "monolithic" doesn't mean that there aren't subsystems that your primary kernel architect needs to fully understand. For example, a video driver. That's very specific knowledge that you wouldn't expect Torvalds to really need to know.
In a lot of ways Torvalds has moved on to more of a project manager/architect role than just a straight code hacker.
Linus steered clear of toxic community issues and the interviewer softballed him on it, or actually completely glossed over it. Can't see that as a good thing, it looks a lot like the ostrich defence.
When all you have is a hammer, every problem starts to look like a thumb.
the real OS is systemD. PS: I win the "first systemD comment" contest.
"and that's okay"
I agree in this case. However. Stop trying to tell me how to feel and think, you cunt. Your job is to report.
And that's how backdoors can be slipped into the Kernel by the big bad guys who are pretending to be fixing something or updating its drivers.
Complexity (in software) is indeed the enemy of security.
"When you have complexity you can't manage it in a closed environment"
Try working in a manufacturing environment some time, Linus, because we manage the complexity all the time. For example, solar panels - HUGE amounts of detail you need to pay attention to (even one bad solder joint destroys a panel during lamination) and yet we manage this all the time, with all of our documentation very much closed off to the outside world. Hell, we even manage our constantly-changing crew, and there's not much of a problem there, either.
You just can't manage the complexity it because you lost 100% control. Admit it. Just like you lose control of your mouth.
Monolithic is a problem if you don't update subsystems and there's inertia to push out a new kernel version for that reason, anything else is whining.
then disclose it.
"Performance is not really doubling every two years and that's good," Torvalds said. "It means we'll maybe go back to the time when you cared more about performance on the software side and you had to be more careful and couldn't just rely on hardware getting better."
He's wrong: it means we'll just get slower and slower software because hardly anyone knows how to do anything besides paste libraries together.
"First they came for the slanderers and i said nothing."
Gates and his "640k ought to be enough for anybody" proved he didn't know what was happening within his OS.
Slashdot, fix the reply notifications... You won't get away with it...
So when a microkernel exists that has a reasonable amount of usage and performance Iâ(TM)ll listen to the microkernel whiners.
You donâ(TM)t have jack and shit... and jack left town.
...Linux's foot is in the grave. Don't @ me.
Mach. Good enough?
Circumcision is child abuse.
There is something in-between.
... I'm not talking about a few years after stabilizing, I'm talking as soon as the monster code base called the kernel no longer depends on them. Forget that there are external modules build from DKMS other other tools that need to handle different versions. And we're not talking about massive APIs, we're talking about things that support name changes.
I love microkernels and have written a few for personal entertainment. I have even been following Fuschia OS's development quite closely... though I can't believe anyone would make something that shitty from the ground up in 2018.
A more or less monolithic with a pluggable and stable ABI and API is often the best of both worlds. Add some form of module signing and code review process and we're in a good place.
After all these years, there has been ample opportunity to optimize microprocessor design to make a better microkernel CPU. There is generally just too much cost involved in the constant context swaps for a desktop OS. Remember that we spent many years trying to trim the fat between application and graphics subsystems. Even today, it's pretty simply to almost devastate the performance of a micro kernel with a massive amount of disk I/O operations.
Last month, I wrote a new Linux kernel module. I needed to implement the Cisco Discovery Protocol on Linux in a less stupid way than it's been done until now. I believe I achieved doing it equally dumb from an opposite direction. This is because the Linux Kernel is an enormous disaster of crap on crap.
Let's talk about something relatively simple that should have been ripped out and entirely moved out of the kernel a LONG time ago... the network stack.
The Linux network stack is amazingly fast and should be given credit for being that way.
It's also a cesspool of shit code from almost end to end. Probably the most important piece of code in the Linux kernel is sk_buff. Oh... sk_buff, you are the biggest, ugliest and shittiest piece of code in the entire world. I mean... you're a buffer... a kinda sorta reference counted buffer which never really gets deleted...except when you do. You're a huge chunk of trash code that looks like you were designed by a drug fuddled preschooler that wanted to try daddy's C compiler while on a series trip.
sk_buff is probably the most critical piece of code to keep the documentation up to date on. This is the code which makes things like kernel panics in the weirdest ways when even the slightest thing goes wrong. And yet, after 27 years of Linux, you guys still can't seem to stabilize the API for the frigging network buffer!!!
Then let's consider procfs...
So... procfs is basically the ability for a user mode application to access a file and it calls procedures for reads and writes... well all files in the kernel work that way, but procfs is a bit special, it was meant to be informative and work in a printf'ish kinda way.
procfs for the most part is nothing more than, open, read, write, seek, close. It doesn't need anything other than that. It's a simple random access file stream.
Somehow, the API is still changing like mad... and worse the transition mechanisms that support the newer APIs are actually getting removed from the kernel
I was looking into adding a new address family to the kernel for CDP... and I tried looking at other code for a good example. I absolutely refuse to make pull requests to the Linux kernel unless it's a bug fix (and deleting the whole tree and starting over doesn't count). I believe that tools like DKMS should point to git repositories and download drivers and build them against the kernel. CDP does not need to be part of the mainstream kernel. It's a tool.
Well, as I said, I was looking into it. And after this many years, because of the absolute shitty state of the kernel... I'd at least have to register AF_CDP and PF_CDP somewhere so that I could have my very own protocol number. And for the most part, that would prob
Doesn't matter if it's open. If one person (or one entity in AI) can't absorb all of the complexity, it becomes increasingly unlikely that one person/entity will identify vulnerabilities and exploits that arise by chance or by malicious intent.
Well lets hope one day you're lucky enough to be one of the 30 people who get to write software for it
Sadly, this probably means more shit spaghetti code than ever before. Another good reason for Google to ditch this in favor of Fuchsia / Zircon.
After all these years, there has been ample opportunity to optimize microprocessor design to make a better microkernel CPU. There is generally just too much cost involved in the constant context swaps for a desktop OS
The reason that microkernels suck has nothing to do with context swap inefficiencies. The biggest problem is trying to maintain a synchronised state between the different tasks across different memory protection areas.
A simple example is a file system. Imagine a dozen different tasks, all working on the same file system, As soon as one task makes a change (say, delete a file), all other tasks are working with an outdated snapshot of the file system state. Unless notified of the change, this will lead to corruption. And notifying all tasks of every little change would be hugely inefficient, not just because of all the overhead of sending the messages, but also because tasks would need to be made with frequent check points. In the end, it would do nothing to simplify the overall system, because you'd basically be implementing a virtual shared memory, and you'd have to deal with exactly the same issues as with a real shared memory.
The tradition solution in microkernels is to have a single task running the file system. This may be a practical idea on some small scale single user systems, but it's totally unacceptable on a larger server, say a big web server with a few hundred simultaneous connections.
I love microkernels and have written a few for personal entertainment. I have even been following Fuschia OS's development quite closely... though I can't believe anyone would make something that shitty from the ground up in 2018.
I haven't looked at it, but you saying that doesn't surprise me. And it's an important hint.
How does this work? Someone up and says "I want a better OS". I myself have thought this on many occasions. And true to form, someone with half a clue but no experience just goes and builds it. And because they didn't look very closely at what had come before, they end up with something shitty. linux is a case in point, and look how popular it is.
After all these years, there has been ample opportunity to optimize microprocessor design to make a better microkernel CPU. There is generally just too much cost involved in the constant context swaps for a desktop OS. Remember that we spent many years trying to trim the fat between application and graphics subsystems. Even today, it's pretty simply to almost devastate the performance of a micro kernel with a massive amount of disk I/O operations.
There's a long way between the hardware and the software guys. The hardware guys get idiotic ideas like buying a third party company for eight milliard dollars so they can patch up certain specific software's failings in hardware. Turns out that's a lot of money wasted. The software guys just want to fix it all in software.
Yeah, there's been lots and lots of opportunity, but somehow the few in the rigth places didn't manage to even try.
Shit, you could have had a dual core 10 Watt CPU, MIPS64 laptop back in 2000 or so, but nobody thought to build one. Well, I did but I had neither the money nor the contacts to make it happen. You could have had something good but all anybody did was build more laptops for the suitwearing crowd. What do they know of good software? Look at what they're using!
The same with servers. "Real servers are headless", but these days they have entire secondary computers built very deeply into the system to emulate that head over the network. That's just bad and wrong on so many levels. But it's what "everyone" does. Reasons why left as an exercise.
So no, this is what you get when there's nobody with serious vision around: A lot of derping. There has been no meaningful pressure from the market to make microkernels work well in the hardware, so all that opportunity, wasted.
The Linux network stack is amazingly fast and should be given credit for being that way.
Not the only one that's nice and fast, mind.
Then there's the fun we call "finding shit in the Linux headers".
Unix traditionally came with very good documentation. Something that didn't quite carry over into the linux (and gnu) world.
And finally... the programming language thing.
So you'd make a "better" C just to make a "better" Unix in? I don't think that flies. Again, reasons why left as an exercise.
I suppose I could go on and on. But let's face it... sooner or later things like Redox will get proper support.
Because having a language with a CCoC is a neat feature to have. Nevermind its dependencies on C++ and the inability to bootstrap the compiler on other hardware. It's the CCoC that makes all the difference.
We can't keep depending on Linux. It's just ancient technology which isn't even proven... it's more like "what else is there?"
It's very well proven. It's a giant pile of poo but it's been in use so widely that people feel comfy with it. Same with windows, by the by. That is an even bigger pile of poo and even more popular.
But if your best answer to "what else is there?" is an OS with its one and only reason to be promotion of a shitty language with a large ideological chip on its shoulder, you haven't looked around very much.
I semi-agree with a lot of your rant until you went off on C. Writing secure code has very little to do with the language and a lot to do with the programmer. I've seen code in python, java, c#, c++, typescript just this year that you can drive a bus through .. simply because the programmers haven't got a clue about security. Writing secure code is about understanding all possible interactions and making sure you code still works as designed. It's then about ensuring that "works as designed" is the safe thing to do
You better be trolling.
Circumcision is child abuse.
Considering the mess that is the whole embedded clusterf**k
What did Linus say when you used your obviously elite dev skills to rewrite sk_buff elegantly and submitted it to him and his team for review?
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
You have no idea what a microkernel is right?
You better be trolling.
Mach (and BSD) are used in Apple's various OSes. Does that qualify as "reasonable amount of usage and performance" of the original comment?
Also, L4:
* https://en.wikipedia.org/wiki/L4_microkernel_family#Commercial_deployment
Linux is too complex. Let's start over.
Of course I do. The key feature of a microkernel is that it uses different access spaces. with protection barriers between them. In contrast, a monolithic kernel has shared access spaces, so different tasks can modify the same data and keep common state synchronized.
That's the only really important difference.
After all these years, there has been ample opportunity to optimize microprocessor design to make a better microkernel CPU. There is generally just too much cost involved in the constant context swaps for a desktop OS
The reason that microkernels suck has nothing to do with context swap inefficiencies. The biggest problem is trying to maintain a synchronised state between the different tasks across different memory protection areas.
A simple example is a file system. Imagine a dozen different tasks, all working on the same file system, As soon as one task makes a change (say, delete a file), all other tasks are working with an outdated snapshot of the file system state.
That example is silly. "Outdated snapshot?" You don't keep copies of state, you keep only one state and query that at every use. That way, nothing to 'synchronize', no "snapshots" to 'get old'.
Of course, now you have the cost of querying that filesystem state. In the monolithic kernel, this cost is no higher than looking at some local snapshot, which is the reason no local snapshots are ever made. Linux tries to do "zero copying", and a snapshot is a copy that simply takes too much time. And of course, a snapshot provides zero benefit for its substantial cost in the monolithic kernel.
In a microkernel, looking at some other tasks state may cost more - and that is precisely the context swaps you claimed wasn't the problem. Looking at some other tasks state involves two context switches - to that other task and back to the current. Just like when userspace makes a system call into the kernel. Only now you get this overhead inside the microkernel too - whenever a task boundary has to be crossed.
Call it "messages" if you like, but that is a high-level construct. Passing a message between two tasks really is a couple of context switches, on the low level of things. If you want to talk about kernel architectures, you should understand that much.
The tradition solution in microkernels is to have a single task running the file system. This may be a practical idea on some small scale single user systems, but it's totally unacceptable on a larger server, say a big web server with a few hundred simultaneous connections.
Huh? A file system had better be in a single memory context (a "task"), so it won't be slowed down by internal context switching. To be efficient on a large server, it obviously has to support many parallel threads of execution - but should still be a single memory context. For these reasons, a microkernel with only one thread per memory context won't be useful for big servers. (Either losses to internal context switching, or single-thread inefficiency.) But if you design a kernel, you can of course have a multithreaded microkernel. If each task is big enough - such as one task being the entire file system - then maybe you can have an efficient microkernel with a little more internal protection than a monolithic kernel. But only a little. If the file system task crashes, the other tasks may be isolated from that disaster. But your server is dead for all practical purposes anyway - so where is the micorkernel advantage?
And additionally - a webserver with a few hundred simultaneous connections is nowhere near "big", and it will work just fine with a single-threaded file system. "Big" is more than that!
I think most OS's that have a long history also have basically grown their kernels into a spaghetti mess. Well especially desktop OS which have certainly bloated up over the years. Even mobile OS to some extent require better hardware just to maintain a parallel level of performance. Its not surprising that many stick with older kernels and OS releases because of this performance gain. Yes, you lose the advantage of new features and hardware support. But older hardware likes older kernels.
The Linux kernel has consisted of millions of lines of code for many years. It is doubtful that anyone can understand, really understand, all the ins and outs of more than a few tens of thousands of lines of code.
Looking at some other tasks state involves two context switches - to that other task and back to the current.
Looking at another task's state means making a copy of the state, a snapshot if you will. As soon as you make a copy, you now have two versions. Right after you make a copy, before the 2nd task can even examine it, the original state can change again.
To be efficient on a large server, it obviously has to support many parallel threads of execution - but should still be a single memory context
Yes, that's very efficient. That's what a monolithic kernel does.
this guy has no idea what he's talking about. usually you would have one server managing the filesystem that would manage filesystem state, no need to synchronize data. the big problem has long been the context switches because internally the kernel has to context switch between different kernel subsystems, when the kernel is running, to perform its various tasks, whereas with monolithic, it does not need to.
mvs makes the linux kernel look like a phone app - tell ibm how closed development doesn't work
Wow. I don't want to hear why you don't/can't write a few simple string or collection functions in C.
>I love microkernels and have written a few for personal entertainment.
Fucking nerd. You should try getting laid once in a while.
Systemd is the human expression of perpetual and covert alien telepathic pressure
"It's a complicated world and the only way to deal with complexity is the open exchange of ideas."
This is a quote that Mr. Torvalds should be known for, forever. It applies to much more than just software.
It is pitch black. You are likely to be eaten by a grue.
C++ has some of the features you mention, such as vectors. Proposals to introduce C++ to the kernel are quickly rejected.
Kernel code is bound to be difficult. I've often thought, for drivers, maybe a compatibility layer that presents a nice stable API can be provided for people to write drivers for (not forced, the internal API can also be exposed), but as another option.This would avoid cluttering kernel internals with backwards compatability stuff but would provide a nice stable API for drivers to use.
If people did that it would certainly address a lot of their concerns about C without needing a new language
Of course you can always roll your own, but is it preferable to have a million different custom solutions in place of 1 or 2 standard supported features?
L4, QNX are probably the top 2. And performance is tied to old hardware limitations and models, which make context switches expensive. A cpu better suited to micro-kernels, like the Mill could bring about a different story. If a context switch cost 5-10 cycles instead of hundreds, all of a sudden the micro-kernel architecture looks a lot better.