Linux Getting Extensive x86 Assembly Code Refresh

jones_supa writes: A massive x86 assembly code spring cleaning has been done in a pull request that is to end up in Linux 4.1. The developers have tried testing the code on many different x86 boxes, but there's risk of regression when exposing the code to many more systems in the days and weeks ahead. That being said, the list of improvements is excellent. There are over 100 separate cleanups, restructuring changes, speedups and fixes in the x86 system call, IRQ, trap and other entry code, part of a heroic effort to deobfuscate a decade old spaghetti assembly code and its C code dependencies.

Comments

Debt

[Ducho_CWB]

Technical Debt haunts you.

Re:Debt

[Darinbob]

Yes, if it weren't for the idea that I could change jobs if I needed to, I'd have been full of hopless dread at just about most places I've worked. The sad thing is, in some places the majority of technical debt is creating in the first year of the company's existence, during the hurry-up-already startup phase.

Re:Debt

[Ryanrule]

Thats not really the debt though. The debt is when you get a giant wad of funding and dont take the time to greenfield your cludge app.

Re:Debt

[swillden]

Thats not really the debt though. The debt is when you get a giant wad of funding and dont take the time to greenfield your cludge app.

No, that is debt, and in many cases it is the biggest single source of debt, as Darinbob said. Failing to rewrite your kludge app is just failing to pay down your debt. Whether or not the moment you get a giant wad of funding is the right time to do that depends on the context.

Re:Debt

[Anonymous Coward]

I agree completely: I spent 5 years trying to fix an application that was built over the course of a few months.

Re: Debt

[Anonymous Coward]

Fixing that sort of thing is my business model. ... I haven't yet figured out how to explain it, though

Cruft

[Anonymous Coward]

I wonder how much cruft there is in Windows or OSX.

Re:Cruft

[Anonymous Coward]

Proof? I can't find such posts in Mark's blog.

Re:Cruft

[benjymouse]

For some time now, Mark Russinovich at Microsoft has been talking about just how bad the Windows kernel was/is in his blog.

I think you are confused. It was not Mark Russinovich, but rather Linus Torvalds, and he was talking about the Linux kernel, not the Windows kernel.

"I mean, sometimes it's a bit sad that we are definitely not the streamlined, small, hyper-efficient kernel that I envisioned 15 years ago...The kernel is huge and bloated, and our icache footprint is scary. I mean, there is no question about that. And whenever we add a new feature, it only gets worse."

Glad I could help.

Re:Cruft

[Anonymous Coward]

With their install size, do you really need to wonder?

If linux and other open source projects keep the issue in a spotlight, then maybe the clumsy closed source companies might be compelled to follow suit.

Open source does it better. GUI OS fits on a single floppy disk:
http://menuetos.net
http://kolibrios.org

Re:Cruft

[Anonymous Coward]

That Torvalds guy is obviously a M$ $hill to say something like that.

obfuscation mk2

[Anonymous Coward]

Don't worry, the obfuscation is being replaced by kdbus and its metadata

Re:obfuscation mk2

[ThePhilips]

Not replaced, you dummy.

Elevated to a new level.

Re:obfuscation mk2

[Anonymous Coward]

nooooooo

Re:obfuscation mk2

[X0563511]

I find your signature incredibly relevant to your post.

Can we be sure there are no exploits?

[Anonymous Coward]

We live in interesting times these days. With a changeset so big, and involving assembly code that isn't as easy to understand as C code, how can we really be sure that no exploits have been introduced? How extensively have these changes been reviewed to ensure there are no exploits or potential exploits being sneaked in?

Re:Can we be sure there are no exploits?

[Lunix Nutcase]

The only way to truly understand C code is to read the disassembly. Otherwise you are only assuming what the compiler is emitting.

Re:Can we be sure there are no exploits?

[Bing Tsher E]

That does not fit in with the concepts of 'abstraction' that are fashionable. No 'coder' should really need to understand the actual code the hardware runs. It's all rife with gotos and other icky things. Pointers (physical memory addresses) and stuff get revealed.

No! Mommy! Make the bad hardware go away!

Re:Can we be sure there are no exploits?

[cheater512]

It isn't really hard? Big job = lots of people involved.

Re:Can we be sure there are no exploits?

[Anonymous Coward]

It's not even hardware, assembly language is just yet another level of abstraction. The difference is that assemblers tend to be a bit more predictable to what it generates from the supplied source.

Re: Can we be sure there are no exploits?

[Anonymous Coward]

Even the latest java magazine talk about diagnostic and optimization of jited asm code. Ignore the hardware if you want to make less money....

Re:Can we be sure there are no exploits?

[EzInKy]

x86 assembly is not that difficult to understand. You MOV, CMP, and JMP to memory addresses according to the values in the processors registors. My guess is that exponetially more exploits are introduced when what you MOV and CMP and where you JMP to is abstracted away than when it is not.

Re:Can we be sure there are no exploits?

[Anonymous Coward]

There are way more assembly commands than those.

Re:Can we be sure there are no exploits?

[m.dillon]

Well... basic procedures using only MOV/CMP/JMP is not something that even linux really needs to code in assembly. What is being talked about here is primarily the trap, exception, syscall, signal trampoline, and interrupt entry and exit mechanisms. Also thread switch code can get pretty complex because there is a lot more hardware state involved than just the basic register set. When you start having to deal with SWAPGS and MSR registers, you've really gone down the rabbit hole.

-Matt

Re:Can we be sure there are no exploits?

[TechyImmigrant]

I program in logic gates you insensitive clod?

Re:Can we be sure there are no exploits?

[EzInKy]

Of course there are. The point is that when you "abstract" them away you hide them.

Re:Can we be sure there are no exploits?

[Rigel47]

I do enjoy how we tech people less-than-quietly display our scorn of all the varying fields in IT that aren't our own. Because, hah, stupid Javascript developer, couldn't explain northbridge from south! Or what DMA is! He probably doesn't know a single assembly instruction! Clearly an inferior being.

Javascript guy meanwhile regards the C guy as a primitive, bearded man from the hills who labors all day on some stupid library that is ten layers of stack between that mortar and pestle and the awesome browser-art he's creating.

Systems administrators wish everyone would run off and die because they are all irritating, stupid whiners.

DBAs are just smug because nobody else understands their schemas and, hey, this is where it all happens.

Networking would rather be back below the hold, No, the network isn't broken, your app is buggy or the stupid website you're trying to load is just slow..

Help desk guys meanwhile consider themselves the cocks of the walk because, generally, their camaraderie and opportunity to interact with more regular people means their souls haven't been totally crushed.

Re:Can we be sure there are no exploits?

[Anonymous Coward]

Logic is also another abstraction. Come back when you care about signal levels.

Re:Can we be sure there are no exploits?

[Anonymous Coward]

The only way to truly understand C code is to read the disassembly. Otherwise you are only assuming what the compiler is emitting.

The only way to truly understand C code is to read the opcodes. Otherwise you are only assuming what the assembler is emitting.

Re:Can we be sure there are no exploits?

[TechyImmigrant]

Signal levels are another abstraction. Come back when you care about electron band gaps.

Re:Can we be sure there are no exploits?

[Anonymous Coward]

Jokes on you. I'm building a VM language in ASM.js and VM bytecode. I get to deal with memory management and multiprocessing issues that are unique to ASM.js. Since the code compiles to opcode which can then be output as "native" ASM.js (which is then mostly compiled to machine code by browsers) or it can run blazing fast on the C VM implementation with its (native) JiT optimizations. Since the platform is meant to be an extremely portable high level and systems level language I'm currently implementing the VM and compiler in its own bytecode, whith the goal of also outputting as x86, ARM, ASM.js, JavaVM bytecode, and Perl6 bytecode.

That means I have to deal with OS level things like spin locks, memory paging, etc. However, it also gives me the chance to radically change the security model such that all code is sandboxed, ran in a VM as bytecode and only compiled and cahed as "native" to the machine / platform / browser if trusted.

Networking abstraction happens to be entire point of the project. Instead of everything being a file, as in UNIX, on my system everything is a network connection. Even Local "native" connections can be seamlessly wrapped as TCP or UDP connections at the OS level, so that IPC is transparent and computing can become distributed without modifying the code. Yes, that means your business can harness the CPU of its visitors to scale availability along with traffic.

Silly single platform devs, still not harnessing every tier of development like plebians. The future is distributed, and you'll get on my level eventually.

Re:Can we be sure there are no exploits?

[Anonymous Coward]

That was the most single most hipster post I've read on /. all year.

Re:Can we be sure there are no exploits?

[Anonymous Coward]

Because a physics model isn't an abstraction?

Re:Can we be sure there are no exploits?

[Anonymous Coward]

You know how management doesn't actually need to know the jobs they're managing? Because, well, people are people everywhere and how well they do the jobs they do can be adequately summed up by a couple of sheets full of numbers you expect your underlings to regularly hand to you. It's easy and pays well. And, in fact, that is how middle management got its well-deserved bad reputation: By treating people as "black boxes" that poop productivity.

Computer technology is full of things that we treat as black boxes. And, on a certain level, we absolutely have to, because otherwise there are simply too many details to all keep track of. So we abstract them out. But at another level, our abstractions are rather poor, and so our use of the technology is likewise rather poor. Mix with widespread ignorance, a bit of ego, some groupthink, and you get what you describe.

I really do wish every "web developer", or at least the ones that deal with backends, writing interfaces for "experience designers" to build upon, would at least understand relational algebra and what ACID embodies, as it'd mean nobody would deploy mysql or mariadb for any new project. I'd also like web developers to understand just how much of the design that props up PHP is poor to excreable, so nobody serious would contemplate deploying that any longer. I'd like people to understand that spending too many cycles on generating webpages, then spending even more resources on caching so the generation doesn't hurt so much, is only a band-aid at best. Similarly I'd wish they would understand that javascript is not free, that there are real costs to spending other people's cycles, even if they don't mind because they don't see it and hey, their employer pays for the latest desktop AND laptop anyhow, so they'd not be so free with slapping on javascript, even reinventing basic features like "links" brokenly, badly, and with gobs of unnecessary and very very slow javascript.

I think these are valid complaints that directly touch on web developers' chosen field, moreso than on tangentially related fields. I could probably think up similar complaints about other fields. It's just that there's currently a lot of web development going on, most of it very very poor indeed. Including some really big and hip names. The point is: There is quite a lot of room, oodles of room, big blue rooms full of room, for improvement here, and you don't need to sneer on the neighbours to find it.

Re:Can we be sure there are no exploits?

[smittyoneeach]

Probably not without the right framework, though.

Re:Can we be sure there are no exploits?

[Anonymous Coward]

Yeah well, I'm using transistor glitches.

Re:Can we be sure there are no exploits?

[RabidReindeer]

Chorus:

"Well, It's Simple! All You Have To Do Is..."

Re:Can we be sure there are no exploits?

[TheRaven64]

The first instruction in the Intel Architecture Reference (Part 2: Instruction Set Reference) is AAA, which is named after the noise made by people forced to read x86 assembly. It is short for 'ASCII Adjust After Addition' (yes, that should be AAAA, but that would be too consistent for Intel). This instruction exists to convert the result of binary addition into the result of the corresponding BCD addition.

Or, to put it in simpler terms: Anyone who thinks x86 assembly is not that difficult to understand is certifiably batshit insane.

Re:Can we be sure there are no exploits?

[flowsnake]

The only way to truly understand C is to read the CPU circuit design on which the application will run. Otherwise you are only assuming what the CPU will actually do with the opcodes generated during assembly. See 6502 opcode 8B (XAA)

Re:Can we be sure there are no exploits?

[RabidReindeer]

Yeah, yeah, All You Have To Do, yada yada.

Unlike the old dinosaur mainframes, modern microprocessors run internal pipe lines and what instructions follow what other instructions can make a big different.

Aside from that, one reason that raw assembly is no longer the cult favorite it used to be is that the average C statement is probably going to produce 5-10 machine-language instructions. More instructions means more work to create or replace them, more places for potential bugs, and less flexibility to make high-level architectural changes.

Re:Can we be sure there are no exploits?

[EzInKy]

Why are you bringing up such a high level abstractive language as C in this discussion of assembly language?

Re:Can we be sure there are no exploits?

[EzInKy]

So knowing that you seem to have some understanding of it. Interesting!

Re:Can we be sure there are no exploits?

[houghi]

And all this fingerpointing leads to confused users (who are the worst as they are demanding it to just work).
I had an issue with displaying things on Windowmaker, while it worked on XFCE.
Windowmaker pointed to X.org. They pointed to NVidia who pointed to Windowmaker.

"It wasn't me." was all the feedback I got. No way they would talk to each other, because their code was perfect.

As if two ISP's only check their network so the connection problem MUST be with the other ISP and nobody checks the patch cable between the two of them, because that is not our problem. (Yes, that really happened. Yes, it was a patch cable. Yes it took weveral hours before somebody pushed it back into place.)

Re:Can we be sure there are no exploits?

[Anonymous Coward]

The only way to truly understand C is to implement the CPU design yourself, along with most of the remaining hardware. Otherwise you are only assuming that the CPU is actually implemented in conformance with the design documents you're looking at.

Re:Can we be sure there are no exploits?

[Etcetera]

Because a physics model isn't an abstraction?

Obligatory https://xkcd.com/435/

Re:Can we be sure there are no exploits?

[RabidReindeer]

Because, among other things, compilers (such as C) can automatically generate pipeline-friendly machine language. All that extra bookkeeping is trivial for an automated process.

And a compiler can re-optimize the entire module every time they compile.

Hand-optimized assembly lost it when IBM introduced Pascal/VS. I COULD optimize code as good or better than it did, but considering that it was doing large-scale register reassignments, the equivalent amount of hand optimization for just one minor source change/recompile would have taken me a full day or more.

And my boss wouldn't have tolerated that.

Re:Can we be sure there are no exploits?

[Anonymous Coward]

I do enjoy how we tech people less-than-quietly display our scorn of all the varying fields in IT that aren't our own. Because, hah, stupid Javascript developer, couldn't explain northbridge from south! Or what DMA is! He probably doesn't know a single assembly instruction! Clearly an inferior being.

Javascript guy meanwhile regards the C guy as a primitive, bearded man from the hills who labors all day on some stupid library that is ten layers of stack between that mortar and pestle and the awesome browser-art he's creating.

Systems administrators wish everyone would run off and die because they are all irritating, stupid whiners.

DBAs are just smug because nobody else understands their schemas and, hey, this is where it all happens.

  Networking would rather be back below the hold, No, the network isn't broken, your app is buggy or the stupid website you're trying to load is just slow..

Help desk guys meanwhile consider themselves the cocks of the walk because, generally, their camaraderie and opportunity to interact with more regular people means their souls haven't been totally crushed.

For my money, it's between SysAdmins and Developers nowadays, mostly. IT is mostly concerned with desktops and offices, and there's been a general reduction in thick/native apps across the board since the various web 1.0/2.0/3.0 paradigms came on board. Networking is important, but its impact on usability is usually a function of bandwidth limitations, packet loss, reliability when a device goes down, and live re-routing decisions being made. The first three are (ideally) invisible.

DBAs may have a lot of sway, depending on your shop.

SysAdmins/SysEngineers, though, have to deal heavily with operational performance and scaling. Often, they're interacting with Developers who are "that Javascript guy meanwhile regards the C guy as a primitive, bearded man from the hills who labors all day on some stupid library that is ten layers of stack between that mortar and pestle and the awesome browser-art he's creating"

We have to keep the web stack running because Developers are operating at such a high level of abstraction that they have no idea, and have never been training to even think of, the underlying load on the systems.

If you go into Development *from* a SysAdmin Linux base, sure.... you'll keep these considerations in mind intuitively. If you're fresh out of school and have never had to think about it you are the bane of my existence.

Re:Can we be sure there are no exploits?

[Cajun Hell]

You have to admit, though, those other guys really are idiots.

Re:Can we be sure there are no exploits?

[Anonymous Coward]

SHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH the kid is reinventing Plan9, let him work.

It keeps them off the streets and off of my lawn.

Re:Can we be sure there are no exploits?

[Enry]

In general, SAs are a dying breed. Yes there's some still around and there will be in the future, but many of them are being folded into 'DevOps' which is really developers that use short-term instances to do the job they need. Since each single instance does one job, does it well, then gets destroyed there's no need to have someone handle the long term care and feeding. Since you block everything in a VPC and you don't have users connecting directly there's no security or user administration issues. Storage? Networking? Logging? Pfft, whatever.

It's fairly easy to see when you have developers that demand the latest and greatest versions of RoR/NodeJS/MongoDB/PostgreSQL/whatever while SAs are tearing their hair out trying to figure out how to install, let alone support, multiple versions of software distributed via github for developers that may themselves be coding against different versions of the above. So IT is seen as inflexible and AWS and the cloud offered developers a way to skirt around the surly SAs and build whatever sandbox they wanted with the versions of software they wanted and in a faster time than the SAs could reply to a ticket.

As a former SA, this is going to go on for a bit but given how the cycles of computing go SAs sill be back in fashion in 10-15 years. Until then, I hope you know Ruby.

Re:Can we be sure there are no exploits?

[Anonymous Coward]

BRO, I WORK IN TECH SUPPORT! Users are whinners, whine if you talk to them or if you dont.

Think I should work digging graves, those clients complain A WHOLE LOT LESS

Re:Can we be sure there are no exploits?

[jeremyp]

Get yourself a 65CE02. Every opcode is well defined and supported. 8B is STY abs,X

Re:Can we be sure there are no exploits?

[TechyImmigrant]

You were supposed to reply with a lower level abstraction. Please keep up.

Re:Can we be sure there are no exploits?

[Anonymous Coward]

This is not unique to C. The only way to truly understand the runtime generated by any high level language is to read the disassembly - and the higher (more abstracted) the language the more this is the case. However reading the disassembly of the typical bloat library, whatever the dev environment, is [a] really hard work, and [b] usually illegal under DMCA unless you can prove it's for the narrow specific purpose of 'interoperability'. So all the more credit to open source and the folks doing this work on the Linux kernel.

Should be micro kernel

[Billly Gates]

Linux has been obsolete since introducing assembly code. Minux touches the hardware in just 100 lines of code and macosx is a micro kernel as well.

It is 2015 and not 1985

Re:Should be micro kernel

[Anonymous Coward]

Sometimes theory doesn't live up to reality.

Re:Should be micro kernel

[Anonymous Coward]

What does the presence of assembly code have at all to do with being a microkernel? Microkernels can be written entirely in assembly and have been.

Re:Should be micro kernel

[Anonymous Coward]

Linux is being de-obsoleted slowly. Anyone interested in that is sending heroic pull requests. When Linux Vista comes, who is to say they haven't refactored the whole driver architecture so it runs in "driverspace" and it only kills itself when it segfaults?

Re:Should be micro kernel

[Lunix Nutcase]

XNU contains thousands upon thousands of lines of assembly code for both x86 and ARM. Being a microkernel does not preclude the presence of assembly code

Re:Should be micro kernel

[EmeraldBot]

Linux has been obsolete since introducing assembly code. Minux touches the hardware in just 100 lines of code and macosx is a micro kernel as well.

It is 2015 and not 1985

Better not use Minix then, because it also has assembly.

Re:Should be micro kernel

[stox]

All C code ends up as assembler, anyway. It just gets deleted, most of the time, before you see it.

Re:Should be micro kernel

[Anonymous Coward]

I've never seen a true microkernel that has the performance of a monolithic kernel. Nobody wants to buy a new computer and drag it down to a crawl. The people who want microkernel features are getting them with no issue. What's the point? It's not like my system has dumped core and full-on bombed out since about 2006. It's not like I have problems finding the features that are supported by microkernel OSs. It's not even like most "microkernel" operating systems perform all of their system calls through pipe or socket interfaces in reality. So, what's the point?

AC is correct. Great in theory, but in *reality* some programmer just bites the bullet and makes this stuff all solid in a monolithic kernel, and the result is a faster OS. I'll take that practice over the microkernel theories any day.

Re:Should be micro kernel

[Anonymous Coward]

I'd insinuate that you're actually Andrew Tanenbaum in disguise, but he'd know how to spell MINIX.

Re:Should be micro kernel

[unixisc]

OS-X is not a microkernel - has never been. Even when NEXTSTEP was based on Mach 3.0, that too was not microkernel. Apple doesn't use any of the rules of microkernels - user space drivers, et al - in OS-X or iOS

Re:Should be micro kernel

[OzPeter]

Sometimes theory doesn't live up to reality.

Yes, I've hurd that before.

Re:Should be micro kernel

[caseih]

Just because Minux has only 100 lines of assembly doesn't mean anything about Darwin, even if Darwin has microkernel components, so your association there is a bit fallacious. Unless it's changed recently, Darwin does have microkernel (mach in fact) underpinnings, but a complete FreeBSD subsystem is grafted onto that. So if anything Darwin is a hybrid kernel. Like most real systems out there, it's not a complete microkernel system.

Re:Should be micro kernel

[OzPeter]

I've never seen a true microkernel that has the performance of a monolithic kernel. Nobody wants to buy a new computer and drag it down to a craw

Did you ever use OS-9 from Microware? (not to be confused with OS 9 from Apple)

Back in the day I ran OS-9 on a Tandy Co-Co and had a fully multi-user, pre-emptive multitasking system* running on a 6809E, 8 bit, sub 2MHz CPU. Later on I worked with a variety of industrial computers running OS-9 on 68K based systems and they worked just fine.

* I will give you that I only ever fired up the graphical desktop all of once just to see if it worked. After that I stayed in the command line.

Re:Should be micro kernel

[Lunix Nutcase]

Yep, XNU is a hybrid kernel. It has microkernel parts and monolithic kernel parts.

Re:Should be micro kernel

[Lunix Nutcase]

To add your claim that XNU does not follow any microkernel rules is simply false. XNU uses microkernel-style message passing.

Re:Should be micro kernel

[snowgirl]

I've never seen a true microkernel that has the performance of a monolithic kernel.

I've never seen a RISC processor that can match the performance of the best CISC processors. You know, nevermind the fact that tons of money has been poured into CISC processors making them faster and faster.

Sometimes, it's just a matter of where the attention has been placed.

Re:Should be micro kernel

[Anonymous Coward]

Every other paper ever written on the topic of making microkernels fast has reduced to "fake the performance of a pipe by providing the interface, and then using a shared memory segment between the processes."

That's cheating. That's wrapping microkernel functionality into a macrokernel. They second you try a cool trick like migrating a thread to another machine, the performance drops to exactly the level you would expect.

Re:Should be micro kernel

[Darinbob]

Micro kernel is not necessarily better. There was a period of time when everyone felt that way, but the fad wore off quickly when no one managed to actually create a micro kernel that ran rings around the competition. Yes, micro kernel is great for adaptability and debugging but in practice the actual consumers don't care about those features.

It's like software layers. You'll see some groups that are utterly adamant about keeping things strictly in layers, yet there are often very noticeable barriers between the layers that are inefficient both in run time and in developer productivity. For network stacks, I don't think anyone ever creates a pure layered architecture as envisioned by the OSI model.

Re: Should be micro kernel

[Anonymous Coward]

OS-9 Was spectacular! As befits the outstanding 6809. Had a long life, up to the early 2000s in deep embedded systems.

Radio Shack COCo was an ok machine too, hated the keyboard tho.

Re: Should be micro kernel

[Anonymous Coward]

Release cycle is a bit slow

Re: Should be micro kernel

[Anonymous Coward]

Streams was layered networking, excellent architecturally, died from bad performance vs. sockets.

Re:Should be micro kernel

[Guy Harris]

To add your claim that XNU does not follow any microkernel rules is simply false. XNU uses microkernel-style message passing.

XNU has system calls to allow messages to be sent between processes, including sending large amounts of data by page flipping.

It just doesn't happen to use that to implement very much of the UNIX API; it's not used to implement file I/O (that goes through ordinary BSD system calls to in-kernel file systems that are called through a BSD-style VFS) or network I/O (that goes through ordinary BSD system calls to in-kernel networking stacks that are called through a BSD-style kernel socket layer) or much of the process/thread management or VM code (that goes through ordinary system calls that end up calling Mach task, thread, and VM management calls).

It is used for communication between user processes, and for some kernel user communication, but that's the same sort of use that happens in systems with Boring Old Monolithic Kernels.

Re:Should be micro kernel

[jblues]

OS-X is not a microkernel - has never been. Even when NEXTSTEP was based on Mach 3.0, that too was not microkernel. Apple doesn't use any of the rules of microkernels - user space drivers, et al - in OS-X or iOS

As I understand it, NeXT / OSX started with a micro-kernel philosophy and then introduced some monolithic kernel concepts to address the performance bottleneck of messaging between true micro modules.

Meanwhile Linux starts as a monolithic kernel, but introduced (un)loadable modules to address maintainability and extendability.

So if we described it as a continuum with 'pure microkernel' being a '1' and pure monolithic kernel being a '10', then OSX would be something like a '3' and Linux would be a '7'.

Re:Should be micro kernel

[Anonymous Coward]

What would happen if you implemented the pipe functionality in hardware like custom registers and cache memory? Seems like you would up with something like OCCAM/Transputer processors which had at least four input and out pipes each.

Re: Should be micro kernel

[zoid.com]

The chicklet keyboard was horrible. There were a few 3rd party replacement keyboards that worked pretty well.

Re:Should be micro kernel

[Guy Harris]

As I understand it, NeXT / OSX started with a micro-kernel philosophy and then introduced some monolithic kernel concepts to address the performance bottleneck of messaging between true micro modules.

Meanwhile Linux starts as a monolithic kernel, but introduced (un)loadable modules to address maintainability and extendability.

So if we described it as a continuum with 'pure microkernel' being a '1' and pure monolithic kernel being a '10', then OSX would be something like a '3' and Linux would be a '7'.

Loadable kernel modules have nothing to do with microkernels. A truly micro microkernel wouldn't need loadable kernel modules because all the loadable functionality would run in userland; plenty of monolithic kernels have loadable kernel modules.

And OS X is a lot further from "pure microkernel" than 3. The "monolithic kernel concepts" include "running the networking stack in the kernel, standard BSD-style", "running the file systems in the kernel, standard BSD-style", and "implementing most process and VM management operations with standard system calls in the kernel".

Re:Should be micro kernel

[Anonymous Coward]

According to their own site, the kernel is not a microkernel.

http://www.microware.com/index...

Re:Should be micro kernel

[m.dillon]

Nobody does message passing for basic operations. I actually tried to asynchronize DragonFly's system calls once but it was a disaster. Too much overhead.

On a modern Intel cpu a system call runs around 60nS. If you add a message-passing layer with an optimized path to avoid thread switching that will increase to around 200-300ns. If you actually have to switch threads it increases to around 1.2uS. If you actually have to switch threads AND save/restore the FPU state now you are talking about ~2-3uS. If you have to message pass across cpus then the IPI overhead can be significant... several microseconds just for that, plus cache mastership changes.

And all of those times assume shared memory for the message contents. They're strictly the switch and management overhead.

So, basically, no operating system that is intended to run efficiently can use message-passing for basic operations. Message-passing can only be used in two situations:

(1) When you have to switch threads anyway. That is, if two processes or two threads are messaging each other. Another good example is when you schedule an interrupt thread but cannot immediately switch to it (preempt current thread). If the current thread cannot be preempted then the interrupt thread can be scheduled normally without imposing too much overhead vs the alternative.

(2) When the operation can be batched. In DragonFly we successfully use message-passing for network packets and attain very significant cpu localization benefits from it. It works because packets are batched on fast interfaces anyway. By retaining the batching all the way through the protocol stack we can effectively use message passing and spread the overhead across many packets. The improvement we get from cpu localization, particularly not having to acquire or release locks in the protocol paths, then trumps the messaging overhead.

#2 also works well for data processing pipelines.

-Matt

Re:Should be micro kernel

[Anonymous Coward]

OS-X is not a microkernel - has never been. Even when NEXTSTEP was based on Mach 3.0, that too was not microkernel. Apple doesn't use any of the rules of microkernels - user space drivers, et al - in OS-X or iOS

Screw you Mac OS X guys with your bloated microkernel! I use classic Mac OS with a nanokernel .

Captcha: miserly

Re:Should be micro kernel

[TechyImmigrant]

I implemented Transputer drivers for OS9 running on a 68k in a VME rack back in the day.

It was a clean, simple and well designed OS.

Re:Should be micro kernel

[0123456]

You'd presumably need to add new CPU functionality to allow fast context switches. If I remember correctly, a 20MHz Transputer took about one microsecond to switch threads, because that was one of the primary design goals. Of course, that lead to them building a stack-based CPU where almost nothing had to be saved on a context switch...

Re:Should be micro kernel

[thogard]

It was a monolithic kernel. One of the interesting features were devices drivers were modules and there was a small device node module which would say stuff like "used module 'serial driver', call it tty4 at IRQ 2 and address 0x454040". The kernel would deal with all IRQs in the hardware and then run the IRQ callback funtion in the proper module. That allowed user level device drivers back in the early 1980s.

Another cool feature was each software module had a CRC so it could detect bad binaries. There were ways to whitelist and blacklist based on CRC values.

Re:Should be micro kernel

[jblues]

So they're both further towards the monolithic side?

Loadable kernel modules have nothing to do with microkernels. A truly micro microkernel wouldn't need loadable kernel modules because all the loadable functionality would run in userland; plenty of monolithic kernels have loadable kernel modules.

I'm sure you're right, though they have something to do with micokernels. There was Linus interview from a few years back explaining his preference for the monolithic approach, and he explained that modules were introduced to give most of the benefits of the microkernel, without the drawbacks.

Re:Should be micro kernel

[Guy Harris]

I'm sure you're right, though they have something to do with micokernels. There was Linus interview from a few years back explaining his preference for the monolithic approach, and he explained that modules were introduced to give most of the benefits of the microkernel, without the drawbacks.

I'd have to see that interview to believe that's exactly what he said. In this essay by him, he says

With the 2.0 kernel Linux really grew up a lot. This was the point that we added loadable kernel modules. This obviously improved modularity by making an explicit structure for writing modules. Programmers could work on different modules without risk of interference. I could keep control over what was written into the kernel proper. So once again managing people and managing code led to the same design decision. To keep the number of people working on Linux coordinated, we needed something like kernel modules. But from a design point of view, it was also the right thing to do.

but doesn't at all tie that to microkernels.

Loadable kernel modules in UN*Xes date back at least to SunOS 4.1.3 and AIX 3.0 in the early 1990's. I'm not sure they were introduced to compete with microkernels.

Re:Should be micro kernel

[jblues]

I can't find the article now. It was years ago. Perhaps I misunderstood it. But I think it meant something like:

• Microkernels allow non-fundamental features (such as drivers for hardware that is not connected or not in use) to be loaded and unloaded at will. This is mostly achievable on Linux, through modules.

Re:Should be micro kernel

[Guy Harris]

I can't find the article now. It was years ago. Perhaps I misunderstood it. But I think it meant something like:

• Microkernels allow non-fundamental features (such as drivers for hardware that is not connected or not in use) to be loaded and unloaded at will. This is mostly achievable on Linux, through modules.

That's more like "mechanisms X or Y both allow Z to be accomplished"; the only thing that says X and Y have to do with one another is that they both allow Z to be accomplished, which isn't that much.

Re:Should be micro kernel

[Anonymous Coward]

I've never seen a RISC processor that can match the performance of the best CISC processors.

Look who's counting mainframes! ;)

Re:Should be micro kernel

[jblues]

though they have something to do with micokernels

Which isn't that much.

Great, can we agree now that not much is something and not nothing?

In other news Thylacines and Jackals have nothing to do with each other, except they both look like canids and fill similar ecological niches. Apples and oranges . . .

Re:Should be micro kernel

[itzly]

Micro kernel is not necessarily better.

No, micro kernels are plain worse. The biggest problem with micro kernels is the synchronisation problem you get with distributed state. Imagine a file system that is split up in different tasks, instead of one monolithic blob. Now, one task makes a change, like removing a file. Before the other tasks can make changes to the filesystem, they first need to synchronize to get the latest state. This becomes either terribly inefficient, or a huge mess, and most likely both at the same time.

Minix, for example, solved this particular problem by letting the entire filesystem be supported by a single task, but results in terrible scaling performance when adding more users/processes.

Re:Should be micro kernel

[Teun]

Ah the days when umacs was the editor of choice, a large HD was a full 10MB and backing up required 70 floppies.

Re: Should be micro kernel

[Anonymous Coward]

so you have never used a big powerpc machine.

Re:Should be micro kernel

[phantomfive]

It's like software layers. You'll see some groups that are utterly adamant about keeping things strictly in layers, yet there are often very noticeable barriers between the layers that are inefficient both in run time and in developer productivity

Everyone builds things in layers. At a minimum, you have the layer between the CPU and assembly language, between the kernel and userland, and between storage and the 'file' abstraction. There are plenty of other layers because they are helpful.

If the layers get in the way instead of helping, it means that the layers were designed poorly.

Re: Should be micro kernel

[Anonymous Coward]

That's relatively recent marketspeak. Back in the pre-Radisys days Microware most definitely touted it as a microkernel. I bet they just wanted to stay away from the term to seem 'hip' since microkernels have become unfashionable.

Re: Should be micro kernel

[Anonymous Coward]

It's possible to boot OS-9 for the 68k via emulation (barebones demo here):

OS-9/68k (OSK) running on FAME 68k emulator core â¦: http://youtu.be/odXcexmpfpE

Re:Should be micro kernel

[Guy Harris]

though they have something to do with micokernels

Which isn't that much.

Great, can we agree now that not much is something and not nothing?

Sure, if we'll also agree that "[introducing] (un)loadable modules" to a monolithic kernel "to address maintainability and extendability" does not in the least make that kernel any closer to a microkernel (because, in fact, it doesn't).

In other news Thylacines and Jackals have nothing to do with each other, except they both look like canids and fill similar ecological niches. Apples and oranges . . .

In other other news, Felis catus and Loxodonta africana have nothing to do with each other, except that they have four legs and bear live young.

Srsly, "both are kernels" and "both let you load and unload stuff" isn't much of an ecological niche. True microkernels (not "hybrid kernels" like the NT kernel or XNU) and monolithic kernels (with our without loadable modules) are sufficiently different from one another than "you can add or remove stuff at run time" isn't much in the way of commonality.

Re:Should be micro kernel

[Darinbob]

Well yes, normally you'd keep related functionality in the same task. In many RTOS environments everything's in a task anyway, it's the normal way to do things. There's isn't necessarily much synchronization of state, as each task maintains its own state (you don't need the file system's latest state in order to interact with it).

The distinction between kernel and application is blurred in many embedded systems. Similarly, the difference between a microkernel and a monolithic kernel can get blurred as a mono kernel may have kernel threads, or a micro kernel may still have a core kernel that controls access to basic hardware functions. Though you rarely see people use monolithic kernels in smaller embedded systems, there's significant overhead there, but monolithic starts being used when the system gets larger or there's less need for real time.

I wouldn't say minix is a good example here, performance wise, as it's intended primarily to be an educational tool.

Re:Should be micro kernel

[jblues]

From the operating systems that have survived out in the wild today (general, widespread use):

• Kernels with a microkernel philosophy have made compromises towards monolithic to address performance bottlenecks
• Kernels with a monolithic architecture introduce loadable modules to address maintainability/extensibility, which happens to be one of the advantages of the microkernel.

• I see your point that this addresses short-coming, without moving towards a microkernel philosophy.

Re:Should be micro kernel

[serviscope_minor]

I've never seen a RISC processor that can match the performance of the best CISC processors. You know, nevermind the fact that tons of money has been poured into CISC processors making them faster and faster.

I hate to be pedantic (I'm lying, I love to be pedantic), but for what measure of performance? For out right single threaded MIPS/FLOPS and Flops/W in certain situations, they do win. For other measures of performance, not so much.

Re:Should be micro kernel

[Anonymous Coward]

Or a seriously heavy cpu load, where it may show an advantage. The typical desktop user, nope.

Re:Should be micro kernel

[itzly]

I wouldn't say minix is a good example here, performance wise, as it's intended primarily to be an educational tool.

Still, it's worthwhile to examine the problems, and think about how to solve them. Minix has a clear and serious problem with the single threaded filesystem, and its lack of performance. If you want to scale it up to a general purpose computer system (not a small embedded, single purpose design), you're either going to have to a) keep it as a single task but make it smarter, or b) split it up in multiple tasks. Keeping a filesystem as a single task is problematic. For instance, a request comes in that requires waiting for physical hardware. While the task is waiting, other requests come in that can be served from the cache, but since the task is waiting for the hardware, these requests will have to wait, wasting performance. Trying to fix that by building giant state machines makes a huge mess out of the design. On the other hand, trying to improve performance with multiple, independent, threads basically turns the filesystem into a distributed filesystem, with all the synchronisation issues.

In a monolithic kernel, none of these problems exist. Every application request that enters the filesystem layer automatically continues in its own independent thread. When it hits an area that requires synchronisation, it briefly acquires a lock (usually without contention), does the work, and releases the lock. This is a much simpler design, with higher performance.

Re:Should be micro kernel

[Anonymous Coward]

nS [...] ns [...] uS

Please don't do that. Capitalisation is important. A lower-case "s" means a second, a unit of time, and an upper-case "S" means a Siemens, a unit of electrical conductance. At least you've used the correct case for the prefixes. (A "u" for "micro-" is fine when you're using a typeface that doesn't have a Greek "mu".)

Other important ones to remember: lower-case "b" for bit, upper-case "B" for byte, lower-case "k" for kilo-, upper-case "M" for mega-, lower-case "m" for milli-, upper-case "W" for watt, etc. Remember, MB/s, mb/s, Mb/s and mB/s all mean different things.

Re:Should be micro kernel

[itzly]

The major problem is not the performance of the CPU.

To pass a message across a protected barrier, it means that you can't use two references to the same data. You must actually make a copy of the data. Everybody thinks that the time spent making the copy is a problem with microkernels that needs to be addressed. But the real problem is not the time to make a copy. The real problem is that you now have two different copies of the same data. And when you modify one copy of the data, to update some of the state, the other copy remains in the old state, which is generally not a good idea, unless you have carefully made a design that can tolerate such differences. Or, you would have to build in frequent synchronisation points where all the new state information is distributed to all the tasks. This means poor performance, because the tasks will spend a lot of time waiting for other tasks to catch up.

Re:Should be micro kernel

[Anonymous Coward]

So does Linux. TUN/TAP and Fuse come to mind. We still consider Linux a monolithic kernel, though. Because even though it still supports running network drivers and filesystems in userspace, like a microkernel, it still contains all those other non-micro things.

Re:Should be micro kernel

[fisted]

Way to miss the point.

Besides, 'assembler' is a part of the toolchain. The language(s) are called 'assembly'.

Re:Should be micro kernel

[Anonymous Coward]

Not all C compilers compile through assembly. There are also C interpreters.

Re:Should be micro kernel

[Anonymous Coward]

I've never seen a true microkernel that has the performance of a monolithic kernel.

I've never seen a RISC processor that can match the performance of the best CISC processors. You know, nevermind the fact that tons of money has been poured into CISC processors making them faster and faster.

Sometimes, it's just a matter of where the attention has been placed.

Back when RISC was a big thing (early 90's) there was plenty of RISC that was faster than CISC.

The main reason CISC is faster today is probably more related capital investment needed in production. Intel just have so much more.

ARM still does pretty well in its segment though.

Re:Should be micro kernel

[Anonymous Coward]

It's funny how in a grammar-nazi post on capitalization you spelled the unit "siemens" with a capital "S", like the surname of the guy it's named after.
Yours truly, millibyte per siemens

Re:Should be micro kernel

[JesseMcDonald]

In a monolithic kernel, none of these problems exist. Every application request that enters the filesystem layer automatically continues in its own independent thread. When it hits an area that requires synchronisation, it briefly acquires a lock (usually without contention), does the work, and releases the lock. This is a much simpler design, with higher performance.

Any particular reason you couldn't do the same thing in a microkernel? I'm envisioning some form of IPC primitive that automatically spawns a lightweight thread to handle each incoming message, which isn't too different from the monolithic kernel approach apart from not having a fixed 1:1 correspondence between the client and server contexts. You would be able to use your shared data structures and locks just as you would in a monolithic kernel, at least within the filesystem code. For anything else, of course, you'd need to use IPC.

Re:Should be micro kernel

[Anonymous Coward]

> Nobody does message passing for basic operations.

QNX does: http://en.wikipedia.org/wiki/QNX

Re:Should be micro kernel

[snowgirl]

The main reason CISC is faster today is probably more related capital investment needed in production. Intel just have so much more.

This was basically what I was trying to say. More capital investment typically means better outcomes.

Re:Should be micro kernel

[snowgirl]

I wasn't intending on saying that CISC was superior to RISC... what I was more saying is that there has been more money put into CISC processors, and so they develop faster.

It's just a simple fact of money == better access to stuff to make more money.

Re:Should be micro kernel

[snowgirl]

That's cheating.

And inside, a modern x86 processor is actually a giant hardware emulation of x86 instructions with a RISC/VLIW core... You call it cheating, and I call it optimizing.

They second you try a cool trick like migrating a thread to another machine...

But this would happen with a macrokernel as well... you can't just magically make networking overhead disappear...

Re:Should be micro kernel

[unixisc]

I've never seen a RISC processor that can match the performance of the best CISC processors. You know, nevermind the fact that tons of money has been poured into CISC processors making them faster and faster.

Sometimes, it's just a matter of where the attention has been placed.

That's b'cos active development of RISC processors stopped in the last decade, most of them sunk by the hype around the Itanic. Otherwise, the last time you had the big 3 RISC processors - POWER2, PA-RISC and Alpha 21264, they were heavily ahead of the Pentium. Except that most people don't run Dhrystones or SPEC## or other such benchmarks - they run real software, which just wasn't available for most of the above CPUs.

Two things happened since the demise of the PA-RISC, Alpha and MIPS III & IV: first, Intel continued to shrink their CPUs faster, thereby increasing their individual performance and closing the gap w/ RISC CPUs. But the biggest coup for Intel was Windows NT becoming the underlying OS for all Windows OSs in Windows 2000 and then XP. While Windows 9x wasn't SMP capable, suddenly, Windows XP was. So Intel could take an optimal CPU core, toss in as many as made sense - 2, 4, 8, whatever - and run them against the fastest of RISC workstations. That had 2 big advantages over RISC - first, it continued to run native Wintel software, regardless of whether they were multithreaded or not, or optimized for multi-core or not. The second was that at a given price point, Intel could toss several cores to not just match, but even exceed the performance of a RISC workstation. Once that happened, the reason to prefer RISC at all went away. RISC could have gotten the same attention that the x64 got, but unless Microsoft took the initiative in porting the bulk of their applications to, say, the Alpha, that wasn't gonna move. Compaq saw that and pulled the plug on that platform.

Re:Should be micro kernel

[unixisc]

HURD never looked at seriously good micro-kernels. Or else, they'd have forked Minix3, and used that as a basis for their platforrm. Instead, they went through 4 different micro-kernels before settling on the most archaic of them all - Mach 3, which is least micro-kernel by the most current definitions of the concept

Adeos and Xenomai

[Anonymous Coward]

I have a feeling that this is going to mean a *big* hump for everyone who uses real-time Linux. It could be months before we're able to upgrade. Adeos tinkers with that code, and Xenomai works on Adeos patched kernels.

Re:Adeos and Xenomai

[Anonymous Coward]

With some luck it will fix some of the jitter problems that makes real-time Linux unusable on computers that aren't a decade old for some use cases.

Re:Adeos and Xenomai

[wonkey_monkey]

this is going to mean a *big* hump for everyone

Giggity.

hmmm

[Anonymous Coward]

linux is obsolete, but there is nothing better

Re:hmmm

[Anonymous Coward]

In which way obsolete?

Re: hmmm

[Anonymous Coward]

Systemd, of course.

Re:hmmm

[Anonymous Coward]

Systemd SJW trash code.

Re:hmmm

[Anonymous Coward]

Yes, linus needs people skills training from lennart.. there's some irony for you..

micro vs. mono

[turkeydance]

r u redi to rumble?

efi 32bit

[Anonymous Coward]

does it support low end x86 tablets with efi32 now, like windows has for years, or is that still a no.

Re:efi 32bit

[Anonymous Coward]

I imagine it supports far more low end x86 tablets than Windows supports ARM tablets, but hey, if you want to lamely try to imply that Windows is better at system compatibility I guess you have to grasp at straws.

Re: efi 32bit

[Anonymous Coward]

Electronic fuel injection?

Re: efi 32bit

[fisted]

yes. 32-valve, too.

yay

[Anonymous Coward]

wtg guys

Risk of regression

[labradore]

> There's a risk of regression when exposing the code to many more systems

The risk of regression is due to refactoring, not due to testing. Ironic, given that the post cites de-obfuscation as a reason for doing this. Or perhaps our submitter just got an MBA and is learning to think and speak in management-ese.

Re:Risk of regression

[Anonymous Coward]

It's fairly common to speak of regressions "occurring" during testing. It's a test-centric point of view, rather than a developer-centric point of view. Obviously the regressions actually come from bugs that are introduced during coding, but that's such a trite observation I'm amazed you even bothered to make it. Test-centricity is not the same as MBA management speak. Do you not have testers where you work, or are you just not allowed to interact with them?

Speedups?

[darthsilun]

> There are over 100 separate ... speedups

The last time I looked, which was quite a few years ago TBH, the BSDs have, IIRC, less than 100 lines of x86 assembly, in the bootstrap. It's really only there to manage loading at the right address and then jumps into C code. Maybe there's some more bits scattered in other places too, IDK. Bottom line is that there's just not a lot of time spent in code written in assembly.

What the fsck is in the Linux kernel that could be sped up?

Re:Speedups?

[darthsilun]

What the fsck is in the Linux kernel _assembly_ that could be sped up?

Before some anal retentive navel gazer starts obsessing over my omission.

Re:Speedups?

[Guy Harris]

> There are over 100 separate ... speedups

The last time I looked, which was quite a few years ago TBH, the BSDs have, IIRC, less than 100 lines of x86 assembly, in the bootstrap.

From relatively-recent FreeBSD:

$ find sys/i386 -name '*.[sS]' -print | xargs wc -l
208 sys/i386/acpica/acpi_wakecode.S
40 sys/i386/bios/smapi_bios.S
396 sys/i386/i386/apic_vector.s
78 sys/i386/i386/atpic_vector.s
160 sys/i386/i386/bioscall.s
470 sys/i386/i386/exception.s
900 sys/i386/i386/locore.s
279 sys/i386/i386/mpboot.s
831 sys/i386/i386/support.s
538 sys/i386/i386/swtch.s
179 sys/i386/i386/vm86bios.s
37 sys/i386/linux/linux_locore.s
127 sys/i386/linux/linux_support.s
32 sys/i386/svr4/svr4_locore.s
202 sys/i386/xbox/pic16l.s
494 sys/i386/xen/exception.s
361 sys/i386/xen/locore.s
5332 total
$ find sys/amd64 -name '*.[sS]' -print | xargs wc -l
282 sys/amd64/acpica/acpi_wakecode.S
326 sys/amd64/amd64/apic_vector.S
73 sys/amd64/amd64/atpic_vector.S
541 sys/amd64/amd64/cpu_switch.S
906 sys/amd64/amd64/exception.S
88 sys/amd64/amd64/locore.S
236 sys/amd64/amd64/mpboot.S
56 sys/amd64/amd64/sigtramp.S
732 sys/amd64/amd64/support.S
75 sys/amd64/ia32/ia32_exception.S
161 sys/amd64/ia32/ia32_sigtramp.S
38 sys/amd64/linux32/linux32_locore.s
124 sys/amd64/linux32/linux32_support.s
246 sys/amd64/vmm/intel/vmx_support.S
42 sys/amd64/vmm/vmm_support.S
3926 total

It's about 45,000 lines in Linux 3.19's arch/x86. A fair bit of that is crypto code, presumably either generally hand-optimized or using various new instructions to do various crypto calculations.

Re:Speedups?

[BitZtream]

If you consider that the C compiler itself uses assembly to make the basic operations work in the libraries, and that all C code is built on assembly libraries, then it makes the whole argument kind of silly, doesn't it?

ALL the kernel code is assembly on Linux and BSD, some of it is just raw assembly, and other bits of it are assembly encoded in "C".

The couple in that most languages and VMs are written in C ... as well as all the libraries that these things depend on to actually get something else done ... well then pretty much everything is reduced to assembly ...

Re:Speedups?

[jones_supa]

There's also a cool tool called CLOC which gives a nice report about a source tree including the lines of code and in which languages they are written.

Re:Speedups?

[stooo]

>> ALL the kernel code is assembly on Linux and BSD, some of it is just raw assembly,
Yes, but the compiler is a few orders of magnitude more reliable than a human when generating assembly

Re:Speedups?

[Anonymous Coward]

Not really, just faster.

Re:Speedups?

[Anonymous Coward]

And my favourite line:
locore.S
                call hammer_time /* set up cpu for unix operation */

Re:Speedups?

[Anonymous Coward]

if that was true, inline assembly would never had needed to be implimented

If It Ain't Broke, Don't Fix It!

[Anonymous Coward]

Why modify tested working code? What is this other than an excellent opportunity to inject malware into multiple Linux distros?

Re:If It Ain't Broke, Don't Fix It!

[Anonymous Coward]

if they want to audit those stale old things for correctness and performance, and make them more readable for future generations, and
do the testing and review to make sure they haven't fucked them up

then good for them. i mean really good for them.

any code - especially the kernel - isn't a concrete artifact, its a process. an organism.

heathy organisms eliminate their wastes

and a 2% performance bump in system call overhead isn't anything to sneeze at

Re:If It Ain't Broke, Don't Fix It!

[Anonymous Coward]

It always disappoints me how many people still parrot this old and broken cliche. You're probably not in software, but if you are, please develop an understanding of your craft and good software engineering practices. Please read and learn about testing. Please read and learn about refactoring. Don't be like OpenSSL. This code is going bye bye and will be replaced with something better that will also be tested and just as bulletproof in short order.

You think the old code isn't broken. It is though, by sheer virtue of the fact that someone wrote a cleaner, better version of it.

Re:If It Ain't Broke, Don't Fix It!

[Anonymous Coward]

bullshit.

unless you have metrics and or studies on how rewriting decreases bug density.

captcha: registry. isn't that ironic!

Re:If It Ain't Broke, Don't Fix It!

[Anonymous Coward]

"deobfuscate a decade old spaghetti assembly code and its C code dependencies"

Bug density is directly proportional to spaghetti.

Re:If It Ain't Broke, Don't Fix It!

[Gavagai80]

You're operating on the mistaken assumption that code that works now will always work and never need to be modified. You can't leave anything but the simplest things alone forever, because changes to the context/world will eventually require changes to it. If it's spaghetti code, that's going to be causing future bugs that are going to be non-obvious and difficult to discover.

Re:If It Ain't Broke, Don't Fix It!

[Troed]

I am "in Software" since ~25 years. I also hold a degree as a Software Engineer.

People who obsess about rewriting old code just because it's old tend to forget that in that old code are many bug fixes for edge cases found over the years. It was well documented and part of my education to know and understand that rewriting often caused those same bugs to surface again.

Best practice is to run both the old and new software in tandem for a while and verify the results. In reality no organization besides NASA will do that.

Re:If It Ain't Broke, Don't Fix It!

[TheDarkMaster]

When you need more performance and/or new features. But this said, I also agree that it is stupid to change code just to meet the latest fad from brats of today.

Re:If It Ain't Broke, Don't Fix It!

[Anonymous Coward]

Refactoring is not the same as rewriting. Refactoring is the application of a series of transformations which are non-identity in source code but which are identity transforms on the semantics of the source code. It is possible to make mistakes in refactoring, but it doesn't cause old bugs to reappear unless a mistake was made in refactoring. All the information about the workarounds for those bugs is in the code, and should be preserved, modulo human error. Refactoring is no less safe than adding a new feature.

Maybe you should refresh your understanding of software methodology. You're 25 years out of date.

Re:If It Ain't Broke, Don't Fix It!

[Troed]

"unless a mistake"
"should be preserved"

= best practice is to know and understand that the old bugs will resurface. I.e, there's a cost to do the rewrite (no matter if you call it refactoring or not) that will affect the business for some time after deployment.

Your Software Engineering education seems to be a bit lacking.

Re:If It Ain't Broke, Don't Fix It!

[Parker Lewis]

I have a degree in Computer Science, and I'm in software since 1998. If you do a proper refactoring, at the end of the day, you'll get a much better code, probably better performance, and now that you have more background in the subject, a smaller code. If you're using a code repository, so you'll never lose anything. And if you have a bug regression always coming back, you need a proper test/spec to cover that. So, refactor is really good when: you have a way improved background in the subject, code repository (i.e., history), and tests to cover the recurrent bugs and the main features.

Re:If It Ain't Broke, Don't Fix It!

[gatkinso]

It always disappointing to me when inexperienced software developers confuse the meanings of "refactoring" and "rewriting."

Re:If It Ain't Broke, Don't Fix It!

[Anonymous Coward]

Best practice is to run both the old and new software in tandem for a while and verify the results.

How do you run an interrupt handler "in tandem"?

Re:If It Ain't Broke, Don't Fix It!

[Troed]

... thus "very few organizations besides NASA".

A lot of people seem to miss the point on how the ideal lab condition doesn't carry over into real world organizations.

Re:If It Ain't Broke, Don't Fix It!

[Anonymous Coward]

Because this is literally faster on a noticeable level.

Just because something works, doesn't mean it shouldn't be optimized.
There is almost always some room for improvement in non-algorithmic code.

They, even though still ugly, improved readability quite a bit, and not only kept the same speed, they improved on it quite a bit too.
That is a major win in my eyes.

Security issues can and will be dealt with, of course.
But for all we know, there could be security issues with THIS (that) code that have gone unnoticed for years.
It can take a small period to rewrite code, it can take far longer to crack it even with an army of sweaty basement nerds.
Obfuscation can work if you do it the right way.

Re:If It Ain't Broke, Don't Fix It!

[Ben Hutchings]

Because it wasn't tested well enough? For example, in the case of the system call entry path, Andy Lutomirski found a bunch of bugs over the past few months - including CVE-2014-4508, CVE-2014-9090 and CVE-2015-2830. His changes for 4.1 include the addition of regression tests as well as cleaning up that code.

opportunity for backdoors?

[Anonymous Coward]

Does this mean that a bunch of assembly code is going to be replaced? Who is going to code review that? Who can sight read assembly anymore?

Re:opportunity for backdoors?

[EzInKy]

Who can sight read assembly anymore?

Everybody who is interested in "How Things Work" can read assemblly code. Those who depend on hopes and prayers do not.

Re:opportunity for backdoors?

[Anonymous Coward]

Guess I should put it on my ToDo list.

Re:opportunity for backdoors?

[phantomfive]

Who can sight read assembly anymore?

Anyone who reasonably calls themselves a programmer.

Re:opportunity for backdoors?

[Anonymous Coward]

People who know more than just python and ruby scripting, ie real programmers.

Re:opportunity for backdoors?

[EzInKy]

That depends. Are you more interested in how something happens, or are you more interested in preventing something from happening? I think that is the factor that differeniates technicians from scientists.

Re:opportunity for backdoors?

[Anonymous Coward]

I've never learned C, Java, or Python, but I can sure read x86 assembly.

It's only a modest refresh

[m.dillon]

It's not a major refresh, only a modest one, and it doesn't really fix the readability issues (which would require a complete rewrite). Linux assembly is a mostly unreadable, badly formatted, macro-happy mess. The assembly in the BSDs is much more elegant and minimalistic.

-Matt

Re:It's only a modest refresh

[Anonymous Coward]

So is clean code a modern thing? Or is the skill of the current developers much higher than it used to be?

Whenever I crack open old open source code, there's seems to be absolutely no concept or desire to have readable code, with absolutely vague variable names (all missing vowels of course), and fragile interfaces. It's like the concept of compartmentalization didn't exist. Maybe it was the lack of good optimizing compilers, so what you wrote made a significant difference, more like an interpreted language? Maybe they didn't trust optimization, so screwed us in the future?

lol, captcha: codifies

Re:It's only a modest refresh

[Anonymous Coward]

It is a modern thing, primarily for two reasons:

(1) As you mentioned, optimization trumped cleanliness. It's not just that a given coder couldn't be bothered wasting his time writing it cleaner: it's that often the choice was between a guy who can write clean code and a guy who can write very messy but highly-optimal code, and the latter would win in the marketplaces (the software, hardware, and programming-job ones). Writing optimal assembly code and organizing a modern large clean codebase in a HLL don't have a ton of skill overlap, all things considered.

(2) As you rewind the clock on programming history, keep in mind that further back there were simply far fewer total programmers in existence, and far fewer working on any given codebase, by orders of magnitude. When you look back far enough, you see major companies launching major projects with a total programming staff of like 1-3 guys. Most of the code ever written in the older decades was the result of heroic one-man efforts. Why bother optimizing for others being able to read your code where there's unlikely to be many of them, and they're all likely to be crazy like you anyways?

Re:It's only a modest refresh

[Anonymous Coward]

Whenever I crack open old open source code, there's seems to be absolutely no concept or desire to have readable code, with absolutely vague variable names (all missing vowels of course), and fragile interfaces.

Have you looked at code written by academics? The variable names are the least of their transgressions. There are no comments anywhere in the code except the header of the file which only says the blandest and meaningless of things anyway. Academics releasing source code lacking comments should be terminated with extreme prejudice.

Re:It's only a modest refresh

[Steve Hamlin]

+5 Informative / Insightful for parent

For background, the parent comment is Matthew Dillon, compiler and kernel hacker on Amiga/BSD/Linux and founder of DragonFly BSD (fork of FreeBSD).

http://en.wikipedia.org/wiki/D...

No, the real questions are...

[EzInKy]

...do you know what the machine is doing with the codes that you type? Abstractions neccessarily lead to assumptions.

Ingo Molnár on distributions

[Anonymous Coward]

Ingo Molnár has interesting comments on distributions, basically he wants them more modular.

Tomato Source

[Anonymous Coward]

part of a heroic effort to deobfuscate a decade old spaghetti assembly code

Which is why the 4.1 release should be named as "Tomato Source."

So that is how it happens

[PenguinJeff]

The improved assembly code is what allows the Terminator to be so efficient a killing machine.

Re:So that is how it happens

[Mariner28]

So when we get to kernel version 4.1.15, it will speak with an Austrian accent rather than Finnish?

And of course, when we see the later T-1000 form a pointy sword from its liquid metal arm and kill young John Connor's foster father, it's further proof that there's still old cruft code in the future kernel, since it's just reproducing Linus' most famous gesture.

Wouldn't a re-write be more fruitful?

[Anonymous Coward]

I spent 5 years trying to fix an application that was built over the course of a few months

I've seen similar things keep repeating, in many of the places I know

What I do not understand is, why people do that in the first place ...
 
Wouldn't a total re-write be the right thing to do instead?

Re:Wouldn't a re-write be more fruitful?

[Hognoxious]

I don't really know why.

Users will say "But it works, we don't want to change waaagh scary" while simultaneously reporting 237 bugs all of which are OMG critical. Management will assume that it's cheaper, because existing stuff is already there so it's wasteful not to use it.

Now it's true that once a load of crufty business rules have built up with 17 levels of nested conditionals it can be risky to try and replicate it for fear of missing some obscure case that's bound to occur at an inconvenient time for a key customer. There's no documentation, of course. Or if there is it's the source code, six revisions behind, pasted into a word document with three screenshots taken as BMPs so the whole thing is 1.5G. This alone can make you say "sod it".

I can't find the correct phrase but maybe it's just a false analogy with physical things. Like reusing wood from an old shed to build a deck possibly is cheaper.

Re:Wouldn't a re-write be more fruitful?

[Anonymous Coward]

Because many times the people in charge of deciding on the rewrite are the same people who were responsible for the mess in the first place.
It's essentially asking them to admit failure.

Re:Wouldn't a re-write be more fruitful?

[DrXym]

The problem with total rewrites is they almost always involve a huge amount of effort, introduce new bugs and when they "work", users barely notice the difference. So the company soldiers on, applying patch upon patch to some bullshit codebase and suffering from it but in a incremental way.

Worst of all is when they embark on a rewrite and give up half way through. I was involved in a project to port a C++/ActiveX based system to .NET forms. They ported most of the major views but left a lot of the minor stuff from the old codebase lying around and wrote bridges to host it in the new framework. So they doubled the code, half of it became bitrotten and hidden by the new code and bloated out the runtime. Great project.

Re:Wouldn't a re-write be more fruitful?

[khellendros1984]

Worst of all is when they embark on a rewrite and give up half way through.

I saw something similar happen at my employer. A newer employee was sent on a mostly solo project to rewrite some of the core of our product, to make it easier to make some planned enhancements. Things didn't immediately work perfectly, and some of the founding employees fought the changes. Since they were necessary for the next release, and there wasn't another option for the features that we were required to make available, we ended up with two parallel sections of code that did basically the same thing but in very different ways. That was what we put up with for 4 or 5 *years* before the politics of the situation allowed us to integrate the changes back into a single codepath.
Before that, if you didn't touch that code constantly, it was usually unclear which path was being actively used and which was bitrotted to uselessness.

Re:Wouldn't a re-write be more fruitful?

[tlhIngan]

The problem with total rewrites is they almost always involve a huge amount of effort, introduce new bugs and when they "work", users barely notice the difference. So the company soldiers on, applying patch upon patch to some bullshit codebase and suffering from it but in a incremental way.
Worst of all is when they embark on a rewrite and give up half way through. I was involved in a project to port a C++/ActiveX based system to .NET forms. They ported most of the major views but left a lot of the minor stuff from the old codebase lying around and wrote bridges to host it in the new framework. So they doubled the code, half of it became bitrotten and hidden by the new code and bloated out the runtime. Great project.

And what you described is technical debt.

Rewriting code costs time, effort and money, which is how you repay the debt. After all, when faced with a fix, you can do it quick and dirty and borrow from the bank, incurring debt, or fix it properly but takes longer, with no debt. The former will get it done the quickest and put the fire out, but that stuff you borrowed will haunt you later.

Your new system where you wrote bridges means you got rid of some debt, but the incurred a bunch more.

Re:Wouldn't a re-write be more fruitful?

[Anonymous Coward]

Do you work at the same company as I do? Your story is so familiar, but on our company the PHB also wanted us to change the UI completely every month or two, it seems that if the UI was changed radically often enough into random direction, it will eventually be good just by luck. Of course, the users did not want to keep using software which had a completely randomized UI on each release.

Re:Wouldn't a re-write be more fruitful?

[phantomfive]

I don't really know why. Users will say "But it works, we don't want to change waaagh scary" while simultaneously reporting 237 bugs all of which are OMG critical.

Because if you did it wrong the first time, there's no chance that you're going to do it better the second time. You'll end up leaving out crucial functionality or something.

If you don't know how to clean up a codebase in-place by rewriting a little at a time, then you aren't skilled enough to do a rewrite from scratch.

Re:Wouldn't a re-write be more fruitful?

[toddestan]

Because it's always cheaper and faster to kludge the next feature on to make the next shipment or put out the current fire than it is to rewrite. If they would take a step back, look at what they want to be in a year or two they'd realize that the time invested would pay for itself over and over. But most places don't have that kind of foresight, or the manpower to do that while also dealing with all/ the short-term crises arising from the current codebase.

Re:Wouldn't a re-write be more fruitful?

[Hognoxious]

Why are you assuming the person fixing it is the one who originally wrote it?

Re:Wouldn't a re-write be more fruitful?

[phantomfive]

I'm not. If a different person is fixing it, then the second paragraph applies.

Re:Wouldn't a re-write be more fruitful?

[Anonymous Coward]

There's a few problems, besides incrementalism which others have covered.

1. You might not know there is 5 years of work at first. Then you get into sunk cost fallacies.

2. Some of that 5 years isn't fixing what's broken but implementing what's missing. In other words, doing it right from scratch could take 5 years anyway.

I didn't see anyone asking about feelings.

[0xdeaddead]

were anyone's feelings injured in this kernel?

Are peoples feelings going to be hurt with the assembly changes in 4.1?

Is anyone offended by the words pop/push/mul/div/jmp?

feeligns man, gotta make sure nobody's feelings are hurt.

XNU?!?

[Ice Station Zebra]

Is apple powered by scientology?

Belittling is the way to go!

[Anonymous Coward]

The best way to make yourself big and brilliant is to take someone elses excellent code that you could never write, go through it and implement their logic in "new" code claiming that their "spaghetti garbage" has now been fixed by you even though you did little more than a lint tool would.

Claiming that the work of others, that you build upon and without which you would be fumbling in the dark, is garbage is bad form.

Re:So what now?

[itzly]

I'm waiting for Intel to integrate systemd in the next core update.

80386

[Anonymous Coward]

Never again? Not even emulating 80486 instructions?

Answer to both questions: Management

[mpercy]

Management pressure forces even good developers to produce. Sometimes, against your better judgement, you have to go for the quick fix to meet a deadline (or not exceed a deadline by too much). Or, developers make a proof-of-concept or prototype, and management says "ship that".

Once management has its grubby hands on existing codebase, it ignores the accrued technical debt, poo-poos developers warnings to rewrite some stuff that was thrown together in the heat of battle, and never funds general background cleanup tasking.

Management does not believe in bit-rot.

Re:Answer to both questions: Management

[Anonymous Coward]

Management does not believe in bit-rot.

One of the weaknesses of proprietary. Unlike open source, where you're free to go on a cleanup-spree if you like.

Reroofing is close analogy

[mpercy]

Building codes generally allow homeowners to install a second layer of asphalt shingles over an existing layer of same. But once you're considering a 3rd layer, building codes generally require the entire roof to be stripped and redone.

Software should be the same way.

Not with a bad set of requirements

[johanwanderer]

Wouldn't a total re-write be the right thing to do instead?

Yes, if you can get the proper requirements. (This does not apply to the current article, since I assume that the requirements for these syscalls, etc. are well described.)

On most business systems, especially one that us written over the course of a few months, the requirements are just as spaghetti as the code, so rewriting the system from scratch might also rewriting the requirements from scratch, which is a monumental task if it already have customers with different configurations.

On a more humorous note, I find it funny that this is today's article on The Daily WTF: Seven Minutes In Heaven

B-llshit

[Anonymous Coward]

B-llshit. Linux code quality is superior, because so many developers are there cleaning up everything all the time.

Who cares.

[Anonymous Coward]

"Linux" and "Gaming" don't even belong in the same sentence. You're welcome to keep dreaming, but no amount of it is going to lead you to reality.

Because hacker hack and ...

[Anonymous Coward]

because working on UIs is to hard to make easier.

Yay

[Anonymous Coward]

Oooh yay! Lots of new security holes will be introduced to exploit! Do it like openssl.