Microsoft Developer Explains Why Windows Kernel Development Falls Behind

New submitter mha writes "In a response that truly seems to be from a core Microsoft developer, we are told about why Windows kernel development continues to fall further and further behind that of the Linux kernel. He says, 'The cause of the problem is social. There's almost none of the improvement for its own sake, for the sake of glory, that you see in the Linux world. ... There's no formal or informal program of systemic performance improvement. We started caring about security because pre-SP3 Windows XP was an existential threat to the business. Our low performance is not an existential threat to the business. See, component owners are generally openly hostile to outside patches: if you're a dev, accepting an outside patch makes your lead angry (due to the need to maintain this patch and to justify in in shiproom the unplanned design change), makes test angry (because test is on the hook for making sure the change doesn't break anything, and you just made work for them), and PM is angry (due to the schedule implications of code churn). There's just no incentive to accept changes from outside your own team. You can always find a reason to say "no," and you have very little incentive to say "yes."'"

Comments

Long story short...

[korbulon]

People at M$ only innit for the money. Microsoft's got good people no doubt, but I am reminded of line from Chef in Apocalypse Now: "They lined us all up in front of a hundred yards of prime rib. Magnificent meat, beautifully marbled. Then they started throwing it in these big cauldrons. All of it. Boiling." That's Microsoft: boiled prime rib.

Re:Long story short...

[Anonymous Coward]

Linux: we'll ship when we think it's the right time. Linus will decide to delay a kernel release on the night before it's released, which is more typical than the exception.

MS: we'll ship cause we told all our billions of customers when we will ship. And from their track record--I'd say 98% of the time they do, hence all the zero day bugs.

IT Managers:
+ Hate that a new Linux kernel breaks all their apps, video drivers, doesn't boot (why is it that nearly EVERY time a linux kernel upgrade resulted in either hours of debugging or reinstalling the distro (easy to do but a pain in the ass, especially on a mission critical system). Then their the driver problems which takes week to figure out which distro/dev (not Linus of course) broke the kernel. WE'VE ALL BEEN THERE.

OR

+ Spend ridiculous amounts of MS training time and wasted effort in prepping for a Windows SP update or moving from Windows NT->XP->Server->7->8. Only to find new headaches in integrating new .NET features. WE'VE ALL BEEN THERE.

Either case, it's hard and that's why IT folks are paid the big bucks compared to a Ice Ring operator. So, I'd say: get over it fellas. Both are good OSes--it's the lesser of 2 evils. Now OSX is another story....

Re:Long story short...

[Austerity Empowers]

People at M$ only innit for the money.

The lesson is a lot deeper, and a lot more pervasive than that. Microsoft is a microcosm of all software and hardware development in the corporate world. There is a fundamental difference that the suits don't understand, and in fact think they are superior to: Linux is developed by people who enjoy what they do, and do it of their own free will and volition, in their own time, in their own way. They are creating something they care about, and invest whatever it takes to make it right. It is scary because they are more or less self managing, it's not always a pretty process, and not always monotonically increasing in quality and value, but the long term trend is positive.

In contrast, employees of a company are doing whatever it takes to make a paycheck, trying to balance the horrors of working for a boss and a program manager for a company they increasingly hate, month by month, for a business unit they probably have hated since they day they were hired, on a goal that they have no investment in (i.e. maximizing shareholder value) in work environments that make insane asylums look like club med. All while being constantly reminded how replaceable they are, frequently while training their overseas replacements or at least being forced to work "along side" them. Then, just when you think it can't be worse, someone in a suit reads a Jack Welch screed and tries some "management technique" he understood poorly and was ill conceived to begin with. If a single employee actually cares at the end of the day, 1) he's being disobedient or is a thief or 2) he's got some good meds.

Good engineers, of any kind, are "applied artists", their medium is physics, computer processors, sheet metal, whatever. We are treated at best like "junior managers", at worst like factory workers. We must eat, so we work in the corporate salt mine but I guarantee you that management is never even seeing a fraction of the potential their dollars are paying for, and honestly it's by their own design. The management that learns how to properly harness that energy (or perhaps rediscovers it), is the one that is going to make millions.

Re:Long story short...

[techno-vampire]

(why is it that nearly EVERY time a linux kernel upgrade resulted in either hours of debugging or reinstalling the distro (easy to do but a pain in the ass, especially on a mission critical system).

Maybe because you don't have enough sense to keep more than one kernel installed? I have three; the current one and the preceding two. That way, if a kernel upgrade breaks something, all I have to do is boot into the one I was using before and go on with my life. And, in fact, that's the default in the two distros I'm familiar with (I use Fedora, and my sister uses Ubuntu.) and I'd be a tad surprised to find that the only reason your boxes don't do that is because you "knew better" and changed it.

Re:Long story short...

[shutdown -p now]

As a Microsoft developer, I find your comment deeply offensive. I am not in it for the money, but for an opportunity to work on interesting stuff together with smart people. It certainly helps when something that you really enjoy doing is also paid well, and they do pay well, but spending 8 hours of your life daily doing a job you hate just for the sake of money is a horrible way to run your life, and I'd never go there.

Re:Long story short...

[Austerity Empowers]

Great rant, except that over 75% of the Linux code contributed is contributed by paid corporate employees that are simply doing their job

I know, my little company has two such people. They're interesting to work with, they have the free will of contractors, the job satisfaction of doing what they'd be doing anyway, they are untouchable because my company cannot succeed without linux street cred and they're well respected in their communities. They care about our company, to be sure, but when we got our second round of funding, and it came with strings attached and a new set of management hell bent on offshoring design, they told them to fuck off, in those exact words, in front of the entire company. They didn't get fired or forced out, unlike four of the software engineers we have on our proprietary management OS who also refused to cooperate. Those guys lived and died by the idiocy of our suits, they are slaves.

Linux devs who get paid by "the man", who have the community support required to make them influential, but the corporate support required to put the food on the table are in a great position. They do not reflect the majority of engineers working for private industry. Their "boss" is the community of people who contribute, who deal with their patches and understand the quality of their work. The company is simply paying them to influence by design, and they're irreplaceable. This is an example of a model that really works. Free men work harder, smarter and longer than slaves.

Linux however is the exception, not everything can follow this model. As I said, management who can learn from this will be rich. The rest will merely scrape by.

Re:your mom is fat

[Anonymous Coward]

its because of her genes though

The quality of Slashdot trolling has gone way down recently.

And the retraction

[caywen]

Sounds like the guy was just frustrated and venting. Lots of us do that sometimes, and this one seems ready made to please the slashdot crowd. But do read the retraction the guy posted.

First, I want to clarify that much of what I wrote is tongue-in-cheek and over the top --- NTFS does use SEH internally, but the filesystem is very solid and well tested. The people who maintain it are some of the most talented and experienced I know. (Granted, I think they maintain ugly code, but ugly code can back good, reliable components, and ugliness is inherently subjective.) The same goes for our other core components. Yes, there are some components that I feel could benefit from more experienced maintenance, but we're not talking about letting monkeys run the place. (Besides: you guys have systemd, which if I'm going to treat it the same way I treated NTFS, is an all-devouring octopus monster about crawl out of the sea and eat Tokyo and spit it out as a giant binary logfile.) ...

Re:And the retraction

[fredprado]

M$ is losing more and more of its competent developers to Google, Apple and other companies. It is in no position to start firing people for Internet posts. And no, they can't take legal action against him because of this post. The post was very generic, no specific information was disclosed.

Re:And the retraction

[epyT-R]

and with attitudes like yours, freedom dies a little more, and society becomes a little less tolerant of the truth, instead bottling up the incongruence between it and politics til the pressure blows out via the next weak link.

Re:And the retraction

[just_a_monkey]

It is not his postings that embarrass his manager. It is not fixing the problems that does that.

Re:And the retraction

[SomeKDEUser]

It is because of people who think like you that there is an isotherm of truth in companies. The grunts know. The middle managers suspect. The upper echelons are completely out of it. Eventually, the company drowns in its own shit.

It needs not be like that: this dev did a great and good thing for MS: it made it impossible to ignore the truth. The truth was always there, knowing it changes nothing, except that you can now work on it.

Incidental rant: private sector companies are much better at covering up the shit than public sector ones, thus the myth of private efficiency. But in reality, it is all about people like you who cover up the shit -- whereas in the public sector, you always have some incentive to uncover the problems: you can always blame your predecessor for political gain.

Poor Management

[rudy_wayne]

All of the problems listed there are the direct result of poor management.

Accepting an outside patch makes your lead angry because . . . .
makes test angry because . . . .
and PM is angry because . . . .

There's just no incentive to accept changes from outside your own team.

When this happens, the manager who is in charge of all those people steps in and says "You will co-operate and get things done, or else you will no longer work here". Sadly, too many managers are too lazy and/or gutless to do this.

Re:Poor Management

[centipedes.in.my.vag]

The module-style build is a core part of Linux, not a weakness. If you're looking for a cohesive build style, you should try *BSD.

Re:i know real reason

[jones_supa]

It is possible that you can make some kind of pretty good hybrid tablet/desktop OS if you thoroughly plan and execute it really well. We cannot fully know. The problem is only that Windows 8 is way too far from such vision. They just released a hacked Windows desktop with this Metro screen thingy taped on it. Everything is all over the place with no good integration and smooth workflow. There is no posh: the graphics are only sharp squares with plain colors. It feels like a tech concept demo thrown together over a weekend.

Re:I regretted submitting this story immediately.

[symbolset]

PS: On the other hand, enough people voted this to the front page...

I have discovered recently that it's not as hard to get a submission accepted to the /. front page as it once was.

Re:Not exclusively Microsoft problem

[router]

Dilbert-esqe. Change the details and its the last two "insert Big Co Name Here" jobs I spent a decade and a half at.

andy

Re:I regretted submitting this story immediately.

[ShanghaiBill]

It is no big deal. It is a non-story. Everything described is not "Microsoft", it is human

Yes it is a story, and it is interesting. Of course it is human nature, and all organizations have these problems. But successful organizations overcome these problems. Organizations that don't overcome these problems fail ... except for Microsoft. What makes Microsoft so fascinating, is that it is only successful because of some early chance opportunities that allowed it to establish customer lock-in, and this has allowed it to succeeded despite being utterly dysfunctional. Microsoft has not only failed to overcome these human problems, but has wallowed in levels of backstabbing, empire building, and technical incompetence that would have destroyed any less endowed organization. Anyone interested in organization behavior should look at Microsoft as a fascinating outlier that breaks all the rules, yet still survives.

This is True Of All Companies

[Greyfox]

I never see anyone working to improve the quality of already working code. Unless it's actually not doing the job intended, no one will ever revisit that code. If something needs to be added to it, they'll go do that and everyone will probably hate working with the code because the design sucks, but no one will think to improve the design. If you told them they could, they'd look at you like some strange alien monster.

If a project has enough churn, you can actually justify cleaning up design, interfaces and even entire subsystems in some cases. If all you do is make each piece of code you touch suck just a little bit less, you'll hate having to work on that code less and less over time. All you have to do is look at the code and think "it doesn't HAVE to be this way!" If that old application everyone hates has gotten to the point where it requires a full time position just to maintain it, there's usually no reason why the design couldn't be improved along the way. My goal in maintenance positions is to eliminate the need for that job. There'll always be SOMETHING that needs maintenance, so I don't feel bad about doing so.

Re:i know real reason

[peragrin]

exactly.

Take the ribbon, love it or hate it, if you really look at it the ribbon all it does is change the shape of the menu system of earlier versions of office. The exact same dialog boxes are there behind the scenes, showing up when you least expect them.

Even in windows 8 if you look around you can find the old windows 9X series dialog boxes and components in the seldom accessed areas. They are slowly being phased out but they are still there.

Re:You cannot kill the hosts file

[armanox]

Actually I remember reading that Windows 8 ignores certain entries in the hosts file - this was an article a while back on Slashdot.

The Iron Law of Bureaucracy runs Msoft

[buybuydandavis]

From Pournelle's web site:

Pournelle's Iron Law of Bureaucracy states that in any bureaucratic organization there will be two kinds of people":

  First, there will be those who are devoted to the goals of the organization. Examples are dedicated classroom teachers in an educational bureaucracy, many of the engineers and launch technicians and scientists at NASA, even some agricultural scientists and advisors in the former Soviet Union collective farming administration.

Secondly, there will be those dedicated to the organization itself. Examples are many of the administrators in the education system, many professors of education, many teachers union officials, much of the NASA headquarters staff, etc.

The Iron Law states that in every case the second group will gain and keep control of the organization. It will write the rules, and control promotions within the organization.

*** Finding a way to effectively deal with bureaucratic capture of institutions is probably the number one human problem.

Re:The Iron Law of Bureaucracy runs Msoft

[gweihir]

Indeed. And the bureaucrats believe firmly they the more time of others they can "bind" (i.e. make them waste it against their wills by processes, forms, committees, etc.), the more important they are. These people are natural parasites that try to take over the host slowly, but permanently. Unfortunately they are typically slow enough that the host takes a long, long time dying. Prime example: The former USSR.

While bureaucrats are amoral (i.e. devoid of any morality and ethics, be it good or bad, and as such do not strictly qualify as human, same as many politicians), fighting them is a moral imperative for everybody not utterly immoral.

Re:I'm sure this is on the money, but

[gbjbaanb]

But you'd expect a company that is joined-up, has significant managerial talent and expects to produce a good, core product to do a little better than continually produce internally-incompatible extras - what he said about cmd.exe not being upgraded and getting powershell instead rings too true for everything at Microsoft (he did forget cscript that appeared in between them, and no doubt there will be another one sooner or later). The same definitley applies to serious system components, I know the dev div wrote WPF/Xaml becuase they just didn't want to work with the Windows team - think about that, a graphics display system that sits on top of Windows and appears to all Windows APIs as a black-dialog-box. things like that need to be part of the core system. not something totally incompatible slapped on top. And that's not the only one.

I understand Sinofsky got this abd tried to make things work, but I wonder how much politics supporting the status quo got in the way there and did for him? That's the biggest problem Microsoft has today - not technical but organisational.

Unix vs MS

[Anonymous Coward]

From http://blog.zorinaq.com/?e=74, Bill Pytlovany wrote

He understood, as I have always taught, just changing the text of a string could affect your compiled code in ways you’d never imagine.

This statement encapsulates the entire difference between an MS Programmer and a Unix Programmer.

Unix programmers don't buy into this sorta voodoo, hocus-pocus nonsense. Really.
Every time I see a statement like that from an "experienced" MS programmer, I want to scream!

It's not meant to be a personal attack against Bill, either. But consider C# has absolutely no way to tell where you
are in an application - just a basic little thing like __LINE__ and __FILE__, and you see that that is the Microsoft core
attitude to their product. You can't always run the "wonderful" VS development environment on you target platform;
some developers use logging as their only means of identifying and correcting bugs. And, because it's from MS,
people really believe this is the best way to do things. I'm not a MS hater, but it's their (Bill's) subtle arrogance that
is essentially the ruin of the industry.

Very Sad.

Annon over tor, got to be true

[letherial]

The first rule of the internet is that everything is true, and that if somone says they are something then it must be so, like me, im a astronaut on a super secret mission to be the first to land on mars, go me...we launch in 2017, but you wont know it, it will be disguised as a satellite launch.

I must say though, anonymous person posting over tor does leave a little credibility to be desired. Of course i remember the first rule of the internet, how can i not, its the only truth out there, but if i was truly a critical thinking person id be forced to spend a little bit of time wondering if this was just some jackass trying to get attention, or someone truly from Microsoft. I mean, clearly he is from Microsoft cause he speaks with so much unproven knowledge, he sounds good right? that must mean he is telling the truth, no way he would lie, right? nobody lies on the internet, why should they, being anonymous means you always tell the truth right? anyways, i got to go train now, going to be the Neil Armstrong of mars, to bad nobody will ever know about it but ill probaly get on tor and write a article about it since its annon people will believe me and make a big article and ill be supper popular.

That's not at all true

[SuperKendall]

In contrast, employees of a company are doing whatever it takes to make a paycheck

That is totally wrong. Many programming employees at companies ALSO enjoy what they do. They are ALSO good programmers.

But as this article attests to, what they cannot do is influence code outside the group they are in, even if they have access to it. So the effect they can have, even if they are very good, is often reigned in a great deal beyond what it could be.

The reason Linux does so much better is because restraints are based on ability, not on arbitrary non-technical boundaries.

Re:NTFS

[hairyfeet]

What is wrong with it? Fuck what is wrong with any of it? The Linux guys can laugh at the registry and shit but ya know what? I can actually patch Windows and not have the drivers shit on by devs that don't give a fuck about anything but "works for me!" I have a machine in the shop I'm retired after NINE YEARS of being my netbox...NINE fucking years. That is TWO service packs and probably over 4000 patches and NOT A SINGLE DRIVER BROKEN, not a single one! hell you can't even update Linux without the wireless or sound being shit all over.

When you can show me ONE distro, just one, that can pass "The Hairyfeet Challenge"* then you have something to brag about but until then Linux will stay last place for a REASON, because normal folks aren't gonna deal with dead wireless, sound, graphics getting screwed, and a million other pains in the ass because linus the arrogant ass torvalds thinks he is fucking smarter than every OS designer that has ever lived and can't build a driver interface that works.

*.- For those that don't know "The Hairyfeet Challenge" simulates the typical 5 year cycle of your average PC, we take one random laptop and one random desktop out of the pile, we install ANY distro release from 5 years ago and we update it to current. Wanna guess what happens when you hold Linux up to just HALF the Windows lifecycle? it DIES, it DIES HARD, it shits all over its drivers and by the end you'll be lucky if even 30% of what was working at the start is 100% functional at the end.

We all know what the definition of insanity is and that is the Linux driver model, 20 god damned years of forum hunts, googling for fixes, shit breaking in Foo+1 that worked in Foo and kinda but not really being fixed in Foo+2 before its fixed in Foo+3 only to have something else shit on. Its 2013 guys, that shit is NOT gonna fucking cut it which is why the ONLY gains after 20 years has been Android where a big corp gave a finger to the devs and brought some sanity to the driver model.

If you want the masses to accept you then you are gonna have to stop taking shit sammiches from Torvalds and demand he fix it or step down for somebody who will. Don't you DESERVE better? Do you really think so little of yourself and your community that "free equals shit" is just fine and dandy to you? I mean MSFT has released an OS more hated than Vista, ME, and Bob rolled together and you are gaining NO SHARE...if that doesn't slap some reality into you then i don't know what will. Hell at this rate Win 9 could be "Win Goatse in smell-o-rama" and it would still sell 60 million copies because while it would smell terrible at least the fucking drivers would work.

Re:NTFS

[hairyfeet]

Ya know why? Because they CAN't deny it, because I can fricking wallpaper this post in "update foo broke my driver" posts and they KNOW this, so all they can do is throw a tantrum like a petulant child or say "Ur a bad mean old poo poo head". You see i'm their worst fricking nightmare...A retailer that has actually USED their product and ended up seeing the same thing Dell saw which is their driver model is BROKEN. I mean here is Dell, one of the largest retailers of PCs on the entire fricking planet, and they were selling...what? maybe a dozen different Linux units MAX? And they couldn't even keep drivers functional on a dozen units, maybe 8 fricking pieces of hardware all told?

The sad part is unlike the koolaid drinkers I WANT Linux to get better, I WANT Linux to pass the challenge, I WANT Linux to be a functional OS...but its not. Its really not. And notice how quickly they dismiss my challenge? If their product works and they believe in it... what are they afraid of? hell i even tilt the test in their favor which anybody doing a legitimate test should NEVER do by only giving them HALF the support cycle of Windows, yet all they can do is insult and try to make me out to be a bad man...why? Why are they so afraid if their product works?

Because it doesn't work and they KNOW this, but like any religious zealot (which is why I call 'em FOSSies instead of Freetards) that is told something against dogma it just can't be true, the scripture HAS to be right. this is why I've blocked the Linux articles from my feeds and have learned to ignore them as its always the same routine, "ur a sekret M$ ninja", "let's move the goal posts", "lets talk about web servers" when the topic is the desktop, its the entire circle of loon bit. Know what I find truly depressing and sad? Go to Linux TM Repo, which was set up as a fricking joke site to lampoon the FOSS zealots, and look at their top 20 Linux TMs, go on I'll wait...there you have just seen EVERY argument they have right there, from "it works for me!" to "Linux runs on supercomputers" its just the same excuses, the same insults, they simply can't face reality so instead of doing something about the situation they just insult all those that don't drink the GNUaid.

You watch they will probably downmod us both to hell rather than even attempt to come up with a counter argument,its because they can't. They piss and moan and make excuses like "companies don't support us!" yet I can take my HD4850 graphics card, uninstall the latest driver, and install the nearly 5 year old driver that came on the disc and it WILL work, when you can't even take the driver from last year and use it on the latest kernel thanks to the stupid ass backwards way Linus set up the driver model. I mean here it is 2013 and a company STILL can't just put a penguin on the God damned box because the drivers on the CD won't even work before it gets to the shelves...how pathetic is that?

Re:NTFS

[AmiMoJo]

I'm sceptical. All we have is the word of this one guy, who for all we know might just be an idiot who doesn't understand what he is looking at.

NTFS seems to be fairly robust. I'm sure someone will chip in with an anecdote about how it screwed up and they lost all their data, but even back in the XP/Vista days I used to replace about 50-60 HDDs a month for customers and as long as the drive wasn't totally dead NTFS was usually readable and recoverable. You don't hear credible reports of fatal data loss bugs or corruption issues, and Microsoft doesn't often issues patches for it because it doesn't seem to have many issues.

Of course the codebase may still be horrible, but since that is usually a guarantee of instability and flaws in what is a critical part of the operating system it does seem somewhat unlikely.

Reverse hairyfeet !

[DrYak]

When you can show me ONE distro, just one, that can pass "The Hairyfeet Challenge"* then you have something to brag about.

*.- For those that don't know "The Hairyfeet Challenge" simulates the typical 5 year cycle of your average PC, we take one random laptop and one random desktop out of the pile, we install ANY distro release from 5 years ago and we update it to current. Wanna guess what happens when you hold Linux up to just HALF the Windows lifecycle? it DIES, it DIES HARD, it shits all over its drivers and by the end you'll be lucky if even 30% of what was working at the start is 100% functional at the end.

Well I for one, are introducing the "Reverse Hairyfeet Challenge".
You do the same with Windows. But with one little specific detail: you do it from the point of view not of a corporate user, but a at-home end-user.

So you try surviving going all the way from Win98, all the way though WinME, and end up with Windows XP Home. See if you can keep you sanity going through this mess.
(I could have been even worse, I could have asked to start the challenge at Windows 3.11 and end-up at Vista, but I would probably get arrested for violating international laws against torture just for suggesting this).
And even if you managed to keep sanity you would probably not keep the hardware: at each major jump you'll end up noticing that your hardware is from a noname aisan manufacturer who since long went belly up and didn't bother writting drivers for the newer OS architecture. Requiring you to buy another piece of hardware from another manufacturer).

For the record, the laptop on which I am writing this is happily running opensuse for more than 2 years now, each update being done simply by live-updating to the newer version - while the distro is still running and used at the same time.
And 2 years ago, this laptop wasn't installed clean from scratch. I simply carried over the disk content from its predecessor. (Yup, try doing that with windows without entering a world of pain: you take a running Windows XP from one laptop, then yank out the disk, plug it into another laptop, and have it start. On linux, its mostly without problems. On Windows, your only hope is to clear huge part of the registry and configuration, to put it back into a "fristboot mode" where all the hardware is scanned again).
And I've got probably desktop carrying over the same installation for much longer. I think the jump from 32 to 64 bits was the last time I did a fresh install, then kept simply ugrading over.