Slashdot Mirror
Code Quality In Open and Closed Source Kernels
Diomidis Spinellis writes "Earlier today I presented at the 30th International Conference on Software Engineering a research paper comparing the code quality of Linux, Windows (its research kernel distribution), OpenSolaris, and FreeBSD. For the comparison I parsed multiple configurations of these systems (more than ten million lines) and stored the results in four databases, where I could run SQL queries on them. This amounted to 8GB of data, 160 million records. (I've made the databases and the SQL queries available online.) The areas I examined were file organization, code structure, code style, preprocessing, and data organization. To my surprise there was no clear winner or loser, but there were interesting differences in specific areas. As the summary concludes: '..the structure and internal quality attributes of a working, non-trivial software artifact will represent first and foremost the engineering requirements of its construction, with the influence of process being marginal, if any.'"
Or the summary is completely incomprehensible?
Of course, I could try to RTFA, but hey, this is Slashdot, after all...
No sig for the moment.
That you have neither capitalized on your shared synergies, nor have you recovered your cherished paradigms.
Oh. Wait. This is about propeller-head stuff rather than management stuff. Lemme get my "Handbook of postmodern buzz words"...
"If god did not exist, it would be necessary to invent him" --Voltaire
Which of the samples were closed source? And........ how did he get a hold of it/them? Open source software is still open even if it came from microsoft.
Final line in the paper: "Therefore, the most we can read from the overall balance of marks is that open source development approaches do not produce software of markedly higher quality than proprietary software development."
Interesting, but not shocking for those who have worked with disciplined commercial teams. I wonder what the results would be in less critical areas than the kernel, say certain types of applications.
..that Open Source code is of quality, but at least the point of things like the GPL is that you have the power to change that, and improve that code..
"The OpenSolaris kernel was a welcomed surprise: it was the only body of source code that did not require any extensions to CScout in order to compile."
Given that the Solaris kernel has been compiled by two very different compilers (Sun Studio, of course, and gcc), it isn't that surprising. Because of the compiler issues, it is likely the most ANSI compliant of the bunch.
If I am understanding correctly, you were looking for 'winners' and 'losers' (weasel words in and of themselves, but anyway...) in terms of 'quality' (another semi-subjective term that could make someone go crazy and drive motorcycles across the country for the rest of their lives).
You found that '..the structure and internal quality attributes of a working, non-trivial software artifact will represent first and foremost the engineering requirements of its construction, with the influence of process being marginal, if any.' -- or in plain English: "the app specs had a much bigger influence when compared to internal efficiencies".
I would wonder if you're just seeing a statistical wash-out. Are you dealing with data sets (tens of millions of lines and thousands of functions) that are so large, that patterns simply get washed out in the analysis?
Oh dear, my post is no more clear than the summary...
davejenkins.com |
Um, he already did plenty. I'm sure there's room for a follow-up paper along those lines. Quite possibly by someone else - I'd guess you're up to the task, considering how dismissive you were of his efforts?
Well, you lose your bet, been over five minutes and no anti Microsoft screeds let alone spelling it with a $.
Just so everyone understands, the tactic used here is known as "Poisoning the well." The idea is the discredit an argument's source before the argument is presented. Here, our AC friend is trying to ward off criticism of Microsoft by insinuating that anyone who does so is a 13 year old "Slashbot."
The fallacy is in the fact that even someone who is 13 and often goes along with the Slashdot zeitgeist may still have legitimate criticisms of Microsoft, such as the fact that Microsoft sucks giant hairy donkey balls.
- None can love freedom heartily, but good men; the rest love not freedom, but license. -- John Milton
I'm sorry, but if this is what passes for serious academic computer-science work, close the schools. This all appears to boil down to: quality code (definition left to the reader) is produced by good programmers (can't define, but I know one when I see his/her code) who are given the time to produce quality code. Rushed projects by teams of average-to-crappy programmers results in low-quality code. All the tools and management theories in the world have little impact on this basic fact of life. My PhD, please?
So while looking at the data collected, I had to wonder if some of the conclusions reached were not something of a matter of weighting - I saw some things pretty troubling about the WRK. Among the top of my list was a 99.8% global function count!!!
This would explain some things like lower LOC count - after all, if you just have a bunch of global functions there's no need for a lot of API wrapping, you just call away.
I do hate to lean on LOC as any kind of metric but - even besides that, the far lower count of Windows made me wonder how much there, is there. Is the Windows kernel so much tighter or is it just doing less? That one metric would seem to make further conclusions hard to reach since it's such a different style.
Also, on a side note I would say another conclusion you could reach is that open source would tend to be more readable, with the WRK having a 33.30% adherence to code style and the others being 77-83%. That meshes with my experience working on corporate code, where over time coding styles change on more of a whim whereas in an open source project, it's more important to keep a common look to the code for maintainability. (That's important for corporate code too - it's just that there's usually no-one assigned to care about that).
"There is more worth loving than we have strength to love." - Brian Jay Stanley
the worst for what applications?
If good code and bad code were a simple automated analysis away, don't you think everyone would be doing it? What methodolgy could possibly give a quantitative weighting for "quality"?
"To my surprise there was no clear winner or loser..." Not really a surprise at all, actually.
People make claims about the need for closed source all the time, usually revolving around the need to a predictable level of quality, or some other factor. The fact is, this results proves that its a wash whether you choose open or closed--so why not choose open?
There's a deep significance here I'm failing to capture completely. Someone else word it better if they can. But there didn't need to be some blow-out victory of open source over closed source for this to be a victory. All open source needed to do was compare--which it did, clearly--with closed source, in terms of value, to secure its worth.
The way you choose to license your software doesn't coralate with software quality... Seems logical to me. As how you license your software has very little to do about the code inside the OS.
Closed Source Developer: I will try to do my best job as I possibly can so I can keep my job and make money because that is what I value.
Open Source Developer: I will try to do my best job as I possibly can so I can help the comunity and feel better about myself/get myself noticed in the comunity/Something cool to put on my resume... because that is what I value.
People who choose to license their software OpenSource vs. Closed Source says nothing about their programming ability. There are a bunch of really crappy GNU projects out there as well as a bunch of crappy closed source projects... Yea there is the argument of millions of eyes fixing problems but really when you get millions of people looking at the same thing you will get good and bad ideas so the more good ideas you get the more bad ideas you get and the more people involved the harder it gets to weed out good ones and bad ones. Closed source is effected often by a narrow level of control where bad ideas can be mandated.... All in all everything really ballances out and the effects of the license are negledgeable.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
I haven't seen anybody else comment on the fact that the statement that the quality of the code had more to do with the engineering than the process through which the code was developed is quite interesting.
;)
From my personal experiences, it typically seems code is written to solve a specific need. Said another way, in the pursuit of solving a given problem, whatever engineering is required to solve the problem must be accomplished - if existing solutions to problems can be recognized, they can be used (for example, Gang of Four/GOF patterns), otherwise, the problem must have a new solution engineered.
Seeing as how there are teams successfully developing projects (with both good, and bad code quality) using traditional OO/UML modeling, the software development life-cycle, capability maturity model, scrum, agile, XP/pair programming, and a myriad of other methods, it would seem to be that what the author is saying is, it didn't necessarily matter which method was used, it was how the solution was actually built (the.. robustness of the engineering) that mattered.
Further clarification on the difference between engineering and "process" would strengthen this paper.
I went to a Microsoft user group event some time ago - and the presenter described what they believed the process of development of code quality looked like. They suggested the progression of code quality was something like:
crap -> slightly less crappy -> decent quality -> elegant code.
Sometimes, your first solution at a given problem is elegant.. sometimes, it's just crap.
Anyways, just my two cents. Maybe two cents too many..
SixD
Don't feed the troll, compadre!
No sig for the moment.
"Linux excels in various code structure metrics, but lags in code style. This could be attributed to the work of brilliant motivated programmers who aren't however efficiently managed to pay attention to the details of style. In contrast, the high marks of WRK in code style and low marks in code structure could be attributed to the opposite effect: programmers who are efficiently micro-managed to care about the details of style, but are not given sufficient creative freedom to structure their code in an appropriate manner. "
How ever I was left wondering how it was possible to compare fairly? He already stated:
"Excluded from the kernel code are the device drivers, and the plug-and-play, power management, and virtual DOS subsystems. The missing parts explain the large size difference between the WRK and the other three kernels."
and reading I see even more of the drivers aren't there:
"The NT Hardware Abstraction Layer, file systems, network stacks, and device drivers are implemented separately from NTOS and loaded into kernel mode as dynamic libraries. Sources for these dynamic components are not included in the WRK. "
http://www.microsoft.com/resources/sharedsource/licensing/researchkernel.mspx
So it's not like for like. Maybe you would draw different conclusions if it was, maybe the Linux style issue is because of all the drivers the WRK lacks. So even though I think his conclusion sounds probable, I don't feel I can state it as so with any confidence.
Interesting, but not shocking
Considering that few open source developers are strictly open source, that's hardly a surprise. I'd be willing to bet many open source developers are also part of disciplined commercial teams.
The flip side of that coin is just as intriguing. Open source development models don't produce software of notably inferior quality either. That should send a shivey through Castle Redmondore.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
The metrics used in this paper are lame. They're things like "number of #define statements outside header files" and such.
Modern code quality evaluation involves running code through something like Purify, which actually has some understanding of C and its bugs. There are many such tools. This paper is way behind current analysis technology.
Sorry, I've been in the business for over 25 years and had to hear one pin head after another spout about code quality or productivity. Its all subjective at best.
The worst looking piece of spaghetti code could have fewer bugs, be more efficient, and be easier to maintain than the most modular object oriented code.
What is the "real" measure of quality or productivity? Is it LOC? No. Is it overall structure? no. Is it the number of "globals?" maybe not.
The only real measure of code is the pure and simple darwinian test of survival. If it lasts and works, its good code. If it is constantly being rewritten or is tossed, it is bad code.
I currently HATE (with a passion) the current interpretation of the bridge design pattern so popular these days. Yea, it means well, but it fails in implementation by making implementation harder and increasing the LOC benchmark. The core idea is correct, but it has been taken to absurd levels.
I have code that is over 15 years old, almost untouched, and still being used in programs today. Is it pretty? Not always. Is it "object oriented" conceptually, yes, but not necessarily. Think the "fopen,"fread," file operations. Conceptually, the FILE pointer is an object, but it is a pure C convention.
In summation:
Code that works -- good.
Code that does not -- bad.
It's a well known fact that code will always resemble the institution that produced it, to some extent. To describe the Microsoft code as "poorly structured" is likely a bit out of touch.
The absolutely best kernel code is generally extremely beautiful and descriptive when dealing with the system's abstracts (with nice, long descriptive names for each function) and then unbelievably hellish and ugly in the sections that deal with hardware. Kernels represent an intersection between the idealistic system code and the hideously complex and inhuman machine interaction code. For this reason, we gauge the value of the systems based on how cleanly they compile into assembly, their performance, and ideally how well they do what they were written to do.
Kernel code fills such a complex role in the computer science paradigm that it is likely impossible to gauge the value or quality of any of them through any sort of automated means. What we have here is a mess of a research paper that comes to no obvious conclusions because they didn't really discover anything. If it were of any value, its final summary and conclusions wouldn't be so obfuscated. The researcher may or may not have mastered the art of understanding the zeitgeist of kernels but he certainly hasn't mastered the research paper.
People who choose to license their software OpenSource vs. Closed Source says nothing about their programming ability. There are a bunch of really crappy GNU projects out there as well as a bunch of crappy closed source projects... Yea there is the argument of millions of eyes fixing problems but
You hit the nail on the head, there are millions of eyes looking at the code but how many of those eyes are in the heads of idiots? Probably more than anyone thinks- how many people still buy Britney Spears albums (or ever have)? There are plenty of idiots in the world and every open source project has the same proportion of them... not saying that closed source is any better but this failed argument of millions of eyes is such propaganda... it does nothing to help OSS by perpetuating that myth
By the time you get a car/plane that's accepted by the "market", the insides are going to be of a certain quality, no matter what the process used.
:)
;).
Did I get that right?
So now the question is how long did it take to get that level of quality. And maybe there is a difference in quality but the measurement used is not sensitive enough, or not as appropriate or his conclusion isn't quite correct - he measured a difference, just didn't show in his conclusion
Statements like this: "Indeed the longest header file (WRK's winerror.h) at 27,000 lines lumps together error messages from 30 different areas; most of which are not related to the Windows kernel." Allow me to feel smug in my anti-Windows bias
Awesome furniture, accessories and cabinetry in Santa Rosa, CA: http://humanity-home.com/
This leaves just one closed-source kernel and two open source kernels. If you were to perform a comparitive study between drugs on the basis of three subjects, you'd be laughed out of the research field so fast you'd hit escape velocity and be the first human on Mars.
Now, what OS' could potentially be studied? This depends on who can be talked into supplying software source and whether the researchers are willing to sign NDAs. I'd argue that you want at least three OS' in the closed and open categories, and two in the hybrid - one in each direction.
The closed-source OS' you'd want to examine - if you could get the code - would be Windows (as is already the case), LynxOS-178 (a moderately battle-hardened RTOS) and Trusted IRIX (massively hardened secure OS). In principle, the three design methods and requirements are different to such an extreme that this will give you the best possible picture of the range of code development practices. The idea would be to not test code that would have had similar constraints, because that only tells you about one aspect of software design. If, on the other hand, you pick three points that are more-or-less on different extreme points on the boundary of closed source design, you can get a picture of what that boundary might look like. Substituting any RTOS and Trusted OS for LynxOS-178 and Trusted IRIX respectively would work just as well, the idea is to get the extremes, not get specific implementations of those extremes.
The open-source OS' would include Linux and one of the BSDs - but it has to be a BSD that follows a markedly different ideology, so OpenBSD might be a good choice - but would probably also include one of the BeOS clones and/or a research OS like ExoPC. Again, the idea is to mark out the boundary as carefully as possible. The more data points you have, from points as far apart as possible, the better the understanding of that boundary.
What you will end up with is not a handful of values that have no meaningful basis for comparison, but two overlapping shapes. It'll look something like a Venn diagram. Hybrid methodologies will not merely appear somewhere between the two furthest extremes between the two shapes, but must exist only within the intersection between the two groups, and only there. No other midpoints would be valid.
To test this, we'd use a selection of hybrid designs. Plan9 started closed but has been open the longest, so I'd expect that to be in the intersecion but closer to the midpoint of the open source group than the midpoint of the closd source group. OpenSolaris has been open a relatively short time, so I'd expect the converse to be true for that.
This would still not be a great study, but it would increase the number of data points and provide some measure of control over the experimet. Since you cannot sensibly have a control group in this, the only controls you can assert are those of ensuring the samples are representative of the diversity in techniques, ensuring that the software is correctly classified, and ensuring that the results are sanity-checked.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
Isn't NetBSD the system filled with academics who insist upon clean, manageable, and portable code above all other standards? Too bad the NetBSD kernel didn't get judged here, I suspect it would have taken the cake.
I still recall this exhaustive report comparing several kernels' performance back in 2003 in which NetBSD pretty much beat the pants off of everybody else (note the two updates with separate graphs). The initial poor performance was due to an old revision, and upon seeing that there were some places in which the newer revision wasn't so hot, the developers fixed them and in only two weeks, NetBSD beat out FreeBSD on every scalability test. Their pragmatism and insistence on quality code finally paid off.
Ever since seeing those charts, I've been waiting for Debian/NetBSD to come out...
Use my userscript to add story images to Slashdot. There's no going back.
Oops, it needs a trailing slash: http://bulk.fefe.de/scalability/
Use my userscript to add story images to Slashdot. There's no going back.
wtf.n0x.org
You: "There's obviously a problem with a study that takes 8GB of data and concludes that there's no difference in quality between kernels with legendary uptimes and those that can't manage memory well enough to stay up more than a few weeks." From the summary: "The areas I examined were file organization, code structure, code style, preprocessing, and data organization." These have no direct correlation to uptime. Yes, indirect, perhaps, as in "a better-organized kernel is easier to understand, debug, and reason about", but not direct as in "implementing the scheduler in 3 files instead of one guarantees stability." That said, what defines this as an interesting but impractical study? Doesn't it say something that there's something more fundamental than just high-level software engineering principles at work in the relative qualities of kernels?
- Have managers and an oversight group control quality (+)
- Through its bureaucracy remove incentives to find creative solutions to quality problems (-)
- Pay for developers to attend training courses (+)
- Provide a nice environment free of distractions that allows developers to focus on developing quality software (+)
- Buy expensive tools that can detect quality problems (+)
- Developers take their paycheck for granted and loose interest in what they are doing (-)
- Developers write obfuscated code for job security (-)
And here are some possible ways in which an open source development effort can affect (positively or negatively) the quality of the software:- Volunteers are more motivated than paid employees (+)
- Nobody takes responsibility for the overall quality of the code; responsibility is diffused (-)
- Working conditions can be suboptimal (-)
- Developers work part-time (-)
- Developers eat their own dog food and therefore care about their code (+)
- There are many eyeballs to spot code problems (+)
- There are no marketing pressures to deliver substandard work (+)
- Developers are geographically dispersed and can't communicate easily (-)
Both lists can be expanded, and many of the arguments can be refuted. Still you get the idea: the inputs to the two development processes differ substantially and this could affect quality.Most of the criterions used in this paper are syntactic and/or not clearly related with the actual quality of the code.
E.g., measuring the average length of the files or the average number of files in a directory reflects an underlying assumption: software quality is better when granularity is higher. But it is not true.
The reality is that you need an *appropriate* level of granularity in your software, and this also depends on your overall design. It's a little bit nonsense to try measuring such properties of the code as if they were somehow absolute and context-free !
Such criterions and the related underlying assumptions seem very debatable: thus its no wonder that no significant difference can be extracted out of these measurements.
That subjective conclusion is precise effect reading too much into the metrics.
Sun or Microsoft programmers need to support 2 and 2 platforms respectively. (Sun: SPARC and AMD64; M$: IA32 and AMD64). All portability are of boolean complexity.
But FreeBSD and Linux run on dozen of platforms. I do not know how it is in BSD land, but in Linux first and foremost requirement for platform support, is that it has no negative side-effects on other platforms. Consequently, for example, under Linux most (all? - all!!) locking is still implemented as macros: on uni-processor system with preemptive kernel feature disabled all in-kernel synchronization would miraculously (thanks to preprocessor) disappear from the whole code base. To make sure that on such platform, kernel would run as efficiently as possible - without any locking overhead, because all the locking is not needed anymore.
And that's single example. There are many macros for special CPU features: depending on platform it would be nop or asm statement or function call. No way around using macros.
I think one of the points the author needed to factor in, is portability of OS. Without that, most metrics are skewed too much.
P.S. Actually, Linux affinity to macros is often (at least from words of kernel developers) stems from poor optimization of inlined functions in GCC. Many macros can be converted to functions - but that would damage overall level of performance. In many places significantly.
All hope abandon ye who enter here.
That would be Linux for Canucks.
Don't even try. That's twitter you're arguing with, and all he cares about is bashing Microsoft, not about the engineering principles applied within the respective kernels.
For a site about things like basic rights, Slashdot users sure do like to censor "dissent".
Ahh... yeah, I did look at his profile and notice pretty much all of his replies were scored -1.
Not relevant. The NT kernel itself rarely ever crashes (if the kernel crashes, is it even capable of displaying the blue screen?!?) so the crash count of it would be roughly the same as Linux (uncertain about BSD). It's when you add drivers that it starts to destabilise (much like Linux).
For a site about things like basic rights, Slashdot users sure do like to censor "dissent".
what was the most foul comment you encountered :D ? and where did it reside
Ultimately, I don't believe in the religious wars over commercial vs open source. My guess is that good software is developed by clever people in environments that allow them to thrive. That be just as easily be commercial or open source. It's probably easier for the clever people if they get paid since that does grease the skids, but being paid or otherwise is not the key point.
Those statistics are useless; nobody has any idea how these measures translate into correctness, robustness, or performance.
Twitter - YOU said this? You really should be ashamed of yourselves.
Faster! Faster! Faster would be better!
You could have horribly written code that is reliable and bulletproof. OpenSSH comes to mind (unmodded by Debian).
Or you could have beautifully written code that is useless and full of holes. Handed-in college CS homework assignments come to mind.
Forget what you *think* you're measuring (code quality). Instead, consider whether you're measuring anything at all. That is, is there any information in the data you've measured?
Now given your sample space, it is highly unlikely that you'd measure something (anything!) and the measurement comes out more or less the same for every sample. The more likely conclusion, is that you haven't measured *anything*.
The summary means that getting your requirements right is the most important part of the project's life. How you implement those requirements is less important.
Of course, this is just my interpretation; and the meaning of "requirements" is quite fuzzy. For example, if my specs go beyond the scope of a "usual" SRS, and include programming style guidelines, there will also be an impact on how code is written.
--
SRS = Software Requirements Specifications
The saddest poem
If you've got so much data and still can't find differences, it's because your measurements are not informative enough. The only thing the author can conclude is that his way of measuring code quality cannot pick up differences in the part he has labeled "process".
Well, Visual Studio is nice minimalistic development environment. Using it doesn't make my cry out loud in frustration, unlike what happens with most of its commercial competitors. Microsoft can be proud of it.
But of course, for doing real work there is still no alternative to Emacs.
No, not at all. The only modification would be :
BSD Software promotes Competition.
GPL Software promotes Competition & Collusion.
Closed Source Software promotes Collusion.
Semi-automatic amateur armchair Australian philosopher; conjecture ready at any moment...
Recommendation to guy who wasted his time on a study of quality, without having any idea what quality actually is:
Read "Zen and the Art of Motorcycle Maintenance
Further advice:
When your analysis of the geometry of the planet concludes that the Earth is conical, despite obvious prima facie evidence to the contrary, don't announce your discovery that the Earth isn't spherical after all to the world; take another look at your method of analysis.
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
the question is not a troll, though I'm a Linux server and desktop advocate there are applications for which it's the very worst choice. Like serious CADD / CAM, on Linux or BSD? forget about it.....
I think you've missed something. Many (most?) good programmers have a drive to write elegant and correct code. It doesn't matter who pays them, or if they meet their peers by the corporate coffee machine or on a mailing list.
What's specific to corporate code ("open source" or propietary) is at least one thing: a manager can come up to you and say "you will code feature Foo and be done on date Bar", and the programmer will kludge it, even if he knows Foo cannot be done in a robust way, and the best option would be not to try at all.
winerror.h is also shipped with the WDK. It contains all the error codes for the Win32 environment subsystem, which isn't implemented by the kernel. Win32 exists mostly in user mode: although a sizable chunk of the Win32 server process was moved from winsrv.dll into win32k.sys in NT4, which does run in kernel mode, Win32 really isn't part of the kernel. They did that to cut down on IPC overhead, not to integrate it into lower layers. Win32 services can only be used by user mode processes.
ntstatus.h is the header that contains the kernel's error codes. That file is 16272 lines in the WDK 6000 (with 8 comment lines to every code), and contains few codes not relevant to the kernel.
It's not surprising that winerror.h contains messages not related to the kernel because it's not what I'd consider part of the kernel. AFAICT, this file is included only to support win32k.sys, and possibly display drivers. I'd be interested to know how many of the kernel source files actually include winerror.h.
the layout of the bottom half of Eclipse's 'About' dialog is slightly better than the one in VS. apart from that it's a steaming pile of crap. i curse every time i have to use it, which is far too often these days.