Code Quality In Open and Closed Source Kernels

Diomidis Spinellis writes "Earlier today I presented at the 30th International Conference on Software Engineering a research paper comparing the code quality of Linux, Windows (its research kernel distribution), OpenSolaris, and FreeBSD. For the comparison I parsed multiple configurations of these systems (more than ten million lines) and stored the results in four databases, where I could run SQL queries on them. This amounted to 8GB of data, 160 million records. (I've made the databases and the SQL queries available online.) The areas I examined were file organization, code structure, code style, preprocessing, and data organization. To my surprise there was no clear winner or loser, but there were interesting differences in specific areas. As the summary concludes: '..the structure and internal quality attributes of a working, non-trivial software artifact will represent first and foremost the engineering requirements of its construction, with the influence of process being marginal, if any.'"

Comments

Too few kernels studied.

[jd]

OpenSolaris is hybrid open/closed. Solaris was originally closed, so the style and quality control will still be at least partially along closed lines, but is now open so will contain some style and quality control aspects of open source. Without knowing the ratio, however, it is not a particularly useful kernel to study. It is most useful as a sanity check, as it must be somewhere between the two styles, but since you cannot define where between, it has no value in the initial analysis.

This leaves just one closed-source kernel and two open source kernels. If you were to perform a comparitive study between drugs on the basis of three subjects, you'd be laughed out of the research field so fast you'd hit escape velocity and be the first human on Mars.

Now, what OS' could potentially be studied? This depends on who can be talked into supplying software source and whether the researchers are willing to sign NDAs. I'd argue that you want at least three OS' in the closed and open categories, and two in the hybrid - one in each direction.

The closed-source OS' you'd want to examine - if you could get the code - would be Windows (as is already the case), LynxOS-178 (a moderately battle-hardened RTOS) and Trusted IRIX (massively hardened secure OS). In principle, the three design methods and requirements are different to such an extreme that this will give you the best possible picture of the range of code development practices. The idea would be to not test code that would have had similar constraints, because that only tells you about one aspect of software design. If, on the other hand, you pick three points that are more-or-less on different extreme points on the boundary of closed source design, you can get a picture of what that boundary might look like. Substituting any RTOS and Trusted OS for LynxOS-178 and Trusted IRIX respectively would work just as well, the idea is to get the extremes, not get specific implementations of those extremes.

The open-source OS' would include Linux and one of the BSDs - but it has to be a BSD that follows a markedly different ideology, so OpenBSD might be a good choice - but would probably also include one of the BeOS clones and/or a research OS like ExoPC. Again, the idea is to mark out the boundary as carefully as possible. The more data points you have, from points as far apart as possible, the better the understanding of that boundary.

What you will end up with is not a handful of values that have no meaningful basis for comparison, but two overlapping shapes. It'll look something like a Venn diagram. Hybrid methodologies will not merely appear somewhere between the two furthest extremes between the two shapes, but must exist only within the intersection between the two groups, and only there. No other midpoints would be valid.

To test this, we'd use a selection of hybrid designs. Plan9 started closed but has been open the longest, so I'd expect that to be in the intersecion but closer to the midpoint of the open source group than the midpoint of the closd source group. OpenSolaris has been open a relatively short time, so I'd expect the converse to be true for that.

This would still not be a great study, but it would increase the number of data points and provide some measure of control over the experimet. Since you cannot sensibly have a control group in this, the only controls you can assert are those of ensuring the samples are representative of the diversity in techniques, ensuring that the software is correctly classified, and ensuring that the results are sanity-checked.