Slashdot Mirror
The Effect of Programming Language On Software Quality
HughPickens.com writes: Discussions whether a given programming language is "the right tool for the job" inevitably lead to debate. While some of these debates may appear to be tinged with an almost religious fervor, most people would agree that a programming language can impact not only the coding process, but also the properties of the resulting product. Now computer scientists at the University of California — Davis have published a study of the effect of programming languages on software quality (PDF) using a very large data set from GitHub. They analyzed 729 projects with 80 million SLOC by 29,000 authors and 1.5 million commits in 17 languages. The large sample size allowed them to use a mixed-methods approach, combining multiple regression modeling with visualization and text analytics, to study the effect of language features such as static vs. dynamic typing, strong vs. weak typing on software quality. By triangulating findings from different methods, and controlling for confounding effects such as team size, project size, and project history, they report that language design does have a significant, but modest effect on software quality.
Quoting: "Most notably, it does appear that strong typing is modestly better than weak typing, and among functional languages, static typing is also somewhat better than dynamic typing. We also find that functional languages are somewhat better than procedural languages. It is worth noting that these modest effects arising from language design are overwhelmingly dominated by the process factors such as project size, team size, and commit size. However, we hasten to caution the reader that even these modest effects might quite possibly be due to other, intangible process factors, e.g., the preference of certain personality types for functional, static and strongly typed languages."
Quoting: "Most notably, it does appear that strong typing is modestly better than weak typing, and among functional languages, static typing is also somewhat better than dynamic typing. We also find that functional languages are somewhat better than procedural languages. It is worth noting that these modest effects arising from language design are overwhelmingly dominated by the process factors such as project size, team size, and commit size. However, we hasten to caution the reader that even these modest effects might quite possibly be due to other, intangible process factors, e.g., the preference of certain personality types for functional, static and strongly typed languages."
"We aren't saying that functional, static and strongly typed languages are inherently superior. We're just saying that if you don't prefer them, maybe you aren't really cut out for programming."
The problem with these kinds of studies is that there is no actual way to objectively measure software quality. You can correlate all the data you want, but garbage in means garbage out.
For this study they used two thinfactor gs to determine software quality: one is the number of bugfix commits. Ugh. I'm not even clear if the number of bugfix commits means higher quality, or lower quality. That could go either way. It might mean you had better testers, or that you committed things in small batches, or that you had more branches. The other factor was a natural language processor that read the check-in comments. While this is a really cool idea, you would need a lot of research just to prove that this approach actually works before you can start using the technique to draw conclusions about some other data.
So while this was very cool, and very ambitious, the results are completely meaningless until someone can prove that this technique actually measures software quality at all.