Slashdot Mirror
The Effect of Programming Language On Software Quality
HughPickens.com writes: Discussions whether a given programming language is "the right tool for the job" inevitably lead to debate. While some of these debates may appear to be tinged with an almost religious fervor, most people would agree that a programming language can impact not only the coding process, but also the properties of the resulting product. Now computer scientists at the University of California — Davis have published a study of the effect of programming languages on software quality (PDF) using a very large data set from GitHub. They analyzed 729 projects with 80 million SLOC by 29,000 authors and 1.5 million commits in 17 languages. The large sample size allowed them to use a mixed-methods approach, combining multiple regression modeling with visualization and text analytics, to study the effect of language features such as static vs. dynamic typing, strong vs. weak typing on software quality. By triangulating findings from different methods, and controlling for confounding effects such as team size, project size, and project history, they report that language design does have a significant, but modest effect on software quality.
Quoting: "Most notably, it does appear that strong typing is modestly better than weak typing, and among functional languages, static typing is also somewhat better than dynamic typing. We also find that functional languages are somewhat better than procedural languages. It is worth noting that these modest effects arising from language design are overwhelmingly dominated by the process factors such as project size, team size, and commit size. However, we hasten to caution the reader that even these modest effects might quite possibly be due to other, intangible process factors, e.g., the preference of certain personality types for functional, static and strongly typed languages."
Quoting: "Most notably, it does appear that strong typing is modestly better than weak typing, and among functional languages, static typing is also somewhat better than dynamic typing. We also find that functional languages are somewhat better than procedural languages. It is worth noting that these modest effects arising from language design are overwhelmingly dominated by the process factors such as project size, team size, and commit size. However, we hasten to caution the reader that even these modest effects might quite possibly be due to other, intangible process factors, e.g., the preference of certain personality types for functional, static and strongly typed languages."
e.g., the preference of certain personality types for functional, static and strongly typed languages.
My guess is that this has a bigger impact on most projects than actual features of a chosen language. I was thinking it the whole time I read the summary and then, sure enough, it's mentioned as a disclaimer at the end...
Creationist Textbook Stickers Declared Unconstitutional by CowboyNeal
It's clear that there are more factors here that need to be normalised out. For example, they found that the category that "had" the most performance bugs was the procedural, static, unmanaged memory category, i.e. C, C++ etc, far outstripping languages like ruby. To me, it's clear that that is caused by people using these languages actually caring about performance, while people using languages who's implementations are many orders of magnitude slower, don't really file (or fix) bugs about perf.
it says basically nothing
Wanna buy a shirt?
https://www.redbubble.com/people/stealthfinger/shop?asc=u
"We aren't saying that functional, static and strongly typed languages are inherently superior. We're just saying that if you don't prefer them, maybe you aren't really cut out for programming."
Almost no casual programer uses functional languages and do not tend to be used for large FOSS projects.
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
I would say that there are three other critical factors; which languages beginners chose, which languages are rarely used, and potentially even more importantly which languages become the programmer's only language ever.
If someone is new to programming then their programming is probably going to be poor. So certain languages tend to be "gateway" languages such as PHP, Python, VB(in the past), C#, etc. It is doubtful that someone is going to start out their programming career with the C in OpenCL or Haskell.
I have seen many people learn PowerBuilder and never learn anything else, and while they might master powerbuilder they never really master programming. I have also seen the same thing with accountants who master the VB in excel resulting in some of the strangest agglomerations of code I have ever seen.
But also certain languages are sort of throwaway for many programmers such as whatever the language is inside Make scripts; as most programmers that I know have not mastered make and do what they have to do to get things to compile. The same with bash; I have only met a few programmers who truly knew bash. They did what they had to do and ran away after that.
So it would be very difficult to tease out the quality of a language based on these statistics. But regardless of the results the religious zealots who think their language is the very best and that all others are for children won't be swayed by facts anyway.
The more flexibility and power a language provides, the more opportunities you have to hang yourself with it.
Personally what I hate are loosely, dynamically typed languages. They provide no compile-time checking at all that I can detect, which means that in order to even guess whether the code is "correct" you have to run through all the possible use cases. I realize that it's an ideal to test all possible inputs (especially boundary conditions), but that just isn't practical for most project schedules and budgets.
As powerful as functional languages can be, the restrictions imposed by them can lead to difficulty implementing certain behaviours in the code. In fact, one Erlang project I worked on proved to have such an extreme difficulty implementing an algorithm that we had to cancel the project, even though the rest of the project had been completed. (That function was *the* heart of the system: the scheduling algorithm>)
Much as the researchers discovered, I've never really found the programming language itself to have much of an impact on the code quality or readability of the code if the code was competently written. That said, even the best of languages can be turned into unmaintainable gobbledygook by a dedicated bonehead, especially consultants who know damned well they'll be long gone before the project enters maintenance/enhancement mode.
What I found really degrades quality is not the language, but an overemphasis on code style at some companies. Instead of code reviews focusing on the functionality of the code being reviewed, they spend all their time nit-picking about variable names and whether to use camel-case or underscores.
I consider the maintainability and readability of code to be at least as important as any metrics about the number of bugs in a project. If you can't read and understand the code easily, fixing a bug when it is discovered becomes a hellish nightmare.
I do not fail; I succeed at finding out what does not work.
The KISS-principle is probably the most important thing to keep software quality up, more so than tools and language.
Yes, I'm left. You have a problem with that?
You can code sloppily in any language.
All this tells me is that there's so little difference as it not to be a major factor in your choice of language. As such, other more practical considerations (such as hiring programmers, project time, and even speed of the final code) should take far more precedence than the triviality of what language you happen to use.
As with all things "programming language" - apply them to real language. I'm certain that in some languages, it's easier to mis-speak at a critical moment, or to say the wrong thing, or be misconstrued. I'm also certain that some languages are more prevalent, easier to learn, clearer in their intent and grammar, etc.
But it doesn't mean at any point that you should change what you're doing to the language of the moment, nor that you should choose what language to do a project in taking any notice of the structure or grammar of the language over who you have who can speak it and how well everyone can be understood if they speak it together.
Also, there are languages and dialects that make specific tasks easier (for instance, IT has a language all of it's own, talking about SCSI, buses, cloud, etc.). If everyone is able to "speak the lingo", then that's a good choice, but it's not the be-all and end-all of a good project.
As such, all the programming language discussion is really like saying "We should all speak and write only in Chinese, because the Chinese for death and dearth sound more different and we won't get confused". Don't. Program in the language that you're comfortable with, that the people you hire can read and write fluently, and that is most common and available.
Personally, that's always been C / C99 for me. So I always find it hard to justify the use of other languages except when there's a functional difference that gives a distinct advantage (e.g. a scripting language for scripting, or string-handling in Perl, etc.)
TL;DR version: Who cares what language? Stop arguing about it and start coding.
This may well be because their study cannot discern the amount of programmer effort per check-in, but it is a fatal flaw. Open development methods mean that a lot of dirty laundry gets checked into repositories. If dynamic languages have more bugs per check-in, but require significantly less work per check-in, then measuring bugs per check-in without measuring effort per check-in is meaningless, and that's before you even get to the functionality provided by the checked-in code.
You can code sloppily in any language.
True, but some languages make it more difficult to do so. Ada, for example, won't allow code to compile with (what should be) obvious logic or syntax errors that most C/C++ compilers will happily ignore, and hence allow to go undiscovered until runtime...errors that could be catastrophic in the real world.
Ada has acquired a reputation as a niche-market language, but that niche market takes heavy advantage of Ada's strengths: strong typing and a requirement that the developer properly design the software before writing code. Unfortunately, deciding to develop commercial software in Ada also comes with a fairly steep price tag, because it's a niche market...thus perpetuating the cycle.
DISCLAIMER: I am not affiliated with any company which produces or sells Ada compilers.
All the world's an analog stage, and digital circuits play only bit parts.
The problem with these kinds of studies is that there is no actual way to objectively measure software quality. You can correlate all the data you want, but garbage in means garbage out.
For this study they used two thinfactor gs to determine software quality: one is the number of bugfix commits. Ugh. I'm not even clear if the number of bugfix commits means higher quality, or lower quality. That could go either way. It might mean you had better testers, or that you committed things in small batches, or that you had more branches. The other factor was a natural language processor that read the check-in comments. While this is a really cool idea, you would need a lot of research just to prove that this approach actually works before you can start using the technique to draw conclusions about some other data.
So while this was very cool, and very ambitious, the results are completely meaningless until someone can prove that this technique actually measures software quality at all.
Also striking - they point out that functional languages, in particular Scala, Erlang, and Clojure have more concurrency bugs, without bringing up that concurrency support is basically the primary feature those languages are selected for. I'd love to see the defect number correlated with the percentage of code dealing with concurrency.
I don't care what they say, software written with Emacs is way better than software writen with Vi!
Stop-Prism.org: Opt Out of Surveillance
Absolutely! They write:
"The major languages that contribute concurrency errors
from these classes are Go, C#, and Scala and their regression
coefficient is also statistically significant. These results confirm,
in general static languages produce more concurrency errors than
others. Among the dynamic languages, only Erlang is more
prone to concurrency errors. The regression analysis also shows
that projects written in dynamic languages like CoffeeScript,
TypeScript, Ruby, and Php have fewer concurrency errors"
Well, there isn't much concurrency to be had in TypeScript or CoffeeScript since they operate in single threaded environments, and it wouldn't surpise me if the same goes for the other ones. And saying that Erlan has problems with concurrency is... the cart before the horse.
Python also uses hash tables everywhere, including for global and local variables and instance and class members.
The implementations of the hash table is extremely optimised.
There was an interesting discussion on usenet about someone who wanted to optimise a piece of Python code by writing a C++ module to do the heavy lifting. It was about adding C++ strings into a C++ hash map. The naive version was 10,000 times slower in C++.
Of course the naive function was copying strings many times since this is what the string class does when passing by value. After many posts they came up with a optimised version, it was about 100 lines of code (original was 10 lines), with lots of references passing and explicit and complex template instantiation of the hash map and its iterators. It was still 1.2 times slower than the Python implementation.
Relevant parts highlighted:
"The remaining coefficients are significant and either positive or negative. ... a greater number of defect fixes.
For those with positive coefficients we can expect that
the language is associated with
These languages include C, C++, JavaScript, Objective-C, Php, and Python.
The languages Clojure, Haskell, Ruby, Scala, and TypeScript, all have negative coefficients
implying that these languages are less likely than the average to result in defect fixing commits"
Isn't the real message here to choose static typing if you like fixing bugs?