Slashdot Mirror
"Clinical Trials" For Programming Languages?
theodp writes "High school junior Charles Dawson's New Year resolution is to write a new program in different language each week. It's an ambitious project for someone of any age, and while it won't give him an in-depth appreciation of programming language differences, it'll certainly give him greater insight into the strengths of certain languages than would perusing the Hello World Wikipedia article. Lots of claims are made about the comparative productivity of programming languages, but have there been any landmark studies that measure the efficacy of a programming language's productivity claims in a 'clinical trial' of sorts? Would head-to-head tests against other languages be a better way of sorting out Popularity vs Productivity vs Performance claims, or is relying on more nebulous claims of superiority the best we can do?"
long flamewar
There is already a pretty good collection http://www.99-bottles-of-beer.net/
There is also a website with the implementations of the Perl cookbook in a bunch of languages: http://pleac.sourceforge.net/
...Some languages are good for some things, and other languages are good for other things. Think LISP.
My New Year resolution is to design and implement a new programming language every week.
If Pandora's box is destined to be opened, *I* want to be the one to open it.
Is dice this desperate for clicks?
John McAfee 'It was like that time I hired that Bangkok prostitute; to do my taxes, while I fucked my accountant'
And therein lies the problem of comparisons. An extreme case: a person writing a program that involves concurrency among hundreds of processes will probably be more productive in Erlang than in Perl, but a person writing a program that does massive amounts of text manipulation will be more productive in Perl than in Erlang, because of what the languages were designed for. It's somewhat like asking which is a better tool, a hammer or a screwdriver. A lot of it depends on what you're trying to build.
In any such trial, it is important that aspects such as maintainability, reliability and securability be considered. The ability to hack out a-lot of functionality is not the only criteria that is important, unless you are building a home hobby project.
You'll have trouble getting a consensus as to an agreed-upon operational definition of "Productivity".
I have this same problem -- there are a lot of interesting languages out there that I'm interested in trying, but I always keep going back to languages I already know because:
I was thinking that the solution to this is to have one program that I understand very well implemented as well and completely as possible in a language that I feel proficient in, and have that be my reference. Then, over the course of a couple of weeks (a month?), re-implement the same program in the new language and strive for the implementation to be as idiomatic of the new language as possible. After all, if you're still thinking in the old language but just using the new one's syntax, what's the point?
I feel like this would give you a lot of data to make a reasoned decision -- you can compare language features and how the implementation works in one versus the other; time to implementation (LOC, maybe?); how much of a mental shift the new language requires; the toolchain around the new language; etc.
The problem is figuring out what the reference app is, and having the stomach for implementing it over and over again. Tetris, maybe? ;)
But, back to the resolution (and partially touched on) -- I don't think a week is enough time. A month is even cutting it close, IMO.
C
The Sun is proof that we can't even do fire properly.
do you relly want apps that need 2-5+ run times and are very bloated?
You need a million dollars, for a guy like you anyway...
The answer is research is sparse in this area, but the few times it's been tried (using competent programmers in each language rather than conflating learning the language and productivity in it), is LISP and LISP-like languages win when measuring programmer productivity and ability to express complex algorithms in small amounts of code, and C and C++ like languages win for ultimate ability to make the program run fast. But the variability from programmer to programmer in how fast the program runs can exceed the variability between languages so it pays to get high quality programmers who have an intimate understanding of both efficient algorithms and the underlying machine architecture, rather than think "the language will make the program run fast".
With everything, there are professional users and amateur users. For amateur users, it's important to get reasonably good results with relatively low effort without much learning. For professionals, what counts is the effort for projects the size a professional does, after learning a lot.
Trying a new programming language every week cannot give any useful information to a professional user, because the language can only be judged on how well it works for inexperienced developers on tiny projects. That's not what professionals do.
If your using a programming language, but your language is not enough, then you should consult your doctor.
Call your doctor if your application worsens, or if you have unusual changes in mood, behavior, or thoughts of suicide.
See a doctor if you have high fever, stiff muscles, confusion, or having trouble swallowing.
Some adverse affects are: diarrhea, seizures, and flatulence.
Simon Peyton Jones points out that doing 'clinical trials' for programming languages is hard, because it's expensive, and because "programming language is weak on that score.....culturally we're not well adapted to it."
He also points out that Microsoft does something similar, they do usability testing for their APIs and languages, where a researcher sits behind a glass window, and watches programmers try to figure things out.
For pure efficiency, the benchmark game at least gives some data points. From a gut feeling, I would say LISP is rather good at letting you do the most in the least number of lines (dependency injection is insanely easy, for example, compared to Java where it's painful if you haven't planned for it).
"First they came for the slanderers and i said nothing."
This is a myth. There is only suitability to solve a problem within a given context. At first, it's "how fast can I bring this to market", then "how does this scale" (in terms of execution efficiency). Finally, "can I hire people to do this?"
The starting point is inevitably what the initial implementer is most familiar with (or infatuated with) at the moment.
A few weeks ago, I wrote an article about how asinine this technology argument is.
Fixed
Some key features of "gold standard" clinical trials: 1) large enough sample size to draw statistically significant conclusions; 2) real illnesses, not a simulated laboratory setting; 3) a double-blind control group; and 4) long enough duration to measure real-world outcomes.
The programming-languages version would be to have teams randomly assigned to perform major (6+ month) programming projects in different languages, and then see their outcomes. For example, 40 game studios will continue to write their games in C++ as the control group, while you'll have the other 40 write them in Haskell. You probably want to iterate a few times as well to make sure that there's no first-game-in-a-new-language effect and to ensure that everyone is actually knowledgeable in the language being tested.
Oh, and it should be blind, so neither the teams nor the researchers know which language they're using.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
Enjoy it while you can. Soon enough, job and family concerns will consume most of your attention.
Dr. Lizardo said it best: "Laugh'a while you can, monkey boy!"
it's my observation that programming languages have become the equivalent of fashion and bands.
it's as if choosing and using one is like saying "oh, im listening to arctic monkeys and calvis harris these days, and that makes me feel liberated from those plebs still listening to kings of leon and david guetta..and MAYBE if im lucky someone will be impressed by my trendiness"...now that so many 20-somethings code, they all seem to feel the need to break from the "boring old bonds" of existing languages and define themselves among their peers by how esoteric their language-de-jour is.
what this guy is doing is illustrating the point, he isn't going to learn anything about the benefits of each "language" or how to maximize productivity...it's just a "notice me notice me" stunt.
it's just a silly exercise in syntax-swapping anyway for 90% of it...
please don't get me wrong...it's totally fine by me whatever language you want to use, as long as it's maintainable and there is a large enough pool of existing programmers who know the language so when you leave my company because your bored with maintaining the code you wrote, i can find someone quickly and affordably to replace you.
never bring a twinkie to a food fight.
Yes, some languages do things better than others, but there's more to languages than that. A good question is how well the language meets its goals.
http://xkcd.com/1306/
Some languages are simply more clean, have a more consistent syntax, etc. then others. For example, both Fortran 77 and 90 are aimed at numerical computation, but 77 has a weird syntax made for punch cards and other oddities; 90 is just better. (Flame on!)
Repeatability. Productivity must be repeatable if any measured claims can be made, and no one does the same thing twice.
Even if you write similar code for a similar purpose, you have the backing of experience. The results are different even if you end up with the same code.
And, there are the os and third party libraries. I can save time by using such features, if I take time to learn them.
There is a post below about maintainability, securability, etc, all of which are good points, but will rarely be done exactly the same way. Especially since some are intended to be server side, where obscurity is possible, and some client side where you may need real security.
Any such study will have questionable conclusions due to the number of variables. Writing new code vs inheriting, where the choice might be maintain or rewrite in a new language.
But none of these will be applicable to an entirely new situation, and any study results irrelevant. Repeatability in code means studying something that will probably never happen again, or something so basic it will never represent normal professional usage.
Look on craigs list. You can get it for $1-$2,000.
Do you even lift?
These aren't the 'roids you're looking for.
One of the hallmarks of a clinical trail is that a random group of humans will respond about the same to a given intervention (or lack thereof). And for things like drugs or a medical treatment it's an assumption you want to hold up for safety's sake. It doesn't always, of course. There is a drug on the market that is targeted specifically for African Americans, as it works really well in that population.
However, that assumption just don't hold for a random group of developers. Not even close.
After decades of enjoying the extreme and obvious productivity advantages of programming with dynamic languages in interactive environments, I continue to be baffled by the overwhelming preference for static, compiled languages. I understand there is a place for such things, much as there is a place for programming in assembly language, but I continue to wonder why such a clumsy paradigm is so dominant.
Soon to be featured by a /. outlet in your galaxy , yay !!
Religous speak to God. Insane are spoken to by God. When all shut up, one can finally hear Shostakovich in peace
There are good practical reasons for writing the same program in different languages. (1) People often make the same mistakes in the same language, because they are often taught/trained in similar fashions. (2) Different programming languages have different issues and failure modes.
I'm not an expert, but I worked as an undergraduate research assistant on a project way (way) back in college (1985-87) for a professor on a NASA contract to study issues related to N-version fault tolerance - like used on the Shuttle, where several computers solve the same problems and vote on the answers. One problem (if I remember correctly) was that the different programs, written by different people, in the same, or same type of, language often failed on the same or similar edge cases.
The experiment was to implement the same solution to a problem using much different languages. In this case, Pascal and Prolog. The "gold" Pascal routine was already written and my task was to write the corresponding "gold" routine in Prolog. [ I also did work for a related study in the automatic analysis (and execution) of abstract data types in LISP.] I graduated before the experiment was run, but found the idea as least plausible.
Note that I might be remembering some of this incorrectly as it was a while ago and I was only an undergrad. (They wanted a graduate student, but I was the only one with LISP and Prolog experience... And $9.50/hour wasn't bad for 1985.)
It must have been something you assimilated. . . .
I'm surprised to see that no one has mentioned Rosetta Code yet. It provides many examples of different (real-world) problems being solved in many different languages. It's interesting for comparisons between languages and it's very useful for finding already-implemented solutions to problems in whatever language a user may require.
Irony is that marketshare for C and C++ is going down. Rust or Go or D or M# is the future of systems languages!
It's a good lesson, but for different reasons. Here's why.
In the real world, you pick the right tool for the job. You never pick a language because it's the best language. There is no such thing. Factors going into language selection where technical merit plays no role include what the other developers at the company and/or the project are using, what environment you're using (if Apple, then Objective C; if Android, then Java), what language the code you are maintaining was written in 5, 10, 20, or 30 years ago, and (hopefully if you are a great programmer this will be a minor issue) what languages you're comfortable with using.
After 30 years I've learned that basic computer science concepts are helpful, but only to a point. Google may want you to know specifics about certain types of trees for their interview process, but if you need to know that level of detail for a job, you spend a few hours on Google and learn it. The same goes for languages. Figure out what you need with a bit of research before you start the job. You should have a great idea of what environments a language is nearly always used, so you don't try to do something weird nobody can maintain. If you're going to write an iPhone app, you're going to adhere to whatever specific Objective C thing Apple is doing. Maybe I'm slightly out of date and Apple is doing something else, who knows? I don't work in that space.
Python everywhere, be damned with you, is a quick way to make enemies of people 10-20 years down the road who have to maintain your code. I was doing web development in the 1990s, and everybody used Perl. For everything. Now I work with a legacy Perl code base, and mod_perl seems to be completely abandoned, and it certainly hasn't been released for apache httpd 2.4 yet. We're using Perl because we have to, but not for new stuff. But for the Perl part of our system in bug fix maintenance mode, it is the appropriate language. We didn't have the attitude that we'd continue to use Perl for everything just because that's the way things were done. We were flexible enough to slowly switch over to PHP for certain things that we had been using Perl for.
Avoid fads like the plague. After 30 years of programming, I just ignore marketing. I have no gee whiz attitude about anything. I focus on perfecting my craft, learning how to program better, to debug better, to test better. Learning how to deliver code that works now and five years from now. All that is way more important than figuring how how some language is subjectively the best.
Hi everyone. Thanks for all of your comments. I just want to quickly elaborate on why I'm doing this and what I'm trying to do with this project: I am not trying to start a flamewar, annoy programmers, or hog attention. I am not aiming to master any of these languages after only a week. I'm not even trying to get "okay" at any of them. All I'm trying to get out of this project is a more broad understanding of the languages that are out there, and what, in a very general sense, each language is good for. My thought is that if I can get even one cool new idea or concept out of each week's programming language (like functional programming concepts from this week's foray into Haskell), I'll consider this project a success. Also, it will allow me in future to be able to quickly reference a new language if I need it for something I'm working on (not necessarily remember any syntax or anything, but at least to save myself some googling time). Thanks for your interest. Sincerely, Charles Dawson
As someone who pretty much only codes in python lately, I would agree except that I hate the fact that whitespace actually matters in python. I feel that this limits the utility of python while gaining only a modest increase to readability (according to some people).
I work with a large number of programmers overseas in India who only know one language. Our UI team is made up of 2 guys who understand Javascript/CSS/HTML, 2 who understand PHP/MySQL, and one that understands both (but can't actually do either very well). And while I've observed it to be a terrible handicap to them a lot of the time, and I really wish they would find a new career, that seems to be the norm over there. So I guess if you only know one language you have good prospects in India.
All the world's a CPU, and all the men and women merely AI agents
We all know innately know this, but sometimes it's hard to avoid getting caught up in a religious debate about the apples-to-oranges details.
Once your language is at least complex enough to write a compiler or interpreter for itself - that is, it's no longer a toy language - it tends to be more or less capable of everything every other language is capable of. Sure, it might not be the best tool for a given job, but they're all generally the same aside from some syntactic sugar.
The more important thing is the metrics we can track are directly correlated to a person's experience with a language. The longer an individual spends writing in a language, the less time it takes to complete a program, the less errors it contains, the severity or error incident goes down, the number of security issues is reduced, it is better optimized for the platform, uses less memory or cpu, has more features, etc.
Those are reliable statistics and the trend holds true regardless of the language. That has real world value far above and beyond arguing whether whitespace should be part of a language, or if it uses smalltalk or c++'s object models, to name a few items I've seen above.
The end result; whatever language you tend to use the most is going to be the best language for you to use, often even when it's not a good fit.
Au contraire! Disliked it at first, but love it now. Makes programming a lot faster, less error prone, very easy to read even for the novice. ps I toll some 20+ years in all sorts of curly bracket languages and other dynamic languages, and heck there was even Assembler in the very beginning
I tend to agree. One thing I find annoying is that in order to temporarily comment out the body of a conditional you have to add a "pass" line. The ability to add members to classes at runtime has also created some situations for me that were very confusing to troubleshoot.
Account -> Discussions -> Disable Sigs
Then I'd rank them by time to completion.
Seems simple enough but it ain't.
Which completion--the time it took you to write the code or the time it takes the code to execute?
If one time period is longer and the other shorter, which takes precedence?
It is hard to devise a good test. And it's even harder to articulate the expectations thereof.
Check out the "C2 dot com" wiki for discussions on programming features. We like to debate programming language features over there and there are many existing topics and discussions to get ideas from. Warning: personal preferences vary widely, and it can get heated. Bring a thick skin.
Table-ized A.I.
I believe this is for the Visual Basic 6 or less, not for "Visual Fred" which has the same name but mostly unrelated syntax and semantics. See: http://catb.org/jargon/html/V/Visual-Fred.html I think you have to take those measures with large grains of salt, but it's certainly true that languages affect productivity.
- David A. Wheeler (see my Secure Programming HOWTO)
I didn't post this. I came home from school and started wondering why I was getting so much traffic, and found this post in the referrals tab.
I have points but couldn't figure out how to mod the actual article -1 Flamebait
Tom DeMarco and Tim Lister's Peopleware book covers issues about productivity and is often quoted when people say that some developers are up to 10x more productive that others.
(see http://www.amazon.com/Peopleware-Productive-Projects-Teams-3rd-ebook/dp/B00DY5A8X2/ref=sr_1_1?ie=UTF8&qid=1389069388&sr=8-1&keywords=peopleware)
In summary the book looks into issues of programmer productivity. It explores the role of computer languages and concludes that for the most part which programming language is being used will not have a huge effect on productivity with the exception of Assembler. The jump to 3rd generation computer languages makes programmers MUCH more productive than Assembler, but between those languages and even the so-called 4gls there is not a great difference. (However, it would be interesting to see this study repeated with modern applications + languages, because writing web apps involves so many tools and third party tools that I would guess that there IS a difference between writing a web app in C vs Ruby on Rails)
The book then goes on to note that a far greater impact on productivity is the programmer's environment and the book fixates on the issue of a noise free environment and a door that closes. Interestingly a large part of the industry has forgotten the Peopleware lesson and has moved back to "open floor plans" or "cubicles" while the book cites studies showing that these increase the distraction rate and productivity of programmers.
A great book and and entertaining read.
I agree. Python is nice but the replacing visible braces with invisible whitespace is basically a dumb idea.
If a coder needs to be forced to indent properly then the coder in question should just stop coding all together and save everyone the trouble of reading their code.
yes, seriously.
it's quite difficult to make bash execute input data. You'll need to pipe it into another shell instance or explicitly execute it with -c or eval.
You don't believe that? prove me wrong!
Atari rules... ermm... ruled.
Any proper comparison between languages must leave out the UI, otherwise you're going to entangle the maturity and suitability of the presentation frameworks into the results.
Then there's the whole "stability under changing requirements" metric, which I'm sure wasn't even touched upon...
I'm not sure how useful it is to attempt a comparison like this. If you've ever done Project Euler and looked at all the different posted answers to the problems you solve, you realize that a lot of these hand-waving language metrics on 'productivity' and 'expressiveness' are pure bunk. C#/Python/Haskell/Clojure/etc... static typing, dynamic typing, it doesn't matter - all the good solutions look the same in this class of modern languages. Then there's the less expressive languages like C and assembly, and the reg-ex like languages R/J/K. These aren't really worth comparing with the first category because their proper use cases are so different. Then you have a the occasional Java retard posting eight pages of code where one paragraph was sufficient for everyone else.
I would say it's not the comparison between languages that is useful per se, but rather comparing the cruft and practices that surround them.
I had to do a networking project using a python library. Having to fuck around with a python socket made glaringly obvious the flaws of a language that does not support type casting. Don't call the bullshit python pulls type casting, because those are parsing library functions, not type casting.
What were you doing that you needed to d typecasting as part of a network socket functionality? I've never used python socket, only higher level network libraries like urllib, httplib, etc.
Was it Javascript that terminated their observably short programming lives? Was it what the experience did to them or what others did to them due to the { quality | performance | maintainability | other } of the results?
The execution environment matters if the application runs in one.
Execution size & time matter if execution will be long or done many times, or in a constrained environment.
Compile time matters if it only has to run once. That's why there was WATFIV.
If there are any, the standards of the team or organization that produce or consume the source code matter.
Readability and maintainability sometimes matter. I did a lot with APL, including enhancing a program where the flow chart was over 40 pages, but the program was so well structured, it was easy. In most personal programming cases, however, any complex function had to be written and tested in one day. After that, it was almost undecipherable, even by the author.
Verbosity matters to poor typists. It gets in the way of understanding. Did the person who came up with, "Blah blah = new Blah;" stutter? In what way is groovy inferior to java?
Domain really matters. Once I needed something to generate several drawings for a patent application. I considered generating SVG, I considered macro capabilities in a CAD package. In less than a day, I learned and made the diagrams with MSLogo. If I had to do it again, I might use python turtle graphics, but it would have been more verbose..
One has to get past Python's definition of white space. Just because source loots indented on the page doesn't mean that Python will accept the white space. This is true because of spaces getting mixed up with tab characters, they are not seen as the same and many editors don't help you very well wih this. It is almost as if you must do 'cat -v' to be sure.
I agree that once you get past that, Pythons notion of structure is easier to see than with braket-blocked syntax. It is more compact.