Slashdot Mirror
Choice Overload In Parallel Programming
scott3778 writes to recommend a post by Timothy Mattson over at Intel's Research Blog. He argues, convincingly, that the most important paper for programming language designers to read today is one written by two social psychology professors in 2000. This is the well-known academic study, "When Choice is Demotivating: Can One Desire too Much of a Good Thing?" "And then we show them the parallel programming environments they can work with: MPI, OpenMP, Ct, HPF, TBB, Erlang, Shmemm, Portals, ZPL, BSP, CHARM++, Cilk, Co-array Fortran, PVM, Pthreads, windows threads, Tstreams, GA, Java, UPC, Titanium, Parlog, NESL,Split-C... and the list goes on and on. If we aren't careful, the result could very well be a 'choice overload' experience with software vendors running away in frustration."
This whole idea of 'choice overload' is so much drivel, IMHO. And, no, I'm not trying to flame here.
Have you ever known anybody to say: "There are just too many girls to choose from, I guess I'll go hide in the basement."?
Or: "There are ten thousand restaurants in this city. I just can't cope. I'm going to stop eating."?
A better label for the whole subject would be: " How a small minority of people fail to learn tree-pruning techniques, and dissolve in panic." Then we all could say: "Yep, sounds like my ex-girlfriend. Been there, done that. Next?"
And they'll change it every three years, so as to make more money off of certifications, software sales, and save money by not having to fix bugs in that "old, obsolete" stuff that was so "shiney new" stuff so recently.
If Microsoft wants to tell me what to do, they'd better be ready to sign a check with 6 figures to the left of the decimal point ...
and I say that as an atheist.
Ok, first: he writes as if all choices are equivalent. One jam might as well be the same as another, they just differ by taste. It's not like I walk into the store already invested raspberries. It's not as if Java programmers are going to decide that the Fortran parallel library is better, so why not just switch to Fortran.
Second, I doubt explicit parallel programming is going to be mainstream anytime soon. No, make that ever. Ever! Parallel programming will only happen in the mainstream when it is handled implicitly by the language, like a dataflow language. Asking normal programmers to deal with parallel programming is trouble when basic logic eludes most of them.
Third, all you people, including the author of TFA, who think that more than one or two standards is bad thing ("the great thing about standards is there are so many to choose from!") it's time to wake up: the world is not about to consolidate. The future is going to require C3PO and R2D2: there will be so many fricking languages and standards that your translator is going to require AI and legs to come along with you. For every one thing that fades away, eventually, probably 10 or 100 replace it. The future is a big mess.
must... stay... awake...
Choice is good if it provides different tools for different tasks. The list provided is somewhat silly, since several of the technologies address completely different issues and applications. There's a reason Sears sell thirty different shapes of hammers -- all nails are not the same.
After considerable deliberation and experimentation, I've shosen OpenMP for most task-parallel applications. The syntax is simple, it operates across C, C++, and Fortran, and it is supported by most major compilers on Linux, Windows, and Sun. The only quirk has been problematic support in GCC 4.2, but that will likely be cleared up within a few months. For cluster work, I tend to use MPI, because it has a long history and good support. I'm sure other tools have good versatility in environments different from those I frequent.
All about me
On top of that, if this really is something that affects programmers then why the hell aren't we all rendered utterly useless by the number of programming languages? Or all the possible ways one could format code? Etc.
But hey, the guy's writing in a "research" blog and, as in academia, when you don't have anything real to contribute you can cite something completely unrelated and pretend it has relevance.
Honestly, this sounds vaguely like "there's too much to choose from, so everyone just use Intel Thread Building Blocks, K? You can't possibly do better so just use our stuff because we cover all cases..."
There are hundreds of languages that support loops, variable assignments, recursion, definition of subroutines and Joe knows what else.
Language constructs to support mp are bound to be just as numerous. I'm not normally one to be so dismissive of a post, but I think this one of the more pointless items ever shared with erudite little community.
That's just silly. There are two types of programmers that could be making choices like this, and neither one of them would suffer from too much choice.
The first kind is a programmer just trying to paralelize existing code. In that case, the choice of threading platforms is pretty much obvious. Existing Windows code? Use windows threads. C/C++ on Unix? Pthreads probably. Java code? Java threads... Probably not even 2 seconds worth of thought will go into considering the alternatives (and that's probably fine)
The other type of programmer is one who's actively looking to develop high performance paralelized software. I am talking about cases where performance is the primary objective and it drives the choice of programming language and platform. In these cases, the nuance of the different thread models might matter but the programmer of this type would be happy (rather than scared) to investigate all the options. After all, if he didn't care, he'd just go with the default choices like the first programmer.
http://ed.markovich.googlepages.com
The other choices are whats wrong with Open MP. The market needs to shake out the pretenders, before more people will make the correct choices. Thats the basic idea of this story.
Having said that, I'm praying for Fortran 95 to take over. Its the only Malt Liquor I drink.
Well.. maybe. Or Maybe not. But Definitely not sort of.
A nice video about the The Paradox of Choice is available at Google Video. It is an interesting topic, but I don't think it applies all that much to parallel programming. The issue isn't that there are to many languages, but simply that there are a bunch of very well established languages that provide you little to no help with writing parallel programs properly, so everybody just continues to write their programs the way they did the last 20 years and thus takes little or no advantage of the available multiprocessor systems. And I doubt that just reducing the choice would help much at all about that right now, since we really still don't know how to write parallel programs on a large scale (i.e. in a way that everybody can and does it), so some more research and experimentation is needed.
Uh, most good parallel programming frameworks are cross-language. MPI has APIs in C, Fortran, and C++ (I believe) ... so does OpenMP. Also, most supercomputing programs are either written in C or Fortran. So, YES!, choose Fortran (and MPI) for all of your supercomputing needs. Or write it in C. Or write subroutines/functions in both languages and compile away (the newest version of Fortran will be fully C-compatible).
... the actual implementation of parallel programming! Is it distributed (MPI) or shared (OpenMP)? Does it have elegant syntax for accessing variables across processors (Co-Array) or just function based? Because there is no one true way to write a parallel program (it really depends on the algorithm), there will always be multiple frameworks to choose from. O well! The people who write parallel programs are typically smart enough to deal with excessive choices. (No comment on others).
The main differences between these parallel programming frameworks are
Okay this list seems to be of several different technologies some of which over lap but several are used for very different tasks. You can not replace MPI with Pthreads.
I don't see the problem. Just as we have many different programing languages these different interfaces all have different niches.
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
It's kind of amusing looking at the languages he lists. MPI and OpenMP are by far the most-used environments, but pthreads and java should probably be next not at the end of the list. Ct, intel's new parallel language, hasn't even been formally announced yet let along there being any released documentation / code for it. CUDA however, Nvidia's competing parallel language, isn't even mentioned though it's been released for months now.
What does a web browser need more than one core for? Or a word processor? Or an IM client? The only "desktop PC" type tasks I can think of that might actually be able to saturate even a single CPU are multimedia and gaming. In the case of the former, it's usually enough for the OS put the media player's threads on one core and everything else on another so they don't have to fight. In the case of the latter, well, video games hardly qualify for "just about every application".
Part of the problem is that there isn't a good solution yet, so there's a lot of effort being put into trying to find a way for a bad solution to be more comfortable.
Old-school iterative languages are a clumsy fit. They're night impossible to debug, and ones that let you do clever things at the hardware level will bring the whole project down in screaming flames when someone tries to get clever. So new libraries for old languages seldom fill the bill.
New-hotness functional languages are insane. It's very, very, very difficult for seasoned programmers to get their heads around it, and impossible for n00bz who don't have heavy math backgrounds. Compounding the issue is that the syntax tends to be on the wrong side of horrible with little or no syntactic sugar to make the medicine go down. So re-imagining the paradigm is a bit like picturing a five dimensional sphere - great fun, if you're smart enough to do it. No-one is smart enough to do it.
We're probably looking at a problem space that is best tackled by something that doesn't exist yet - an elegant, easily understood tool that simply makes sense, like objects or everything-is-a-file or scripting languages or regex. We're seeing so many different approaches to MPP because programmers are trying to figure out what that tool is. Once someone hits on it, the field will shake itself out.
Since we haven't hit on it, too much choice is a good thing - it means people will take the initiative to do something on their own that works better, rather than trying to get something suboptimal to work because it's the "standard".
And I say that as an agnostic.
This guy must be missing the point of having different programming languages and environments - parallel or not. He lists ZPL, which is, first and foremost in my opinion, a really cool array-based language. There are certain things you're going to want to do in ZPL as opposed to non-array based languages, such as image processing (which lends itself really well to parallel processing IMHO). For things that don't require non-multi-dimensional array processing, you wouldn't want to use ZPL.
"...today consumers have been conditioned to think of beer when they see a bullfrog..."
- Jams are functionally equivalent; the choice is inconsequential. This is far from the case with programming languages, which have meaningful differences.
- Programming languages solve important problems, so a choice will be made. You can't just give up on the whole idea and walk away as with specialty jams.
- There are so many different aspects of a language, you can have a great number of them, yet they can all be very different from each other.
- Significant resources are devoted to developing and choosing parallel languages. This greatly increases the number of choices that can be evaluated. Consider how much time you spend shopping for the right car vs. a jar of jam.
Now would be a terrible time to stop developing parallel languages, because the problem is just now coming to the forefront with the limits of single-core performance pushing back and multi-cores taking over. I'm suspect the parallel programing paradigm of the next 40 years hasn't been invented yet, and I'm almost certain it hasn't yet been popularized. So I say, let a hundred flowers bloom.MPI, pthreads and so on are really a poor way of doing parallel programming. The reality is that these languages are all simply serial languages with parallelism bolted on. What you really need to do is use a truely parallel language. Way back in 1990 I learned to program transputers using Occam which was parallel through and through. On that platform it was trivial to write pure parallel code and more to the point, you could write it in a very fine grained way which could easily be serialised to run on a smaller number of processors. In some ways it was similar to MPI but far more potent because of the built in support in the transputer architecture. It is very sad that in the intervening 20 years or so since the transputer was first invented, parallel programming has gone largely nowhere. Attempts at automatic parallelisation of serial code are doomed to failure and threading within serial languages is always going to be a blunt tool. Maybe in another 20 years we will be back where we were in the late 1980's.
"I have the attention span of a strobe lit goldfish, please get to the point quickly!"
A 5 dimensional sphere is easy to visualize. The other two dimensions could be color and size, with the other three being the normal x y z coordinates.
People always assume that the extra dimensions are obscure and bizarre extension of space-time. They don't have to be. A dimension can be used for any variable you want. A dimension could be reflectivity of light, smell, fluffyness, firmness. hardness, etc.
Diamonds, for instance, are priced on a four-dimensional scale (carat, color, clarity, cut). Those dimensions have no relationship to space-time.
People always assume that dimension #4 is time and the other dimensions are, well, unknowable. That definitely is not the case.
What would you rather they do, stick with version 1.0 for the rest of time, never incorporating new features?
You have to strike a balance - change stuff too fast and people avoid it as its unstable. Change things too slowly and people avoid it as it doesn't provide a required or desired feature that some competing product provides.
To be honest, depending on what exactly it is, three years doesn't sound unreasonable. It's not like anyone's forcing you to get certified either - I don't know about your country or company, but we barely even look at a person's certs when deciding whether or not to hire them, experience is far preferable.
It's official. Most of you are morons.
You're right; there's nothing more tragic than watching a programmer butcher his well-written program in a futile attempt to make it fit the only concurrency model he knows. Closely associating a language with a single, well-designed concurrency framework would at best do the same thing for it that Rails did for Ruby: bring it a flurry of popularity in the short run and damage its reputation in the long run as people doggedly apply the framework to unsuitable problems and blame the language for the results.
On the other hand, at some point we're all supposed to face up to the end of the free lunch, and a fad for an exotic kind of concurrency might be a clumsy, spasmodic step in the right direction.
Choice overload is a problem, but only if you are actually faced with all the choices. It need not be that way. There are various kinds of parallelism and various angles to attack each. Some of the choices one has will not make a lot of sense for type of problem one is looking
at. So, categorizing can help.
Some technologies will be in rapid development, others will be no longer actively maintained, and yet others will be stable but actively maintained. This also affects which choices are good.
Then there's licensing. Depending on the task, closed-source or copyleft
licenses might not be acceptable.
Some of the solutions may be low-level, allowing programmers to build something matching their application out of the provided building blocks, where other solutions may focus on providing higher level constructs, ready to be used. Sometimes, these will match what you need, and sometimes, they won't.
I am sure there are other axes of differentiation. Setting requirements will narrow ones choices, as well as illustrate why choice is a Good Thing. If there were only a few choices, it is unavoidable that none of them would actually fit some sets of requirements.
Now, the thing is that categorizing the various solutions is not something that every potential user of the solutions has to do. Part of the work can be done by the developers of each solution. Presumably, the solution is developed because a satisfactory solution did not already exist. In my opinion, the developers _should_ list related work, compare their solution to it, and explain why they saw fit to develop their solution. This is a standard part of research.
Another part of the work is comparisons done by third parties. Some independent person would go and investigate a number of solutions, and provide a write-up of the requirements they assumed, the solutions they investigated, how these solutions fit their requirements, and what their overall impression of the solutions was (w.r.t. things like ease of setup, documentation, development status, etc.). This, too, is valid research. It should be published, so everyone benefits.
In the end, what you get to do when you need to pick a solution for parallel programming, is
1. Define your requirements
2. Get a list of possible solutions
3. See what has been written about them
4. Check if that seems to be valid (it might be out of date, for one)
5. Possibly investigate any solutions that you found but that haven't been covered by others.
6. Decide which one to go with, based on the information you have gathered.
Sure, this is a far cry from
1. Find the only available solution
2. There is no step 2
but for that you are almost guaranteed to get a choice that better fits your requirements (you would be very lucky to have the only available solution be a great match), without having to pay the full cost of investigating every solution out there.
The thing to remember about the paradox of choice is that you will probably _feel_ less happy (there is always the nagging feeling that you could have made a better choice), but that you will generally end up with something _better_ than if the choice hadn't been there is the first place.
If you _really_ aren't happy about having to choose, you can always pick one (say, at random) and pretend that was your only choice. I conjecture that this is what the situation of having only one option is really like.
Please correct me if I got my facts wrong.
Seems to me to be a philosophical problem. With a single CPU you have a single gate processing a single sequence of instructions. It's easy to push the instructions and data through that gate in order. When you have 10, 20, 100 gates choosing which direction to push the instructions and data becomes exponentially more complex.
The solution it would seem to me would be to start pulling the instructions and data through the gates instead of pushing it.
Deleted
The overload is just a symptom if the real problem and that is that parallel programming is just plain hard. We've had these issues for over a decade and we haven't seen a step function in use of parallel programming. It is difficult for most people to think of many things happening at the same time and to design and debug this class of program. We tend to start by thinking of a task in serial steps and then look for ways to add a little parallelism.
The folks who are low level systems programmers (OS and networks) tend to be folks who have an aptitude for thinking about parallelism and designing with parallelism in mind. There are a group of people in the scientific space who make use of parallelism, but then again they are Phd mathematicians and physicists. After that it drops off rapidly.
Maybe it has something to do with he way we are educated. perhaps it is a more fundamental issue of brain wiring. After all, we c perform complex physical tasks in parallel, but maybe only a small segment of the population is wired to think about programming problems in parallel.
The chip guys are throwing more cores at us and we can't create the software to fully utilize the hardware due to this issue. Perhaps it is time to take a step back and to stop trying to solve the problem by throwing more and different programming packages at the problem and examine why folks have so much trouble in this area.
So do you want to go to a bar with only one woman in it? Or a bar with 20? If you believe the premise of "choice overload" making you unhappy, you should choose the first, right?
Aside from that in many contexts the "choice overload" hypothesis is flat-out wrong (unless you really can claim to have felt a special rush of happiness just when you went into that bar with only one dame in it - and she wasn't, say, your girl already), there are open questions about how representative the test sample was. Psychological problems run culturally in certain populations. How can we be sure that the population tested for "choice overload" didn't share a psychological problem regarding choice that has no foundation is basic human psychology, but rather was relative to their own cultural limitations?
For most people in most cultures over history, the trick is to be happy with not much choice. That's generally the case for the working class, for the infantry soldiers, and for tribal peoples in environments of scarcity. Yet even in those cultures there are other classes for whom the trick is to be happy with a great deal of choice - the upper class, the generals, and tribal peoples in environments of plenty. Those whose cultures and religions derive primarily from desert (scarcity) environments are those driven craziest by "choice overload" - thus the Islamic meltdown, and the rejection of modern freedom by American fundamentalists. But we do have other cultures here. And the studies associated with the "choice overload" hypothesis, do not, I'll bet dimes to dollars, correct in any way for (sub)culture and psychological diagnosis.
"with their freedom lost all virtue lose" - Milton
You're re-writing history. Microsoft proposed .NET so that they could get people off of web-based applications. THAT is why they stopped developing IE after version 6. Talk to people who were in the financial industry, and you'll see the massive push they made to try to get everyone to switch from web apps to .NET apps.
> "Shall we shit on Ford because they'd like to see everyone driving one of their cars?"
We don't have to - their cars are shitty enough already :-)
As for Mono, and Microsoft "shared-source" and other licensing in general, this guy is repeating what I said 3-4 years ago http://www.linux-watch.com/news/NS3695984068.html