Slashdot Mirror
Choice Overload In Parallel Programming
scott3778 writes to recommend a post by Timothy Mattson over at Intel's Research Blog. He argues, convincingly, that the most important paper for programming language designers to read today is one written by two social psychology professors in 2000. This is the well-known academic study, "When Choice is Demotivating: Can One Desire too Much of a Good Thing?" "And then we show them the parallel programming environments they can work with: MPI, OpenMP, Ct, HPF, TBB, Erlang, Shmemm, Portals, ZPL, BSP, CHARM++, Cilk, Co-array Fortran, PVM, Pthreads, windows threads, Tstreams, GA, Java, UPC, Titanium, Parlog, NESL,Split-C... and the list goes on and on. If we aren't careful, the result could very well be a 'choice overload' experience with software vendors running away in frustration."
Because I'm sick (in the head) I say we go with the Fortran option!
'twas my second language; after BASIC. Ahhh, the fond memories...
http://www.columbia.edu/~ss957/whenchoice.html
Those who would give up essential liberty to purchase a little temporary safety, deserve neither liberty nor safety.
Microsoft will come along and tell you what your choice will be.
Write concurrently in two languages, then you're sure to make full use of available CPU cores.
This whole idea of 'choice overload' is so much drivel, IMHO. And, no, I'm not trying to flame here.
Have you ever known anybody to say: "There are just too many girls to choose from, I guess I'll go hide in the basement."?
Or: "There are ten thousand restaurants in this city. I just can't cope. I'm going to stop eating."?
A better label for the whole subject would be: " How a small minority of people fail to learn tree-pruning techniques, and dissolve in panic." Then we all could say: "Yep, sounds like my ex-girlfriend. Been there, done that. Next?"
Quoth the blogger: "With hundreds of languages and API's out there, is anyone really dumb enough to think "yet another one" will fix our parallel programming problems?"
Yet Intel touts its Threading Building Blocks library as just such a fix to many parallel programming problems. Now, TBB is a very nice product, and in many ways it is superior to a lot of existing libraries, APIs, and languages, but one gets the sense that maybe the left hand doesn't know what the right hand is doing at Intel.
I might also draw an analogy to the open source world, where there are often dozens of solutions to both simple/mundane problems (text editors, media players, command line shells, etc) and more complex ones (window managers, Linux distributions, etc). I wonder if the free and open source software world wouldn't also benefit from a "culling of the herd," so to speak.
and I say that as an atheist.
Ok, first: he writes as if all choices are equivalent. One jam might as well be the same as another, they just differ by taste. It's not like I walk into the store already invested raspberries. It's not as if Java programmers are going to decide that the Fortran parallel library is better, so why not just switch to Fortran.
Second, I doubt explicit parallel programming is going to be mainstream anytime soon. No, make that ever. Ever! Parallel programming will only happen in the mainstream when it is handled implicitly by the language, like a dataflow language. Asking normal programmers to deal with parallel programming is trouble when basic logic eludes most of them.
Third, all you people, including the author of TFA, who think that more than one or two standards is bad thing ("the great thing about standards is there are so many to choose from!") it's time to wake up: the world is not about to consolidate. The future is going to require C3PO and R2D2: there will be so many fricking languages and standards that your translator is going to require AI and legs to come along with you. For every one thing that fades away, eventually, probably 10 or 100 replace it. The future is a big mess.
must... stay... awake...
Choice is good if it provides different tools for different tasks. The list provided is somewhat silly, since several of the technologies address completely different issues and applications. There's a reason Sears sell thirty different shapes of hammers -- all nails are not the same.
After considerable deliberation and experimentation, I've shosen OpenMP for most task-parallel applications. The syntax is simple, it operates across C, C++, and Fortran, and it is supported by most major compilers on Linux, Windows, and Sun. The only quirk has been problematic support in GCC 4.2, but that will likely be cleared up within a few months. For cluster work, I tend to use MPI, because it has a long history and good support. I'm sure other tools have good versatility in environments different from those I frequent.
All about me
On top of that, if this really is something that affects programmers then why the hell aren't we all rendered utterly useless by the number of programming languages? Or all the possible ways one could format code? Etc.
But hey, the guy's writing in a "research" blog and, as in academia, when you don't have anything real to contribute you can cite something completely unrelated and pretend it has relevance.
Honestly, this sounds vaguely like "there's too much to choose from, so everyone just use Intel Thread Building Blocks, K? You can't possibly do better so just use our stuff because we cover all cases..."
It's designed for shared memory boxes. This is great if you own an E25K, not so great if you've chained together a couple thousand itaniums.
the more accurate the calculations became, the more the concepts tended to vanish into thin air. R. S. Mulliken
I've been gradually trying to learn more about functional programming, partly because I think fp techniques and ways of thinking come in handy even if you're programming in a procedurally oriented language, and partly because fp seems like a paradigm that is likely to get more and more useful as we get machines with more and more cores. Okay, fp!=parallel, but, e.g., one of the big selling points of Erlang is supposed to be that it lends itself to completely transparent use of parallel processors.
The choice overload does seem like kind of an issue to me. For as long as I continue to keep programming comfortably in the procedural languages I'm comfy with (e.g., perl), I'm never going to really wrap my mind around the radically different ways of thinking that you get in a more fp world. I'm been thinking for a long time that it would be fun to do a coding project in ocaml ... or haskell ... or lisp ... or erlang ... or -- you get the idea.
The trouble is, it's really not clear what to hitch my wagon to. Ocaml seems to have a very high quality implementation, but its garbage collector isn't multithreaded, the only book you can buy is in French (it's nice that you can download the English version for free, but I'd prefer to buy something bound), and the availability of libraries (and documentation for them) isn't quite as wonderful as I've gotten used to with perl. Lisp could be cool, but I hate the fact that it's not standardized, and I'm not convinced that eschewing arbitrary syntax really carries more pros than cons. Haskell? Maybe, but it sounds like putting on a hair shirt. The list goes on. I really feel like a deer in the headlights.
Find free books.
There are hundreds of languages that support loops, variable assignments, recursion, definition of subroutines and Joe knows what else.
Language constructs to support mp are bound to be just as numerous. I'm not normally one to be so dismissive of a post, but I think this one of the more pointless items ever shared with erudite little community.
That's just silly. There are two types of programmers that could be making choices like this, and neither one of them would suffer from too much choice.
The first kind is a programmer just trying to paralelize existing code. In that case, the choice of threading platforms is pretty much obvious. Existing Windows code? Use windows threads. C/C++ on Unix? Pthreads probably. Java code? Java threads... Probably not even 2 seconds worth of thought will go into considering the alternatives (and that's probably fine)
The other type of programmer is one who's actively looking to develop high performance paralelized software. I am talking about cases where performance is the primary objective and it drives the choice of programming language and platform. In these cases, the nuance of the different thread models might matter but the programmer of this type would be happy (rather than scared) to investigate all the options. After all, if he didn't care, he'd just go with the default choices like the first programmer.
http://ed.markovich.googlepages.com
The other choices are whats wrong with Open MP. The market needs to shake out the pretenders, before more people will make the correct choices. Thats the basic idea of this story.
Having said that, I'm praying for Fortran 95 to take over. Its the only Malt Liquor I drink.
Well.. maybe. Or Maybe not. But Definitely not sort of.
A nice video about the The Paradox of Choice is available at Google Video. It is an interesting topic, but I don't think it applies all that much to parallel programming. The issue isn't that there are to many languages, but simply that there are a bunch of very well established languages that provide you little to no help with writing parallel programs properly, so everybody just continues to write their programs the way they did the last 20 years and thus takes little or no advantage of the available multiprocessor systems. And I doubt that just reducing the choice would help much at all about that right now, since we really still don't know how to write parallel programs on a large scale (i.e. in a way that everybody can and does it), so some more research and experimentation is needed.
Just use RubyMPI when I release it next week :)
The power of MPI wrapped in the beauty of Ruby.
http://www.public.iastate.edu/~crb002/
bash-2.04$
bash-2.04$yes "Don't you hate dialup connections?"| write USERNAME
Okay this list seems to be of several different technologies some of which over lap but several are used for very different tasks. You can not replace MPI with Pthreads.
I don't see the problem. Just as we have many different programing languages these different interfaces all have different niches.
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
It's kind of amusing looking at the languages he lists. MPI and OpenMP are by far the most-used environments, but pthreads and java should probably be next not at the end of the list. Ct, intel's new parallel language, hasn't even been formally announced yet let along there being any released documentation / code for it. CUDA however, Nvidia's competing parallel language, isn't even mentioned though it's been released for months now.
What does a web browser need more than one core for? Or a word processor? Or an IM client? The only "desktop PC" type tasks I can think of that might actually be able to saturate even a single CPU are multimedia and gaming. In the case of the former, it's usually enough for the OS put the media player's threads on one core and everything else on another so they don't have to fight. In the case of the latter, well, video games hardly qualify for "just about every application".
Part of the problem is that there isn't a good solution yet, so there's a lot of effort being put into trying to find a way for a bad solution to be more comfortable.
Old-school iterative languages are a clumsy fit. They're night impossible to debug, and ones that let you do clever things at the hardware level will bring the whole project down in screaming flames when someone tries to get clever. So new libraries for old languages seldom fill the bill.
New-hotness functional languages are insane. It's very, very, very difficult for seasoned programmers to get their heads around it, and impossible for n00bz who don't have heavy math backgrounds. Compounding the issue is that the syntax tends to be on the wrong side of horrible with little or no syntactic sugar to make the medicine go down. So re-imagining the paradigm is a bit like picturing a five dimensional sphere - great fun, if you're smart enough to do it. No-one is smart enough to do it.
We're probably looking at a problem space that is best tackled by something that doesn't exist yet - an elegant, easily understood tool that simply makes sense, like objects or everything-is-a-file or scripting languages or regex. We're seeing so many different approaches to MPP because programmers are trying to figure out what that tool is. Once someone hits on it, the field will shake itself out.
Since we haven't hit on it, too much choice is a good thing - it means people will take the initiative to do something on their own that works better, rather than trying to get something suboptimal to work because it's the "standard".
And I say that as an agnostic.
This guy must be missing the point of having different programming languages and environments - parallel or not. He lists ZPL, which is, first and foremost in my opinion, a really cool array-based language. There are certain things you're going to want to do in ZPL as opposed to non-array based languages, such as image processing (which lends itself really well to parallel processing IMHO). For things that don't require non-multi-dimensional array processing, you wouldn't want to use ZPL.
"...today consumers have been conditioned to think of beer when they see a bullfrog..."
It's useless in cases where you don't have shared memory (though, really, most "general-purpose" solutions are going to be useless in this case).
It's implementation is, essentialy, compiler magic. This automatically rules it out in a lot of cases where you need precise control of what your code is doing. If, for example, you need logic to spin for a few cycles while a DMA operation completes (so as to not interrupt/stall something else - and yes, people do actually optimize to that level on some platforms) you can't really add that to OMP unless your compiler vendor is willing to make the change for you. With a library based solution you can grab the code and hack your little change into the scheduler without relying on third parties who may or may not be willing to accomodate you.
That said, it's certianly not useless. I mean, it's damn amazing when it comes to quickly hacking some parallelism into a loop that wasn't really designed with multi-threading in mind.
- Jams are functionally equivalent; the choice is inconsequential. This is far from the case with programming languages, which have meaningful differences.
- Programming languages solve important problems, so a choice will be made. You can't just give up on the whole idea and walk away as with specialty jams.
- There are so many different aspects of a language, you can have a great number of them, yet they can all be very different from each other.
- Significant resources are devoted to developing and choosing parallel languages. This greatly increases the number of choices that can be evaluated. Consider how much time you spend shopping for the right car vs. a jar of jam.
Now would be a terrible time to stop developing parallel languages, because the problem is just now coming to the forefront with the limits of single-core performance pushing back and multi-cores taking over. I'm suspect the parallel programing paradigm of the next 40 years hasn't been invented yet, and I'm almost certain it hasn't yet been popularized. So I say, let a hundred flowers bloom.The game.
MPI, pthreads and so on are really a poor way of doing parallel programming. The reality is that these languages are all simply serial languages with parallelism bolted on. What you really need to do is use a truely parallel language. Way back in 1990 I learned to program transputers using Occam which was parallel through and through. On that platform it was trivial to write pure parallel code and more to the point, you could write it in a very fine grained way which could easily be serialised to run on a smaller number of processors. In some ways it was similar to MPI but far more potent because of the built in support in the transputer architecture. It is very sad that in the intervening 20 years or so since the transputer was first invented, parallel programming has gone largely nowhere. Attempts at automatic parallelisation of serial code are doomed to failure and threading within serial languages is always going to be a blunt tool. Maybe in another 20 years we will be back where we were in the late 1980's.
"I have the attention span of a strobe lit goldfish, please get to the point quickly!"
to welcome our new Choice Overlord personally until I found that I misread.
Sorry.
is not an excess of choice, is an excess of improvisation.
Long story short... now that hardware speed is not easily doubled every few years, the industry has found a 'simple' way to keep pushing the weel, duplicate cores!. Well, it turns out that after decades of ignoring the parallel programming demand from academics now they are trying to push the 'somewath parallel' mess thay are producing.
The problem is 'duplicated cores' != 'parallel programming', that's the problem.
What's in a sig?
Just use pthreads and forget that other nonsense.
I am not sure you understand the problem, it is *not* how to write a multithreaded program. It is how to write "normal" code, say a for loop, that will automatically be executed in parallel if multiple cores are present.
No, a red fuzzy sphere is NOTHING like a 5-dimensional hypersphere. Even a sphere of 4 spatial dimensions would blow your fucking mind. If we lived in a universe with 4 dimensions of space, then planets would be 4D spheres with a 3D surface. You would be able to walk around, and "turn" up and down without ever leaving the ground, just like you can turn left and right in our universe. It would be impossible to tie a knot. Imagine, if you will, people living on the surface of a circular planet in a 2D universe, then imagine how incredibly different it would be from our 3D universe. If you aren't utterly shocked by thinking about what it would be like to live in 4 spatial dimensions, let alone 5, then you haven't been thinking hard enough.
Choice overload is a problem, but only if you are actually faced with all the choices. It need not be that way. There are various kinds of parallelism and various angles to attack each. Some of the choices one has will not make a lot of sense for type of problem one is looking
at. So, categorizing can help.
Some technologies will be in rapid development, others will be no longer actively maintained, and yet others will be stable but actively maintained. This also affects which choices are good.
Then there's licensing. Depending on the task, closed-source or copyleft
licenses might not be acceptable.
Some of the solutions may be low-level, allowing programmers to build something matching their application out of the provided building blocks, where other solutions may focus on providing higher level constructs, ready to be used. Sometimes, these will match what you need, and sometimes, they won't.
I am sure there are other axes of differentiation. Setting requirements will narrow ones choices, as well as illustrate why choice is a Good Thing. If there were only a few choices, it is unavoidable that none of them would actually fit some sets of requirements.
Now, the thing is that categorizing the various solutions is not something that every potential user of the solutions has to do. Part of the work can be done by the developers of each solution. Presumably, the solution is developed because a satisfactory solution did not already exist. In my opinion, the developers _should_ list related work, compare their solution to it, and explain why they saw fit to develop their solution. This is a standard part of research.
Another part of the work is comparisons done by third parties. Some independent person would go and investigate a number of solutions, and provide a write-up of the requirements they assumed, the solutions they investigated, how these solutions fit their requirements, and what their overall impression of the solutions was (w.r.t. things like ease of setup, documentation, development status, etc.). This, too, is valid research. It should be published, so everyone benefits.
In the end, what you get to do when you need to pick a solution for parallel programming, is
1. Define your requirements
2. Get a list of possible solutions
3. See what has been written about them
4. Check if that seems to be valid (it might be out of date, for one)
5. Possibly investigate any solutions that you found but that haven't been covered by others.
6. Decide which one to go with, based on the information you have gathered.
Sure, this is a far cry from
1. Find the only available solution
2. There is no step 2
but for that you are almost guaranteed to get a choice that better fits your requirements (you would be very lucky to have the only available solution be a great match), without having to pay the full cost of investigating every solution out there.
The thing to remember about the paradox of choice is that you will probably _feel_ less happy (there is always the nagging feeling that you could have made a better choice), but that you will generally end up with something _better_ than if the choice hadn't been there is the first place.
If you _really_ aren't happy about having to choose, you can always pick one (say, at random) and pretend that was your only choice. I conjecture that this is what the situation of having only one option is really like.
Please correct me if I got my facts wrong.
Seems to me to be a philosophical problem. With a single CPU you have a single gate processing a single sequence of instructions. It's easy to push the instructions and data through that gate in order. When you have 10, 20, 100 gates choosing which direction to push the instructions and data becomes exponentially more complex.
The solution it would seem to me would be to start pulling the instructions and data through the gates instead of pushing it.
Deleted
Parallel programming has many facets. Libraries like OpenMP can not be compared to Windows threads, for example: the former is a mechanism for doing task/data parallelism at language level, the latter offers the primitives for making threads. PThreads is a thread API similar to win32 threads. I don't see how there is an overload of choices, really.
I always find it amusing (in a sad kind of way) how people talk about Herb Sutter's "call to action" over this. It's not that I've got anything against Herb himself: he's a decent writer, an excellent speaker, and a guy who can use the word "expert" legitimately in areas like C++. But it's also not like he's the first guy to notice that modern desktop computer architectures have been heading for parallelisation rather than increased speed for several years now.
Despite being right in the thick of this culture shift myself — I'm sure I'm not the only one here who has been talking about this for a while, and is just seeing management catch up — I don't think this is going to be that big of a deal for most people. The harsh reality, for the buzzword-wielding consultants rubbing their hands with glee at a new programming approach they can hype up, is that most people just don't need all this.
Your average desktop PC is more than powerful enough for most things that most people do with it: Internet communications, writing documents, working with databases, shop floor software, and the like. As long as the operating system is reasonably smart about scheduling, the guys writing these common types of applications don't really have to know anything about multithreading, locking, message passing, and all that jazz. Similarly, your average mobile device has more than enough juice to dial another phone, write a quick e-mail, or capture a digital photo.
At the other end of the spectrum, serious servers (database, communications, whatever) have been dealing with parallel processing of many requests since forever. High-end systems doing serious maths (the guys modelling weather systems, say) have also been using massive parallelisation on their supercomputers for zillions of years now.
There is a gap between these different areas, which we might traditionally have called the "workstation" market: the guys doing moderate number crunching for CAD, scientific visualisation, simulations, and the like. Many modern games also fall into this classification. This market is ripe for a parallel processing revolution, because historically it hasn't followed this approach very much because the hardware wouldn't really take advantage of it, yet the extra power is genuinely useful. But I don't think this represents some huge proportion of the software development industry as a whole. The guys working in these areas tend to be pretty smart, and will no doubt adopt useful practices and conventions fairly quickly now that the hardware has reached the point that they are useful.
As to what those conventions are, I just don't buy the whole "choice overload" theory. There are relatively few basic models for parallel processing: for example, you can have no shared state and communicate only through message passing, or you can have shared state. In the latter case, you then have the question of how to make sure that the sharing is safe, which leads to lock-based or lock-free approaches. Funky toys like transactional memory run at a slightly higher level than this, but they are ultimately constructed from the same building blocks, and again there are only a small number of approaches at this level to consider.
I'm not familiar with all of those libraries mentioned in the story, but I'll bet that those three classifications (no shared state, shared state with explicit locking, shared state without explicit locks) probably cover the models used by most if not all of them. If you understand the trade-offs in those, you can produce a sensible design, and then the toolkit or framework you use to code it up is mostly just an implementation detail. Given that the trade-offs are pretty obvious and will often steer projects clearly in one direction, I don't think there's really that much to choose at all.
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
The overload is just a symptom if the real problem and that is that parallel programming is just plain hard. We've had these issues for over a decade and we haven't seen a step function in use of parallel programming. It is difficult for most people to think of many things happening at the same time and to design and debug this class of program. We tend to start by thinking of a task in serial steps and then look for ways to add a little parallelism.
The folks who are low level systems programmers (OS and networks) tend to be folks who have an aptitude for thinking about parallelism and designing with parallelism in mind. There are a group of people in the scientific space who make use of parallelism, but then again they are Phd mathematicians and physicists. After that it drops off rapidly.
Maybe it has something to do with he way we are educated. perhaps it is a more fundamental issue of brain wiring. After all, we c perform complex physical tasks in parallel, but maybe only a small segment of the population is wired to think about programming problems in parallel.
The chip guys are throwing more cores at us and we can't create the software to fully utilize the hardware due to this issue. Perhaps it is time to take a step back and to stop trying to solve the problem by throwing more and different programming packages at the problem and examine why folks have so much trouble in this area.
So do you want to go to a bar with only one woman in it? Or a bar with 20? If you believe the premise of "choice overload" making you unhappy, you should choose the first, right?
Aside from that in many contexts the "choice overload" hypothesis is flat-out wrong (unless you really can claim to have felt a special rush of happiness just when you went into that bar with only one dame in it - and she wasn't, say, your girl already), there are open questions about how representative the test sample was. Psychological problems run culturally in certain populations. How can we be sure that the population tested for "choice overload" didn't share a psychological problem regarding choice that has no foundation is basic human psychology, but rather was relative to their own cultural limitations?
For most people in most cultures over history, the trick is to be happy with not much choice. That's generally the case for the working class, for the infantry soldiers, and for tribal peoples in environments of scarcity. Yet even in those cultures there are other classes for whom the trick is to be happy with a great deal of choice - the upper class, the generals, and tribal peoples in environments of plenty. Those whose cultures and religions derive primarily from desert (scarcity) environments are those driven craziest by "choice overload" - thus the Islamic meltdown, and the rejection of modern freedom by American fundamentalists. But we do have other cultures here. And the studies associated with the "choice overload" hypothesis, do not, I'll bet dimes to dollars, correct in any way for (sub)culture and psychological diagnosis.
"with their freedom lost all virtue lose" - Milton