Slashdot Mirror
Choice Overload In Parallel Programming
scott3778 writes to recommend a post by Timothy Mattson over at Intel's Research Blog. He argues, convincingly, that the most important paper for programming language designers to read today is one written by two social psychology professors in 2000. This is the well-known academic study, "When Choice is Demotivating: Can One Desire too Much of a Good Thing?" "And then we show them the parallel programming environments they can work with: MPI, OpenMP, Ct, HPF, TBB, Erlang, Shmemm, Portals, ZPL, BSP, CHARM++, Cilk, Co-array Fortran, PVM, Pthreads, windows threads, Tstreams, GA, Java, UPC, Titanium, Parlog, NESL,Split-C... and the list goes on and on. If we aren't careful, the result could very well be a 'choice overload' experience with software vendors running away in frustration."
http://www.columbia.edu/~ss957/whenchoice.html
Those who would give up essential liberty to purchase a little temporary safety, deserve neither liberty nor safety.
Microsoft will come along and tell you what your choice will be.
Write concurrently in two languages, then you're sure to make full use of available CPU cores.
Quoth the blogger: "With hundreds of languages and API's out there, is anyone really dumb enough to think "yet another one" will fix our parallel programming problems?"
Yet Intel touts its Threading Building Blocks library as just such a fix to many parallel programming problems. Now, TBB is a very nice product, and in many ways it is superior to a lot of existing libraries, APIs, and languages, but one gets the sense that maybe the left hand doesn't know what the right hand is doing at Intel.
I might also draw an analogy to the open source world, where there are often dozens of solutions to both simple/mundane problems (text editors, media players, command line shells, etc) and more complex ones (window managers, Linux distributions, etc). I wonder if the free and open source software world wouldn't also benefit from a "culling of the herd," so to speak.
Actually, both of your analogies are poor. The problem with "choice overload" in the software context is that with so many platforms to choose from, no one platform builds the critical mass to be useful for a broad range of problems, and developers are almost certain to build systems and components that do not interoperate because they are built in separate frameworks. In software, there's a benefit to having everyone chose the same platform to build on.
On the other hand, I don't know about the benefit of everyone chosing the same girl or the same restaurant, though--unless you like gang-bangs, long lines, etc.
and I say that as an atheist.
Ok, first: he writes as if all choices are equivalent. One jam might as well be the same as another, they just differ by taste. It's not like I walk into the store already invested raspberries. It's not as if Java programmers are going to decide that the Fortran parallel library is better, so why not just switch to Fortran.
Second, I doubt explicit parallel programming is going to be mainstream anytime soon. No, make that ever. Ever! Parallel programming will only happen in the mainstream when it is handled implicitly by the language, like a dataflow language. Asking normal programmers to deal with parallel programming is trouble when basic logic eludes most of them.
Third, all you people, including the author of TFA, who think that more than one or two standards is bad thing ("the great thing about standards is there are so many to choose from!") it's time to wake up: the world is not about to consolidate. The future is going to require C3PO and R2D2: there will be so many fricking languages and standards that your translator is going to require AI and legs to come along with you. For every one thing that fades away, eventually, probably 10 or 100 replace it. The future is a big mess.
must... stay... awake...
Choice is good if it provides different tools for different tasks. The list provided is somewhat silly, since several of the technologies address completely different issues and applications. There's a reason Sears sell thirty different shapes of hammers -- all nails are not the same.
After considerable deliberation and experimentation, I've shosen OpenMP for most task-parallel applications. The syntax is simple, it operates across C, C++, and Fortran, and it is supported by most major compilers on Linux, Windows, and Sun. The only quirk has been problematic support in GCC 4.2, but that will likely be cleared up within a few months. For cluster work, I tend to use MPI, because it has a long history and good support. I'm sure other tools have good versatility in environments different from those I frequent.
All about me
On top of that, if this really is something that affects programmers then why the hell aren't we all rendered utterly useless by the number of programming languages? Or all the possible ways one could format code? Etc.
But hey, the guy's writing in a "research" blog and, as in academia, when you don't have anything real to contribute you can cite something completely unrelated and pretend it has relevance.
Honestly, this sounds vaguely like "there's too much to choose from, so everyone just use Intel Thread Building Blocks, K? You can't possibly do better so just use our stuff because we cover all cases..."
I've been gradually trying to learn more about functional programming, partly because I think fp techniques and ways of thinking come in handy even if you're programming in a procedurally oriented language, and partly because fp seems like a paradigm that is likely to get more and more useful as we get machines with more and more cores. Okay, fp!=parallel, but, e.g., one of the big selling points of Erlang is supposed to be that it lends itself to completely transparent use of parallel processors.
The choice overload does seem like kind of an issue to me. For as long as I continue to keep programming comfortably in the procedural languages I'm comfy with (e.g., perl), I'm never going to really wrap my mind around the radically different ways of thinking that you get in a more fp world. I'm been thinking for a long time that it would be fun to do a coding project in ocaml ... or haskell ... or lisp ... or erlang ... or -- you get the idea.
The trouble is, it's really not clear what to hitch my wagon to. Ocaml seems to have a very high quality implementation, but its garbage collector isn't multithreaded, the only book you can buy is in French (it's nice that you can download the English version for free, but I'd prefer to buy something bound), and the availability of libraries (and documentation for them) isn't quite as wonderful as I've gotten used to with perl. Lisp could be cool, but I hate the fact that it's not standardized, and I'm not convinced that eschewing arbitrary syntax really carries more pros than cons. Haskell? Maybe, but it sounds like putting on a hair shirt. The list goes on. I really feel like a deer in the headlights.
Find free books.
That's just silly. There are two types of programmers that could be making choices like this, and neither one of them would suffer from too much choice.
The first kind is a programmer just trying to paralelize existing code. In that case, the choice of threading platforms is pretty much obvious. Existing Windows code? Use windows threads. C/C++ on Unix? Pthreads probably. Java code? Java threads... Probably not even 2 seconds worth of thought will go into considering the alternatives (and that's probably fine)
The other type of programmer is one who's actively looking to develop high performance paralelized software. I am talking about cases where performance is the primary objective and it drives the choice of programming language and platform. In these cases, the nuance of the different thread models might matter but the programmer of this type would be happy (rather than scared) to investigate all the options. After all, if he didn't care, he'd just go with the default choices like the first programmer.
http://ed.markovich.googlepages.com
Uh, most good parallel programming frameworks are cross-language. MPI has APIs in C, Fortran, and C++ (I believe) ... so does OpenMP. Also, most supercomputing programs are either written in C or Fortran. So, YES!, choose Fortran (and MPI) for all of your supercomputing needs. Or write it in C. Or write subroutines/functions in both languages and compile away (the newest version of Fortran will be fully C-compatible).
... the actual implementation of parallel programming! Is it distributed (MPI) or shared (OpenMP)? Does it have elegant syntax for accessing variables across processors (Co-Array) or just function based? Because there is no one true way to write a parallel program (it really depends on the algorithm), there will always be multiple frameworks to choose from. O well! The people who write parallel programs are typically smart enough to deal with excessive choices. (No comment on others).
The main differences between these parallel programming frameworks are
Okay this list seems to be of several different technologies some of which over lap but several are used for very different tasks. You can not replace MPI with Pthreads.
I don't see the problem. Just as we have many different programing languages these different interfaces all have different niches.
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
- Jams are functionally equivalent; the choice is inconsequential. This is far from the case with programming languages, which have meaningful differences.
- Programming languages solve important problems, so a choice will be made. You can't just give up on the whole idea and walk away as with specialty jams.
- There are so many different aspects of a language, you can have a great number of them, yet they can all be very different from each other.
- Significant resources are devoted to developing and choosing parallel languages. This greatly increases the number of choices that can be evaluated. Consider how much time you spend shopping for the right car vs. a jar of jam.
Now would be a terrible time to stop developing parallel languages, because the problem is just now coming to the forefront with the limits of single-core performance pushing back and multi-cores taking over. I'm suspect the parallel programing paradigm of the next 40 years hasn't been invented yet, and I'm almost certain it hasn't yet been popularized. So I say, let a hundred flowers bloom.Just use pthreads and forget that other nonsense.
I am not sure you understand the problem, it is *not* how to write a multithreaded program. It is how to write "normal" code, say a for loop, that will automatically be executed in parallel if multiple cores are present.
I always find it amusing (in a sad kind of way) how people talk about Herb Sutter's "call to action" over this. It's not that I've got anything against Herb himself: he's a decent writer, an excellent speaker, and a guy who can use the word "expert" legitimately in areas like C++. But it's also not like he's the first guy to notice that modern desktop computer architectures have been heading for parallelisation rather than increased speed for several years now.
Despite being right in the thick of this culture shift myself — I'm sure I'm not the only one here who has been talking about this for a while, and is just seeing management catch up — I don't think this is going to be that big of a deal for most people. The harsh reality, for the buzzword-wielding consultants rubbing their hands with glee at a new programming approach they can hype up, is that most people just don't need all this.
Your average desktop PC is more than powerful enough for most things that most people do with it: Internet communications, writing documents, working with databases, shop floor software, and the like. As long as the operating system is reasonably smart about scheduling, the guys writing these common types of applications don't really have to know anything about multithreading, locking, message passing, and all that jazz. Similarly, your average mobile device has more than enough juice to dial another phone, write a quick e-mail, or capture a digital photo.
At the other end of the spectrum, serious servers (database, communications, whatever) have been dealing with parallel processing of many requests since forever. High-end systems doing serious maths (the guys modelling weather systems, say) have also been using massive parallelisation on their supercomputers for zillions of years now.
There is a gap between these different areas, which we might traditionally have called the "workstation" market: the guys doing moderate number crunching for CAD, scientific visualisation, simulations, and the like. Many modern games also fall into this classification. This market is ripe for a parallel processing revolution, because historically it hasn't followed this approach very much because the hardware wouldn't really take advantage of it, yet the extra power is genuinely useful. But I don't think this represents some huge proportion of the software development industry as a whole. The guys working in these areas tend to be pretty smart, and will no doubt adopt useful practices and conventions fairly quickly now that the hardware has reached the point that they are useful.
As to what those conventions are, I just don't buy the whole "choice overload" theory. There are relatively few basic models for parallel processing: for example, you can have no shared state and communicate only through message passing, or you can have shared state. In the latter case, you then have the question of how to make sure that the sharing is safe, which leads to lock-based or lock-free approaches. Funky toys like transactional memory run at a slightly higher level than this, but they are ultimately constructed from the same building blocks, and again there are only a small number of approaches at this level to consider.
I'm not familiar with all of those libraries mentioned in the story, but I'll bet that those three classifications (no shared state, shared state with explicit locking, shared state without explicit locks) probably cover the models used by most if not all of them. If you understand the trade-offs in those, you can produce a sensible design, and then the toolkit or framework you use to code it up is mostly just an implementation detail. Given that the trade-offs are pretty obvious and will often steer projects clearly in one direction, I don't think there's really that much to choose at all.
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.