Is Parallel Programming Just Too Hard?

pcause writes "There has been a lot of talk recently about the need for programmers to shift paradigms and begin building more parallel applications and systems. The need to do this and the hardware and systems to support it have been around for a while, but we haven't seen a lot of progress. The article says that gaming systems have made progress, but MMOGs are typically years late and I'll bet part of the problem is trying to be more parallel/distributed. Since this discussion has been going on for over three decades with little progress in terms of widespread change, one has to ask: is parallel programming just too difficult for most programmers? Are the tools inadequate or perhaps is it that it is very difficult to think about parallel systems? Maybe it is a fundamental human limit. Will we really see progress in the next 10 years that matches the progress of the silicon?"

Comments

Re:our brains aren't wired to think in parallel

[Anonymous Coward]

I do a lot of multithreaded programming, this is my bread and butter really. It is not easy - it takes a specific mindset, though I would disagree that it has much to do with management. I am not a manager, never was one and never will be. It requires discipline and careful planning.

That said, parallel processing is hardly a holy grail. On one hand, everything is parallel processing (you are reading this message in parallel with others, aren't you?). On the other, when we are talking about a single computer running a specific program, parallel usually means "serialized but switched really fast". At most there is a handful of processing units. That means, that whatever it is you are splitting among these units has to give itself well to splitting this number of ways. Do more - and you are overloading one of them, do less - and you are underutilizing resources. In the end, it would be easier often to do processing serially. Potential performance advantage is not always very high (I know, I get paid to squeeze this last bit), and is usually more than offset by difficulty in maintenance.

Re:Two words: map-reduce

[allenw]

Implementing MapReduce is much easier these days: just install and contribute to the Hadoop project. This is an open source, Java-based MapReduce implementation, including a distrbuted filesystem called HDFS.

Even though it is implemented in Java, you can use just about anything with it, using the Hadoop streaming functionality.

Re:Nope.

[Anonymous Coward]

Functional programming is no harder than procedural/OO programming. A good functional interpreter can already draw this kind of parallelism out of a program. And yes, there are compiled functional languages with parallelizing compilers. (Erlang and OCaml come to mind)

Re:A different approach to parallel programming

[jlarocco]

I've often wondered if parallel programming would be easier if it were done in Chinese characters instead of English/European alphabetical characters.

Perhaps having a Chinese character represent a simple block of pre-compiled code that does one simple thing. Then the characters could be placed in two-dimensional order to form parallel threads. This would require a completely different approach to compiler development. But that would be OK because compilers are stuck in the 1970s anyway.

Maybe I'm a "bit thick", but that doesn't make any sense to me. It's an interesting idea, but I just don't see how it'd help parrallelize things. At the very least, it seems to be solving the wrong problem.

The biggest problem right now is that it's really hard to split most tasks into parts that can be performed at the same time. Once a parrallel algorithm is devised, it's relatively easy to write a program that performs the task in parrallel.

Also, I don't know what you mean about compilers being stuck in the 70s. There have been massive improvements to compilers in the last 40 years.

Just getting away from the idea of having code based on a very limited set of alphanumeric characters strung together like beads on a string might help unlock a whole new era of innovative approaches to parallel program development strategies.

But programming doesn't work like that. Individual characters in a programming language are almost irrelevant.

Lack of skill in the field

[node159]

I've seen it over and over in the industry, there is a distinct lack of parallel programming skill and knowledge, and even senior developers with years under their belt struggle with the fundamentals.

Parallel programming does add a layer of complexity and its inherent lack of general solutions does make abstracting its complexity away difficult, but I suspect that the biggest issue is the lack of training of the work force. It isn't something you can pick up easily without a steep learning curve with many hard lessons in it, definitely not something that can be incorporated as a new thing to be learnt on the fly with deadlines looming.
Another aspect is that its fundamental to the design, parallelism can and often will dictate the design and if the software architects are not actively designing for it or are not experienced enough to ensure that it remains a viable future option, future attempts to parallelise can be difficult at best.

Ultimately the key issues are:
* Lack of skill and training in the work force (including the understanding that there are no general solutions)
* Lack of mature development tool to easy the development process (debugging tools especially)