Is Parallelism the New New Thing?

astwon sends us to a blog post by parallel computing pioneer Bill McColl speculating that, with the cooling of Web 2.0, parallelism may be a hot new area for entrepreneurs and investors. (Take with requisite salt grains as he is the founder of a Silicon Valley company in this area.) McColl suggests a few other upcoming "new things," such as Saas as an appliance and massive memory systems. Worth a read.

Comments

1% of programmers

[LotsOfPhil]

Only around 1% of the world's software developers have any experience of parallel programming.

This seems far, far too low. Admittedly I work in a place that does "parallel programming," but it still seems awfully low.

Re:1% of programmers

[Otter]

Admittedly I work in a place that does "parallel programming," but it still seems awfully low.

I think your experience is wildly skewed toward the high end of programming skill. The percentage of working programmers who can't iterate over an array is probably in the 15-20% range, even without getting into whether "web programmers" are included in that statistic. I'd be astonished if the number with parallel experience is significantly above 1%.

Re:1% of programmers

[Anne Thwacks]

I'd believe 1% as having "some experience". Some experience is what you put on your CV when you know what the buzword means.

If you ask how many can "regularly achieve significant performance through use of multiple threads" then 0.1% is far too high. If you mean "can exchange data between a userland thread and an ISR in compliance with the needs of reliable parallel execution" then its a safe bet that less than 0.1% are mentally up to the challenge. /. readers are not typical of the programming cummiity. These days people who can drag-and-drop call themselves programmers. Poeple who can spell "l337" are one!

evolution, not revolution

[nguy]

the guy has a "startup in stealth mode" called parallel computing. Of course he wants to generate buzz.

Decade after decade, people keep trying to sell silver bullets for parallel computing: the perfect language, the perfect network, the perfect os, etc. Nothing ever wins big. Instead, there is a diversity of solutions for a diversity of problems, and progress is slow but steady.

Re:evolution, not revolution

[jd]

There are hundreds of libraries, yes. That's part of the problem. Too many different ways of slicing the same pie, leading to solutions that are not efficient when trying to get them to play nice with each other.

The next problem is that parallelism is not, as a rule, CPU-bound but network-bound. All the libraries in the world won't work when the network clogs and chokes.

The third problem is that coders are taught serial methods. Parallel thinking is very different from serial thinking. You run into problems that do not exist in the serial world, even on a timeslicing system like Linux. True parallelism, like true clockless computing, is a nightmare to do well. You can't just shove another library in and hope things'll work.

The fourth problem is the level of connectedness. Globus is a great toolkit for some things, but you wouldn't use it for programming a vector computer or - most likely - even a Beowulf cluster. It's a gridding solution and a damn good one, but grids are all it will do well. On the flip-side, solutions like bproc and MOSIX are superb mechanisms for optimally using a fairly tight cluster, but you'd never sanely use them on a grid. The latencies would make the very features that make those solutions useful in a cluster useless on a WAN-based grid.

I'm not sure I'm keen on Java on any parallel solution other than gridding. It's too slow, its threading model is still in its infancy, and the sandboxing makes RDMA an absolute nightmare to do safely. In fact, the very definition of sandboxing is that external entities can't go around poking bits of data into memory, which is the entire essence of RDMA - CPUless networking.

Regardless, there are some things that C++ and Java simply cannot do well that other, parallel-specific languages like Pi-Occam can do with extreme ease and safety. It is possible to prove an Occam program is safe. You cannot do likewise with a C++ or Java program.

Parallelism isn't just about more threads on one CPU. In a totally generalized parallel scenario, there may be any number of threads - a few tens of thousands would not be unusual - running on systems that may be SMP, multi-core, multi-threaded, vectored, clockless, or any combination of the above, where those systems may be on a tightly-coupled or loosely-coupled cluster, and where the cluster may be homogenous or heterogeneous, SSI or multi-imaged, where memory may be local, NUMA or distributed, and where these systems/clusters may be gridded over wide-area networks that may or may not be reliable or operational at any given time, and where threads, processes and entire operating systems may migrate from system to system without user intervention or awareness on the part of the application.

The number of true parallel experts in the world probably number less than a dozen. No, I'm not one of them. I'm good, I understand the problem-space better than the average coder, but I've talked to some of the experts out there and they're as far beyond any traditional programmer as a traditional programmer is beyond the chipmunk. A network engineer might consider themselves OK if they can set up OSPF optimally across a traditional star network of star networks. Any traditional routing protocol over a mesh without getting flaps and maintaining a reasonable level of fault tolerence would be considered tough. A butterfly network, a torroidal network or a hypercube would leave said network engineer a gibbering wreck. Modern supercomputers do not take up buildings. Modern supercomputers take up a few rooms. The interconnects take up entire buildings. And the air conditioning on top systems can be measured in football stadia.

OpenMOSIX is largely dead, because it was impossible to reconcile those who wanted load-balancing with those who wanted HPC. It's not that they can't be reconciled in theory, it's that the mindsets are too different to cram into one brain.

If one solution could solve parallelism, the Transputer would be the only processor in use today and Intel would

Parallel computing pioneer likes Parallelism

[Alzheimers]

A guy who's made it his life's work to study Parallel Computing has come forth to say, he thinks Parallelism is the next big thing?

Shock! And Awe!

"Next hot thing" my hiney

[Rosco P. Coltrane]

For having been in the computer industry for too long, I reckon the "next hot thing" usually means the "latest fad" that many of the entrepreneurs involved in hope will turn into the "next get-rich-quick scheme".

Because really, anybody believes Web-Two-Oh was anything but the regular web's natural evolution with a fancy name tacked on?

More so now, but depends ...

[Midnight Thunder]

Now that we are seeing more and more in the way of multi-core CPUs and multi-CPU computers I can definitely see parallelism become more important, for task that can be handled this way. You have to remember that in certain cases trying to parallise a task can end up being less efficient, so what you parallelise will depend on the task in hand. Things like games, media application and scientific applications are usually likely candidates since they are either doing lots of different things at once or have tasks that can be split up into smaller units that don't depend on the outcome of the other. Server applications can to a certain extent, depending whether they are trying to the same resources or not (ftp server, accessing this disk, vs a time server which does not file I/O).

One thing that should also be noted, is that in certain cases you will need to accept increased memory usage, since you want to avoid tasks locking on resources that they don't really need to synchronise until the end of the work unit. In this case it may be cheaper to duplicate resources, do the work and then resynchronise at the end. Like everything it depends on the size and duration of the work unit.

Even if your application is not doing enough to warrant running its tasks in parallel, the operating system could benefit, so that applications don't suffer on sharing resources that don't need to be shared.

"...with the cooling of Web 2.0,..."

[sm62704]

Sorry guys, web 2.0 was never cool and never will be.

What am I missing?

[James Lewis]

Now that multi-core computers have been out I keep hearing buzz around the idea of parallel computing, as if it is something new. We've had threads, processes, multi-CPU machines, grid computing, etc etc for a long time now. Parallelism has been in use on single processor machines for a long time. Multi-core machines might make it more attractive to thread certain applications that were traditionally single-threaded, but that's the only major development I can see. The biggest problem in parallel computing is the complexity it adds, so hopefully developments will be made in that area, but it's an area that's been researched for a long time now.

Re:About time

[jc42]

When I was in graduate school in the mid '90's I thought Parallelism would be the next big thing.

When I was in grad school back in the 1970s, people thought parallelism would be the next big thing, and it had some interesting technical challenges, so I got into it as much as was possible back then. Then I got out into the Real World [TM], where such ideas just got blank looks and "Let's move on" replies.

Some what later, in the 1980s, I worked on projects at several companies who thought that parallelism was the next big thing. That time around, I got my hands on a number of machines with hundreds of processors and gigabytes (Wow!) of memory, so I could actually try out some of the ideas from the previous decade. The main things that I learned was that 1) many of the concepts were viable, and 2) debugging in an environment where nothing is reproducible is hard. And I moved on, mostly to networking projects where you could actually do loosely-coupled multiprocessing (though management still gave you blank looks if you started talking in such terms).

Now we're getting personal computers with more than one processor. It's been a bit of a wait, but we even have management saying it's the "new^N thing". And debugging parallel code is still hard.

I'll predict that 1) We'll see a lot of commercial parallelized apps now, and 2) those apps will never be debugged, giving us flakiness that outshines the worst of what Microsoft sold back in the 1980s and 1990s. We'll still have the rush to market; developers will still be held to nonsensical schedules; debugging will still be treated as an "after-market" service; and we developers will still be looking for debugging tools that work for more than toy examples (and work with a customer's app that has been running for months when a show-stopper bug pops up).

There's a reason that, despite the existence of multi-process machines for several decades, we still have very little truly parallel code that works. Debugging the stuff is hard, mostly because bugs can rarely be reproduced.

(Of course, this won't prevent the flood of magical snake-oil tools that promise to solve the problem. There's a lot of money to be made there. ;-)

Re:Multithreading Is to Blame

[grumbel]

Just because the majority of developers can't doesn't mean it has "failed" When you want to get parallelism into the mainstream then that is pretty much the exact definition of "fail".