Is Parallelism the New New Thing?

astwon sends us to a blog post by parallel computing pioneer Bill McColl speculating that, with the cooling of Web 2.0, parallelism may be a hot new area for entrepreneurs and investors. (Take with requisite salt grains as he is the founder of a Silicon Valley company in this area.) McColl suggests a few other upcoming "new things," such as Saas as an appliance and massive memory systems. Worth a read.

Comments

About time

[olddotter]

When I was in graduate school in the mid '90's I thought Parallelism would be the next big thing. Needless to say I was a bit early on that prediction. Finally maybe those graduate classes and grant work will pay off. :-)

Re:About time

[pleappleappleap]

As a user of Linux, I have to say the Parallelism is the 'old thing', as Linux has supported parallel operations for over a decade. Compare this to closed source, proprietary operating systems, such as Windows, where this sort of thing is relatively new.

Windows is not the only closed-source proprietary operating system out there. AIX and Solaris have supported parallel functions for a number of years, and various IBM mainframe operating systems have had those functions since the '70's. There are architectures which had it in the '60's.

Proprietary closed-source operating systems had these functions FIRST before Linux was a twinkle in Linus Torvalds's shorts.

Re:About time

[Gerzel]

"Proprietary closed-source operating systems had these functions FIRST before Linux was a twinkle in Linus Torvalds's shorts."

Do not mock the shorts of Torvalds, for they are mighty indeed!

Re:About time

[jc42]

When I was in graduate school in the mid '90's I thought Parallelism would be the next big thing.

When I was in grad school back in the 1970s, people thought parallelism would be the next big thing, and it had some interesting technical challenges, so I got into it as much as was possible back then. Then I got out into the Real World [TM], where such ideas just got blank looks and "Let's move on" replies.

Some what later, in the 1980s, I worked on projects at several companies who thought that parallelism was the next big thing. That time around, I got my hands on a number of machines with hundreds of processors and gigabytes (Wow!) of memory, so I could actually try out some of the ideas from the previous decade. The main things that I learned was that 1) many of the concepts were viable, and 2) debugging in an environment where nothing is reproducible is hard. And I moved on, mostly to networking projects where you could actually do loosely-coupled multiprocessing (though management still gave you blank looks if you started talking in such terms).

Now we're getting personal computers with more than one processor. It's been a bit of a wait, but we even have management saying it's the "new^N thing". And debugging parallel code is still hard.

I'll predict that 1) We'll see a lot of commercial parallelized apps now, and 2) those apps will never be debugged, giving us flakiness that outshines the worst of what Microsoft sold back in the 1980s and 1990s. We'll still have the rush to market; developers will still be held to nonsensical schedules; debugging will still be treated as an "after-market" service; and we developers will still be looking for debugging tools that work for more than toy examples (and work with a customer's app that has been running for months when a show-stopper bug pops up).

There's a reason that, despite the existence of multi-process machines for several decades, we still have very little truly parallel code that works. Debugging the stuff is hard, mostly because bugs can rarely be reproduced.

(Of course, this won't prevent the flood of magical snake-oil tools that promise to solve the problem. There's a lot of money to be made there. ;-)

Re:About time

[ChrisA90278]

I was a systems programming on the CDC6400. This at one time was the world's fastest super computer and was released in 1964 (hence the model number.) I'm not that old. The machine was old when I worked on it. I did patches and custom modification to the operating system. We worked in asembly language so I did have to know the gory details of the hardware.

It was a highly parallel machine. At every level. It could execute 10 instuctions at once. There was a 10 way path to main memory so we could do 10 store or fetches at once. Some machine had multiple CPUs and all machine had at least 10 PPUs, there were smaller CPU where most of the operating system actualy ran keepping the main CPu free for user level tasks.

To this day I've not seen this level parallelsm. Ok the old Pentium had a long pipeline and Sun will sell you an 8-core SPARC but still nothing compared to the old CDC machines

The wiki article below has some details wrong but the ideas are describbed well. The machine was built at a time when electronics tachnology had hit a wall and couldn't be had faster. They had already gone to extreams like giing up on water cooling to use freon cooling where the refridgerent was plumbed directly into the guts of the CPUs card guides. The only option to go faster was parallelium. I think we wil see this again as we hit a wall with IC technology
http://en.wikipedia.org/wiki/CDC_6600

Let me know when I can buy

[geekoid]

32G chips and put them in 4 slots on my 64 Bit PC before talking about 'massive memory'

Re:Let me know when I can buy

[Sancho]

I agree that parallelism is more easily (and transparently) used at the OS level, but that doesn't mean that we don't need to start moving that way in applications, too. As we move towards a point where extracting more speed out of the silicon becomes harder and harder, we're going to see more and more need for parallelism. For a lot of applications, it's going to be irrelevant, but for anything at all CPU intensive (working with video, games, etc.) it's going to eventually be the way of the future.

TRIPS

[Bombula]

Someone in another recent thread mention the TRIPS architecture. It's quite interesting reading.

Re:TRIPS

[The Living Fractal]

TRIPS and EDGE are interesting approaches to parallel processing. The thing is, data interdependency is going to make execution speed remain as the trump card for computing. That is to say: you cannot parallelize an algorithm that requires step-by-step completion. EDGE simply identifies, at the compilation level, what parts of a program can be parallelized, based on interdependencies, and then it creates the program based on this in the form of 'hyperblocks'.

If each subsequent step is dependent on the previous step then this by nature makes the program impossible to parallelize. So what we're most likely going to find out is that we'll hit the 'wall' of parallelization in programs in a relatively short period and be back to the familiar place: increasing the clock speed.

Re:TRIPS

[The Living Fractal]

I certainly agree with your post from a system-level perspective of abstraction. I.E. if we design it into the system at every level, from hardware up through the layers of abstraction to software and the OS, then we will see the largest possible gain from parallelization. Computers will be able to utilize hundreds, thousands or millions or more 'micro-cores' to perform complex tasks faster than ever before.

I guess my point is that I think we'll actually create the basic, expandable model fairly quickly. Would you agree that today's supercomputing, which utilizes parallelization on a scale far beyond desktop computing, has successfully harnessed parallelization? I hope you would. If so, then the next step is miniaturization of what supercomputing is already doing. That step is just now taking place. It's not something that will happen overnight, but I do think that after we've fully integrated parallelization into everyday computing we'll be back to the same old game again: that of looking for ever better ways to increase FLOPS through transistor/switch speed.

My basic thoughts on this are that it is, in theory, easier to model the perfect parallelization of a program, and the optimum number of cores for a specific type of computer, than it is to model the fastest possible clock speed of a CPU. Because of this we'll probably see diminishing returns in advancement of parallelization at an accelerated rate compared to CPU design and clock speed.

Re:TRIPS

[olddotter]

Unless we are going to multi-cores because increasing clock rate is getting hard to do.

I don't think "supercomputer" approaches (which I admit I think of as the same as "massively parallel scientific computing") are applicable to most applications today. Exceptions might be video, picture, and audio editing; and maybe certain types of database operations. If you have other examples I'd be most interested in hearing about them.

1% of programmers

[LotsOfPhil]

Only around 1% of the world's software developers have any experience of parallel programming.

This seems far, far too low. Admittedly I work in a place that does "parallel programming," but it still seems awfully low.

Re:1% of programmers

[Shados]

I'd be sceptical of the source of the information too (Intel's blog as you posted), but that doesn't seem that low to me...

The entire hierarchy system in the IT fields has to deal with the painfully obvious fact that less than 1% of programmers know what they're doing: that is, in an ideal scenario, everyone would know what they're doing, and you'd have a FEW hardcore computer scientists to handle the nutso theoritical scenarios (most parallel programming for example can be done with only basic CS knowledge... like, let say, trimming all of the strings in an array...everyday stuff). But thats not how it works: In the current world, you usually have a few douzan coding monkeys with some understanding of the theory (but no clue how to use it), and a few software architects who call the shots (Note i'm talking about software architects...those that work on software design, not system architects).

In an ideal world where programmers know how to do the -basics- of their job, software architects would be an obsolete job, yet they're really not.

Now, thats a side note, but my point is that the typical IT shop hierarchy is MADE with the "all programmers suck" in mind...so it really has to be a problem, and thus the 1% number isn't far fetched.

As for anecdotal evidence... I'm a .NET and Java dev, and those languages, while it could be better, make parallel programming relatively simple (at least, it is definately within the grasp of a code monkey). Yet, I have -never ever ever-seen any async/parallel code in anything but my own code, everywhere I worked (and I'm doing consulting, so I worked in a lot of places in a short amount of time).

The closest to parallel I've seen was people using a queue dispatcher (MSMQ) and have multiple triggers watching the same queue, thus resulting in some kind of parallel execution... thats it though.

So, 1% from my point of view actually seems pretty -high-, as sad as it is.

Re:1% of programmers

[postbigbang]

Your number is a bit insulting.

Consider that parallel computing means keeping extra monolitic cores busy. There are a number of programmers that need the discipline to know how to spawn, use, and tear down threads to keep them busy. But there are a helluva lot of them that plainly don't need to know. What we lack are reasonable compilers that allow the hardware layer to be sufficiently abstracted so that code can adapt to hardware infrasturcture appropriately. If that doesn't happen, then code becomes machine-specific rather than task-specific/fulfilling. If an app is written, then it should be able to take advantage of parallelism, and the plumbing should take care of the substrate, be that substrate a core-duo, or 64 cores and more, perhaps on separate systems with the obvious latencies that distance might inject.

Those layers are only nominally addressed in operating systems, which should be the core arbiter of how parallelism can be manifested to an application instance. It's up to the kernel makers to figure out how to take advantage and make that advantage useful to applications writers-- who should be at least knowledgable about how to twig those features in the operating system. But app writers needn't have to know all of the underlaying differential to write flexible code.

Re:1% of programmers

[Otter]

Admittedly I work in a place that does "parallel programming," but it still seems awfully low.

I think your experience is wildly skewed toward the high end of programming skill. The percentage of working programmers who can't iterate over an array is probably in the 15-20% range, even without getting into whether "web programmers" are included in that statistic. I'd be astonished if the number with parallel experience is significantly above 1%.

Re:1% of programmers

[AKAImBatman]

In an ideal world where programmers know how to do the -basics- of their job, software architects would be an obsolete job

Not really. The structure of a large system still has to be defined by someone. The key difference is that the architect would get a lot more feedback from his team, and could possibly even farm out high-level pieces of the design to be further architected by other developers.

I'm a .NET and Java dev, and those languages, while it could be better, make parallel programming relatively simple (at least, it is definately within the grasp of a code monkey). Yet, I have -never ever ever-seen any async/parallel code in anything but my own code, everywhere I worked (and I'm doing consulting, so I worked in a lot of places in a short amount of time).

That's because parallel code is the job of the J2EE or IIS server. Programmers develop modules that are loaded by the app server and run asynchronously. Granted, things are divided along he lines of one connection == one thread (though some app servers use poll/select for more granular control), but that's good enough to where systems like Sun's T1 "Niagara" processor can churn through a web load WAY faster than the single-threaded monstrosities we've been using to date.

And before you try to tell me that "application servers don't count", ask any parallel computing researcher if he thinks the threaded programming model is a good idea. You'll almost always get a resounding, "NO". Most of the parallel computing folks I've spoken with rave on and on about lambda and the inherent parallel nature of lambda functions.

The thing is, once parallelism takes off (if one can reasonable argue that it hasn't already), coding will be more about creating parallelizable modules rather than creating threads. The theory will be that the platform running the modules knows more about its resources and how to balance them than the code does. So by exposing areas where parallelism can occur, a language exposes the opportunity to split execution across many processors.

Let's take an ideal example: Let's say you create a raytracing function to cast a ray. Well, that function is inherently parallel in nature. All you need is to map a list of rays to the function and let the platform work out how to balance that function across many processing units.

A slightly less ideal (yet still parallelizable) example is collision detection in a video game. Collision detection is a matrix of objects that can interact. In general, that means that you want to test two objects against each other to see if they have collided or not. If they have, trigger an event to update the state of those objects. (e.g. explode, reverse direction, etc.) Once again, you can have a collision function that takes two items and works out if they have collided or not. A very parallelizable situation. Even firing any resulting events can be done in parallel, as long as the platform is careful not to dispatch multiple events on the same object in parallel. (That creates an out-of-order code issue which can be a bit tricky to resolve.)

Long story short: Parallelism is hard; expect the solution to be as invisible as possible.

Re:1% of programmers

[AKAImBatman]

To date the only frameworks I know of that allow fully asynchronous efficient processing of requests and database queries (so that you don't need to spawn a new thread/process for every request) is ASP.NET, WCF (SOAPish stuff in .NET), and some of .NET's database APIs.

How do you see that as different from what Java J2EE does? Most J2EE servers these days use pools of threads to handle requests. These threads are then utilized based on poll/select APIs so that one thread can handle many requests depending on the availability of data. Database connections are similarly pooled and reused, though any optimization on the blocking level would need to be done by the JDBC driver.

Re:1% of programmers

[Anne Thwacks]

I'd believe 1% as having "some experience". Some experience is what you put on your CV when you know what the buzword means.

If you ask how many can "regularly achieve significant performance through use of multiple threads" then 0.1% is far too high. If you mean "can exchange data between a userland thread and an ISR in compliance with the needs of reliable parallel execution" then its a safe bet that less than 0.1% are mentally up to the challenge. /. readers are not typical of the programming cummiity. These days people who can drag-and-drop call themselves programmers. Poeple who can spell "l337" are one!

Re:1% of programmers

[Otter]

Do you have anything, even anecdotal evidence to support that?

Conveniently, the DailyWTF steps in to provide some anecdotal evidence:

When I showed my lead the old code and the new, he responded, ah, that must have been Jed Code; yeah, he really hated anything that had to use arrays or loops, he couldn't see the point of them ... I think each month he would uncomment the next month and redeploy the application

In fact, I've encountered quite a few programmers (whom I don't hire, so don't blame me) who don't understand anything past variable assignment and flow control. I also know that the people who do hire programmers routinely ask the most basic questions about iteration and weed out quite a few candidates that way.

Incidentally, I didn't mean to denigrate web developers, who come in great, good, adequate and DailyWTF, just as everyone else in IT does. But I'd be surprised if including them in "developers" didn't further drive down the percentage with experience in parallelization.

Re:ACtually

[garett_spencley]

Paul ?

Paul Otellini ?

I didn't know you posted on slashdot !

So what's up man ? Can I buy you a beer ?

evolution, not revolution

[nguy]

the guy has a "startup in stealth mode" called parallel computing. Of course he wants to generate buzz.

Decade after decade, people keep trying to sell silver bullets for parallel computing: the perfect language, the perfect network, the perfect os, etc. Nothing ever wins big. Instead, there is a diversity of solutions for a diversity of problems, and progress is slow but steady.

Re:evolution, not revolution

[jd]

There are hundreds of libraries, yes. That's part of the problem. Too many different ways of slicing the same pie, leading to solutions that are not efficient when trying to get them to play nice with each other.

The next problem is that parallelism is not, as a rule, CPU-bound but network-bound. All the libraries in the world won't work when the network clogs and chokes.

The third problem is that coders are taught serial methods. Parallel thinking is very different from serial thinking. You run into problems that do not exist in the serial world, even on a timeslicing system like Linux. True parallelism, like true clockless computing, is a nightmare to do well. You can't just shove another library in and hope things'll work.

The fourth problem is the level of connectedness. Globus is a great toolkit for some things, but you wouldn't use it for programming a vector computer or - most likely - even a Beowulf cluster. It's a gridding solution and a damn good one, but grids are all it will do well. On the flip-side, solutions like bproc and MOSIX are superb mechanisms for optimally using a fairly tight cluster, but you'd never sanely use them on a grid. The latencies would make the very features that make those solutions useful in a cluster useless on a WAN-based grid.

I'm not sure I'm keen on Java on any parallel solution other than gridding. It's too slow, its threading model is still in its infancy, and the sandboxing makes RDMA an absolute nightmare to do safely. In fact, the very definition of sandboxing is that external entities can't go around poking bits of data into memory, which is the entire essence of RDMA - CPUless networking.

Regardless, there are some things that C++ and Java simply cannot do well that other, parallel-specific languages like Pi-Occam can do with extreme ease and safety. It is possible to prove an Occam program is safe. You cannot do likewise with a C++ or Java program.

Parallelism isn't just about more threads on one CPU. In a totally generalized parallel scenario, there may be any number of threads - a few tens of thousands would not be unusual - running on systems that may be SMP, multi-core, multi-threaded, vectored, clockless, or any combination of the above, where those systems may be on a tightly-coupled or loosely-coupled cluster, and where the cluster may be homogenous or heterogeneous, SSI or multi-imaged, where memory may be local, NUMA or distributed, and where these systems/clusters may be gridded over wide-area networks that may or may not be reliable or operational at any given time, and where threads, processes and entire operating systems may migrate from system to system without user intervention or awareness on the part of the application.

The number of true parallel experts in the world probably number less than a dozen. No, I'm not one of them. I'm good, I understand the problem-space better than the average coder, but I've talked to some of the experts out there and they're as far beyond any traditional programmer as a traditional programmer is beyond the chipmunk. A network engineer might consider themselves OK if they can set up OSPF optimally across a traditional star network of star networks. Any traditional routing protocol over a mesh without getting flaps and maintaining a reasonable level of fault tolerence would be considered tough. A butterfly network, a torroidal network or a hypercube would leave said network engineer a gibbering wreck. Modern supercomputers do not take up buildings. Modern supercomputers take up a few rooms. The interconnects take up entire buildings. And the air conditioning on top systems can be measured in football stadia.

OpenMOSIX is largely dead, because it was impossible to reconcile those who wanted load-balancing with those who wanted HPC. It's not that they can't be reconciled in theory, it's that the mindsets are too different to cram into one brain.

If one solution could solve parallelism, the Transputer would be the only processor in use today and Intel would

Parallel computing pioneer likes Parallelism

[Alzheimers]

A guy who's made it his life's work to study Parallel Computing has come forth to say, he thinks Parallelism is the next big thing?

Shock! And Awe!

"Next hot thing" my hiney

[Rosco P. Coltrane]

For having been in the computer industry for too long, I reckon the "next hot thing" usually means the "latest fad" that many of the entrepreneurs involved in hope will turn into the "next get-rich-quick scheme".

Because really, anybody believes Web-Two-Oh was anything but the regular web's natural evolution with a fancy name tacked on?

Re:"Next hot thing" my hiney

[SanityInAnarchy]

Oh, now that's not entirely fair.

Web 2.0 was a single name for an amorphous collection of technologies and philosophies. It was even worse than AJAX.

Parallelism is a pretty simple, well-defined problem, and an old one. That doesn't mean it can't turn into a buzzword, but I'm not convinced Web 2.0 can be anything but a buzzword.

More so now, but depends ...

[Midnight Thunder]

Now that we are seeing more and more in the way of multi-core CPUs and multi-CPU computers I can definitely see parallelism become more important, for task that can be handled this way. You have to remember that in certain cases trying to parallise a task can end up being less efficient, so what you parallelise will depend on the task in hand. Things like games, media application and scientific applications are usually likely candidates since they are either doing lots of different things at once or have tasks that can be split up into smaller units that don't depend on the outcome of the other. Server applications can to a certain extent, depending whether they are trying to the same resources or not (ftp server, accessing this disk, vs a time server which does not file I/O).

One thing that should also be noted, is that in certain cases you will need to accept increased memory usage, since you want to avoid tasks locking on resources that they don't really need to synchronise until the end of the work unit. In this case it may be cheaper to duplicate resources, do the work and then resynchronise at the end. Like everything it depends on the size and duration of the work unit.

Even if your application is not doing enough to warrant running its tasks in parallel, the operating system could benefit, so that applications don't suffer on sharing resources that don't need to be shared.

Re:More so now, but depends ...

[Shados]

Well, actually...I think more things can be paralleled than one would think at first glance... The very existance of the foreach loop kindda shows this... Looking at most code I have to work with, 90% of such loops simply do "I'm iterating through the entire list/collection/array and processing only the current element", some simple aggregates (the kind where you can split the task and aggregate the result at the end), etc. Virtually all applications have those, and call them often.

Being able to simply say "the order in which these tasks are made doesn't matter" lets you run a lot of tasks in parallel right there.

Then you have a lot of tasks which do not depend on each other at ALL, but will still wait on each other... Let say a typical MVC web app ala Rail or Strut... you have a lot of ressource handling going on in the view that doesn't even rely on the Model/database whatsoever...and then you have the operation of rendering the model data in the view... so querying the data, and handling the ressources, could be done at the same time, and once both are done, do the actual render. ASP.NET has a decent mechanism for that, but no one uses it.

These are all things that are being worked on (and some already exist but most people ignore them), and can dramatically boost performance of virtually ANY application...but there needs to be a certain awareness... And some of these features are just freagin late (Microsoft is working on PLINQ...ways to make it easy to process data structure in parallel... for example: int[] blah = listOfInts.ForAll( item => item * 2 );... and it will do it in parallel. Seems so obvious, but why hasn't this been around for years in .NET?)

Didn't we have this debate last week?

[Nursie]

Oh yes, here it is.

And the conclusion?

It's been around for years numbnuts, in commercial and server applications, middle tiers, databases and a million and one other things worked on by serious software developers (i.e. not web programming dweebs).

Parallelism has been around for ages and has been used commercially for a couple of decades. Get over it.

Re:Didn't we have this debate last week?

[spottedkangaroo]

But, at the end of the day (where the rubber meets the road) this will utilize the core competencies of solutions that specialize in the new ||ism forefront.

Please no

[Wiseman1024]

Not parallelism... Why do MBA idiots have to fill everything with their crap? Now they'll start creating buzzwords, reading stupid web logs (called "blogs"), filling magazines with acronyms...

Coming soon: professional object-oriented XML-based AJAX-powered scalable five-nines high-availability multi-tier enterprise turnkey business solutions that convert visitors into customers, optimize cash flows, discover business logic and opportunities, and create synergy between their stupidity and their bank accounts - parallelized.

Most companies need parallel developers

[1sockchuck]

This sure looks like a growth area for qualified developers. An audience poll at the Gartner Data Center conference in Las Vegas in November found that just 17 percent of attendees felt their developers are prepared for coding multi-core applications, compared to 64 percent who say they will need to train or hire developers for parallel processing. "We believe a minority of developers have the skills to write parallel code," said Gartner analyst Carl Claunch. I take the Gartner stuff with a grain of salt, but the audience poll was interesting.

McColl's blog is pretty interesting. He only recently started writing regularly again. High Scalability is another worthwhile resource in this area.

Re:Most companies need parallel developers

[Hasmanean]

In common usage, threading usually implies different "streams" of execution doing independent things, at the same time. If the same function is executing in n different threads then you might call it "parallel" programming. A lot of multithreaded programming involves taking pieces of program functionality and breaking them out into separate threads, each executing independently.

Calling the latter architecture parallel computing is misnomer, it is really "simultaneous" computing i.e. things can happen at the same time, but there is a big difference between the same thread executing n times in parallel, and different threads doing different things simultaneously.

For example, a "Trivial" program which reads in a list of numbers from a file, computes something (say the sum of the magnitudes squared), and prints the result out to the screen might be implemented as follows:

while not eof
        read n numbers
        compute something from them
        print result


a multithreaded version might look something like this

Thread 1 (Disk IO): read n number from disk, write them to a queue/shared memory, repeat
Thread 2 (Outputting): wait for outputs to become available, print them, repeat
Thread 3 (Compute): wait for inputs to arrive in a queue, process them, write output to another queue, repeat



A parallel version would just have more than 1 Compute thread, and they would subdivide the work between them (for example 2 threads dividing the input array into stripes, one handling even indices, the other odd...or a bunch of threads computing different slices of the array). Note that the threads would still have to combine their results at the end of the computation, and that is not always simple to do in parallel.

Some problems or algorithms simply cannot execute in parallel. Also issues of memory access patterns, caching, branch divergence (if they threads take different code paths will this affect performance) come into play. It requires a whole new set of issues to worry about, but they are not too difficult. As the professor who teaches the course http://courses.ece.uiuc.edu/ece498/al1/ says, learning parallel programming is not hard, he could teach it to you in 2 hours. But doing parallel programming well and efficiently is difficult. You can write a trivial parallel program which uses just 1 processor and has just 1 thread, which is identical to the sequential version, and it will work logically, although the performance will be severely limited. You can then extend it to use n threads, and it will experience a speedup. But to take full advantage of the hardware on your board, you will need to know a few tricks, and understand the hardware and your program's behavior intimately.

Another issue is that programmers were spoiled by processor upgrades coming along and speeding up their programs "for free" by virtue of their higher clock speed. Now with clock speeds reaching physical limits, the only evolution in new processors will be in the number of cores they have. So the only way to coax more out of a program will be to make it more parallel, and that might be trivial or it might be difficult. We're going to have to think laterally to get more performance out of software.

"the bastards say 'welcome'"

[david.emery]

So all-of-a-sudden people have discovered parallelism? Gee, one of the really interesting things about Ada in the late 80s was its use on multiprocessor systems such as those produced by Sequent and Encore. There was a lot of work on the language itself (that went into Ada95) and on compiler technologies to support 'safe parallelism'. "Safe" here means 'correct implementation' against the language standard, considering things like cache consistency as parts of programs get implemented in different CPUs, each with its own cache.

Here are a couple of lessons learned from that Ada experience:
1. Sometimes you want synchronization, and sometimes you want avoidance. Ada83 Tasking/Rendezvous provided synchronization, but was hard to use for avoidance. Ada95 added protected objects to handle avoidance.
2. In Ada83, aliasing by default was forbidden, which made it a lot easier for the compiler to reason about things like cache consistency. Ada95 added more pragmas, etc, to provide additional control on aliasing and atomic operations.
3. A lot of the early experience with concurrency and parallelism in Ada learned (usually the hard way) that there's a 'sweet spot' in the number of concurrent actions. Too many, and the machine bogs down in scheduling and synchronization. Too few, and you don't keep all of the processors busy. One of the interesting things that Karl Nyberg worked on in his Sun T1000 contest review was the tuning necessary to keep as many cores as possible running. (http://www.grebyn.com/t1000/ ) (Disclosure: I don't work for Grebyn, but I do have an account on grebyn.com as a legacy of the old days when they were in the ISP business in the '80s, and Karl is an old friend of very long standing....)

All this reminds me of a story from Tracy Kidder's Soul of a New Machine http://en.wikipedia.org/wiki/The_Soul_of_a_New_Machine. There was an article in the trade press pointing to an IBM minicomputer, with the title "IBM legitimizes minicomputers". Data General proposed (or ran, I forget which) an ad that built on that article, saying "The bastards say, 'welcome' ".

dave

"...with the cooling of Web 2.0,..."

[sm62704]

Sorry guys, web 2.0 was never cool and never will be.

What am I missing?

[James Lewis]

Now that multi-core computers have been out I keep hearing buzz around the idea of parallel computing, as if it is something new. We've had threads, processes, multi-CPU machines, grid computing, etc etc for a long time now. Parallelism has been in use on single processor machines for a long time. Multi-core machines might make it more attractive to thread certain applications that were traditionally single-threaded, but that's the only major development I can see. The biggest problem in parallel computing is the complexity it adds, so hopefully developments will be made in that area, but it's an area that's been researched for a long time now.

Dont believe? Intel & MS have made a $20M bet

[kiyoshilionz]

You think that nobody has a real interest in parallel computing? Intel's put their money on it already - they've allotted $20 million between UC Berkeley and University of Illinois to research parallel computing, both in hardware and software.

I am a EECS student at Cal right now and I have heard talks by the UC Berkeley PARLab professors (Krste Asanovic and David Patterson, the man who brought us RAID and RISC), and all of them say that the computing industry is going to radically change unless we figure out how to efficiently use parallelism. This is the first time in history that software performance is beginning to lag behind how fast we can make our hardware. The failure of the frequency scaling to continue to improve system performance has been shown in the failure of the NetBurst microarchitecture - remember the Prescott? And the failure of the Tejas and Jayhawk? Building faster chips is over, it's a mechanical engineering issue - we can make chips put out more heat per area than the surface of the sun. Quoting professor Hennessey from Stanford:

"...when we start talking about parallelism and ease of use of truly parallel computers, we're talking about a problem that's as hard as any that computer science has faced. ... I would be panicked if I were in industry. ... you've got a very difficult situation."

To whoever is saying that parallelism is just a fad, you're really missing a lot of what's going on in the computing world. We've already switched to dual- and quad-core CPU's, and it doesn't look like it's going to stop any time soon.

Race conditions

[microbox]

Being able to simply say "the order in which these tasks are made doesn't matter" lets you run a lot of tasks in parallel right there.

Mucking around with language design and implementation highlights some of the deep problems that the "parallize-everything" crowd often don't know about.

In your example, the loop can only be efficently parallized if it doesn't have any side-effects. If any variables are written to out of scope of the loop, then they are exposed to race conditions in both the read and write direction. It's still possible to parallize the code if the machine code doesn't take advantage of the memory model of your architecture, and no other thread is modifying those variables, and it doesn't matter in which order the variables are modified, and modifications are atomic. As an implementation detail, memory reads and write will be signficiantly slower, and sychronization of atomic operations is also not without cost. So much in fact, that you'd probably be better off just running the loop on a single thread. Your code will be more predictable regardless.

At the moment, it seems that effective parallization requires some planning and effort on behalf of the programmer. The bugs can be extraordinarily subtle, impossible to debug and difficult to reason about. Expect single-threaded programming models to be around for a long time.

Then we are all doomed

[DrJokepu]

You see, the majority of the programmers out there don't know much about parallelism. They don't understand what synchronization, mutexes or semaphores are. And the thing is that these concepts are quite complex. They require a much steeper learning curve than hacking a "Web 2.0" application together with PHP, Javascript and maybe MySQL. So if now everybody will start writing multithreaded or otherwise parallel programs, that's going to result in an endless chain of race conditions, mysterious crashes and so on. Rembember, race conditions already killed people.

Multithreading Is to Blame

[MOBE2001]

Needless to say I was a bit early on that prediction.

The reason is that all academic researchers jumped on the multithreading bandwagon as the basis for parallel computing. Unfortunately for them, they could never get it to work. They've been at it for over twenty years and they still can't make it work. Twenty years is an eternity in this business. You would think that after all this time, it would have occurred to at least one of those smart scientists that maybe, just maybe, multithreading is not the answer to parallel computing. Nope: they're still trying to fit that square peg into the round hole.

Both AMD and Intel have invested heavily into the multithreading model. Big mistake. Multiple billion dollar mistake. To find out why threads are not part of the future of parallel computing read Nightmare on Core Street. It's time for the computer industry to wake up and realize that the analytical engine is long gone. This is the 21st century. It's time to move on and change to a new model of computing.

Re:Multithreading Is to Blame

[Anonymous Coward]

Unfortunately for them, they could never get it to work.

Hyperbole much? Parallel systems such as MPI have been the staple of high performance computing since the mid 90's, and there are plenty of developers (including myself) who can write multi-threaded code without breaking into a sweat, and get it right.

At what point did parallel and concurrent programming "fail"? I really must have missed that memo.

Re:Multithreading Is to Blame

[grumbel]

Just because the majority of developers can't doesn't mean it has "failed" When you want to get parallelism into the mainstream then that is pretty much the exact definition of "fail".

Always Was, Always WIll Be

[StCredZero]

Parallelism was a hot area when I was in grad school 10 years back. We used to say about Gallium Arsenide -- "Gallium Arsenide is the semiconductor of the future -- always was, always will be." Maybe the same thing is true with parallelism -- paradigm of the future -- it's always useful as a way of getting more work done per unit time, but it has high costs associated with it, so people will only use it if it feels like they absolutely have to.

Parallelism has had broad applicability in graphics. It's definitely useful. But I doubt it's going to obsolete what we can do sequentially.