Slashdot Mirror
More Interest In Parallel Programming Outside the US?
simoniker writes "In a new weblog post on Dobbs Code Talk, Intel's James Reinders discusses the growth of concurrency in programming, suggesting that '...programming for multi-core is catching the imagination of programmers more in Japan, China, Russia, and India than in Europe and the United States.' He also comments: 'We see a significantly HIGHER interest in jumping on a parallelism from programmers with under 15 years experience, verses programmers with more than 15 years.' Any anecdotal evidence for or against from this community?"
Old programmers don't want to learn new things -- trust the tried and true.
Young bucks want to be on the cutting edge to get the jobs that the old people already have.
----
Oh, and the people see the benefit in the other countries more than those in the U.S.? Probably not, we're just lazy American's though.
Which is more efficacious? To take up as much simultaneous processor time as possible in order to finish faster, or to leave extra cores open for other processes to run simultaneously.
Given that the management of threads isn't exactly the easiest thing in the world (not the hardest either, mind you), perhaps it would be more beneficial to let the OS determine which threads to run rather than trying to optimize a single app.
For example, many webservers use a semaphore rather than multiple threads to handle request dispatch. The result is that there is less overhead creating and cleaning up threads.
Q1) Why did the multithreaded chicken cross the road?
A1) to To other side. get the
Q2) Why did the multithreaded chicken cross the road?
A4) other to side. To the get
It is funnier in the original Russian.
echo -e 'global _start\n _start:\n mov eax, 2\n int 80h\n jmp _start' > a.asm; nasm a.asm -f elf; ld a.o -o a;
One reason could be that software engineers with more experience simply already know about these things, and have faced off against the many problems with concurrency. Threads can be hell to deal with for instance. So because of things they don't show any interest.
That being said I think that if you want to actually make use of many cores you really do have to switch to a language that can give you usage of many threads for free. Writing it manually usually ends up with complications. I find Erlang to be pretty nifty when it comes to these things for instance.
Parallel programming is going to go mainstream, not because people find it interesting, but because that's the way hardware is forcing us to go. First mainframes, then workstations, and now desktops and laptops have moved away from single CPU cores. In every case, it has been a necessary evil due to the inability to pack enough power into a single monolithic processor. Does anyone actually think Intel, if they could, wouldn't happily go back to building single-core CPU's with the same total power as the multi-core CPU's they're making now?
Right now, parallel development techniques, education, and tools are all lagging behind the hardware reality. Relatively few applications currently make even remotely efficient use of multiple cores, and that includes embarrassingly parallel code that would require only minor code changes to run in parallel and no changes to the base algorithm at all.
However, if you look around, the tools are materializing. Parallel programming skills will be in hot demand shortly. It's just a matter of time before the multi-core install base is large enough that software companies can't ignore it.
One day soon, the computer industry will realize that, 150 years after Charles Babbage came up with his idea of a general purpose sequential computer, it is time to move on and change to a new computing model. The industry will be dragged kicking and screaming into the 21st century. For over 20 years, researchers in parallel and high-performance computing have tried to come up with an easy way to use threads for parallel programming. They have failed and they have failed miserably. Amazingly, they are still continuing to pursue the multithreading approach. None other than Dan Reed, Director of scalable and multicore computing at Microsoft Research, believes that multithreading over time will become part of the skill set of every professional software developer (source: cio.com). What is wrong with this picture? Threads are a major disaster: They are coarse-grained, they are a pain in the ass to write and hard to debug and maintain. Reinders knows this. He's pushing threads, not because he wants your code to run faster but because Intel's multicore CPUs are useless for non-threaded apps.
Reinders is not an evangelist for nothing. He's more concerned about future-proofing Intel's processors than anything else. You listen to him at your own risk because the industry's current multicore strategy will fail and it will fail miserably.
Threads were never meant to be the basis of a parallel computing model but as a mechanism to execute sequential code concurrently. To find out why multithreading is not part of the future of parallel programming, read Nightmare on Core Street. There is better way to achieve fine-grain, deterministic parallelism without threads.
Whether it's a good idea or not, you will have a VERY hard time convincing an old dog programmer that he should jump on the train. Think back to the time when relational models entered the database world.
Ok, few here are old enough to be able to. But take object oriented programming. I'm fairly sure a few will remember the pre-OO days. "What is that good for?" was the most neutral question you might have heard. "Bunch o' bloated bollocks for kids that can't code cleanly" is maybe more like the average comment from an old programmer.
Well, just like in those two cases, what I see is times when it's useful and times when you're better off with the old ways. If you NEED all the processing power a machine can offer you, in time critical applications that need as much horsepower as you can muster (i.e. games), you will pretty much have to parallelize as much as you can. Although until we have rock solid compilers that can make use of multiple cores without causing unwanted side effects (and we're far from that), you might want to stay with serial computing for tasks that needn't be finished DAMN RIGHT NOW, but have to be DAMN RIGHT, no matter what.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
I'm writing a scientific library that is supposed to scale for many many cores. Using a mutex lock is not an option. Unfortunately right now I am spending all my time trying to figure out how to get compare and swap working on all the different platforms. I am saddened to see the lack of support since this is such a fundamental operation. Also, the whole 32 vs 64-bit thing adds more pain because of pointer size.
----
Go canucks, habs, and sens!
One of the reasons more seasoned programmers are not particularly interested is that in most cases someone else has already doen the hard work.
Want to serve multiple user on multiple cpus with your web pages? Then write a single threaded program and let Apache handle the parallelism. Same goes for JavaEE, database triggers, etc. etc. going all the way back to good old CICS and COBOL.
It is very rare that you actually need to do parallel programing yourself. Either you are doing some TP monitor like program which schedules tasks other people have written in which case you should use use C and POSIX threads (anything else will land you in deep sewage) or you are doing serious science stuff in which case there are several "standard" fortran libraries to do multithreaded matrix math -- but if the workload is "serious" you should be looking at clustering anyway.
Old COBOL programmers never die. They just code in C.
Been there, done that. Good from far, but far from good.
As an engineer straight out of college, I was very interested in parallel programming. In fact, we were doing a project on parallel databases. My take is that it sounds very appealing, but once you dig deeper, you realise that there are too many gotchas.
Considering the typical work-time problem, let's say a piece of work takes n seconds to complete by 1 processor. If there are m processors, the work gets completed in n/m seconds. Unless the parallel system can somehow do better than this, it is usually not worth the effort. If the work is perfectly divisible between m processors, then why have a parallel system? Why not a distributed system (like beowulf, etc.)?
If it is not perfectly distributable, the code can get really complicated. Although it might be very interesting to solve mathematically, it is not worth the effort, if the benefit is only 'm'. This is because, as per Moore's law, the speed of the processor will catch up in k*log m years. So, in k*log m years, you will be left with an unmaintainable piece of code which will be running as fast as a serial program running on more modern hardware.
If the parallel system increases the speed factor better than 'm', such as by k^m, the solution is viable. However, there aren't many problems that have such a dramtic improvement.
What may be interesting are algorithms that take serial algorithms and parallelise them. However, most thread scheduling implementations already do this (old shell scripts can also utilise parallel processing using these techniques). Today's emphasis is on writing simple code that will require less maintenance, than on linear performance increase.
The only other economic benefit I can think of is economy of volumes. If you can get 4GHz of processing power for 1.5 times the cost of a 2GHz processor, it might be worth thinking about it.
It's only been twelve years since I entered the workforce in the US, but I have been studying parallel programming for almost 15 years (three years in university).
The future isn't "multi-threaded" unless you count SPMD, because architecturally the notion of coherent shared memory is always going to be an expensive crutch. Real high-performance stuff will continue to work with distributed, local memory pools and fast inter-node communication... whether the nodes are all on chip, on bus, in the box, in the datacenter, etc.
As they have been since the 80s at least, many CS researchers will be trying to find the holy grail of programming models and tools to automatically parallelize larger classes of algorithms for naive programmers. And in the meantime, just as we have since the 80s at least, programmers will often go to bare metal and hand-optimize important libraries and applications that are too important to leave to the vagaries of the immature tools.
If there are fewer programmers in the US or Europe who worry about parallelism, it is only because the economy is such that they can still satisfy their customers without it. There are many parallel programmers here too, and maybe there isn't a pressing need for even more because their work is being reused. I'm not sure what sort of statistical analysis you should use to determine prevalence of parallel systems usage, but I am pretty sure it is not by counting programmer interest. How many deployed systems are there? How many CPU-hours of parallel work are being done? What fraction of IT budgets are supporting production use of parallel systems?...
To be frank, I think the vast majority of computer cycles are spent executing code written by a small minority of the programmers in the world. These are the ones that matter, as far as optimizing for parallel environments. Sorry if that sounds elitist, but it's a bit like trying to analyze the distribution of race car drivers by surveying the interests of all licensed motorists.
Those of us doing server side development for any medium to large company will have already been doing multi-threaded and/or multi-process applications for ages now:
... until "strange" problems start randomly to pop-up</rant>).
... or is this just part of a PR campaign to sell an architecture which, in desktops, is fast growing beyond it's usefulness (a bit like razors with 4 and 5 blades)
- When Intel was still a barely known brand, other companies were already selling heavy-iron machines with multiple CPUs for use in heavy server-side environments (didn't ran Windows though). Multi-cores are just a variant of the multiple-CPU concept.
The spread of Web applications just made highly multi-threaded server-side apps even more widespread - they tend naturally to have multiple concurrent users (<rant>though some web-app developers seem to be sadly unaware of the innate multi-threadness of their applications
As for thick client apps, for anything but the simplest programs one always needs at least 2 threads, one for GUI painting and another one for processing (either that or some fancy juggling a la Windows 3.1)
So, does this article means that Japan, China, India and Russia had no multi-CPU machines until now
That's an oversimplified view.
It's more like when you've got enough experience, you already know what can go wrong, and why doing something might be... well, not necessarily a bad idea, but cost more and be less efficient anyway. You start having some clue, for example, what happens when your 1000 thread program has to access a shared piece of data.
E.g., let's say we write a massively multi-threaded shooter game. Each player is a separate thread, and we'll throw in a few extra threads for other stuff. What happens when I shoot at you? If your thread was updating your coordinates just as mine was calculating if I hit, very funny effects can happen. If the rendering is a separate thread too, and reads such mangled coordinates, you'll have enemies blinking into strange places on your screen. If the physics or collision detection does the same, that-a-way lies falling under the map and even more annoying stuff.
Debugging it gets even funnier, since some race conditions can happen once a year on one computer configuration, but every 5 minutes on some hapless user's. Most will not even happen while you're single-stepping through the program.
Now I'm not saying either of that is unsolvable. Just that when you have a given time and budget for that project, it's quite easy to see how the cool, hip and bleeding-edge solution would overrun that.
By comparison, well, I can't speak for all young 'uns, but I can say that _I_ was a lot more irresponsible as the stereotypical precocious kid. I did dumb things just because I didn't know any better, and/or wasted time reinventing the wheel with another framework just because it was fun. All this on the background of thinking that I'm such a genius that obviously _my_ version of the wheel will be better than that built by a company with 20 years of experience in the field. And that if I don't feel like using some best practice, 'cause it's boring, then I know better than those boring old farts, and they're probably doing it just to be paid for more hours.
Of course, that didn't stop my programs from crashing or doing other funny things, but no need to get hung up on that, right?
And I see the same in a lot of hotshots nowadays. They do dumb stuff just because it's more fun to play with new stuff, than just do their job. I can't be too mad at them, because I used to do the same. But make no mistake, it _is_ a form of computer gaming, not being t3h 1337 uber-h4xx0r.
At any rate, rest assured that some of us old guys still know how to spawn a thread, because that's what it boils down to. I even get into disputes with some of my colleagues because they think I use threads too often. And there are plenty of frameworks which do that for you, so you don't have to get your own hands dirty. E.g., everyone who's ever wrote a web application, guess what? It's a parallel application, only it's the server which spawns your threads.
A polar bear is a cartesian bear after a coordinate transform.
- There are often other ways to do it, e.g. multiple processes communicating over sockets, or multiple processes that share memory.
- Threads are hard to get right. Really, really hard.
When your library of mutexes, semaphores, etc. doesn't have exactly the construct you need, and you go to write your own on top of them, it's really, really hard not to introduce serious bugs that only show up very rarely. As one random example, consider the Linux kernel team's attempts to write a mutex, as descried in Ulrich Drepper's paper "Futexes are Tricky."If these people take years to get it right, what makes you think *you* can get it right in a reasonable time?
The irony is that threads are only practical (from a correctness/debugging point of view) when there isn't much interaction between the threads.
By the way, I got that link from Drepper's excellent "What Every Programmer Should Know about Memory." It also talks about how threading can slow things down.
You just haven't noticed.
Multi process apps have been common in the business and server app space for almost two decades.
Multi thread apps have been common in the business and server world for a few years now too.
To all having the will it/won't it go mainstream argument: You missed the boat. It's mainstream.
Parallel programming is simply harder than typical sequential programming. Not only does the design take more time and effort, but the debugging is VERY much harder. tools for parallel programming are poor but debugging tools are basically pathetic. Worse, today's project and development methodologies don't focus on getting something up and hacking, not on careful upfront design that is needed to really parallelize things. We get most of our parallelism from the web server being multi-threaded and the database handling concurrency.
As many have said, large scale parallel systems are not new. Just because we need a solution to the problem doesn't mean that it will appear any time soon. Some problems are very difficult and involve not only new technologies and programing models but major re-educational efforts. There are many topics in physics and mathematics that only a small number of people have the intellectual skill and predilection to handle. Of all the college graduates, what percent complete calculus, let alone take advanced calculus? Pretty small number.
My prediction is that the broad base of programmers will have the tools and be able to do some basic parallelism. A small number of programmers will do the heavy duty parallel programming and this will be focus on very high value problems.
BTW, this Intel guy, while addressing a real issue, seemed to be doing marketing for his toolkit/approach. Sounds like a guy trying to secure his budget and grow it for next year.
There are two problems: thread sychronization, and race conditions.
Race conditions
A modern CPU architecture uses multiple levels of cache, which aggrivate the race condition scenario. For a programmer to code multi-threaded code, and "not-worry", then the architecture must always read and write every value to memory. This worst case scenario is only needed in a tiny fraction of cases. So the compiler can do much better by working with the memory model of the architecture, instead of assuming that no memory model is in place.
Any significant improvement in speed will require solving the race condition problem. I believe research into transactional memory may be the way to go - but it is still half the speed of normal memory access.
The synchronization problem
This requires that the programmer reason about multiple threads (at least two). It doesn't really matter what buildling blocks you're using, you simply must be aware that two pieces of code are effectively executing at the same time - or at least their cpu slices are interwoven. This type of reasoning significantly raises the bar for writing bug-free code, because a whole new class of subtle problems arise. The mind must wrap itself around something that is simply more complex, and we have trouble enough with single threaded programs.
I like the flex/javascript solution. Just one thread - with asynchronous-like behaviour through events. It means there are no race conditions, and you can get enough async behaviour to get the job done. I hope, in the future, that both these technologies allow you to create a seperate "process" (thread but with different memory context) that executes asychronously and returns the results on the main event queue. It's a bit clunky, but at the same time it's much more idiot proof. And I'm speaking as someone who's done a reasonable amount of parallel programming.
Like all pain, suffering is a signal that something isn't right
> Name a single real world problem that doesn't parallelize.
Childbirth. Regardless of how many women you assign to the task, it still takes nine months.
(feel free to reply with snark, but that's a quote from Fred Brooks, so if your snarky reply makes it look like you haven't heard the quote before you will seem foolish)
Anything harder than a problem needs to be is too hard. The question is whether concurrency can be made easier while preserving the benefits it provides.
For your examples you threw out a bunch of problems that are currently being parallelized just fine by today's software, which would indicate we're not having problems with parallel programming. Name a search engine, database query engine, weather simulation software, etc, that isn't already multithreaded. Where's the issue?
Problems that can and should be parallelized in software already are for the most part. There is no issue here.
Business processes are often serial (step B depends on output of step A). That's what a lot of corporate programmers work on. And even these have steps that are done in parallel or can run multiple instances of a process in parallel. Anyone working on a web application or j2ee infrastructure is probably running lots of their small, serialized problems, in parallel. Again, there is no issue here.
Anybody else remember the great clock cycle stall of the 1980's? During that period, Moore's Law operated in a manner closer to its original statement - the big news was the drop in cost per transistor, not raw CPU speed. The general wisdom at the time was that parallelism was going to be the way to get performance.
And then, we entered the die shrink and clock speed up era, clock speeds doubled every 14 months or so for ten years, and we went smoothly from 60 MHz to 2 GHz. Much of the enthusiasm for parallel programming died away - why sweat blood making a parallel computer when you can wait a few years and get most of the same performance?
Clock speeds hit a wall again about five years ago. If the rate of increase stays small for another five years, the current cycle drought will have outlasted the 1980's slowdown. I have a great deal of sympathy for parallel enthusiasm (I hacked on a cluster of 256 Z80's in the early 80's), but I think it won't really take off until we really have no other choice, because parallelism is hard.
To a Lisp hacker, XML is S-expressions in drag.