Slashdot Mirror
New Languages Vs. Old For Parallel Programming
joabj writes "Getting the most from multicore processors is becoming an increasingly difficult task for programmers. DARPA has commissioned a number of new programming languages, notably X10 and Chapel, written especially for developing programs that can be run across multiple processors, though others see them as too much of a departure to ever gain widespread usage among coders."
I'm waiting for parallel libs for R, even if i'm told that scripted languages won't have much of a future in parallel processing. All I can do is hope. Sigh.
Parallel is not going to go anywhere but is only really valid for certain types if applications. Larger items like operating systems or most system tasks need it. Whether it is worthwhile in lowly application land is a case by case decision; but will mostly depend on the skill of programmers involved and the budget for the particular application in question.
"Maybe this world is another planet's hell"
Aldous Huxley
All you need is Fortran.
A lot of problems are I/O driven -- I would like to see more database client libraries allow a full async approach that lets us not block the threads we are trying to do concurrent work on.
Not trying to troll or anything, but I'd always hear of how parallel programming is very complicated for programmers, but then I learnt to use pthread in C to parallelise everything in my C program from parallel concurrent processing of the same things to threading any aspect of the program, and I was surprised by how simple and straightforward it was using pthread, even creating a number of threads depending on the number of detected cores was simple.
OK, maybe what I did was simple enough, but I just don't see what's so inherently hard about parellel programming. Surely I am missing something.
You just got troll'd!
Agree. Adding a new language isn't going to help much. Maybe extending an old one is the best way to go. However, GCC has added (if you choose) a small subset of parallel programming constructs automatically at the compiler level for at least a year now.
Huh? What spoken language is Perl supposed to be?
Erlang is an older established language designed for parallel processing.
Erlang was first developed in 1986, making it about a decade older than Java or Ruby. It is younger than Perl or C, and just a tad older than Python. It is a mature language with a large support community, especially in industrial applications. It is time tested and proven.
It is also Open source and offers many options for commercial support.
Before anyone at DARPA thinks that they can design a better language for concurrent parallel programming then I think they should be forced to spend 1 year learning Ada, and a second year working in Ada. If they survive they will most likely be cured of the thought that the Defense department can design good programming languages
The example in the article is atrocious.
Why would you want the withdrawal and balance check to run concurrently?
Bullshit.
Tell that to apache, and oracle, and basically anything that runs in a server room.
Threading i don't count as parallel processing for the desktop. I don't even hear of any games or applications built for parallel.
Uhhhhhhhhhhh? Yes, well done with that...
Look up the term "race condition" here -> http://en.wikipedia.org/wiki/Race_condition , & you'll get an idea of what the "problems inherent" are - I feel the same as you do though, as long as I keep 1 thread doing 1 task, & another thread of execution doing another (albeit, on 2 diff./discrete sets of data, not working on the SAME set of data) - this is known as "coarse multithreading" (keeping multiple threads of execution off the same dataset) vs. "fine multithreading" (see here for more on that -> http://www.cs.bu.edu/~best/courses/cs551/lectures/lecture-02.html ) where the multiple threads of execution work on the same data involved.
(Bit of an "oversimplification" on my part possibly, but the broad strokes are there - the 'finer points' with examples are on those pages from the URL's above I posted)
APK
I see some potential in combining innovations meant for the netbooks with multiple processors. Low power & lightweight software may mix well with multiple CPUSs.
Check out Clojure. The only programming language around that really addresses the issue of programming in a multi-core environment. It's also quite a sweet language besides that.
Not even all scientists. I use a lot of programs that are computationally intensive - stuff that you start running and walk away for a week - and I'd guess about half of the new programs are not written to take advantage of parallel processing. For us, this is seriously frustrating because... well, I don't like having to wait a week for the software to return a collection of answers, even using powerful machines.
A big problem for scientific computing - and maybe it's a problem elsewhere, too - is that too many programs are collections of legacy code squashed just-so to make it compilable on a new machine. While I typically applaud the path of least resistance when it comes to work, it makes the software inefficient as heck.
Rehash time...
Parallelism typically falls into two buckets: Data parallel and functional parallel. The first challenge for the general programming public is identifying what is what. The second challenge is synchronizing parallelism in as bug free way as possible while retaining the performance advantage of the parallelism.
Doing fine-grained parallelism - what the functional crowd is promising, is something that will take a *long* time to become mainstream (Other interesting examples are things like LLVM and K, but they tend to focus more on data parallel). Functional is too abstract for most people to deal with (yes, I understand it is easy for *you*).
Short term (i.e. ~5 years), the real benefit will be in threaded/parallel frameworks (my app logic can be serial, tasks that my app needs happen in the background).
Changing industry tool-chains to something entirely new takes many many years. What most likely will happen is transactional memory will make it into some level of hardware, enabling faster parallel constructs, a cool new language will pop up formalizing all of these features. Someone will tear that cool new language apart by removing the rigor and giving it C/C++ style syntax, then the industry will start using it
This is a subject near and dear to my heart. I got to participate in one of the early X10 alpha tests (my research group was asked to try it out and give feedback to Vivek Sarker's IBM team). Since then, I've worked with lots of other specialized programming HPC programming languages.
One extremely important aspect of supercomputing, a point that many people fail to grasp, is that application code tends to live a long, long, long time. Far longer than the machines themselves. Rewriting code is simply too expensive and economically inefficient. At Los Alamos National Lab, much of the source code they run are nuclear simulations written Fortran 77 or Fortran 90. Someone might have updated it to use MPI, but otherwise it's the same program. So it's important to bear in mind that those older languages, while not nearly as well suited for parallelism (either for programmer ease-of-use/effeciency, or to allow the compiler to do deep analysis/optimization/scheduling), are going to be around for a long time yet.
To make laws that man cannot, and will not obey, serves to bring all law into contempt.
--E.C. Stanton
Huh? What spoken language is Perl supposed to be?
Read the Wikipedia. Not PERL itself, but, from the article:
Wall and his wife were studying linguistics with the intention afterwards of finding an unwritten language, perhaps in Africa, and creating a writing system for it. They would then use this new writing system to translate various texts into the language, among them the Bible.
Great, just what we need. More clueless crusaders giving otherwise peaceful peoples the tools to become planet cancer. But there's more:
Wall's Christian faith has influenced...Perl, such as the name itself, a biblical reference to the "Pearl of great price" (Matthew 13:46). Similar references are the function name bless, and...Perl 6 design documents with categories such as apocalypse and exegesis. Wall has also alluded to his faith when he has spoken at conferences, including a rather straightforward statement of his beliefs...
I would never work for anybody who used PERL.
If threading isn't parallelism, then what is? At what level of separation between separate streams of execution does an application become "parallel"?
We all know what to do, but we don't know how to get re-elected once we have done it
I think it's just too darned difficult for most of us to write mainstream apps whilst figuring out the how to take advantage of multiple cores, or figuring out how to write a web browser in erlang etc. I'll be curious to see what comes out of the stanford pervasive parallelism lab, I think they have the right approach to tackling the 'coding for many cores' problem. http://www.theregister.co.uk/2008/04/30/stanford_funding_ppl/
But my first thought upon reading "Chapel" was...
I'm multi-threaded bitch!
Creating threads is extremely easy to do in Java, .NET and C++. We don't need "inherit" parallelism in every loop where the performance gains are negligible. The programmer knows best. The programmer can synchronize more efficiently and divide tasks more efficiently than any automated "parallelizer" ever will, and having such automated retarded languages will only reduce overall performance when you have lots of applications running, all of which might as well have been sequential, but are now all fighting for multiple cores.
Of the billions of lines of code that runs on most of the world's fastest supercomputers, 99% of it is in FORTRAN. This will NEVER CHANGE. PERIOD. Anybody who tries to change this, should be shown the door. Granted, most if it is still Fortran77, but it works, runs the fastests and the easist to maintain. This is why the next generation of Fortran (Fortran 2008) will be hard-core, parallel driven. You "C" beanies will need C/OPENMPI/OPENMP/TAU to just even try to match Fortran2008's power. Intel knows this, this is why all their parallel libraries, codes and tools are geared for Fortran and NOT C. Live with it...
You know what, I don't think I'm going to use modern English, either.
Don't you know that early modern English was invented to have something standard into which the bible could be translated? For shame!
As a devoted secularist, I'll just burn all my shakespeare and rushdie after I delete all my perl code.
The good and new comes from no quarter where it is looked for, and is always something different from what is expected.
A big problem for scientific computing - and maybe it's a problem elsewhere, too - is that too many programs are collections of legacy code squashed just-so to make it compilable on a new machine.
It's not solely a problem with scientific computing by any means, but as a bioinformatics grad student who used to be a business programmer, I think I can say it's worse in scientific computing than in most other places. As a rule, when you design a business app, you design it for longevity. The company wants something that will not just work for the moment, but work for years, and preferably be maintainable after the person who wrote it leaves the company. Most scientific programs, in contrast, are written to solve a specific problem for a specific project in a specific lab; and the people writing them are most often grad students or postdocs who are pretty much guaranteed to leave at some point, but this isn't taken into account when the programs are being written. (At three in the morning, to meet the deadline for a paper submission or conference presentation the next day ...) If they live on after that, it's pretty much by accident.
There's a reason for this. Most scientists are not computer scientists, and they don't want to be. They've had neither a rigorous education in best practices, nor the production-coding experience to see those practices put into effect. They were concentrating on learning other things -- the core knowledge of their field -- and that knowledge is vital to their research. No one, unfortunately, has the time to learn it all.
I'm not sure what the answer is. I'm a better programmer than most of the people I work with (he says, trying not to break his arm patting himself on the back) but I'm also forty years old and still a year or two away from finishing my PhD. I'm glad to have the experience that's useful in my work, but I kind of doubt that my fellow students (or faculty!) would trade away their youth for that experience. Ultimately I think the answer comes down to having a mix of people in the lab: at least a few who have business programming and/or formal CS education in their backgrounds, working alongside those who have focused their entire careers on the science itself. That way everyone can learn from each other.
The correlation between ignorance of statistics and using "correlation is not causation" as an argument is close to 1.
"MT apps on a single CPU core can have benefits-- such as, your UI can remain responsive to the user during serious number crunching-- but at 100% CPU load, this necessarily comes with the cost of your number-crunching taking longer - by synaptik (125) on Sunday June 07, @03:23PM (#28243359) Homepage
Actually, I have heard tell that multithreaded applications (not specifically SMP designed ones that use Win32 API calls like SetThreadAffinity etc. et al) actually have MORE overheads than single-threaded code, & are therefore actually slower on single-core/single CPU machines, than multithread designed code are on SMP/MultiCore systems.
AND??
As far as the UI being responsive? Yes, I suppose you could use a thread to update/refresh (using form objects methods such as Application.MainForm.Update OR Application.MainForm.Refresh as examples from Borland Delphi), but, then again/alternately?
You can use Application idletime to do that also (Borland Delphi, as just 1 example language I am using here, once more), as Delphi provides an "Application.OnIdle" event you can leverage for that type of work, & it does the job well (I have used it for this before is why I mention it now)...
Using application or systemwide idletime, you can safely perform any screen/ui updates therein, foregoing the use of a thread there (the ONLY reason I mention this, & SPECIFICALLY with Delphi, though it may also be true of other programming languages, is that the Borland GUI RAD controls you're given are NOT "guaranteed" to be 100% thread-safe is why I note this point).
APK
Can Apache use 2 CPUs to serve one request? Well, I thought so.
Oracle, AFAIK, can't use several CPUs to service one query too (I might be wrong).
Parent is kinda flamebait, and it's exactly the opposite of my experience.
Scientists (I am one) who also write some of their own code, have much better things to do with our time than to try and make the software efficient. When we figure out what we want done, we hand it over to professional programmers who, if the cost:benefit analysis works out, will parallelize or optimize it as they're told is needed. Even lousy programmers are expensive, and hardware is cheap.
I 100% agree with the end of his statement - was it 10, 15 years ago scientific computing was still done in fortran FOR A REASON - the optimizing compiler didn't completely suck? Some scientific computing is still done in FORTRAN but that's been purely a legacy thing since the optimizing compilers for C caught up. I'm sure someone clever will find some way to get an interpreted language to figure out what depends on what and parallelize your code for you. This is a very hard problem to do perfectly, but sensible people will quickly realize that's okay. For some cases, I can beat an optimizing compiler by writing assembly - am I ever going to do that? Hell no.
Now, this may result in additional good coding practices which will be required of us so that the optimizing compiler can make easier sense of our code. Might it be lower overhead to create an optimization friendly programming language, which I suspect will end up amounting to making such practices an explicit requirement? Probably not, but it depends on how closely these new programming languages adhere to existing languages (I haven't looked at either example discussed in the article.)
The good and new comes from no quarter where it is looked for, and is always something different from what is expected.
That's funny, because Fallout 3 for instance is enhanced for multicore. I'd assume there were other games.
I'd be skeptical of allowing smart compilers to do the programming for us, a compiler no matter how smart isn't the programmer.
except scientists who use supercomputers
I don't really consider myself a scientist but I've had to write parallel systems twice now. It goes much better when you either don't need internode communication, or you're allowed to implement it that way (when nobody's around to complain they might have to configure something).
Threading I agree is not the same thing... with a decent threading API available you can fan out a loop across all your CPU cores with a small paragraph of code, if the iterations are independent of one other. Setting up that independence (if you have to) is most of the work. But that's a trick you can also use in a UI to speed up animations etc.
Of course there are standard libraries now to handle this stuff, so some of the fun is gone.
I've been very disappointed in parallel programming support. The C/C++ community has a major blind spot in this area - they think parallelism is an operating system feature, not a language issue. As a result, C and C++ provide no assistance in keeping track of what locks what. Hence race conditions. In Java, the problem was at least thought about, but "synchronized" didn't work out as well as expected. Microsoft Research people have done some good work in this area, and some of it made it into C#, but they have too much legacy to deal with.
At the OS level, in most operating systems, the message passing primitives suck. The usual approach in the UNIX/Linux world is to put marshalling on top of byte streams on top of sockets. Stuff like XML and CORBA, with huge overhead. The situation sucks so bad that people think JSON is a step forward.
What you usually want is a subroutine call; what the OS usually gives you is an I/O operation. There are better and faster message passing primitives (see MsgSend/MsgReceive in QNX), but they've never achieved any traction in the UNIX/Linux world. Nobody uses System V IPC, a mediocre idea from the 1980s. For that matter, there are still applications being written using lock files.
Erlang is one of the few parallel languages actually used to implement large industrial applications.
Network stack doesn't run in parallel therefore there is no need to use multiple CPUs to service network requests. Even worse, multi-tasked servers perform much worse then single threaded servers. The bottleneck is the network end not the CPU anyway.
There needs to be an equivalent of Donald Knuth's "Art of Computer Programming" as a definitive reference for parallel algorithms. Until then, I don't care how many cores you have, you won't get the most out of them.
I void warranties.
If they are inherently sequential, then obviously they cannot be made to run in parallel. The truth is that the vast majority of computing applications, both existing and future, are inherently parallel. As soon as some maverick startup (forget the big players like Intel, Microsoft, or AMD because they are too married to the old ways) figures out the solution to the parallel programming crisis (see link below), get ready for a flood of super complex parallel applications to hit the market, especially in the AI, gaming and simulation fields. Cars will drive themselves and robots will maintain your home, that kind of stuff. The possibilities are mind boggling.
Now the reason that the old timers cannot solve the problem is that they are all addicted to the Turing Machine model of computing and last century's multithreaded approach to concurrency. The Turing Machine model is evidently no help in solving the crisis and threads are inherently non-deterministic. There is an urgent need to move away from antiquated and flawed paradigms that do not contribute to the solution. Indeed, they got us into this mess to begin with.
How to Solve the Parallel Programming Crisis
Why would you even try that? For relevant workloads there are more requests than CPUs, handling multiple requests in parallel is inherently more efficient than trying to parallelize a single request. As for Oracle, the query level parallelization is pretty primitive ... but a cursory google search shows it exists http://www.orafaq.com/wiki/Parallel_Query_FAQ
Two points? If that request access both web and a database then yes.
Second why would you want to CPu's to do one task? Why not have each CPU handle their own request and handle twice as many users.
i thought once I was found, but it was only a dream.
Threading is parallelism, if it's not then what is?
Also, Team Fortress 2 has multiprocessor extensions.
I'm trying to tell that servers are a special case. They are trivially parallelizeable.
We still have not solved more complex parallelization tasks. For example, GUI is still mostly single-threaded.
I remember back in the 90's uc berkeley had developed split-c and titanium.
Split-c was interesting in it was C based and had some interesting concepts such as running block on one or many processors, synchronizing processors and spread pointers (pointers across memory across machines).
Titainium was a Java like language for parallel processing, but at the time didn't have multithreaded implemented.
MPI seemed to be the main api used on standard languages.
Being on the inside could you perhaps explain to me why they went with threading instead of message passing?
Multiple threads on a single CPU may not be parallel, but the moment you add more than one core, of course it is parallel.
Syllable : It's an Operating System
Ever heard of pipe-lining?
Looking at the 99 bottles Chapel code (from original article)
http://99-bottles-of-beer.net/language-chapel-1215.html
This looks like the way you do stuff in Haskell. Functions compute the data and the I/O routine is moved into a "monad" where you need to sequence. This doesn't seem outside the realm of the possible.
I have not read the article (par for the course here) but I think there is probably some confusion among the commenters regarding the difference between multi-threading programs and parallel algorithms. Database servers, asynchronous I/O, background tasks and web servers are all examples of multi-threaded applications, where each thread can run independently of every other thread with locks protecting access to shared objects. This is different from (and probably simpler than) parallel programs. Map-reduce is a great example of a parallel distributed algorithm, but it is only one parallel computing model: Multiple Instruction / Multiple Data (MIMD). Single Instruction / Multiple Data (SIMD) algorithms implemented on super-computers like Cray (more of a vector machine, but it's close enough to SIMD) and MasPar systems require different and far more complex algorithms. In addition, purpose-built supercomputers may have additional restrictions on their memory accesses, such as whether multiple CPUs can concurrently read or write from memory.
Of course, the Cray and Maspar systems are purpose-built machines, and, much like special-build processors have fallen in performance to general purpose CPUs, Cray and Maspar systems have fallen into disuse and virtual obscurity; therefore, one might argue that SIMD-type systems and their associated algorithms should be discounted. But, there is a large class of problems -- particularly sorting algorithms -- well suited to SIMD algorithms, so perhaps we shouldn't be so quick to dismiss them.
There is a book called An Introduction to Parallel Algorithms by Joseph JaJa (http://www.amazon.com/Introduction-Parallel-Algorithms-Joseph-JaJa/dp/0201548569) that shows some of the complexities of developing truly parallel algorithms.
(Disclaimer: I own a copy of that book but otherwise have no financial interests in it.)
...and not CPU bound!
Game! - Where the stick is mightier than the sword!
Er, what do you mean by that? Most good apps now keep a thread just for the interface, and new GUIs ship the compositing out to the graphics card, which contains vector processors to perform data parallel operations.
That's at least three levels of parallelism, as I see it.
My experience agrees with yours -- scientific parallel computing seems to be a lot like a battle between the scientists, who want to write quick, sloppy, easy code that just runs, and the parallel computing people who want them to write better code that will run with greater efficiency and in less time.
When the project was put to IBM, Cray and Sun to design these new languages (Sun's Fortress didn't make the cut), one of the specifications was that they should have parallelism not as a quite hard to use bolt-on, like MPI, but that the parallelism should be intrinsic, built-in, and transparent where possible. (As far as I recall.)
I really like the UPC and Coarray Fortran models, which give parallel features that -look- intrinsic: UPC looks like C but with shared type variables (eg. shared int shared_array[4]). The PGAS model works well for this sort of thing, and doesn't scale too badly if you know enough to work out roughly how it works. Fortunately, IBM have picked up UPC and look to be interested in implementing a native compiler for it (alphaWorks has a shared memory parallel version) as well as the Berkeley UPC source-to-source compiler, and a couple of people are implementing Coarray Fortran modules for Fortran compilers.
There are some good people working on these new parallel methods -- and working on bringing them at least part-way to the programmers, rather than making the programmers come to them.
I think you're confusing two different uses of parallelism. One is "small" parallelism -- the kind you see in graphical user interfaces. That is to say, if Firefox is busy loading a page, you can still click on the menus and get a response. Different aspects of the GUI are handled by different threads, so the program is responsive instead of hanging. That's done by using threading libraries like Posix and the like. But that's really a negligible application of parallelism. The really important use of parallelism is for really large programs that require lots of hardware to run in a reasonable amount of time.
The threading libraries that you use for GUI applications don't work well for computationally intensive applications requiring parallelism. They require a shared-memory architecture. (A shared memory architecture is one in which all processors see the same values in the RAM. E.g, if processor #1 writes X to memory block 0x87654321, and processor #2 reads 0x87654321, it returns X instead of whatever value processor #2 last wrote there) Shared-memory architectures don't scale -- the biggest ones you can buy have about 64 CPUs. If you do want to run computationally intensive applications on shared memory architectures, then OpenMP is the library of choice. It's also fairly simple to use.
If you want to run a big applications, you need to use a distributed memory architecture. And MPI (message passing interface) is pretty much the only game in town where that is concerned. It's by far the dominant player.
To make laws that man cannot, and will not obey, serves to bring all law into contempt.
--E.C. Stanton
You are absolutely correct. Oracle does not do true parallel processing. As a result of this, companies like Mark Logic, that specialize in XML databases utilizing XQuery are able to beat Oracle in performance for large server farms. This is due to the fact that XQuery operates in a truly functional manner, like Erlang, allowing for peak parallellization.
Multi-threaded networking isn't new. Most requests are networked. Multi-threaded data sorting and searching isn't new either. Multi-threaded disk handling isn't either.
Whether these tasks have historically been made multi-threaded or not has more to do with the lack of threading support on the CPUs themselves IMHO.
- Michael T. Babcock (Yes, I blog)
No widely spoken human natural language was "invented," Modern English included. Where do people come up with these things? Modern English evolved out of Middle English, just as Spanish evolved out of Latin, etc. Modern English was not "invented" in any meaningful sense of "invernted."
reference for WikiWeenies.
Shared-memory architectures don't scale -- the biggest ones you can buy have about 64 CPUs
The shared memory solutions by SGI and Cray provide up to 8024 processors (Cray XMT).
I use threading as short hand for communication through shared memory with manual locking. PGAS to me seems firmly targeted at the shared memory with manual locking type of programming, the G in PGAS isn't really necessary for message passing
PS. I realize that up to a certain point just accessing remote memory is more efficient than passing on all the data in a message, but those kind of optimizations can be done transparently with pure message passing too (with MOBILE data types in modern Occam for instance). So a completely global memory model is not necessary.
As far as I'm aware, those larger systems are ccNUMA -- cache-coherent non-uniform memory access, in which processors have local memory, but can also access other processors' memory via a network layer. It doesn't quite support the same kind of scaling as proper, uniform access shared memory.
A lot of commenters don't seem to understand this distinction. Personally, I like to use the terms concurrent program and parallel program to distinguish.
There is a lot of concurrent software out there... the often cited examples of web servers, database engines, UI threads and the like are concurrent. There is parallel execution going on, but fundamentally each thread is working on a separate task and the problem that needs to be solved is making sure that the threads don't step on each other's toes (deadlocks, priority inversions, etc.). This is tough, but it's simpler than parallel programming.
In a truly parallel program, you have parallel streams of execution solving one problem. This is two orders of magnitude harder, since most of the algorithms we use regularly aren't easily parallelizable. I do this as my day job; it's two orders of magnitude harder since for every day we spend working on the concurrency problems, we spend a hundred working on the parallelization problems.
It is easy to create a single worker thread that does some offline processing, say creating a document to print. It is harder to split a task into n threads where n scales over time (years after your program was shipped) to match the number of available cores in the machine. Harder still to ensure that all these threads are kept busy with low synchronisation overhead.
I argue that a program gets more "Parallel" as it becomes more scalable in the sense I just described.
I like to think of all programming languages as a way to translate our requirements into a highly explicit form that the computer can understand. We then choose how explicit we want to be (assembly -> C -> Python (say) -> ?)
While the GPU will certainly parrellelize the drawing operations with its specialty hardware, the calling thread is going to have to wait / block for that to complete before doing a refresh.
The blocking portion of that goes back to being a serial operation (waiting for something else to complete), the software is just using the appropriate hardware for the task at hand. I'm don't think that really qualifies as "parrallel" in this case.
Sorry to be nit-picky.
But other threads can do processing on the CPU while that's taking place.
Threading isn't parallelism, it's concurrency. They aren't the same thing.
A concurrent program is logically divided into several interacting threads. Concurrency is about the program's semantics, not its performance.
A parallel program is physically divided into several (not necessarily interacting) operations (usually threads, but might be vector ops) which run at the same time on different pieces of hardware. Parallelism is about the program's performance, not its semantics.
A web browser should be concurrent in that the network thread shouldn't block the Javascript thread shouldn't block the interface. It's less important that it be parallel, though this will improve performance and probably responsiveness when dealing with Javascript-intensive sites and flash movies.
A web server should usually be parallel for better performance. it's not particularly concurrent because its threads don't interact much.
It happens that writing a program concurrently makes it easier to parallelize, but it doesn't guarantee that it will be run in parallel. For example, Concurrent ML performs pretty well despite not supporting SMP.
I hereby place the above post in the public domain.
Mandarin was invented, quite cleverly too(at least the written form).
Did you read the parent? Perl was not invented from whole cloth either, it was based on awk and C. Even most constructed spoken languages, Esperanto for example was based on modern derivatives of Latin. Was Esperanto not therefore "invented"?
The good and new comes from no quarter where it is looked for, and is always something different from what is expected.
Threading i don't count as parallel processing for the desktop.
There are no vehicles that carry more than two people. Anything with more than two wheels I don't count as a vehicle.
SQL Server most definitely can use multiple CPUs to process a single query. For nasty report queries etc it can speed things up significantly.
The problem is there is a large amount of cost in splitting the query into multiple parallel tasks and then merging the results, and for a simple query this outweighs the gains made. SQL Server's default config can be a bit trigger-happy in deciding to do a parallel query, for OLTP applications I normally disable it.
So the bless keyword had to be a biblical reference, it couldn't have been in the form of an augmented constructor without biblical literary reference?
HORSESHIT, I say.
Recent versions of gcc support OpenMP, and there's now experimental support for a multithreading library that I gather is going to be in the next c++ standard. These don't solve everyone's problems, but certainly it's getting easier, not harder, to take better advantage of multi-processor multi-core systems. I recently test retrofit some of my own code with OpenMP, and it was ridiculously easy. Five years ago it would have been a much more irritating process. I realize not everyone develops in c/c++, nor does everyone use a compiler that supports OpenMP. But I doubt it's actually getting harder, probably just the rate at which it's getting easier is not the same for everyone.
"Any program with a "for" or "while" loop in which the results of one iteration"
Think about it... In most real world applications, a for or while loop do depend on some variable in a previous iteration. If something inside a loop does not change you have an infinite loop. Something has to change. In a for loop this is the index variable but in this case, something has changed.
Elvish? Klingon?
Oh, wait, a human language. Huff.
Wait, aren't programmers human?
Well, if an application is written to use threads, then it is both parallel and concurrent, in that the threads could be run on different cores/processors/whatever if they are available.
So you haven't heard of any games or other applications written to be parallel? If they are threaded, they are written to be parallel.
i'd hit it so hard, if you pulled me out you'd be the king of britain [bash.org]
The main problem faced by each new language is "How do I access all the stuff that's already been done?"
The "Do it over again" answer hasn't been successful since Sun pushed Java, and Java's initial target was an area that hadn't had a lot of development work. Sun spent a lot of money pushing Java, and was only partially successful. Now it probably couldn't be done again even by a major corporation.
The other main answer is make calling stuff written in C or C++ (or Java) trivial.Python has used this to great effect, and Ruby to a slightly lesser one. Also note Jython, Groovy, Scala, etc. But if you're after high performance, Java has the dead weight of an interpreter (i.e., virtual machine). So that basically leaves easy linkage with C or C++. And both are purely DREADFUL languages to link to, due to pointer/integer conversions and macros. And callbacks. Individual libraries can be wrapped, but it's not easy to craft global solutions that work nicely. gcc has some compiler options that could be used to eliminate macros. Presumably so do other compilers. But they definitely aren't standardized. And you're still left not knowing what's a pointer so you don't know what memory can be freed.
The result of this is that to get a new language into a workable state means a tremendous effort to wrap libraries. And this needs to be done AFTER the language is stabilized. And the people willing to work on this aren't the same people as the language implementers (who have their own jobs).
I looked over those language sites, and I couldn't see any sign that thoughts had been given to either Foreign Function Interfaces or wrapping external libraries. Possibly they just used different terms, but I suspect not. My suspicion is that the implementers aren't really interested in language use so much as proving a concept. So THESE aren't the languages that we want, but they are test-beds for working out ideas that will later be imported into other languages.
I think we've pushed this "anyone can grow up to be president" thing too far.
Well there is a non interpreted i.E. compiled language where the language designers did not shy away from tasking:
http://en.wikibooks.org/wiki/Ada_Programming/Tasking
And tasking had been there since the original standard in 1983. Only problem: Ada was always considered heavy weight and difficult. In my opinion both is not true - after all the ISO standard for C and Ada are only a few pages apart in size.
Martin
Not just 2 CPUs. Can Apache use 2 CPUs, located on different machines to serve one request, almost twice as fast as it can serve the same request using just one CPU?
Can someone explain me why we would need a new language... There are languages designed from the ground-up to allow parallel processing. Yes, structures inside de core language, not some extra libs. Just to name one: Ada, designed to allow parallel computing since before 1979 (yes, that's 30 years ago). Reference: http://www.adaic.org/whyada/multicore.html. Even on a single processor, doing stuffs memory-bound (such as a sort on a huge amount of data) parallelized can show significant improvement.
In the let the compiler decide attitude of the C language families ... Inmos C had the correct solution. You add two new keywords to the language, parallel and sequential.
sequential
{
stmt1;
stmt2;
stmt3;
}
as opposed to
parallel
{
stmt4;
stmt5;
stmt6;
}
The stmt1 must be executed before stmt2 which must be executed before stmt3 in the sequential construct. C languages actually already support this in a bit more awkward way with the ravel operator. But sequential is an easier to understand and read method, and balances nicely the parallel keyword. The compiler and runtime have been told that stmt4, stmt5, and stmt6 can be executed in parallel. There is implicit synchronization at the end of the statement block.
This is all well and good and many people look and say that it would not be so tough to do this in other ways, and so on. But combine this with fast iterators as are in Objective C 2.0 and it gets much more interesting. Or for the generalized case where any place a left brace is permissible, either of these two constructs could be substituted. This generalizes to braces enclosing a conventional block of statements as exists now, a forced sequential block of statements (so that side affects from say external inputs or other volatile entities can be dealt with at the specific case where needed) or a statement block where the contained statements may be executed in parallel. The programmer still has to have a bit of knowledge here, but the compiler and runtime can really lighten the load. And it does not have a syntax clash with either C, C++ or Objective C so could be adopted by all of them.
I used this back in 1980s and it was awesomely easy to deal with dispatch of hundreds of lightweight instances. Essentially fibers in a more modern vernacular. By partitioning the work between the complier and the runtime systems I ran the same binary code across quad processor and 64 processor arrays. (Ancillary to this discussion was that Inmos Transputers had also built in message passing on dedicated links in hardware. Of course Fortran was also supported as was Pascal, but the main pushed language was Occam. And hardware timers were there as a data type too to make scheduling a breeze. Processors w/o hardware timers just mimicked them in the runtime. And locks were supported in the hardware as well...)
The point is this was a elegantly solved problem in the 1980s that was mostly forgotten. It was a simple matter to have the runtime aware of the fabric an individual process could access and just turn stuff loose. But that part is a bit outside the main discussion, like I don't drift enough already!
- Tjp
I am in wallow with my inner money grubbing capitalistic pig. ... Oink!
imho you shouldn't be worrying about parallelizing things, the compiler/interpreter/whatever should take care of that. How about erlang for the job?
The written form of Korean was invented by a King Sejong sometime in the 16th century I think. He hired a bunch of scholars who devised a simplified phonetic language (as opposed to the ideographic Chinese characters they were using). That's why Korean looks really really different from Chinese, the characters are supposedly designed to represent the throat and tongue configuration needed to pronounce the character.
He forced the populace (under pain of death?) to use it but it worked, literacy rates shot up. All Koreans now use it North and South (but with some Chinese characters used occasionally) and King Sejong is widely revered, I believe he's on South Korean currency (Kim Jong-Il is probably on the North, surprise).
Elvish? Klingon?
Oh, wait, a human language. Huff.
Wait, aren't programmers human?
Esperanto?
A big part of the problem as we have it already is rooted in the ways operate computers, ever since it was the only way to do so (slow, little memory, etc).
We tell our computers not only what to do but out of sheer paranoia HOW to do it. This is because we are not confident we have taught (programmed) the computer to make good decisions and map the road of solutions from the problem to that what we want computer to do, so we employ languages like C to map out every turn the program blindly has to take, no matter the road it is put on really. As most of the programming world, out of habit and what not, employs most slavish forms of imperative programming, what is the chances that a compiler-translator or an operating system (or the underlying hardware which IS INHERENTLY IMPERATIVE by nature) are able to override the decisions that the programmer itself has explicitly made on its behalf and thus ordered it to follow strictly? Granted, some compilers/translators do have freedom of interpretation, but is is also subject to language specification, I mean if you express A is implemented in a B way, then there is so much the compiler can do and no more.
To make an analogy, if your teenage son/daughter interns at your law firm as a your private secretary of sorts, when told "fetch me that contract from the finance dep. on second floor and bring me a good pencil from third for signing it" he/she might not catch and comment on the fact that you don't sign a contract with a pencil, and just follow through your order blindly. If you taught him/her the art of contracts though, he/she might become a much better secretary, might eventually replace you as well :-)
When the programmer assumes he knows most (of all parties involved) EXACTLY HOW the program should solve the problem across time/space, not only for their own testing hardware but for all the combinations of architectures and environments of his programs' users, then those chances are even slimmer. This is program optimization problem, which surfaces when we try to compile our serial source code to run on very parallel systems.
So here we are, discussing solving ways to parallelize our solutions to common problems, when we are like a one-eyed master who tells a slave not only what to do but also how to do it, instead of educating the slave so that he which has better depth-perception can better guide the master. And I am not talking about sloppy out-of-college programmers, this happens to the very best, because the habit was there for so long, I mean we had to tell the slave what to do because historically, that slave character was much more blind than the master and severely handicapped in many areas.
In essence, if we put parallel programming paradigms into an imperative language, how is this going to prevent even great programmers from assuming too much? We need to teach computers how to map the solution themselves, with us only specifying the constraints of such solution, or goals of the program. You might say that such assumptions on human part are always a necessity, because we are just not there yet to have sufficiently intelligent HCI translators, but we should try nevertheless, for the sake of solving several problems at once with one broad look at things. Like one guy here said, how sad is that opening a bunch of pages in Firefox that do absolutely nothing maxes out a modern multi-core, superscalar, out-of-order executing CPU. Is it the faul of a) Firefox slavishly told what to do by programmers that wrote its C code? b) Operating system slavishly doing what kernel calls coming from Firefox tell it do? or c) the underlying hardware slavishly doing what the CPU tells it to do? I'd guess all of the above are equally involved. But can you blame either? All three are, by design, doing their job as they are told.
I hate it when I get asked if I've finished my Forking program by the boss. You think he'd mind his language!
it wasn't "invented" - it was CREATED! (something as beautiful & wide-spread as the English language could only be created by the spaghetti-monster!!)
"Getting the most from multicore processors is becoming an increasingly difficult task for CODERS" - Fixed. Real programmers have no problems handling multiple threads/processes.
How exactly was mandarin invented?
And what exactly do you mean with "the written form"?
The written form is basically just a way of writing the spoken language, it isn't language on its own.
I recommend reading for example this . There's a lot of other great stuff to read on the same site.
Threads Cannot be Implemented as a Library. That means pthreads is bad. Read: http://www.hpl.hp.com/techreports/2004/HPL-2004-209.pdf
Then after a few years, work on Java memory model has found a good solution. Read: Foundations of the C++ concurrency memory model [based on the Java memory model] http://www.hpl.hp.com/techreports/2008/HPL-2008-56.pdf
How fugly can this be for all you C++ wannabe fanguys??? (Phun intended!)
Multitasking is a great idea. So great that even Windows does it :-) Now, what happens when you have one program doing some heavy stuff, and another waiting for user input? The one waiting for user input becomes annoyingly slow, to the point where people tend to give up on multitasking, and just let the heavy program run while they go outside to do something else.
Then some brilliant people decided to fix the problem. If you have two programs running, and two cores, the heavy program cannot slow down the other program (well, as long as at least one of them stays away from the hard drive). Great, now multitasking works at it is supposed to. You don't notice at all, that there is another program running in the background.
Now, some asshole decides that the one heavy program - the one that you finally don't even notice is running - should be able to use multiple cores. Once again, the heavy program will be using the same CPU as the one the user is waiting for. The system feels slow. But this time, more cores is not going to help. The heavy program is built not for 2 or for cores, but for perhaps 64 or even more cores. So no matter how how big a CPU he can afford, it will still slow him down.
Why do some people seem to hate when computers get fast enough to multitask smoothly?
I'm sure I'll get a sure I'll get the Java is slow and it eats all my memories crap but here it goes.
The JVM (Java Virtual Machine) is one of the few platforms that have a well defined memory model (Short Description Wikipedia)
The main problem in parallel programming is dealing with data across different threads, knowing when data written in one thread is visible from another thread, and efficiently communicating between threads. The JVM platform can handle all of this in a deterministic manner, which is key.
Now i say JVM here because it's the platform, and not the Java language, that makes it all work. Java the language (as of 1.5) has great concurrency support, but there are also other languages built with concurrency in mind from the get go like Clojure and Scala.
Plus it all works cross platform.
Had never heard of Inmos before... and it's generally true that many problems in the computing world have been addressed before elsewhere...
Tweet, tweet.
Mandarin is/was the court language of China, it was created in a rather clever way.
The country, being very large had a number of different dialects. Mandarin was developed so that while the spoken word might be different in different areas of the country the Mandarin text was identical regardless of where you were. It was a language of scholars and regular people never really learned it(they didn't need to). The spoken form wasn't used much by anyone who wasn't a courtier(though neither was the written form).
For example, Peking and Beijing are the same place, Peking is what they call(ed) it in the south, and Beijing is what they call it in the north, the written mandarin for both words is the same.
To a certain extent nearly all written languages were initially created because so very few people actually wrote in the early days that there wasn't really any sort of natural evolution for a lot of written languages. For a more specific example, during the early years of the soviet union when the soviets were encouraging the development of their ethnic minorities(as opposed to kicking them off their land and putting them in work camps as they did a few years later. The Soviet government actually sent linguists out to some of the more nomadic of these groups to develop a written form of their language, which prior to this effort had never existed. That's not even counting resurrected languages which likely bear no significant resemblance to their previous forms, but which are spoken by real live people, or made up languages like Klingon that regular folks .
Languages are both created and naturally evolve, and written languages and spoken languages do not always begin at the same time, are not always used by the same people, and are sometimes rather arbitrary.
I think that the whole parallel processing notion is being stood on its head by logic developers working on embedded FPGA apps. The languages of choice are mostly based on VHDL and verilog. The processors involved can be DSP hardware blocks, embedded processors (implemented hardcore or softcore logic), and an arbitrary number of custom state machines, all operating independently or in various locked-step arrangements. Communications between processors are defined as required. Latest FPGAs offer a couple of thousand DSP blocks implemented in hard logic, all with localized memory stores. They can be arbitrarily grouped and/or can run independently, and can be clocked at different rates according to requirements. The resulting parallel processing power and versatility is astounding.
Agreed. Looking at how some computing resources are set up, there is as much (maybe?) of an emphasis on high-throughput computing as on parallel computing ( http://en.wikipedia.org/wiki/High-Throughput_Computing ). As in "how do we make all these long-running jobs complete reliably and so as to use the available hardware efficiently?"
Any discussion of parallel programming would benefit from have read and understood the resources and history covered by Russ Cox at http://swtch.com/~rsc/thread/
sequential as opposed to parallel
That's been tried many times, especially in concurrent FORTRAN variants. The problem is that it doesn't help with locking and interlocking unless the compiler is able to determine that "parallel" tasks really are independent. In some number-crunching applications, that's possible. If the "parallel" tasks are allowed to have any access to shared data, more structure is needed in the program. It's easy to say that you want something done in parallel. The problem is detecting race conditions, providing the locking necessary to deal with them, and figuring out the invariants of the shared data.
Many, many ideas have been tried in this area. Most of them break down when there's a need for complex interaction between threads, as in a window system, browser, or MMORPG server. Partitioning of problems like inverting big matrices in parallel is well understood. But that's not why most people get multi-core CPUs today.
As an Ada advocate I often get crap answers making Ada down. But your answer is better - you know what you are talking about.
The String point is indeed valid. We have to thank the high integrity guys who wanted language which can be used without heap memory - all data on stack.
Same goes for garbage collection - GC is actually advocated by Ada - but the embedded guys wanted it optional. Note that Ada has a distributed extension as well - which too should offer GC.
And the C interface - well yes it is difficult for a high integrity language it interface with the array is pointer mess and still try to keep up at least a bit integrity -do then remember: most viruses and trojans exploit precisely those C weaknesses which Ada does not like.
A Desktop Ada would be nice which does have garbage collection - and perhaps easier to use string. But it won't happen - it is not cool enough, it's difficult to design and develop and also all new languages seem to be interpreted.
Martin
OpenMP Sample Programs
-metric
I suspect the commenter is trying to make something of the first dictionaries in english which standardised spelling. Someone had to decide, (invent ?), spellings for these books. Obviously the spelling wasn't locked down perfectly but modern differences are smaller than those that people dealt with in the past. I suspect that translating the bible into a language spoken more commonly was a motivation among the protestant reform movements more than the other more traditional sect but was this the principal motivation? I find that hard to believe (though I don't really know).
Also, probably the commenter was joking.
The easiest way I've found to take advantage of all the user's cores:
Build COM server exes. Automatically call UnregisterClassObjects() after the first object is created, so the next attempt to create an object creates a new process. Now I have:
1. Multiple processes running in parallel on multiple cores.
2. An easy programming API (the COM interface).
3. No unintended memory sharing, so very little chance of race conditions.
4. A very testable surface. I can write test code in VB that exercise the COM objects.
5. Ability to write different components in different languages.
Nothing like 1990's technology no solve today's problems.
human NATURAL language, which (more or less by definition, actually) means a language which was not consciously designed by humans.
This refers to the vast majority of languages (including modern english, as noted by GP) that people use to communicate with eachother on a day-to-day basis (i.e. excluding math and logic journals)
"So while the application isn't idle (eg crunching large datasets) the UI is unresponsive. Gotcha, that really solves the issue." - by Anonymous Coward on Monday June 08, @03:09PM (#28254419)
You probably made the mistake I & others made a decade++ ago then: You didn't put in 'timeslicing' into your loops in your DB apps... how does this matter? Read on:
It's GOOD, say while "crunching datasets", to put in a SLEEP API call in loops (especially during returned recordsets populating controls)
AND, the really nice part about the SLEEP API call in Win32 is, is that you can set the "sleep interval" itself!
(I.E./E.G.-> lowering it for some more "important users", based on their logon credentials which you can read from a "users table", & secure DB side... just so it works faster for they, vs. those that are "less important" who would get a higher rate inserted, making it slower for them, basically)...
This comes into play MOSTLY in my experience, with returned recordsets from say, Oracle &/or SQLServer & populating grids or listboxes & yes, it can cause problems in TERMINAL SERVER/CITRIX run multiuser + multicampus scenarios...
To solve it? Yes, the sleep API call worked, oddly when DoEvents would not!
(Myself, & 3 others coded that information system, 1.5 million lines of VB code alone not counting server-side SQL, in VB6 to Oracle (PL-SQL) via both OO40 for writes, & native MS middleware in ADO for reads + to this day, 10++ yrs. later? It has been running without major modification running a major companies sales, shopfloor, & inventory info. dead-solid perfect + "bulletproof & bugfree" (yes, one middleware was faster than the other, hence using 1 middleware for reads, & the other for writes)).
Without that SLEEP api call though in loops that took returned data & populated grids & such (RAD controls)?
The app WOULD HANG, & across every remote user across Terminal Server connections (shared line) over 25 miles away @ the companies' other remote location (factory part, whilst we were @ the administrative part most of the time).
APK
P.S.=> Between that Win32 API "SLEEP" api call, & using application idletime? You can do what was asked for, without threads... & just letting them go "high torque" ALL THE TIME, especially over TS shared sessions? A recipe for disaster... hung apps/hung shared sessions galore! apk
Except that firefox isn't multithreaded.
This isn't because its a hard parallelization task, but because GUIs are generally user-bound, not CPU-bound. Until you parallelize the software running on the unit between the chair and the keyboard, you don't need to worry too much about parallizing the GUI.
There are too many qualifiers in this statement. It should simply be "no natural language is invented", since that is true by definition of a natural language. An "invented" language like Esperanto, Klingon, Lojban, or Elvish is a "constructed language", not a "natural language".
All a compiler does is write programming code, given an input which is also programming code (usually, in different languages.)
So, if you don't want a compiler to do that for you, do you hand code all your object code, and then just run a linker?
Shared-memory architectures don't scale -- the biggest ones you can buy have about 64 CPUs.
Try telling that to SGI :-) [see their Altix 4700 line]. But you are correct in that it is hard to scale up shared-memory machines, at least to the scale used by anything on the top500 list.
Nope. It's because UI _is_ a hard task to parallelize.
For example, a lot of programs are still single-threaded. Even the ones like Microsoft Word.
While I pretty much agree with you, a few comments:
But for suitably complex spreadsheets, I can imagine some accountants wishing there was some parallel processing done.
If the spreadsheet is that complicated, then a spreadsheet is probably the wrong tool for the job. But that has to do with how you use spreadsheets, not whether they are parallelized.
I curse at kmail for not being parallel. Sometimes when it runs spamassassin against incoming email, it can get annoyingly slow
That stems from a PC mentality which makes the MUA do double duty as both MUA and MTA. Remember that "Unix" is not a "desktop" OS, but as a do-all OS, happy to simultaneously run "desktop" and "server" apps.
Install fetchmail/gotmail, an MTA (I like postfix, but exim4 works just as well), spamassassin, an MDA (I like maildrop, but procmail is more popular) and an IMAP server (I like courier, but dovecot is also popular).
Once you have it all integrated (it's not as complicated as you might think), with new mail being dropped into ~/Maildir (I have a cron job that runs fetchmail every 5 minutes), then config kmail to also look at the "localhost" IMAP server. (Also, there are scripts on the 'net to convert mbox files to Maildir format, or you could drag-drop using kmail.)
Thus, in practice, you have "parallelism" even though the apps are single-threaded.
(This scheme also allows you to switch MUAs or access your mail from a different machine. (For example, I sometimes read my mail from my wife's Windows box, and one time when Debian Sid broke X, I was still able to read my email with Mutt from the console.)
"I don't know, therefore Aliens" Wafflebox1
I think they come up with this stuff based on their narrow one sided public school education - English class, which teaches them the prescriptive rules of "Standardized English". That really was invented, and now it's taught to all students (in the US at least), and our culture dutifully squeezes every gaff into that narrowly prescribed rule set.
IMO, all public schools should be running linguistics programs in parallel with there English classes. It's important, because the way language works linguistically, is more accurate and useful when you want to really understand something. The culture here could really benefit from more basic understandings of many things - including the way language really works and evolves.
http://www.unfocus.com/
You know I looked it up, and apparently I had some basic facts wrong about the origin of "Standard English", but my point still stands I think. ;-P
http://en.wikipedia.org/wiki/Standard_English
http://www.unfocus.com/
No, again, its because users aren't parallel, so there is very little to gain by parallelizing the UI.
Yes, and because users only do one thing at a time, that isn't a problem.
And Microsoft Word won't benefit from a parallelized UI until you have users trying to click on menu buttons at the same time they are typing text.
"And Microsoft Word won't benefit from a parallelized UI until you have users trying to click on menu buttons at the same time they are typing text."
Or maybe, you know, until user starts spell-checking in the background? Or maybe pagination of a large document?
Those could benefit from parallelization of the engine, but there not so much the UI, where a fairly traditional single-threaded event-driven UI would probably do nearly as well as a multithreaded one, even with a parallel backend.
Well, if an application is written to use threads, then it is both parallel and concurrent, in that the threads could be run on different cores/processors/whatever if they are available.
Not really. Most threaded apps have at most one CPU-intensive thread. The threads are about concurrency, not parallelism... you just want to make sure that mostly-independent parts of the app don't block each other. Usually either everything is waiting for input or just the CPU-heavy thread is running. These apps benefit very little from multicore.
For example, Firefox is threaded (poorly), but it isn't particularly parallel: it rarely or never brings the CPU over 100%, so it can't take much advantage of a dual-core CPU. It could theoretically be made parallel by pipelining or otherwise parallelizing the renderer, or making it run in parallel with the Javascript engine, but these are hard enough and gains small enough that they won't happen anytime soon.
I hereby place the above post in the public domain.
What you are describing as parallel is simply more concurrent. You don't need a separate word. It just confuses people.
i'd hit it so hard, if you pulled me out you'd be the king of britain [bash.org]