The Father of Multi-Core Chips Talks Shop

pacopico writes "Stanford professor Kunle Olukotun designed the first mainstream multi-core chip, crafting what would become Sun Microsystems's Niagra product. Now, he's heading up Stanford's Pervasive Parallelism Lab where researchers are looking at 100s of core systems that might power robots, 3-D virtual worlds and insanely big server applications. The Register just interviewed Olukotun about this work and the future of multi-core chips. Weird and interesting stuff."

Comments

That's a lot of systems.

[Cheesebisquit]

That's a lot of core systems.

Re:That's a lot of systems.

[K. S. Kyosuke]

"Sun Microsystems's Niagra" And they decided to, uhm, "increase their performance" as well...

The Future Is Non-Algorithmic

[Louis Savain]

It is time for professor Olukotun and the rest of the multicore architecture design community to realize that multithreading is not part of the future of parallel computing and that the industry must adopt a non-algorithmic model. I am not one to say I told you so but, one day soon (when the parallel programming crisis heats up to unbearable levels), you will get the message loud and clear.

Re:The Future Is Non-Algorithmic

[hostyle]

Indeed. Its turtles all the way down.

Re:The Future Is Non-Algorithmic

[Anonymous Coward]

That strikes me as crackpottery. The stuff that link describes as "nonalgorithmic" is also easily algorithmic, just in a process calculus.
And guess what? Non-kooks in the compsci community are busily working on process calculi and languages or language-facilities built around them.

Re:The Future Is Non-Algorithmic

[lenski]

To simplify: Dataflow. It's been too many years, but I recall that DataFlow was a company name. Their lack of commercial success was based on the combination of being way ahead of their time.

The recent advent of multiple on-die asynchronous units ("cores") is leading to a resurgence of interest in the dataflow model.

Anyone who has implemented networked event-driven functionality has already started down the path of dataflow model of computation, though obviously it's not fine-grained. The "non-algorithmic model" looks like a fine-grained implementation of a normal network application. (I agree with a downthread post that claims that current and classical Java-based server applications are already there, accepting the idea that event-driven multithreading applications are essentially coarse-grained dataflow applications.) And when the research gets going hot and heavy, I'll wager that the research will end up focusing on organizing the connectivity model.

As far as I am concerned, one place to look for multicore models to shine would be in spreadsheets and similar applications where there is already a well-defined pattern of interdependency among computational units (which in this case would be the spreadsheet cells). I also think that database rows (or row groupings) would be naturals for dataflow computing.

An efficient dataflow system would be the most KICK-ASS life computation engine! :-) (Now you know how old I am...)

Re:The Future Is Non-Algorithmic

[Anonymous Coward]

Hi MOBE2001. Trying Twitter's tricks as well now?

Re:The Future Is Non-Algorithmic

[lenski]

I can see your point... I can imagine a thing that looks a whole lot like an FPGA whose cells are designed to accept new functional definitions extremely dynamically.

(As you can tell, I don't agree with using the name "non-algorithmic": It's algorithmic by any reasonable theoretical definition. This is why I refer to it as being an extremely fine-grained data flow model.)

However, if you look at modern FPGAs, you will discover that even there, the macrocells are fairly large objects.

I guess that when it comes down to it, the "non-algorithmic" model proposed in the page you cite seems so fine-grained that benefits would be overwhelmed by connectivity issues. By this I mean not simply bandwidth among functional components, but in defining "who talks with whom under what dynamically changing circumstances". Any attempt to discuss fine-grained data flow must face the issue of efficiency in connecting the interacting data and control "elements".

There's the possibly even more interesting question about how many of each sort of functional module should be built.

What do you say to meeting in the middle, and thinking about a system that isn't so fine-grained, while also thinking of "control functions" as being just as movable as the data elements? Here's why I ask: In my opinion, there might well be some very good research work to be done in applying techniques related to functional programming to a system of extremely large number of simple functional units that know how to move functionality around with the data.

Re:The Future Is Non-Algorithmic

[hostyle]

Sorry about your turtle, John. We'll get you a new one. Or maybe some Japanese Fighting Fish? They are fun until they start programming your VCR and recording Leno.

Re:The Future Is Non-Algorithmic

[K. S. Kyosuke]

Oh yes, that is why we have (or are currently developing) purely functional programming languages that can often mimic this model quite nicely, and efficient compilers capable of compiling the code into (potentially) whatever paralellism model you are using. Threads should ideally be just a means of implementing paralellism for such languages, of for parallel computing frameworks. Today, you are probably not supposed to write threaded code by hand in most cases. Once you have a reasonable compiler (latest versions of GHC look really promising in this respect), you can stop worrying about how the damned thing works (in most cases) and just write your code.

Just one question - as "algorithm" is essentially just a "recipe" for computing certain results from its input. How a supposedly universal computing machine could work without such recipes to change its mode of operations? No matter whether the machine is a Turing machine, a lambda calculus abstract machine or just a reconfigurable electric circuit.

Re:The Future Is Non-Algorithmic

[Anonymous Coward]

I made myself a lot of enemies apparently.

I don't think its so much enemies you make, its the attitude you take towards the community you are trying to influence. There are many very intelligent computer scientists, and you seem to suggest that most are idiots. You will not be seen as insightful if you cannot recognize the great accomplishments already made.

Personally, I disagree with your positions on physics, and (especially) mathematics. Statements like "Continuity ... leads to an infinite regress" belie your lack of understanding of these mature fields. Who is going to trust your analysis when you make these statements without any real argument? Down-modding these statements is not censorship, it's moderation: we do not need any more of this crap on /.

With this kind of broader view of your posts, its tempting to just throw away all of your comments as "crackpot posts." Which is, by the way, what happened with your previous post: someone just though, "Oh it's that crackpot Louis Savain again; time for a downmod." This is bad, because that post in particular was actually insightful.

The thing is, we do need new programming languages; we do need implicit concurrency; we do need simplicity. Unfortunately, we don't need your arrogance or extremism. You may have something to offer, but it isn't your hate.

Your COSA project doesn't get traction because it requires the world to change; for better or worse, you must take the world as it is and nudge it where you think it should go. There are many people smarter than you who should and do have more sway in the matter: you should be seeking to convince them. Why not code a working version of COSA which can run on a single-core computer, but can exploit arbitrarily many additional cores? People would be less unimpressed by you if you produced a functional product.

Re:The Future Is Non-Algorithmic

[Cheesey]

Right, so you split your computation up into small units that can be efficiently allocated to the many core array. This allows you to express the parallelism in the program properly, because you're not constrained by the coarse granularity of a thread model. Cool.

But the problem here is how you write the code itself. Purely functional code maps really well onto this model, but nobody wants to retrain all their programmers to use Haskell. We're going to end up with a hybrid C-based language: but what restrictions should exist in it? This depends on what is easy to implement in hardware - because if we wanted to stick with what was easy to implement in software, we'd carry on trying to squeeze a few extra instructions per second out of a conventional CPU architecture.

The biggest restriction turns out to be the "R" in RAM. Most of our programs use memory in an unpredictable way, pulling data from all over the memory space, and this doesn't map well to a many core architecture. You can put caches at every core, but the cache miss penalty is astronomical, not to mention the problems of keeping all the caches coherent. Random access won't scale; we will need something else, and it will break lots of programs.

This is going to lead to some really shitty science, because:

• Many core architectures will only be good for running certain types of program: not just programs that can be split into tiny units of computation, but programs that access RAM in a predictable way.
• The many core architects will pick the programs that work best on their system; these may or may not have anything to do with real applications for many core systems (And what is an application for a many core system anyway? Don't say graphics...)
• It will be hard to quantitatively compare one many core architecture with another because of the different assumptions about what programs are able to do in each case. There are too many variables; there is no "control variable".

I think that the eventual winning architecture will be the one that is easiest to write programs for. But it will have to be so much better at running those programs that it is worth the effort of porting them. So it will have to be a huge improvement, or easy to port to, or some combination of the two. However, those are qualitative properties. Anyone could argue that their architecture is better than another - and they will.

Re:The Future Is Non-Algorithmic

[cobaltnova]

My goal is to bash them every chance I get.

Hence the downmod. You just do not learn.

They don't put food on my table or a roof over my head.

Really? I take it you don't use the internet or a computer then? Fail.

Wisdom is 90% guts and 10% sweat.

I just do not see this: considering the amount of guts you've got, where is the COSA toolchain? The COSA OS, with the COSA web-browser, with the COSA frigging interactive editor? Why should I believe anything you say? You HAVE DONE NOTHING.

You are a prime example of what I mean by an ass kisser.

Who's ass am I kissing? Turing? Hawking? Zeno? Have you heard of proof by intimidation? It is not effective. I mean, there are discrete topologies with "infinite divisibility" which are (consequentially) non-continuous, take

{2^{-n}|n\in N}

Where the frig is the contradiction?! I WANT TO SEE; PLEASE SHOW ME!

PS. Why be a gutless coward? Sign your work if you stand by it.

I am not here to make an enemy or collect blood-karma from you. I am here to make a point. I am here because I see fallow potential in you.

Re:The Future Is Non-Algorithmic

[Anonymous Coward]

So sir, what does 0.999... = ?
I guess you're the kind that would say this isn't 1.

Multi-core chips will be constrained by

[Skapare]

Multi-core chips will be constrained by, among other things, the memory bandwidth going off-chip. Maybe they need larger caches. Maybe they just need to put all the RAM on the chip itself instead of so many other cores. How about 4GB of RAM at 1st level cache speed.

Ultimately, we'll end up with PCs made from SoCs, and direct SATA, USB, Firewire, and DVI interfaces coming out instead of a RAM access bus. By the time they are ready to make 256 core CPUs, software still won't be ready to work well on that. So in the interim, they might as well just do tighter integration (that can also run faster there, too). No more north bridge or south bridge. Just a few capacitors, resistors, and maybe a buffer amp or two, around the big CPU.

About the only thing that won't be practical to put in the CPU for a long time is the power supply. They could even put the disk drive in there (flash SSD).

Re:Multi-core chips will be constrained by

[ddrichardson]

That sounds ideal and in the long term is probably what will happen. But you need to overcome two massive issues first - leakage and interference between that many components in one space and of course heat dissapation.

Re:Multi-core chips will be constrained by

[BrentH]

How do videocards handle feeding data to 800 (latest AMD chip) separate processors? The memory controller is onchip of course, and it has a bandwidth of about 50-60GB/s I believe. So, for normal multicore cpu's, try bumping up that DDR2 ram from a measly ~10GB/s (when used in dual channel) up to the same level (AMD again already has the memorycontroller onchip, Intel is going there I believe). DDR(2) being 64bits wide (why?) doesn't either help I'd say.

Re:Multi-core chips will be constrained by

[pjt33]

Three - three massive issues! Leakage, interference between that many components in one space, of course heat dissipation, and having a single, expensive, point of failure. Wait, I'll come in again.

Re:Multi-core chips will be constrained by

[lenski]

Actually, Sun's Niagara has that "problem". The way they solved it is to place Gbit networking close to the cores. There are also multiple DDR-2 memory buses and (I think) PCI-E lanes to feed the processor's prodigious need for memory bandwidth.

The comments to the Register article include a comment about the Transputer. (In case it's not familiar history, the transputer was a really slick idea that went nowhere... 4 high bandwidth connections, one for each neighbor CPU, with onboard memory. I recall that they were programmed in "Occam", a dataflow-oriented language.)

I believe that large-count multi-core systems will remain niche solutions until dataflow "locality", "discovery", and unification of control and data become well understood in a theoretical framework. The niches are nice places to be, though. A high quality game is "merely" a simulation of some virtual reality and simulations are perfectly matched to high-count multi-core systems.

The idea of unifying control and data is no new invention, and anyone trying to patent it should be shot. It's just an ordinary spreadsheet cell. Or possibly a neuron.

Re:Multi-core chips will be constrained by

[mikael]

the transputer was a really slick idea that went nowhere... 4 high bandwidth connections, one for each neighbor CPU, with onboard memory. I recall that they were programmed in "Occam", a dataflow-oriented language.)

Mainly because CPU clock speed and data bus speed were doubling every year. By the time an accelerator card manufacturer had a card out for six months, Intel had already ensure that the CPU was faster and so the accelerator card rapidly became a de-accelerator card. If you look at the advert pages of old Byte magazines, you will see all sorts of accelerator card that tried to offload work away from the CPU (i860's, TMS34020's, quad transputers, video cards with built in networking). All of these were squelched one way or another (Intel created a custom video bus, added the 80x87 FPU, then put it on-core, created the Xeon with a built in i860, added a larger cache, multi-stage pipelines, superscalar architecture, doubled register size from 16-bits to 32-bits, and so on...)

Also, most applications that benefit from parallel processsing required data to be stored in three-dimensional grids, which needed both floating-point acceleration and six and more memory accesses (up and down as well as north, south, east and west). Both Parallel C and Fortran were available for the transputer, but the problem was cost - a transputer accelerator board cost well over 500 pounds just for a four transputer board.

It's a shame the transputer never made it, but in the PC world a manufacturer needed to have a new product out every six months to keep up against Intel.

Re:Multi-core chips will be constrained by

[Fweeky]

The memory controller is onchip of course, and it has a bandwidth of about 50-60GB/s I believe

Which is in fact, around the amount of memory bandwidth Niagara systems have, with 6 memory controllers per socket.

Re:Multi-core chips will be constrained by

[TheRaven64]

Niagara has enough memory bandwidth to keep its execution units happy. The last chip I remember that didn't was the G4 (PowerPC). The problem is more one of latency. This isn't such a problem in a GPU, since they are basically parallel stream processors - you just throw a lot of data at them and they process it more-or-less in that order.

There was a study conducted ages ago (70s or 80s) which determined that, on average, there is one branch instruction in every seven instructions in general purpose code. This means that you can pretty much tell where memory accesses are going to be for 7 instructions, you've got a 50% chance for 14 (assuming it's a conditional jump, not a computed jump), a 25% chance for 21 instructions and so on. The time taken to fetch something from memory if you guessed wrongly is around 200 cycles.

This is a big reason why the T1/2 have lots of contexts (threads). If you need to wait for memory with one of them, then there are 3 or 7 (T1 or T2) waiting that can still use the execution units.

Most CPUs use quite a lot of cache memory. This does two things. First, it keeps data that's been accessed recently around for a while. Second, you access memory via the cache, so when you want one word it will load an entire cache line and data near it will also be fast to access. This is good for programs which have a lot of locality of reference (most of them).

Illogical, Donald Knuth is smarter than that.

[lenski]

Silly. I cannot believe Donald Knuth would be that dense, there must be more to the conversation.

Every major system in existence today is already a "multiprocessor" system, we just don't think of them that way. The average PC is a parallel system running at least 14 CPUs in parallel. (two or three for every spindle, one or two for keyboard, a few for your broadband modem, a few in your firewall, etc etc etc).

Multicore systems are simply an extension of the existing computational model. Plus, every supercomputer built in the last 20 years has been massively parallel.

Out of ideas? I Don't think so.

Re:Illogical, Donald Knuth is smarter than that.

[cnettel]

Those processors have all been part of maintaining the illusion of the von Neumann machine, and to maintain common interfaces between different pieces of hardware. What's happening now is that the model of a single instruction feed is breaking down completely, no matter what task you want to do, if you want it done efficiently. And that's for the very reason that the very smart people designing chips have run out of ideas on how to make them faster while maintaining that very convenient illusion for the smart people writing software for those chips.

No one wants to write threaded code for computations, if it can be done serially. (Parallelism can be quite convenient for processing of requests, like a server, but even then most designs unless done with great care will contain synchronization or non-locality in one way or another.)

Re:Illogical, Donald Knuth is smarter than that.

[lenski]

I'll accept the argument that the single-threaded model is (temporarily) being preserved in current systems. That said, I believe that there is a natural progression toward multithreaded computing as the technologies become more pervasive.

What do you think of such things as SQL and spreadsheets already starting down a road of declarative style of "programming", which would implicitly allow the engines to make their own decisions about how to run multi-threaded?

I had good experience with a quad-phenom running a classical web application recently: The system used all 4 cores very effectively without our needing to make a single adjustment to our extremely simple application.

To me, it appears that we (the developers, theoreticians as well as practical implementers) are already naturally moving to use the resources that the hardware developers are providing. However, I really don't see multi-core systems as "cruft"y as your first comment claims.

Re:First mainstream multicore?

[TheRaven64]

Read more carefully. He created the Stanford Hydra in 1994, and the Niagara is based on this design. They are not claiming the Niagara was first.

Multiprocessor Programming

[yumyum]

I just finished taking a course at MIT on multiprocessor programming. It was taught by the authors of The Art of Multiprocessor Programming, Maurice Herlihy and Nir Shavit. I highly recommend their book, their classes, their expertise. They are now focused on transactional memory, which may make things a bit easier to program in the multiprocessor universe. Of course we can stick with course-grained locking, but as they pointed out early on, Amdahl's Law shows that throwing hardware at a problem may not be successful in upping performance by the amount you expect if the system's scheduler has no hopes of keeping the cores busy due to how you've written your code.

Re:Well

[maeka]

Multicores in the average PC is just marketing.

I'm sorry you do not have a need to run more than one CPU intensive process at a time.

The laptop I am typing this on is 8 years old 2.2GHz celeron

You are significantly off on your estimate of its age.

Horse Pucky.....

[FlyingGuy]

We already have servers for INSANELY HUGE internet apps, its called a main-frame.

It amazes me to no end, how many people still think its about the CPU. It about throughput, ok? Can we just get that fucking settled already? I don't give a rats ass how many damn cores you have running or if the are running 100 gigahertz, if you are still reading data across a bus, over an ethernet connection, ANYTHING that does not work at CPU speed then it makes little difference, that damn CPU will be sitting there spinning waiting for the data to come popping through so it can do something!

Mainframes use 386 chips for I/O controllers and even those sit there and loaf, talk about a waste of electricity! About .01% of the worlds computers need the kind of power that a CPU with more then say 4 cores provide. Those that do are rather busy doing insanely complex mathematics, but even then I doubt that the CPU(s), even when running at "100%" utilization are actually doing the work that they were programmed to do, rather they are waiting for I/O to a database or RAM and fetching data.

Until someone figures out how to move data in a far far more efficient manner then we currently understand, these mega-core CPU's, while nice to think about, are simply a waste of time and silicon with the possible exception of research.

Re:Great Researcher, but...

[Anonymous Coward]

You misperceive the role of a Stanford computer science professor. They're not there to educate you; they're there to create their startups in a risk-free environment with cheap talent. Teaching is just the price of admission.

Re:First mainstream multicore?

[Anne Thwacks]

Well, the fundamental idea behind it was used in the National Semiconductors COP - a 4 bit processsor in the late 1970s.

Incidentally, I worked with Transputers,and the concept died for many reasons

1) The comms channel was a wierd, proprietry protocol, and not HDLC - completely fatal

2) In the event of an error, the entire Transputer netowork locked up - competely fatal

3) Mrs Thatcher eventually agreed to fund the project with $50,000,000 the same day that United Technology (can you say 6502, or was it Z80) cancelled a project saying "in the world of Microprocessors $50,000,000 is nothing". - Two fatal errors here (a) expecting the UK government to fund anything reasonably sensible, and (b) Making it clear that the project is insufficiently funded to survive

4) The project was taken over by the French - whose previous achievements in both hardware and software are [white space here]. 5) Inmos, who made it, (a) tried to force people to use a new language, at a time when there was a new language every month, (b) took two years to discover that the target market wanted C, and (c) never discovered the appropriate language was Algol68.

In short, the company was run by a clever but narrow minded geek, who failed to take advice from others in the industry (including other narrow minded geeks, like me, etc).