So there are some interesting arguments in your comment that I'd like to address.
the entanglements between logic and data differ between paradigms and that entanglement remains because it's built into the semantics of the programs.
That's true for some (perhaps many) languages. Data models are often difficult to translate. That doesn't invalidate the statement that your use of "always" is simply incorrect. Perhaps my use of "usually" was a bit strong. I was thinking in terms of mainstream languages. They mostly all look similar to each other. Common implementation of C++ inheritance is "only" structs and arrays of function pointers. It's not great semantics for the compiler but projects like Self show that you can get good performance if you're willing to put in the resources to get there. Often it's not worth it and current implementations are "good enough."
The compiler's job is indeed to map the semantics, but where the semantics are highly inefficient on that architecture the compiler can't map any better than that. THAT is why code for the Itanium 2 and later has generally been horrible.
Itanium is a really interesting case study. I remember reading and discussing the original IMPACT papers. We quickly came to the conclusion that it wasn't going to work, long before Intel ever shipped an Itanium. Why? Because it required deep dependence analysis on codes primarily written in languages that don't support that (C). The IA64 ISA is a house of cards. If you can't do all of the required static speculation, it falls apart. With enough analysis, profile feedback,etc. I'm sure one can do it. But it's not worth the effort where the powerful hardware of x86 is so cheap and easy to use. So yes, language semantics matters a lot and hardware engineers need to be aware of the issues. I don't think I've ever claimed otherwise.
so it's relatively easy (emphasis on relatively) to design programs that will be fast on a given hardware platform
Tell that to anyone trying to get their C code to perform on vector hardware. The aliasing rules kill it. Things like restrict help but it requires the programmer to check all the call sites and add the keyword when possible.
you lose a lot in the translation of higher-order concepts to machine language
Not really. It depends on the semantics of the language. C++ can often perform better than C even though it is at a slightly higher level of abstraction. That's because the semantics (particularly around templates) are compiler-friendly. C++ retains higher-level information (primarily typing) than C, so the compiler can exploit that to perform transformations it couldn't do on an equivalent C code.
The short story is, "let the compiler do the optimization" has never been the answer to generating fast code.
I've got a couple decades of compiler design and writing experience under my belt and given that experience, I can state pretty definitively that the above statement is hogwash. Of course one can always find examples where the compiler missed something (compiler writers are human!) but I've seen many, many, many examples where the programmer tried to "help" the compiler and destroyed the semantic information available to the compiler, making program transformation impossible. A typical example is taking multi-dimensional array expressions and hand-linearizing them into pointer arithmetic. I weep every time I see that.
Wait a minute, your other post claimed that the paradigm didn't matter.
In the large, that's true. One can make a perfectly fine imperative language that compilers will love. It just happens to be that K&R didn't do that.
well-written code that can work on both will compile to smaller, faster code using the more modern compiler.
Well of course. The more modern compiler has decades of research to exploit. That doesn't change the fact that the Fortran aliasing rules were deliberately designed to be compiler-friendly.
Your claim, if taken literally, is that the first variable is absolute zero.
You're being absolutist. No, it's not absolute zero. But the advances in general program analysis and transformation dwarf the advances of understanding how to exploit a particular language's semantics, especially a language whose fundamental semantics haven't changed much in 20 years. Much of the current whole-program advances, for example, simply wouldn't have been practical 10 years ago due to hardware resource limitations.
My experience is that GCC
GCC is a poor optimizing compiler. In certain cases it can beat other compilers but it's pretty rare. That's not a knock on GCC, BTW. The GCC project's focus is different from, say, HPC compilers.
It's very difficult to compare compilers. The best way to do it is to find the set of options for each compiler on a particular code that performs best. Then compare the performance you can squeeze out of each compiler. This is very labor-intensive work. It can require thousands of experiments on codes that take hours or days to compile. So I very much doubt anyone can say anything about the relative "goodness" of the compilers you've listed.
Does experience with a language matter? Sure. But general analysis and transformation improvements matter more.
A language whose architecture mirrors that of the CPU can ALWAYS be compiled better and cleaner than a language whose architecture differs from that of the CPU
Beyond the hyperbole here ("always" is almost never a good word to use), there is no fundamental law that requires this to be true. The language conveys the program semantics. The compiler's job is to map those semantics onto the hardware. Sometimes this is easier than other times but the language design usually has little to do with it. We did used to have special purpose hardware for LISP, object-oriented designs and even polynomials for God's sake. We found out we didn't need them.
Compiler design is often flawed - GCC shouldn't optimize at all, for example, as early optimization is often bad optimization and makes platform assumptions that are increasingly invalid; optimization should either be in the linker or moved into a completely different program and run as a distinct stage in the chain
I don't even know what this means but it displays a great misunderstanding of compiler architecture. Compilers are highly modular. There is no one place where program transformation takes place. It is a steady refinement of the program down to the instruction level.
Compilers are crappy, for the most part, which is why Fortran is still used in maths. The level of effort that has gone into understanding how to compile Fortran programs is enormous
You don't know what you're talking about. The reason people use Fortran is because it compiles so well to the hardware and the reason it compiles so well is that it has well-defined, compiler-friendly semantics at the procedure boundary. Namely, harmful aliases are forbidden.
In that sense, functional languages are also easy to compile because side-effects don't exist. Compilers aren't crappy, language sematics are and C has some of the worst.
Bulldozer has apparently been a disappointment. Why? One wonders. One suspects that AMD encountered some late-in-the-game, unexpected, and very unpleasant surprises.
Possibly. It's also the case that none of the reports I've seen specify that the benchmarks were recompiled and tuned for Bulldozer. You really have to tune the hell out of your compiler to get good BD code. Code compiled for Barcelona, MC12, etc. will perform pretty poorly on BD.
That's not quite right. Codes need completely different tuning for BD. Take FMAs. No legacy software is going to use them. BD is certainly a challenge but it's a bit much to claim it can never outperform its predecessor.
You are absolutely wrong. My wife experiences it every single day.
And the first thing the doctor almost always tries is "Here's a prescription for antibiotics."
My wife has seen some irresponsible providers operate that way. They tend to be older, probably trained at a time when antibiotics were relatively new and amazing. Younger providers rarely behave this way.
Medicine is in a second dark age.
Medicine is not a science. Do doctors get things wrong? Sure. Do they get it wrong half the time? Not even close. Most doctors I've encountered are far from arrogant. That's a stereotype that just doesn't hold true to my experience.
Your experience is tragic. I'm sorry you got a rotten physician. But you can't label all providers from that experience.
Even though I've always been prone to respiratory infections (I used to get bronchitis and/or pneumonia pretty much every winter), and have become quite a connoisseur of antibiotics
Were the pneumonia episodes diagnosed as such? Bronchitis is almost always viral. Antibiotics won't help and demanding them leads to resistant strains. Many people who claim "pneumonia" or "flu" don't actually have either.
As such, they give a number of antibiotics that we would not do.
That's just not true at all. My wife, a provider, experiences overbearing patients every single day. They demand and demand and demand antibiotics until the provider breaks down from the pure stress of the situation. She rarely gives in, but when she does, she's not thinking about lawsuits. She's thinking, "get these damn people out of my office so I can have some peace and quiet!"
No, that really doesn't happen. My wife is a provider. No clinic in her health system allows drug representatives. Most responsible clinics restrict drug reps in some way, at the very least banning gifts, lunches, etc.
No, it's not rational. They do not ease most illnesses. Most of the diseases that people demand antibiotics for (sinus problems, cough and so on) are viral. Antibiotics will do no good.
Yes, I know you included antivirals but the best treatment is to take a few days off work (if possible) and let the disease run its course.
Of course, the bloodsuckers in corporate America won't let their hourly workers take a day off and that is a serious problem for multiple reasons.
My wife is a provider and sees firsthand every day how patients whine until the providers give in. Drug company subsidies have little to do with it. Responsible clinics ban those anyway. It's patients unwilling to suffer even a day of illness insisting on antibiotics that won't work anyway.
I get a regularly recurring bronchitis. I don't run to the doctor in a couple of days. If it isn't better in about a month and I'm feeling chest congestion, then I'll go see the doctor.
Since the point we reached the ability to handle the power and heat dissipation requirements economically. Engineering is about tradeoffs. Until we get better materials, multicore is more cost-effective than push the clock beyond the reasonable cost envelope.
Sure they do. What do you think a GPU is? History has shown over and over that we can never have enough computing power. Now that we're at the physical limits of clock speeds, parallelism is going mainstream.
If they are so willing to adapt, why weren't they willing to accommodate IBM's change requests?
I don't have any knowledge of what those change requests were, so I don't know the answer. Everything I have read indicates that IBM wanted too much money.
It's not like IBM was totally unwilling to build a $200 million machine.
From what I have read, it seems that they were. They couldn't keep their costs low enough to justify the expense.
I was referring to annotations for GPU offloading. Codes that run on GPUs are in fact so common nowadays that in fact you'll be asked on conferences why you didn't try CUDA if you present any performance measurement sans GPU benchmarks.
Ah, I misunderstood. I don't think directives have been around all that long (PGI's earilier directives and CAPS's directives come to mind) and they certainly weren't standardized. OpenACC, like OpenMP, should allow scientists to write more portable accelerator-enabled code. In fact the OpenACC stuff came out of the OpenMP accelerator committee as explained here. I think it's highly likely some version of it will be incorporated into OpenMP.
You are absolutely right. That's why I wrote, "not a small task."
You seldom achieve competitive e performance with this annotation type parallelization, simply because the codes were written with different architectures in mind.
The nice thing about this is the restructuring one does for GPUs generally also translates into better CPU performance on the same code. So one can enhance the code in a performance-portable way. That isn't possible to do without compiler directives. With directives, one can avoid littering the code with GPU-specific stuff.
They didn't go into that avenue because they knew that their codes wouldn't scale to the number of cores well.
This article explains that five years ago when NCSA made the bid, accelerators were very exotic technology. The move toward GPUs was actually at the behest of scientists who now see a way forward to speed up their codes with accelerators. Technology shifts and we adapt.
None of the supercomputer codes I know uses such a type of parallelization or accelerator offloading.
None? I know of several. It's all still in its infancy of course, but I'm convinced it's possible to get good speedup from GPUs on real science codes. It's not applicable to everything, but then that's why they aren't CPUs.
The mystique is in scaling. It's very hard to run codes on hundreds of thousands of cores and get decent performance. Communication is a huge problem which is why you still see custom interconnect on the high-end systems. Memory architectures on these machines are pretty exotic. It's not just about having a fast processor. It's more about making sure that you can feed that fast processor.
Well, they won't have to completely rewrite all of their codes thanks to OpenACC. They will probably still have to do a bit of restructuring (and that's not a small task) but the nitty-gritty low-level stuff like memory transfers should be handled and optimized by the compiler.
They didn't just buy the name. They also bought all of the people who designed and built those earlier Cray machines. There are still people at Cray who had a hand in the original Cray 1. It's actually a rather nice mix of expertise, multithreading experience from the Tera side, scalable MPP and vector experience from the Cray Research side.
Slashdot Mirror
So there are some interesting arguments in your comment that I'd like to address.
the entanglements between logic and data differ between paradigms and that entanglement remains because it's built into the semantics of the programs.
That's true for some (perhaps many) languages. Data models are often difficult to translate. That doesn't invalidate the statement that your use of "always" is simply incorrect. Perhaps my use of "usually" was a bit strong. I was thinking in terms of mainstream languages. They mostly all look similar to each other. Common implementation of C++ inheritance is "only" structs and arrays of function pointers. It's not great semantics for the compiler but projects like Self show that you can get good performance if you're willing to put in the resources to get there. Often it's not worth it and current implementations are "good enough."
The compiler's job is indeed to map the semantics, but where the semantics are highly inefficient on that architecture the compiler can't map any better than that. THAT is why code for the Itanium 2 and later has generally been horrible.
Itanium is a really interesting case study. I remember reading and discussing the original IMPACT papers. We quickly came to the conclusion that it wasn't going to work, long before Intel ever shipped an Itanium. Why? Because it required deep dependence analysis on codes primarily written in languages that don't support that (C). The IA64 ISA is a house of cards. If you can't do all of the required static speculation, it falls apart. With enough analysis, profile feedback ,etc. I'm sure one can do it. But it's not worth the effort where the powerful hardware of x86 is so cheap and easy to use. So yes, language semantics matters a lot and hardware engineers need to be aware of the issues. I don't think I've ever claimed otherwise.
so it's relatively easy (emphasis on relatively) to design programs that will be fast on a given hardware platform
Tell that to anyone trying to get their C code to perform on vector hardware. The aliasing rules kill it. Things like restrict help but it requires the programmer to check all the call sites and add the keyword when possible.
you lose a lot in the translation of higher-order concepts to machine language
Not really. It depends on the semantics of the language. C++ can often perform better than C even though it is at a slightly higher level of abstraction. That's because the semantics (particularly around templates) are compiler-friendly. C++ retains higher-level information (primarily typing) than C, so the compiler can exploit that to perform transformations it couldn't do on an equivalent C code.
The short story is, "let the compiler do the optimization" has never been the answer to generating fast code.
I've got a couple decades of compiler design and writing experience under my belt and given that experience, I can state pretty definitively that the above statement is hogwash. Of course one can always find examples where the compiler missed something (compiler writers are human!) but I've seen many, many, many examples where the programmer tried to "help" the compiler and destroyed the semantic information available to the compiler, making program transformation impossible. A typical example is taking multi-dimensional array expressions and hand-linearizing them into pointer arithmetic. I weep every time I see that.
Wait a minute, your other post claimed that the paradigm didn't matter.
In the large, that's true. One can make a perfectly fine imperative language that compilers will love. It just happens to be that K&R didn't do that.
well-written code that can work on both will compile to smaller, faster code using the more modern compiler.
Well of course. The more modern compiler has decades of research to exploit. That doesn't change the fact that the Fortran aliasing rules were deliberately designed to be compiler-friendly.
Your claim, if taken literally, is that the first variable is absolute zero.
You're being absolutist. No, it's not absolute zero. But the advances in general program analysis and transformation dwarf the advances of understanding how to exploit a particular language's semantics, especially a language whose fundamental semantics haven't changed much in 20 years. Much of the current whole-program advances, for example, simply wouldn't have been practical 10 years ago due to hardware resource limitations.
My experience is that GCC
GCC is a poor optimizing compiler. In certain cases it can beat other compilers but it's pretty rare. That's not a knock on GCC, BTW. The GCC project's focus is different from, say, HPC compilers.
It's very difficult to compare compilers. The best way to do it is to find the set of options for each compiler on a particular code that performs best. Then compare the performance you can squeeze out of each compiler. This is very labor-intensive work. It can require thousands of experiments on codes that take hours or days to compile. So I very much doubt anyone can say anything about the relative "goodness" of the compilers you've listed.
Does experience with a language matter? Sure. But general analysis and transformation improvements matter more.
Er, wow.
A language whose architecture mirrors that of the CPU can ALWAYS be compiled better and cleaner than a language whose architecture differs from that of the CPU
Beyond the hyperbole here ("always" is almost never a good word to use), there is no fundamental law that requires this to be true. The language conveys the program semantics. The compiler's job is to map those semantics onto the hardware. Sometimes this is easier than other times but the language design usually has little to do with it. We did used to have special purpose hardware for LISP, object-oriented designs and even polynomials for God's sake. We found out we didn't need them.
Compiler design is often flawed - GCC shouldn't optimize at all, for example, as early optimization is often bad optimization and makes platform assumptions that are increasingly invalid; optimization should either be in the linker or moved into a completely different program and run as a distinct stage in the chain
I don't even know what this means but it displays a great misunderstanding of compiler architecture. Compilers are highly modular. There is no one place where program transformation takes place. It is a steady refinement of the program down to the instruction level.
Performance doesn't matter any more.
Oh really? Is that why I just spent the bulk of the last two months optimizing a set of applications for a supercomputer?
Performance doesn't matter everywhere, but it certainly does matter in lots of places. Quite a bit.
Compilers are crappy, for the most part, which is why Fortran is still used in maths. The level of effort that has gone into understanding how to compile Fortran programs is enormous
You don't know what you're talking about. The reason people use Fortran is because it compiles so well to the hardware and the reason it compiles so well is that it has well-defined, compiler-friendly semantics at the procedure boundary. Namely, harmful aliases are forbidden.
In that sense, functional languages are also easy to compile because side-effects don't exist. Compilers aren't crappy, language sematics are and C has some of the worst.
Bulldozer has apparently been a disappointment. Why? One wonders. One suspects that AMD encountered some late-in-the-game, unexpected, and very unpleasant surprises.
Possibly. It's also the case that none of the reports I've seen specify that the benchmarks were recompiled and tuned for Bulldozer. You really have to tune the hell out of your compiler to get good BD code. Code compiled for Barcelona, MC12, etc. will perform pretty poorly on BD.
That's not quite right. Codes need completely different tuning for BD. Take FMAs. No legacy software is going to use them. BD is certainly a challenge but it's a bit much to claim it can never outperform its predecessor.
Patients never "demand them unnecessarily".
You are absolutely wrong. My wife experiences it every single day.
And the first thing the doctor almost always tries is "Here's a prescription for antibiotics."
My wife has seen some irresponsible providers operate that way. They tend to be older, probably trained at a time when antibiotics were relatively new and amazing. Younger providers rarely behave this way.
Medicine is in a second dark age.
Medicine is not a science. Do doctors get things wrong? Sure. Do they get it wrong half the time? Not even close. Most doctors I've encountered are far from arrogant. That's a stereotype that just doesn't hold true to my experience.
Your experience is tragic. I'm sorry you got a rotten physician. But you can't label all providers from that experience.
Even though I've always been prone to respiratory infections (I used to get bronchitis and/or pneumonia pretty much every winter), and have become quite a connoisseur of antibiotics
Were the pneumonia episodes diagnosed as such? Bronchitis is almost always viral. Antibiotics won't help and demanding them leads to resistant strains. Many people who claim "pneumonia" or "flu" don't actually have either.
As such, they give a number of antibiotics that we would not do.
That's just not true at all. My wife, a provider, experiences overbearing patients every single day. They demand and demand and demand antibiotics until the provider breaks down from the pure stress of the situation. She rarely gives in, but when she does, she's not thinking about lawsuits. She's thinking, "get these damn people out of my office so I can have some peace and quiet!"
No, that really doesn't happen. My wife is a provider. No clinic in her health system allows drug representatives. Most responsible clinics restrict drug reps in some way, at the very least banning gifts, lunches, etc.
No, it's not rational. They do not ease most illnesses. Most of the diseases that people demand antibiotics for (sinus problems, cough and so on) are viral. Antibiotics will do no good.
Yes, I know you included antivirals but the best treatment is to take a few days off work (if possible) and let the disease run its course.
Of course, the bloodsuckers in corporate America won't let their hourly workers take a day off and that is a serious problem for multiple reasons.
Or you could just let the disease run its course.
My wife is a provider and sees firsthand every day how patients whine until the providers give in. Drug company subsidies have little to do with it. Responsible clinics ban those anyway. It's patients unwilling to suffer even a day of illness insisting on antibiotics that won't work anyway.
I get a regularly recurring bronchitis. I don't run to the doctor in a couple of days. If it isn't better in about a month and I'm feeling chest congestion, then I'll go see the doctor.
OK, so 1.2GHz cuts that down to 16flops/clock. Since when can anything Intel Architecture achieve that many flops per cycle?
Since LRBni and its 512-bit vectors. A double-precision FMA gets you 16 ops in a clock.
But can they really keep the FP unit running continuously at that rate? On all 50 cores?
Easily. HPC codes regularly keep thousands of cores busy.
Since when?
Since the point we reached the ability to handle the power and heat dissipation requirements economically. Engineering is about tradeoffs. Until we get better materials, multicore is more cost-effective than push the clock beyond the reasonable cost envelope.
Your average consumer doesn't need 50 cores.
Sure they do. What do you think a GPU is? History has shown over and over that we can never have enough computing power. Now that we're at the physical limits of clock speeds, parallelism is going mainstream.
I used the same to build an IVI system for my car and I agree, it's a great setup!
Man, seeing articles like this makes me want to get back into HPC stuff.
Cray is hiring!
If they are so willing to adapt, why weren't they willing to accommodate IBM's change requests?
I don't have any knowledge of what those change requests were, so I don't know the answer. Everything I have read indicates that IBM wanted too much money.
It's not like IBM was totally unwilling to build a $200 million machine.
From what I have read, it seems that they were. They couldn't keep their costs low enough to justify the expense.
I was referring to annotations for GPU offloading. Codes that run on GPUs are in fact so common nowadays that in fact you'll be asked on conferences why you didn't try CUDA if you present any performance measurement sans GPU benchmarks.
Ah, I misunderstood. I don't think directives have been around all that long (PGI's earilier directives and CAPS's directives come to mind) and they certainly weren't standardized. OpenACC, like OpenMP, should allow scientists to write more portable accelerator-enabled code. In fact the OpenACC stuff came out of the OpenMP accelerator committee as explained here. I think it's highly likely some version of it will be incorporated into OpenMP.
That's similar to what PGI is doing.
Yes. In fact they've been working together on it.
It's not that simple.
You are absolutely right. That's why I wrote, "not a small task."
You seldom achieve competitive e performance with this annotation type parallelization, simply because the codes were written with different architectures in mind.
The nice thing about this is the restructuring one does for GPUs generally also translates into better CPU performance on the same code. So one can enhance the code in a performance-portable way. That isn't possible to do without compiler directives. With directives, one can avoid littering the code with GPU-specific stuff.
They didn't go into that avenue because they knew that their codes wouldn't scale to the number of cores well.
This article explains that five years ago when NCSA made the bid, accelerators were very exotic technology. The move toward GPUs was actually at the behest of scientists who now see a way forward to speed up their codes with accelerators. Technology shifts and we adapt.
None of the supercomputer codes I know uses such a type of parallelization or accelerator offloading.
None? I know of several. It's all still in its infancy of course, but I'm convinced it's possible to get good speedup from GPUs on real science codes. It's not applicable to everything, but then that's why they aren't CPUs.
The mystique is in scaling. It's very hard to run codes on hundreds of thousands of cores and get decent performance. Communication is a huge problem which is why you still see custom interconnect on the high-end systems. Memory architectures on these machines are pretty exotic. It's not just about having a fast processor. It's more about making sure that you can feed that fast processor.
Well, they won't have to completely rewrite all of their codes thanks to OpenACC. They will probably still have to do a bit of restructuring (and that's not a small task) but the nitty-gritty low-level stuff like memory transfers should be handled and optimized by the compiler.
It's one petaflops sustained performance, not peak. That means actual real scientific codes running at one petaflops, not just Linpack.
They didn't just buy the name. They also bought all of the people who designed and built those earlier Cray machines. There are still people at Cray who had a hand in the original Cray 1. It's actually a rather nice mix of expertise, multithreading experience from the Tera side, scalable MPP and vector experience from the Cray Research side.