Slashdot Mirror
High Performance Linux Kernel Project — LinuxDNA
Thaidog submits word of a high-performance Linux kernel project called "LinuxDNA," writing "I am heading up a project to get a current kernel version to compile with the Intel ICC compiler and we have finally had success in creating a kernel! All the instructions to compile the kernel are there (geared towards Gentoo, but obviously it can work on any Linux) and it is relatively easy for anyone with the skills to compile a kernel to get it working. We see this as a great project for high performance clusters, gaming and scientific computing. The hopes are to maintain a kernel source along side the current kernel ... the mirror has 2.6.22 on it currently, because there are a few changes after .22 that make compiling a little harder for the average Joe (but not impossible). Here is our first story in Linux Journal."
Compilers shouldn't need to be compatible with each other; code should be written to standards (C99 or so) and Makefiles and configure scripts should weed out the options automatically.
IMHO This is a great development, for one important reason.
Portability of the kernel.
GCC is a great compiler, but relying on it excessively is a bad thing for the quality of kernel code, the wider range of compilers used, the more portable and robust the code should become.
I know there will be the usual torrent of its-just-not-open-enough rants, but my reasoning has nothing to do with that, it is simply healthy for the kernel to be compilable across more compilers.
It also could have interesting implications with respect to the current GCC licensing 'changes' enforcing GPL on the new plugin structures, etc.
GCC is a wonderful compiler however it has in the past had problems with political motivations rather than technical, and moves like this could help protect against those in the future (some of us still remember the gcc->pgcc->egcs->gcc debarcle).
Of course no discussion of compilers should happen without also mentioning LLVM, another valuable project.
Ingo A. Kubblin is quoted as saying:
is that 8-9% overall speedup of applications, or just kernel tasks?
The real "Libtards" are the Libertarians!
Looking at Amdahl's law (golden oldie here) how much time does a PC spend on kernel tasks these days?
You see, I'm a consultant and am paid by the hour.
So GCC is slow compared to the Intel compiler?
Amazing. You have no idea what you're talking about :D
C99 doesn't stop you writing interrupt code OR threaded code.
I completely agree. I ran into this when I was working as a software architect on a project that had been around for a while. Contracts were required compiler compatibility instead of standard compatibility. It made updates to the dev environment much more complicated. The contracts should have specified standards, but its writers didn't know any better-- the customer had no need to stick to a compiler product/version. It also makes your code more dependent upon the compiler's quirks. I would mod you up if I had the points.
:)
I think the point is that ICC has been made "gcc compatible" in certain areas by defining a lot of pre-baked defines, and accepting a lot of gcc arguments.
In the end, though, autoconf/automake and cmake and even a hand-coded Makefile could easily abstract the differences between compilers so that -mno-sse2 is used on gcc and --no-simd-instructions=sse2 on some esoteric (non-existent, I made that up) compiler. I used to have a couple of projects which happily ran on BSD or GNU userland (BSD make, GNU make, jot vs. seq, gcc vs. icc vs. amiga sas/c :) and all built fairly usable code from the same script automatically depending on the target platform.
The over-reliance of the Linux kernel and it's hardcoded options for GCC means you have to port GCC to your platform first, before you can use a compiler which may already be written by/for your CPU vendor (a good example was always Codewarrior.. but that's defunct now)
Of course there is always configure script abuse; just like you can't build MPlayer for a system with less features than the one you're on without specifying 30-40 hand-added options to force everything back down.
A lot of it comes down to laziness - using what you have and not considering that other people may have different tools. And of course the usual Unix philosophy that while you may never need something, it should be installed anyway just because an app CAN use it (I can imagine using a photo application for JPEGs alone, but they will still pull in every image library using the dynamic linker, at load time.. and all these plugins will be spread across by disk)
A few years ago someone figured out that Intel's compiler was engaged in dirty tricks: it inserted code to cause poor performance on hardware that did not have an Intel CPUID.
http://techreport.com/discussions.x/8547
But perhaps they have cleaned this up before the 10.0 release:
http://blogs.zdnet.com/Ou/?p=518
steveha
lf(1): it's like ls(1) but sorts filenames by extension, tersely
They should think about moving to a Java kernel. They could just bootstrap one of the new, clever "Just-In-Time" Virtual Machines at powerup.
These JVMs are able to dynamically optimize the running code in real-time, far beyond what could be achieved by C or C++ compilers, without any performance degradation.
A Java kernel would likely run at least 50 times faster then the very best hand coded assembler - and since the language is completely type-safe and doesn't implement dangerous legacy language features such as pointers or multiple-inheritance then it would be unlikely to ever crash.
If your program is malloc-intensive and you care about performance, you may as well just use a memory pool in userland. It is very bad practice to depend upon specific platform optimisations when deciding which optimisations not to perform on your code. Then you move to another operating system like FreeBSD or Solaris and find your assumptions were wrong and you must now implement that optimisation anyway.
Sam ty sig.
It severely cripples maintenance. Any optimisation, especially one that forks you into multiple parallel implementations (raw C, x86 asm, amd64 asm, amd64 ASM with SSE4, PPC, ....), has to be carefully weighed against its extra maintenance cost.
The parts that do benefit from optimisation, such as RAID parity calculation, symmetric encryption, etc. are already optimised. At any rate I think the kernel developers know a lot more about this than you or I do.
Sam ty sig.
That's being done too. GCC 4.3 with Profile Guided Optimisation is SWEET. I don't think plain PGO can be run on a kernel (but that would be an awesome project), but it would definitely close the gap between ICC and GCC. ICC's PGO is not as good, or rather, ICC itself is better at making the kind of fuzzy predictions that PGO makes definite.
Sam ty sig.
And what is the C99 standard to tell the compiler to pack structures with a 1 byte alignment?
(Hint: there is no standard way)
Why don't they improve GCC to have a 8-9 to 40% performance gain? it's not like intel has some kind of secret magical piece of code that lets them have a better compiler.
--
Stay tuned for some shock and awe coming right up after this messages!
Unfortunately, writing an OS inherently requires making use of functionality not addressed in the C standards. If you stick only to behavior well defined by the ISO C standards you *can* *not* write a full kernel. Doing stuff that low level requires occasional ASM, and certainly some stuff dependent on a particular hardware platform. I think that being as compiler-portable as it is hardware-portable should certainly be a goal. The ability to build on as many platforms as possible certainly helps shake out bugs and bad assumptions. But, just saying "clean it up to full C99 compliancy, and don't do anything that causes undefined behavior" would be ignoring the actual reality of the situation, and makes as much sense as porting the whole kernel to Java or Bash scrips.
To a large extent, they have. ICC really no longer has the performance lead that it once did over gcc. There was absolutely a time when the difference was consistent, and significant. But, a lot has changed since gcc 2.95, when egcs existed. The 4.x branch in particular has been about improving the optimisation capabilities of the compiler. These days, I generally reccomend just going with gcc to anybody who asks me.
GCC itself is rather prolific... Is there any noteworthy platform that it doesn't already support?
Commodore 64?
We were not impressed.
-- Erich
Slashdot reader since 1997
This kernel is so ancient that any possible performance gains are outweighed by the new kernels performance, bug fixes, and improved driver support. Plus why would someone want to toss away their freedom by using a non-free compiler? Also, does the Intel compiler work with AMD processors?
There is so much against this that it is useless, until Intel open sources, can work with up to date kernels, and can work on all x86 and x86_64 compatible hardware (im not sure if this is a problem) then im not interested.
There isn't one, so what you do is use pragmas (I remember #pragma pack(1)) or attributes (__attribute__((packed)) or something similar.
Of course they're compiler-specific but there's no reason that code can't be written wrapped in defines or typedefs to stop compiler-specific stuff getting into real production code nested 10 directories down in a codebase with 40,000,000 lines.
Linux does an okay job of this - but since coders usually reference the compiler manual to use these esoteric pragmas and types, they are usually told "this is specific to GCC" (GCC does a good job of this anyway) so they should be wrapping them by default to help their application be portable and maintainable to future compilers (especially if they change the attribute name or the way it works - as has been done on many a GCC, let alone other compilers).
What usually nukes it (and why linux-dna has a compiler wrapper) is because they're hardcoding options and doing other weird GCC-specific crap. This is not because they are lazy but because the Linux kernel has a "we use GCC so support that, who gives a crap about other compilers?" development policy and it usually takes some convincing - or a fork, as linux-dna is - to get these patches into mainline.
There are actually quite a few ARM processors that do. See Jazelle.
I'm afraid the boost of kernel code won't help you much. Since you're doing fluid physics, I guess the hotspots are in the floating point math computation, and your code doesn't do context switching often. In that case, kernel speed isn't that important.
Well, I'm just saying it. I hope I'm wrong :)
Colorless green Cthulhu waits dreaming furiously.
Java is not a "systems language", meaning you don't write operating systems and systems level code in it for very good reasons.
Funny cause Sun already did that like 13 years ago.
One of them being, name me a processor that can run Java bytecode nativly.
The ARM9E.
I find it hard to believe that the Linux kernel developers never heard of ICC. Or, to take another example, never used Codewarrior or XL C (IBM's PPC compiler, especially good for POWER5 and Cell) or DIAB (or Wind River Compiler or whatever they call it now). Or even Visual C++. Personally I've had the pleasure of using them all.. they all do things differently, but when you have a development team which is using more than one.. I once worked on a team where most of the developers had DIAB, but they didn't want to pay for licenses for EVERYONE, so it was just for the team leaders and release engineering guys, so we all got GCC instead. We had to be mindful not to break the release builds.. and the work ethic meant everything went pretty much fine all round.
All of them have at one time or still today produce much better code and have much better profiling than GCC and are used a lot in industry. If the commercial compiler doesn't do what you want or is too expensive, GCC is your fallback. Linux turns this on it's head because it "wants" to use as much free, GNU software, but I don't think the development process should be so inhibited as to ignore other compilers - especially considering they are generally always far better optimized for an architecture.
As a side note, it's well known that gcc 2.95.3 generates much better code on a lot of platforms, but some apps out there are refusing to compile with gcc 2.x (I'm looking at rtorrent here.. mainly because it's C++ and gcc 2.x C++ support sucks. This is another reason why commercial compilers are still popular :) and some only build with other versions of gcc, patches flying around to make sure it builds with the vast majority, significant amounts of development time is already "wasted" on compiler differences even on the SAME compiler, so putting ICC or XCC support in there shouldn't be too much of a chore, especially since they are broadly GCC compatible anyway.
Like the article said, most of the problem, and the reason they have the wrapper, is to nuke certain gcc-specific and arch-specific arguments to the compiler, and the internal code is mostly making sure Linux has those differences implemented. There is a decent white-paper on it here. The notes about ICC being stricter in syntax checking are enlightening. If you write some really slack code, ICC will balk. GCC will happily chug along generating whatever code it likes. It's probably better all round (and might even improve code quality generated by GCC, note the quote about GCC "occasionally" doing the "right" thing when certain keywords are missing) if Linux developers are mindful of these warnings, but as I've said somewhere in this thread, Linux developers need some serious convincing on moving away from GCC (I've even heard a few say "well, you should fix GCC instead", rather than take a patch to fix their code to work in ICC)
It depends, if the system is distributed, the hotspots (ie performance bottlenecks) could quite easily be in network latency and throughput, something that could be reasonably impacted here.
Of course if its not, you are 100% right, however dont underestimate the proportion of cpu time the kernel spends in some situations (databases and distributed apps, for example).
Depends on the CPU... gcc has reasonable performance on x86, but on ia64 or ppc the vendor supplied compilers have a big advantage. even on x86 icc leads by a considerable margin in some areas, especially on very new processors.
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
All my feature-incomplete software is really fast, too.
XML causes global warming.
On the contrary - there is no support for threads or interrupts whatsoever in C99. Sure, there's pthreads and the like - but those are not part of C99, nor can you implement them in pure C99.
C itself (all versions) tries very hard to avoid tying itself to any specific hardware or OS. It even supports weird things like platforms with more than 8 bits in a char, or with reserved bits in their integers. But as a result, it has only the bare minimum featureset common across all platforms imaginable, and this is why it's very hard to write anything useful with only pure C. (No networking, no listing the contents of a directory, no executing any other programs except via system()...)
For most userland applications, C plus some OS-dependent libraries are good enough, of course. Things like the POSIX API can't be implemented in regular C (at some level you have an assembly call to the OS's syscall interface), but if you treat it as opaque, no problem.
But for an OS kernel, things aren't that easy. In the quest for high performance, Linux does all kinds of neat hacks, including things like inlining assembly code into C functions - and later rewriting that code on the fly (google for 'smp alternatives' for more information). It also makes use of CPU-level atomic operations - and exactly which ones are available depend on the architecture. Because of these kinds of hacks, which produce noticeable speed improvements, it is utterly impossible to stick purely to standards like C99.
A Java kernel would likely run at least 50 times faster then the very best hand coded assembler
I'm going to have to agree with you here. However with all the major browser producers concentrating on JavaScript speed recently, I'd say it's much better to use JavaScript instead of plain Java. Think about it, JavaScript is where the speediness is. Also, since almost every browser supports it, you could just boot the kernel using any browser. This could potentially get the kernel out of the hands of that bunch of self-righteous, elitist Linux hackers who are currently totally disconnected from users like you and I.~
8 of 13 people found this answer helpful. Did you?