Porting Linux Software to the IA64 Platform

axehind writes "In this Byte.com article, Dr Moshe Bar explains some of the differences between IA32 and IA64. He also explains some things to watch out for when porting applications to the IA64 architecture."

Comments

Awesome!

[Wakko Warner]

Now I, and the other two IA64 users, will have some programs to run on our Linux-64 boxes!

Can someone please port nethack for us?

- A.P.

Re:Awesome!

[rcw-home]

Can someone please port nethack for us?

Maybe you could try the patches here?

MOSIX + porting

[Ed Avis]

Well obviously what we'll see next is a kernel extension that dynamically 'ports' all your applications to IA-64 and transparently migrates them to IA-64 machines elsewhere in the cluster. When Intel's next Great Leap Forward is released, you'll be able to transparently migrate to that as well. In fact it will be so transparent, you won't notice any difference and you can continue working at your 80286-based machine without any interruption.

The more things change .....

[binaryDigit]

Ah, porting to homogeneous isa but with a bigger word size. Funny how it's the same old issues over and over again. Structs change in size, bad assumptions about the size of things such as size_t, sizeof(void *) != sizeof(int) (though sizeof(void *) == sizeof(long) seems to be pretty good at holding true here), etc. Of course now there are concerns about misaligned memory accesses, which on IA32 was just a performance hit. Most IA32 types are not used to being forced to be concerned about this (of course many *NIX/RISC types are very used to this).

When things were shifting from 16 to 32 bit (seems like just yesterday, oh wait, for M$ it was just yesterday), we had pretty much the same issues. Never had to do any 8 -> 16bit ports (since pretty much everything was either in BASIC, where it didn't matter, or assembler, which you couldn't "port" anyway).

Speaking of assembler, I guess the days of hand crafting code out of assembler is really going to take a hit if IA64 ever takes off. The assembler code would be so tied to a specific rev of EPIC, that it would be hard to justify the future expense of doing so. It would be interesting to see what type of tools are available for the assembler developer. Does the chip provide any enhanced debugging capabilities (keeping writes straight at a particular point in execution, can you see speculative writes too?). It'd be cool if the assembler IDE could automagically group parallelizable (is that a word?) together as you are coding.

Re:The more things change .....

[CFN]

Well, they days of hand crafted assembly, except for a few special purposes, have long since past. And no one expects assembly writers to be competitive with the compiler's ability to discover and explot ILP.

But the example you mention won't actually cause assembly writers any problems: the code won't be tied to a specific version of EPIC.

The IA-64 assembly contains so-called "stop bits", which specify that the instruction(s) following the bit cannot be run in parallel with those before the bit.
Those bits have nothing to do with the actual number of instructions that the machine is capable of handling.
For example, if a program consisted of 100 independent instructions, the assembly would not contain any stop bits. Now the actual machine implementation might only handle 2 or 4 or 8 instructions at a time, but that does not appear anywhere in the assembly. The only requirement is that the machine respect the stop bits.

Now, you might question how it deals with load-value dependencies (ie. load a value into a register, use that register). Obviously, the load and use must be on different sides of a stop bit, but that would still not guarantee correctness. I'm not sure how IA64 actually works (and someone should reply with the real answer) but I imagine that either: a) loads have a fixed max latency, and the compiler is required to insert as many stop bits between the load and the use to ensure correctness, or b) the machine will stall (like current machines).

Either way, the whole point of speculative loads is to avoid that being a problem.

i386 not designed for servers?

[Ed Avis]

From the article:

Back in the early '80s, nobody at Intel thought their microprocessors would one day be used for servers; the inherent architecture of the i386 family shows that clearly.

That's funny, I thought that the i386 was specifically designed to run MULTICS, which was the very definition of a 'server' operating system (computing power as a utility, like water and electricity). The early 80s was the time Intel designed the i386 wasn't it?

Re:i386 not designed for servers?

[Russ Steffen]

What's really funny is that I have an Intel propoganda book for the "brand new 80386." It spends two whole chapters talking about how the 386 is the perfect CPU for LAN servers. Of course, it also had to spend almost that much space describing what a LAN is and what a server might do, since very few people had ever heard of a LAN at that point, much less had one.

Re:i386 not designed for servers?

[hey!]

386 designed for Multics? I doubt it. Running multics on a 386 would be like scoring Beethoven's ninth for a kazoo.

Multics was pretty much tied to it's unique mainframe hardware with loads more weird addressing and virtual memory management features that would never have fit the paltry 275,000 transitors of the 80386. Also, at the time (1985) Multics was a legacy system; Unix was seen the operating system of the future, in particular because it was portable to microprocessors and didn't require much special hardware.

Debian on the IA64

[hereward_Cooper]

Debian is already ported to the IA64 -- not sure about the number of packages ported yet, but I know they intend to release the new 3.0 (woody) with a IA64 port.

See here for more details

Re:Debian on the IA64

[BacOs]

From #debian-ia64 on irc.openprojects.net:

Topic for #debian-ia64 is 95.70% up-to-date, 96.07% if also counting uploaded pkgs

There are over 8000 packages for i386 (the most up to date architecture) - ia64 currently has about 7650 or so packages built

More stats are available at buildd.debian.org/stats/

PA-RISC and IA32 Native Execution

[morbid]

In the article he mentions that itanic can execute IA32 code _and_ PA-RISC code natively, as well as its own, but these features will be taken away sometime in the future.
Does anyone remember the leaked benchmarks that showed the itanic executing IA32 code at roughly 10% of the speed of an equivalently-clocked PIII?
I wonder how it shapes up on PA-RISC performance?
It has to offer some sort of advantage over existing chips, or no one will buy it.
On the other hand, maybe its tremendous heat dissipation will reduce drastically when they remove all that circuitry for running IA32 and PA-RISC code.
Which leads me to think, why didn't they invest the time and money in software technology like dynamic recompilation, which Apple did very successfully when they made the transition from 69k to PPC?

Why can't i386 assembler be used?

[Ed Avis]

From the article:

Quite obviously, inline assembly must be rewritten from scratch.

I don't see what is so obvious - isn't one of the selling points of Itanium its backward i386 compatibility? Even if running the 64-bit version of Linux it should still be possible to switch the processor into i386-compatible mode to execute some 386 opcodes and then back again. After all, the claim is that old Linux/i386 binaries will continue to work. Or is there some factor that means the choice of 32 bit vs 64 bit code must be made process-by-process?

Interesting question: which would run faster, hand-optimized i386 code running under emulation on an Itanium, or native IA-64 code produced by gcc? They say that writing a decent IA-64 compiler is difficult, and I'm sure Intel has put a lot of work into making the backwards compatibility perform at a reasonable speed (if not quite as fast as a P4 at the same clock).

Re:Why can't i386 assembler be used?

[NanoGator]

" isn't one of the selling points of Itanium its backward i386 compatibility?"

If I remember clearly, the 386 instructions are interpreted instead of being on the chip. That means that those instructions will execute alot slower. It would work, but it wouldnt work well. Its nice because you could transition to IA 64 now and wait for the new software to arrive.

Personally, I dont think that selling point is that worthwhile, but Ill let Intel do their marketing without me.

Re:Why can't i386 assembler be used?

[iabervon]

Changing modes for a single assembly block is not going to work. All of your data is in IA-64 registers, the processor pipeline is filled with IA-64 instructions, and so forth. Switching is a major slowdown (might as well be another process), and the point of having sections in assembly is to speed up critical sections.

In any case, what makes it difficult to write an IA-64 compiler is taking advantage of the things that the new instruction set lets you tell the processor. It's not hard to write code for the IA64 that's as good as some code for the i386. It's just that you won't get the benefits of the new architecture until you write better code, and the processors aren't optimized for running code that doesn't take advantage of the architecture.

Re:Why can't i386 assembler be used?

[Chris Burke]

I don't see what is so obvious - isn't one of the selling points of Itanium its backward i386 compatibility?

Yes. Compatability. Nothing more. Your old apps will run, but not fast. It's basically a bullet point to try to make the transition to Itanium sound more palatable.

Or is there some factor that means the choice of 32 bit vs 64 bit code must be made process-by-process?

It is highly likely that the procedure to change from 64 to 32 bit mode is a privileged operation, meaning you need operating system intervention. Which means the operating system would have to provide an interface for user code to switch modes, just so a small block of inline assembly can be executed. I highly doubt such an interface exists (ick... IA-64 specific syscalls).

Interesting question: which would run faster, hand-optimized i386 code running under emulation on an Itanium, or native IA-64 code produced by gcc?

An interesting question, but one for which the answer is clear: gcc will be faster, and by a lot. Itanium is horrible at 32-bit code. It isn't designed for it, it has to emulate it, and it stinks a lot at it.

They say that writing a decent IA-64 compiler is difficult, and I'm sure Intel has put a lot of work into making the backwards compatibility perform at a reasonable speed (if not quite as fast as a P4 at the same clock).

Writing the compiler is difficult, but a surmountable task. And your surety does not enhance IA-64 32-bit support in any way. It is quite poor, well behind a P4 at the same clock, and of course at a much lower clock. Even with a highly sub-optimal compiler and the top-notch x86 assembly, you're better off going native on Itanium.

NULL barfage

[dark-nl]

The examples he gives for usage of null pointers are both wrong. When a null pointer (whether written as 0 or NULL) is passed to a varargs function, it should be cast to a pointer of the appropriate type. See the comp.lang.c faq for details. The relevant questions are 5.4 and 5.6. But feel free to read them all!

Re:NULL barfage

[ScottMaxwell]

The examples he gives for usage of null pointers are both wrong. When a null pointer (whether written as 0 or NULL) is passed to a varargs function, it should be cast to a pointer of the appropriate type.

Indeed. In the particular case in question, passing a pointer to printf(), this should be (void *) 0 or (void *) NULL.

At least he's right when he says "The following is coded wrong." :-)

Bar is also mistaken on at least one other ANSI/ISO C-related point. He writes:

values of type size_t should use the Z size modifier [to printf], like so:

In fact, the Z modifier in the %Zu construction is non-standard. There was no portable way to print a size_t in the original ANSI/ISO C (C89). C99 (the 1999 revision of the ISO C standard) uses a lower-case z instead, so portable code should use %zu instead. Of course, the kernel is intended for compilation with gcc, not just any compiler, so Bar's example is correct for the kernel but is not (as he claims) standard.

How is that different from a PPC?

[jmv]

A while ago, I tried compiling and running my program (http://freespeech.sourceforge.net/overflow.html) on a Linux PPC machine and (to my surprise) everything went fine. Does that mean that it should work on ia64 too since (AFAIK) both are big-endian 64-bit architectures?

Will 64 bit chips ever make it?

[00_NOP]

When I started messing about with computers 8 bit chips were stanard on the desktop and 4 bit in the embedded sphere.

Within four years 16 bit was the emerging standard for the desktop and four more than that 32 bit was emerging.

In the 12 years since then, well...

32 bit rules in both the desktop world and in the embedded world. Can someone tell me why we aren't on 128 bit chips or more by now? Why do 64 bit chips not amke it - is this a problem of the physics of mobos or what?

Re:Will 64 bit chips ever make it?

[Chris Burke]

It's really not that complicated.

While 4-bit and 8-bit chips were cool and all, no one really thought they were -sufficient-. The limitations of an 8-bit machine hit in you in the face, even if you're coding fairly simple stuff. 16 bits was better but, despite an oft quoted presumption suggesting otherwise, that as well was clearly not going to work for too long.

Then, 32 bits came around. With 32-bit machines, it was natural to work with up to around 4 GB of memory without any crude hacks. Doing arithmetic on fairly large numbers wasn't difficult either. The limitations of the machine were suddenly a lot farther away. Thus it took longer for those limitations to become a problem. You'll notice that for those spaces where 4GB was a limiting factor the switch to 64 bits happened a long time ago. The reason we are hearing so much about 64 bits now is that the "low end" servers that run on the commodity x86 architecture are getting to the point where 4GB isn't enough anymore. Eventually I imagine desktops will want 64 bits as well. I've already got 1.5GB in the workstation I'm typing this on.

When will 128 bit chips come about? I don't know, but I'm sure it will take longer than it will take for 64 bits to become mainstream. The reason is simple: Exponential growth. Super-exponential, in a way. 64 bits isn't twice as big as 32 bits, it's 2^32 times bigger. While 2^32 was quite a bit of ram, 2^64 is really, really huge. I won't say that we'll never need more than 2^64 bytes of memory, but I feel confident it won't be any time soon.

An interesting end to this: At some point, there -is- a maximum bit size. For some generation n with a bit size 2^n and a maximum memory space of 2^2^n you have reached the point where you could use the quantum state of every particle in the universe to store your data, and still have more than enough bits to address it. Though this won't hold true if, say, we discover that there are an infinite number of universes (that we can use to store more data). Heh.

Re:No FP in kernel?

[descubes]

There are two reasons:

1/ The massive amount of FP state in IA-64 (128 FP registers). So the linux kernel is compiled in such a way that only some FP registers can be used by the compiler. This means that on kernel entry and exit, only those FP registers need to be saved/restored. Also, by software conventions, these FP registers are "scratch" (modified by a call), so the kernel needs not save/restore them on a system call (which is seen as a call by the user code)

2/ The "software assist" for some FP operations. For instance, the FP divide and square root are not completely implemented in hardware (it's actually dependent on the particular IA-64 implementation, so future chips may implement it). For corner cases such as overflow, underflow, infinites, etc, the processor traps ("floating-point software assist" or FPSWA trap). The IA-64 Linux kernel designers decided to not support FPSWA from the kernel itself, which means that you can't do a FP divide in the kernel. I suspect this is what is more problematic for the application in question (load balancer doing FP computations, probably has some divides in there...)

XL: Programming in the large

Use Java

[matsh]

And forget about the problem!

Mats

Re:What's the deal with IA64?

[Master Bait]

I think the IA-64 has the potential to be exactly what is needed in an X86 replacement. Its performance isn't necessarily worse than X86, either. Last time I saw SPECint marks, an 800MHz IA-64 actually outperformed an Athlon 1.4GHz in the SPECint-per-MHz deparment. Sure, such benchmarks are unreliable, and even moreso when you try to get per-MHz figures, but really, the IA-64 wasn't doing that bad.

What SPEC needs to benchmark is SPECInt-per-$. Considering that commodity Athlons, Pentiums, Celerons and Durons handily beat the extremely expensive Itanic in a straight SPECInt benchmark, what's the advantage of the IA64 performing more efficiently per mhz?

What IA-64 isn't doing well at is simple integer based desktop stuff like web browsing and word processing. The IA-32 compatibility is also painfully slow.

It was very silly of Intel to graft a 386 unit onto the IA64 chip, that's for sure. Fast int ops are important for running databases. They are essential in supporting that 64-bit I/O.

However, where it really matters, Itanium II might really shine. With much more cache, faster RAM, and some architectural bottlenecks nullified, you can bet this processor is going to smoke the competition if clocked high enough.

That's been Intel's promise since they announced the chip project many, many, many years ago. They also promised that the chip would be inexpensive. It isn't very fast, it isn't a good value compared to todays 32-bit commodity CPUs.

Itanium also scales very well.

From what I've read, the Itanic scales in a way very similar to the Hammer -- 8 CPUs at a time and if you want more than you have to run a pipe between each group of eight. Hammer claims a Hypertransport link between each set with a one cycle wait state (Intels simply calls their a pipe), but really, anything more than 8-way is still going to be the realm of POWER4, UltraSparc, etc. IMO. To tell the truth, the Itanic and the X86-64 will have very similar scaleability, the x86-64 is less than half the die size of the Itanic and better performing. It's NUMA setup gives greater throughput between multiple CPUs in an 8-way or less. It may be ugly on the inside, but both CPUs do the about same thing. And one will be faster and a whole lot cheaper. And don't forget AMD's 4-way chipset. The Taiwanese motherboard makers are going to be moving into that space with this chipset. Commoditization.

X86-64 really can't compete in the high end market. It will put up a fight on desktops, however. Intel is going to make a strong showing with Yamhill or IA-64 simply because of its clout.

Well, just take a 32-bit commodity CPU and kludge it to 64 bits, gain about 25% speedup in doing so and SELL IT FOR AROUND $400 maximum and you will quickly see that the Itanic is sinking! Sure the x86 instruction set is lame, but that's the roll of the dice. If the Motorola 68000 had been chosen by IBM for the PC, we would be singing the same tune. I think the x86 instruction set will be around ad infinitum. Just like the accellerator pedal is on the right side, the clutch is on the left and the brake pedal is in the middle. Totally arbitrary, but it somehow stuck.

Anyway, the original point of my post was just to dispute the fact that IA-64 is a "piece of crap architecture." I think most would agree that it's a very nice and interesting architecture with a lot of potential for performance, if not for success in the market place.

The Itanic wasn't a piece of crap 5 years ago, but it is obsolete today. Intel raves about its "266mhz" memory bus and its 66mhz-64-bit PCI support. You can get this in a commodity motherboard and two Athlon CPUs for around $600. You can get the Pentium 4 with 133mhz X4 quad-pumped memory bus nowadays. The Itanic's parallel execution method is nice, but why did they wait till the CPU was released before they began making compilers that took advantage of this? Completely useless without the right tools (assuming decent tools can be made).