Hardware Selection for AMD64 + Linux?

MrClever asks: "After a disaster involving my cat, a pot of coffee and my workstation, I am now in the market for a new machine. I thought I'd jump on the AMD64 wagon and keep running Linux. After some initial investigation, it became clear that ATi, Promise and other manufacturers don't have 64bit drivers for Linux, which rules out most motherboards with onboard P/SATA RAID, thus limiting my available choices. I know you can run 32bit on AMD64, but if I wanted that I'd get an AthlonXP. So, what AMD64 hardware is the best supported in 64bit mode under Linux? Seems NVidia have 64bit drivers, does anyone else?"

Comments

don't buy an Athlon 64 until new socket comes out!

[Tumbleweed]

Wait for Socket 939 boards & CPUs - the current Socket 754 has a very limited lifespan. Socket 939 processors are due VERY soon now (just saw the first leaked report on one yesterday). FYI.

Of course, this doesn't apply if you're thinking about the Opteron, with its Socket 940.

Hardware RAID on the motherboard

[jmauro]

Most of the RAID on the motherboards are really software RAID that runs in a Windows driver or Linux driver. Since each slot usually shows up as a normal PATA or SATA device, one could then just use Software RAID under linux and get the same effect as the "on board" RAID under 64-bit x86.

Re:Hardware RAID on the motherboard

[Paladin128]

The difference with the on-board RAID on most mobo's is that you can boot to the RAID array. With pure software RAID, you need a non-RAID drive to boot from.

Re:Hardware RAID on the motherboard

[photon317]

The real difference is just that the exact layout of the raid is a pre-set standard by the BIOS vendor, and thus if you run Promise or whoever's softraid drivers in both OSes, you can have multi-platform softraid for a dual-boot setup. Linux boots just fine from a software raid device on it's own without this stuff, I assume windows can do the same.

Re:Hardware RAID on the motherboard

[aminorex]

But that "drive" can be a ramdrive, or a tiny partition which is mirrored and never mounted after
boot. I don't see the disadvantage, frankly.

Even the CPU cost of soft RAID is vanishingly small
these days, when caches are 2-8MB and CPUs are
approaching 5000 bogomips.

I see the difference as being MORE PORTS. Lets
me cram more drives into the box.

Promise works fine

[OrenWolf]

The Promise controller on the Tyan Opteron motherboards works perfectly in both Red Hat Enterprise Linux (with Update 2), and Fedora Core 1/2 for AMD64.. That same chipset (PDC20378) is available on Athlon64/AthlonFX motherboards as well.

Consider 3ware...

[isaac]

3ware cards work a treat in amd64 systems with one caveat - using the PATA Escalade 7500-series cards on the Tyan Thunder K8W (opteron) MB is asking for trouble. The SATA cards work fine.

Promise is junk anyhow; it's not a hardware raid controller, just a dumb ATA controller card with software RAID drivers.

Just do your own software raid in Linux or buy a real (e.g 3ware) controller.

-Isaac

Re:Consider 3ware...

[iamcadaver]

Please elaborate.
I just deplowyed two Tyan Thunder K8W workstations equiped with Escalade 7500 RAID controllers.

Its on a research vessel ready to SAT and deport by mid next month. I've never heard anything about this.

*suddenly nervous*

Software selection

[hackstraw]

Maybe you should either tell us what kind of applications that you want to run. Video drivers aren't too important for a headless box that sits in a closet.

I'm guessing that you are planning on running very large memory applications (> 2 Gig per process), otherwise 64bit support is useless. Especially since _many_ of Linux's applications still have 32bit limitations, even when compiled for 64bit platforms. I've run 64bit linux for 6 or 7 years now, and I'm still pissed that I run into 2Gb file size limits. Remember an int on 64bit linux is still 4 bytes as it is on 32bit systems, so each application has to either use size_t or long to get 64bit integers (which will work on either a 32bit or 64bit machine). Just today I had a user mail me with an error with rcp because it could not transfer a file that was 2.1Gigs. I believe 'cat' has the same limitation, unless it is done as a pipe. For example, cat over_2Gig_file > /dev/null will fail, but cat /dev/null will not. You will find these limitations from time to time, and rarely does the platform matter.

Also, Linux has other limitations like it cannot access a block device over 1 or 2 Tb (depending on the kernel version).

I think that the 64bit hype is amusing. I'm not sure, but an amd64 system running int 64bit mode might be slower than a 32bit offering from either intel or amd. You will have to look at the numbers, but they are hard to find. All of the benchmarks for the opteron that I have seen were run on 32bit applications that were complied with the _Intel_ compiler, or sometimes gcc (and then I believe that they were in 32bit mode).

My recommendation is to 1) kill you cat (just kidding), and 2) just by a stock machine that is either 32bits or look for an integrated 64bit system for linux already, or get a really nice 64bit system (but I wouldn't put Linux on one of those).

Re:Software selection

[Paladin128]

• I'm guessing that you are planning on running very large memory applications (> 2 Gig per process), otherwise 64bit support is useless.

Not at all true! AMD64 has twice the number of general-purpose registers available in 64-bit mode. Some apps also just run faster in 64-bit, like POVray.

Re:Software selection

[Paladine97]

Having twice the general purpose registers will typically improve performance 10-20% just by recompiling everything into 64 bit mode. The grandparent is smoking some major crack.

Re:Software selection

[hackstraw]

Having twice the general purpose registers will typically improve performance 10-20% just by recompiling everything into 64 bit mode.

Data please? this thread mentions povray, well this povray benchmark site clearly shows that the $259 amd64 chip is slower than the $200 Intel offering.

this site has some benchmarks. Note that they use gcc for the pentium machines, which is not a very good optimizing compiler. For floating point apps, I typically see 2x speedup when using the Intel compiler (like oggenc, povray, etc). I cannot say which is faster, but being that there is no good (free) compiler for the amd64 you will just have to take the numbers for what they are (meaningless).

The grandparent is smoking some major crack.

Damn, I've gotta be more discreet.

Re:Software selection

[alienw]

Ummm... Lay down the crack, man. The AMD64 platform offers many more advantages than just it's 64-bitness.

First, the Opteron has an integrated memory controller. That means FAST memory access. If you are running two of them on a dual-channel board, you get a really fast NUMA configuration. That's very important for applications that actually need to calculate stuff, assuming your OS supports it.

Second, it has twice the number of registers. That gives you a large performance advantage over IA-32 because apps don't need to constantly swap out variables into RAM.

Third, the Opteron has 1 meg of L2 cache. That is twice than what Athlon 64 or Mac G5 has, for about the same price. It sure as hell makes a difference, even for normal desktop use.

Also, I see no reason whatsoever to buy an expensive pre-built system when a really nice machine can be put together in a few hours for well under $900. I just upgraded my workstation to an Opteron 140 for only about $600. That's with a server-class board, 400W power supply, and 512 megs of DDR400 registered ECC RAM. Apple doesn't even offer the same features, and a comparable machine costs about $2000 from them. Not to mention that OS X is 32-bit.

Re:Software selection

[aminorex]

You *do* know that registered RAM is shooting your
bandwidth to hell, right? Don't use registered
unless you need it for SMP.

Re:Software selection

[Too Much Noise]

the recompile has to do with performance jumps of amd64 in 32bit mode vs. 64bit mode with the same compiler (gcc). Now, if you compiled the 32bit code with icc and had fp math that icc likes, you won't see much of an increase for gcc-compiled 64bit code vs. icc-compiled 32bit one. But you might still see faster code, depending on the particular app. That is due to a bunch of factors:

• scalar math w/ 16 SSE2 registers vs. vector math with only 8: you can balance out the timings between loads and math ops; in particular, unaligned loads hurt less (besides, AMD's SSE2 unit behaves about the same in scalar and vector mode anyway from typical benchmarks - if you really want timings for that I can check it)
  
• 64-bit registers mean easier data moves between GPRs and SSE2
  
• not least, cache and integrated memory controller issues.
  
• drawback: clock speed is actually important for fp math crunching, so athlon64 is handicapped here.
  

besides, those benchmarks don't show such a 'clear' difference. An amd64 3200+ slower by 5% than a p4@3GHz in 32bit mode w/ WinXP. Not exactly relevant for 64bit.

Re:Software selection

[Too Much Noise]

not just instability. Try filling all the slots in a desktop box (read: non-ECC memory) and you'll see the RAM speed throttle down. You get a trade-off, speed for size. Server-type memory doesn't have that problem with all slots full (different addressing scheme for ECC, if I remember correctly).

Search the support databases of linux distributors

[Rauchbier]

Maybe the support database of a linux company like SuSE may be helpful.

Dyslexia?

[itwerx]

After a disaster involving my cat, a pot of coffee and my workstation...

I can just see it:

"So there I was, petting my cup of coffee and drinking my cat..." :)

SATA with promise chipset works

[MerlynEmrys67]

I am running Fedora Core 2 on a dual opteron Tyan motherboard. I have a 80Gig SATA drive that works fine right out of the box (FC 1 didn't see it though, in fact FC 2Test 3 didn't either)

Dual opteron just rocks

VIA BIOS problems

[S.I.O.]

I'm using a VIA-based Gigabyte m/b and basically everything works fine (SATA, sound), but the famous AMD HALT bug is still not fixed in the latest BIOS, so the kernel is running in polling mode. It means that the CPU cannot switch to sleep mode when there's nothing to do. *Very* irritating. BIOS coders are eventually more evil than trade mark/IP lawyers.

New Cat

[lcde]

After a disaster involving my cat, a pot of coffee and my workstation, I am now in the market for a new machine.

And a new cat.

Re:New Cat

[Cyrano_De]

I don't know, the penny pincher in me says to just format and re-install the old one.....

This proprietary driver situation sucks

[7-Vodka]

For some reason I'm finding myself extremely upset at companies refusing to release the specs to their harware and forcing linux users to use proprietary drivers. That's not what linux is about. There are a lot of us who actually care about using free software.
What's more it's really fucking inconvenient and I hate being forced to use lower-quality software because of greed.

I'm sure there are more than a few upset nforce users for example. The ones who aren't upset, wait till they find a bug or find their performance isn't up to par and take their problems to the lkml. They'll find out that since their platform is a black box they can't get any support and are stuck with what they're forced to use.

I was going to buy an nforce3 dual opteron motherboard but I can't stomach having to use a bunch of proprietary drivers.

I also used to think that it was ok if just the video card had a proprietary driver. It's just one driver afterall right? well apparently the slippery slope has slipped. From now on I will refuse to use any drivers which taint the kernel.

On top of all this, I have to really question the legality of all these proprietary drivers that are popping up. I know there were some threads on the lkml about this recently.
Basically they came to the conclusion that if a driver was written for another OS and merely released for linux as an afterthought it was legal. However if it was written for linux it came under a derrivative work and was not legal.

Either way... PROPRIETARY DRIVERS SUCK

Precisely ...

[vlad_petric]

For modern processors accessing the register file is considerably faster than accessing the memory (even if the memory access doesn't miss in the L1 cache). With 8 GPRs, the compiler can barely alocate 3 for variables (the rest are needed by the compiler for other stuff). That number increases to 10->11 when going to 16 GPRs. This means that for small leaf functions you don't need to go to memory at all ...

The performance increases of 10-20% is precisely what people got by recompiling with gcc for AMD64. This is, indeed, the reason 64-bit architectures perform better than standard x86/32 bits (Itanium has 128 GPRs for instance ...)

My amd64 linux system

[riprjak]

is simple
MSI k8t neo FSIR2 motherboard (some issues with slow bios upgrades)
MSI Geforce FX5950ultra 256MB
Soundblaster Audigy
2 x 120GB ATA4 HDD
1 x 36GB SATA 10k drive
1 x dvd+/-rw CDrw combo
amd64 3200+
1 GB (2 x 512MB) kingston ddr333

This system runs gentoo 2004.1 64bit linux fine. SATA and PATA work fine, but there is not now nor, hopefully, will there ever be support for Software RAID as you find on motherboards (it is pointless feature creep IMHO).

Whilst I would say that ASUS appear to be on the ball with bios updates compared to MSI, my system runs fine (even manages wine using 32bit compatibility libraries and runs windows progs...).

I wholeheartedly recommend 64bit linux and would say that EVERYTHING works except high end ATI radeons (ATI couldn'f find their arsehole with two hands and a roadmap in 64bit terms) and many 802.11g cards (mostly due to the atheros binary driver crap, but support is slowly improving). Couple this with *no* support for software RAID (which is no real use anyway) and you nicely encapsulate most of the problems with 64bit linux. Sure, grub and lilo dont play well at 64 bit, so you will need a liveCD or a chrooted 32bit environment to build them (and some other apps); but 32bit apps execute fine as long as you have a set of 32bit libraries for them to play with.

Go for it, join us, we tools who double as early adopters... then you too can whine at manufacturers for their tardiness in supporting "production ready" 64bit OS'... lol

hope this helps...

err!
jak.

64-bit performance revisited

[kompiluj]

This is a comparison done on sparc platform between 64 and 32 bit modes highliting some performance issues. There are two most important things:

• Amount of performance you'll gain/lose when switching to 64 bit mode depends on the application you intend to run (for instance big gains on SSH/SSL )
• sizes of executables (programs, libs) are significantly larger in 64 bit mode

Of course in the case of AMD64 you will also gain something because of ability to use more registers, which is not the case with sparc.
And one more thing - do take a look at the Solaris 64-bit Developer's Guide. They have done the migration 32->64 long time ago. Learn from them.

platform DOES matter

[David Jao]

Most of this post is just pure misinformation. Some of it has already been corrected in the replies. I will focus on the bits that have not yet been corrected.

Just today I had a user mail me with an error with rcp because it could not transfer a file that was 2.1Gigs.

For example, cat over_2Gig_file > /dev/null will fail

I could see this as being true "6 or 7 years" ago, but any remotely modern linux distribution will NOT have a problem with this, even on 32-bit platforms. Here is a screenshot debunking your claims for both rcp and cat.

Perhaps your problem is that you're not compiling your applications using the large file support API, which all modern distributions do for you whenever possible.

You will find these limitations from time to time, and rarely does the platform matter.

You take a look at this screenshot and try to tell me that platform doesn't matter.

Also, Linux has other limitations like it cannot access a block device over 1 or 2 Tb (depending on the kernel version).

Ironically, the one limitation that you do state correctly is also one of the limitations that only applies to 32-bit systems. It sounds like you need a 64-bit system after all, preferably a modern distribution like Fedora 2 or SuSE 9.1 that doesn't have any of the weird userspace problems that you bring up.

Don't own one

[anti-NAT]

however, I think the ATI 9200 series meets your requirements.

The http://dri.sf.net project is the place to get the scoop on fully open source supported 3D cards.