Best Motherboards With Large RAM Capacity?

cortex writes "I routinely need to analyze large datasets (principally using Matlab). I recently 'upgraded' to 64-bit Vista so that I can access larger amounts of RAM. I know that various Linux distros have had 64-bit support for years. I also typically use Intel motherboards for their reliability, but currently Intel's desktop motherboards only support 8GB of RAM and their server motherboards are too expensive. Can anyone relate their experiences with working with Vista or Linux machines running with large RAM (>8GB)? What is the best motherboard (Intel or AMD) and OS combination for workstation applications in terms of cost and reliability?"

Comments

Tyan?

[therufus]

Have you looked into Tyan mainboards. They're more for the server market, which is really what you're aiming for.

Re:Tyan?

[arivanov]

I would concur. Tyan Opteron motherboards are probably the best choice for this. The only annoyance is that most of them are EATX and fit only in high end huge cases.

The other thing to do is to abandon Windows. Matlab behaves considerably better on Linux or Solaris than on Windows (especially on big data sets). Most Matlab users I know have long stopped trying to run it on Microsoft platforms. They are simply not fit for purpose. AFAIK Vista is no exemption. So if you really make a living off matlab you should move your other windows stuff onto a cheap and cheerfull small PC and switch the matlab monster to a "proper" OS. That is the way I have maintained it for my matlab users in the past and they have been happy with the arrangement.

Re:Tyan?

[Erpo]

If you're running Matlab on Linux, you'd better pick one version of one Linux distribution and make sure the version of Matlab you're using supports it. If you change distros or get updates, expect problems, like crashes when you multiply [1,0]*[1;0].

If you're a free software advocate, you could blame this on the mathworks for not providing the source to Matlab so that it can be endlessly tweaked and rebuilt to keep up with FOSS development.

If you've got any common sense, you can blame this on OSS developers who do things like making binary incompatible changes to libraries and doing nothing to make sure old programs don't accidentally load the new, incompatible libs. It doesn't make one iota of difference if it's "wrong" for a program to access errno directly! There are programs that depend on being able to do it, and taking away "extern int errno" breaks those programs (including Matlab).

Then there was that whole NPTL mess. *sigh*

Re:Tyan?

[krilli]

Are EATX boards really that big?

I got an Athlon MP board a while ago. As far as I can remember, it was EATX and fit into a regular old case. There wasn't a lot of space left, but it did fit. If I'm not mixing things up, EATX have the same mounting holes, and the extra board area just flows into normally unused areas of the case.

The listed size is deceiving - I am prepared to be wrong, but I urge you to check again.

Re:Tyan?

[Erpo]

Oh me too. But if I'm forced to choose between threads and Matlab, I'll take my Matlab. Especially if Matlab is the whole reason the computer is there in the first place.

Re:Tyan?

[jedidiah]

If enterprise database vendors can manage to support Linux, then some desktop application vendor shouldn't have any problem.

Things get deprecated in all manner of environments that coders have to deal with. Linux may be more annoying in this regard but it's hardly unique.

Nothing really forces you to alter a Linux installation once it's been deployed. Running a 5 or 8 year old copy of Linux doesn't quite have the same problems as doing the same for Windows. You can do the same with MacOS or Solaris too (safely run a 5 year old copy of the OS).

Tyan

[B5_geek]

Look no further then Tyan. The Tempest line (Intel CPUs) can hold 32GB of ram and the Thunder line (AMD CPUs) can hold 64GB of ram.

Now I am curious about one thing you said about Intel mobos:

and their server motherboards are too expensive If you are too cheap to buy a mobo that in your own words was "reliable, and solid", how the heck are you going to pay for the 32GB of ECC RAM?

I run a Tyan Thunder with two Opteron 270's (and 4GB of RAM) as my primary workstation, and I have never been happier. I can honestly say that this is the last workstation I will buy until it dies, I no longer need to worry about "but my computer can't run X".

With the memory sizes and data sets that you are talking about I wouldn't consider anything other then AMD CPU's. The bandwidth that the CPU and memory are shared on Intel boards, and each AMD cpu has a dedicated memory controller and dedicated RAM slots.

You posted this on /. so you know that Linux will be the preferred OS.

Go with AMD, you won't be disappointed.

Re:Tyan

[ZorbaTHut]

Funny thing about this, actually.

I jumped on the dual-processor bandwagon pretty much the instant that commodity CPUs officially supported it. Namely, the Athlon MP. I got a Tyan Tiger motherboard and a friend did the same. Shortly thereafter I lost contact with that friend.

A few years later I went to turn on my computer as usual and it wouldn't turn on. A bit of troubleshooting later and I realized that the PSU connector had burned itself into the motherboard power socket because something on the motherboard had randomly decided to short itself. Four of the pins had fried (in a distinctive pattern, see here and here) and I ended up buying a new motherboard from a different manufacturer and a new power supply (thankfully, the other components had survived fine.)

About a year after that I ran into my friend. We were talking about upgrades and I dug out those pictures. Turned out he'd lost three Tyan Tiger motherboards, with the exact same burn pattern, before changing manufacturers.

So, yeah, I'm not touching Tyan again. I've never actually had a computer component burn itself to death before, and one time was enough.

Re:Tyan

[cowbutt]

A few years later I went to turn on my computer as usual and it wouldn't turn on. A bit of troubleshooting later and I realized that the PSU connector had burned itself into the motherboard power socket because something on the motherboard had randomly decided to short itself. Four of the pins had fried (in a distinctive pattern, see here and here)

What make/model of PSU were you using? Looks to me that the PSU's power connector couldn't cope with the current the board was pulling. Of course, that may or may not be down to a fault on the board, but seeing as you haven't told us anything about the PSU, I'm betting you were using a cheapo one that came with the case you were using.

Re:Tyan

[phantomlord]

S2460?

I just went through the same thing in late October.I built the system in June, 2002. Had a problem with the initial power supply (Antec 430W) that came in my case not being big enough and I was getting random lockups. Switched to another power supply and everything was fine for years. I went away for a weekend and sshed in to read my email. On my way home, a friend called me and noticed that I wasn't logged onto AIM. Hmm, ssh was still up. I must have gotten disconnected and it didn't full reconnect right (gaim likes to do that from time to time).

So I get home and notice that my computer had rebooted rather than having a simple AIM disconnect. Odd, but whatever. A week goes by and no problems. I attribute it to a freak power problem since I notice my UPS battery isn't holding a charge. Suddenly, my computer reboots in the middle of playing nethack. Ok, that was weird. Power failure is set to turn the computer off, not reboot. Computer hangs before LILO runs. I reset it and it hangs at LILO again. Odd. So I take the side of the case off to make sure the CPU/GPU fans are spinning. No problems. I let it sit for a few minutes. Turn it on and everything is fine again.

I grumble about losing my nethack game and start anew. I get about 5 minutes in and my computer reboots again. This time, LILO starts loading Linux and the computer reboots before the image is uncompressed. It does the same thing again. I start smelling that aroma of burning electric and plastic. Ok, it's too late to deal with this. I power off for the night and decide to come back in the morning.

Same problem in the morning, as soon as the computer gets warm, it starts rebooting. Electrical smell is getting heavier. I start taking PCI cards and drives out to make sure they aren't causing a problem. I swap in a known working video card and that's the only thing connected to the motherboard. Same problem. Ok, maybe the power supply is flaking out. I go to disconnect it from the motherboard and it was stuck pretty good. Bad enough that I had to get out some pliers and really start yanking.

Exact same pattern as you, all the red (5V) connectors are burned out. Fearing the worst, I ordered parts for a new computer. Later that night, I decided to see how bad the problem really was. I took a scalpel and small finger drill and cleaned all the melted plastic out of the motherboard connector until I could plug my backup power supply in. Some quick testing showed everything worked, so I reassembled it and used it for a week until my new parts got here.

I think what happened, in my case, is that the UPS batteries went bad (I've since replaced them) and the minor power fluctuations caused something to burn out in the power supply, which in turn, affected the current it was providing. It's worth noting the rating on my power supply says it provides 42A across the 5V wires, so we're talking some significant juice to start with.

That said, it is interesting that we've all had the same failure result.

Re:Tyan

[toddestan]

I jumped on the dual-processor bandwagon pretty much the instant that commodity CPUs officially supported it. Namely, the Athlon MP.

Wandering off-topic, but you were kind of late for that bandwagon. Dual processors was supported by the original Pentium, though the Athlon MP might have been the first attempt by AMD to do so.

Re:Tyan

[kjs3]

And you'd be wrong, since he specifically said 64-bit versions.

Re:Tyan

[everphilski]

Yup, I was using 2 Intel 366 MHz Celeron processors (overclocked to 550MHz) back in high school, using an Abit BP6 motherboard. At $30 apiece for the processors and picking up the motherboard for $70 at a trade show, it wasn't a bad deal at the time for a gigahertz when most people were running less than half of that.

Re:Tyan

[Bastardchyld]

To elaborate a little more on the previous answer. This memory limitation is not a limitation with xp/vista this is a memory limitation with 32 bit xp/vista. If you are using 64 bit xp/vista the 3+gb limit disappears.

-matt

Re:Tyan

[ZorbaTHut]

Yep, S2460.

I like your diagnosis about the PSU. I'll have to dig out that old PSU (I still have it somewhere because the burnt connector is so awesome) and see if it has bad caps. But yeah, even if it was the PSU, the motherboard definitely contributed somehow - I've posted that picture quite often and the only people who've ever said "Hey, I had the exact same pattern!" were people with S2460s.

I doubt the need for that much ram.

[djcapelis]

Is your working set honestly over 8GB? Your dataset might be extremely large... but I would think that for the most part you'd get along just fine with swapping out to a decently fast device and your working set would be considerably below 8GB.

Consider swapping to and from a flash device or a series of flash devices. That will get you better latency over a spindle. If you want bandwidth though, you'll need to go with a hard drive. I find it very unlikely even with matlab (bloated as it is) that you honestly will improve performance considerably with >8GB of physical memory... Then again, I have no idea how good Vista is at swapping these days. But they talked about ReadyBoost and all that, so I assume it doesn't suck at it completely. :)

If you really are worried about I/O performance, you should consider getting multiple chips (and cores, but mostly multiple chips) so you have more L1/L2 cache available to access. Though this assumes your applications are somewhat parallelizable...

Generally this question is a lot more complex than simply assuming throwing more ram in the box is going to be the best use of your money.

Re:I doubt the need for that much ram.

[jacquesm]

I love that attitude...

Some guy comes and asks an honest question. Then people go and tell him that can't be right and then go and give all kinds of suggestions taking into account that he isn't right.

Let's just for a second assume that the OP has a dataset that large. I can easily imagine it:

- complicated physics model
- computational biology problem
- datamining

and any one of a thousand other not so trivial computational problems.

If his 'luck' is the problem is not trivially parallelizable (I hope that's spelled right) then he's got two choices:

1) try to set up some kind of pipeline
2) get a single machine that can handle all the data

Apparently he has chosen for door #2 because that seems to be just about feasible.

There are some top of the line dell machines that will hold up to 128G of ram, the R900 series.

Re:I doubt the need for that much ram.

[16384]

Is your working set honestly over 8GB? Your dataset might be extremely large... but I would think that for the most part you'd get along just fine with swapping out to a decently fast device and your working set would be considerably below 8GB.

When doing computer simulations it's really easy to need that much RAM. I currently have 4 GB (2xQuad Xeons on a Tyan motherboard -- To the OP: Get Opterons instead if you can), but could sometimes use much more. Swap is not an options: When the memory hits the swap the performance simply drops to pathetic levels.

In computational physics you are in a constant struggle between the need for more accuracy and the limits of the machine.

Re:I doubt the need for that much ram.

[Jeff DeMaagd]

If it's that kind of data, then it's really worth paying more for a solid workstation class board. And it almost assures you of ECC compatibility. ECC isn't necessary for home use and gaming, but if you have a need for 8GB+ of memory, then you probably should protect that data, and it's not terribly expensive either, in my opinion, last year's FB-DIMM pricing notwithstanding, but even that's very affordable now too.

Re:I doubt the need for that much ram.

[Jeff DeMaagd]

ECC might not be that important for you. ECC memory only helps resist bit flipping while the data is in memory. It won't make your backups much more reliable as it's mostly the reliability of the medium, when backing up, the amount of time data is in memory during the transfer is very short. If you keep gigabytes of data in RAM for days at a time, or if the data is valuable, then ECC would be one step, in conjunction with mirrored or RAID-5 storage and off-line backups.

Re:I doubt the need for that much ram.

[jacquesm]

even the not-so-wealthy can have skills that allow them to ask difficult questions but may not have a corresponding budget.

I used to be in that position. Now I run my more interesting software on a 5 node dual opteron cluster (small for a cluster, I know... see that's those budget constraints again), each node has 8G of ram and 3TB storage. Before that it was 10 pentium machines at 600 Ghz (See http://clustercompute.com/ , which has inspired numerous people to build copies) and before that it was 10 pentium 225's (overclocked 200's :) ). What used to take weeks now takes at the most days. My applications are mostly in datamining, but I find computational biology to be very interesting.

You have to love it when people overcome their financial limitations with cleverness, why not give the guy a break and simply help him to solve his problem, starting out from the assumption that his problems and limitations are real.

It would have been nice to have a few more bits of information about the kind of data and the nature of the calculations, I'm pretty sure that 'cheap' is also relative but it seems that cheaper is better for this guy. How many people are at their most brilliant periods in their lives when they're also poor is not easy to figure out but I would not be surprised if it was the majority.

Re:I doubt the need for that much ram.

[TheSkyIsPurple]

Maybe he figured that most everyone else would answer his direct question, but he thought might have deeper insight into the problem?

I don't know how many times I've been focused on a problem for a long time, ventured down a solution path, and ended up asking for help for something complicated; only to have that guy ask me what I was thinking. When I explained the problem, it turns out I had missed something that drastically reduced it.

Sort of like the ol' America space pen vs. Russian Pencil story.

In other words, he was getting at the underlying concern, not the question asked. (think "Do I look fat?"... that's really not what they're asking)

Re:I doubt the need for that much ram.

[Detritus]

Sort of like the ol' America space pen vs. Russian Pencil story.

Which is a myth. Do you have any more pearls of wisdom?

Re:I doubt the need for that much ram.

[try_anything]

Is your working set honestly over 8GB? Your dataset might be extremely large... but I would think that for the most part you'd get along just fine with swapping out to a decently fast device and your working set would be considerably below 8GB.

...

more L1/L2 cache available to access. Though this assumes your applications are somewhat parallelizable...

That's a big assumption. Give the guy a break! Maybe he's just working on a problem where there's no known way to achieve predictable data access patterns. After all, not everyone doing math on computers is solving differential equations. When somebody says their working set is over 8GB and you make the jump all the way down to L1 and L2 cache, it's obvious that you are used to working on nicely behaved numerical problems. Not everybody is so lucky! And, indeed, a lot of heavy work goes into making those problems so "nice." Differential equations have been the center of the applied math world for over two hundred years, and they have important military and industrial applications. Centuries of brilliant mathematical work, massive investment, decades of clever programming, and all this for problems that naturally lend themselves to partitioning and parallelization anyway. The field is so mature that people who work on these kinds of problems get used to the idea that arbitrarily large datasets can be processed in arbitrarily small chunks just by using common sense and known techniques. In general, this assumption is much too optimistic. There are plenty of problems that are not so nice or not so well understood.

Chipsets

[niceone]

To narrow things down a bit, it's not about Motherboards - it's about chipsets. I've only been looking at Intel (AMD don't have the performance right now for music stuff) - Intel's current P35 and X38 chipsets both support 8GB memory max. If you need more then you have to look at one of the Xeon chipsets: the 5000X workstation chipset is the one to look at if you want to be able to run 2 processors (not sure what the equivalent one is for a single processor) - it supports up to 32GB of memory.

Best motherboards for 64-bit and large memory

[redstar427]

Standard motherboards are typically limited to 8 GB of ram, since they are designed for home users and gamers.
Server/workstation motherboards are the best solution at this time to go beyond this. Most people are only running 32-bit software, with 1-3 GB of ram, so it's not a problem for them.

Currently at work, I use a Tyan Tempest i5000XT (S2696) motherboard, with dual quad-core Intel Xeon cpu's, and 8 GB of ram. I will expand to 16 GB in 2008. This board can upgrade to 32 GB of ram, with 4 GB Dimms, which should be available sometime in the future.

I dual boot with 64-bit Fedora 8 Linux, and 64-bit Windows Vista Ultimate. I run Fedora 8 for all my productive work, and use VMWare with different versions of Linux and Windows, for testing and standard Windows work. I dual boot into 64-bit Vista Ultimate when I need Windows with direct hardware support for some multimedia apps and gaming. 64-bit Vista Ultimate seems a lot more compatible with current apps than 64-bit Windows XP Pro.

For my next home computer, I will choose a similar, but different Tyan Server/workstation motherboard.
The Tyan Tempest i5400PW (S5397) is also a dual socketed motherboard for dual quad-core Xeon cpus.
It has 16 memory sockets and can be expanded up to 128 GB of ram, with future dimms of 8 GB each.
I believe this is the best long-term solution for those that really need a lot of ram, at a reasonable price.
Even with just reasonable priced 2 GB dimms, it can hold 32 GB ram, which is a lot, even for large 64-bit apps.

While $450 for these motherboards is fairly expensive, they provide a lot of value, and good quality desktop motherboards cost $150-400, so it's not really that much more.

Battleship

[eddy]

>Consider swapping to and from a flash device or a series of flash devices.

Good performance. Gets expensive though. $7000 for nine Mtron 16GB Solid State Drives alone, then you need very high end RAID cards to cope with the throughput.

Your AMD Options

[this great guy]

All current socket AM2/AM2+ AMD processors (Opteron 1000 series, Phenom, Athlon X2, etc) support a maximum of four unbuffered DDR2 memory sticks. All current socket F AMD processors (Opteron 8000 and 2000 series) support a maximum of eight registered DDR2 memory sticks. (You can find this info in AMD's public datasheets).

As of today, unbuffered and registered DDR2 memory sticks of 4 GB or more are extremely expensive because the technology cannot be inexpensively mass-produced (yet). Only 2-GB DDR2 sticks can be found at reasonable prices.

For these financial and technical reasons, your are restricted to a total of 8 GB per socket AM2/AM2+ processor, or 16 GB per socket F processor. Therefore the cheapest option for an AMD mobo supporting more than 8 GB of memory is to buy a single socket F model. Newegg sells one for $136 (open box, though). Add a $180 Opteron 2212 processor, $240 for eight 2-GB sticks of registered DDR2-667, and you end up spending only $556 for a dual-core 2.0 GHz 16 GB barebone server assuming you have a chassis and a PSU lying around.

I'll leave other people comment on your Intel options. I am not very familiar with Intel server motherboards.

Re:A different algorithm may be needed

[EsbenMoseHansen]

However, the problem is that he uses Matlab. Perhaps he could get better performance using Octave with Atlas optimization, but in the end, only compiling in C with assembly language optimization will guarantee the best results. I have heard from several people that Matlab has problems when the data sets become large. Well, looking at the price list, switching to octave should buy him a good deal more hardware, even if the performance is the same :)

Re:Audio recording

[gazbo]

I'm not the OP, but you may want to check out the Studio Central forums. Filled with equal measures of twats and great advice. You'll probably want to check out the DAW forums (digital audio workstaion, in case you've not come across that term before).

Don't expect to find very much about Linux though.

Apple Mac Pro or XServe?

[foniksonik]

The latest Mac Pro supports 16 GB of RAM and the latest XServe (a better option IMHO) supports 32GB of RAM.

Mac Pro Specs

XServe Specs

XServe is a quad-core XEON 64bit at 3GHz as is the Mac Pro

They will both run Matlab w/ stunning execution.

Here's a nice case study for the XServe w/ Matlab: Induquímica Laboratorios

Re:Audio recording

[niceone]

I use XP for music stuff (linux for everything else)... I'll probably get an Intel DP35DP motherboard (pretty popular with DAW builders) and Core 2 quad Q6600 (best bang for buck). There's good advice to be had over at the SoundonSound forums (the PC Music board) and there's even a Linux section: http://www.soundonsound.com/forum/postlist.php?Board=LinMus . The other way to go is to look at what the pro DAW builders are using - www.adkproaudio.com are pretty well respected and seem to be on top of the issues.

Good luck!