Best Motherboard for a Large Memory System?

kimanaw writes "Due to a particular infrastructure need, and increasing OS support for 64 bitness, I'm looking into building a large memory server box (at least 16 gigs, possibly up to 64 gigs, probably config'ed into a big ramdisk). I only need a single CPU, and just minimal disk; most prebuilt systems w/ large memory seem to focus on more CPus and big RAID, all of which (over)inflate the pricetag. I've searched several websites (including Tom's Hardware), and I've googled, but can't seem to locate any commercially available AMD MBs supporting more than 4 sticks of RAM, or 4 gigs. Have any Slashdotters built a big-RAM server? Any pointers, hints, and tips much appreciated."

Comments

Out of curiosity

[elid]

what do you need so much RAM for in an individual single-processor server?

Re:Out of curiosity

[Tsiangkun]

But, He wouldn't be limited by the amount of RAM, instead he could have up to $FILESYSTEM_LIMIT amount of space. Pretty much any mother board will work with it, and he doesn't need to go 64 bit unless he has to for other reasons.

Re:Out of curiosity

[kpharmer]

Actually, a well-tuned high-end database typically doesn't need to be completely in-memory to get 98% of the speed of one on a ramdisk.

For example, let's say that you've got 15 gbytes of raw data on db2/oracle/informix, as well as 4-8 gbytes of memory.

First step is storage: don't use raid-5 (requires extra writes), instead you'll want something like raid-10. Then ideally get 28-56 or so drives, and you can spread data, indexes, logs, and tempspace across all drives. Obviously 15k rpm ultra320 drives with a caching controller would be best (if you can't afford fibre).

Next make sure that your memory is tuned right - multiple gbytes set aside for buffering, sorting, asynchronous agents, etc. With these databases it's easy to ensure that all of your indexes are always in memory, that all writes to disk are asynchronous, and that most sorts are in-memory. The buffering should ensure that 98% or so of your disk access actually hits the bufer instead of the disk.

Given the above scenario a database running on a ramdisk could be faster - what? 0-5%? And in return for that possible speed increase it'll cost more and lack the reliability of the disk-based database. Probably not a good deal.

Re:Out of curiosity

[Aeiri]

Not a single sysadmin who actually deserves the title is going to run a db out of system memory.

There is a way (haven't done this myself) to send out the update/insert/delete to the harddisk immediately after it applies to the RAM, but the application responds to the webserver or whatever is calling it immediately after it applies to RAM. This creates a lightning fast database, with storage on a hard drive as well in case of a server crash.

Re:Out of curiosity

[letxa2000]

Disk times would probably be sufficient for the type of work that a single processor system could handle.

Disk times would be "sufficient" because most people define what a single processor system could handle based on a disk-based, OS-based system. A RAM-based system is entirely different.

You'd be absolutely amazed at how much computing power we already have available on our single-processor systems. Get rid of Windows and, yes, even Linux and the sheer speed and power available is quite mind-boggling.

In most cases, the practical need for additional processors is not so much a limitation of the processor--which is lightning fast already--but of the operating systems that just consume so much resources to do what they do. And that's not a slam. The OS serves a very necessary purpose. But do some work on embedded systems and you'll realize just how much power there really is under the hood of even a consumer PC.

And when I say embedded, I don't mean the new definition of "embedded" that seems to be putting Windows or Linux in a small form factor but, otherwise, just a normal PC. Embedded used to mean getting into the nuts and bolts with a program that does its job and nothing more. No OS overhead, just pure code and hardware. That's embedded. And when you work at that level for awhile, you really begin to appreciate how much of a hit performance takes for the flexibility of running under an OS.

2GB DIMMs and Tyan

[Jeffrey Baker]

You might want to get 2GB DIMMs. I have never seen any that worked reliably, but I also haven't looked at them in about a year. Get some and burn them in with memtest and see what happens. As for 8-DIMM motherboards, check out something from Tyan. Since you say you don't need that much CPU power, you could load up a Tyan Thunder board with two Opteron 240 CPUs for rather little money (or, a very small amount compared to what the 16GB of memory is going to cost you). If you seriously need 64GB of memory, the only PC system I've seen with that much are the HP ProLiant DL585, DL740, and DL760 machines. These are very expensive, especially once loaded with 64GB of memory.

Re:I don't know of such things...

[Bradee-oh!]

A few points -

-The submitter didn't explicitly state it but I'm guessing since he did mention the popularity/existence of 64 bit processors that he intends on using one.

-32bit CPUs are not limited to 4gigs of ram. Only 4 gigs directly addressable at a time, yes. But google "page address extension" to find all you'd ever want to know about it. The Linux kernel has supported 64gigs on 32bit processors since the late 2.5.x's

But, of course, I agree 64 bit is the way to go because it avoids all that address-table-translation nastiness but the submitter's problem remains.

Is there a motherboard on the consumer market within a reasonable price range ($1000, maybe?) that can actually, physically take 64+ gigs of ram?

Tyan anyone!

[mwfolsom]

What's the deal?

Tyan currently lists 6 2 processor opteron boards that hold 16 GB and 1 4p board that holds 32 GB - actaully got one of those the other day. Very nice!

Iwill has one 8p system that holds 64GB.

See:
http://www.tyan.com/products/html/opteron. html

IBM eServer 326

[Stack_13]

If 16 GB is enough, IBM has eServer 326 is just about expandable to that amount.

Strangely, 16 GB memory is only available in a dual-CPU configuration. DIMMs 5-8 can only be used if another CPU is installed.

But since the extra Opteron CPU costs about the same as a 2 GB memory module, that shouldn't be much of a concern. Price is in the vicinity of $15K with 2 x Opteron 248 and 16 GB RAM.

MSI dual opteron board, 12 slots

[Anonymous Coward]

MSI make a 12 slot dual opteron board, needs ECC tho. (MSI K8D Master3-FS Dual AMD® Opteron(TM) E-ATX Mainboard) In South Africa it goes for the equivalent of about USD 525, so it'd probably be a bit cheaper elsewhere

A couple options

[Anonymous Coward]

Your best bet might be a dual CPU system with slow/cheap CPU's. (Opteron 240's). The dual Opteron boards out there are set up with two banks of RAM, each dedicated to a CPU, so you habe to fill both CPU sockets to make use of all the DIMMS.

There are a couple dual processor motherboards that I know of out there with 12 DIMM slots. That would get you up to 24GB per machine using "fairly inexpensive" RAM. The 2GB DIMMS these days are only around $600-800 bucks a piece, so it's feasable to load up one of these boxes for around ten grand. 4GB DIMMS do exist but AFAIK they're 3-4x the cost of a 2GB DIMM.

You might scope out HPC Systems (www.hpcsystems.com), they sell boxes pre-integrated with large memory footprints. (not an advertisement, but I've purchased gear from them before).

-A

Re:I don't know of such things...

[fishbowl]

"Is there a motherboard on the consumer market within a reasonable price range ($1000, maybe?) that can actually, physically take 64+ gigs of ram?"

There are lots of dual Xeon boards that take 64GB.
The trick is finding the chips. A $375 board that needs two $300 chips is one thing. $20,000 in RAM chips takes it to the next level...

A G5 will hold at least 16G.

[JackAxe]

The 2G sticks have been out for about a year now. 4G sticks are what's new.

Apple doesn't list higher then 8G, because that is what they tested. 4G sticks will most likely also work. There is really no reason why they wouldn't.

"All that RAM." "far More data than you have resources to process." *LOL* It's safe to say that you don't work with video, nor as an artist. Are you working on a 386SX. :D *Kidding* I could be wrong though. :)

I have over a Terabyte of HD and my system handles it just fine. My 7 to 8 minute video jobs export single video files that average about 12Gigs. If I had 16GB of RAM and AEPro supported it, even with that Much RAM I would still need to "purge it all" every so often. I also generally run most of my pro apps at the same time since I need to jump between them. When they support 64-bit memory, my 5 G of RAM will not be enough. I'm upgrading to Tiger since it allows 64-bit memory support for apps now and CS2 will take advantage of that, at least on the Mac. I'll be allocating 4 gigs of RAM into it, since some of the poster illustrations I work on have exceeded over 3.5G in RAM (And since PS could only see 2G, it had to rely on scratch disk(s) to make up the difference.). That would only leave 1G for everything else, so needless to say I'll also be upgrading my RAM when I buy Tiger to more then 8G.

What I work with is peanuts compared to what a friend of mine does. He works with satellite imagery and a small plot of land can easily exceed over a Terabyte for one image.

Anyways, 16G of RAM is nothing with current comps and I bet that in about 4 years you'll probably have at least 6Gs in your system. Go back just a few years and 1G of RAM was considered more then most peeps would ever need or use.

Re:Oh shit, wrong account

[slashdot_commentator]

One of my favorite geek memories of childhood was listening to Radio Moscow on my shortwave radio. (I think I was 13 years old back then.) Initially, there was the childish visceral thrill of doing something "forbidden". (There was a COLD WAR ON, back then...) But really, the greatest aspect of the broadcasts were how the "Commies" were unintentionally being hilarious in some of their less defendable points of view. It would be like listening to Rush Limbaugh because you got bellylaughs from mocking his opinions. (But I personally don't find stupidity or his propaganda all that amusing...).

My most notable memory was how the Radio Moscow program was sponsoring an essay contest to win a free trip to the USSR. I don't remember the specifics, but they had the winner (American or Canadian) read it over the air. It was a foreign accented individual spieling communist ideology in a dead monotone right out of a bad Manchurian Candidate movie clone for like 15-20 minutes.

Basically, it could only be one of two possibilities. They could have been too broke or cheap to sponsor a real contest, and got it some academic ideologue to write it, and foreign embassy staff to read it. Mind you, you're not going to convince the audience there were loyal communists in the US if they speak in a slight foreign accent. The other possibility was that it actually was a North American, but hooboy, selecting a mumbling, monotonic guy out of a psychiatric ward would hardly encourage people to embrace communism.

Anyway, I found that post hilarious. The moderator is an obvious moron for labeling it a troll. I'm curious as to what traumatic brush with communists causes you to lash out in parody. (And *bang* is a nice touch.) Keep alive the Revolution, Comrade!

Re:Any reason why you are building it yourself?

[maraist]

Plus what sane company is going to go with some homebuilt machine that, presumably, is important to the business when product is out there?

Start ups, where dollars matter.. What I've seen people linking to are $3,000 machines at their cheapest.. Then the IBM and other solutions are probably in the tens of thousands of dollars.. Yes, what is your time worth? If you work for a fortune 500 company, then it and the consequences are worth a few measly 10 grand. But if as I suspect the auther DOESN't need CPU horsepower, DOESn't need high-end disk arrays.. They just need memory, then you do the math. The likelyhood of the project needing a 20,000% increase in price ($500 to $10,000) to save a few man-hours...

I quote $500, because a premium MB + 1G memory + 3000+ CPU can be had for much under $500. If you want a rack, sure you're going to pay more, but considering that he was bitching about now wanting a multi-CPU server, I seriously doubt he was looking at racks.

And why would you want memory instead of hardware? Because there are an increasing number of applications which are memory-space intensive, not CPU or even disk intensive.. Running VMware sessions, user-mode-linux, running many-many java servlet engines (each JVM hapilly can consume 1Gig of memory), caching your database. These are various instances that I personally run into on a regular basis where memory is NEVER enough. Especially when I hear people claim "memory is cheap".. No it isn't! Not when you hit artificial limits like 4 DIMM's per AMD64 CPU. It's a fixed resource where a programming environment which assumes an infinite amount of available memory can in certain configurations and situations become a huge bottle-neck..

The most reasonable solution is to purchase multile machines to divide the work.. But this doesn't scale too well as your administrative over-head (as well as cost of UPS's, power/heat constraints, shelf/rack space, etc).

One of the points of going to a 64bit architecture was to remove the architectural limit on addressing. Well, much like the early moterola's, you can address significantly higher than the specific architecture can facilitate.. This is to be expected, but to realistically have an upper bound of 16 chips or 64Gb (assuming 4Gig chips, which I don't even know if exist; certainly not whole-sale) is just kind of sad (considering 1Gig servers are the norm, and 64bitness is well over a decade old)

A 2G chip costs at least $200 (for a crappy server-need-not-apply brand). Half gig's start at $30 (or $120 for the 2Gig counter-part). Server-worthy half-gigs start at $60. Often, however, vendors that sell memory with their system charge enormous premiums (nearly a thousand dollars) for upgrading to the larger chip sizes. If instead of being limited to 4 slots / CPU the architecture facilitated 8 or 16 slots, then vendors wouldn't have as much justification for uprading to the more expensive DIMMs and thereby passing the premium on.

It's simply impractical to afford purchasing a system with 16Gig of memory.. There are many other cheaper solutions which provide a significantly faster machine (including using a swap-file and RAIDing the hell out of it). But this is insane, there is no good reason why a larger memory solution should cost so much as to cause system administrators and developers to have to spend hundreds of thousands of development/deployment dollars working around such limits.

I recognize that you can't just put more memory slots on a BUS, but I don't see why the x86-64 architecture didn't facilitate fully banking the memory out into 4 parallel busses of 4 slots each.. To my knowledge, the most that is supported are two banks of two slots each.

The AMD64, while a GREAT chip series, seems just a little on the cheap side to me.