Slashdot Mirror
Best Motherboard for a Large Memory System?
kimanaw writes "Due to a particular infrastructure need, and increasing OS support for 64 bitness, I'm looking into building a large memory server box (at least 16 gigs, possibly up to 64 gigs, probably config'ed into a big ramdisk). I only need a single CPU, and just minimal disk; most prebuilt systems w/ large memory seem to focus on more CPus and big RAID, all of which (over)inflate the pricetag. I've searched several websites (including Tom's Hardware), and I've googled, but can't seem to locate any commercially available AMD MBs supporting more than 4 sticks of RAM, or 4 gigs. Have any Slashdotters built a big-RAM server? Any pointers, hints, and tips much appreciated."
...but there are 2 GB chips now.
http://ask.slashdot.org/article.pl?sid=05/02/22/09 30238&tid=198&tid=126&tid=4&tid=137
I really like the specs on the Tyan S4882. Quad Opteron, 32GB RAM, and lots of stuff you probably don't want/need. It's a sweet motherboard, nonetheless.
RHCE; are you certified? Karma: ambiguous.
IBM xSeries 445. We have around 50 of these IBM servers at work and they support up to 64 GB of RAM. Any reason why you are trying to build a server out of cheap commodity parts? Save yourself the trouble and buy one of these.
CLICK HERE
fuvoo: watch something
I know you're looking for an MB, not a system, but if you go to http://www.xicomputer.com/ and check out their NetRaider 64 LT. You can custom configure it to have up to 12GB of ram w/ a single Opteron 244.
Have fun!
has several boards, but none of them are single proc systems. If you need a stock Mobo, and want that much memory, you may need to go for a tyan board. I don't know if you can purchase the quad cpu board, use one cpu and all sixteen slots or not, but I think it will work. Obviously if you are going to shell out for that much memory, buying that mobo shouldn't be a big deal, even if you don't use all four cpu slots.
FWIW, that's the only thing I know of.
"We don't know what we are doing, but we are doing it very carefully,..." Wherry, R.J. Personnel Psychology (1995)
RAM DISK. He said it in the question. Ever run a 15GB DB in RAM instead of on a physical disk? It's very fast. There are a ton of options that would be good in RAM. Albeit if he were running a DB application more than one processor would probably be beneficial as well.
Sun Fire V40z: up to 32 gb of memory
8 of 13 people found this answer helpful. Did you?
Get a Sun. You can get very large memory systems. You can go up to 192 gigs of memory. Marc
Tyan Thunder K8QS Pro Quad Socket 940 Motherboard AMD Solution Motherboard - Operton Motherboard: Outpost #: 4311135 http://shop2.outpost.com/product/4311135/
-Chipset: AMD-8131
-Processor Support: Quad Socket 940 for AMD Opteron 800 Series Processors
-Memory Support: Up to 32GB Registered PC3200 DDR
-Expansion Slot: Two 64-bit 66/33 MHz PCI-X, two 64-bit 133/100/66/33 MHz PCI-X, One 32-bit PCI Slots
-Other Features: GigaLAN, Integrated Video, SCSI
if you have the money to fork over, pick one of these up at your local Fry's Electronics or try fry's online at http://www.outpost.com/
There are all sorts of scientific applications where you need as much RAM as possible. One where the amount of RAM is almost the only limiting factor is model checking. It refers to a systematic exploration of an entire system. Even though a lot of techniques have been invented to reduce the complexity of the problem, the problem itself is exponentially complex in the size of a system. A 16-bit counter has 65536 states. Just imagine how many states a simple ALU or an entire CPU can have. This shown you that there is no limit to the amount of RAM one can use for this.
All major players in the CPU industry use this technology, so I am sure AMD has some systems like that in their design labs...
Model checking is also used in some software projects where the complexity of the system is "manageable", such as for device drivers or embedded systems.
Other scientific uses needing a lot of RAM are simulations. The level of detail simulations can manage is mostly limited by the amount of RAM (and of course also CPU speed). Again, the sky is the limit for how much RAM one could use if it was available... so you see it is not problem to fill 32 GB of RAM in a couple of seconds with such simulations.
I've always read it's inadvisable even if one has a lot of RAM. Here's some quotes from this page:
Can the Virtual Memory be turned off on a really large machine?
Strictly speaking Virtual Memory is always in operation and cannot be "turned off." What is meant by such wording is "set the system to use no page file space at all."
Doing this would waste a lot of the RAM. The reason is that when programs ask for an allocation of Virtual memory space, they may ask for a great deal more than they ever actually bring into use -- the total may easily run to hundreds of megabytes. These addresses have to be assigned to somewhere by the system. If there is a page file available, the system can assign them to it -- if there is not, they have to be assigned to RAM, locking it out from any actual use.
--
Why is there so little Free RAM?
Windows will always try to find some use for all of RAM -- even a trivial one. If nothing else it will retain code of programs in RAM after they exit, in case they are needed again. Anything left over will be used to cache further files -- just in case they are needed. But these uses will be dropped instantly should some other use come along. Thus there should rarely be any significant amount of RAM 'free'. That term is a misnomer -- it ought to be 'RAM for which Windows can currently find no possible use'. The adage is: 'Free RAM is wasted RAM'. Programs that purport to 'manage' or 'free up' RAM are pandering to a delusion that only such 'Free' RAM is available for fresh uses. That is not true, and these programs often result in reduced performance and may result in run-away growth of the page file.
I'm on a road shaped like a figure eight; I'm going nowhere but I'm guaranteed to be late.
What you're interested in is solid-state storage, which you can google. The main problem with your plan is, if the power goes out, you've lost everything. Even trying to flush everything to the hard drive will take dangerously long. (Writing 15GB to harddisk is no small task!) The solid state storage machines out there are basically computers with lots of RAM, but the RAM is backed by battery, so even if it's unplugged, it stores its state, and then backs up to hard disk, automatically. Trying to get a PC to do this for you would be much trouble. Remember: UPSing a whole computer is a lot different than keeping RAM charged! :)
Digital Sailor
The only way to get anywhere close to 64GB is by going multi processor. I have 2 Thunder K8S Pro (S2882) motherboard with 16GB of RAM each, running Postgres 8 on FreeBSD 5.3 in production. These systems are dual opterons. They have 2GB sticks. I also have a test system with a Tyan Thunder K8QS Pro (S4882), this one is a quad opteron and supports upto 32GB with 2GB sticks. The only way I know to get 64GB with 2GB sticks is with an IWILL H8501, that's an 8-way opteron barebone system.
I think you can get 4GB stiks now, but they are very expensive still.
The ASUS K8N-DL is a dual Opteron motherboard which officially supports up to 24GB RAM in 6 DIMMs. It costs only about $300, too. Not sure of current availability, it's pretty new.
Management\DisablePagingExecutive to 1.
That only stops Windows from paging out parts of the OS itself (like the kernel and currently-idle device drivers).
On XP, you can just set the pagefile size to zero. And yes, it does work, in the sense that it stops Windows from hitting the disk twice per second even when doing nothing at all.
On 2000, you need to assign the pagefile to a RAMdrive (and one that supports NTFS and doesn't identify itself as RAM, since Windows won't normally let you put the pagefile on a RAMdrive).
Finally, for anyone disabling paging - Also disable memory dumps and automatic reboot on bluescreens, or you will regret it.
Otherwise, it works just fine, regardless of what the naysayers and MS Knowledgebase fanboys might say. Quite a boost in performance, too.
In order to populate all the RAM slots in an Opteron system, you have to populate all the processor sockets as well. And a 4-socket system requires expensive Opteron 8xx processors.
Instead, consider an IBM x366 or an HP DL580; either one can be configured with 32GB RAM and 1 processor for under $30K.
Doing this would waste a lot of the RAM. The reason is that when programs ask for an allocation of Virtual memory space, they may ask for a great deal more than they ever actually bring into use -- the total may easily run to hundreds of megabytes. These addresses have to be assigned to somewhere by the system. If there is a page file available, the system can assign them to it -- if there is not, they have to be assigned to RAM, locking it out from any actual use.
Not true. When a program allocates a large chunk of ram, all it gets are the page table entries. Only when it actually writes to a page does the ram get used. This means that I can allocate 1.5G of ram and scribble in half of it and only be using 750M
"We returned the General to El Salvador, or maybe Guatemala, it's difficult to tell from 10,000 feet"
There are a few possible solutions you might want to look at for a big-RAM server. Now, if you really want 64GB and AMD Opteron processors than you really only have one choice, the HP Proliant DL585. That's the only Opteron solution that I know of which supports 64GB of memory.
If you can get by with a bit less memory then you have some other solutions. Tyan carries quite a number of boards with varying capabilities. The trouble here is that the Opteron processors are limited to 8GB of memory per processor, so to get 16GB you're going to be looking at a dual-processor board (quad processor for 32GB). Since the memory controller is right on the CPU with the Opteron you will actually need a second processor in the socket to use this memory.
For this reason, you might actually want to consider one of Intel's new 64-bit Xeon chips. I know that Supermicro offers some boards that can handle up to 32GB with only a single Xeon processor. Something like the X6DHE-XB seems like it might fit you're bill reasonable well. Fairly inexpensive to get you up to 16GB of memory, though going to 32GB is quite expensive. Crucial has a list of compatible memory for this board, including some 4GB modules.
Of course, if you're not limited to x86 systems then there are other solutions that would work. You could get something like an IBM Power system or Sun UltraSparc system with pretty much any amount of memory you need (or can afford).
Caching is often a bigger win than RAM Disk.
:-) But I know these can go up to 1/2 TB, addressable by a single application. And you can't beat their IO.
Profile your application before making such committments. You might be surprised at what you find.
What's interesting here is how securely "in the box" people are thinking. They are on a do-or-freaking-die mission to press a consumer architecture into service as a mid-range engineering system.
My shop uses Sun hardware for midsize applications like this, but we also run some Alpha systems (Both older DEC and newer HP).
We have a couple of Sun E20K but I don't know how they are configured exactly (I don't get to touch stuff that belongs to financial depts, nor would I want to
But somebody who is thinking in terms of "What motherboard should I buy?" probably isn't prepared for the sticker prices of SunFire servers, EMC storage, etc.
When a skunkworks shop needs some large scale system on a budget approaching zero dollars, some creativity is needed. So we get into storage solutions like consumer RAID, and we hear questions like the OP, "what's the largest memory configuration on a commodity 64-bit microcomputer?"
Dual Xeon boards that take 32GB RAM are common enough, but I think that's 16GB per processor and I have no idea how it's addressed or whether any given application knows how to address it.
32GB in premimum RAM will run over $20 grand. Nothing compared to the half-million for an E20K, of course.
-fb Everything not expressly forbidden is now mandatory.
As to the purpose of the box, I can't say too much, except its intended as a sortof main-memory database cache solution (ala TimesTen ) to serve a potentially very large user base, with a lot of cached images associated with the cached data.
I also considered a piecemeal commodity approach, e.g., filling racks with cheap 4 gig RAM 1U's running Linux, but then I'd have to come up with s/w that can coordinate those systems (basically, a head-end to hash the requests to the right 1U). memcached looks intriguing, but I'm still concerned about interconnect latencies.
Based on my cocktail napkin estimates, the h/w cost of a big RAM system (if it can be got) would probably be about 2x the cost of the piecemeal system, but hopefully signifcantly reduce interconnect latency, and use simpler s/w (assuming I can get Linux to config a ramdisk that big).
Since latency is the prime concern, solidstate disks using the usual HD interconnects just didn't seem an acceptable solution, esp since the cost reduction doesn't seem that significant.
As for failure recovery, my thoughts were just a couple fast HDs that would ping-pong taking snapshots of the ramdisk; hopefully, the ramdisk recovery would never lose more than an hour's data (the lost data wouldn't be critical, and would be recoverable from other sources...its really the delivery speed thats essential)
But I may need to go back to my napkin and do some more queueing analysis to see if there are better commodity solutions.
007: "Who are you?"
Pussy: "My name is Pussy Galore."
007: "I must be dreaming..."
Windows XP doesn't swap, it uses a page file, which preemptively makes copies of the least used pages in your ram to your disk so that if there is an unexpected jump in memory usage, it can wipe those pages from memory quickly and just put them back from disk when they are needed. There really isn't a performance hit since it's a pretty much passive system in reference to RAM writes. Adrian's Rojak Pot has a great guide on Windows virtual memory at http://www.rojakpot.com/default.aspx?location=3&va r1=143&var2=0
-Julius
You can configure an Apple Xserve this way, with either one or two CPUs. Not an AMD board, but it is 64bit, and you can still put Linux on it if you like.
$14,599--One CPU, 80GB HD, CD-RW, no video support, but 16GB of RAM and an unlimited user Server OS. You can do better if you purchase through the Education or Government channels, and you can do better if you purchase the 2GB DIMMs elsewhere.
Although I actually couldn't find 2GB DIMMs at the popular aftermarket places, but they are now available from Apple direct (just be sure to get the Xserve with at least one 2GB DIMM, to be sure it has support for 2GB DIMMs on the MLB).
--
$tar -xvf