Slashdot Mirror
Best Motherboard for a Large Memory System?
kimanaw writes "Due to a particular infrastructure need, and increasing OS support for 64 bitness, I'm looking into building a large memory server box (at least 16 gigs, possibly up to 64 gigs, probably config'ed into a big ramdisk). I only need a single CPU, and just minimal disk; most prebuilt systems w/ large memory seem to focus on more CPus and big RAID, all of which (over)inflate the pricetag. I've searched several websites (including Tom's Hardware), and I've googled, but can't seem to locate any commercially available AMD MBs supporting more than 4 sticks of RAM, or 4 gigs. Have any Slashdotters built a big-RAM server? Any pointers, hints, and tips much appreciated."
...but there are 2 GB chips now.
http://ask.slashdot.org/article.pl?sid=05/02/22/09 30238&tid=198&tid=126&tid=4&tid=137
I really like the specs on the Tyan S4882. Quad Opteron, 32GB RAM, and lots of stuff you probably don't want/need. It's a sweet motherboard, nonetheless.
RHCE; are you certified? Karma: ambiguous.
IBM xSeries 445. We have around 50 of these IBM servers at work and they support up to 64 GB of RAM. Any reason why you are trying to build a server out of cheap commodity parts? Save yourself the trouble and buy one of these.
You might want to get 2GB DIMMs. I have never seen any that worked reliably, but I also haven't looked at them in about a year. Get some and burn them in with memtest and see what happens. As for 8-DIMM motherboards, check out something from Tyan. Since you say you don't need that much CPU power, you could load up a Tyan Thunder board with two Opteron 240 CPUs for rather little money (or, a very small amount compared to what the 16GB of memory is going to cost you). If you seriously need 64GB of memory, the only PC system I've seen with that much are the HP ProLiant DL585, DL740, and DL760 machines. These are very expensive, especially once loaded with 64GB of memory.
CLICK HERE
fuvoo: watch something
I know you're looking for an MB, not a system, but if you go to http://www.xicomputer.com/ and check out their NetRaider 64 LT. You can custom configure it to have up to 12GB of ram w/ a single Opteron 244.
Have fun!
has several boards, but none of them are single proc systems. If you need a stock Mobo, and want that much memory, you may need to go for a tyan board. I don't know if you can purchase the quad cpu board, use one cpu and all sixteen slots or not, but I think it will work. Obviously if you are going to shell out for that much memory, buying that mobo shouldn't be a big deal, even if you don't use all four cpu slots.
FWIW, that's the only thing I know of.
"We don't know what we are doing, but we are doing it very carefully,..." Wherry, R.J. Personnel Psychology (1995)
RAM DISK. He said it in the question. Ever run a 15GB DB in RAM instead of on a physical disk? It's very fast. There are a ton of options that would be good in RAM. Albeit if he were running a DB application more than one processor would probably be beneficial as well.
You ough to know that 500megs of ram is enough for everyone.
Sun Fire V40z: up to 32 gb of memory
8 of 13 people found this answer helpful. Did you?
Get a Sun. You can get very large memory systems. You can go up to 192 gigs of memory. Marc
Tyan Thunder K8QS Pro Quad Socket 940 Motherboard AMD Solution Motherboard - Operton Motherboard: Outpost #: 4311135 http://shop2.outpost.com/product/4311135/
-Chipset: AMD-8131
-Processor Support: Quad Socket 940 for AMD Opteron 800 Series Processors
-Memory Support: Up to 32GB Registered PC3200 DDR
-Expansion Slot: Two 64-bit 66/33 MHz PCI-X, two 64-bit 133/100/66/33 MHz PCI-X, One 32-bit PCI Slots
-Other Features: GigaLAN, Integrated Video, SCSI
if you have the money to fork over, pick one of these up at your local Fry's Electronics or try fry's online at http://www.outpost.com/
Any good to you? You can easily get up to 128Gb in a single drive, 3GB/s bandwidth, fits in a drive bay. Who cares what "RAM" the motherboard supports? Just a though, don't know if it would be any good for your environment.
*sigh* I think I'm getting tired of this....
*bang!*
The biggest board I know of is the Thunder K8QS Pro (S4882). With 16 DIMM slots it supports 32 GB of RAM. But it is a quad processor board.
Most of the rest of Tyan's Opteron server boards do support 16 GB. Again they are dual processor.
Tyan's boards will run with a single processor installed, but only 4 DIMM slots (1 bank) will be active. All processors can see all the RAM installed, but because the memory controller is integrated into the CPU, the CPU must first be installed. Then it can pass access to the RAM to the other CPUs. Local RAM always being the fastest. (Linux with Opteron NUMA support tries to keep memory associated with the task running on a specific CPU local to that processor.)
Ah you've returned. Now be loyal troll-pet.
*pets troll on head*.
Tom
Someday, I'll have a real sig.
Insightful my ass. Some of us have real JOBS that require Windows, and saying "Sorry, I missed the deadline because of problems with Wine" are a fast way to get yourself fired.
He asked about how to fix it on Windows. How about I respond to every problem with Gnome and KDE or Linux with "Maybe you should switch to a supported platform like Windows"? We'll see how often that tripe gets modded "Insightful".
There are all sorts of scientific applications where you need as much RAM as possible. One where the amount of RAM is almost the only limiting factor is model checking. It refers to a systematic exploration of an entire system. Even though a lot of techniques have been invented to reduce the complexity of the problem, the problem itself is exponentially complex in the size of a system. A 16-bit counter has 65536 states. Just imagine how many states a simple ALU or an entire CPU can have. This shown you that there is no limit to the amount of RAM one can use for this.
All major players in the CPU industry use this technology, so I am sure AMD has some systems like that in their design labs...
Model checking is also used in some software projects where the complexity of the system is "manageable", such as for device drivers or embedded systems.
Other scientific uses needing a lot of RAM are simulations. The level of detail simulations can manage is mostly limited by the amount of RAM (and of course also CPU speed). Again, the sky is the limit for how much RAM one could use if it was available... so you see it is not problem to fill 32 GB of RAM in a couple of seconds with such simulations.
I've always read it's inadvisable even if one has a lot of RAM. Here's some quotes from this page:
Can the Virtual Memory be turned off on a really large machine?
Strictly speaking Virtual Memory is always in operation and cannot be "turned off." What is meant by such wording is "set the system to use no page file space at all."
Doing this would waste a lot of the RAM. The reason is that when programs ask for an allocation of Virtual memory space, they may ask for a great deal more than they ever actually bring into use -- the total may easily run to hundreds of megabytes. These addresses have to be assigned to somewhere by the system. If there is a page file available, the system can assign them to it -- if there is not, they have to be assigned to RAM, locking it out from any actual use.
--
Why is there so little Free RAM?
Windows will always try to find some use for all of RAM -- even a trivial one. If nothing else it will retain code of programs in RAM after they exit, in case they are needed again. Anything left over will be used to cache further files -- just in case they are needed. But these uses will be dropped instantly should some other use come along. Thus there should rarely be any significant amount of RAM 'free'. That term is a misnomer -- it ought to be 'RAM for which Windows can currently find no possible use'. The adage is: 'Free RAM is wasted RAM'. Programs that purport to 'manage' or 'free up' RAM are pandering to a delusion that only such 'Free' RAM is available for fresh uses. That is not true, and these programs often result in reduced performance and may result in run-away growth of the page file.
I'm on a road shaped like a figure eight; I'm going nowhere but I'm guaranteed to be late.
You will likely be better served if you just use a solid state disk for this.
What you're interested in is solid-state storage, which you can google. The main problem with your plan is, if the power goes out, you've lost everything. Even trying to flush everything to the hard drive will take dangerously long. (Writing 15GB to harddisk is no small task!) The solid state storage machines out there are basically computers with lots of RAM, but the RAM is backed by battery, so even if it's unplugged, it stores its state, and then backs up to hard disk, automatically. Trying to get a PC to do this for you would be much trouble. Remember: UPSing a whole computer is a lot different than keeping RAM charged! :)
Digital Sailor
First, I would recommend going with a server vendor. I honestly do not mean any offense by this, but if you are looking at places like Tom's Hardware for recommendations (a website which is frequently incompetent even at reviewing l33t g4m3r d00ds hardware, let alone server-grade hardware), you are probably not qualified to build a system which would actually need 16GB of RAM (e.g. a corporate server which must be relied upon). I do not know what the system will be used for, but if it is for a many-person organization, my recommendation stands, and if it is just for you and some friends, 16GB of RAM is almost always going to be absurd overkill.
Now that that's out of the way, Tyan has several dual Opteron boards which support 8 DIMMs. Look at their Thunder line, and put just one Opteron in them, and 2GB DIMM modules.
Computer Science is no more about computers than astronomy is about telescopes. --E. W. Dijkstra
The only way to get anywhere close to 64GB is by going multi processor. I have 2 Thunder K8S Pro (S2882) motherboard with 16GB of RAM each, running Postgres 8 on FreeBSD 5.3 in production. These systems are dual opterons. They have 2GB sticks. I also have a test system with a Tyan Thunder K8QS Pro (S4882), this one is a quad opteron and supports upto 32GB with 2GB sticks. The only way I know to get 64GB with 2GB sticks is with an IWILL H8501, that's an 8-way opteron barebone system.
I think you can get 4GB stiks now, but they are very expensive still.
Rather than saying, "I need this to solve my problem, and I'm not saying what my problem is!" it is better to say, "Here is my problem, I am thinking of doing this. How can I do that, or are there better solutions that I haven't thought of?" He'll get better answers that way.
If he doesn't know where to purchase a large memory system then it is possible that he doesn't really know if that is what he wants.
Lasers Controlled Games!
The ASUS K8N-DL is a dual Opteron motherboard which officially supports up to 24GB RAM in 6 DIMMs. It costs only about $300, too. Not sure of current availability, it's pretty new.
Management\DisablePagingExecutive to 1.
That only stops Windows from paging out parts of the OS itself (like the kernel and currently-idle device drivers).
On XP, you can just set the pagefile size to zero. And yes, it does work, in the sense that it stops Windows from hitting the disk twice per second even when doing nothing at all.
On 2000, you need to assign the pagefile to a RAMdrive (and one that supports NTFS and doesn't identify itself as RAM, since Windows won't normally let you put the pagefile on a RAMdrive).
Finally, for anyone disabling paging - Also disable memory dumps and automatic reboot on bluescreens, or you will regret it.
Otherwise, it works just fine, regardless of what the naysayers and MS Knowledgebase fanboys might say. Quite a boost in performance, too.
In order to populate all the RAM slots in an Opteron system, you have to populate all the processor sockets as well. And a 4-socket system requires expensive Opteron 8xx processors.
Instead, consider an IBM x366 or an HP DL580; either one can be configured with 32GB RAM and 1 processor for under $30K.
Actually, a well-tuned high-end database typically doesn't need to be completely in-memory to get 98% of the speed of one on a ramdisk.
For example, let's say that you've got 15 gbytes of raw data on db2/oracle/informix, as well as 4-8 gbytes of memory.
First step is storage: don't use raid-5 (requires extra writes), instead you'll want something like raid-10. Then ideally get 28-56 or so drives, and you can spread data, indexes, logs, and tempspace across all drives. Obviously 15k rpm ultra320 drives with a caching controller would be best (if you can't afford fibre).
Next make sure that your memory is tuned right - multiple gbytes set aside for buffering, sorting, asynchronous agents, etc. With these databases it's easy to ensure that all of your indexes are always in memory, that all writes to disk are asynchronous, and that most sorts are in-memory. The buffering should ensure that 98% or so of your disk access actually hits the bufer instead of the disk.
Given the above scenario a database running on a ramdisk could be faster - what? 0-5%? And in return for that possible speed increase it'll cost more and lack the reliability of the disk-based database. Probably not a good deal.
Doing this would waste a lot of the RAM. The reason is that when programs ask for an allocation of Virtual memory space, they may ask for a great deal more than they ever actually bring into use -- the total may easily run to hundreds of megabytes. These addresses have to be assigned to somewhere by the system. If there is a page file available, the system can assign them to it -- if there is not, they have to be assigned to RAM, locking it out from any actual use.
Not true. When a program allocates a large chunk of ram, all it gets are the page table entries. Only when it actually writes to a page does the ram get used. This means that I can allocate 1.5G of ram and scribble in half of it and only be using 750M
"We returned the General to El Salvador, or maybe Guatemala, it's difficult to tell from 10,000 feet"
There are a few possible solutions you might want to look at for a big-RAM server. Now, if you really want 64GB and AMD Opteron processors than you really only have one choice, the HP Proliant DL585. That's the only Opteron solution that I know of which supports 64GB of memory.
If you can get by with a bit less memory then you have some other solutions. Tyan carries quite a number of boards with varying capabilities. The trouble here is that the Opteron processors are limited to 8GB of memory per processor, so to get 16GB you're going to be looking at a dual-processor board (quad processor for 32GB). Since the memory controller is right on the CPU with the Opteron you will actually need a second processor in the socket to use this memory.
For this reason, you might actually want to consider one of Intel's new 64-bit Xeon chips. I know that Supermicro offers some boards that can handle up to 32GB with only a single Xeon processor. Something like the X6DHE-XB seems like it might fit you're bill reasonable well. Fairly inexpensive to get you up to 16GB of memory, though going to 32GB is quite expensive. Crucial has a list of compatible memory for this board, including some 4GB modules.
Of course, if you're not limited to x86 systems then there are other solutions that would work. You could get something like an IBM Power system or Sun UltraSparc system with pretty much any amount of memory you need (or can afford).
Caching is often a bigger win than RAM Disk.
:-) But I know these can go up to 1/2 TB, addressable by a single application. And you can't beat their IO.
Profile your application before making such committments. You might be surprised at what you find.
What's interesting here is how securely "in the box" people are thinking. They are on a do-or-freaking-die mission to press a consumer architecture into service as a mid-range engineering system.
My shop uses Sun hardware for midsize applications like this, but we also run some Alpha systems (Both older DEC and newer HP).
We have a couple of Sun E20K but I don't know how they are configured exactly (I don't get to touch stuff that belongs to financial depts, nor would I want to
But somebody who is thinking in terms of "What motherboard should I buy?" probably isn't prepared for the sticker prices of SunFire servers, EMC storage, etc.
When a skunkworks shop needs some large scale system on a budget approaching zero dollars, some creativity is needed. So we get into storage solutions like consumer RAID, and we hear questions like the OP, "what's the largest memory configuration on a commodity 64-bit microcomputer?"
Dual Xeon boards that take 32GB RAM are common enough, but I think that's 16GB per processor and I have no idea how it's addressed or whether any given application knows how to address it.
32GB in premimum RAM will run over $20 grand. Nothing compared to the half-million for an E20K, of course.
-fb Everything not expressly forbidden is now mandatory.
I'm curious how a device like this would get 3GB/s bandwidth - what kind of bus is it using? It's certainly way past PCI. Perhaps PCI-X, or plugging into AGP or something?
Another possibility, if cost is more of a problem, and bus speed is less of a problem, is to network a couple of motherboards together, with as much RAM as possible on each of them, and either GigE or Firewire.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
As to the purpose of the box, I can't say too much, except its intended as a sortof main-memory database cache solution (ala TimesTen ) to serve a potentially very large user base, with a lot of cached images associated with the cached data.
I also considered a piecemeal commodity approach, e.g., filling racks with cheap 4 gig RAM 1U's running Linux, but then I'd have to come up with s/w that can coordinate those systems (basically, a head-end to hash the requests to the right 1U). memcached looks intriguing, but I'm still concerned about interconnect latencies.
Based on my cocktail napkin estimates, the h/w cost of a big RAM system (if it can be got) would probably be about 2x the cost of the piecemeal system, but hopefully signifcantly reduce interconnect latency, and use simpler s/w (assuming I can get Linux to config a ramdisk that big).
Since latency is the prime concern, solidstate disks using the usual HD interconnects just didn't seem an acceptable solution, esp since the cost reduction doesn't seem that significant.
As for failure recovery, my thoughts were just a couple fast HDs that would ping-pong taking snapshots of the ramdisk; hopefully, the ramdisk recovery would never lose more than an hour's data (the lost data wouldn't be critical, and would be recoverable from other sources...its really the delivery speed thats essential)
But I may need to go back to my napkin and do some more queueing analysis to see if there are better commodity solutions.
007: "Who are you?"
Pussy: "My name is Pussy Galore."
007: "I must be dreaming..."
Windows XP doesn't swap, it uses a page file, which preemptively makes copies of the least used pages in your ram to your disk so that if there is an unexpected jump in memory usage, it can wipe those pages from memory quickly and just put them back from disk when they are needed. There really isn't a performance hit since it's a pretty much passive system in reference to RAM writes. Adrian's Rojak Pot has a great guide on Windows virtual memory at http://www.rojakpot.com/default.aspx?location=3&va r1=143&var2=0
-Julius
The 2G sticks have been out for about a year now. 4G sticks are what's new.
:D *Kidding* I could be wrong though. :)
Apple doesn't list higher then 8G, because that is what they tested. 4G sticks will most likely also work. There is really no reason why they wouldn't.
"All that RAM." "far More data than you have resources to process." *LOL* It's safe to say that you don't work with video, nor as an artist. Are you working on a 386SX.
I have over a Terabyte of HD and my system handles it just fine. My 7 to 8 minute video jobs export single video files that average about 12Gigs. If I had 16GB of RAM and AEPro supported it, even with that Much RAM I would still need to "purge it all" every so often. I also generally run most of my pro apps at the same time since I need to jump between them. When they support 64-bit memory, my 5 G of RAM will not be enough. I'm upgrading to Tiger since it allows 64-bit memory support for apps now and CS2 will take advantage of that, at least on the Mac. I'll be allocating 4 gigs of RAM into it, since some of the poster illustrations I work on have exceeded over 3.5G in RAM (And since PS could only see 2G, it had to rely on scratch disk(s) to make up the difference.). That would only leave 1G for everything else, so needless to say I'll also be upgrading my RAM when I buy Tiger to more then 8G.
What I work with is peanuts compared to what a friend of mine does. He works with satellite imagery and a small plot of land can easily exceed over a Terabyte for one image.
Anyways, 16G of RAM is nothing with current comps and I bet that in about 4 years you'll probably have at least 6Gs in your system. Go back just a few years and 1G of RAM was considered more then most peeps would ever need or use.
You can configure an Apple Xserve this way, with either one or two CPUs. Not an AMD board, but it is 64bit, and you can still put Linux on it if you like.
$14,599--One CPU, 80GB HD, CD-RW, no video support, but 16GB of RAM and an unlimited user Server OS. You can do better if you purchase through the Education or Government channels, and you can do better if you purchase the 2GB DIMMs elsewhere.
Although I actually couldn't find 2GB DIMMs at the popular aftermarket places, but they are now available from Apple direct (just be sure to get the Xserve with at least one 2GB DIMM, to be sure it has support for 2GB DIMMs on the MLB).
--
$tar -xvf