Slashdot Mirror
Best Motherboard for a Large Memory System?
kimanaw writes "Due to a particular infrastructure need, and increasing OS support for 64 bitness, I'm looking into building a large memory server box (at least 16 gigs, possibly up to 64 gigs, probably config'ed into a big ramdisk). I only need a single CPU, and just minimal disk; most prebuilt systems w/ large memory seem to focus on more CPus and big RAID, all of which (over)inflate the pricetag. I've searched several websites (including Tom's Hardware), and I've googled, but can't seem to locate any commercially available AMD MBs supporting more than 4 sticks of RAM, or 4 gigs. Have any Slashdotters built a big-RAM server? Any pointers, hints, and tips much appreciated."
...but there are 2 GB chips now.
http://ask.slashdot.org/article.pl?sid=05/02/22/09 30238&tid=198&tid=126&tid=4&tid=137
High End Macs support 8gigs
I really like the specs on the Tyan S4882. Quad Opteron, 32GB RAM, and lots of stuff you probably don't want/need. It's a sweet motherboard, nonetheless.
RHCE; are you certified? Karma: ambiguous.
IBM xSeries 445. We have around 50 of these IBM servers at work and they support up to 64 GB of RAM. Any reason why you are trying to build a server out of cheap commodity parts? Save yourself the trouble and buy one of these.
CLICK HERE
fuvoo: watch something
I know you're looking for an MB, not a system, but if you go to http://www.xicomputer.com/ and check out their NetRaider 64 LT. You can custom configure it to have up to 12GB of ram w/ a single Opteron 244.
Have fun!
has several boards, but none of them are single proc systems. If you need a stock Mobo, and want that much memory, you may need to go for a tyan board. I don't know if you can purchase the quad cpu board, use one cpu and all sixteen slots or not, but I think it will work. Obviously if you are going to shell out for that much memory, buying that mobo shouldn't be a big deal, even if you don't use all four cpu slots.
FWIW, that's the only thing I know of.
"We don't know what we are doing, but we are doing it very carefully,..." Wherry, R.J. Personnel Psychology (1995)
RAM DISK. He said it in the question. Ever run a 15GB DB in RAM instead of on a physical disk? It's very fast. There are a ton of options that would be good in RAM. Albeit if he were running a DB application more than one processor would probably be beneficial as well.
Troll.
Where did you get these facts?
The kernel has three memory models. = 1GB, = 4GB and = 64GB.
I've run Linux on a 1.5GB box before just fine [they were PC-133 sticks on sale bought three 512MB sticks...]. You just have to enable the different memory model.
Tom
Someday, I'll have a real sig.
Sun Fire V40z: up to 32 gb of memory
8 of 13 people found this answer helpful. Did you?
Get a Sun. You can get very large memory systems. You can go up to 192 gigs of memory. Marc
Tyan Thunder K8QS Pro Quad Socket 940 Motherboard AMD Solution Motherboard - Operton Motherboard: Outpost #: 4311135 http://shop2.outpost.com/product/4311135/
-Chipset: AMD-8131
-Processor Support: Quad Socket 940 for AMD Opteron 800 Series Processors
-Memory Support: Up to 32GB Registered PC3200 DDR
-Expansion Slot: Two 64-bit 66/33 MHz PCI-X, two 64-bit 133/100/66/33 MHz PCI-X, One 32-bit PCI Slots
-Other Features: GigaLAN, Integrated Video, SCSI
if you have the money to fork over, pick one of these up at your local Fry's Electronics or try fry's online at http://www.outpost.com/
There are all sorts of scientific applications where you need as much RAM as possible. One where the amount of RAM is almost the only limiting factor is model checking. It refers to a systematic exploration of an entire system. Even though a lot of techniques have been invented to reduce the complexity of the problem, the problem itself is exponentially complex in the size of a system. A 16-bit counter has 65536 states. Just imagine how many states a simple ALU or an entire CPU can have. This shown you that there is no limit to the amount of RAM one can use for this.
All major players in the CPU industry use this technology, so I am sure AMD has some systems like that in their design labs...
Model checking is also used in some software projects where the complexity of the system is "manageable", such as for device drivers or embedded systems.
Other scientific uses needing a lot of RAM are simulations. The level of detail simulations can manage is mostly limited by the amount of RAM (and of course also CPU speed). Again, the sky is the limit for how much RAM one could use if it was available... so you see it is not problem to fill 32 GB of RAM in a couple of seconds with such simulations.
I've always read it's inadvisable even if one has a lot of RAM. Here's some quotes from this page:
Can the Virtual Memory be turned off on a really large machine?
Strictly speaking Virtual Memory is always in operation and cannot be "turned off." What is meant by such wording is "set the system to use no page file space at all."
Doing this would waste a lot of the RAM. The reason is that when programs ask for an allocation of Virtual memory space, they may ask for a great deal more than they ever actually bring into use -- the total may easily run to hundreds of megabytes. These addresses have to be assigned to somewhere by the system. If there is a page file available, the system can assign them to it -- if there is not, they have to be assigned to RAM, locking it out from any actual use.
--
Why is there so little Free RAM?
Windows will always try to find some use for all of RAM -- even a trivial one. If nothing else it will retain code of programs in RAM after they exit, in case they are needed again. Anything left over will be used to cache further files -- just in case they are needed. But these uses will be dropped instantly should some other use come along. Thus there should rarely be any significant amount of RAM 'free'. That term is a misnomer -- it ought to be 'RAM for which Windows can currently find no possible use'. The adage is: 'Free RAM is wasted RAM'. Programs that purport to 'manage' or 'free up' RAM are pandering to a delusion that only such 'Free' RAM is available for fresh uses. That is not true, and these programs often result in reduced performance and may result in run-away growth of the page file.
I'm on a road shaped like a figure eight; I'm going nowhere but I'm guaranteed to be late.
What you're interested in is solid-state storage, which you can google. The main problem with your plan is, if the power goes out, you've lost everything. Even trying to flush everything to the hard drive will take dangerously long. (Writing 15GB to harddisk is no small task!) The solid state storage machines out there are basically computers with lots of RAM, but the RAM is backed by battery, so even if it's unplugged, it stores its state, and then backs up to hard disk, automatically. Trying to get a PC to do this for you would be much trouble. Remember: UPSing a whole computer is a lot different than keeping RAM charged! :)
Digital Sailor
The only way to get anywhere close to 64GB is by going multi processor. I have 2 Thunder K8S Pro (S2882) motherboard with 16GB of RAM each, running Postgres 8 on FreeBSD 5.3 in production. These systems are dual opterons. They have 2GB sticks. I also have a test system with a Tyan Thunder K8QS Pro (S4882), this one is a quad opteron and supports upto 32GB with 2GB sticks. The only way I know to get 64GB with 2GB sticks is with an IWILL H8501, that's an 8-way opteron barebone system.
I think you can get 4GB stiks now, but they are very expensive still.
The ASUS K8N-DL is a dual Opteron motherboard which officially supports up to 24GB RAM in 6 DIMMs. It costs only about $300, too. Not sure of current availability, it's pretty new.
Not so strangely, each set of 4 DIMM banks is hard wired to the memory controller of each processor.
Accesses from one processor to the further away bank must go through hypertransport, but as long as the OS scheduler is NUMA-aware, jobs should tend to run on the processor which is local to the memory it has been allocated, making for some screaming memory performance relative to single memory controller solutions (most Intel SMP setups).
XML is like violence. If it doesn't solve the problem, use more.
More than 4 DIMM slots on a UP box? Forget it, it isn't going to happen. The problem isn't with addressing limitations; it's a physical hardware problem, along with issues of economics. DDR SDRAM is a parallel, shared bus architecture. The more memory slots you add to a channel, the higher the load on the channel and the more issues you have with signal integrity. You also run into trace routing issues trying to cram all the DIMM slots in while keeping your trace length equal so you don't get signal timing issues.
Now, these are all solvable, but it isn't cheap. The UP server market does not carry nearly enough of a premium for it to be cost effective to load a UP board up with more slots. So, the bottom line is if you're looking for massive amounts of memory, you're pretty much forced to buy at least a dual proc system, and more likely a quad proc system. High density memory modules do help, but again, they put more load on the memory bus, so look at the fine print to determine how many of those 2 and 4GB DIMMs you can toss on a memory channel.
Management\DisablePagingExecutive to 1.
That only stops Windows from paging out parts of the OS itself (like the kernel and currently-idle device drivers).
On XP, you can just set the pagefile size to zero. And yes, it does work, in the sense that it stops Windows from hitting the disk twice per second even when doing nothing at all.
On 2000, you need to assign the pagefile to a RAMdrive (and one that supports NTFS and doesn't identify itself as RAM, since Windows won't normally let you put the pagefile on a RAMdrive).
Finally, for anyone disabling paging - Also disable memory dumps and automatic reboot on bluescreens, or you will regret it.
Otherwise, it works just fine, regardless of what the naysayers and MS Knowledgebase fanboys might say. Quite a boost in performance, too.
In order to populate all the RAM slots in an Opteron system, you have to populate all the processor sockets as well. And a 4-socket system requires expensive Opteron 8xx processors.
Instead, consider an IBM x366 or an HP DL580; either one can be configured with 32GB RAM and 1 processor for under $30K.
Doing this would waste a lot of the RAM. The reason is that when programs ask for an allocation of Virtual memory space, they may ask for a great deal more than they ever actually bring into use -- the total may easily run to hundreds of megabytes. These addresses have to be assigned to somewhere by the system. If there is a page file available, the system can assign them to it -- if there is not, they have to be assigned to RAM, locking it out from any actual use.
Not true. When a program allocates a large chunk of ram, all it gets are the page table entries. Only when it actually writes to a page does the ram get used. This means that I can allocate 1.5G of ram and scribble in half of it and only be using 750M
"We returned the General to El Salvador, or maybe Guatemala, it's difficult to tell from 10,000 feet"
There are a few possible solutions you might want to look at for a big-RAM server. Now, if you really want 64GB and AMD Opteron processors than you really only have one choice, the HP Proliant DL585. That's the only Opteron solution that I know of which supports 64GB of memory.
If you can get by with a bit less memory then you have some other solutions. Tyan carries quite a number of boards with varying capabilities. The trouble here is that the Opteron processors are limited to 8GB of memory per processor, so to get 16GB you're going to be looking at a dual-processor board (quad processor for 32GB). Since the memory controller is right on the CPU with the Opteron you will actually need a second processor in the socket to use this memory.
For this reason, you might actually want to consider one of Intel's new 64-bit Xeon chips. I know that Supermicro offers some boards that can handle up to 32GB with only a single Xeon processor. Something like the X6DHE-XB seems like it might fit you're bill reasonable well. Fairly inexpensive to get you up to 16GB of memory, though going to 32GB is quite expensive. Crucial has a list of compatible memory for this board, including some 4GB modules.
Of course, if you're not limited to x86 systems then there are other solutions that would work. You could get something like an IBM Power system or Sun UltraSparc system with pretty much any amount of memory you need (or can afford).
Caching is often a bigger win than RAM Disk.
:-) But I know these can go up to 1/2 TB, addressable by a single application. And you can't beat their IO.
Profile your application before making such committments. You might be surprised at what you find.
What's interesting here is how securely "in the box" people are thinking. They are on a do-or-freaking-die mission to press a consumer architecture into service as a mid-range engineering system.
My shop uses Sun hardware for midsize applications like this, but we also run some Alpha systems (Both older DEC and newer HP).
We have a couple of Sun E20K but I don't know how they are configured exactly (I don't get to touch stuff that belongs to financial depts, nor would I want to
But somebody who is thinking in terms of "What motherboard should I buy?" probably isn't prepared for the sticker prices of SunFire servers, EMC storage, etc.
When a skunkworks shop needs some large scale system on a budget approaching zero dollars, some creativity is needed. So we get into storage solutions like consumer RAID, and we hear questions like the OP, "what's the largest memory configuration on a commodity 64-bit microcomputer?"
Dual Xeon boards that take 32GB RAM are common enough, but I think that's 16GB per processor and I have no idea how it's addressed or whether any given application knows how to address it.
32GB in premimum RAM will run over $20 grand. Nothing compared to the half-million for an E20K, of course.
-fb Everything not expressly forbidden is now mandatory.
Intel Xeon64bit Dual up to 32GB of DDR 266 http://www.supermicro.com/products/motherboard/Xeo n800/E7520/X6DH8-XB.cfm
Intel Xeon64bit Dual up to 32GB of DDR2-400
http://www.tyan.com/products/html/thunderi7525_spe c.html
Windows will run very slowly without a swap file in many cases.
Linux and Windows have fundamental virtual machine differences. Linux swaps pages to disk, Windows keeps a page backed store. What this means is that every memory address in Winodws is backed up by some kind of virtual memory. For code and some data segments, the "page" is the executable file on disk itself, so it doesn't need to ever copy those pages to swap. (Incidentally, this will show up as "swap file in use" even though it's not in your swap file).
Windows also uses swapping for memory mapped files, of which many apps make use of.
Finally, if there is no swap file, there is no "reserved" memory either. That is, memory that is allocated but not used. This can cause some apps to freak out if they reserve more memory than you physically have.
If you need web hosting, you could do worse than here
As to the purpose of the box, I can't say too much, except its intended as a sortof main-memory database cache solution (ala TimesTen ) to serve a potentially very large user base, with a lot of cached images associated with the cached data.
I also considered a piecemeal commodity approach, e.g., filling racks with cheap 4 gig RAM 1U's running Linux, but then I'd have to come up with s/w that can coordinate those systems (basically, a head-end to hash the requests to the right 1U). memcached looks intriguing, but I'm still concerned about interconnect latencies.
Based on my cocktail napkin estimates, the h/w cost of a big RAM system (if it can be got) would probably be about 2x the cost of the piecemeal system, but hopefully signifcantly reduce interconnect latency, and use simpler s/w (assuming I can get Linux to config a ramdisk that big).
Since latency is the prime concern, solidstate disks using the usual HD interconnects just didn't seem an acceptable solution, esp since the cost reduction doesn't seem that significant.
As for failure recovery, my thoughts were just a couple fast HDs that would ping-pong taking snapshots of the ramdisk; hopefully, the ramdisk recovery would never lose more than an hour's data (the lost data wouldn't be critical, and would be recoverable from other sources...its really the delivery speed thats essential)
But I may need to go back to my napkin and do some more queueing analysis to see if there are better commodity solutions.
007: "Who are you?"
Pussy: "My name is Pussy Galore."
007: "I must be dreaming..."
Here. Retail, looks like you can get these intended for Compaq servers. They even sell them at CompUSA.
That's called overcommit. You can switch this off or on with linux (it's usually on, but there might be cases this is not desired behaviour and you absolutely have to guarantee that enough memory is actually available). I'm not sure what windows does, it would surprise me if it wouldn't overcommit memory.
Amazing to see incompetant advice first start by proclaiming the incompetance of other sites.
Opteron's all have built in memory controllers. That means, you cannot populate the second bank without another Opteron. Real simple. You need the additional memory controller, or you are stuck with getting to 8 GB via 2 GB dimms.
More importantly, I would not recommend Tyan boards. Having used them (we design, sell, service large linux clusters with huge memory configs), I can tell you that they have caused use nothing but grief. The s4882 has design errors heaped on top of design errors. For example, the dimm slots on 2 of the processors cannot be fully opened due to collisions on either side of the mechanism. This isn't the only problem, there are some that are significant in terms of operation that effectively make these boards useless for large memory operations.
Look at either Iwill motherboards, the new MSI that gives you 12 slots, or go with Panta Systems, where you can get 128 GB over 4 CPUs (Opterons).
It is not worth it building this yourself, go to a knowledgable source, and ignore the idiots who proclaim Tyan from the tallest tree.
I was able to fit 32GB into one of these babies:
0 /index.html
:^)
:^)
http://h18002.www1.hp.com/alphaserver/archive/es4
It ran very well with that much RAM.
Just don't try to load Windoze on it.
Windows XP doesn't swap, it uses a page file, which preemptively makes copies of the least used pages in your ram to your disk so that if there is an unexpected jump in memory usage, it can wipe those pages from memory quickly and just put them back from disk when they are needed. There really isn't a performance hit since it's a pretty much passive system in reference to RAM writes. Adrian's Rojak Pot has a great guide on Windows virtual memory at http://www.rojakpot.com/default.aspx?location=3&va r1=143&var2=0
-Julius
When spec'ing a large RAM Opteron make sure to remember that unlike Intel Xeon systems, the memory controller is on the cpu die. This provides for great memory performance.. but requires that on a 2, 4 or 8 way system you must purchase a CPU to use the bank of memory adjacent to the CPU socket.
Blantant Ad: My employer, Western Scientific, has recently released their 8-way Opteron solution, the FusionA8, which supports 32 memory sticks when 8 Series 800 processors are used.
There are a couple problems with what you propose. Firstly, most applications requiring 16 GB or more of memory usually require (or at least benefit from) multiple CPUs. Such applications would be databases, sophisticated modelling systems, etc.
The next problem is the electronic signaling. Getting 4 memory slots to work together at today's speeds can be tricky. That's why in the past couple of years it has become increasingly important to use memory modules that are listed as supported by your motherboard manufacturer. Wiring together more than four slots and getting the signalling/timing down right is much more difficult. This isn't as big of an issue on SMP Opteron systems because each CPU has it's own memory interface and dedicated memory, so 8 slots on a 2-way equals 4 slots per CPU, still with easily achieveable goals.
That leaves you with having to fit larger memory modules into your four memory slots. The largest that I have seen generally available are the 2GB modules. I wouldn't be suprised if someone were selling 4GB modules, but they will be very hard to come by and very expensive. Right now you can buy 1GB ECC modules for around $280-$300 each. The 2GB ECC modules are about $800 each. I can't imagine what 4GB modules would cost, but I know that I wouldn't want to pay for them.
Since you were talking about using it as a RAMdisk a better option might be a solid state hard disk.
You can configure an Apple Xserve this way, with either one or two CPUs. Not an AMD board, but it is 64bit, and you can still put Linux on it if you like.
$14,599--One CPU, 80GB HD, CD-RW, no video support, but 16GB of RAM and an unlimited user Server OS. You can do better if you purchase through the Education or Government channels, and you can do better if you purchase the 2GB DIMMs elsewhere.
Although I actually couldn't find 2GB DIMMs at the popular aftermarket places, but they are now available from Apple direct (just be sure to get the Xserve with at least one 2GB DIMM, to be sure it has support for 2GB DIMMs on the MLB).
--
$tar -xvf
> You are right but only for OLTP types of applications. Full table scans can't be buffered adequately.
yeah, cache hits are more critical in oltp - where a single percentage point in the hit rate can be a noticable difference.
but even on non-oltp applications (data warehouses, data marts, olap, whatever), it makes a difference - and you (typically) get can get good hit rates. I'm looking at stats on a 500 gbyte data warehouse right now, and see:
1. dimension tables: 99% cache hit rate
2. summary tables: 98% cache hit rate
3. fact indexes: 99% cache hit rate
4. fact tables: 92% cache hit rate
In the above list the vast majority of data is in the fact tables - where over 90% of the queries are coming out of memory. Since many queries actually hit the summary tables instead of detail, those queries are getting a 98% hit rate. And when the cache is missed - it doesn't necessarily mean a tablescan: this particular database is partitioned via db2, so a cache miss generally just results in a partition scan of 2-4 seconds.
So, while you're right - caching is tougher in a reporting environment, it still helps enormously in most implementations. At least in my experience.
ken
This means that on a dual mobo, half the RAM slots are wired directly to the second CPU socket, and with no second CPU installed, that half of the RAM is simply inaccessable to the first CPU. You need that extra CPU - or a single-CPU mobo with a lot of RAM slots.
Why would anyone engrave "Elbereth"?