Slashdot Mirror
Where are the Large RAM Systems?
CaptCanuk asks: "I've been charged with finding a system with 16 GB of memory and have had a really hard time in acquiring one (especially with a PCIE 16x slot). Linux is at the forefront of these 'large system memory' systems and beyond beta versions of Windows XP, is the only OS that supports the 64 bit memory addressing required to use this much RAM. When I asked large beige box wholesalers, I'd get comments from 'Why do you want a 16GB harddrive...you want MEMORY? are you sure?' to 'No motherboard supports more than 4GB of memory; everyone knows that'. Where are these mythical large memory systems? Do you think such workstation configurations will become pervasive in the future? Will it take Microsoft's Windows XP 64 bit to legitimize their existence in larger quantities?"
Because if you're a company looking for reliability and ongoing maintenance, a self-build isn't necessarily the first thing you think of. You're a beancounter looking for an ongoing support contract from a reputable OEM you've heard of before, possibly with onsite or couriered-in replacement clause.
I'd like to throw out the possibility of clustering instead, though (mostly cause it's on my mind because I've been dealing with several support cases on clusters recently). Why is this not an option for extra power, resilience, etc..
Screw you all! I'm off to the pub
...there all over the place:
Dell Itanium
HP Itanium
IBM Itanium
Or, I went to AMD's page here and clicked on one of the manufacturers listed. Where I found this dual opteron supporting 16GB ram. Took me all of 2 minutes.
-- Hulver's site
http://www.appro.com/ do same damn fine boxes, including 1U (yes, 1U) quad (yes, yes, quad) operton boxes that take 32GB of RAM.
:(
I only wish the company I work for could afford boxes like that
Oh, and there's that "need" thing I keep hearing about.
I just found a number of boards within about 30 seconds. That's a new low for an ask slashdot.
Here's a few
Every board there except for the single processor ones supports at least 16 GBs of memory. Many have 16x pcie slots and at least one has 2.
Best slashdot comment
That is the addressable memory space. Curiousley the x84 instruction set doesn't have a 32bit (4Gb) wall but reather an 64Gb wall due to the segment offset. This is the original hack which gave us 1Mb limit reather than the 64K.
The only reason linux gives you the choice between the two when compiling is to allow the address to be stores in one 32bit int.
Mouse powered Chips, Open source Processors and Lego
There are quite a few motherboards that can handle 16G (or 32G) memory, they're mostly dual/quad Opteron boards. Tyan has a line.
If you also want PCIe x16, it's harder - Tyan lists this baby (Thunder K8WE), but I don't know if that one is actually available already.
I believe posters are recognized by their sig. So I made one.
The Apple XServe supports 16Gb of RAM, the just don't like to admit it. I found this image on their site while looking for stuff last year.
I can think of a couple of things...
How about working on enormous multilayer images in Gimp that are ultimately
destined to be printed as large, high-gloss posters? That'll eat some RAM.
The piddly little images I have worked with (you know, 600dpi for 8.5x11
letter-size, tiny little things) can use up more than a gigabyte each, with
only four layers; a complex image can easily have over fifty layers...
Video editing springs to mind.
Databases perform better if they can fit all the data in RAM, especially if
the data are read far more often than written. It's easy to imagine a DB
that goes past 4GB.
I'm sure there are other applications where you could want that much RAM.
Cut that out, or I will ship you to Norilsk in a box.
Whether to use a cluster or not depends heavily on the problem domain. Until very recently, clusters didn't work so well with large databases, etc.
You also seem to be shopping specs rather than throughput. Your mention of 16x PCIexpress is what gives this away. The only cards that support this now are high, high end graphics cards, and these cards don't even need it. There's no real difference between the AGP 8x and PCIex versions of these cards.
That said, you're not going to find what you're looking for in the beige box world. You're looking (realistically) at about 4 different venders: Windows: Dell, IBM, and HP. UNIX: IBM, HP, and Sun.
You're also only looking at servers (not desktop or towers).
My experience is with Sun, and a little Dell and IBM. So I'm going to speak to those. Sun makes magnificient hardware. Their support organization has had problems recently, but the hardware is good enough that we don't need it often. Sun's V880 servers are amazing. up to 8 CPU's and up to 32 gigs of ram, with great growth potential (12 PCI slots, several of them 64 bit, 66 MHz).
We've had lots of problems with our Dell hradware. Whole lines of their servers have been crap, and dell replaced thier 16xx line with their 17xx line for us for free. Our exchange server runs on a 6550 IIRC, which has at least 8 gigs of ram. This model probably can go higher in ram, but I'm not sure.
We've been really impressed with the IBM hardware we've started to purchase. It's been pretty stable, fun to work with, etc. IBM has a long history of making great servers. They probably have several models that will help.
Zapman
OS X does not totally take advantage of more than 4GB of RAM. It can address tons of RAM, but each running application is limited to a maximum of 4GB of addressable space in Panther.
When Tiger comes out, non-gui applications will be able to address the full 64 bit address space, however, GUI apps will remain limited to the 32 bit address space. See here for more info.
Doh!
Buy an SMP opteron box, they'll support all the memory you want and then some. Most of the Opteron motherboards I've seen in use have 4 memory slots per cpu socket. So for instance with a quad opteron boards you could stick 16x 4G sticks in it for 64G of ram. Incidentally, it's not that only linux supports "64-bit addressing". The memory addressability is a function of the processor and/or memory controller (which is integrated in the processor in the case of the Opteron). There is no processor I know that can actually physically address 64 bits of memory (which would require something on the order of 65,536x 256Terabyte sticks to fill). IIRC correctly, the Opteron memory controller can physically address 40 bits of physical memory, which puts the theoretical limit for it at 1TB of RAM.
11*43+456^2
Most CPUs starting from PentiumPro had 36-bit physical addressing, and as long as the mainboard and chipset supported it too, CPU could address up to 64GB of PHYSICAL memory. For this however, paging must be switched to the PAE mode, so page table entries will have another format and could contain bigger physical address base, that's what the "64G" Support in Linux' kernel was for. It has nothing to do with memory segments, or the ability of 8086 to address 1MB using 16-bit pointers at all.
On 32-bit CPUs, VIRTUAL addresses and segment sizes are still 32-bit and could not exceed the 4G size.
One additional thing to consider if you are planning to use Windows is the 4GB process limit (which is NOT the same as a total memory limit) in a 'normal' Windows server.
/3GB switch, bla bla bla, ....).
i d= 69
The operating system (Windows Server Enterprise Edition) will work with more than 4GB memory, but a process running on that server can only address 4GB of memory, of which 2GB is reserved kernel space (in normal circumstances, not including the
Check out:
http://www.brianmadden.com/content/content.asp?
Of course there are some tricks and things you can do, but still... keep this in mind.
This is due to the fact that you are working on 32-bit hardware that can only address 4GB directly, as far as I understand. Does Linux have this limit too? Or are there other 'tricks' that the Linux kernel applies to go above 4gb? Maybe other Slashdotters can elaborate on this.
Dell PowerEdge 6600, 6650, 7250...b oard/Xeo n800/. html
IBM xSeries 336, 346...
http://www.supermicro.com/products/mother
http://www.tyan.com/products/html/barebone
In short, every place I've checked so far.
-Uberhund
Really, I gotta wonder, what the hell are you running that requires that many pages to be in memory at the same time.
How about the entire genomic sequences of >4 organisms? That way you can compare them to each other simultaneously, and learn which sequences are similar and which are different.
Here's another application, off the top of my head: simulate the gravitational mechanics of any large system of objects. Think you want to swap that kind of thing to disk?
I submit that there are many scientific applications of this much RAM; and you're not likely to recognize or understand the need unless you're in the field yourself. A LOT of bleeding edge computing work is being driven by scientific researchers who demand, really, a heavy amount of resources to do their simulations on--and computing structures that are designed for database work/gaming is just not comparable.
Personally, we use HP quad Opterons, with 64GB of RAM each (running Linux, btw); and while you could build that kind of thing yourself, the reliability issues at that scale just aren't worth it.
--
$tar -xvf
The recently released hp xw9300 is exactly what you want. It has room for 2 Opteron processors, up to 16gb of ram, and dual PCIe x16 graphics cards.
It starts at around $1900, a decent price for a dual-proc workstation. It has SATA II 300, an NVIDIA chipset (NForce Professional 2200; based on NForce4) and 8 dimm slots for registered DDR.
http://tyan.com/products/html/thunderk8we.html That bad boy has two PCI-Express slots to boot. Son, you just have to look for them...
The large system there has 4 GB RAM (4 1Gig memory sticks - substitute 8 2 GB RAM sicks gets you 16 GB memory). True, these don't have PCIe - Sun won't be getting PCIe until later this year, but the IO on this system isn't to be beatten.
If you want even more memory, try the 40z and 16 2GB RAM sticks for even more memory.
Don't expect Intel systems with Dual memory controllers to get you there - you need real systems.
I have mod points and I am not afraid to use them
According to the docs at http://www.intel.com/support/motherboards/server/s e7520bd2/sb/CS-013543.htm, this board supports up to 24GB with the right kind of RAM, assuming you can find 4GB RAM. With 2GB sticks, you could get 12GB.
Tyan just released a new series of Opteron boards that have PCIe & 16GB: They're the ones with the "E" at the end of their names [e.g. Thunder K8WE -vs- the older Thunder K8W].
http://www.penguincomputing.com/products/workstati ons/niveus800.php
Features:
Full-Tower Workstation Chassis
Dual Intel® Xeon® Processors w/ EM64T
800MHz Front Side Bus
Up to 16GB of PC2700 DDR RAM
Two External 5.25" Optical Drive Bays
Four Internal or Hot-swappable 3.5" SATA Hard Drive Bays
One PCI-Express x16 Slot
One Gigabit LAN port on Motherboard
If you like what I've said here, and want to read more, go to http://www.krillrblog.com
"A: "Legitimize" is a word I don't"...
XEON based Intel solutions have had extended RAM support for years. There's nothing new with Intel based systems having more than 4GB or ram. You just need an OS that supports the PAE extension. This boosts the memory capacity of the OS from 4GB (32bit) to 64GB (36bit). Linux and Windows have supported PAE for quite a while. (Microsoft artificially disables the ram based on the version of windows you're using)
The difference between 32bit w/PAE and 64bit is that a program running on the 32bit OS will only access a maximum virtul memory space of 4GB, while a program running on the 64bit variant can access the entire virtual memory space that the OS can.
If you're running a 32bit compiled app on a 64 bit processor, you're also limited to the 4GB space as well.
Bye!
Including segments, an 80386 virtual address consists of 48 bits: 16 bits of segment selector, and 32 bits of offset within that segment. Of that segment selector, 2 bits are used for the "requested privelege level" and select different views of the same segment. 1 bit is used to select the local vs. global segment table. Generally, a process can use all of its local segment table (13 bits), and some of the global segment table, but the latter is shared with the operating system.
So call it 13.5 bits. The actual number of different segment+offset combinations that can be specified in 45.5 bits.
Each segment selector value causes a lookup in the appropriate (local or global) segment descriptor table. The descriptor contains a 32-bit base address and a 32-bit limit. (Actually, a 20-bit limit, plus a "granularity" bit wqhich says whether the limit is in bytes or 4K pages.)
If the offset is beyond the segment descriptor's limit, the address is invalid. Otherwise, the Intel-unique "linear address" (also a virtual address) is formed by adding the 32-bit segment offset to the 32-bit segment base.
The 32-bit linear address is an addressing space bottleneck. Unless you have segments overlapping in linear address space (which would defeat the purpose here), you can only have 4 GB of segments marked as present in the operating system at any one time. And that limit includes the operating system itself as well as the user process.
The 32-bit linear address is then put through a page table system like any other processor's, which produces a 32- or 36-bit physical address. 36-bit addressing can't expand any single process' address space beyond the 32-bit linear address space limit, but you can have multiple such processes in memory at once.
Now, it is possible to mark segments as "not present", so you can do swapping and virtual memory on a segement basis, but this raises a couple of issues:
What it boils down to is that there's an extremely theoretical possibility, but you could do almost as well in straight software (especially using so-called pointer-swizzling tricks), and so no operating syste, has ever tried to break the 4 GB virtual address barrier that way. And it takes a complete and total overhaul of the operating system's memory management to try.
Indeed, precisely because all modern operating systems just set the segment registers to point to some large fixed values and leave them alone, modern x86 processors have inefficient support for segment register loading. They d