Slashdot Mirror
64GB RAM Under 64-bit Linux?
gary.flake asks: "My group at NECI is in need of a machine that can address 64GB of ram in a single process. This means we need 64-bit addressing. We'd prefer to go with a Linux solution because all of our development is under Linux. We've spec'ed out some reasonable machines (Dell can do 32GB, and Compaq can do 64GB) but they seem to be lacking in that they can only be loaded up with 4 x 800Mhz Itaniums. We would really like to have more processing power (2 Ghz x 4 would be a dream). Does anyone know of any monster Itanium machines that will meet our needs? (Please, no Alphas). Finding such a beast is harder than you'd think."
This article from The Register has a some info. on the Intel server processor roadmap, although not much about processor speeds. Last I heard the McKinely was going to ramp up to 1.5GHz and seeing that the Madison will start at 1GHz that one is probably the best bet for getting to 2GHz and according to the Register article, its not set to be released until early 2003.
"Karma can only be portioned out by the cosmos." -Homer Simpson
4GB SDRAM DIMMs exist. A quick trip over to Pricewatch shows that Mushkin, for one, makes these things. Prices start at about $1,300. (I'm not saying that Pricewatch components are necessarily approriate for this application, but they do exist.)
You don't have too many options if you need 64 GB of ram in a single image (not a cluster)...
SunFire 3800
According to the techpub, you'll need both CPU trays (but perhaps with each only half populated with CPUs -- 2 + 2 = 4 CPUs) to house enough memory modules for 64 GB. Base price for a 3800 is $160,000 for two boards. Plus unique RAM (currently $1700 per GB). Keep in mind that the 3800 cannot be upgraded beyond 64 GB or 8 CPUs.
AlphaServer GS80
Looks like you'll need at least 4 CPUs to handle the 64 GB. Base price seems to be around $140,000. Plus unique RAM. Keep in mind that the GS80 cannot be upgraded beyond 64 GB or 8 CPUs.
IBM P660
2 - 8 CPUs. Up to 64 GB RAM. Starts at $66,000. Plus unique RAM. Runs the Linux-Friendly IBM AIX 5L. Keep in mind that the P660 cannot be upgraded beyond 64 GB or 8 CPUs.
SGI Origin 3000
Looking at about $220,000 for one that can accomodate 64 GB. Plus unique RAM (about $900 per GB). The 3000 can be upgraded to 1024 GB (1TB) of RAM and 512 CPUs as a supported configuration. 1 TB and 1024 CPUs unsupported and requiring unsupported OS patches.
Origin 300 would be *much* cheaper, but it only supports 32 GB right now (it will support 64 GB when SGI ships their high density memory modules) and it's nowhere near as expandable or upgradable as the 3000. Origin 300 cannot be upgraded beyond 32 CPUs and will most likely never support more than 64 GB of RAM.
IBM is your best best.
Typical big iron uses very custom components. For example, the Origin 3000 uses DDR-SDRAM chips, but on a memory module that also includes "directory ram" and some other oddball components. The O3K has also been using this ram since about 6 months before desktop PCs began using DDR, so they didn't really have a chance to embrace now-popular standards.
Sun's ram is similar. Sun uses slightly slower chips, but has many in parallel with some ungodly wide bit paths to and from the memory modules. Again, it's much different than what we have in our desktop PCs.
Keep in mind that these monster machines were designed with some insane requirements and low tolerance for error. As such, they often require much different components to keep everything working. Recall that most DDR-SDRAM based PC mobos up until the past few months shipped with two or three DIMM slots rather than four or eight because of timing and signal issues that hadn't been resolved yet. Big iron systems from Sun, IBM, and Compaq support over 100 CPUs, over 200 GB RAM, and do so without (many) problems. SGI's Origin 3000 supports up to 1 TB of RAM and 512 CPUs per single system (and several such huge machines can be clusterd via multiple 8 gigabit GSN networking connections if a task requires insane compute power).
I can't believe the number of people implying that this guy is an idiot for needing 64GB RAM.... There are some scientific applications out there, that are extremely important to people with lots of money, where having 64GB worth of fast-access matrices/hash table/etc. will let you do experiments that are simply not possible if you are limited to 1GB
Very well put. I do a great deal of work with my university's high performance computing lab and have been exposed to this area for almost four years and continue to be amazed and blown away. By far our most demanding users are from the meteorology research group -- an organization that has been on the forefront of many new weather modeling and weather analysis applications. Their batch jobs range from crazy to insane when it comes to CPU and RAM usage and have allowed for many interesting case studies with single machines, clusters, and different platforms.
Some random observations:
Many of the jobs the meteorology folks run consume 8, 16, or even 32 GB of RAM per instance. Most jobs keep the disk arrays clattering away like a drive stress test. They've tried our cluster and tried our SunFire, but prefer the Origin 3000 (and for lesser jobs, the Origin 2000). Unlike "simple" partitionable tasks such as rendering (or SETI@home), their batch jobs don't take to clustering too well.
Many of our users put preference to total system size and maturity/support of the compilers moreso that the speed of each CPU or the newness of the software. We still have groups that prefer the Origin 2000 over anything else. One user wants us to invest in commercial fortran compilers for the Linux cluster. Many users simply prefer a single large system such as the Origins and the SunFire.
A few users continue to push the envelope. One that comes to mind has been asking for some adjustment to job quotas so that his app can utilize 128 GB of RAM. We're still not sure *what* he wants to do with that ram and are still waiting to hear back from him.
Not everyone uses their machines to surf the web, compile software, or even manipulate 256 megabyte files in Photoshop... for those that handle several terabytes of data every half hour and need to have half a terabyte or more in ram at any given time.... for those people there is truly big iron and it's available today (and yesterday). I belive we had an SGI Challenge XL with 36 R10K/195 CPUs in 1995. Two years ago we got a 128 CPU Origin 2000 and a 64 CPU SunFire. This summer we finally recieved a 256 CPU Origin 3000 with 512 GB of RAM. Big Iron costs a fortune and isn't for your highschool, your ISP, or even for your bank. But for those that *need* it, it's worth every penny.
A suprisingly large number of responders have asked why anyone would need 64GB, and have speculated that we are either lazy, stupid, or have money to burn. While I can't reveal too much, I'll try to give you an approximate idea of the sort of problems that we are working on.
Basically, we are doing two things: graph theoretic algorithms on graphs with hundreds of millions of vertices and billions of edges, and robust classification systems trained on training sets of a similar size.
For our purposes, we need to access these data structures in a deterministic order that cannot be predicted in advanced. We could cache all of our data structures on disk, but the algorithms would require years to complete because of disk seek time. Instead, we are going to try to keep a compressed version of the data in memory. The difference is that the in-memory approach will take minutes as opposed to years.
And if you are thinking that we need a new algorithm, trust me on this, we are using the faster algorithm. Most of the other candidates run in exponential time.
If you want a real clue as to what we're doing, then read this. If you like what you've read and think you can help, then contact me. I am hiring.
The Computational Beauty of Nature