Best Motherboard for a Large Memory System?

kimanaw writes "Due to a particular infrastructure need, and increasing OS support for 64 bitness, I'm looking into building a large memory server box (at least 16 gigs, possibly up to 64 gigs, probably config'ed into a big ramdisk). I only need a single CPU, and just minimal disk; most prebuilt systems w/ large memory seem to focus on more CPus and big RAID, all of which (over)inflate the pricetag. I've searched several websites (including Tom's Hardware), and I've googled, but can't seem to locate any commercially available AMD MBs supporting more than 4 sticks of RAM, or 4 gigs. Have any Slashdotters built a big-RAM server? Any pointers, hints, and tips much appreciated."

Comments

Dunno if 8'll work for you

[Rick Zeman]

...but there are 2 GB chips now.

Re:Dunno if 8'll work for you

[PhoenxHwk]

In fact, Crucial has 4GB sticks. They're obscenely expensive though (~$3k), and are fairly slow - PC2100. Impractical, but worth a mention. The 2GB sticks are much more reasonable.

I have.

[Colin Smith]

Or at least, I took it out the box and plugged it in. With the help of the vendor engineers. Cost quarter of a million quid.

HTH.

inherent limitation of the processor

[Anonymous Coward]

Opterons only support 4 banks per channel

which translates to 4 two-sided memory modules for dual channel per processor

Demand?

[tomstdenis]

Maybe you don't see many home-user 64G boxes because there is little demand for it.

As for "expensive addons/etc". You can buy cheaper motherboards and just add on IDE if that suits your fancy. Promise [iirc] controllers give decent performance and aren't that expensive [~$60 CAD]

Tom

If you want eight sticks

[DmitryProletariat]

With AMD I've only seen 8 stick support on 4-way SMP motherboards. And yes, they're expensive as hell. But you do get up to 32GB RAM support, and it's all shared. Which is perfect for absurdly large matrix calculations, but less than helpful for your situation.

Re:If you want eight sticks

[Anonymous Coward]

There is the Tyan Thunder K8WE, a 2-way Opteron motherboard with 8 memory slots for around $500-600 USD. There are probably others like this.

Re:If you want eight sticks

[Anonymous Coward]

Good call. Actually, that link might be helpful to me as I know someone who was looking for a 16GB system in the next few months to calculate huge matrices in Mathematica. Latest version support 64 bits in Linux...

And now back to playing DmitryProletariat (which has been a source of much amusement) :)

Re:If you want eight sticks

[william_w_bush]

http://www.newegg.com/app/ViewProductDesc.asp?desc ription=13-151-134&depa=0

tyan 2-way boards do 8 sticks. you need 2 opteron mem controllers to do 8 sticks, and the timings suffer somewhat. a 4/8-way board might give you more. opterons are limited to 4 sticks per cpu for the most part, the price of onboard controllers.

you said amd, but intel can do 32gb with 1 cpu, and ddr2 scales better than ddr.

I don't know of such things...

[Nate53085]

But in order to get more the 4gigs of ram on that badboy with a single processor you will need a 64bit system since 32 bit system maxes at 4 gigs of ram.

Re:I don't know of such things...

[Anonymous Coward]

Intel 32bit x86 chips can use 48bit addressing you dipshit

Re:I don't know of such things...

[Anonymous Coward]

Wow... way to read the summary.

Re:I don't know of such things...

[Anonymous Coward]

Actually, I think there is an memory extention pse36 which allows 36-bit addressing of memory.

Allows addressing of up to 64GB of physcial memory on 32-bit systems. Introduced around Jan 2000 if the press release from Compaq is right.

Now, 32 bit proceses can only handle 2GB of ram I think.

Re:I don't know of such things...

[Anonymous Coward]

2^32 is 4 Gig of addresses

If the new Intel 32 bit architechures support 2^36 memory addresses, then you are looking at 64 Gig max.

Re:I don't know of such things...

[Bradee-oh!]

A few points -

-The submitter didn't explicitly state it but I'm guessing since he did mention the popularity/existence of 64 bit processors that he intends on using one.

-32bit CPUs are not limited to 4gigs of ram. Only 4 gigs directly addressable at a time, yes. But google "page address extension" to find all you'd ever want to know about it. The Linux kernel has supported 64gigs on 32bit processors since the late 2.5.x's

But, of course, I agree 64 bit is the way to go because it avoids all that address-table-translation nastiness but the submitter's problem remains.

Is there a motherboard on the consumer market within a reasonable price range ($1000, maybe?) that can actually, physically take 64+ gigs of ram?

Re:I don't know of such things...

[Anonymous Coward]

That would be 36-bit addressing, Einstein.

4GB * 2^4 = 64GB

Re:I don't know of such things...

[Anonymous Coward]

It was the individual processes. Although looking at the other thread someone else posted, the proceeses running are limited to 32-bit addressing, 4 GB, while the processors can physcially address 36-bit, 64 GB of memeory.

Although since the original person wants a 64-bit proccessor, the physcial limit doesn't apply.

Re:I don't know of such things...

[fishbowl]

"Is there a motherboard on the consumer market within a reasonable price range ($1000, maybe?) that can actually, physically take 64+ gigs of ram?"

There are lots of dual Xeon boards that take 64GB.
The trick is finding the chips. A $375 board that needs two $300 chips is one thing. $20,000 in RAM chips takes it to the next level...

maybe this will help?

[carterhawk001]

http://ask.slashdot.org/article.pl?sid=05/02/22/09 30238&tid=198&tid=126&tid=4&tid=137

Re:maybe this will help?

[Anonymous Coward]

Nothing for you to see here. Please move along.

Re:maybe this will help?

[Anonymous Coward]

fixed link

Re:maybe this will help?

[Anonymous Coward]

remove the space that slashcode added.

High End Macs support 8gigs

[Anonymous Coward]

High End Macs support 8gigs

Out of curiosity

[elid]

what do you need so much RAM for in an individual single-processor server?

Re:Out of curiosity

[Anonymous Coward]

What do you need so much RAM for in an individual single-processor server?

CAD. EDA (Electronic Design Automation), databases, any serious engineering application.

Re:Out of curiosity

[BoomerSooner]

RAM DISK. He said it in the question. Ever run a 15GB DB in RAM instead of on a physical disk? It's very fast. There are a ton of options that would be good in RAM. Albeit if he were running a DB application more than one processor would probably be beneficial as well.

Re:Out of curiosity

[Tsiangkun]

I'm guessing it's for a database that doesn't do a lot of changing, and needs to respond quicker to queries.

Re:Out of curiosity

[Anonymous Coward]

Yeah I've run a 15GB DB in RAM before, but never thought about going through the penalty of forcing RAM access through filesystem calls before.

Re:Out of curiosity

[temojen]

Filesystem calls are a pimple compared to the mountain of disk seeks required if access paterns to the database are random. They may also make the difference between using an off the shelf database and writing one yourself.

Re:Out of curiosity

[Florian Weimer]

If you need a very fast database (even for random access), another option would be a solid-state disk. Of course, they aren't exactly cheap, either.

Re:Out of curiosity

[Anonymous Coward]

RAM DISK. He said it in the question. Ever run a 15GB DB in RAM instead of on a physical disk? It's very fast. There are a ton of options that would be good in RAM. Albeit if he were running a DB application more than one processor would probably be beneficial as well.

Many databases will cache information in memory without the need for a separate RAM disk. If the DB doesn't many operating systems will (like Solaris).

One has to question the large memory requirement on a single processor system. Disk times would probably be sufficient for the type of work that a single processor system could handle.

Re:Out of curiosity

[Anonymous Coward]

He's getting ready for Longhorn !
sin

Re:Out of curiosity

[Tsiangkun]

But, He wouldn't be limited by the amount of RAM, instead he could have up to $FILESYSTEM_LIMIT amount of space. Pretty much any mother board will work with it, and he doesn't need to go 64 bit unless he has to for other reasons.

Re:Out of curiosity

[anthony_dipierro]

Ever run a 15GB DB in RAM instead of on a physical disk? It's very fast.

Depends on the application. If you're running a database for an internet application, then a four disk RAID-5 with 10K RPM drives are fast enough to saturate a T3 anyway (and I'm talking about random access). Actually it's hard for me to think of an app which doesn't require more than one CPU and needs faster access than a properly configured RAID can give it. Just saying "a DB application" isn't enough to justify using that much ram. Maybe if you had a multi-terabyte database, and the indexes themselves take up 16 gigs, but then the key to the architechture is going to be building that multi-terabyte disk array.

Re:Out of curiosity

[John Harrison]

No, he qualified that by saying "probably". In any case I don't want to know how it will be configured, I want to know what he wants to use it for and what he thinks the benefits will be. /.ers are pretty smart and might be able to suggest something better to him.

Rather than saying, "I need this to solve my problem, and I'm not saying what my problem is!" it is better to say, "Here is my problem, I am thinking of doing this. How can I do that, or are there better solutions that I haven't thought of?" He'll get better answers that way.

If he doesn't know where to purchase a large memory system then it is possible that he doesn't really know if that is what he wants.

Re:Out of curiosity

[Anonymous Coward]

Because 64 GBytes aught to be enough for anyone.

Re:Out of curiosity

[pz]

Not a single sysadmin who actually deserves the title is going to run a db out of system memory.

Why?

Because if the system crashes, the db is lost.

Ergo, the system will either dribble output to magnetic media, or the db will be run off disk directly. Since people who design databases for a living understand caching issues, they'll probably do a much better job at managing 4+GB of main memory than someone asking for motherboard advice on Slashdot.

Sure, big memory will help this a lot, but the fact that the original poster is suggesting using a RAM disk means his position is suspicious.

Re:Out of curiosity

[mpcooke3]

Why have the overhead of using a database then? the application could just be allocated 15GB and hold it in memory.

Re:Out of curiosity

[kpharmer]

Actually, a well-tuned high-end database typically doesn't need to be completely in-memory to get 98% of the speed of one on a ramdisk.

For example, let's say that you've got 15 gbytes of raw data on db2/oracle/informix, as well as 4-8 gbytes of memory.

First step is storage: don't use raid-5 (requires extra writes), instead you'll want something like raid-10. Then ideally get 28-56 or so drives, and you can spread data, indexes, logs, and tempspace across all drives. Obviously 15k rpm ultra320 drives with a caching controller would be best (if you can't afford fibre).

Next make sure that your memory is tuned right - multiple gbytes set aside for buffering, sorting, asynchronous agents, etc. With these databases it's easy to ensure that all of your indexes are always in memory, that all writes to disk are asynchronous, and that most sorts are in-memory. The buffering should ensure that 98% or so of your disk access actually hits the bufer instead of the disk.

Given the above scenario a database running on a ramdisk could be faster - what? 0-5%? And in return for that possible speed increase it'll cost more and lack the reliability of the disk-based database. Probably not a good deal.

Re:Out of curiosity

[Aeiri]

Not a single sysadmin who actually deserves the title is going to run a db out of system memory.

There is a way (haven't done this myself) to send out the update/insert/delete to the harddisk immediately after it applies to the RAM, but the application responds to the webserver or whatever is calling it immediately after it applies to RAM. This creates a lightning fast database, with storage on a hard drive as well in case of a server crash.

Re:Out of curiosity

[Anonymous Coward]

ummm I bet you think mysql is a great database...

Re:Out of curiosity

[kpharmer]

> Actually it's hard for me to think of an app which doesn't require more than one CPU and needs faster
> access than a properly configured RAID can give it.

was wondering this too, how about this unlikely scenario:

1. very low concurrency application - in which you never or almost never have more than one connection at a time to a database.

2. you need the fastest possible response to the query.

3. you've got 15 gbytes of data along with 500 mbytes of indexes

4. your queries don't fit into any pattern - so normal database buffering seldom gives you above a 50% hit rate.

5. all access to data is via indexes, and is never for more than a few rows: so parallel queries won't help.

6. availability doesn't matter - you'll just get two of these, and they are read-only.

In this scenario I imagine that a ramdisk would provide a tiny performance margin over a properly designed disk-based database.

But the other 99 out of 100 scenarios would do better on disk. :-)

Re:Out of curiosity

[fishbowl]

Caching is often a bigger win than RAM Disk.

Profile your application before making such committments. You might be surprised at what you find.

What's interesting here is how securely "in the box" people are thinking. They are on a do-or-freaking-die mission to press a consumer architecture into service as a mid-range engineering system.

My shop uses Sun hardware for midsize applications like this, but we also run some Alpha systems (Both older DEC and newer HP).

We have a couple of Sun E20K but I don't know how they are configured exactly (I don't get to touch stuff that belongs to financial depts, nor would I want to :-) But I know these can go up to 1/2 TB, addressable by a single application. And you can't beat their IO.

But somebody who is thinking in terms of "What motherboard should I buy?" probably isn't prepared for the sticker prices of SunFire servers, EMC storage, etc.

When a skunkworks shop needs some large scale system on a budget approaching zero dollars, some creativity is needed. So we get into storage solutions like consumer RAID, and we hear questions like the OP, "what's the largest memory configuration on a commodity 64-bit microcomputer?"

Dual Xeon boards that take 32GB RAM are common enough, but I think that's 16GB per processor and I have no idea how it's addressed or whether any given application knows how to address it.

32GB in premimum RAM will run over $20 grand. Nothing compared to the half-million for an E20K, of course.

Re:Out of curiosity

[mobiGeek]

the mountain of disk seeks required if access paterns to the database are random.

No database application (or RDBMS) worth its weight is going to be doing "random data access"...and if the access truly is random, then the filesystem will be much *less* efficient than an RDBMS will be.

RDBMS's have things such as caches (dynamically sizing ones even), indexes, and clustered indexes to reduce the impact from applications that seek "random" data. But the overwhelming majority of applications do in fact have a pattern to their access.

Also, the size of a given database is usually irrelevant. Typical applications only access "recent" data, with batch systems or other reporting mechanisms generating reports against data outside of that "recent" portion. The "recent" data is typically sitting in cache.

Re:Out of curiosity

[fishbowl]

"Filesystem calls are a pimple compared to the mountain of disk seeks required if access paterns to the database are random. "

There are no physical seeks if the backing store is RAM instead of disks, at which point the fsio calls start to be more significant. I think that was the point the OP was making.

On the other hand, it may be much simpler to write an optimized fsio than to try to mess with memory management.

Re:Out of curiosity

[Anonymous Coward]

That is called write-through cache. If you're doing more than a couple of writes to the database a minute, it's bloody slow.

Re:Out of curiosity

[letxa2000]

Disk times would probably be sufficient for the type of work that a single processor system could handle.

Disk times would be "sufficient" because most people define what a single processor system could handle based on a disk-based, OS-based system. A RAM-based system is entirely different.

You'd be absolutely amazed at how much computing power we already have available on our single-processor systems. Get rid of Windows and, yes, even Linux and the sheer speed and power available is quite mind-boggling.

In most cases, the practical need for additional processors is not so much a limitation of the processor--which is lightning fast already--but of the operating systems that just consume so much resources to do what they do. And that's not a slam. The OS serves a very necessary purpose. But do some work on embedded systems and you'll realize just how much power there really is under the hood of even a consumer PC.

And when I say embedded, I don't mean the new definition of "embedded" that seems to be putting Windows or Linux in a small form factor but, otherwise, just a normal PC. Embedded used to mean getting into the nuts and bolts with a program that does its job and nothing more. No OS overhead, just pure code and hardware. That's embedded. And when you work at that level for awhile, you really begin to appreciate how much of a hit performance takes for the flexibility of running under an OS.

Re:Out of curiosity

[pz]

My point exactly: this isn't being run on a RAM disk.

Re:Out of curiosity

[Anonymous Coward]

Duh. Our dell server for our DB has 36Gb ram 8 processors and 16*4 162 gig U320 scsi drives.

our database is huge, it takes up 1/2 the raid 50 disk array. we typically use all 8 processors all the tije but the system rarely uses more than 50% of the ram. I have tried to force oracle to use more ram but it's moot. The scsi cards have 2 gig cache ram on them and our cache hits are huge compared to misses.

I see it as silly to spend a crapload on ram when you can get more from u320 raid. the performance difference is minimal when compared to risk of database loss or corruption running in a ramdisk.

Re:Out of curiosity

[Anonymous Coward]

Some databases are read-only.

Re:Out of curiosity

[Anonymous Coward]

Ever run a 15GB DB in RAM instead of on a physical disk? It's very fast.

It's also very stupid.

It's extremely dumb to keep the data files on a RAM disk. Better to let the DB use the RAM itself to cache data and queries and write data to stable media. A lot of thought has gone into making sure DBs don't lose data, and it's unlikely you (as an individual, or your team) will do it better.

Speed may be important, but making sure your data doesn't go "poof" is usually more important.

Just my $0.02 of course.

Re:Out of curiosity

[Anonymous Coward]

Sounds like one of "Jerry's kids". Clueless at best.

Re:Out of curiosity

[Anonymous Coward]

You are right but only for OLTP types of applications. Full table scans can't be buffered adequately.

Re:Out of curiosity

[gCGBD]

I could also imagine this being used as a cache server. If you are collecting data of some sort much faster than your disks can absorb it, you might need a lot of RAM (but not a lot of CPU) to take in the data (bursts?).

Re:Out of curiosity

[kpharmer]

> You are right but only for OLTP types of applications. Full table scans can't be buffered adequately.

yeah, cache hits are more critical in oltp - where a single percentage point in the hit rate can be a noticable difference.

but even on non-oltp applications (data warehouses, data marts, olap, whatever), it makes a difference - and you (typically) get can get good hit rates. I'm looking at stats on a 500 gbyte data warehouse right now, and see:
1. dimension tables: 99% cache hit rate
2. summary tables: 98% cache hit rate
3. fact indexes: 99% cache hit rate
4. fact tables: 92% cache hit rate

In the above list the vast majority of data is in the fact tables - where over 90% of the queries are coming out of memory. Since many queries actually hit the summary tables instead of detail, those queries are getting a 98% hit rate. And when the cache is missed - it doesn't necessarily mean a tablescan: this particular database is partitioned via db2, so a cache miss generally just results in a partition scan of 2-4 seconds.

So, while you're right - caching is tougher in a reporting environment, it still helps enormously in most implementations. At least in my experience.

ken

Re:Out of curiosity

[Anonymous Coward]

If you need a very fast database (even for random access), another option would be a solid-state disk. Of course, they aren't exactly cheap, either.

Solid state disks are nowhere near as fast as a RAM disk residing within system RAM in sequential transfer terms. The bottleneck of going through the PCI bus and perhaps then through an IDE and SCSI bus kill transfer rates. I say perhaps because you can get solid state "disks" on PCI cards or actual units that look like 3.5" drives complete with SCSI or IDE connections, but are full of RAM.

I guess the huge difference in performance between external solid state disks and RAM disks within system RAM might be reduced dramatically when comparing random access times.

Out of interest, I played around with RAM disks (mfs) within OpenBSD and NetBSD and found that in NetBSD the RAM disk was not as fast as data from a real disk that had already been cached. OpenBSD did not exhibit this, I guess due to the current lack of UBC.

Re:Out of curiosity

[corngrower]

Still, can't see the need for such a large memory. With an appropriate disk configuration, interleaving writes among the disks, one should be able to write data to disks as fast the system would be able to acquire it. And with disk speeds in the 150Mb/sec range, it's hard to imagine a system that would be acquiring data so fast.

Re:Out of curiosity

[ottothecow]

It's possible that he asked these questions elsewhere (with a small group of professionals etc.) and got an answer. The only problem is that none of them knew where to get a system quite like what they want even though the technology is there.

Thus they turn to /.

Re:Out of curiosity

[pantherace]

Dual Xeon boards that take 32GB RAM are common enough, but I think that's 16GB per processor and I have no idea how it's addressed or whether any given application knows how to address it.

It's addressed via PXE/"Highmem" extensions, allowing the p6 core to use the 36-bit real memory bus to remap that into the 32-bit virtual address. It requires any app using more than 4 gig to do some "weird shit" with memory requests, as well as in general only having 3-3.5Gig available at any one time (OS virtual overhead), with parts of the program duplicated a number of times. (Or at least that's how it was, Hopefully it's improved since then)

And we aren't even considering how horribly that would perform on anything more than a dual. Shared bus = easy 'multi-core' processors = Shit for scaling to more processors than 2. Any server chip is going to have better than that, hell, the original athlons did (because they used an alpha ev6 bus)

One of the few advantages Sparcs have over Xeon, has evaporated as Intel follows AMD's x86-64 spec, and replaces the memory with 48-bit virtual (Opterons/A64s only do 40-bit physical. Oh dear, by the time I've got that much ram, I'll have a better processor!)

Re:Out of curiosity

[devilspgd]

You've heard of usenet? -- Usenet is already approaching 150Mb/s, plus you're not just writing, you're reading too.

Exceeding 150Mb/s isn't difficult.

Re:Out of curiosity

[devilspgd]

Sure, and that's important if you're writing to the DB much.

This isn't always the case if the DB is being read constantly but updates infrequently.

Re:Out of curiosity

[tigersha]

There are lots of scenarios where you have a large fixed database which is used for analasys: ie, read only.

Think stock time series, scientific data read by some instrument, census data, geographic information systems. Quite often people just want to look and examine the stuff and update it never or once in a while, in which case you can reload the db.

Or you could have a large dataset which is appended to (stock time series, some instrument reading stuff) and writing to both the master copy on disk and the memory database. If anything goes wrong, reload the master copy.

Re:Out of curiosity

[Anonymous Coward]

he isn't running a database.. he's got plans on a self-destructing web/ftp server for p0rn and warez. run everything off of ram, so when the cops come and bust up party... poof goes the evidence when they unplug the server to haul it away.

Re:Out of curiosity

[Daengbo]

I like your suggestion, but I suspect that an Ask Slashdot question phrased that way would mostly get "You want US to do your homework for you? That's what Google is for..." responses. I could be wrong, though.

Re:Out of curiosity

[pegr]

Except for the concurrency bit, your describing ATM and credit card authorization processing. Response times are supposed to be milliseconds including communication latency. If you don't keep up your service agreements, you pay big bucks to the network... Lots of strange custom boxes doing this work on the backend.

Re:Out of curiosity

[r.muk]

He could tell you but then he'd have to shoot you soon after.

Re:Out of curiosity

[John Harrison]

It seems to me that what he asked for is much more of a "That's what Google is for..." than what I suggested is. In any case, posting the question the way I suggested is more likely to generate a conversation that is interesting to me than, "I don't want to buy a large memory system from IBM or Sun. How can I build my own."

Re:Out of curiosity

[Daengbo]

I'm sorry. I wasn't giving a critique of your suggestion, just bashing slashdot mindlessly.

Re:Out of curiosity

[Anonymous Coward]

There's no reason to spend half a million on a new E20K if you need 32 - 64 Gb of RAM. YOu can source second-hand/grey/refurbished SPARCs, like the smaller E4000's or even E10K's way below that, and probably cheaper than any new 8-way Opteron with 32 Gb RAM, just make sure you know the architecture before you buy, and often these specs and questions come up when apps bound to the x86 platform need more oomph, so SPARC won't really do you much good if that's the case.

big ram server..

[gl4ss]

is not going to be cheap.

simple as that. you will have fork over premium for something that would support what you feel is big ram.

offtopic: anyone know how to stop windows from swapping when there is 500mb+ of free ram? it's really annoying, and just putting the swap to 0 on all drives doesn't really solve the problem either(and some soft freak off from it, this is on XP). I hate having 1.5gb of ram and only seeing half of it used regularly while having windows swap horrendously.

Re:big ram server..

[Tester]

offtopic: anyone know how to stop windows from swapping when there is 500mb+ of free ram? it's really annoying, and just putting the swap to 0 on all drives doesn't really solve the problem either(and some soft freak off from it, this is on XP). I hate having 1.5gb of ram and only seeing half of it used regularly while having windows swap horrendously.

You ough to know that 500megs of ram is enough for everyone.

Re:big ram server..

[Anonymous Coward]

Try moveing the swap file on a ramdisk. Horrendous, I know.

Re:big ram server..

[Anonymous Coward]

Insightful my ass. Some of us have real JOBS that require Windows, and saying "Sorry, I missed the deadline because of problems with Wine" are a fast way to get yourself fired.

He asked about how to fix it on Windows. How about I respond to every problem with Gnome and KDE or Linux with "Maybe you should switch to a supported platform like Windows"? We'll see how often that tripe gets modded "Insightful".

Re:big ram server..

[Anonymous Coward]

search on google, there's some registry key u can mess with

Re:big ram server..

[Anonymous Coward]

You could try setting Windows to optimize for background services in the virtual memory settings. In theory, this means Windows won't swap out applications in the background because they are doing stuff. In practice, I haven't seen much of a change.

Re:big ram server..

[thdexter]

I've always read it's inadvisable even if one has a lot of RAM. Here's some quotes from this page:

Can the Virtual Memory be turned off on a really large machine?

Strictly speaking Virtual Memory is always in operation and cannot be "turned off." What is meant by such wording is "set the system to use no page file space at all."

Doing this would waste a lot of the RAM. The reason is that when programs ask for an allocation of Virtual memory space, they may ask for a great deal more than they ever actually bring into use -- the total may easily run to hundreds of megabytes. These addresses have to be assigned to somewhere by the system. If there is a page file available, the system can assign them to it -- if there is not, they have to be assigned to RAM, locking it out from any actual use.

--

Why is there so little Free RAM?

Windows will always try to find some use for all of RAM -- even a trivial one. If nothing else it will retain code of programs in RAM after they exit, in case they are needed again. Anything left over will be used to cache further files -- just in case they are needed. But these uses will be dropped instantly should some other use come along. Thus there should rarely be any significant amount of RAM 'free'. That term is a misnomer -- it ought to be 'RAM for which Windows can currently find no possible use'. The adage is: 'Free RAM is wasted RAM'. Programs that purport to 'manage' or 'free up' RAM are pandering to a delusion that only such 'Free' RAM is available for fresh uses. That is not true, and these programs often result in reduced performance and may result in run-away growth of the page file.

Re:big ram server..

[d1g1t4l]

There is an option called "No Paging File" in
Control Panel
\System: Properties
\Advanced
\Performance: Settings
\Advanced
\Virtual Memory: Change

Re:big ram server..

[baadger]

Apparently you can disable paging, to do so

Edit the Registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Contro l\Session Manager\Memory Management\DisablePagingExecutive to 1.
Exit the Registry and reboot.

There are many ways to optimize your swap file and lots of tweaks out there to reduce harddrive activity.

Here is one useful looking tweak set

Re:big ram server..

[Anonymous Coward]

I bet you would have got modded up if you used Linux.

Re:big ram server..

[pla]

Management\DisablePagingExecutive to 1.

That only stops Windows from paging out parts of the OS itself (like the kernel and currently-idle device drivers).

On XP, you can just set the pagefile size to zero. And yes, it does work, in the sense that it stops Windows from hitting the disk twice per second even when doing nothing at all.

On 2000, you need to assign the pagefile to a RAMdrive (and one that supports NTFS and doesn't identify itself as RAM, since Windows won't normally let you put the pagefile on a RAMdrive).

Finally, for anyone disabling paging - Also disable memory dumps and automatic reboot on bluescreens, or you will regret it.

Otherwise, it works just fine, regardless of what the naysayers and MS Knowledgebase fanboys might say. Quite a boost in performance, too.

Re:big ram server..

[Fulcrum of Evil]

Doing this would waste a lot of the RAM. The reason is that when programs ask for an allocation of Virtual memory space, they may ask for a great deal more than they ever actually bring into use -- the total may easily run to hundreds of megabytes. These addresses have to be assigned to somewhere by the system. If there is a page file available, the system can assign them to it -- if there is not, they have to be assigned to RAM, locking it out from any actual use.

Not true. When a program allocates a large chunk of ram, all it gets are the page table entries. Only when it actually writes to a page does the ram get used. This means that I can allocate 1.5G of ram and scribble in half of it and only be using 750M

Re:big ram server..

[Anonymous Coward]

offtopic: anyone know how to stop windows from swapping when there... Try using the performance tweaks at: http://www.worldismine.org/computers/tips.php

Re:big ram server..

[StarWreck]

You ough to know that 500megs of ram is enough for everyone.

Well, you ought to know that 64K is enough for anyone. Don't you listen to our god Bill Gates?

Re:big ram server..

[Junta]

Here's a counter argument. My knowledge is mostly based on linux.

For very special conditions (i.e. custom hpc apps) where you know almost *exactly* how much resource is going to be used, some of the platform design aspects can impact performance in writing to disk even if swapping is never needed.

In general:
-Inactive pages are speculatively copied to disk, but not overwritten unless needed.
-If application needs to access memory if both in memory and paged out, no swap reads necessary, no harm, no foul.
-An application comes and needs to alloc memory. If inactive pages are using memory, but marked disposable as they are already copied to swap/pagefile, they are overwritten and the allocation of memory is quick.
-If nothing was speculatively swapped out, memory allocations in the above situation would crawl, being blocked on I/O writes to disk. This is a tremendous price to pay if memory usage indicates this to be a realistic possibility. Keep in mind the OS has no way of knowing what you are about to do, and can only make assumptions based on general principles.

However, those I/O write operations can be expensive, particularly if otherwise the system isn't needing to touch the storage at all during this time. Invoking the storage driver to do lengthy writes of pages to swap can be devastating to performance. If your environment is *very* tightly controlled and deterministic with respect to resource usage (i.e. customized apps running with little else in the background, not serving and not doing typical desktop/workstation stuff), the OS assumptions/worst case provisions hurt more than help.

Nowadays linux is pretty tunable with how aggresive it is about copying pages to swap, but even still it happens pretty often and those writes can affect HPC applications.

Note that you are still write about the general case, and that people shouldn't flip out about seeing swap/paging file usage while they still have free memory, or even seeing almost no free memory (actually, for those really concerned in linux at least, ignore the first row free number and look at the second row under free, it shows free memory not counting disk buffers and cache, and is a more true reflection of available physical memory).

Re:big ram server..

[Anonymous Coward]

Just out of interest - I have a couple of mini-apps that I know will never require more than a few K of heap-space. Is there anyway to tell Linux not to allocate more than X amount, as I know in advance that it will not be needed (whereas Linux, presumably, won't)? Would this increase performance, if small, non-heap-hungry apps in general (e.g. the large amount of services commonly left running in modern desktops) were able to say in advance that they would require only a small amount of RAM?

Re:big ram server..

[man_of_mr_e]

Windows will run very slowly without a swap file in many cases.

Linux and Windows have fundamental virtual machine differences. Linux swaps pages to disk, Windows keeps a page backed store. What this means is that every memory address in Winodws is backed up by some kind of virtual memory. For code and some data segments, the "page" is the executable file on disk itself, so it doesn't need to ever copy those pages to swap. (Incidentally, this will show up as "swap file in use" even though it's not in your swap file).

Windows also uses swapping for memory mapped files, of which many apps make use of.

Finally, if there is no swap file, there is no "reserved" memory either. That is, memory that is allocated but not used. This can cause some apps to freak out if they reserve more memory than you physically have.

Re:big ram server..

[merlin_jim]

That's not swapping, that's paging.

In Windows, paging is done concurrently with memory writes; this way when you swap the memory out, it doesn't require you to actually write the memory contents to the drive, as they're already there.

It also helps in case of kernel panic (it allows Windows to report its entire state when it comes back from a blue screen)

IIRC, there is no way to disable this behaviour, unfortunately.

Re:big ram server..

[x_codingmonkey_x]

offtopic: anyone know how to stop windows from swapping when there is 500mb+ of free ram? it's really annoying, and just putting the swap to 0 on all drives doesn't really solve the problem either(and some soft freak off from it, this is on XP). I hate having 1.5gb of ram and only seeing half of it used regularly while having windows swap horrendously.

I think the clear answer to your question is ...
...
...
...
...
Linux! :P

Re:big ram server..

[Anonymous Coward]

That's 640k!

Re:big ram server..

[BouffeMoiLaChatte]

Windows virtual memory sucks.

Never tried to copy a big iso file over local network?

windows try to put it all in memory , eating instantly all your precious physical ram and get your machine on his knee , even on those costly high end machines.

i never found any great solution for this.
only one program do the trick : total commander with disk cache disabled for big files.
try this at home. and enjoy the trick. /Z

Re:big ram server..

[Anonymous Coward]

Yes, I do it too, on anything over 512mb. It is much faster, especially on a laptop with slow drive. The only problems were:

1) warcraft crashing running out of ram on a game... solved by upgrading to 1G ;)

2) adobe photoshop refuses to run without a swap file... idiots. Solved by switching to the gimp.

Re:big ram server..

[mczak]

That's called overcommit. You can switch this off or on with linux (it's usually on, but there might be cases this is not desired behaviour and you absolutely have to guarantee that enough memory is actually available). I'm not sure what windows does, it would surprise me if it wouldn't overcommit memory.

Re:big ram server..

[Anonymous Coward]

Windows XP doesn't swap, it uses a page file, which preemptively makes copies of the least used pages in your ram to your disk so that if there is an unexpected jump in memory usage, it can wipe those pages from memory quickly and just put them back from disk when they are needed. There really isn't a performance hit since it's a pretty much passive system in reference to RAM writes. Adrian's Rojak Pot has a great guide on Windows virtual memory at http://www.rojakpot.com/default.aspx?location=3&va r1=143&var2=0

-Julius

Re:big ram server..

[yppiz]

You do pay more for a dual processor machine, but it's not an obscene price. You can put together an 8GB RAM machine for well under $4k. Memory is the main cost in these systems.

As a data point, I built a machine around a Tyan S2882 motherboard with 8GB of RAM for just over $3000. This is a dual processor machine with 8 1GB modules. Half this cost is memory. This price includes the assembly cost, shipping, and tax.

Going to 16GB means using more expensive 2GB modules, meaning memory is an even greater percent of the system cost.

Going to 64B means going to a quad board with 2GB modules. You'll add perhaps $1000 for the upgrade to the quad CPU motherboard and additional CPUs, but the RAM is probably now 65% of the system cost.

--Pat / zippy@cs.brandeis.edu

Re:big ram server..

[jgold03]

Every process has its own independent address space. This address space is "ideal" in that it appears like the machine has 2^32 or 2^64 bytes (depending on your processor-type) of physical memory, regardless if you do or don't. When the process needs to be able to actually access a memory location, the OS creates a mapping from that specific 4 KB region to some random 4 KB region (a "page") of physical memory. If we don't have any physical pages to distribute, thats when we have to start using the hard disk for storage, which is called swapping.

Now as to why Windows would be swapping if all you have open is AIM and Firefox and you have 2 GB of ram... well you'll just have to ask Microsoft.

Re:big ram server..

[dayton967]

One solution that you could do is make a ram disk and put the swap file on that.

Re:big ram server..

[Anonymous Coward]

Have a look @ CachemanXP - I sit here with 770MB free after boot, right now after 3 days uptime and several hibernates, a lot of Firefox use and lots of networking I'm @ 686MB free in the SysTray display. I have never had to click the "Free RAM now" button.

The URL is Outertech.com.

Posted AC because of modpoints

Re:big ram server..

[cpct0]

My non-Mac computer is currently running Windoze XP. I went to 1G of RAM when I know that usually I use 500M. Sometimes, I go to 700-800. And I disabled the virtual memory.

Second computer I did that. So far, I find that a worthwhile investment.

Beware, though. On WinXP, you do a malloc of 64 megs, it will TAKE 64 megs of RAM if you disable the virtual memory. If you do a malloc of 64 megs and never access it with virtual memory on, it will take minimal space. Some software assume that. So make sure you run the task manager after a good day of work to see the dedicated (peak) if it fits on your RAM, add up a good 25% to that and you'll know if you are ok.

Mike

Re:big ram server..

[kyhwana]

Don't forget windows will use unused (for apps/data) physical RAM for disk cache..

Re:big ram server..

[B4RSK]

AC Wrote: Oh for fuck's sake, like Linux is any better. As soon as you install KDE or GNOME, plus a selection of apps necessary to get close to the functionality of XP, it takes up just as much memory. Maybe more, as I hear XP will comfortably run on 64MB RAM - would KDE even fit into this amount of RAM?

I know you're just a troll, but have you ever tried to run XP (or 2K for that matter) on 64MB?

"Painful" doesn't even begin to describe the experience. It's so agonizingly slow as to be useless.

Re:big ram server..

[mindstrm]

Err, yes, it's a page file....

I've seen lots of justifications for why having the pagefile enabled is good... but in the end, if you have enough memory for the tasks you are going to perform, windows performs faster with the pagefile disabled completely.

Re:big ram server..

[mindstrm]

s/many cases/someo very specific weird cases.

I've run windows without a swapfile for years, doing a huge variety of tasks. It runs fine, and is indeed faster without swap (page, whatever).

Re:big ram server..

[edwdig]

Swapfile, pagefile, same difference. Linux systems use a swap partition - do you really think that means Linux doesn't do paging?

Try running an XP box 24/7 for a while. I leave my work machine on 24/7, and after its been on for a few days, it will start swapping things out overnight. There aren't any active processes running when I leave at night, yet the next morning each application takes 30+ seconds to load back into RAM when I click it's taskbar icon. Once everything is loaded back in, I have no more issues for the rest of the day.

My guess is once your total memory usage gets above a certain percentage of your RAM, XP will automatically page out to disk applications that are idle for long periods of time.

Re:big ram server..

[kschawel]

offtopic: anyone know how to stop windows from swapping when there is 500mb+ of free ram? it's really annoying, and just putting the swap to 0 on all drives doesn't really solve the problem either(and some soft freak off from it, this is on XP). I hate having 1.5gb of ram and only seeing half of it used regularly while having windows swap horrendously.

I know exactly what you mean and I don't think that many people understand. When I have Firefox minimized for a few hours, Windows likes to put the whole program in swap (or something) because when I try to restore it, it takes like 30 seconds. There is a way to change that. In the [386enh] section of your system.ini file, add:

ConservativeSwapFileUsage=1

That seemed to help me. The site that I got this from is here, but it seems to be down now.

Re:big ram server..

[SunFan]

Windows XP doesn't swap, it uses a page file...

So, you're saying Windows XP doesn't append the disk storage to it's virtual memory space? That's a fairly inefficient way to use storage, if true. Solaris (Linux?) appends swap space to the physical RAM, so I can literally have a program use nearly RAM+swap amount of space without a hitch. Solaris' /tmp directory is also mounted as a RAM disk into virtual memory, which makes for an awesome place to put intermediate files from scripts. Solaris also uses all unused physical RAM as a disk cache.

I know this sounds like a solaris-solaris-solaris fanboy post...but it just seems that Windows XP is still seven years behind UNIX in kernel-land.

Re:big ram server..

[japa]

> Windows XP doesn't swap, it uses a page file, which preemptively makes copies of the least used pages in your ram to your disk

I have 1 gig of memory in my home PC and not that many applications or services running on it, it should have lots of free memory. One day it swapped about 30 secods when I tried to get minimized processes back to foreground. That kind of pissed me off and I started to investigate why does my PC swap, how to read the memory information given by task manager and do I need more memory.
Unfortunately, I can't find the URL any longer, but it was some m$ developer forum (think it had something to do with sql server), where I read that the swapping algorithm is optimized for systems with 128 or 256 megs of ram. For systems with more ram, it's too agressive. You can see that yourself: Have some applications open. Look at task manager, processes view. Sort processes by memory usage. open opera, word or some other app that uses nice amount of memory. Now press the minimize button. The process is no longer listed using that big amount of ram and windows will swap it out. Doesn't matter how much you have free ram! That's totally stupid! Since then I've set my system to run without swap file and had no problems and enjoyed increased speed.

s/swap/page

Re:big ram server..

[NoRemorse]

if you slipstream your cd with a program called nlite it will give you the option to disable the paging/swap so it runs off of ram :)

Re:big ram server..

[gregfortune]

Yep, 'cause "Sorry, I missed the deadline because the latest Nimbda variant took over my system" sounds so much better.

Re:big ram server..

[mormota]

It does overcommit for sure. I turned off my pagefile as an experiment on my windows xp box, and I saw page faults in the task manager when I fired up notepad. (this means that page table entries were allocated for notepad, but no actual memory, which caused a PF at the firs access)

Re:big ram server..

[Gr8Apes]

Windows isn't 7 years behind, it took a wrong fork 15 years ago.

Re:big ram server..

[afidel]

regedit: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Contro l\Session Manager\Memory Management\DisablePagingExecutive set to 1 to disable kernal swapping to disk on large memory systems.

Re:big ram server..

[Anonymous Coward]

I have nt 4.0 but i think it's the same on windows xp.
MmAdjustWorkingSetSize in ntorknl.exe calls _MiEmptyWorkingSet@4 if it is called with (-1,-1).
You have to disable that by patching the kernel. Make the function return if it is called with -1,-1.
win32k.sys calls adjustworkingsetsize every time you put an application in the background.

Re:big ram server..

[default luser]

I'll agree with you about XP, but 2k was designed to work with 64MB ram.

For example, a default system load of 2k on my old P133 (VX chipset, 64MB ram) allocated 43MB of memory on load (this includes drivers for everything), and this included IE resident. It was perfectly usable as an email / browsing machine, provided you didn't browse image-heavy sites or need flash / java. After turning off the transparency transition and mouse effects (two options unchecked), the interface even felt peppy.

Currently, the machine is doing duty as a network router / firewall with Debian...headless. I wouldn't try to run Gnome on it, under any circumstances. I run Gnome on a Celeron 800 with 256MB ram and it feels sluggish.

Re:big ram server..

[smitty_the_smith]

This may sounds very simular... you could make a large ramdisk and use it for for "virtual ram" or just switch to linux or win2k or something...

Funny how...

[Anonymous Coward]

the biggest advantage of 64 bit processing is memory access and x86-64 falls flat on its face. Way to go AMD.

overkill, but...

[Bad Boy Marty]

I really like the specs on the Tyan S4882. Quad Opteron, 32GB RAM, and lots of stuff you probably don't want/need. It's a sweet motherboard, nonetheless.

Re:overkill, but...

[Anonymous Coward]

With dual-core CPUs! Mmm... eight cores. This borders on geek porn.

Re:overkill, but...

[Florian Weimer]

I really like the specs on the Tyan S4882. Quad Opteron, 32GB RAM, and lots of stuff you probably don't want/need. It's a sweet motherboard, nonetheless.

I doubt it's really overkill. To reach the 32 GB, you must use 4 Opteron CPUs because you can't get a board/CPU combination which supports more than 8 GB per CPU.

Re:overkill, but...

[agw]

I really like the specs on the Tyan S4882. Quad Opteron, 32GB RAM, and lots of stuff you probably don't want/need. It's a sweet motherboard, nonetheless.

No, it is not.
Just look at the diagram at ftp://ftp.tyan.com/datasheets/d_s4882_100.pdf. Those four SATA channels must share the bandwidth of 133MB/s with all other legacy PCI cards, USB and the graphics controller.
Also the Dual SCSI-320 and the Dual GB NICs share the same 533MB/s bus bandwidth.
This means, if you want fast I/O, you have to buy an extra PCI-X card for the other PCI-X bus.

Any reason why you are building it yourself?

[Anonymous Coward]

IBM xSeries 445. We have around 50 of these IBM servers at work and they support up to 64 GB of RAM. Any reason why you are trying to build a server out of cheap commodity parts? Save yourself the trouble and buy one of these.

Re:Any reason why you are building it yourself?

[Ryan Amos]

Very, very good advice. PC parts (especially very high end stuff like this guy is looking for) are usually at a premium on the consumer market anyway. With an IBM server you get support and a guarantee that everything you buy will work properly under Linux. With bleeding edge high-end stuff, this is not always the case, at least not immediately. Try pricing out an equivalent machine from commodity parts; and remember that for a server you're going to want ECC registered memory. Also remember that, if this is for a business, they are essentially paying you by the hour to build a PC. It's probably cheaper to pay an assembly worker to build a computer than a system admin (and if it's not, you should ask for a raise ;) Like the saying goes, it's only free if your time is worthless.

This goes for home-built systems as well. Often it's just cheaper to buy a system built in taiwan than to build your own out of the exact same parts. I'm all for geeking it out and building something to your own specs, but for gaming machines or just basic desktop apps a pre-built system saves a lot of headaches.

Re:Any reason why you are building it yourself?

[Anonymous Coward]

Informative perhaps, but flippant as hell.

"Any reason you are trying to build a server out of cheap commodity parts?"

Hmm. There must be a reason somewhere. Let's think this one over real slowly and try to come up with a possible answer on why someone might try to save money. Hmm. It looks like a tough one. I can see why you were stumped.

Hmm. Why why why. Why would he do that?

Hey. I got it.

Maybe . . . maybe he was thinking about . . well you know . . . perhaps he could spend the money on something else?

What do you think?

Re:Any reason why you are building it yourself?

[Duncan3]

Hrm, well, I don't know, maybe it's the 1000% markup IBM uses? Or that the 445 is nothing like what he asked for?

Re:Any reason why you are building it yourself?

[Rooked_One]

> - but for gaming machines or just basic desktop apps a pre-built system saves a lot of headaches. I've got to WHOLEHEARTEDLY disagree here. If I were to buy a pre-assembled gaming rig, it would cost about 100% more than if I were to build it myself - even if it had the exact same components in it. Don't believe me? Just go look at alienware or something equivilant. Now for a desktop only machine for granny, an emachine is fine and dandy, but you'd never catch me buying a pre-built computer.

Re:Any reason why you are building it yourself?

[Anonymous Coward]

perhaps he could spend the money on something else?

As his time is free to debug stuff that other people have done?

Plus what sane company is going to go with some homebuilt machine that, presumably, is important to the business when product is out there?

Re:Any reason why you are building it yourself?

[NutscrapeSucks]

Another thing to keep in mind is that for high-end motherboards, you can't just pricewatch random power supply and RAM. You need to be very careful to select components that are fully compatibile, and this stuff comes at a premimum.

If the poster is serious about building this box, I would recommend he ask over at Ars Technica or some other board where people have detailed knowledge of these highend motherboards.

My datapoint is that a couple years ago I looked into building a dual Xeon, and ended up finding a complete system on Dell Outlet for only a couple hundred dollars more than the parts cost.

Re:Any reason why you are building it yourself?

[Elshar]

I can very easily find better quality parts for the same price or even cheaper than the mass-produced, lowest-possible-price pieces of tapeworm infested animal excrement parts that wholely make up the boxes you can buy from taiwan.

Also, considering that any geek worth thier salt can throw any machine of any moderate complexity together and working perfectly (minus os install and possible driver hunting/install) in roughly 60 minutes or less, why would you pay the same amount for crappy parts and even crappier 'support'?

Yes, I like runon sentances :)

Re:Any reason why you are building it yourself?

[Anonymous Coward]

I found monarchcomputer.com to be quite good at specing out and building a linux-based system for me. You might wanna look into monarch.

Re:Any reason why you are building it yourself?

[Grey_14]

I Agree with you, except for your example, Alienware is terribly overpriced, partially for their brand name, check out dell, gateway, whoever, and find a gaming machine from them, first thing you'll notice? Most of their "Gaming" Machines are underpowered, if you want a powerful gaming rig from a name brand, you pay through the nose, but if you build it yourself, you save a lot of cash

Re:Any reason why you are building it yourself?

[maraist]

Plus what sane company is going to go with some homebuilt machine that, presumably, is important to the business when product is out there?

Start ups, where dollars matter.. What I've seen people linking to are $3,000 machines at their cheapest.. Then the IBM and other solutions are probably in the tens of thousands of dollars.. Yes, what is your time worth? If you work for a fortune 500 company, then it and the consequences are worth a few measly 10 grand. But if as I suspect the auther DOESN't need CPU horsepower, DOESn't need high-end disk arrays.. They just need memory, then you do the math. The likelyhood of the project needing a 20,000% increase in price ($500 to $10,000) to save a few man-hours...

I quote $500, because a premium MB + 1G memory + 3000+ CPU can be had for much under $500. If you want a rack, sure you're going to pay more, but considering that he was bitching about now wanting a multi-CPU server, I seriously doubt he was looking at racks.

And why would you want memory instead of hardware? Because there are an increasing number of applications which are memory-space intensive, not CPU or even disk intensive.. Running VMware sessions, user-mode-linux, running many-many java servlet engines (each JVM hapilly can consume 1Gig of memory), caching your database. These are various instances that I personally run into on a regular basis where memory is NEVER enough. Especially when I hear people claim "memory is cheap".. No it isn't! Not when you hit artificial limits like 4 DIMM's per AMD64 CPU. It's a fixed resource where a programming environment which assumes an infinite amount of available memory can in certain configurations and situations become a huge bottle-neck..

The most reasonable solution is to purchase multile machines to divide the work.. But this doesn't scale too well as your administrative over-head (as well as cost of UPS's, power/heat constraints, shelf/rack space, etc).

One of the points of going to a 64bit architecture was to remove the architectural limit on addressing. Well, much like the early moterola's, you can address significantly higher than the specific architecture can facilitate.. This is to be expected, but to realistically have an upper bound of 16 chips or 64Gb (assuming 4Gig chips, which I don't even know if exist; certainly not whole-sale) is just kind of sad (considering 1Gig servers are the norm, and 64bitness is well over a decade old)

A 2G chip costs at least $200 (for a crappy server-need-not-apply brand). Half gig's start at $30 (or $120 for the 2Gig counter-part). Server-worthy half-gigs start at $60. Often, however, vendors that sell memory with their system charge enormous premiums (nearly a thousand dollars) for upgrading to the larger chip sizes. If instead of being limited to 4 slots / CPU the architecture facilitated 8 or 16 slots, then vendors wouldn't have as much justification for uprading to the more expensive DIMMs and thereby passing the premium on.

It's simply impractical to afford purchasing a system with 16Gig of memory.. There are many other cheaper solutions which provide a significantly faster machine (including using a swap-file and RAIDing the hell out of it). But this is insane, there is no good reason why a larger memory solution should cost so much as to cause system administrators and developers to have to spend hundreds of thousands of development/deployment dollars working around such limits.

I recognize that you can't just put more memory slots on a BUS, but I don't see why the x86-64 architecture didn't facilitate fully banking the memory out into 4 parallel busses of 4 slots each.. To my knowledge, the most that is supported are two banks of two slots each.

The AMD64, while a GREAT chip series, seems just a little on the cheap side to me.

Re:Any reason why you are building it yourself?

[Fweeky]

AMD64's support 8 slots per CPU; that just happens to be a bit much to fit on most motherboards.

Re:Any reason why you are building it yourself?

[(negative video)]

I recognize that you can't just put more memory slots on a BUS, but I don't see why the x86-64 architecture didn't facilitate fully banking the memory out into 4 parallel busses of 4 slots each.

Pin count. Opteron already has 940 pins, which is getting pretty ridiculous in terms of physical size, number of tiny fragile pins, reliability, power consumption to drive those pins at high speed, and chip area for the massive output driver transistors. Adding two more memory buses would require around 300 more pins at which point it gets absurd.

Given the high cost of adding buses, and that most users don't need them, AMD made the right choice IMHO.

Re:Any reason why you are building it yourself?

[Anonymous Coward]

just have to say, posting from an emachine with an athlon 64 and ati 9600 laptop, their higher end stuff is pretty nice.

Re:Any reason why you are building it yourself?

[Anonymous Coward]

Spend the money on something else like what? He said the machine was for work. Do you have any idea how much 64 GB of ECC RAM cost? A lot more than the cost of 1 IBM server. If he is running an application that needs this much RAM then money should not be an issue.

Re:Any reason why you are building it yourself?

[Infinite Entropy]

Dell and thier ilk exploit the general population's ignorance of graphics cards by always using a really slow one even in what are supposed to be high end machines.

Re:Any reason why you are building it yourself?

[Rich0]

This goes for home-built systems as well. Often it's just cheaper to buy a system built in taiwan than to build your own out of the exact same parts. I'm all for geeking it out and building something to your own specs, but for gaming machines or just basic desktop apps a pre-built system saves a lot of headaches.

I was able to put together a pretty decent AMD64 system last year for about $600. I saved a little by recycling DVD, floppy, and video card (not a gamer). I didn't recycle the parts that mattered most (power supply, motherboard, CPU, RAM, hard drive).

First, there aren't many vendors out there a year ago which even sold AMD64 systems. Second, for the performance level I got you'd easily spend double to triple with a major brand. Now, I value my time, but I don't value an hour or two of it at $1000+. Besides, it was fun, and I got a chance to catch up on the latest gear.

When you buy a name-brand PC, you get whatever the latest advertising buzzwords are. Probably lots of GHz. Probably a big hard drive, and probably even "7200RPM." You'll get a fast frontside bus since that is a buzzword now. You'll get lots of RAM. You won't get a decent motherboard, since most people don't know what one is. You will get a power supply that cost $20 which you'd never use for a comparable system since the only way the major brand got away with this is by buying 500 brands of $20 power supplies and figuring out which 2 of the 500 actually didn't burn out in a week. You won't get a hard drive with decent cache (cache - what's that?). You'll get a nice video card if you pay more, and you may or may not have an option to save money on a cheap one. You'll get some licenses for MS Works and Windows XP which you may or may not use (possibly the later, probably not the former).

Many benchmarks have shown motherboards to have a significant impact on system performance - sometimes equivalent to going up a notch in processor speed. Instead of spending $400 more for that extra bleeding-edge MHz level, you could spend $20 more on a motherboard which isn't a piece of junk.

I didn't spend a premium on any of my parts. However, I didn't get junk either.

I've been given brand-name PCs from friends who have had them stop working. I fix them up for the kids, when it makes sense to even do so. Often half the parts in them are just junk. I've read horror stories on Compaq power supplies which don't follow standard ATX pinouts despite having ATX plugs.

When buying commodity office workstations I'd just pay the premium and go with the name brand at work. However, at home I don't mind putting a little TLC into my systems and getting something which isn't junk...

Re:Any reason why you are building it yourself?

[omb]

That was (a) not what the poster asked

(b) premium from Blue Chip often has insane mark up

If you are saying, "Buy from a Blue-Chip who will give you support" just reccomend that

Re:Any reason why you are building it yourself?

[jovlinger]

Just curious; how come there aren't any slot splitters? Fits into one ram slot on the mb, attach 2 sticks of ram to that?

The idea is obvious enough that there must be some technical reason why this doesn't exist.

Re:Any reason why you are building it yourself?

[Anonymous Coward]

If you want an AMD based box then you can try the COMPAQ DL585 - specs say it can handle 64GB.

http://h18004.www1.hp.com/products/servers/platf or ms/index-dl.html

Re:Any reason why you are building it yourself?

[Anonymous Coward]

LOL! I'm sorry, but this is a discussion about Opteron and 64-bit computers. Even Intel is seriously lagging AMD for 64-bit x86 CPUs. Their 64-bit Xeons just don't compare at all.

Re:Any reason why you are building it yourself?

[maraist]

I'll make a cheap reply by not checking the facts out for myself, but doesn't the Opteron have 4 external hyper-transport BUS's. The diagrams I remember looking at implied the 4-way symmetrical external interconnect could be attached to either adjacent CPU's or memory busses. So I was under the impression that each hyper-transport bus could be used to connect to 4 separate memory devices, or 4 CPU's. And THAT was why there were so many pins. If this is the case, then pin-out isn't the limitation.. I'll accept that the internal memory controller may be limited to two busses, but a single CPU's pin-out should support 4 (in single-CPU mode).

Now, again, I'll cheat by not checking the facts, but I know that there are 3 Opteron models which support 1, 2 or 4 CPU configurations, which to me implies different pin-counts (or at least internal wiring). So it may be true that you'd have to purchase a 4-way CPU to get 4 [hyper-transport] banks ANYWAY. And in such a situation, it would make the most sense to simply produce motherboards that support 4-way. Which answers all our questions.

Re:Any reason why you are building it yourself?

[angle_slam]

This goes for home-built systems as well. Often it's just cheaper to buy a system built in taiwan than to build your own out of the exact same parts.

The problem is finding a system built out of the exact same parts. Specifically, graphics cards and hard drives. The big name manufacturers trick out the systems with GHz, but give you either lame graphics cards or top of the line graphics cards, not a mid-line card like the GeForce 6600. And most give you a paltry 80 GB hard drive and charge through the nose to give you a larger hard drive.

So to build my system was more expensive than just getting a Dell, but I got what I wanted, instead of ordering something from Dell and immediately swapping out the graphics card and adding a new hard drive.

But there is a way to get stuff cheaper--upgrading. I could buy a brand new computer to replace my daughter's old computer. Or I could buy a motherboard, Sempron, and a stick of RAM for under $300.

Re:Any reason why you are building it yourself?

[Anonymous Coward]

Even though this is not what was asked... BUILD YOUR OWN...

I built a new AMD 64 system... 3500+, 1 gig of ram, Soundblaster Live 5.1, nForce 4 mobo, 480w Thermaltake PSU, NEC dvd burner, Nvida 6600 GT

All for under $900... granted, I didn't need a case or HDD becuase I had them already, but everything else was new. (Fans included)

Not to mention, it is faster than anything Dell has for $1700 or $1800

Re:Any reason why you are building it yourself?

[drinkypoo]

Is that 8 slots per CPU, or 8 slots per channel? Some CPUs have one channel and some two, so saying 8 slots per AMD64 CPU is probably not correct.

Re:Any reason why you are building it yourself?

[Fweeky]

Per CPU, according to one (admittedly somewhat vague) document from AMD. Also an old example system with 8 slots per processor. And here's a more recent motherboard with an 8+4 configuration: K8D Master 3.

Re:Any reason why you are building it yourself?

[(negative video)]

HyperTransport certainly could connect to memory, but there are no memory modules with an HT interface, and I don't know of any memory controller chips with an HT interface either. The only HyperTransport memory controller you can get comes with an AMD processor on the same chip. It's likely to remain that way, since not many people need huge amounts of RAM.

Now, again, I'll cheat by not checking the facts, but I know that there are 3 Opteron models which support 1, 2 or 4 CPU configurations, which to me implies different pin-counts (or at least internal wiring).

That's true. The one-way chips have one HyperTransport interface, the two-way have two, and the four-way have three. Unfortunately each HT interface has only 38 pins (yes, I had to look that up ;-), so even if you sacrificed two of them you wouldn't have enough pins for another memory interface. So multiple CPUs it is.

BTW, each memory bus can theoretically support up to four DIMMs. The catch is that they have to be clocked slow--PC2100 IIRC. So hardly any vendors have gone for that option.

2GB DIMMs and Tyan

[Jeffrey Baker]

You might want to get 2GB DIMMs. I have never seen any that worked reliably, but I also haven't looked at them in about a year. Get some and burn them in with memtest and see what happens. As for 8-DIMM motherboards, check out something from Tyan. Since you say you don't need that much CPU power, you could load up a Tyan Thunder board with two Opteron 240 CPUs for rather little money (or, a very small amount compared to what the 16GB of memory is going to cost you). If you seriously need 64GB of memory, the only PC system I've seen with that much are the HP ProLiant DL585, DL740, and DL760 machines. These are very expensive, especially once loaded with 64GB of memory.

Re:2GB DIMMs and Tyan

[Anonymous Coward]

If you don't mind switching platforms, IBM sells their "OpenPower" systems that can hold up to 64GB ram. They run Linux on POWER5 CPU's (can you say twice as fast as the Apple G5?) and do come in desktop style cases. You can even get them bare, so you can throw Debian on it if you feel like it. :)

Re:2GB DIMMs and Tyan

[JoeMerchant]

I did this research project about 3 months ago, and my results agree with J.Baker's post - only the HP systems support a full 8 sticks per processor.

AMD specs a limit of 8 sticks per processor, but all motherboards except the HP limit to 4 sticks. It seems a waste to buy extra processors just to use them as memory controllers, but for your application (large RAM drive), in essence, that is what you will be doing. I suspect going to 8 sticks per processor will drastically limit your memory speed, not as slow as a SATA (hard drive) bus, but considerably down from 400. Loaded with a full 4 sticks of 1GB each, my AMD64 motherboard drops from 400 to 333 because of the bus loading.

I'd look at TYAN for motherboards, and possibly an IBM box as suggested above. If you want the RAM to be fast, be prepared to pay for extra processors to get that speed.

Products

[Virtual Karma]

Here is a link for a list a motherboards (supporting various amounts of RAM). You can read product reviews, compare prices and store ratings

CLICK HERE

Re:Products

[Anonymous Coward]

Yeah, and the largest there is 4GB which is the limit he complained about. Thanks dude!

Re:Products

[dinojemr]

Did you even look at that site?
While the first ram size on the list is 4gb, it also has motherboards supporting up to 16gb.
Although it would be more intuitive for them to be listed in some kind of order, that site does have larger motherboards.

Power Mac G5, perhaps

[diamondsw]

At least will get you up to 8 slots. I'm not sure if 2GB DIMMs are shipping yet, however, so you still may be limited to 8GB.

Also, what are you going to do with all of that RAM? You'll likely end up with far more data than you have the resources to process.

Re:Power Mac G5, perhaps

[mkelley]

2gb dimms are shipping, I just purchased some for my G5 Xserve. Yeah, I thought the same thing, the G5 PowerMacs hold 8gb (prob. more now since the 2gb are shipping) with 8 slots. We're using the hell out of those on my graphic designers' boxes.

Re:Power Mac G5, perhaps

[Anonymous Coward]

The high-end Xserves will take 16 GB.

Have to have multiple CPUs

[Anonymous Coward]

Most A64 systems require multiple CPUs. Generally it's 8GB per CPU. This is because the on die memory component can only handle 4 slots. 4 slots * 2GB sticks = 8 GB per CPU.

Now if you had an 8 CPU system with 8 GB a piece, that's 64 GB of RAM.

Huh?

[zsadecki]

4 DIMMs = 4GB ?!

You know there are 4GB DIMMs around nowadays. So, 4 slots = 16GB. And 2G DIMMs are all over the place and relatively cheap, so you could get 8G with 4 slots at a minimum!

I've seen a 4P Opteron system with 16 DIMMs running with 64GB of memory. So it can be done! But that much memory is EXPENSIVE (4G DIMMs are just now getting under $1/MB)!

Re:Huh?

[Anonymous Coward]

And was under the impression there are mobo limitations on the amount of RAM which can be installed, regardless of the physical capacity (bays) and the apparent storage of the DIMMs themselves (Someone got anything more defintive on this front?)

Re:Huh?

[mlyle]

And was under the impression there are mobo limitations on the amount of RAM which can be installed, regardless of the physical capacity (bays) and the apparent storage of the DIMMs themselves (Someone got anything more defintive on this front?)

The anonymous coward would be correct; you're still limited by the number of address lines hooked up (in the memory controller itself and on the traces on the motherboard); and limited as well by the memory controller's ability to refresh the memory possibly. (The worst case scenario is being able to address the memory but not have it refreshed properly-- resulting in information in upper memory quickly decaying to garbage).

Look at Xi

[PseudononymousCoward]

I know you're looking for an MB, not a system, but if you go to http://www.xicomputer.com/ and check out their NetRaider 64 LT. You can custom configure it to have up to 12GB of ram w/ a single Opteron 244.

Have fun!

tyan

[DarkSarin]

has several boards, but none of them are single proc systems. If you need a stock Mobo, and want that much memory, you may need to go for a tyan board. I don't know if you can purchase the quad cpu board, use one cpu and all sixteen slots or not, but I think it will work. Obviously if you are going to shell out for that much memory, buying that mobo shouldn't be a big deal, even if you don't use all four cpu slots.

FWIW, that's the only thing I know of.

Re:Tyan

[elteck]

True. But be aware that their Quad opteron boards require a minimum of two processors to run!

Re:tyan

[turm]

I don't know if you can purchase the quad cpu board, use one cpu and all sixteen slots or not, but I think it will work.

No, it won't. The memory controller is in the processor. If you don't populate a processor socket, you can't use the DIMM banks attached to that socket.

Re:tyan

[InvalidError]

For optimum results, this should be two DIMMs per channel, two channels per CPU, four CPUs, 16 DIMMs total.

In this optimum (generally) case, a CPU must be fitted in the socket to which the DRAM banks are connected - the Opteron has built-in DRAM controllers, no CPU = no controller = dead slots.

Also, probably only one or two of the CPUs are connected to the chipset so these slot(s) have to be used first to provide an IO bridge for the others - signals do not mysteriously jump across sockets when the CPUs use point-to-point busses, HyperTransport in AMD's Opteron/A64 case.

Re:tyan

[ottffssent]

No, it will not work. Each CPU supports up to 4 DIMM slots - to use all 16, you will need 4 CPUs.

The Tyan board you want is the S4882, as mentioned by at least one other poster. Available for about $1500 from such places as Alvio: http://www.alvio.com/pmoreinfo.asp?iid=3256

2GB DIMMs have come down in price - they're now only about 3x the price of 1GB DIMMs, or 50% more expensive per MB. Using 2GB DIMMs rather than twice as many 1GB DIMMs is probably cost-effective, assuming you don't mind limiting your upgradeability. Otherwise, a quad board with 2 CPUs and 2GB DIMMs would leave quite a bit of room. That Tyan board supports up to 32GB of DDR400.

Re:tyan

[Anonymous Coward]

you cant with opertons. memory is access though each other cpu. each cpu handels 4 DIMM slots. So if a cput needs memory from the farthest cpu from it, them data has to travel though all the other cpus to reach. Thus you cant use 16 DIMM slots and have 1 cpu

Re:tyan

[DarkSarin]

Neat thing is, though that if the poster (of the story) wants that much memory, they are probably willing to spring for the board and the memory.

Also, methinks that tyan has a newer version available.

Limitations

[TEMM]

32 bit processors cant address more than 2^32 bits of memory, or 4 GB. This means you can only have up to 4 GB of memory per CPU in a system. You can have more than 4 GB if you have more than one CPU, or you use a 64 bit CPU that uses larger than 32 bit addressing. Also, you cant even use above 4GB for a ram disk because of the same problems.

Re:Limitations

[Anonymous Coward]

Whatever gave you the idea that Opterons are 32bit processors?

Re:Limitations

[Anonymous Coward]

FTFsummary
support for 64 bitness

Re:Limitations

[Anonymous Coward]

Intel x86 Processors have 32bit, 36 bit and 48 bit addressing modes.

Gee with only 32bits how do we have 8 gb files on our file system? Maybe use extra bits dipshit.

You can have a maximum of 4GB per process (and more likely 2GB (OS Dependant))

Re:Limitations

[Anonymous Coward]

This is mostly incorrect- modern x86 processors support 36-bit addressing, also called PAE or Page Address Extensions. This means that you can address 64G in a system, although individual processes are still limited to 4G.

Re:Limitations

[Anonymous Coward]

That is exactly why you should get an Opteron based system to take full advantage of all that memory in both the OS & User space. Otherwise the 32 bit CPU's need to waste cycles if they need to address more then 2G.

Re:Limitations

[Anonymous Coward]

Nope, that's not right. Nonetheless, the question relates to 64 bit processors, so you're not even on topic.

The limiting factor is the number of address bits available from the memory controller. 4GB modules use up to 16 address bits, but you'll probably be hard-pressed to find a commodity motherboard that supports that many address lines.

Re:Limitations

[Sancho]

You may have missed the fact that he is using a 64 bit machine.

Re:Limitations

[Anonymous Coward]

Actually 8GB files are the result of hacks like offset addressing and such, not the extended addressing capabilities of Intel processors.

Re:Limitations

[TEMM]

ah crap i read the thing twice and i didnt notice that lol

Dont bother :-/

[Anonymous Coward]

Linux ramdisk support is actually pretty slow. I did performance testing because I was investigating the possibility of loading the 'hot' portions of an FS into memory to speed things up.

The results were less than stellar on a system with 1.5GB of DDR400.

Re:Possible Problem

[tomstdenis]

Troll.

Where did you get these facts?

The kernel has three memory models. = 1GB, = 4GB and = 64GB.

I've run Linux on a 1.5GB box before just fine [they were PC-133 sticks on sale bought three 512MB sticks...]. You just have to enable the different memory model.

Tom

Sun

[cerberusss]

I didn't even Google when I read what you wanted, but went straight to Sun's page. I don't know if this is what you're looking for?

Sun Fire V40z: up to 32 gb of memory

Re:Sun

[fm6]

He did make it clear that cost was an issue. I don't know how much more expensive the V40z is than equivalent AMD-based boxes -- but the extra cost of SPARC-based systems is the main reason Sun is in trouble.

Re:Sun

[Jeffrey Baker]

I've seen v40z, with 2 CPU and 2GB RAM, for around $6k, promotional deals. You could get one and add the 3rd and 4th CPUs and a shitload of RAM yourself, without paying Sun's prices on the parts, but beware you'll also need to find a source for the removable (and not installed by default) voltage regulator modules for each CPU, and for each CPU's bank of DIMMs. That's 8 VRMs total, and the machine only comes equipped with 4.

Re:Sun

[Anonymous Coward]

The v40z is an opteron based box. :-) But Sun definately does charge a premium for these machines, especially for the add-ons. If you were to load this guy up with 2GB DIMMS for the full 32GB footprint, you'd probably spend twice as much as you would on a "beige box" machine.

-A

Re:Sun

[Spy Hunter]

The V40z is Opteron-based, his link was wrong. The real link is here.

Re:Sun

[jarich]

I built a dual opteron w/2 gigs of ram complete (80 gig drive, video, case, etc) for $1,700 when the opterons were brand new (about a year ago). Much cheaper the Sun box... but then again, Sun's support is better. ;)

Re:Sun

[cowbutt]

If you don't care about Sun's support, you can buy the same chassis from the designers, Newisys. Sun didn't design these Opteron systems themselves! A UK-based reseller of Newisys hardware is Rainbow-IT. Never used 'em, just found 'em when I was googling a while back.

Re:Sun

[fm6]

That's expensive. You can get a 2GB Intel-based machine for maybe 1/3 of that. OK, maybe not one that upgrades to the amount of RAM this guy needs. But if he were able to spend $4K just get some extra RAM slots, he wouldn't be having this discussion.

Re:Sun

[Anonymous Coward]

Much cheaper the Sun box... but then again, Sun's support is better. ;)

The Sun box is also built like a server. It's in a rackmount case. It's got redundant power supplies. It's got a admin network port with its own CPU so you can telnet into the machine's console and reboot it even if the main CPUs are hung. It uses expensive, well specced disks. The build quality is exceptional. *That's* what you're paying for.

Re:Sun

[ckaminski]

Care to enumerate your parts, please? I'm looking at getting a replacement for my workstation. Apache + postgres + mysql + vmware + gnome&kde apps is killing my 2GB of memory finally.

Re:Sun

[jarich]

It's been a year, so I'll reference Pricewathc for costs. I honestly don't remember the details but the price was equivalent to a dual P4 at the time.

I got one of the Tyan motherboards and it's still selling for ~350, but you can get a dual Tyan motherboard now for ~250.

Two CPUs at $125 each (the "slow" 240s).

Two sticks of registered ECC ram, 1 gig each, at $110 each

80 gig harddrive w/8 meg cache (IDE) are $53 now.

That leaves a case and power supply. I'll ballpark that $75.

Although the Tyan board has onboard video, I put a dual head NVidia GeForce 5200FX on for about another $100.

So at today's prices, my rig would be $250 (mb) + 250 (cpus) + 220 (ram) + 53 (hd) + 75 (case) + 100 (video). That comes to $948.

I paid an extra few bucks to have them assemble it for me because the boards were brand new and I didn't want to have to mess with memory incompatibilities, but today you could build your own. I also scavenged a CD/DVD from another box and opted for no floppy drive.

As to the platform, I have been very pleased with the performance. I've run 32 bit XP, 32 bit RedHat, 32 bit Knoppix, 64 bit Knoppix, etc on it and had no problems at all.

Re:Sun

[Anonymous Coward]

That's expensive.

No it isn't. I'm really tired of people on Slashdot comparing home-built PeeCees with real datacenter-grade servers. Sun's Opteron servers come with an independent lights-out processor, redundant power supplies, and Ultra320 SCSI, for god's sake. Please, inform yourself.

Re:Sun

[SunFan]

Apparently, Sun's next-gen Galaxy servers were designed in-house. It'll be interesting to see how they spec out whenever they start shipping.

Re:Sun

[drinkypoo]

Case and power supply should cost more than $75. That MIGHT pay for an adequate power supply. Most power supplies are garbage and I imagine you'd need QUITE a supply to handle dual opterons and very much memory. It really is marvelous what dual opterons cost, though.

SGI has such systems

[Anonymous Coward]

Copy the profesionals , sgi has such systems.

Large memory systems

[mknewman]

Get a Sun. You can get very large memory systems. You can go up to 192 gigs of memory. Marc

Re:Large memory systems

[Florian Weimer]

Get a Sun. You can get very large memory systems. You can go up to 192 gigs of memory.

192 GB is no longer very large nowadays, and Sun indeed offers systems with larger per-node memory sizes. The price tag is hefty, of course, but so is anything which goes beyond 32 GB (maybe even 64 GB).

Re:large memory systems

[rcpitt]

Seems to me if the ramdisk was set as swap the effect would be similar to an increase in RAM, just funneled through the OS. It would be far faster than swapping to disk but slower than having directly addressable ram on the CPU bus.

Note that if you're using ramdisk for file system, your statement only holds if you are hitting less than cache sized chunks and the rest of the system is not using the cache for something else. When you have something like a huge sequential file or a bunch of them (streaming video anyone?) the ramdisk will perform better than spinning disks. The key is rotational latency coupled with speed of the data under the read head (and having hardware on the disk that can push that data out to the bus). New 15krpm disks minimize this but it still exists.

Re:large memory systems

[geekboy642]

Depends on the use...typical filesystem access wouldn't be sustained read-write over a large portion of the disk. In that case, caching would be much less useful.

Re:large memory systems

[Anonymous Coward]

MOD PARENT OFFTOPIC

Keep your rants to your own blog, idoit.

Re:Large memory systems

[Anonymous Coward]

Actually, the current limit for Sun systems is 576 gigabytes per domain. The 192GB limit is only for systems with 24 CPUs (e.g., the SunFire 6800 or 6900 series). The bigger systems like the 25K go up to 72 dual-core CPUs with 576GB of memory.

For Opteron-based systems, then you generally get four slots per CPU, with up to 2GB per slot. So the two-way V20z can have up to 16GB, and the four-way V40z can have up to 32GB.

Re:Large memory systems

[Anonymous Coward]

Actually, the current maximum for a Sun system is 1152MB of memory. Yes that's over a terabyte of ram.

The E15K/E25K server has 18 memory/CPU boards which each have 32 DIMM slots and four CPU sockets. With the availability of 2GB DIMMs, that's 18 * 64GB totalling 1152MB.

You might say that's a lot of DIMMs in one system (576 DIMMs.) When you have that much memory, you'll probably want an OS that handles memory faults very well. Sun's Solaris will automatically retire failing memory pages--parts of physical memory with persistent errors. Heck you can even take a memory board out of the system, replace the DIMM, add the board back to the system without any downtime. Handy!

Re:Large memory systems

[kkerwin]

You can go up to 192 gigs of memory.

Damn!

Yeah. I'm going to take my entire hard drive, and write it to RAM. Six times.

Yay for geek penis-envy: "My RAM stick is bigger than your RAM stick."

Re:Large memory systems

[Anonymous Coward]

Dude you only have a 32GB hard drive? Where do you put all your pr0n?

Re:Large memory systems

[Anonymous Coward]

... and you'd expect Oracle to run pretty well on that machine, too (why else would you need that much memory, anyhow?)

Re:Large memory systems

[dwater]

> 192 GB is no longer very large nowadays

Indeed.

SGI Altix 350, thanks to memory only nodes, (IINM, which I could very well be) goes to 384GB per system and only requires a single CPU and single disk. They even have a special on 'remarketed' ones at the moment.

Re:Large memory systems

[Anonymous Coward]

I roll it up and store it all in Goatse.cx' ass for safe keeping.

for $1500 you can get 32GB

[JRW129]

Tyan Thunder K8QS Pro Quad Socket 940 Motherboard AMD Solution Motherboard - Operton Motherboard: Outpost #: 4311135 http://shop2.outpost.com/product/4311135/

-Chipset: AMD-8131
-Processor Support: Quad Socket 940 for AMD Opteron 800 Series Processors
-Memory Support: Up to 32GB Registered PC3200 DDR
-Expansion Slot: Two 64-bit 66/33 MHz PCI-X, two 64-bit 133/100/66/33 MHz PCI-X, One 32-bit PCI Slots
-Other Features: GigaLAN, Integrated Video, SCSI

if you have the money to fork over, pick one of these up at your local Fry's Electronics or try fry's online at http://www.outpost.com/

Re:for $1500 you can get 32GB

[CaptnMArk]

However, since each CPU has it's own RAM he will need 4 CPUs to max out the board.

Re:for $1500 you can get 32GB

[etymxris]

That's a NUMA opteron. You need to fill all the CPU sockets to access all that memory. Given the price of 8xx's, that's not very affordable.

Re:for $1500 you can get 32GB

[Anonymous Coward]

ermm, not quite. http://www.hypertransport.org/ - read the spec. get a clue.

Re:for $1500 you can get 32GB

[Anonymous Coward]

Er, uh, it's you who needs the clue. You need to fully populate the Opteron CPUs to fully populate all the RAM slots.

large memory systems

[Binder]

I know tyan has several opteron motherboards with 8 dimm slots and at least one with 16 (Thunder K8QS Pro (S4882)). Those may be a good starting place.

As a side note, I did some testing in college dealing with ramdisks and in general running a filesystem off of a ramdisk is actually slower than from disk. The linux filesystem buffer is very good and will cahce the information in ram anyway.

Solid State Disk Drives

[Xerp]

Any good to you? You can easily get up to 128Gb in a single drive, 3GB/s bandwidth, fits in a drive bay. Who cares what "RAM" the motherboard supports? Just a though, don't know if it would be any good for your environment.

Re:Solid State Disk Drives

[Anonymous Coward]

solid state drives have fast read but are slow as hell for write.

the battery backed ram drives can be super fast but ram is more unreliable than hard drives. over a 24 hour period there is a huge chance of getting data corruption in your database in a 120gig ramdrive compared to a harddrive.

go Proliant or go home... :p

[mattjw916]

http://h71016.www7.hp.com/dstore/ctoBases.asp?Prod uctLineId=431&FamilyId=1770&jumpid=re_hphqiss/Ovw_ Buy/DL585

Here you go...

[Anonymous Coward]

http://www.tyan.com/products/html/thunderi7525.htm l
"support for up to 32GB of Registered DDR2-400 (PC2-3200) memory with ECC"

More boards here:
http://www.tyan.com/products/html/xeon.html

Total time searching...20 seconds.

no shit

[Anonymous Coward]

uh, no shit. i think he communicates well in his post that he knows that.

Re:no shit

[Anonymous Coward]

If you view the original poster's post history it becomes apparent he is an idiot.

Apple Xserver G5 supports 16gb ram

[Dtyst]

The Apple single processor Xserver G5 supports 16gb ram cost and additional 11600$ though.

Check:
http://store.apple.com/1-800-MY-APPLE/WebObjects/A ppleStore.woa/71701/wo/qF4yxwL1DfYE2umn6Pq1GNkmABL /0.0.11.1.0.6.15.0.3.1.3.0.3.1.6.1.1.0

Oh shit, wrong account

[DmitryProletariat]

You do not need a great deal of RAM capitalist pig! The entire population of China will easily be enough human labor to meet these needs. And so through this socialist distribution humanity becomes valued more than simple machinery. Because people count more than computers! Does your computer love a wife at night? Does it eat morning gruel? NO! It is a thing. And as long as you value CPU more than people, the cycle of money more than the cycle of life, you will forever be a money grubbing MCM overlord of labor stealing, evil capitalist, ideologues!!! When the revolution comes we will cover you in plaster, make statues of you, and pour hot grits down your calcified pants!!! REVOLT!!!

*sigh* I think I'm getting tired of this....

*bang!*

Re:Oh shit, wrong account

[slashdot_commentator]

One of my favorite geek memories of childhood was listening to Radio Moscow on my shortwave radio. (I think I was 13 years old back then.) Initially, there was the childish visceral thrill of doing something "forbidden". (There was a COLD WAR ON, back then...) But really, the greatest aspect of the broadcasts were how the "Commies" were unintentionally being hilarious in some of their less defendable points of view. It would be like listening to Rush Limbaugh because you got bellylaughs from mocking his opinions. (But I personally don't find stupidity or his propaganda all that amusing...).

My most notable memory was how the Radio Moscow program was sponsoring an essay contest to win a free trip to the USSR. I don't remember the specifics, but they had the winner (American or Canadian) read it over the air. It was a foreign accented individual spieling communist ideology in a dead monotone right out of a bad Manchurian Candidate movie clone for like 15-20 minutes.

Basically, it could only be one of two possibilities. They could have been too broke or cheap to sponsor a real contest, and got it some academic ideologue to write it, and foreign embassy staff to read it. Mind you, you're not going to convince the audience there were loyal communists in the US if they speak in a slight foreign accent. The other possibility was that it actually was a North American, but hooboy, selecting a mumbling, monotonic guy out of a psychiatric ward would hardly encourage people to embrace communism.

Anyway, I found that post hilarious. The moderator is an obvious moron for labeling it a troll. I'm curious as to what traumatic brush with communists causes you to lash out in parody. (And *bang* is a nice touch.) Keep alive the Revolution, Comrade!

Re:Oh shit, wrong account

[Black Copter Control]

Now needs one more 'funny' mod to be rated '+1 funny'.

hardware ram disk

[Datoyminaytah]

If you're going to config it all as a ram disk anway, why not search for a hardware device that appears to be an IDE or SCSI drive but is actually a bank of DIMMs? Then your limitation is the size hard drive the motherboard/OS can handle, not the amount of memory the motherboard supports.

I'm pretty sure I've heard of something like this but you can do the googling from here...

Re:hardware ram disk

[stoborrobots]

why not search for a hardware device that appears to be an IDE or SCSI drive but is actually a bank of DIMMs?

Can't speak for the OP, but possibly because IDE maxes out at 1.06 Gbps (Gigabits per second) for ATA-133 and 1.20 Gbps for SATA, whereas PC3200 DDR SDRAM can push 25.60 Gbps?

(all numbers above handily supplied by http://www.forret.com/tools/bandwidth.asp )

Re:hardware ram disk

[morcheeba]

exactly. I did the googling ... try this 2-20GB SCSI RAM drive, or similar products.

Re:hardware ram disk

[Kent Recal]

A while back I read about something like that called RocketDrive.
I think it was limited to four DIMMS though but maybe they have worked that out...

Re:hardware ram disk

[Anonymous Coward]

It's called a Solid State Disk.

Re:hardware ram disk

[Anonymous Coward]

Any ramdisk tied to a standard PCI bus is going to be bandwidth limited to 133 MB/s. You'd do better with PCI-X or -E; but unless you put devote a lot (read: 8 or 16) of PCI-E lanes you won't come close the the avalible bandwidth of the ram.

The low latency is more important than bandwidth in most ramdisk applications.

Re:hardware ram disk

[stoborrobots]

Any ramdisk tied to a standard PCI bus is going to be bandwidth limited to 133 MB/s. You'd do better with PCI-X or -E;

Ok, I'll question it: why would a ramdisk be tied to a peripherals bus? A ramdisk fakes a drive by applying filesystem calls directly to RAM, and the processor-RAM data path is clocked to provide those transfer rates which I mentioned earlier.

So where does PCI/PCI-X/PCI-E come into the picture?

Re:hardware ram disk

[(negative video)]

Ok, I'll question it: why would a ramdisk be tied to a peripherals bus?

The devices being discussed are hard drive-style boxes with dynamic RAM chips instead of magnetic platters.

Well titled!

[Anonymous Coward]

turd psot (Score:0)
by Anonymous Coward on Sunday April 17, @11:44AM (#12263162)
lol

Every so often I see a post with a very good title. This is one of them. The title concisely sums up the entire post. Better the poster didn't post it at all though.

Re:Possible Problem

[Anonymous Coward]

Don't feed the trolls, Tom.

Re:Possible Problem

[Zontar The Mindless]

Huh? You should be able use up to 4GB on 2.6. You can run up to 64GB in a single 32-bit machine with extended addressing, but you're still limited to 4GB per process, IIRC.

(I'm no g00r00, but I can search kerneltrap.org as well as the next guy, and I call either ignorant or an attempt to troll. Given the Windows reference, I'm inclined to assume the latter.)

Itanium? Alpha? UltraSparc?

[nick-less]

This one or
This one
or This one

ok, some of them are end of life, but hey you'll never know... ;-)

ps:I hope you have some spare money lying around...

Tyan

[Neon Spiral Injector]

The biggest board I know of is the Thunder K8QS Pro (S4882). With 16 DIMM slots it supports 32 GB of RAM. But it is a quad processor board.

Most of the rest of Tyan's Opteron server boards do support 16 GB. Again they are dual processor.

Tyan's boards will run with a single processor installed, but only 4 DIMM slots (1 bank) will be active. All processors can see all the RAM installed, but because the memory controller is integrated into the CPU, the CPU must first be installed. Then it can pass access to the RAM to the other CPUs. Local RAM always being the fastest. (Linux with Opteron NUMA support tries to keep memory associated with the task running on a specific CPU local to that processor.)

Any way to break it up?

[anthony_dipierro]

Is it possible for you to write software which can handle this application across several machines? 4 machines with 4 gigs of ram connected to each other via gigabit ethernet would probably cost less than 1 machine with 16 gigs.

If cheap is what you care about, you've gotta use scalable software. If you want to just buy something out of the box which can handle 16 gigs of ram, you can expect to pay more.

Re:tom, you're a faggot

[tomstdenis]

Ah you've returned. Now be loyal troll-pet.

*pets troll on head*.

Tom

Quad Opteron

[Anonymous Coward]

Any Quad Opteron system/board running Linux!

Why do you care?

[Anonymous Coward]

You interested in him or something?

If you dont mind that its Intel

[klauskaan]

Then Supermicro has made some quite good motheboards supporting up to 32 Gig RAM
They are really inexpensive too.

http://www.supermicro.com/products/motherboard/Xeo n800/

Re:If you dont mind that its Intel

[chipace]

Intel is the best architecture for a single cpu with a butt-load of DIMM slots... all the memory controllers are in the non-integrated northbridge.

Good recommendation.

External enclosure

[m50d]

Have you looked at getting an external scsi ramdisk? That might be easier than trying to find a motherboard you can fit it all in

Re:Possible Problem

[Anonymous Coward]

Looking at its other posts, they're ALL trolling.

PowerMac G5

[for_usenet]

Actually, if you are looking for something relatively cheap, the 2.0 and 2.5 GHz PowerMac G5 models actually support 16 GB of RAM, if you can find matched 2G PC 3200 DIMMs to fill the slots. This was the info given in the developers docs for the memory controller. And compared to other boxes that can support this much memory, they aren't that expensive. Linux support is also coming along for them, though not all quite there yet.

If you think your memory needs are going to rise above 16 GB, you'll need to look elsewhere, and I don't think there's a single board system out there now, that can do this, and qualifies as "cheap" ;-)

Re:PowerMac G5

[Saint Aardvark]

Linux support is fine for the dual 1.8GHz G5 we've got at work -- even the fan cooling is fine. I'm running Gentoo on it with [quick ssh] 2.6.10-gentoo-r4-g5 #3 SMP. Is there something different about the 2.0/2.5 G5s?

Tyan

[josh42]

Tyan's motherboards have consistently large amounts of DDR slots. Try the "Thunder K8W" (S2885), with 8 slots and 2 processors, or the "Thunder K8QS" (S4882) with 16 slots and 4 processors.

That would be 16 or 32 gigs, respectively.

The K8W is $500 or so, the K8QS around $1700. Your RAM will go around $600 for a 2GB stick. Opterons are anywhere from $200 (for a 2-way) or $750 (for a 4-way) and up - that's just 1.4GHz.

In summary: You can do it, but it's pretty darn expensive.

The law of supply and demand.

[jafo]

I think there's a misconception made by the poster that pricing is going to be cheaper for a single CPU system with no SCSI or RAID built into the motherboard, and they're trying to cut the cost down. However, the law of supply and demand asserts that if most people who want big memory systems also want lots of CPU and disc, then the price for a single CPU motherboard may actually be MORE expensive (because there's less demand).

Personally, I'd say just suck up the extra few hundreds of dollars (from what I've seen) and consider it part of the cost of the 16 to 64GB of RAM.

Sean

Re:The law of supply and demand.

[Dantu]

supply and demand asserts that if most people who want big memory systems also want lots of CPU and disc, then the price for a single CPU motherboard may actually be MORE expensive (because there's less demand)

Supply and Demand says that if more people want x, x will be MORE expensive. This only applies to systems with fixed supply (note that short term supply of many items is fixed).

Economies of scale says that if more x is produced/consumed then x will be less expensive. This only applies up to a point where diseconomies of scale come into play - but this is a /. post not Econ 101.

Re:The law of supply and demand.

[Armadni General]

Please, your application of Supply and Demand is all wrong. Low demand equals high cost? Come on.

They taught us this in tenth grade History class.

Re:The law of supply and demand.

[FueledByRamen]

Low demand equals high cost when the company manufacturing the product is trying to recoup its initial outlay for that product. More products moved means that the R&D and production runup costs (and other one-time expenses) can be amortized across more units, therefore (in theory) bringing the individual unit cost down.

If I make a single motherboard, it'll have a higher per-unit cost than some company that is spreading the pre-production costs across 100,000 units.

Re:The law of supply and demand.

[Armadni General]

You are being substantially more specific than the situation calls for.

Tyan anyone!

[mwfolsom]

What's the deal?

Tyan currently lists 6 2 processor opteron boards that hold 16 GB and 1 4p board that holds 32 GB - actaully got one of those the other day. Very nice!

Iwill has one 8p system that holds 64GB.

See:
http://www.tyan.com/products/html/opteron. html

Dodge Ram

[Anonymous Coward]

Grab life by the gigs.

Applications: Scientific uses

[Telcontar]

There are all sorts of scientific applications where you need as much RAM as possible. One where the amount of RAM is almost the only limiting factor is model checking. It refers to a systematic exploration of an entire system. Even though a lot of techniques have been invented to reduce the complexity of the problem, the problem itself is exponentially complex in the size of a system. A 16-bit counter has 65536 states. Just imagine how many states a simple ALU or an entire CPU can have. This shown you that there is no limit to the amount of RAM one can use for this.
All major players in the CPU industry use this technology, so I am sure AMD has some systems like that in their design labs...
Model checking is also used in some software projects where the complexity of the system is "manageable", such as for device drivers or embedded systems.

Other scientific uses needing a lot of RAM are simulations. The level of detail simulations can manage is mostly limited by the amount of RAM (and of course also CPU speed). Again, the sky is the limit for how much RAM one could use if it was available... so you see it is not problem to fill 32 GB of RAM in a couple of seconds with such simulations.

Re:Applications: Scientific uses

[jacksonj04]

From what I hear Longhorn will chew up 32gb quite happily...

That aside, I can see serious uses for up to 4gb in the power user/gamer scenario. I have 2gb, and frequently have to dip into pagefile when playing games because they just uncompress *everything* into RAM. Means I can get some nice load times on HL2 tho.

Re:Applications: Scientific uses

[gnuman99]

Graph partitioning in mathematics (can be used for processor layouts, etc..) can use up enourmous amounts of RAM. Just image a weighted graph (or hypergraph to make things more interresting) with 1 billion nodes and then you want to partition it. Lots of ram and CPU power needed for this (especially since you can run in parallel).

I worked on something similar (a little bit less nodes :), but we filled up a 128 processor, 256GB machine for the duration of the partitioning. I think only 30GB was left free...

Now, I don't think a machine like this is in the budget of the poster.

Re:Applications: Scientific uses

[bfizzle]

I was thinking the same thing. The OP could very well be a Grad student somewhere trying to run sims on a budget. Who knows maybe he's got a buddy who works at Kingston or Micron and can hook him up fat with RAM.

Low end Opterons are cheap

[vincecate]

The Opteron 242 is only $169 on newegg.com. So getting an extra low end Opteron is not much money. And these actually are still very fast CPUs.

Re:Low end Opterons are cheap

[Xross_Ied]

For an Opteron board to go past 16GB, you need a quad opetron board + Opteron 800 series CPUs.

Opteron 800 series CPUs are much more expensive http://www.pricewatch.com/h/mn.aspx?i=3&f=1

DON'T!

[Anonymous Coward]

You must not stop Windoze from swapping --- it's a key part of the technology driver that keeps America in business by supplying pointless upgrades of computing power to underpin pointless usage of resources in a pointless operating system.

It would be totally unAmerican to get around this.

Re:I recommend an Apple

[LittleLebowskiUrbanA]

It's Darwin not FreeBSD. Common myth.

Upgrade to Alpha or Sparc if possible

[Anonymous Coward]

It'll be difficult designing a RELIABLE large-memory system while remiaining contrained to an Intel-compatible architecture. It may be a fun experiment, but does your business really want to be "bleeding edge?"

If it's a server you're building, perhaps you can host it on something truly reliable, tried and true, fully designed to handle large memory with interleaving.

Get an AlphaServer or an UltraSparc if at all possible. Price is hardly a reason to settle for Intel, since some pretty high-end, high-availability hardware can be found on the used market for cheap.

Don't be fooled by comparing the clock speed of RISC versus CISC architectures. A single Alpha processor for example, can execute several 64-bit instructions per clock cycle. They really rock!

Re:Upgrade to Alpha or Sparc if possible

[Anonymous Coward]

Alpha is EOL'd (End of Life).

Best bet is IBM right now.

Re:Upgrade to Alpha or Sparc if possible

[Anonymous Coward]

Alphas are not quite EOL yet. That term has a certain meaning in the hardware world. You can still buy new AlphaServers from HP, which will continue to be supported for the reasonable future. And of course HP is indeed hoping to migrate its Tru64 customers onto HP/UX on its PA-RISC architecture eventually. But Alpha/Tru64 will remains a viable platform for years.

Of course the 21264 will be the last chip in the Alpha line, thanks to guess who? INTEL! (arrgh!) Yes they bought the rights to this competing architecture, and nixed any development beyond the 21264 Alpha line. This is yet another reason to avoid Intel, I'm afraid.

Though there won't be a 21364, at least the 21264 clock speeds can continue to rise, and also OS enhancements can continue to take greater advantage of this chipset's features.

Yes, IBM's would also offer an excellent choice. :-)
Plus, IBM's support of Java and Linux make them the good guys, truly deserving of one's business.

Cluster you fool.

[Anonymous Coward]

Little boxes 4GB each running little NASed ramdisks.

512GB ...on PPC

[a3217055]

You can try an IBM Regatta, I think they max out at about 512GB, their are some Power5 chip boxes that IBM makes with about 1/2 a terrabyte of RAM etc.. There are also these PCI cards that are RAM DISKS don't know the size or that they do but stuff like that does exist. Also I don't think ther are any x86-(64) motherboards beyond 192 GB of memory. If you are willing maybe some company can get you a custom solution. Over all of the shelves no.. but there are other 'types of computes' you can buy which might have the same functionality.

Re:I recommend an Apple

[tomstdenis]

Unless you have to use libtool ...

Libtool ... a tool written to have a portable input interface to make compiling/linking a standard process between different OSes ...

What does apple do? Adds their own spin to it.

Yeah.... that's totally meant to not be anti-competitive...

Tom

IBM eServer 326

[Stack_13]

If 16 GB is enough, IBM has eServer 326 is just about expandable to that amount.

Strangely, 16 GB memory is only available in a dual-CPU configuration. DIMMs 5-8 can only be used if another CPU is installed.

But since the extra Opteron CPU costs about the same as a 2 GB memory module, that shouldn't be much of a concern. Price is in the vicinity of $15K with 2 x Opteron 248 and 16 GB RAM.

MSI dual opteron board, 12 slots

[Anonymous Coward]

MSI make a 12 slot dual opteron board, needs ECC tho. (MSI K8D Master3-FS Dual AMD® Opteron(TM) E-ATX Mainboard) In South Africa it goes for the equivalent of about USD 525, so it'd probably be a bit cheaper elsewhere

Re:I recommend an Apple

[Anonymous Coward]

Because he said "inexpensive", not "overpriced piece of crap".

A couple options

[Anonymous Coward]

Your best bet might be a dual CPU system with slow/cheap CPU's. (Opteron 240's). The dual Opteron boards out there are set up with two banks of RAM, each dedicated to a CPU, so you habe to fill both CPU sockets to make use of all the DIMMS.

There are a couple dual processor motherboards that I know of out there with 12 DIMM slots. That would get you up to 24GB per machine using "fairly inexpensive" RAM. The 2GB DIMMS these days are only around $600-800 bucks a piece, so it's feasable to load up one of these boxes for around ten grand. 4GB DIMMS do exist but AFAIK they're 3-4x the cost of a 2GB DIMM.

You might scope out HPC Systems (www.hpcsystems.com), they sell boxes pre-integrated with large memory footprints. (not an advertisement, but I've purchased gear from them before).

-A

Just use a solid state "disk" for this...

[shawnce]

You will likely be better served if you just use a solid state disk for this.

Re:Just use a solid state "disk" for this...

[imsabbel]

Same price, a factor 100 slower.
Whats the advantage?

Solid State storage

[soniCron88]

What you're interested in is solid-state storage, which you can google. The main problem with your plan is, if the power goes out, you've lost everything. Even trying to flush everything to the hard drive will take dangerously long. (Writing 15GB to harddisk is no small task!) The solid state storage machines out there are basically computers with lots of RAM, but the RAM is backed by battery, so even if it's unplugged, it stores its state, and then backs up to hard disk, automatically. Trying to get a PC to do this for you would be much trouble. Remember: UPSing a whole computer is a lot different than keeping RAM charged! :)

My recommendation

[Sivar]

I've searched several websites (including Tom's Hardware), and I've googled, but can't seem to locate any commercially available AMD MBs supporting more than 4 sticks of RAM, or 4 gigs.

First, I would recommend going with a server vendor. I honestly do not mean any offense by this, but if you are looking at places like Tom's Hardware for recommendations (a website which is frequently incompetent even at reviewing l33t g4m3r d00ds hardware, let alone server-grade hardware), you are probably not qualified to build a system which would actually need 16GB of RAM (e.g. a corporate server which must be relied upon). I do not know what the system will be used for, but if it is for a many-person organization, my recommendation stands, and if it is just for you and some friends, 16GB of RAM is almost always going to be absurd overkill.

Now that that's out of the way, Tyan has several dual Opteron boards which support 8 DIMMs. Look at their Thunder line, and put just one Opteron in them, and 2GB DIMM modules.

Re:My recommendation

[ionpro]

As mentioned above, that won't work. The 4 DIMM slots attached to the 2nd processor require a 2nd processor to be present before they'll work -- the other CPU has no way of getting to that memory except though a 2nd Opteron's coherent HyperTransport link, and the memory controller on that 2nd proc itself.

Re:My recommendation

[Sivar]

Two of the cheapest Opterons should do it then.
Other than that, would probably have to wait for 4GB memory modules, and those aren't likely to be released any time soon at any even remotely affordable price . :(

Re:My recommendation

[Anonymous Coward]

Amazing to see incompetant advice first start by proclaiming the incompetance of other sites.

Opteron's all have built in memory controllers. That means, you cannot populate the second bank without another Opteron. Real simple. You need the additional memory controller, or you are stuck with getting to 8 GB via 2 GB dimms.

More importantly, I would not recommend Tyan boards. Having used them (we design, sell, service large linux clusters with huge memory configs), I can tell you that they have caused use nothing but grief. The s4882 has design errors heaped on top of design errors. For example, the dimm slots on 2 of the processors cannot be fully opened due to collisions on either side of the mechanism. This isn't the only problem, there are some that are significant in terms of operation that effectively make these boards useless for large memory operations.

Look at either Iwill motherboards, the new MSI that gives you 12 slots, or go with Panta Systems, where you can get 128 GB over 4 CPUs (Opterons).

It is not worth it building this yourself, go to a knowledgable source, and ignore the idiots who proclaim Tyan from the tallest tree.

Re:My recommendation

[Sivar]

Opteron's all have built in memory controllers. That means, you cannot populate the second bank without another Opteron

This is correct, however, there are boards in which both Opterons access the same bank of memory, the second through an HT link to the first. This is not the most efficient method in terms of performance (though it is likely cheaper to manufacture). It would not be impossible to design a dual Opteron board with 16 or 32GB of memory through just one processor, though it would be very difficult to make just one controller handle so many banks of DIMMs at once, even registered ones.

My recommendation was not for Tyan above all else, it just mentioned that they were an option. I have built a number of servers using Tyan boards, and other than an some minor issues with an early S2460 (which was, to their credit, a low-end board), haven't had any problems, though admittedly none were equipped with 16GB of RAM.

All motherboard manufacturers release occasional lemons. Tyan is no different, and they are every bit as qualified to design and manufacture server-grade boards as any companies you mentioned (which area both great manufacturers). Tyan was making SPARC-based Sun-compatible servers 15 years ago, and was chosen by AMD to be their primary partner when developing AMD's first SMP Athlon boards (er, not that those boards were that great).
It's easy to fling personal insults from behind AC, just as it's easy to proclaim that since you have had problems with some products from brand X, brand X must therefore suck.
I would appreciate references to your claims of Tyan's problems (not that I am doubting them--more out of personal curiousity), but I do not appreciate an anonymous coward, who does not know me or my experience in the slightest, liberally spewing meaningless insults.

Re:My recommendation

[Anonymous Coward]

Drive a car do we? Could you build one from parts? OH NO! You can't you must be some sort of "l@|\/|er" - you mean you actually use technology you don't fully comprehend? Did you build your own TV out of Lego? Hmmm? How about your telephone? I suppose you designed that too, and the whole internet while you where at it.

I love techincal primadonnas. People who think they're so much better than other people, just because they have a tiny amount of knowledge about a niche area of technology. The poster actually wanted help with something, not abuse.

I have a friend who also wants a massive RAM disk. He works in video editing and hard disks don't cut it. Maybe it's not the ideal solution but it's something he thought of by himself. His area of expertise is editing videos, not nerding it up on computers and abusing people who aren't at his "l33t" level.

But you know what? He couldn't build an enterprise 's server - but it's not his concern. He's damn good at what he does so why should he have to meet your desired level of IT-knowledge to not be riducled by you?

On behalf of the poster - get a life, you sad git. anyone who must stoop to mocking the lack of IT knowledge of someone who simply uses a computer as a tool obviously has some serious issues.

Re:My recommendation

[Woy]

I use your sig in conversation all the time since i first saw it. Thanks.

Re:My recommendation

[Sivar]

Yeah, you're probably right.

As far as motherboards go

[Anonymous Coward]

The only way to get anywhere close to 64GB is by going multi processor. I have 2 Thunder K8S Pro (S2882) motherboard with 16GB of RAM each, running Postgres 8 on FreeBSD 5.3 in production. These systems are dual opterons. They have 2GB sticks. I also have a test system with a Tyan Thunder K8QS Pro (S4882), this one is a quad opteron and supports upto 32GB with 2GB sticks. The only way I know to get 64GB with 2GB sticks is with an IWILL H8501, that's an 8-way opteron barebone system.

I think you can get 4GB stiks now, but they are very expensive still.

Re:As far as motherboards go

[jarich]

Opteron boards can support 8 gigs per CPU. The singles support 8, duals support 16, quads 32.

This is the only way to get an affordable high ram board.

Sounds like for the poster's needs, he'd want to buy a dual and fill it with 2 gig memory sticks. After he's tinkered with 16 gigs of memory for a bit, he can decide if he needs more. The 2 gig dimms can be pulled over to his quad (or 8 way board) if he needs it.

Re:As far as motherboards go

[magarity]

The only way to get anywhere close to 64GB is by going multi processor.

You need to append to the end of the above: "motherboard but only buy one CPU".

Let's face it, one CPU with memory in the range required is a teeny tiny niche market. Nearly everyone else wanting that much RAM is going to want more processors as well. There are plenty of decent 2-CPU motherboards that can handle 32GB; it's not like you need to buy a Superdome or somesuch, so the original poster's complaint about multiple CPUs adding a lot to the price isn't very legitimate, IMO. For example, here's a URL for some dual CPU boards that take up to 32GB and start at a lousy $243 on pricewatch:
http://www.supermicro.com/products/motherboard/X eon800/

Re:As far as motherboards go

[RzUpAnmsCwrds]

"Opteron boards can support 8 gigs per CPU. The singles support 8, duals support 16, quads 32."

Not true. At my office, we routinely configure HP DL585 servers (4way Opteron) with 64GB of memory.

You need 2GB DIMMs to get 64GB, plus the right motherboard (most only have 4 DIMMs per CPU). But you definately can put 16GB on a single Opteron.

Re:As far as motherboards go

[Anonymous Coward]

Those boards have piss poor software raid via the Silicon Image 311x chipsets.

Re:As far as motherboards go

[Anonymous Coward]

The only way to get anywhere close to 64GB is by going multi processor.

You need to append to the end of the above: "motherboard but only buy one CPU".

You can't use the extra RAM banks if there is no CPU connected to them. If you want to populate all the RAM slots on an SMP board, you'll also NEED to populate all the CPU sockets.

Re:As far as motherboards go

[jarich]

You're right! I didn't know that... I guess the chipsets I've read about were the "consumer" grade.

A dual CPU w/32 gigs of ram... http://www.costcentral.com/proddetail/HP_ProLiant_ DL585/380125001/F85632/froogle/

How do I justify that one to the wife? "But think of all the money we'd save not having to buy a hard drive!"

Seriously, a few years ago at a bio-tech, building a ~huge~ database, we'd have done anything for something like this!

Re:As far as motherboards go

[Zigbigadoorlue]

Is it not true, of course with ought implementation, that an AMD64 technically can support up to 1 terabyte of RAM, this to avoid the ram limitations that have so plagued the x86 chips.

Re:As far as motherboards go

[AlexCV]

Remember, the Opteron's memory controller is on-chip. So to get 32GB on a dual-capable board, you need 2 CPUs. Those DIMM banks are hardwired to the pins on the CPU.

Re:As far as motherboards go

[SunFan]

No, the GP poster is now correct--the moderators have voted!

Re:As far as motherboards go

[CropCircleSystems]

good two others already set this dude straight. as far as concerning MP boards of procs that have their memory controller built in like an opteron, to max out the ram on a MB you must also max out the num of procs the MB supports, but there's nothing stoping a oem or mobo maker to fitting a nForce4-for-intel based MP board to one proc and virtually unlimited banks of memory since the intel version of the nForce4 has its memory controller in it and has an HTX pipe so as to allow a single chain type connection of one proc and as many ram banks as chained nForce4 SPP's as needed to fill this niche of needing lots of ram with relatively less processing power to use it.

Re:As far as motherboards go

[ColaMan]

Far out man, your post is just a blast from the past!

If you replace every GB with MB, it'd be soooo 1992 again. Well, except for the opteron references.... but you know what I mean, dammit.

Re:As far as motherboards go

[magarity]

. So to get 32GB on a dual-capable board, you need 2 CPUs

The MB I linked is a Xeon. The original article only mentions 64-bit as an aside. The primary objective of the questioner is a heck of a lot of RAM.

So to all you yammering about the Opteron's on-chip memory controller: tell someone who mentions an Opteron MB.

Re:As far as motherboards go

[magarity]

the moderators have voted!

Maybe the moderators actually bothered to check that link.

Re:As far as motherboards go

[Man Eating Duck]

I especially liked the disclaimer on the bottom of this page:

**4MB memory modules not fully tested due to limited availability

Yeah, I was happy with 2x2 MB :)

Re:Possible Problem

[Anonymous Coward]

You're one sick sockpuppet. All your posts reek of mental illness.

64 gigs as a RAM disk? Why not a HyperDrive?

[strredwolf]

HyperOS makes a IDE interfaced RAM drive named "HyperDrive III". It basically is a power-backed drive that stores the data onto ordinary DIMMs. Speedy drive, unfortunately it's up to 12 gigs now.

Re:I recommend an Apple

[Anonymous Coward]

I'm a linux user; I'm curious, what has apple done with libtool?

Skip Apple, go IBM OpenPower

[Anonymous Coward]

They run Linux, can hold up to 64GB of RAM and you can get a dual POWER5 (each of which is two to three times faster than the G5 used by Apple).

http://www-1.ibm.com/servers/eserver/openpower/

Re:Skip Apple, go IBM OpenPower

[waffffffle]

The Xserve is a fraction of the price. The clock speed of the G5 is considerably higher, so it should compensate at least partially for the differences between the POWER5 and the G5.

An Xserve seems like the most economical way to build the machine described, but don't buy all that RAM from Apple.

You can also get Xserves running linux from Terrasoft for the same price that Apple charges for Xserves running OS X.

http://www.terrasoftsolutions.com/products/apple /x serves.shtml

Another option: ASUS K8N-DL 24GB Dual Opteron

[alienmole]

The ASUS K8N-DL is a dual Opteron motherboard which officially supports up to 24GB RAM in 6 DIMMs. It costs only about $300, too. Not sure of current availability, it's pretty new.

Re:Another option: ASUS K8N-DL 24GB Dual Opteron

[Jeff DeMaagd]

Tyan K8QS supports four processors, has sixteen slots for up to 32GB. 24GB RAM in only six slots assumes 4GB ram DIMMS which I'm not certain are available yet.

Re:Possible Problem

[Zontar The Mindless]

http://kerneltrap.org/node/3816

Nice try, AC.

Server boards

[tweek]

"I've searched several websites (including Tom's Hardware), and I've googled, but can't seem to locate any commercially available AMD MBs supporting more than 4 sticks of RAM, or 4 gigs."

I'm actually building a system right now. Dual Opteron 224. The Tyan Thunder server boards are your best bet. This is the one going in my system:

http://www.tyan.com/products/html/thunderk8we.ht ml

It has 8 DIMM slots and supports 16GB of memory.

You just aren't going to find any i386 archs to support more than that right now. We have a few x345 servers from IBM that support 32GB but we run with memory mirroring for a total of 16GB rather than the full 32GB.

If you REALLY need more memory than that you're going to have to jump to pSeries/POWER5 line. You might be able to find a used POWER4 on ebay.

Even then I think our p570's only support up to 32GB using 2GB memory modules PER chassis/CEC/building block.

I'm not sure what your application need for the memory is but if you really need 64GB, you may have to buy a couple of servers and use something like memcached to get the full benefit. There is support for memcached in almost every language out there.

Re:Possible Problem

[Anonymous Coward]

I have acute intermittent porphyria, you insensitive clod!

Re:IBM eServer 326

[Junta]

Not so strangely, each set of 4 DIMM banks is hard wired to the memory controller of each processor.

Accesses from one processor to the further away bank must go through hypertransport, but as long as the OS scheduler is NUMA-aware, jobs should tend to run on the processor which is local to the memory it has been allocated, making for some screaming memory performance relative to single memory controller solutions (most Intel SMP setups).

Re:I recommend an Apple

[tomstdenis]

If you run the MacOS distro and not just gentoo/etc/other the libtool program has been fucked up.

It doesn't have the same interface as what you normally see on a Linux/BSD distro. The man page has Apple branding all over it, etc...

I don't actually have a Mac but when I've had to shell into one to do a build it's always been a pain in the ass.

Not that apple's Libtool is hard to use. Just that it's purposefully different as to make it not standard.

Tom

High memory density + UP = no

[Anonymous Coward]

More than 4 DIMM slots on a UP box? Forget it, it isn't going to happen. The problem isn't with addressing limitations; it's a physical hardware problem, along with issues of economics. DDR SDRAM is a parallel, shared bus architecture. The more memory slots you add to a channel, the higher the load on the channel and the more issues you have with signal integrity. You also run into trace routing issues trying to cram all the DIMM slots in while keeping your trace length equal so you don't get signal timing issues.

Now, these are all solvable, but it isn't cheap. The UP server market does not carry nearly enough of a premium for it to be cost effective to load a UP board up with more slots. So, the bottom line is if you're looking for massive amounts of memory, you're pretty much forced to buy at least a dual proc system, and more likely a quad proc system. High density memory modules do help, but again, they put more load on the memory bus, so look at the fine print to determine how many of those 2 and 4GB DIMMs you can toss on a memory channel.

Re:Possible Problem

[Anonymous Coward]

Sorry, not you. All the OP's posts are trolling.

Consider Intel

[Wesley Felter]

In order to populate all the RAM slots in an Opteron system, you have to populate all the processor sockets as well. And a 4-socket system requires expensive Opteron 8xx processors.

Instead, consider an IBM x366 or an HP DL580; either one can be configured with 32GB RAM and 1 processor for under $30K.

Re:Consider Intel

[ottawanker]

Just as a comparison, I offer the following. All Prices are Canadian, I could probably buy this system tomorrow, and I think the RAM may be a bit overpriced:

Tyan Thunder K8QS PRO Quad Opteron - $1817.00
AMD Opteron MP Model 846 Processor - $935.00
2GB PC2100 DDR ECC/Reg Memory - $769.00

Processors - 4x $935=$3740
Memory - 16x$769=$12304
Mainboard - 1x$1817=$1817

Total: $17861.00

Re:Consider Intel

[calc]

You can get the following for ~ $15K:

4 - Opteron 844 ($2176)
1 - Tyan K8QS Pro ($1370)
16 - Crucial 2GB ECC ($10720)

Toss in a few other devices (hd/case/etc) and you still end up well under $20K. So why would he want to waste nearly double the money for a 1 processor system unless he really has to only have 1 processor?

Re:Consider Intel

[dieman]

We got an 8-way opteron *with* processors for under $30k with 32gb of memory. With a .edu discount, but sitll it can't be much more for a business.

http://www.wsm.com/amd/servers/fusiona8/index.ht ml

Re:Consider Intel

[RzUpAnmsCwrds]

"an HP DL580"

Or consider the DL585, which supports 32GB with two Opterons. It runs $31K with 2x 1.8GHz Opterons and 32GB, plus you have the option to upgrade to 4-way/64GB later (or, with dual core, 8-way).

Re:Consider Intel

[kpharmer]

> So why would he want to waste nearly double the money for a 1 processor system unless he really has to only have 1 processor?

how about:
1. hot-swappable power-supplies
2. pair of scsi internal disk in a hardware mirror configuration
3. onsite 24x7 support
4. significant vendor testing that ensures that all components really are compatible

Re:Consider Intel

[calc]

The poster in question seemed to want it as cheap as possible to the point he didn't even want more than 1 processor. I don't think the poster really understands how much ram costs though since the ram alone costs over $10K, even a low end quad processor system is cheap compared to the cost of the ram alone. So he is unlikely to really care about any of those other options. But for items 1 and 2 you can definitely get that a LOT cheaper than another $15K.

Re:Consider Intel

[kpharmer]

Good points.

It all comes down to how reliable he wants it, and what his time is worth. If his time isn't worth much, and he can afford to scrap some parts a few times, the build-it-yourself route can certainly save money.

On the other hand, if his time is worth money, and if a loss of this machine's availability will cost money, it's worth spending that extra cash on a full-tested vendor solution.

Cenatek Rocket Drives

[Anonymous Coward]

You should look into getting a couple of physical ram drives. Cenatek used to make a physical card called the Rocket Drive ( I don't know if they still make them). Single cards can support up to 4Gb and I'm not sure on the possibility of using multiples together. You should take a look and maybe you don't even need a new/special motherboard.

USB Memory Sticks!

[Anonymous Coward]

This could be done using a USB 2.0 hub and lots of memory sticks! And it's not as daft as it sounds.

USB can handle up to 128 devices, and memory sticks are available at 2 gig now for $150 newegg dollars.

Read speeds should be good if you RAID them!

Re:USB Memory Sticks!

[Anonymous Coward]

I've been lurking on Slashdot since time immemorial, and that is the stupidest idea I have ever heard of. In fifty words or less, no, read speeds will not be anywhere near what he wants, and the costs are fucking stratospheric for something that could be solved with RAID 0 and some hugeass caching, or much better by mister infinite-budget here, with shitloads of RAM.

Re:64 gigs as a RAM disk? Why not a HyperDrive?

[tomstdenis]

Seems like a cool product but mentioning "makes no noise" like 10 times isn't that impressive ;-)

Also I wouldn't use that for installing an OS... much easier to screw that up than a normal drive. So at best you could use it as a "mirror" with commits to a real disk...

Almost like you could RAID-1 it with a real disk ... Just gotta patch the RAID driver to "favour" one disk over the other on read operations...

Tom

Power Processor...

[Anonymous Coward]

Go for one of those Power Processor systems. Mac uses Power5 (most). However, I think they can't get up to 64GB memory. Try to look at IBM pSeries. I looked at the entry level pSeries p5 550 Express, it takes up to 64GB of memory. The mid range server p5 570 Express takes up to 512GB of memory, which should be more then enough for your requirement.

It might also worth mentioning that the pSeries can run AIX / RedHat / SuSE. You may be able to try Debian and/or other distribution but might need a bit more thinkering..

Some possible solutions

[Hoser McMoose]

There are a few possible solutions you might want to look at for a big-RAM server. Now, if you really want 64GB and AMD Opteron processors than you really only have one choice, the HP Proliant DL585. That's the only Opteron solution that I know of which supports 64GB of memory.

If you can get by with a bit less memory then you have some other solutions. Tyan carries quite a number of boards with varying capabilities. The trouble here is that the Opteron processors are limited to 8GB of memory per processor, so to get 16GB you're going to be looking at a dual-processor board (quad processor for 32GB). Since the memory controller is right on the CPU with the Opteron you will actually need a second processor in the socket to use this memory.

For this reason, you might actually want to consider one of Intel's new 64-bit Xeon chips. I know that Supermicro offers some boards that can handle up to 32GB with only a single Xeon processor. Something like the X6DHE-XB seems like it might fit you're bill reasonable well. Fairly inexpensive to get you up to 16GB of memory, though going to 32GB is quite expensive. Crucial has a list of compatible memory for this board, including some 4GB modules.

Of course, if you're not limited to x86 systems then there are other solutions that would work. You could get something like an IBM Power system or Sun UltraSparc system with pretty much any amount of memory you need (or can afford).

Re:Some possible solutions

[homerj79]

For this reason, you might actually want to consider one of Intel's new 64-bit Xeon chips [intel.com]. I know that Supermicro offers some boards that can handle up to 32GB with only a single Xeon processor.

If you're hellbent on using an AMD solution, Supermicro is supposedly working on an Opteron board. This should provide a better "server" solution than a lot of the current boards offer.

Re:Some possible solutions

[TimSee]

Even better - the DL585 now supports Dual-Core Opteron. If you can afford 64GB of memory - but a good server like the DL585 - it supports Hot Plug memory and memory mirroring to boot. It also has hot plug redundant fans and power supplies so with 8 cores and 64GB of memory it should be able to handle a LOT of work.

Re:Some possible solutions

[TheNetAvenger]

The trouble here is that the Opteron processors are limited to 8GB of memory per processor

You're kidding right?

The 64bit AMD processors support a 40bit physical address space and a 48bit virtual address space, this means it can technically use up to 1 Terabyte of Physical Memory, and reference 256 Terabytes of virtual Memory.

Now lets put this in a real world senerio...

It is going to be impossible to find hardware or chipsets that support this much RAM, or even OSes that will natively support this much RAM.

So we will use WindowsXP64 (AMD Version). It fully supports 128GB of RAM as hardware allows for the AMD 64bit processor. (And this is not a Multi-processor configuration, as the Pro version of Windows XP64 (AMD) is only licensed for 2 processors.

As for Virtual memory, it supports 16 terabytes of space. This would be a rather large page file, but not impossible.

The 128GB physical limitations in WindowsXP64 are designed to meet initial and near future hardware requirements, as new hardware is developed that actually can even use 128GB for a DESKTOP OS, then WindowsXP will be updated to take advantage of the next jump in using the potential of 1 Terabyte of physical RAM.

As for the 8GB limit you quote, not sure where you got it or why you purport it...

(And I am NOT an AMD fanboy. We use Intel and AMD in our systems as demand needs for our desktop and medium server configurations.)

Re:Some possible solutions

[TheNetAvenger]

As for Virtual memory, it supports 16 terabytes of space. This would be a rather large page file, but not impossible.

Self Correction... Should read 512 terabytes - thats what I get for typing faster than my brain.

Re:Some possible solutions

[drinkypoo]

A single Opteron memory controller can address at most 8 DIMM sockets. A dual-channel Opteron system can therefore address as most 8 slots x 2 controllers x 2GB per slot or 32GB per processor, assuming it was 8 slots per channel, and not 8 slots per dual-channel processor. That's the real limitation here... not that it's much of a limitation. In fact the real limitation in terms of this ask slashdot is finding fully-populated boards in terms of DIMM slots.

A good Dual Opteron Board

[vectorian798]

Hey, here is what I think is one of the awesomest Dual Opteron boards, from Tyan:
Tyan Thunder K8WE

Specifications

It supports only up to 16GB, but has like almost any other feature you could want - has an EATX footprint, and fits int an ATX case if you are a small business type or something, or if you want a rigged-out workstation (dual PCI-Express x 16, for SLI'd Quadro's!!!).

However, if you wanted something more powerful for a larger-scale machine, then this board is probably not it. This is probably it:

Tyan Transport TX46

Specifications

Supermicro

[KingDaveRa]

Don't forget Supermicro. They do boards that support up to 32Gb. http://www.supermicro.com/products/motherboard/Xeo n800/

Re:Possible Problem

[SorcererX]

There is a performance hit with the current implementation going from the 1 GB model to the 4 GB model. This is quite noticeable with 1 GB RAM where the 1 GB model yielding ~900 MB of RAM versus the 4 GB model yielding 1 GB RAM actually gives you better performance.

Probably the right direction

[billstewart]

If what he's looking for is a "ramdisk", then that has a good chance of being the right direction, depending on how fast I/O his application needs (e.g. is the problem just that he needs 0-latency retrievals, or does he also need a fast bus speed?) (Cost is obviously an issue also - these devices are sometimes pricy.)

I'm curious how a device like this would get 3GB/s bandwidth - what kind of bus is it using? It's certainly way past PCI. Perhaps PCI-X, or plugging into AGP or something?

Another possibility, if cost is more of a problem, and bus speed is less of a problem, is to network a couple of motherboards together, with as much RAM as possible on each of them, and either GigE or Firewire.

Re:Probably the right direction

[Anonymous Coward]

The "ramdisk" is not needed. Simply purchase one or more Rocket Drive Solid State Disk PCI drives from CENATEK: http://www.cenatek.com/product_rocketdrive.cfm They sell them on e-bay for just a couple of grand in 2GB and 4GB sizes: http://cgi.ebay.com/ws/eBayISAPI.dll?ViewItem&item =5181816376&category=167&sspagename=rvi:1:1v_home

Re:Probably the right direction

[djdanlib]

While this is a nice device, it's still bottlenecked by the PCI bus. Notice the speed on the specifications page - a true SATA drive can achieve this, and Ultra320 SCSI can shame it as long as you get a 64-bit PCI or PCI-X controller (like http://www1.shopping.com/xPF-IBM_ULTRA320_SCSI_CTR LR_2, although I suggest Adaptec instead of IBM). See http://www.scsita.org/aboutscsi/ultra320/faq.html for info on Ultra320. You get a lot more storage for the buck here. It's not solid-state, but a 73GB drive (15k RPM) -- see http://www.storagereview.com/articles/200209/20020 901ST373453LW_1.html -- can be had for under a grand.

Now... about large memory systems... what is the purpose of such a large device? Databases? Image manipulation? Analysis? If the PHB just told you to go out and buy something with more buzzwords, I'm sorry, but that's probably not going to solve whatever problem you have. What would the benefit be of having a single machine, as opposed to an expandable cluster of machines? Besides the cooling and electrical, anyway. If you have a legitimate need for such a large amount of memory in a SINGLE computer, you should definitely consider a Sun workstation or (more appropriately) a brand new Sun server.

You really don't want to do that

[InfiniteWisdom]

Like any other modern OS, Windows finds pages that are in memory but haven't been used in a long time and writes them out to disk. In most cases it will mark the pages but won't actually evict them. If they get used, it simply unmarks them, if not the next time someone allocates lots of memory there is a ready pool of pages that can be used.

Its a tradeoff between extra disk activity when the system is idle or finding, writing and evicting pages when you actually need them. The former might result in some pages getting written out that didn't need to be, but the latter will result in extra work in the critical path.

Re:tom, you're a faggot

[tomstdenis]

Ah isn't he cute, he thinks he's people!

Tom

How's about these?

[jowaju]

Intel Xeon64bit Dual up to 32GB of DDR 266 http://www.supermicro.com/products/motherboard/Xeo n800/E7520/X6DH8-XB.cfm Intel Xeon64bit Dual up to 32GB of DDR2-400 http://www.tyan.com/products/html/thunderi7525_spe c.html

Got any links?

[Anonymous Coward]

See subject.

Re:SGI

[MikeBabcock]

The SGI Altix 350:

Advanced design that scales to 32 processors and 384GB of memory per system.

x86-64: HP for 64GB, Sun for 32GB.

[kalgen]

I've recently looked into exactly this question. So far as I can tell, the only x86-64 system that currently supports 64GB is HP's DL585 with 32 RAM slots. It uses 4 Opteron daughterboards each with 8 slots and you can put 2GB into each. You can put only 2 CPUs into it and still get 32GB -- again, the only thing out there that does 2 CPU 32GB in the x86-64 space.

Your other viable option is Sun v40z, with 4 CPUs with 4 slots each, for 32GB.

Re:Possible Problem

[tomstdenis]

Where did this come from? I just killed X, shut down apache/sshd/nfs and saw 55MB used [still have modules loaded like the 5MB nvidia driver...].

After I restarted them all and went back into X I see ~900MB free but that's *after* loading apache2, nfs, devfsd, sshd, a bunch of modules, firefox, X, icewm, tvtime and an xterm

If you say 1GB you will get upto 1GB for kernel+user processes. Maybe you only see ~900MB because you have too many background processes loaded. I know stock Fedora Core installs will have a bunch of useless stuff loaded [my box at work routinely uses >100MB of swap with only Gnome+mozilla running...]

Tom

Re:Tyan - try MSI

[ion]

i believe that MSI has, og used to have a dual opteron motherboard, MSI K8D Master 3-FA4R, that had 12 memoery slots and thus could have 24GB memory. However to use that much memoery you need 2 cpus. I can not find it at their american homepage. Maybe they discontinued it? Besides, those other comments about buying from a vendor is most likely worth considering.

many memory slots are a big problem for mobo maker

[asserted]

basically, with current signaling levels frequencies, etc. it has becoime increasingly hard to get many memory slots wired to the same controller.
i think it maxes out at 8 slots/controller for current DDR designs (i see this in intel E75XX 2 socket design, though i'm not sure they all share the same wire so it might be less actually).
so basically, 4-6 slots for 1-socket mobo is the most you can expect. for more you will have to get mb with more sockets/memory controllers. and yes, for amd you will have actually to put CPUs in to utilize those slots.

my recommendation would be one of current intel E75xx-serias mobos, where you get many (8) dimms and don't have to put in the 2nd cpu to use them.
to reach 64 gbytes you'll definitely have yto use some specialized hardware, most likely not x86.

Re:tom, you're a faggot

[Anonymous Coward]

For fuck's sake.

Stop responding to the god damn troll.

Someone mod this guy down.

Tried googling?

[neurophys]

Found some boards through Google:
First link:
http://www.mainboard.cz/mb/tyan/Thunderi7525S2676. htm

Pål

Re:IBM eServer 326

[Anonymous Coward]

I don't believe any OS treats 2-way Opteron systems as NUMA. (Solaris, Linux, Windows 2003 x64). The penalty for going to the other memory bank is less than the overhead for NUMA.

AMD 64 bit Arch == Mem controller in each CPU

[Anonymous Coward]

I'm surprised no one has mentioned that in AMD's 64bit cpu's, they have the memory controller built-in. So in a 4-way system, you WILL have to use 4cpu's to fill every memory bank. Ahhh, sweet NUMA architectures :)

Thanks, and some answers for the curious

[kimanaw]

Thanks to all for the pointers, guess I need to use better google search terms...

As to the purpose of the box, I can't say too much, except its intended as a sortof main-memory database cache solution (ala TimesTen ) to serve a potentially very large user base, with a lot of cached images associated with the cached data.

I also considered a piecemeal commodity approach, e.g., filling racks with cheap 4 gig RAM 1U's running Linux, but then I'd have to come up with s/w that can coordinate those systems (basically, a head-end to hash the requests to the right 1U). memcached looks intriguing, but I'm still concerned about interconnect latencies.

Based on my cocktail napkin estimates, the h/w cost of a big RAM system (if it can be got) would probably be about 2x the cost of the piecemeal system, but hopefully signifcantly reduce interconnect latency, and use simpler s/w (assuming I can get Linux to config a ramdisk that big).

Since latency is the prime concern, solidstate disks using the usual HD interconnects just didn't seem an acceptable solution, esp since the cost reduction doesn't seem that significant.

As for failure recovery, my thoughts were just a couple fast HDs that would ping-pong taking snapshots of the ramdisk; hopefully, the ramdisk recovery would never lose more than an hour's data (the lost data wouldn't be critical, and would be recoverable from other sources...its really the delivery speed thats essential)

But I may need to go back to my napkin and do some more queueing analysis to see if there are better commodity solutions.

Re:Thanks, and some answers for the curious

[ckaminski]

Why not couple a number of smaller boxes together with memcached and/or openMosix, so your 5 machines look like one super-server, and they manage response?

Re:Thanks, and some answers for the curious

[Anonymous Coward]

I saw these people at SuperComputing 2004.
RAM-SAN

why?

[RelliK]

I was with you until you said "ramdisk". Why? Linux is very efficient at caching data, so that ramdisk doesn't actually improve performance. What exactly are you building that requires such a weird configuration?

Itanium

[koafc]

How about an Itanium? They'll run Oracle (or other databases) and many support that much memory. I've seen some early Itanium 1's on Ebay for not a lot of money.

Are you on drugs?

[Some Random Username]

If network speeds were fast enough, he wouldn't be looking for a 64GB system, so memcached isn't terribly helpful. AMD64 is its own arch, just like i386 is, i386 isn't multiple architectures. And you can get 64GB AMD64 machines from both IBM and HP, and Sun goes up to 32GB on their AMD64 machines. And of course Sun goes up to hundreds of GB on their sparc64 machines, you don't "have to" use POWER5.

Re:Are you on drugs?

[tweek]

Well I was simply using power5 as an example. And I was using i386 in a generic sense to cover commodity pc architechure (allthough the term "PC Architecture" is slowly loosing identity.

The point is really that based on what he said in a later post about the purpose, interconnects COULD be fastenough depending on the budget. An IP over infiniband solution would kill the latency issue dead. And I would argue that even latency at gigE copper is fine with the proper infrastructure. Then again he could go to a FiberIP solution and really get solid.

Re:Possible Problem

[SorcererX]

no. "Some area of memory is reserved for storing several kernel data structures that store information about the memory map and page tables. This on x86 is 128 MB. Hence, of the 1 GB physical memory the kernel can access, 128MB is reserved. This means that the kernel virtual address in this 128 MB is not mapped to physical memory. This leaves a maximum of 896 MB for ZONE_NORMAL. So, even if one has 1 GB of physical RAM, just 896 MB will be actually available." - http://kerneltrap.org/node/2450

How about a RAM disk

[tacocat]

Instead of big RAM and a small hard drive, why not get a more sane amount of RAM, a small hard drive, and a solid state Ram disk?

That's probably going to be easier to find than a 64GB single CPU AMD motherboard...

Re:Possible Problem

[tomstdenis]

Want a shell on my box?

kernel + apache + sshd + nfs + modules + X + icewm + firefox + xterm + tvtime != 0MB of ram.

tom@tombox ~ $ free -m
total used free shared buffers cached
Mem: 1000 921 78 0 42 766
-/+ buffers/cache: 112 888
Swap: 976 0 976

What you're saying is JUST NOT TRUE for 2.6 on an x86_64.

My bios reports roughly 1022MB available to Linux, of which Linux makes ~1000MB available for modules/processes/etc.

If you're talking about the memory used for pagetables ... Um keep in mind that modern processors can map 4MB pages not just 4KB as in the 386 case. So you use 1024 times less memory by mapping larger pages.

So yes, if you have "many small" applications you're going to waste space for page table data, process information, book keeping, etc...

But you can alternatively load fewer larger apps...

For instance, I could load an application that takes 800MB of ram and not have it swap. In fact I rarely see swap space used even when I used to build KDE and the like... [hint: huge C++ source code...]

Tom

Re:Possible Problem

[SorcererX]

It is a problem on x86 with 1 GB max set in the kernel, it is not an issue if you enable HIGHMEM (4 GB memory support) in the kernel. The kernel used was 2.6.10 on Gentoo Linux.

Why not a Flash Drive?

[randomErr]

Instead of big RAM, why not install a 200 gig flash drive? Put the virtual memory on the flash drive. That way you don't have to play with funky mobo configurations.

IWILL 8-way server

[tonyhill]

IWILL makes an 8-way motherboard. They claim 128 Gb of RAM will fit.

I'm pretty sure that the 1U rack statement is an error, but there are 32 DIMM slots. I have no idea if one could actually get a 4 GB stick, but with 2 GB sticks that's 64 GB.

http://www.iwillusa.com/product_2.asp?p_id=90

Tony

Re:Possible Problem

[Unknown Lamer]

No, you are a retard. The 128M is virtual memory. It isn't mapped to physical memory unless it needs to be.

4GB DIMMs

[alienmole]

Here. Retail, looks like you can get these intended for Compaq servers. They even sell them at CompUSA.

Re:Possible Problem

[SorcererX]

on x86 doing cat /proc/meminfo yields: MemTotal: 1034424 kB HighTotal: 131008 kB LowTotal: 903416 kB Now, obviously if there was no HighMem support, I'd only have 903416 kB of memory available. it'd be interesting to see what you have on that x86_64 system of yours.

Re:Possible Problem

[tomstdenis]

really? Because on both my AMD64 and Prescott P4 linux boxes the vmallocused is less than 128M ...

At anyrate all I know is I really do have >896M of memory to use. I really do use the memory and I really think you're trolling about the stupidest issue.

Tom

Yep

[RJabelman]

http://www.cenatek.com/product_rocketdrive.cfm

I seem to remember that reviews of this thing said it was good, but pricey, and had firmware locked to the ram config you bought it with, making upgrades expensive.... You can't just buy an empty one and fill it. (All IIRC though.)

Re:I recommend an Apple

[Fancia]

Nothing's stopping you from installing a more traditional libtool, though. :3 You can build a Linux-compatible libtool with Fink; I have it installed for some software that requires it.

Re:Possible Problem

[Captain Segfault]

It is possible to get the best of both worlds with a 1GB system by changing the PAGE_OFFSET #define in include/asm-i386/page.h from 0xC0000000 to 0xB0000000.

There are two lines that need to be changed, right next to each other. Of course, if you do this then only 2.75G of address space will be usable by each process, which might cause problems with a few odd programs. (unlikely) Also, since the performance hits have been reduced recently it probably does not make sense to do this for any case other than 1G.

However, this is nice if you have exactly 1G; it's certainly better than just running with 896M.

Re:Possible Problem

[tomstdenis]

tom@tombox ~ $ cat /proc/meminfo
MemTotal: 1024648 kB
MemFree: 55804 kB
Buffers: 47580 kB
Cached: 788596 kB
SwapCached: 0 kB
Active: 184800 kB
Inactive: 724960 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 1024648 kB
LowFree: 55804 kB
SwapTotal: 1000432 kB
SwapFree: 1000432 kB
Dirty: 96 kB
Writeback: 0 kB
Mapped: 114424 kB
Slab: 36988 kB
CommitLimit: 1512756 kB
Committed_AS: 145732 kB
PageTables: 3400 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 20940 kB
VmallocChunk: 34359717255 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB

Re:Possible Problem

[SorcererX]

ah, seems the issue is only a concern on x86 then, not x86_64.

Most Dual Opteron Boards

[un1xl0ser]

Go to Monarch Computer. Look at the Opteron Boards.

Pick one.

128 GB od RAM!!!

[Sargeant Slaughter]

The company I work for sells an 8-way opteron box that can take 32 sticks of RAM, and it is compatible with 4gig sticks. We have had them up and running benchmarks with 128 GB of RAM at our lab. I don't know if we have them on our website yet, http://www.verari.com but we our engineering group finished testing and released them to sales on Tuesday. The thing can run with just 4 cpus too, with up to 64 GB of memory.

Re:Possible Problem

[tomstdenis]

Ah well then that could make the diff I guess...hehehe..

My P4 only has 512M of ram [1M eaten by an onboard video card thingy] so it's not applicable here...

Tom

and which OS would you like to run on it ??

[Anonymous Coward]

...it just could help us to point U in the right direction...

Re:I recommend an Apple

[tomstdenis]

Perhaps. My point is what benefit did Apple get from an apple branded libtool other than breaking compability?

Since I didn't own the MacOS [only needed it for one build] I wasn't going to install new tools on it. Instead I wrote a macos specific makefile to use their version of libtool.

Tom

ASUS K8N-DL

[zendal]

24Gb RAM with 6 slots and Dual Sockets if you want to chnage your mine and add another processor with the nForce Professional chipset that includes PCI-Express for video and regular PCI slots. Built in 10/100/1000 LAN, 8 channel audio, SATA II and etc. The price is $339 for its relase of 4/15. If that is not what you are looking for but similiar start looking under server/workstation boards. Instead of in regular motherboard section for the major motherboard manufacturers.

Supermicro boxes

[jonesboy_damnit]

Supermicro servers can be had in just about any configuration you like with up to 16 slots for RAM on the dual Xeon boards. Populate a single Xeon and a bunch of RAM and call it a RAM server. We've had good luck with Supermicro machines and Linux. The peripherals all have drivers, the sensors are all supported, etc. Good luck!

MSI K8D MASTER3-F4AR

[Anonymous Coward]

This might be the only (commodity) motherboard with more than the standard 4 slots per processor. The first processor has 8 slots, for a total of 12 slots on the board, at ~$450.

*IF* it supports 4 GB dimms, then you're pretty close to the target...

Uhhh...

[Anonymous Coward]

If you need all that RAM, but you're only looking for one CPU, maybe you don't really need all that RAM?

It seems to me like you're having a problem with scale. Other things you're not likely to find: Socket 940 MoBo's with ISA slots, Socket 940 MoBo's without SATA, Socket 940 MoBo's Socket 940 MoBo's without RAID capability.

Unfortunately the motherboard industry cannot make custom mobo's for people (unless people are willing to spend far more money, which you obviously aren't). People who need the amount of RAM you're talking about need dual CPU's as well almost 100% of the time.

Basically: spend the extra 100 dollars for the dual CPU MoBo, and then save the extra 200+ and only put in one CPU. Then you'll always be able to scale up in the future.

A system with a lot of memory?

[scottv67]

I was able to fit 32GB into one of these babies:

http://h18002.www1.hp.com/alphaserver/archive/es40 /index.html

It ran very well with that much RAM. :^)

Just don't try to load Windoze on it. :^)

Re:Consider the Tyan GX28 1u servers good bang

[RoundSparrow]

I also considered a piecemeal commodity approach, e.g., filling racks with cheap 4 gig RAM 1U's running Linux, but then I'd have to come up with s/w that can coordinate those systems (basically, a head-end to hash the requests to the right 1U). memcached looks intriguing, but I'm still concerned about interconnect latencies.

I personally agree with heavy RAM caching approach you have and find that we still have foolish RAM limits even though we are in the 64 bit world :)

That said, I have been running for 5 months in production the Tyan GX28 1U servers... both Windows 2003 32bit and Gentoo Linux 64-bit servers (several of each).

-- Onboard SATA is on the legacy PCI bus from what I understand and we have seen it max out at speed of two 7200 RPM drives (no raid, just copy drive 1 to 2 and drive 3 to 4 at same time). Good enough for our purpose, but notable.
-- As you have read here, to actually use > 4 memory slots you have to populate the 2nd cpu.

But overall a very solid system for the money. Bargain.

8 Way Opteron && the on-cpu memory archite

[hanulec]

When spec'ing a large RAM Opteron make sure to remember that unlike Intel Xeon systems, the memory controller is on the cpu die. This provides for great memory performance.. but requires that on a 2, 4 or 8 way system you must purchase a CPU to use the bank of memory adjacent to the CPU socket.

Blantant Ad: My employer, Western Scientific, has recently released their 8-way Opteron solution, the FusionA8, which supports 32 memory sticks when 8 Series 800 processors are used.

Why not just buy a ramdisk and install it?

[elrond2003]

For instance the M-Systems' Fast Flash Disk (FFD) Supports up to 34 GB website . Never tried it myself but it seems simpler than trying to roll your own. Also you can debug and develop with a conventional HD and switch to ramdisk only when necessary. I suspect there will be performance problems using the 32GB of a 4 processor board as a single extent of RAM.

My favorite

[Anonymous Coward]

The AM-CORE File Server
Max CPUs: 3
Max Memory: 256Gq
Max Security: 2
Max Bandwidth: 6Gq/s

"This Gateway has a huge amount of memory space, at 256gq (GigaQuads), you will be able to do 2 database missions with this and still have space left. The 3 CPU slots is a bit of a let down, but it is sufficient for most things, though you may be left wanting more power towards the later missions. 2 Security devices make it a nice safe gateway once you have the equipment fitted, but you really have to ask your self, 'Do I really need that much memory?'"

A G5 will hold at least 16G.

[JackAxe]

The 2G sticks have been out for about a year now. 4G sticks are what's new.

Apple doesn't list higher then 8G, because that is what they tested. 4G sticks will most likely also work. There is really no reason why they wouldn't.

"All that RAM." "far More data than you have resources to process." *LOL* It's safe to say that you don't work with video, nor as an artist. Are you working on a 386SX. :D *Kidding* I could be wrong though. :)

I have over a Terabyte of HD and my system handles it just fine. My 7 to 8 minute video jobs export single video files that average about 12Gigs. If I had 16GB of RAM and AEPro supported it, even with that Much RAM I would still need to "purge it all" every so often. I also generally run most of my pro apps at the same time since I need to jump between them. When they support 64-bit memory, my 5 G of RAM will not be enough. I'm upgrading to Tiger since it allows 64-bit memory support for apps now and CS2 will take advantage of that, at least on the Mac. I'll be allocating 4 gigs of RAM into it, since some of the poster illustrations I work on have exceeded over 3.5G in RAM (And since PS could only see 2G, it had to rely on scratch disk(s) to make up the difference.). That would only leave 1G for everything else, so needless to say I'll also be upgrading my RAM when I buy Tiger to more then 8G.

What I work with is peanuts compared to what a friend of mine does. He works with satellite imagery and a small plot of land can easily exceed over a Terabyte for one image.

Anyways, 16G of RAM is nothing with current comps and I bet that in about 4 years you'll probably have at least 6Gs in your system. Go back just a few years and 1G of RAM was considered more then most peeps would ever need or use.

Re:A G5 will hold at least 16G.

[diamondsw]

I'm not saying you don't need that much memory; my point was that any task that uses that kind of memory would also benefit hugely from additional processors. Once you have 16GB+ of *any* kind of data you're going to need to run it through the CPU at some point, so you better have all the horsepower you can get - otherwise you shift your bottleneck from I/O to the CPU.

Why x86?

[Ewan]

I can't see any obvious reason for sticking to amd or intel here? Unix hardware has been doing this for years, and the open power servers from ibm are pretty cheap, and extremely reliable.

IBM open power 710

size of pointers

[Anonymous Coward]

one significant issue is how large your pointers can be since you would need to have an unique address for everything in ram. most commodity CPUs are
32 bit -> 2^32 ~ 4.3 billion -> 4 GB ram
but with a 64 bit processor, you can have substantially more. obviously, you would need a 64 bit operating system to match.

AMD's offerings can't address 64-bit memory.

[JackAxe]

I'm not trolling, but most peeps I encounter don't know that AMD's flag ship the Opteron can only address 40-bit physical and 48-bit vitural memory. That's 1TB of RAM, which is of course is a crapload, but it's not even close to what 64-bit memory can address; 16 Exabytes. I'm just being anal, since no single system that I know of can hold that much ram. =P

DmitryProletariat Troll Moderations

[Anonymous Coward]

Oh, I suppose I really am "Trolling" in the sense that it's not honestly ontopic. But neither is it rude or personally insulting. It's meant just as fun satire and parody in the old slashdot tradition of OOG_THE_CAVEMAN and "Natalie Portman, covered in steaming hot grits." I think most of us old timers still have fond memories of those days, and that's what I'm trying to evoke. BTW: my subnet just got banned (using a nearby wireless net), so no more posting for DmitryProletariat! At least for a while... Sheesh, try to entertain a few folks and this is what ya get...

Thanks for a fascinating comment. Hope you're noticed and modded up! Now capitalist pig, against that brick wall!

*bang!*

Re:DmitryProletariat Troll Moderations

[Anonymous Coward]

Greetings from the notorious underground freedom fighters, the Subnet Liberation Front.

We bring grave news to our honored comrade Dmitry Proletariat-we are unable to locate our account on slashdot. Although the lying capitalist pigs claim we have merely forgotten the account name, it is plain as day that the evil slashdot editors have deleted the account outright in retaliation for our valiant trolling protesting the unfair banning of our subnet. Beware, friend Dmitry, for the trecherous editors may soon delete your account as well. Lock your doors at night and keep nothing written down!

Down with the editors! Free the subnets!

Re:DmitryProletariat Troll Moderations

[drinkypoo]

Trolling is when you espouse a view which you do not personally hold for the purpose of taking someone in and eliciting a desired response. It is not surprising that you do not understand what trolling actually is, in that 99% of Slashdot does not understand it either (at least, based on how I have been moderated in the past.) The worst you are guilty of is Offtopic, Redundant, or perhaps Overrated. Personally, I dislike all three of those moderation options, though Offtopic does have its uses.

Due to a particular infrastructure need...

[SpaghettiPattern]

You say "Due to a particular infrastructure need"

I/we read "Damn them memory leaks!"

;)

Re:Possible Problem

[calc]

The 896MB limit does not affect x86-64 as Tom has already mentioned, though it definitely does affect x86 (32bit). You can not access above 896MB in the normal 1GB mode on 32bit x86. To be able to use the full memory available you have to enable the 4GB or PAE options which cause a ~ 15% performance hit from what I have read in the past.

Well, Careful, you're limited with some still.

[OS24Ever]

The EM64T dual proc systems are going to be limited to 16GB Maximum due to the lindenhurst limits, and to top it off they're also rank-limited so you're going to have to buy single rank 2GB DIMMs to get to 16GB, not a inexpensive option.

I've not seen a dual opteron system go beyond 8 DIMM slots total, or four per proc. This is a limitation to DDR1 I believe. DDR2 will help some (speed of DIMM) but really FBDs next year will help out a lot too. Again, not an inexpensive proposition.

Your best bet for 'lots of slots' would be a quad or higher capable system. IBM's x366 and the other 4-way boxes out there have around 16 memory slots, and with 4GB DIMMs you could crank it up to 64GB.

64G ramdisk??!

[Maljin Jolt]

By my opinion the idea of wasting a precious, fast and costly memory for just a ramdisk of that hideous size is a product of pure lunacy and clearly indicates there is something extremely wrong with your software. For the price of the memory itself, you can certainly design and built a more adequate hardware infrastructure for your task if you use some brain first. But wait, you are not a Longhorn core developer, aren't you?

Re:tom, you're a faggot

[Black Copter Control]

For fuck's sake.
Stop responding to the god damn troll.

Personally, I find it kinda funny -- not funny enough to mod up (I have the points), but funny nontheless.

Probably won't find one

[ocbwilg]

There are a couple problems with what you propose. Firstly, most applications requiring 16 GB or more of memory usually require (or at least benefit from) multiple CPUs. Such applications would be databases, sophisticated modelling systems, etc.

The next problem is the electronic signaling. Getting 4 memory slots to work together at today's speeds can be tricky. That's why in the past couple of years it has become increasingly important to use memory modules that are listed as supported by your motherboard manufacturer. Wiring together more than four slots and getting the signalling/timing down right is much more difficult. This isn't as big of an issue on SMP Opteron systems because each CPU has it's own memory interface and dedicated memory, so 8 slots on a 2-way equals 4 slots per CPU, still with easily achieveable goals.

That leaves you with having to fit larger memory modules into your four memory slots. The largest that I have seen generally available are the 2GB modules. I wouldn't be suprised if someone were selling 4GB modules, but they will be very hard to come by and very expensive. Right now you can buy 1GB ECC modules for around $280-$300 each. The 2GB ECC modules are about $800 each. I can't imagine what 4GB modules would cost, but I know that I wouldn't want to pay for them.

Since you were talking about using it as a RAMdisk a better option might be a solid state hard disk.

Re:Probably won't find one

[ocbwilg]

How odd. The INQ ran an article that addresses this very issue today:

http://www.theinquirer.net/?article=22589

Like this...

[Anonymous Coward]

http://www.sun.com/servers/entry/v40z/index.jsp

As noted above, Choose a Sun

[Chitlenz]

http://store.sun.com/CMTemplate/CEServlet?process= SunStore&cmdViewProduct_CP&catid=116125

specifically...

We use them exclusively now for large scale visualization systems (MRI).

The v40z is a monster, check the review at anandtech

-chitlenz

ps - run win2k3 great =)

Xserve

[Johnny Mnemonic]

You can configure an Apple Xserve this way, with either one or two CPUs. Not an AMD board, but it is 64bit, and you can still put Linux on it if you like.

$14,599--One CPU, 80GB HD, CD-RW, no video support, but 16GB of RAM and an unlimited user Server OS. You can do better if you purchase through the Education or Government channels, and you can do better if you purchase the 2GB DIMMs elsewhere.

Although I actually couldn't find 2GB DIMMs at the popular aftermarket places, but they are now available from Apple direct (just be sure to get the Xserve with at least one 2GB DIMM, to be sure it has support for 2GB DIMMs on the MLB).

What about a cluster?

[eaman]

I'don't know what you have to run on this large amount of ram, but did you consider some kind of cluster setup? One key benefit appears repeatedly with respect to clusters: Within a given budget, a cluster may be the most cost-effective, scalable way to provide large amounts of hardware resources (RAM and CPU) to special applications that can make use of them. A simple comparison between SMP and cluster hardware costs illustrates one of the reasons that clusters are candidates for replacing large, expensive SMP systems. Figure 1 shows this comparison. Cite from: http://www.informit.com/articles/article.asp?p=372 008&rl=1 Regards.

HP DL585

[RzUpAnmsCwrds]

The HP DL585 is a 4U rackmount Opteron server.

It takes up to 64GB in a 4CPU configuration, or 32GB in a 2CPU configuration.

Loaded with 2 1.8GHz Opterons and 32GB, it runs $31,538.00. Not cheap, but it *does* have redundant power/cooling, remote management (ILO), and the capability to expand to 8way/64GB (with dual-core CPUs).

PAE? Ick...

[Omega996]

If you've got a server that is actually working with more than 4GB of RAM, you might take a significant performance hit to use PAE. HP (nee Compaq) used to have a whitepaper up that showed the performance difference between two of their DL-series servers when used as Windows Terminal Servers. It showed an increase in the number of simultaneous sessions the server could support when going up to 4GB of installed memory. The number of supportable sessions didn't increase, though, with more than 4GB of RAM. PAE was enabled on the version of 2000 Server that was used for the test, but I suppose it's possible it's a limitation of Windows, rather than PAE. YMMV with *BSD or Linux, I suppose.
Incidentally, this little tidbit wasn't the point of the whitepaper, but it was an interesting artifact. Saved my company some money (to put into the veep's back pocket) to put no more than 4GB of RAM into a bunch of WTS boxes we purchased.

Tyan

[Anonymous Coward]

I've googled, but can't seem to locate any commercially available AMD MBs supporting more than 4 sticks of RAM, or 4 gigs.

Some Tyan motherboards will take quite a lot of RAM. But this one means you'll need to have 4 CPU's to use all those banks.

Go buy a DEC alpha off of EBay

[Anonymous Coward]

There are systems like you are looking for that are being let go all the time. If you troll ebay for a while you can find a nice one. I put "alpha 8gb memory" into google and got a link to an rs60000 with 8gb and another with 16gb.

Some answers if 16GB will do it

[Anonymous Coward]

1. Current Xeons support what Intel calls EMT64, which is basically the same thing as what AMD does where it can do 32 or 64 bit code. There are plenty of Supermicro brand motherboards that certify 2GB sticks of memory available for Xeon processors and have at least 8 slots.

2. Tyan S2885 (dual AMD Opteron) certifys that 2GB sticks will work and has 4 slots per CPU, 8 slots total.

3. Although 16GB is not listed on their website as a possible configuration, they do sell the above configurations if you call them, they are J&N Computers, http://www.jncs.com/.

Solid State Disks

[lesinator]

probably config'ed into a big ramdisk

Why not get a regular server and a solid state disk? Check out the Solid State Disk Buyers Guide

Optimize your system

[BobPaul]

offtopic: anyone know how to stop windows from swapping when there is 500mb+ of free ram? it's really annoying, and just putting the swap to 0 on all drives doesn't really solve the problem either(and some soft freak off from it, this is on XP). I hate having 1.5gb of ram and only seeing half of it used regularly while having windows swap horrendously.

Search for a decent cache manager. I recommend cacheman, but there are others. They often want to remain resident and free memory.. Since we're trying to fill memory, don't do this. CacheMan, in particular, lets you run a bunch of tweaks. Set your disk cache up on full, turn off "executive paging" etc. You might also want to try TweakUI, and I've had great luck with a cheap utility called "Advanced Registry Optimizer" although that's not exactly a program to solve your particular question.

Between a cachemanager program and MS TweakUI you should be able tell windows not to swap the system files out to disk, which will then actualyl cause a 0mb pagefile to have meaning...

Writing better software.

[Anonymous Coward]

There is a good chance you should rethink the software running on your server before considering spending as much as you are considering on hardware.

What is it you are running on said server?

I am guessing you are looking to store and read a lot of data very quickly. I would suggest (for data integrity's sake) that you look into solid state harddrives. If you loose power, you loose your data. Flash harddrives I am told (although I have never actually seen one) are I am sure quite speedy, and can store upwards of 60GB. You can use your RAM (4GB should be plenty) for caching.

knoppix

[Anonymous Coward]

I just found out the other day that lot's? of knoppix supports a 'tomem' boot parameter that sticks the whole cd in memory.
I don't have the memory to try it myself, but I've always wanted to do that - everything im memory... I guess you would want a cpu with a couple megs of cache.

spend the $$$ on programming lessons instead of hw

[Anonymous Coward]

dumbass probably doesn't know how to add an index, so every time his app executes a query against the DB, it does a full table scan :)

MSI K8D Master 3

[pogson]

This board has 8 sockets on one Opteron and 4 on the other. If you read the fine print in the manual, you will find you must go down to 266 DDR to use 8 memory modules on the one CPU.

Sounds like an SGI is exactly what you want ...

[Anonymous Coward]

With an SGI system you can scale almost any part of the architecture independently of the others.

A 1CPU Altix350 Module can take up to 24GB of RAM. Above that you have to add memory expansion modules.

The same system can scale up to 32CPUs and 384GB of RAM, all in a shared memory system with big low latency pipes.

I/O Scales too. It's 64-bits, runs Linux, has good fp perf yada yada yada...

Dell PE6850/6800

[Matt_Bennett]

The Dell 6850 (rack) or 6800 (tower) can take 16 DIMMSs- 32gig right now, 64 Gig when dial rank 4G DIMMs are available- it even supports hot-plug memory, up to 1.5 TB of storage (5 300GB SCSI drives). Up to 4 processors (Intel 64 bit).

Its not the memory, stupid

[Anonymous Coward]

If you want to run multithreaded apps, buy an Intel board, if you want singlethreaded, amd is the way to go.
If you can wait till FBDs are available, it is worth the wait. That will take one year exactly. You get 8GB sticks and on a board with 8 DIMM slots, you get 64Gigs.

My Previous Ask Slashdot.

[CaptCanuk]

It wasn't exactly the same, but there might be some good leads in the "Ask Slashdot" I had on 16GB RAM system purchases:
http://ask.slashdot.org/article.pl?sid =05/02/22/09 30238&tid=198

Re:MSI K8D Master 3

[ottothecow]

although, if he is looking to use them as a RAMDisk, 266DDR is still going to outpreform a HDD.

nforce?

[SanityInAnarchy]

I thought I read that the new nforce chipsets support 32-64 gigs. But that's the chipset. I haven't seen any mobos that do that.

No, he's not kidding.

[Anonymous Coward]

Integrated northbridge, remember? That give a 4-slot max, combined with 2 GB/slot, makes 8 GB/processor. Not too unreasonable, and you get a big latency bonus from the integrated northbrodge.

Now, you can, in theory, interface more RAM via HyperTransport, but nobody does that. It's simpler just to add more processors.

Bus timing

[Anonymous Coward]

As memory gets faster, the layout of the wiring gets more critical. Back in the days of asynchronous RAM (SIMMs), there were huge arrays of SIMM sockets. Then came SDRAM, and everything had 4 DIMM slots... until it started getting faster and faster, when a lot of motherboards fell to 3, because the timing was just too marginal.

Now we're at DDR, and 2 slots per bus is standard. Northbridges or AMD processors often have dual DDR buses to get 4 slots.

Actually, a two-sided DIMM *is* two separate DIMMs logically, with two chips connected to each data line. But that's the limit on the capacitance you can connect to the bus without slowing it down too much, and on the stub length.

Branches in buses are Bad - the "stubs" lead to weird electrical echoes (reflections) that mess up the signals. It's much easier to make a single wire longer (Gbit ethernet, or SATA, or FireWire, or USB 2, or...) than to have a branch - even a very short one! - and make the system work.

Parity Anyone?

[Anonymous Coward]

Has anyone given any thought to data corruption? A multigigabyte RAM disk ought to have some serious ECC. A non-parity RAMdisk would fall easy prey to psycadelic cosmic rays.

SGI

[wheezl]

http://www.sgi.com/products/servers/

Re:SGI

[SunFan]

SGI could build a 50,000 node single-image system with 15,000TB of RAM, but it doesn't matter if no one buys it. Sun has already shipped tens of thousands of Opteron servers just in the last six months.

Ain't gonna work with one CPU

[Namarrgon]

As others here have said already, Opteron is a NUMA design - each CPU has its own bank of memory, and other CPUs have to ask that CPU to retrieve data on their behalf.

This means that on a dual mobo, half the RAM slots are wired directly to the second CPU socket, and with no second CPU installed, that half of the RAM is simply inaccessable to the first CPU. You need that extra CPU - or a single-CPU mobo with a lot of RAM slots.

Re:Ain't gonna work with one CPU

[magarity]

As others here have said already

As others have not bothered to notice, the MB I linked wasn't an Opteron so don't tell me about needing two CPUs.

Re:Ain't gonna work with one CPU

[Namarrgon]

So, which 64-bit OS will you be running on that MB then? Or perhaps you have a particular Linux app in mind that can take advantage of 32 GB via PAE?

Re:Ain't gonna work with one CPU

[magarity]

Windows Server 2003 Enterprise Edition, in 32-bit, supports 32GB of memory. So no, I don't have a particular Linux app in mind and I didn't say a 64 bit OS was needed to be run on a 32bit system, which would be a little strange.

gotta love the Chinese megamachines

[DABANSHEE]

Oh & those $29.95 DVD players too.

Buy an IBM / SUN / HP ....???

[rew]

I see lots of reactions that suggest to buy an IBM or other "High quality" computer. ("For under $30k").

But this is not the issue. The idea behind the question is that RAM now costs some EUR 120 per Gb in my local store, so for EUR 1000,- I can buy 8G of RAM. Spend another EUR 1000,- and I can have an 8G machine, right? (For americans, read dollars where I say EUR.)

Well, it turns out that this is difficult. Modern CPUs have the memory controller integrated in the CPU and it is not easy to make a motherboard that can address more than the standard amount per CPU.

This means that the motherboard is not going to be available for say "under EUR 1000".

Re:I recommend an Apple

[mollymoo]

And Darwin is based on... FreeBSD. There is a lot of other stuff but there's a whole bunch of BSD too. Common knowledge.

If you can't find it...

[Sloppy]

..maybe consider getting a multiprocessor board, but just use old'n'obsolete CPUs. The price of two "slow" Opteron 240s, is totally dwarfed by the cost of 64GB of RAM.

solid state disks

[Anonymous Coward]

why not use one of these

Why use system memory?

[elemur]

If you are only looking at a RAM disk, look at solid-state disks. You can get ATA/SATA attached solid state drives to give you some capacity.. or go for more professional products, like the equipment from Texas Memory Systems for example. Their products are Fibre Channel attached with memory and drive backups for the RAM. You can see them in use for very fast storage, as well as for speeding databases (put logs on the RAM drives, for example).. It would be difficult to loose your data from that sort of drive, unless you wanted to.

Those sorts of systems are better engineered for concurrent access anyway. If you have a bunch of system memory, with a great deal of concurrent access, you are going to start hitting bus bottlenecks..

You should run MySQL

[Anonymous Coward]

Really, you should run MySQL instead of Postgres. It's faster and better.

The better solution is

[Anonymous Coward]

a hardware RAID card- one with a fully intelligent controller and a 64-bit PCI interface. You won't believe the access speeds, and the more disks you use, the faster it will be. Like a nice chunk of 10 drives, with some extra hot spares is good, in a RAID-5 for the best overall performance/efficiency.

Another idea- how about a cluster solution? Like a bunch of cheaper computers with some kind of shared networked storage, RAIDed together? Anyone know of such a system?

In the really bad old days of PCs we had these wonderful "extended" and "expanded" memory boards.
Anyone know of such a thing for PCI bus?

MySQL Native Clustering?

[ooglek]

MySQL supports Native Clustering over 100mbps/1gbps ethernet. It's like RAID, only you are using PCs instead of disks. You can add storage nodes as you grow (storage node == cheap PC with lots of memory).

Here's the setup:

2+ front end SQL boxes -- run all your SQL queries here. I _believe_ you store the full DB on disk here, but maybe you store it on the storage nodes.

3+ storage nodes -- the data resides here, in memory. You need 40GB? 8GB memory in 6 storage nodes will do ya. (giving you 8GB to grow)

1 back end administration machine, doesn't have to be 100% for this task, but should be highly available.

This way if one of your SQL boxes goes down, there is a second (or more) to cover. If one of your storage nodes goes down, you have enough to cover. Just like RAID baby, except for databases.

Why build it yourself when you could use an existing system to do it for you?

Though I might not store the images in the DB, it is a possibility...

ram drives

[Anonymous Coward]

i seem to recall a battery backed up pci type card that you fill with ram and acts like a scsi device....if that is your goal anyway

Penguinfy!

[Roadkills-R-Us]

I don't recall which mobos by themselves hadnle 8 sticks, but for a complete system, Penguin does. Of course, that's a 2 way, and it's 4 sticks per CPU so far...

Can't use Panta, they don't seem to exist yet

[Roadkills-R-Us]

At least, their web site is useless. It looks like a placeholder for a startup that has yet to actually produce anything. No idea who they really are, but that's what their website looks like.

That's a yes then?

[Some Random Username]

Dude, infiniband is still an order of magnitude slower than RAM, its not a substitute in any way, shape or form. The point is you are acting like many machines == a single machine with lots of RAM, and that is not the case. If the original poster doesn't actually need a single machine with lots of RAM in the first place, then of course many machines will work, and of course that would mean his question was completely retarded too.