Clustering with Consolidated Physical Storage?

Davathar asks: "My friend and I have been toying with the idea of building a cluster server. And in my research I've come up with a few questions I haven't found answers to yet. Right now this is just an idea we haven't put any money into. But if it seems feasible and fun, we may organize a LUG and build it. With dozens of individual machines standing alone and drawing power, the bill can add up quickly. So I've been trying to find ways to reduce power consumption and parts cost."

"It's always seemed to be stupid to convert AC to DC for the UPS, then convert back to AC for the built in power supply and then back to DC for the hardware. Does anyone know of single conversion solutions in the UPS market? Something that goes from AC to DC and then straight to the hardware with protection and backup?

And the next and probably bigger question is about hard drive storage. With the low cost of RAM these days it's very reasonable to put a GB or more RAM in a single unit. And with that much space in RAM, who needs a swap file? So my idea is to design the workhorse units with a heap of RAM and no local storage. This should in theory save power draw, equipment costs and increase reliability. But is it workable?

Linux should be able to boot from network, and I know there are Motherboards out there that support it as well. But I've never tried this. Do they still require a local disk for some other purpose after the initial boot? Would it be possible to run everything from RAM? Would the network traffic become a big problem when the units needed to read or write to the disk array?

What resources are out there that I may have missed?"

Comments

Silly!

[tunah]

So my idea is to design the workhorse units with a heap of RAM and no local storage

No local storage? Then where will you put the ram? This guy is nucking futs...

Re:Silly!

[JCMay]

Memory is not storage.

Storage is where the computer puts things that it isn't dealing with at the moment; disk space is the normal storage medium.

Memory is where the computer holds things that it is currently dealing with.

I like the hands analogy. The computer's memory is like your hands: if you're working with something, it's got to be in your hands. When you're done fiddling with it, you put it down (send it to storage).

This guy is just wanting to make a powerful diskless compute server, which is great except for one thing: most modern operating systems use, if not require, virtual memory (a region of storage that is used as a temporary holding area for thing that are not currently being worked on, but will be needed shortly). To the user, VM looks like memory, but it's really just storage in a funny suit. Having VM over a network would be painfully slow, I imagine. (Imagine how slow a Beowulf cluster of these diskless compute engines would be!)

PSUs that are UPSs

[Yarn]

Kinda in the right ball park:
http://www.magnumpower.com/sectors.cfm?mp_sector =e posfpos

Alternatively make something yourself, you lazy person! :P

Re:PSUs that are UPSs

[dotstar]

Hasn't the telco network (at least here in the states) been powered by DC power for years? Look for NEBS compliant gear, which will almost certainly provide 48Volt DC power inputs.

I think you can learn more regarding NEBS from http://www.telcordia.com/

Sorta

[CounterZer0]

It's possible to run everything in RAM (just use memfs's), and to run without a swap, but should a runaway process or memory leak get you with no swap, your machine is as good as dead ;) As far as the power conversion...such conversion smooths the line noise out of the current, giving cleaner, steadier power to vital internals.

Re:Sorta

[vrt3]

It's possible to run everything in RAM (just use memfs's), and to run without a swap, but should a runaway process or memory leak get you with no swap, your machine is as good as dead ;)

True, when too much memory is used in one way or the other, it can kill machine. But, I wonder, is it better when you have swap? I mean, I used to have 256 MB RAM, and 256 MB swap space. Now I have 896 MB of RAM, and no swap space. I would think chances of low memory conditions occuring are less with 896 MB RAM, even without swap space.

I guess it would even be better to have lots of RAM and a few gigs of swap space, but I'd rather use those extra gigs for an MP3 copy of my music CDs.

Gigabit network

[MJArrison]

Yes, your newtwork will quickly become saturated. Do the math, IDE66 is 66 megaBYTES per second. That's about 500 megabits/sec. So even a single computer could (theoretically) burst 5 times standard fast ethernet. I highly suggest a little Cat6 and gigabit networking for any clustering project.

Re:Gigabit network

[larien]

That's a peak value sustained data rates are likely to be much less, probably around the 20MB/s area; in practice, you could probably get by with Gigabit to the switch and 100Mbit (~8MB/sec) to the desktop. It's still slower than local disk, but you might be able to get by that with agressive caching on the client side.

Re:Gigabit network

[cloudmaster]

If your cluster nodes are wasting time by filling the IDE bus constantly, you need to move your data into a RAMdisk anyway. The reason for clustering machines is to get more aggregate *CPU* power - not to read data off of a disk. The nodes should be spending more time processing data than they do transferring it to disk. If they're not, then the solution should probably be tuned differently.

The cluster should be fine with commonly available 100Mb equipment - maybe with multiple NICs in the head node to keep network utilization down a little (since all of the other nodes will just be talking to the head node).

Some pointers...

[Manic Miner]

For UPS and Power supply discussions you can find previous ask slashot article on UPS's here and discussion on power drain from PC's here

As for network accses to storage etc. You need to be careful. People claim that ATA66 drives can transfer at 66 MB/s put this isn't the case. You need to take into account the file system and OS overheads on the file transfers. This said harddisk storage and access speed is the big bottle neck in servers. Having everything in memory would be great, if not then you need raid to ensure that you can get a decent transfer rate.

On the network front, use CAT6 cables. That way if you find 100Mb is too slow as the cluster expands you can upgrade to gigabit easily.

More UPS fun

[thogard]

What the UPS does it takes 12 or 24 or some other voltage (96 in the case of the one I just installed) and convert it to 110v (or 220 or 240) at 60hz (or 50). It does this by makeing lots of steped squarewaves and if you look at them with a scope, they look like little steps.

Your PC power supply then takes that AC and when it gets above 90 V, its starts charging a cap. when the voltage gets too high (like above 100 or so), it stops charging the cap. That energy stored in that cap is then switched at a very high rate through a transformer to charge up another cap to 5V and a different one to 12V and one for -12v etc. which goes off to your cpu and memory and hard drive.

The problem with running just 12 volts is you need to swtich it anyway down to 5 and if you put 12v on one end of a wire, it will be less on the other end. Pulling out the Disney book of electronics... Assume you get have a wire from lusers electrical supply and you put 12 v on one end and measure it at the other. Since its very bad wire, you get 11.5V. Now if you put 1000V on that wire (assume it has good insulation!), you will find the other end is 999.5V. So basicly your wires are going to have a fixed drop based on their length (the current remains the same). This is why the power compaines run 7500V on the power poles and switch it down close to your house.

Back towards the correct tangent...
If you have a large battery pack...
I've got 16 deep cycle batteries (think big car batteries) in two banks of 8. I get 96 Volts accross them in series. Now my over priced inverter takes that and chops it into nice square waves which get run into a toridal transfomer. somevoltage comes out the other side and its switched to chage up different caps at different voltages. Those caps are then switched so the voltage sort of looks like a sine wave. If I put an inductor on the line, it will look more like a sine wave but I'm not sure how the iverter will like it. If you plug in a cheap audio amp, you will hear the noise. The PC's don't care.

With the hassles I've had with UPSs over the years, I was tempted to just run the 96 volts off the battery to the PC's. The next time the UPS dies, this may happen. They don't care since they swtich on at about 90 volts, they will like the battery voltage.

When playing at home, just keep in mind that more than 48v dc can kill you and if it grabs you, you won't be able to let go.

And a final drift in the history direction...
Edison's electrical system was all DC at low voltage. He tried to show A/C (the Westinghouse & Telsa way) was dangerous because of its higer voltage and even killed an elephant which I'm sure you can find on the web somewhere.

Re:More UPS fun

[john@iastate.edu]

When playing at home, just keep in mind that more than 48v dc can kill you and if it grabs you, you won't be able to let go.

CURRENT KILLS, not voltage. Shuffle across the floor and touch a doorknob, that's many, many times more than 48volts and I doubt it has killed anyone yet who isn't a silicon lifeform.

Re:More UPS fun

[Phork]

a high quality ups will make true sine waves, not psuedo sine waves out of square waves. but with a pc, it doesnt really matter because it gets converted to dc anyways. but with things like televisions(and proably some monitors) it can cause problems.

Re:More UPS fun

[thogard]

High qulaity means motor generator pairs and more than $10k. The solid state ones can only produce square waves its just a mater of how many steps per cycle. The "true sine wave" UPSs will generate lots of square waves of different voltages and the sum will track a sine wave but if you have a good scope, you will still see the steps.

Re:More UPS fun

[thogard]

Current won't matter unless there is enough potental energy (aka voltage) for the current to flow. For thouse that are too caught up in forumlas they don't understand, think about a water system. Voltage is the height of the water tower and directly related to the water pressure. The resistance is related to the size of the pipe.

My 96V battery pack should be good for about 800 amps till something blows.

The thing will run an entire rack of gear for about 6 to 12 hrs and a reduced load for maybe a day but will take nearly 4 days to completely recharge.

Edison electrocuting an elephant

[bleeeeck]

There's a video on this page, and here's the story.

Diskless ?

[Anonymous Coward]

I don't see the difference between what you're trying to do and the thin-client diskless workstations that are on the market. Just put some boot roms on the network cards, stuff the machines with RAM and go for it !!!!

a few suggestions

[Phork]

It is no problem to have a machine with no local storage, people have been doing this for decades(well, at least one decade), you use nfs for the root file system. The are 2 ways to handle /etc, you can make a seperate nfs share for each machine, or you can make /etc generic enough that it can be used by all the machines, and just hand out IPs vai dhcp. the latter is the preferable method. It makes it very easy to make changes to all the machines in the cluster, all you have to do is edit 1 file.

there are 2 ways you can handle booting the machines, the wrong way to do it is to have every machine boot off of a floppy drive. The right way to do it is to get network cards that can accept a boot rom(there are lots of cheap ones that can), and use etherboot.

If you really need to, you can use swap over nfs, but its not going to be fast.

I suggest at least 2 nics in each node, one for the booting and nfs, and the other for for the the actual work happening on the cluster. gigabit is a good idea. switched is a necesity.

Is it just me?

[ReidMaynard]

Any place where your cluster server would eventually reside with either
1) be in a modern corporate building where whole circuits are on UPS; or
2) In you home (dreams) where UPS is not really a factor, because, after all, you are just screwing around.

having said that making a bootp server, then making a net-boot diskette for each node is pretty easy...

however, have you given thought to clustering programming projects?...unless you have some (one) of those, yer cluster is just a big heater...

DC-powered equipment

[Wesley Felter]

Almost all telco equipment runs on 48V DC. In particular, I know there are DC Sun Netras. Good luck affording them, though.

Re:DC-powered equipment

[stilwebm]

Occaisionally an older generation of sun equipment shows up for sale (on eBay even) that uses 48VDC. They usually get few bids so the price never goes high, but unfortunately this often means the reserve is not met. Still worth looking in to.

LTSP w/ MOSIX

[WarWizard]

There has already been some work into this. LTSP for your diskless boxes. There is also some info on running MOSIX with LTSP, here.

Clustering with Consolidated Physical Storage?

[Class V]

I apologize in advance for the commercial.
You may wish to look at www.rocketcalc.com. The diskless clusters might be of interest to you.

True, but...

[cr0sh]

Yes - current kills - not many milliamps of current are needed directly across the heart to stop it. However...

While I am not sure of the insulative breakdown voltage of skin, I know that it is possible to "feel" lower voltages, but typically they are "switched" or "fluctuating" voltages (like the phone ringer voltage - 50 volts (?) I think). Now, if you can feel that (probably due to less resistance due to sweat and other factors), I would think 48 volts from a battery bank could possibly be felt.

But, it would depend on the battery bank, of course - a bank of hearing aid cells could generate 48 volts, but at such a low current rating there wouldn't be any issue. Now, replace those cells with 12 volt deep-cycle marine batteries - and it is a MUCH different story. Such a bank could easily allow for a hundred or more amps - which could fry you.

As for "it grabbing you and not letting go" - because it is DC, and not AC (which crosses a "0" voltage threshhold per half cycle), your muscles will "clinch" up, not allowing you to let go of the source - I would imagine it would feel like a very bad muscle cramp - given enough current, I would think it might even tear the muscles (maybe, if they didn't cook first). Not a pretty thing to think about.

So, in conclusion - I agree with your statement that "current kills", not voltage. However, it is more complicated than that, because of the difference in breakdown voltage of an insulator (in this case, skin), the current needed to help that along, sweating, etc...

Re:True, but...

[Anonymous Coward]

Unfortunately for us humans, as we start to burn our skin resistance goes down, which means more current, etc. until you literally catch on fire.

You'd already be dead, but just in case you weren't, you'd still be dead.

That didn't make sense, sorta. You know what I mean.

What do you plan to run on it?

[zsa]

The software you run on the nodes is going to largely determine the network performance ... I would look pretty closely at where any intermediate files are created and make sure that they get stored in a ram disk instead of sent back and forth across the network.