Clustering with Consolidated Physical Storage?

Davathar asks: "My friend and I have been toying with the idea of building a cluster server. And in my research I've come up with a few questions I haven't found answers to yet. Right now this is just an idea we haven't put any money into. But if it seems feasible and fun, we may organize a LUG and build it. With dozens of individual machines standing alone and drawing power, the bill can add up quickly. So I've been trying to find ways to reduce power consumption and parts cost."

"It's always seemed to be stupid to convert AC to DC for the UPS, then convert back to AC for the built in power supply and then back to DC for the hardware. Does anyone know of single conversion solutions in the UPS market? Something that goes from AC to DC and then straight to the hardware with protection and backup?

And the next and probably bigger question is about hard drive storage. With the low cost of RAM these days it's very reasonable to put a GB or more RAM in a single unit. And with that much space in RAM, who needs a swap file? So my idea is to design the workhorse units with a heap of RAM and no local storage. This should in theory save power draw, equipment costs and increase reliability. But is it workable?

Linux should be able to boot from network, and I know there are Motherboards out there that support it as well. But I've never tried this. Do they still require a local disk for some other purpose after the initial boot? Would it be possible to run everything from RAM? Would the network traffic become a big problem when the units needed to read or write to the disk array?

What resources are out there that I may have missed?"

Comments

PSUs that are UPSs

[Yarn]

Kinda in the right ball park:
http://www.magnumpower.com/sectors.cfm?mp_sector =e posfpos

Alternatively make something yourself, you lazy person! :P

Some pointers...

[Manic Miner]

For UPS and Power supply discussions you can find previous ask slashot article on UPS's here and discussion on power drain from PC's here

As for network accses to storage etc. You need to be careful. People claim that ATA66 drives can transfer at 66 MB/s put this isn't the case. You need to take into account the file system and OS overheads on the file transfers. This said harddisk storage and access speed is the big bottle neck in servers. Having everything in memory would be great, if not then you need raid to ensure that you can get a decent transfer rate.

On the network front, use CAT6 cables. That way if you find 100Mb is too slow as the cluster expands you can upgrade to gigabit easily.

More UPS fun

[thogard]

What the UPS does it takes 12 or 24 or some other voltage (96 in the case of the one I just installed) and convert it to 110v (or 220 or 240) at 60hz (or 50). It does this by makeing lots of steped squarewaves and if you look at them with a scope, they look like little steps.

Your PC power supply then takes that AC and when it gets above 90 V, its starts charging a cap. when the voltage gets too high (like above 100 or so), it stops charging the cap. That energy stored in that cap is then switched at a very high rate through a transformer to charge up another cap to 5V and a different one to 12V and one for -12v etc. which goes off to your cpu and memory and hard drive.

The problem with running just 12 volts is you need to swtich it anyway down to 5 and if you put 12v on one end of a wire, it will be less on the other end. Pulling out the Disney book of electronics... Assume you get have a wire from lusers electrical supply and you put 12 v on one end and measure it at the other. Since its very bad wire, you get 11.5V. Now if you put 1000V on that wire (assume it has good insulation!), you will find the other end is 999.5V. So basicly your wires are going to have a fixed drop based on their length (the current remains the same). This is why the power compaines run 7500V on the power poles and switch it down close to your house.

Back towards the correct tangent...
If you have a large battery pack...
I've got 16 deep cycle batteries (think big car batteries) in two banks of 8. I get 96 Volts accross them in series. Now my over priced inverter takes that and chops it into nice square waves which get run into a toridal transfomer. somevoltage comes out the other side and its switched to chage up different caps at different voltages. Those caps are then switched so the voltage sort of looks like a sine wave. If I put an inductor on the line, it will look more like a sine wave but I'm not sure how the iverter will like it. If you plug in a cheap audio amp, you will hear the noise. The PC's don't care.

With the hassles I've had with UPSs over the years, I was tempted to just run the 96 volts off the battery to the PC's. The next time the UPS dies, this may happen. They don't care since they swtich on at about 90 volts, they will like the battery voltage.

When playing at home, just keep in mind that more than 48v dc can kill you and if it grabs you, you won't be able to let go.

And a final drift in the history direction...
Edison's electrical system was all DC at low voltage. He tried to show A/C (the Westinghouse & Telsa way) was dangerous because of its higer voltage and even killed an elephant which I'm sure you can find on the web somewhere.

a few suggestions

[Phork]

It is no problem to have a machine with no local storage, people have been doing this for decades(well, at least one decade), you use nfs for the root file system. The are 2 ways to handle /etc, you can make a seperate nfs share for each machine, or you can make /etc generic enough that it can be used by all the machines, and just hand out IPs vai dhcp. the latter is the preferable method. It makes it very easy to make changes to all the machines in the cluster, all you have to do is edit 1 file.

there are 2 ways you can handle booting the machines, the wrong way to do it is to have every machine boot off of a floppy drive. The right way to do it is to get network cards that can accept a boot rom(there are lots of cheap ones that can), and use etherboot.

If you really need to, you can use swap over nfs, but its not going to be fast.

I suggest at least 2 nics in each node, one for the booting and nfs, and the other for for the the actual work happening on the cluster. gigabit is a good idea. switched is a necesity.