Linux Clustering w/Bootable CD-ROMS?

Cameron asks: "Has anyone tried to make a Linux cluster on a typical company/school network? I am trying to make a Linux cluster by taking bootable CDs and putting them in computers on an existing network. Red Hat (or another distro if it's better suited) would boot and run off the CD without needing any (or much) HD space. This way the computers aren't changed and I can have a virtual supercomputer for a while. Mmm 18Ghz. Anyway has anyone tried this before? Also I'd appreciate any suggestions on which distro to use and what cluster software/daemons I'd need. i.e. Beowolf or something like it." While an interesting approach to clustering, unless each node has quite a bit of RAM, I would think that you might need a marginal swap partition. What problems do you see with this idea, and would they be surmountable? Might such an idea be useful for quickly converting a computer lab to a cluster when it's not being used for other things?

Comments

Diskless cluster

[crow]

You should probably look at people who have set up diskless Linux systems. The root directory is NFS. You can even use a network device for swap space. You'll probably need some sort of network file system for doing your work anyway.

The advantage of booting off of NFS is that you don't have to burn new CDs when you udate the cluster. I don't have any experience with this, but I suspect that while most diskless systems use special network boot ROMs on the network cards, you could do the same thing with a CD boot. You might even be able to have a Windows application that shuts down cleanly and boots from the network (something like loadlin).

So I guess I don't have answers, only more questions to research.

Re:Diskless cluster

[heliocentric]

You should probably look at people who have set up diskless Linux systems. The root directory is NFS. You can even use a network device for swap space. You'll probably need some sort of network file system for doing your work anyway.

Yes, NFS would be needed for a distributed thing such as a beo since you need to maintain a shared storage for the applications to be run. However, you're going to have enough cross chatter on a decent network with this NFS and the beo already that adding the OS as an NFS thing would have a severe impact.

The advantage of booting off of NFS is that you don't have to burn new CDs when you udate the cluster.

Everything in life is a trade off - if you don't want the work of burning the disks, then you'll need the trade off of waiting extra long for a job to complete due to the network bottleneck.

Re:Diskless cluster

[larien]

Well, there is the option of having two networks; one for NFS booting, the other for inter-machine communication. Of course, this isn't really relevant for this question as it's about using existing systems in a Uni. However, for a planned system, it might well be a viable option, but a 1000 node cluster booting at the same time might be, ah, problematic! :)

zipslack

[rogerl]

Slackware (http://www.slackware.com) has a product called zipslack (http://www.slackware.com/zipslack/) which (as its name implies) runs off of a zip drive. That might work for you.

simpler application

[Anonymous Coward]

Here's a similar/simpler application I have been wondering about recently.

On my home network, I have an old P166 firewall/NAT box, it's no speed demon w/ tiny HD&RAM. My other machine is a Dual Cel500 with oodles of resources. I want to recompile a new kernel on the firewall but it just takes too damn long, so...
How would I setup the smallest/simplest cluster system to make the firewall box transparently compile on the dually box?

Well, gonna go try this bproc stuff now

(Of course, I realize the time to set up cluster support would be greater than just recompiling on the P166 :) )

Re:simpler application

[John Wilson]

Hmmm. Maintain a seperate kernel tree on your dual Celeron box which you also make available to the P166. Compile on the celeron, copy the kernel over to the P166, enjoy. Don't forget to check the little box which says "Pentium" when you do make menuconfig.

I don't know to what extent this has anything to do with "clustering" but it works if you really can't wait to compile the new kernel on the P166 (or if you were using a 486 with 8 megs of ram and a 100 meg drive as your gateway -- now that would be a case of wanting to compile on a different box. Unless your old distro you had on the 486 only had glib5. That's never happened to me nope nope nope.)

Diskless clustering.

[John Wilson]

I looked into doing something like this with a web server cluster a while back. The goal was to eliminate as many single points of failure as possible. The design goes something like this: Buy 4 machines with a CDROM drive, and a gig or so of RAM. Have a stripped down and hardened version of your favorite distro on it, and set it to create a ramdisk on bootup with the distro entirely in RAM. Put them all behind a firewall. Set up an internal gigabit ethernet network connecting all of these machines together, and put your backend database with all of your web pages on the internal network. Now, put in sysconnect gigabit ethernet cards in them (with the two ports on each card and drivers for linux/freeBSD/solaris/almostanyothernix which do automatic failover if you yank a network cable). Make the firewalls, routers, and database backend failover properly, and assuming you have two completely independently routed incoming network connections, the only single point of failure is incoming power and the fact that all your machines are in one place, so if you have a fire, you're kinda screwed.

Get F5 BigIP firewall thingies. They do 900Mb/s of real world traffic (USA Today's web page, Supreme Court website (remember Florida?)), etc. They don't pay me, and I've never worked for them; I'm just really impressed with their product. I don't even know if they make these anymore, but if they do, they're way better than anything Cisco makes (or, to be more precise, they were way better than any Cisco firewall product at the time I looked into this.)

Anyway, the main point, as it related to the poster's question, is that if anyone managed to get past your firewall and invade your box, he's only modifying what's going on in RAM. If you detect him there, and plug the security hole, you don't have to worry about what things he changed on what disks, and so on. All you have to do is burn 4 more CDs, take the machines down one by one (the F5s do load balancing as well as firewalling, so you aren't going to have any downtime upgrading your machines -- did I mention they're the shiznit?) and reboot off the new CD. Yes, I know that the database server still has disks and an attacker could get in and mess that up (again, this is assuming he got past the firewall, which is a pretty big assumption if you're blocking everything but port 22 and port 80) but by and large, it's a pretty interesting way of dealing with possible breakins. If you're really hardcore, you can run openbsd on the boxen (if openbsd allowed that whole foofy "boot into ramdisk" thing -- I have no idea whatsoever if it does. You could run it on the database backends tho.)

For various social and political reasons, we had to make this design work for either linux servers or solaris servers; hence the use of sysconnect gigabit ethernet cards, which work in both solaris and linux. However, I've heard that Solaris now has netcard failover built in; I'm not sure if the linux kernel has the infrastructure to do that or not (I don't think it would be too terribly hard to write it tho, if you had to.)

one word

[QuietRiot]

"Mosix"

You may need to configure one machine separately from the others to be the "master." You'll probably need either some kind of disk space availible or use some sort of ramdrive.

From the Mosix web page:

How to configure MOSIX clusters with a pool of servers and a set of (personal) workstations:

Single-pool = all the servers and workstations are used as a single cluster: install the same "mosix.map" in all the computers, with the IP addresses of all the computers.

Advantage/disadvantage: your workstation is part of the pool.

Server-pool = servers are shared while workstations are not part of the cluster: install the same "mosix.map" in all the servers, with the IP addresses of only the servers.

Advantage/disadvantage: remote processes will not move to your workstation. You need to login to one of the servers to use the cluster.

Adaptive-pool = servers are shared while workstations join or leave the cluster, e.g. from 5PM to 8AM: install the same "mosix.map" in all the computers, with the IP
addresses of all the servers and workstations, then use a simple script, to decide whether MOSIX should be activated or deactivated.

Advantage/disadvantage: remote processes can use your workstation when you are not using it.

There's this thing called a search engine.

[Wakko Warner]

For God's sake, USE IT!

Google is your friend.

Scroll down to the one that says "Beowulf Install".

Has anyone tried to make a Linux cluster on a typical company/school network?

Yes, and he was sued for it. Hopefully you have permission to do this.

- A.P.

Re:There's this thing called a search engine.

[heliocentric]

Has anyone tried to make a Linux cluster on a typical company/school network?

Yes, and he was sued for it. Hopefully you have permission to do this.

There are several things wrong with that statement. First is that the application the guy was running without permission was just an application - not an entire boot device setup to perform a beo. Secondly the application the guy was running was using the hardware to caculate things that the institution really wasn't looking for. Now, I can not comment for the person who asked /. as I'm not them, but I suppose he is asking because he has calculations he would like to do and very likely can tie these in to a school project (speaking for myself I know what I have stated is something I've often considered for some sort of school project).

Ok, now is your (what I think was your) basic message that "you should have permission because in a slightly related case someone was sued [and ended up with $2,100 fine and serve 80 hours of community service]" still valid - you betcha. Just don't give the guy any ideas that there is some criminal on the run who has all the answers about making a nice clean redhat boot CD for beo'ing.

ClumpOS

[barsigor]

See http://labs.psoftware.org it's a bootable CD-ROM Distro that can turn your school/office Windows PC into perfect mosix client !!! Bye

try this

[spongman]

The LWN Distibution list has a "CD-Based" section. I suggest starting with one of those, configuring it to use swap on a file in /tmp on /dev/hd??, and go from there.

scyld beowulf

[raulmazda]

Check out scyld.com for their beowulf distribution. It does exactly what you need (though there is a need for a dedicated head node system). You can run all of the slave nodes entirely diskless, and control booting into the beowulf stuff via a floppy, cdrom, or the hard disk.

You can pick up the CDs at cheapbytes (I think), so it's only a few $$$ for a basic install. Support or buying the professional edition will cost you bigger $$$.

The advantage of this is that all of your files and state and management is all done on the head node. The slave nodes boot up and pick up their configurations from the head node and go. Scyld beowulf is also significantly easier to install/maintain than rolling your own.

Finnix distribution

[djp3]

I've fooled around with Finnix , a bootable-CD distibution of Linux. It's pretty lightweight and worked well for me, but I'm not sure how well the networking support is. Anyway, it might be a nice place to start. -Don

LEAF

[Stinking Pig]

Have a look at leaf.sourceforge.net -- Dachstein and Oxygen both support booting from CD with configuration on floppy, and having your root on RAM disk will help improve speed and reduce the impact of a machine failure.

There isn't a Mosix or Beowulf setup for either yet, but if they'll compile in glibc 2.0.7 and kernel 2.2.19 you're good to go. If not, look under the devel section for Jacques Nilo's work; he's been doing a LEAF distro which gives up the floppy-sized goal and uses the latest glibc and kernel.