K12LTSP + MOSIX Howto

Paul Nelson writes "Richard Camp posted a very complete, step by step guide to building a MOSIX cluster. "...The objective of this howto is to guide the reader on setting up a Mosix cluster with diskless nodes. The setup is based on K12ltsp Project. This should provide an easily scalable system."

Comments

MOSIX Deployment

[PatJensen]

There is a much better, easier way to deploy a MOSIX cluster. This article is poorly written, and missing several important sections. It is not for the kernel-phobic or beginner users. The preferred way to bring up a diskless cluster with easy tear down, no maintenance, and no network booting required is to use ClumpOS.

ClumpOS is a bootable CD with network drivers that is pre-setup with a custom kernel that contains MOSIX and MFS out of the box with no work required. You can download and burn ClumpOS and then boot it on your slave machines.

As far as building your MOSIX master goes, I prefer Debian with the prebuilt easy to deploy MOSIX packages and kernel patches. The links to find both are below:

Clump/OS: A CD-based mini distribution
MOSIX on Debian

MOSIX is a fun, extremely useful tool. Just remember when building your Debian kernel to make sure to turn ALL options on for MOSIX, this includes MFS. Otherwise, you will have weird problems with not being able to migrate processes to your cluster.

-Pat

Re:MOSIX Deployment

[Halo5]

The only problem with Clump/OS is that it doesn't handle multiple NICs well (it can handle it, but the installation process is a little tougher). This kinda sux because with MOSIX, its better to dedicate a NIC to MOSIX communication and use a second NIC for standard networking. The Clump/OS group is working to better this situation; hopefully they are making progress...

On a brighter note, the Clump/OS has a cool monitor program called ClumpView, which looks awesome!

Re:Desktop clustering

[distributed.karma]

This idea is used at CERN. Many desktops belong to a cluster (managed with Condor), but only when not in active workstation use. Therefore full clustering effect only becomes at night, but then again the daytime desktop use is not slowed down by batch work.

PXE

[GigsVT]

Having set this up myself, it seems the author has a few misconceptions about PXE. These seem to be common, as I get into heated discussions on IRC with people who have never done this themselves, but seem to think they know better than I do for some reason. I may have some minor errors in my description below, but I think it's mostly correct.

First off, his cluster isn't really diskless, since he uses floppies.

PXE is an Intel specification, but it is open as far as I know. Intel provides binary only daemons for PXE for Linux. PXE is a way to get around the 640k limitation that is inherent when using the bootp(or dhcp)/tftp boot methods.

PXE is not something that is supported in the kernel as the author implies. PXE is a userspace daemon that allows the workstations to download the whole kernel and also it can present some pretty complicated menus to the user. It is one type of bootstrap, and it is pretty complicated to set up. The PXE daemon for Linux isn't documented very well either, and requires some strange configuration of itself, and also of the DHCP daemon on the server.

Basically, the way I understand it, the DHCP process begins normally from the workstation boot ROM, and the DHCP returns a specific value that tells the workstation information about PXE. The PXE client then connects to the PXE server, and the user is presented boot options, which can be complex.

I didn't use PXE in my final cluster though, due to the extra complication. What I found out was that the SYSLINUX people write something called PXELINUX. PXELINUX is misnamed because it does not use PXE, rather, it is a bootloader that loads over the normal BOOTP/TFTP method, which is loads simpler to set up and maintain. PXELINUX should be thought of as a replacement for PXE.

Without a boot loader, a lot of the docs say you can just send the kernel to the directly to the client. This would work, but iff your kernel is less than 640k, as tftp/bootp operate in real mode, and they have to download the whole thing before they begin booting. (BTW the docs on diskless setups in Linux are extremely out of date for the most part)

With a raw kernel setup, it's also impossible to pass the kernel any boot options. It's the same as if you dd the kernel to a floppy device.

I gained a lot of knowledge about diskless booting in modern Linux in my setup, if anyone wants me to write a book, I'm open to offers. :)

-Gigs
gigs(at)vt(dot)edu-cational

Re:PXE

[Chad Page]

With Debian and the 82559-based Intel NICs I've used with PXE, I did not need any special PXE settings, just pxelinux and the dhcp-server and tftp-hpa daemons in Debian woody/unstable. I've successfully booted a kernel with a ~30-31MB initrd.gz (decompressing to 128MB) and it's not difficult to use at all. I'm building three diskless cluster boxen using the D810EMO which also works very well for this. You can even load memtest86 to test the box, since it uses the Linux loader.

OpenMOSIX

[GigsVT]

Also, be sure to support OpenMOSIX

Apparently MOSIX is going to go closed source, so test out OpenMOSIX if you can, the project is really taking off and has several contributers, but it needs your help in testing the kernels. OpenMOSIX is being sucessfully used in major installations now, so it should be fine for what you want to use it for, and also you won't be getting yourself going on a (soon to be) proprietary path.

Re:Any smart uses out there?

[GigsVT]

I'm using it at home, mostly it just crunches dnet packets all day. Dnet doesn't migrate over MOSIX due to its use of shared memory between threads, so a shell script in normal Beowulf style is used to start it:

ssh user@node1 dnetc
ssh user@node2 dnetc
[...]

Then a corresponding:

ssh user@node1 killall dnetc
ssh user@node2 killall dnetc
[...]

For something like dnet, this works well.

I've also played with distributed John the Ripper, but it also doesn't parallelize, so the way it has to be done is break the target shadow file up into equal chunks, the same number of chunks as you have nodes ideally. John does migrate though, so you can start and stop all the processes on a single node, and MOSIX will migrate them out.

I'm currently limited because I havn't set up MFS, which means I/O bound processes don't migrate. Once I set that up, it will open up the cluster to a whole new class of applications. Generally, MOSIX@home hasn't been as useful as I first thought it would be, as most desktop applications don't migrate very well, but if you do any heavy CPU bound stuff, then MOSIX might be for you.

Re:At last...

[benjamindees]

Distributed servers is an interesting proposal, and might be the solution in the end, but in the interim it is just not feasible. Clusters *always* run better when there is a central server to coordinate tasks. The algorithms for distributed coordination are just not there. Network protocols (P2P) for distributed file-sharing are getting closer, but are still not scalable without huge performance hits. And the most important point of all, $2000 will still buy a machine that is up to the task of serving hundreds of clients. When/if processing speed hits a ceiling (price/performnce) and home users no longer need twice the power they needed a year ago, distributed serving will be price effective. Until then, a structured environment offers so much more in terms of manageability that the price is worth it.

clustermatic!

[Anonymous Coward]

http://www.clustermatic.org/

Similar to MOSIX - bproc and a suite of tools for getting a diskless cluster up and going quickly. Very cool - used for clusters of 128->1024 nodes currently.