Slashdot Mirror
K12LTSP + MOSIX Howto
Paul Nelson writes "Richard Camp posted a very complete, step by step guide to building a MOSIX cluster. "...The objective of this howto is to guide the reader on setting up a Mosix cluster with diskless nodes. The setup is based on K12ltsp Project. This should provide an easily scalable system."
This is exactly where I feel Linux should be used. The idea of dumb terminals and a central server has proven to be the most cost effective way for companies to implement computer technology.
It's becoming clear that Intel/AMD etc are going to crush most other general purpose CPUs. Be it with SMP or SMT or both. With the increase in PCI bandwidth coming and the heralded 64bit chips intel will start to take over more and more server machines. Remember in the steel industry people scoffed at mini mills, kodak scoffed at digital cameras etc etc.
In the future most companies will have dumb terminals and a server room with racks of cheap intel boxes. The OS on the server will be fault tolerant to the max, oh I lost a node ahh well only 255 left. Uptimes measured in years. Hang on a sec that sounds like an IBM or SUN mainframe.
What is rapidly becoming apparent is that network speed is now more important than CPU/MEMORY speed.
The Mandrake Mosix Terminal Project is extremely similar and is based on the k12ltsp concept. Check it out if you can. K12ltsp is great for rolling-out massive amounts of LTSP servers quickly.
put the what in the where?
ClumpOS is a bootable CD with network drivers that is pre-setup with a custom kernel that contains MOSIX and MFS out of the box with no work required. You can download and burn ClumpOS and then boot it on your slave machines.
As far as building your MOSIX master goes, I prefer Debian with the prebuilt easy to deploy MOSIX packages and kernel patches. The links to find both are below:
Clump/OS: A CD-based mini distribution
MOSIX on Debian
MOSIX is a fun, extremely useful tool. Just remember when building your Debian kernel to make sure to turn ALL options on for MOSIX, this includes MFS. Otherwise, you will have weird problems with not being able to migrate processes to your cluster.
-Pat
I think this is where clustering should be done, for now at least, at the thread level. Most programs are multi-threaded. Most people don't want to rewrite programs to support MPI or PVM. Lots of projects that previously had to implement their own clustering protocols can just utilize Mosix instead. If I could talk my boss into it, I would put Linux/Mosix on every desktop at work and have a giant Mosix cluster. This is the future of computing.
"I assumed blithely that there were no elves out there in the darkness"
This is exactly where I feel Linux should be used. The idea of dumb terminals and a central server has proven to be the most cost effective way for companies to implement computer technology.
[...]
In the future most companies will have dumb terminals and a server room with racks of cheap intel boxes. The OS on the server will be fault tolerant to the max, oh I lost a node ahh well only 255 left.
I'm trying to figure out what the benefit of this is. You'd have to maintain the user clients - which will still break down - and the server nodes on top of this.
You get fault tolerance - but user terminals don't need uptimes of years with transparent failover. You get centralized administration - but there are many ways of making this happen with user workstations too (witness the NT systems here that re-image their own drives every week).
Performance will always be worse with a centralized solution than with user workstations, because you have no local disk for fast scratch space (used by many applications in the environments I've worked in).
If computers cost $10k apiece, I can see cost being an issue, but if the cost of hardware and maintenance for a user's machine is much, much less than the cost of the user sitting at the machine, I don't see any justification on the basis of cost either.
How is this supposed to be a "most cost-effective" solution, again?
[Disclaimer: I think dumb terminal systems are nifty; I just don't think they're useful under most business conditions.]
This idea is used at CERN. Many desktops belong to a cluster (managed with Condor), but only when not in active workstation use. Therefore full clustering effect only becomes at night, but then again the daytime desktop use is not slowed down by batch work.
--
If you moderate this, then your children will be next.
Having set this up myself, it seems the author has a few misconceptions about PXE. These seem to be common, as I get into heated discussions on IRC with people who have never done this themselves, but seem to think they know better than I do for some reason. I may have some minor errors in my description below, but I think it's mostly correct.
:)
First off, his cluster isn't really diskless, since he uses floppies.
PXE is an Intel specification, but it is open as far as I know. Intel provides binary only daemons for PXE for Linux. PXE is a way to get around the 640k limitation that is inherent when using the bootp(or dhcp)/tftp boot methods.
PXE is not something that is supported in the kernel as the author implies. PXE is a userspace daemon that allows the workstations to download the whole kernel and also it can present some pretty complicated menus to the user. It is one type of bootstrap, and it is pretty complicated to set up. The PXE daemon for Linux isn't documented very well either, and requires some strange configuration of itself, and also of the DHCP daemon on the server.
Basically, the way I understand it, the DHCP process begins normally from the workstation boot ROM, and the DHCP returns a specific value that tells the workstation information about PXE. The PXE client then connects to the PXE server, and the user is presented boot options, which can be complex.
I didn't use PXE in my final cluster though, due to the extra complication. What I found out was that the SYSLINUX people write something called PXELINUX. PXELINUX is misnamed because it does not use PXE, rather, it is a bootloader that loads over the normal BOOTP/TFTP method, which is loads simpler to set up and maintain. PXELINUX should be thought of as a replacement for PXE.
Without a boot loader, a lot of the docs say you can just send the kernel to the directly to the client. This would work, but iff your kernel is less than 640k, as tftp/bootp operate in real mode, and they have to download the whole thing before they begin booting. (BTW the docs on diskless setups in Linux are extremely out of date for the most part)
With a raw kernel setup, it's also impossible to pass the kernel any boot options. It's the same as if you dd the kernel to a floppy device.
I gained a lot of knowledge about diskless booting in modern Linux in my setup, if anyone wants me to write a book, I'm open to offers.
-Gigs
gigs(at)vt(dot)edu-cational
I've had enough abrasive sigs. Kittens are cute and fuzzy.
Also, be sure to support OpenMOSIX
Apparently MOSIX is going to go closed source, so test out OpenMOSIX if you can, the project is really taking off and has several contributers, but it needs your help in testing the kernels. OpenMOSIX is being sucessfully used in major installations now, so it should be fine for what you want to use it for, and also you won't be getting yourself going on a (soon to be) proprietary path.
I've had enough abrasive sigs. Kittens are cute and fuzzy.
I'm using it at home, mostly it just crunches dnet packets all day. Dnet doesn't migrate over MOSIX due to its use of shared memory between threads, so a shell script in normal Beowulf style is used to start it:
ssh user@node1 dnetc
ssh user@node2 dnetc
[...]
Then a corresponding:
ssh user@node1 killall dnetc
ssh user@node2 killall dnetc
[...]
For something like dnet, this works well.
I've also played with distributed John the Ripper, but it also doesn't parallelize, so the way it has to be done is break the target shadow file up into equal chunks, the same number of chunks as you have nodes ideally. John does migrate though, so you can start and stop all the processes on a single node, and MOSIX will migrate them out.
I'm currently limited because I havn't set up MFS, which means I/O bound processes don't migrate. Once I set that up, it will open up the cluster to a whole new class of applications. Generally, MOSIX@home hasn't been as useful as I first thought it would be, as most desktop applications don't migrate very well, but if you do any heavy CPU bound stuff, then MOSIX might be for you.
I've had enough abrasive sigs. Kittens are cute and fuzzy.