OpenMosix

Francesco Taurino writes "Moshe Bar has released a new Mosix system: openMosix. From the site: "For thousands of users Mosix has been a reliable, fast and cost efficient clustering platform. There are hundreds of Mosix installations in life sciences, finance, industry, high tech, research and government environments. The goal of openMosix is to give to these users a continued support and an up-to-date platform. openMosix is initially fully compatible with the last Mosix (1.5.2 for 2.4.13) kernel, but is now growing in its own direction. If you would like to contribute to the openMosix project, drop a line to moshe@openmosix.org.""

Comments

OpenVMS and clustering

[ASyndicate]

OpenVMS has the most robust, and fastest clustering ability I have ever seen . I just cant wait until linux can do this.

This is great... I think

[GigsVT]

I just finished implementing my 6 node MOSIX cluster, and I ran across several bugs, and I couldn't find any place to report them. The MOSIX development list is closed subscription, and apparently the good Professor ignores his email.

I'm not clear about some things though... How is MOSIX currently licensed? Why are they being so closed about development?

Re:Simple question: Why the split?

[VP]

Follow the link: About openMosix

After a difference of opinions on the commercial future of Mosix, I have started a new clustering company - Qlusters, Inc. - and Prof. Barak has decided not to participate for the moment in this venture (although he did seriously consider joining) and held long running negotiations with investors. It appears that Mosix is not any longer supported openly as a GPL project. Because there is a significant user base out there (about 1000 installations world-wide), I have decided to continue the development and support of the Mosix project under a new name, openMosix under the full GPL2 license. Whatever code in openMosix comes from the old Mosix project is Copyright 2002 by Amnon Bark. All the new code is copyright 2002 by Moshe Bar.

1.5.2 isn't latest MOSIX

[NicolaiBSD]

MOSIX 1.5.7 for Linux 2.4.17 (K-MOSIX) is out, according to Freshmeat. Therefore "the last Mosix (1.5.2 for 2.4.13) kernel" seems incorrect.

I never heard of mosix,

[whovian]

and since information is a bit lacking at the link provided, here's a link to the regular mosix FAQ.

Why the fork...

[Raleel]

I guess I wanna know why there was a fork. I respect both the big Prof and Moshe from what I have read of theirs. Moshe says that Mosix is going in other directions, which sounds kinda...vague ;)

Re:Why the fork...

[crome]

Well, Prof. Barak did not release any new versions until this morning (after my announcement). He closed off the mailing list and bulletin board. He complained to me repeatedly about the GPL and that he was going to close off Mosix and that Linus is a fool.

In the mosix website they speak about going to user-space, which is probably being done to avoid the GPL.

Just ask yourself, if they had Mosix for 2.4.17 (they released it soon after my openMosix, so it must have been there all along) ready, why didn't Prof. Barak give it to the user-community?

That's what I mean they are going proprietary.

Moshe Bar

Re:Why the fork...

[Raleel]

He closed off the mailing list and bulletin board.
Ok...that's not a great sign..

He complained to me repeatedly about the GPL
There are other licenses out there. Prof. Barak himself I suppose would ahve to answer what his issues with the GPL are. I wrote to him a while back and got the impression that he was not interested in the GPL

that Linus is a fool
If that's a quote, I've really lost respect for him. Linus is certainly not a fool.

In the mosix website they speak about going to user-space, which is probably being done to avoid the GPL.
While this is a possible outcome, this is a bit of fear-mongering...

Just ask yourself, if they had Mosix for 2.4.17 (they released it soon after my openMosix, so it must have been there all along) ready, why didn't Prof. Barak give it to the user-community?
actually, my first thought was that whomever hadn't gotten it to work with the new VM code, not that he was withholding it.

I guess my thought it is...He has the right to, I suppose. I'm glad that someone is taking it and keeping it not proprietary. It has the potential in some areas that I work with.

Re:Building a mosix cluster

[Hamshrew]

I don't have a link handy, as it's been a few months. I found it to be fairly simple to install... I had 4 PIII machines, set them all up on an internal network and nfs mounted a directory from the head. From there, it was a simple series of steps:

Unpack kernel sources.
Run the Mosix install script.

Did that on each node, then started the mosix service on each.

It worked like a charm for large computations, but had three flaws for normal use.

1) By default, it does not auto-migrate, which was pretty dumb. And getting it to auto-migrate was buried deep in the docs, though it could be guessed from reading up on locks. (echo 1 > /proc/self/lock, I think it was)

2) Migration only occurrs after a certain load average is maintained... if your job involves spawning multiple short-lived processes, like a large compile, it doesn't migrate anyway.

3) Network usage for migration was very heavy over Fast Ethernet.

There you have it. It's the last reason that MOSIX isn't used often in commercial clusters, but it seems well-suited for other distributed computing applications, and has some interesting features, especially for NOW configurations.

What MOSIX can't do

[Anonymous Coward]

http://www.ai.mit.edu/lab/sysadmin/cluster.html

There are several limitations to what MOSIX can
currently do. Java native-threads is one thing,
since MOSIX cannot migrate apps using shared
memory all native-threads applications will stay
on the node they started on, green-thread
based application can migrate though the
internal threads aren't exposed to the OS so no
real parallelism is achieved. MOSIX also can't
migrate sockets so I/O bound problems also
stay at home. Mixed I/O and CPU jobs can
migrate for CPU cycles, but are brought back for
the I/O ops. In the limited testing to date,
processes that can't take advantage of MOSIX
don't seem to be hindered at all by it

Re:What MOSIX can't do

[Hooya]

i'm been working with a 3 node test cluster (mosix) for viability for our apps. we tried mosix first since we don't want to write to a specific clustering API. we already have a ton of programs and introducing a new API into all of those would be a massive headache. plus a number of our developers dont' know much about parallelisms and the headaches it brings to the table.

with mosix, you just 'fork and forget'. that's simple enough. but here are some of the issues that i've run into and hope it's useful for someone else since it took me about a whole week to figure it out.

Sockets don't migrate. fileHandles dont' migrate. ie. for any I/O to occure, the process is brought back home, then upon completion of the I/O, if appropirate it is transparently sent out. what that means is that if the ratio of i/o to other normal crunching stuff is low, the process is probably going to spend most of it's time at home. what i'm going to do next is test if a process migrates at all by just having an open file handle without doing any I/O. i might post it as a follow-up to this comment. if i have time, i'll also play around with mfs. but probably not. if someone has more info regarding I/O using mfs i would appriciate much. ie. if mfs is used to do file I/O can the process still migrate while doing the I/O? apart from file sharing, what other benefits does mfs bring?

Another thing i've tested is java. as you might already know, native-thread java uses shared memory so that can't be migrated. i was using sun's VM so i turned on the '-classic' switch to use green threads. it migrates. for 5 instances of the JVM, on (3) pentium MMX boxes, i had a 70% (roughly) performance gain (timewise). for about 20 instances of the JVM (i know, that's a waste but more on that later..) i had about 47% gain in time. and i assume it will reduce drastically after that for just spawning that many JVM vs. doing actual work.

to optimize, what i'm going to do is first figure out how many nodes are running. then spawn that many JVMs and split the job between those VMs. and hope mosix sends out each VM to a seperate machine. this is not much different from running a rsh call (or ssh for the security minded folks) except, we don't have to figure out the load on each box we run the command on and we don't have to worry about collecting the output at the end of the run. if someone has a better idea about running java on mosix, i'd love to hear about it. just post your ideas as a reply to this message. i'm sure others will find it useful too. with the mailing list going away and no real documentation for mosix... bummer. also, has anyone tested it with apache? i don't know much about apache but know that it uses shared mem. so can the process forks of apache be migrated? + there's the issue of open sockets from a specific IP. how does that play into this. i've read stuff on web sites that say they're running apache on a mosix cluster. is it one of those 'ahh... i want a beowulf of these' nuts? or is that legit?

FAQ from Website

[lw54]

Before it gets slashdotted by everyone trying to figure out WTF OpenMosix is, here's the FAQ from their website.

Can I mix Mosix and openMosix nodes in the same cluster?

No. Just like the older Mosix, you should not mix nodes because the protocols are subject to un-announced changes from version to version. On top of that, every new version has bug fixes which warrant updating to the new kernels.

Whois the copyright holder of openMosix?

All the old Mosix code is copyright by Prof. Amnon Barak of Hebrew University of Jerusalem. All new code of openMosix is copyright 2002 by Moshe Bar, Tel Aviv.

How do I upgrade to openMosix?

openMosix maintains for now compatibility with the user-land tools of Mosix 1.5.2. I also have a port to openMosix of the user-land tools which will be released soon. To upgrade to openMosix, simply download the openMosix patch from www.openmosix.org and apply the patch with

patch -Np1 < openMosix1.5.2moshe

to a stock Linux kernel of 2.4.14 or 2.4.16 respectively. Make sure to get your old .config file (the .config file remains compatible) and recompile your kernel and modules. Then, reboot.

Is openMosix a fork of Mosix?

Right now, it is a pure fork. Eventually, it will become a separate clustering platform with its own distinct feature set and behaviour.

Good to see.

[GiMP]

I had been using mosix for quite a while until recently. The 2.4.13+ext3+mosix release didn't even work properly with all of my hardware, oddly my networking refused to work at all; other kernels had no problems, a kernel built with the same .config worked fine on another box.

Anyway, I had noticed that 2.4.13 was sticking around awhile.. meanwhile, the page was being updated.. adding some information about a user-space version and the 'disclaimer' on the download page.

Mosix sounds like a good thing, but in reality it isn't very suitable for many of my common tasks. The biggest problem is the lack of support for programs using shared memory. Apache, Mysql, these do not migrate. Also, programs using Xwindows will not migrate well.. as they are network bound and will migrate back to their home node once they need to report the the Xserver. Basically, don't install mosix on 10 of your home machines and expect any kind of performance increase.. besides, you could probably toss out your dual 1ghz PIII, replace it with a 400mhz celeron and not notice a difference.

It would be nice if there was a transparent pseudo layer for things like X.. where all of the machines have their IP for communicating via mosix.. but all 'outside' communication would be made via mosix though a 'public' ip. That public IP would then be used to connect to the Xserver; hence, apps network bound could migrate easily as they would still have the same IP and (spoofed) Mac address. Basically it would be building a NAT router into Mosix for the idea of being able to migrate network-bound applications. It sounds more complicated then it is, but less complicated then it is to impliment.. and I probably don't make any sense, but I know what I mean :) I do of course, assume that source IP is the biggest stumbling block in migrating network-bound applications.

Re:MOSIX question

[Dr.Dubious DDQ]

From my brief experimentation with Mosix and a bit of reading, this sounds correct.

Basically, mosix is a very "chunky" sort of clustering - it works on the level of "whole processes". Because of this, you don't need to write your software to do the splitting and migrating yourself as you do with "less chunky" pvm and mpi. On the other hand, a process split off from a pvm program running can be handled by mosix like any other process and migrated to the cpu that mosix thinks can get the process finished fastest.

Mosix seems like an ideal way to 'lend' processing power to slower machines. This is what I was doing when I played with it previously - I had a K6/2-350 and a P-100 laptop with no L2 cache. I got Mosix set up on them both and used a command-line mp3 encoder as a benchmark. On the P-100, encoding speed was about 15% of realtime. On the K6/2, it was about 110%. Running Mosix between the two over 10Mb Ethernet, I could encode mp3 at about 85% - I suspect it'd have been significantly closer to 100% if I'd had 100Mb Ethernet at the time...

Hopefully OpenMosix will keep up with current kernel versions better. Better still, maybe they'll be able to get it merged into 2.5 at some point...

A variety of things, and your milage may vary

[On Lawn]

We run Mosix, and have had jobs fail. If a node goes down that is running a process two things can (and have) happened. The process dies, or it restarts on another node. What determins this? I have no idea. I can only speak from observation.

It is suggested in the documentation that you have a large swap space on your disk to handle nodes going down. Perhaps with a cached copy of a process it will live on.

In any case, its not scientific but I thought I'd throw that in.