Slashdot Mirror

If you want a monitor that can display useful information about thousands of nodes on a single display try clumon. We use it for our 1000+ node clusters. The software was developed in-house but is available under the University of Illinois/NCSA Open Source License Copyright (noticeware). If you're just going to use this in-house, the license shouldn't be an issue.

You can see a sample clumon display of a working cluster at NCSA Linux Cluster Monitor.The clumon page for that cluster shows you each the job status of each individual node (if the node is colored, it has a job assigned), the load on the machine (the height of the line is proportional to the load, and red tips show loads over 1.0 per cpu) and the service status (green underline is ready, yellow/black stripes is offline, and red is unexpected offline/no comms). If you mouse-over a node, a status box pops up with more information on that specific node.

As this was designed for a cluster with the Torque resource manager, it won't be exactly what you need, but since you are willing to write a monitor from scratch, it might be a really useful starting point. Design-wise, this monitor allows the engineer or manager to see what's going on in general, with problem areas being immediately obvious, and without being overly cluttered.

The open source Performance Co-Pilot software runs on each node to collect information, which is polled by the central server. Back end is MySQL. The dynamic display is PHP.

Straightforward, useful and very configurable.

Funnily enough this is a project I am working on right now.

I'm coming at it from an HPC (high performance computing) perspective. We'll have a cluster in-house supporting the base load and overflow to a utility computing provider.

Job scheduling software (currently torque but also trialing slurm) is used and once the total load has passed a threshold more remote compute VMs are fired up.

We should have it in production by - /me checks gantt chart - last month.

It seems like an idea whose time has come.

Most Beowulf clusters run parallel codes written to use the Message_Passing_Interface (MPI). MPI programs really don't want to be migrated to different nodes while they're running, so load management is achived through schedulers such as Grid Engine, TORQUE, and others. These schedulers avoid the need for process migration by preallocating the resources (compute nodes) in advance, and prevent the load imbalance from happening in the first place. openMosix waits for the imbalance to slow down the computation before it migrates a process to relieve the problem.

If you check the archives of the Beowulf mailing list, you'll see that while the Beowulf community knows about openMosix, very few Beowulfers use it.

And, computer related, it's also the name of some cluster management software made by Cluster Resources of Utah.

Yes it can manage across a cluster, but I'm pretty sure you can run it with a single (local) node in the cluster and all jobs with run locally. I haven't tried that, but in theory...

Have a look at the msub command

The syntax is rather weird, but it seems to have everything you wanted. Except I can't quite work out whether it does dependency management. It seems like it should, but I haven't worked out how.

As for Quartz, you didn't actually specify what language you wanted, or how much programming you were willing to do.
Java can call native processes. Quartz has builtin (but poorly documented) support for calling out to the OS to run commands.
It has the foundations to do everything you asked for. Maybe java isn't an option for you, but if you dig a bit, it probably will work.

I am aware that you're after a batch scheduluer, but the thing is, once you get past the cron style schedulers, the next level up tend to have remote job management (etc) so don't let the fact that Torque markets itself as a cluster manager put you off too much. (of course if it really doesn't handle dependencies, then that's a good reason to ditch it)

What you're really asking for is something like Control-M from BMC software. But it's not open source, and it's not cheap.

• http://www.opensymphony.com/quartz/
• http://www.clusterresources.com/products/torque/
• http://www.OpenPBS.org/
• http://saturn.sourceforge.net/

TORQUE (Tera-scale Open-source Resource and QUEue manager) is a resource manager providing control over batch jobs and distributed compute nodes. It is a community effort based on the original *PBS project and has incorporated significant advances in the areas of scalability, fault tolerance, and feature extensions contributed by NCSA, OSC, USC, the U.S. Dept of Energy, Sandia, PNNL, U of Buffalo, TeraGrid, and many other leading edge HPC organizations.