Implementing Better Task Scheduling for Servers?

trifakir asks: "We are running some quite expensive SunFire servers with Solaris 8. In the 'crontabs' of these hosts we have scheduled maybe some hundred odd jobs, which are constrained by multiple factors: dependencies on other jobs, time constrains, CPU and memory usage, network bandwidth, and so on. Obviously this imposes a CSP. On the other hand the number of these jobs, each one of them can take from minutes to hours is growing and we are now experiencing performance problems given the limited resources we have. Of course we have opened the bag-of-tricks with our best *ad-hoc* solutions, using mostly Open Source software, to turn our system into an event-based and less dependent on the scheduling expertise of the admins. At certain point we were considering using AutoSys and I was looking for a grid-like scheduler like OpenPBS, both of which were discarded for various reasons. I am curious, how you guys, would solve this problem, which seems very trivial for many environments. Both advice about theory (scheduling) and practice will help us and any other readers who may be tackling this difficult problem."

Comments

MAKE, or PMAKE

[fdragon]

I am not sure on the details as I have not done this myself.

But in your situation I would be creating a make file to schedule the jobs with. Make can handle concurrency and with available patches can be made to distribute jobs to multiple nodes. Parallel Make Patches for GNU Make.

In a method like this I would recomend a small shared file system so that as you complete each job you can touch a file. This would allow you to continue from the point you left off, or if you wish, clear it out and start over.

How I solved similar prob: make + daemontools

[Anonymous Coward]

First of all I don't know if you actually need to run certain tasks at certain specific times. If so, you will need to use cron after all. But here's some ideas:

I had a app server that ran a number of critical tasks, and they had a somewhat arbitrary and complex dependency graph. A good analogy would be eBay's indexing cycle: a bunch of stuff has to happen as often as possible, but it's not really important if it takes 30 minutes or 35 minutes or 1 hour to do each cycle.

Also it needed to be easy to extend: a programmer should be able to write the code and "stick it somewhere" to extend the system.

The previous admin had set some tasks to run in cron every 5 minutes, not realizing that after a while some of the jobs actually took 6-7 minutes and growing (you can imagine what the process table looked like after a while, and some of these were not locking resources properly.....)

I came up with a system using Makefiles (takes care of interdependencies and the -j flag will run indepent processes in parallel) and djb's daemontools package.

If you're not familar with daemontools, it is an incredibly tight little set of tools that lets you *atomically* and *reliably* start, stop, and configure daemons, and it lets you turn ANY script into a daemon. It just runs a "run" script you supply, and when the script dies for any reason, it restarts it. So you can create a script like this:

#!/bin/sh
make -C /foo/bar/baz update
sleep 60

and it will run it over and over again. Combine this with resource limits and multilog logging and you have a bulletproof way to keep things going in the background.

So I set up the dependencies in the Makefile, threw in a couple scripts to run all the scripts in a couple "drop box" directories that programmers can use, and documented everything and made a web interface for checking the results. Now it doesn't really matter if the cycle takes 5 minutes or 10 minutes or 30 minutes, the makefiles are run over and over again in a loop, keeping things up to date.

Again, I don't know the specifics of your needs but this is definitely something to consider. Especially if your crontab has grown into a huge confusing mess, and you don't actually care what exact time things are running.

Topological sorting

[Sesse]

Almost sounds like a candidate for simply topologically sorting the entire dependency graph and then processing the tasks in order. You'll eventually end up with several non-connecting subgraphs which can be parallelized; the big remaining problem seems to be prioritizing the tasks if you don't want them to be run equally often, but I'm not really sure if I've understood your problem correctly. :-)

/* Steinar */

spot on!

[Triumph The Insult C]

we use make and lp to handle dependencies, queueing to other machines, etc. not too difficult to setup if you know what you're doing