Open Source Batch Management?

Asgard asks: "My employer is currently running a commercial batch management platform. Unfortunately the licensing model makes it unfeasible to run it in the development / testing environments, leading to poor usage of the tool and unexpected failures in production. I'm looking for an equivalent Open Source tool and am wondering how others have approached the problem. Does Slashdot have any suggestions?" Imagine a system like cron, but with job dependencies. Are there any batch systems out there like this? "The tools I've found through web searches mostly treat 'batch management' from the cluster perspective -- a user submits an ad-hoc job and the tool figures out where and when to run it based on load and architecture requirements. Instead I am looking for something that manages daily schedules of jobs based on their dependencies with other jobs and external events, such as files arriving or time.

An example might be that every day jobs a, b, c, and d must run. Job a must not run before 9pm and requires file X to be present. Jobs b and c depend on a completing successfully. Job d must run after 2am and after b and c have completed successfully. If job c fails then an operator must fix the issue and rerun it, after which the tool will move on to job d. "

Comments

Systems like this are handy

[blincoln]

I don't know of any OSS systems like this, but they are *very* useful for larger companies.

A few years ago I was working in change control, and updates to software stored on network shares across the company were handled using a decrepit old VB app that generated linear xcopy scripts that updated each server (of which there were about 160 spread across the US) one by one. Most of the servers were on slow links, so distributing a 10MB file could take twelve hours or more.

I hadn't learned to code properly at that time, but we used an enterprise batch scheduler called Control-M* that worked like the original post describes. What I did was wrote a batch script that read a config file and then executed a single robocopy command targeted at the server in the Control-M job definition.

I had a whole array of these jobs, one for every target server, and they all depended on another job that would run at - for example - 11PM. So when that time rolled around, all of the dependent jobs could run. As-is, that would have overloaded the WAN and source server bandwidth. So I assigned what Control-M called a "resource" to all of the jobs. It was just an integer counter that I capped at 16. So at any given time, there were 16 "threads" of robocopy running. It ended up being between 20 and 30 times more efficient than the crappy xcopy scripts.

Anyway, they're really handy, and if there isn't an OSS project like this, it would be a great idea.

* This is not an endorsement of Control-M. In my new(er) job, I'm working as an engineer, and I discovered that the encryption system that it uses for storing account passwords in the registry is so poor that I was able to write a universal decoder for it using only vbscript and Excel. There are certainly other downsides to the app as well, although one cool thing is it runs on just about any platform - Unix, AS/400, OS/390, Windows, etc.

a system like cron, but with job dependencies

[hankaholic]

Ummmm... cron+make?

Build systems aren't just for running compilers. :)

Use Gridengine

[Anonymous Coward]

http://gridengine.sunsource.net/

It handles batch jobs, dependancies etc etc.