Open Source Batch Management?

Asgard asks: "My employer is currently running a commercial batch management platform. Unfortunately the licensing model makes it unfeasible to run it in the development / testing environments, leading to poor usage of the tool and unexpected failures in production. I'm looking for an equivalent Open Source tool and am wondering how others have approached the problem. Does Slashdot have any suggestions?" Imagine a system like cron, but with job dependencies. Are there any batch systems out there like this? "The tools I've found through web searches mostly treat 'batch management' from the cluster perspective -- a user submits an ad-hoc job and the tool figures out where and when to run it based on load and architecture requirements. Instead I am looking for something that manages daily schedules of jobs based on their dependencies with other jobs and external events, such as files arriving or time.

An example might be that every day jobs a, b, c, and d must run. Job a must not run before 9pm and requires file X to be present. Jobs b and c depend on a completing successfully. Job d must run after 2am and after b and c have completed successfully. If job c fails then an operator must fix the issue and rerun it, after which the tool will move on to job d. "

Comments

Please, some manners...

[Anonymous Coward]

Please don't join conversations in which you have no interest.

cron + make + caffeine

[MarkusQ]

It works great for me. Just have to do a caffeine check before making major changes (and remember to stop the cron job plus test in a sandbox).

Some handy tips:

• Use pid files to keep new instances from starting up if a job goes long.
• "-j" can be your friend, but (like a real human friend) it can also get you into a heap of trouble if you aren't careful.
• Running the make in a permanent loop and just touching things with cron can be a handy trick, especially if you need to let users (or external processes) tweek the process.

--MarkusQ

TORQUE Resource Manager

[Bryan_Casto]

I think TORQUE Resource Manager will do what you're looking for. From their page:

TORQUE (Tera-scale Open-source Resource and QUEue manager) is a resource manager providing control over batch jobs and distributed compute nodes. It is a community effort based on the original *PBS project and has incorporated significant advances in the areas of scalability, fault tolerance, and feature extensions contributed by NCSA, OSC, USC, the U.S. Dept of Energy, Sandia, PNNL, U of Buffalo, TeraGrid, and many other leading edge HPC organizations.

Suggestions

[RyanGWU82]

I'm working with systems like this right now. You might have better luck if you search for "workflow" instead of "batch." Googling for "open source" workflow management also brings back a bunch of promising hits. And if you're Java-centric, there's a great page which summarizes all the open source workflow engines available for Java.

OSS Systems like this are handy

[Roadkills-R-Us]

We use PBS at work. I didn't pick it, but it works. There are other around, as well, though I don't recall their names off the top of my head. (PBS is avaoilable free, or supported, for a fee. We use the latter-- a commercial version of an OSS project. 8^/

A search of google or any of the OSS sites should turn up several more.

For example:

[Ayanami Rei]

in cron.daily...
make -j $NCPUs -C /working/dir /working/dir/Makefile -

all: tasks/1 tasks/2 tasks/3

tasks/1:
foo bar baz
frob fritz
touch tasks/1

tasks/2: tasks/2.1 tasks/2.2 some_make_test(tasks/2.3)
bar baz qyzzy and touch tasks/2

etc. etc. etc.

Clarifications

[Ayanami Rei]

1) one thing make can't do is run tests that generate dependancies at runtime... it does it in one pass at the beginning. Since you're running it iteratively this isn't a big deal.

2) For a batch automation system, you'll need to use make -k, and if you need to, put targets in .DELETE_ON_ERROR if you don't do something like manually touching a status file at the end of a command.

3) If you have a dependancy chain of targets and you don't want to have to clean up explicitly (or you want your job to run entirely in phases), you can label intermediate targets with .INTERMEDIATE, and if make finishes processing these things in one invocation, it wil delete the outputs/status files when all the dependant jobs are run. If it doesn't make it, then it will be forced to restart from the preconditions.

4) Make sure to fully outline dependancies. If you need to somehow prevent two things from running in parallel, you need to create an artificial barrier with the script itself unfortunately. The easiest way to do this would be perl and IPC::SysV, I should think. You might know of some other shell tricks or opening a device that blocks like a FIFO... but it sucks that gnu make doesn't have it. (However HP-UX and SCO's make have a .MUTEX pseudo-target that prevents two things from being run in parallel... shame)