Slashdot Mirror
Implementing Better Task Scheduling for Servers?
trifakir asks: "We are running some quite expensive SunFire servers with Solaris 8. In the 'crontabs' of these hosts we have scheduled maybe some hundred odd jobs, which are constrained by multiple factors: dependencies on other jobs, time constrains, CPU and memory usage, network bandwidth, and so on. Obviously this imposes a CSP. On the other hand the number of these jobs, each one of them can take from minutes to hours is growing and we are now experiencing performance problems given the limited resources we have. Of course we have opened the bag-of-tricks with our best *ad-hoc* solutions, using mostly Open Source software, to turn our system into an event-based and less dependent on the scheduling expertise of the admins. At certain point we were considering using AutoSys and I was looking for a grid-like scheduler like OpenPBS, both of which were discarded for various reasons. I am curious, how you guys, would solve this problem, which seems very trivial for many environments. Both advice about theory (scheduling) and practice will help us and any other readers who may be tackling this difficult problem."
move all of your cronjobs to at scripts.
In the beginning of each script, you have conditional checks as to what conditions need to be in place for the job to run, if they are met, then do what you have to do, if they don't, reschedule it for 5 minutes from now, and write it to a log. If the next run doesn't work, reschedule for 10 minutes, then 15 until a max is reached, when it dies and sends and email to the admin. Pretty basic, but with a little work it would work fine.
RandomAndInteresting.comdefending the world from stupidity since 1979
Have you come across the ARMTech resource management product from Aurema?
Instead of scheduling when jobs run to take care of CPU/memory constraints, you can set a policy of resource usage, which ARMTech then enforces (eg Application A can only have X amount of memory at maximum). Policy can be adjusted whenever the need arises. It might help in solving your problem.
Disclaimer: I am an employee of the above company.
As much as I love to support Open Source, there's times when it's just not worth the pain. If you're already spending tens (if not hundreds) of thousands of dollars for Sun hardware (nothing wrong with that, we do it here) then pony up the cash for a REAL Enterprise Scheduler. Look at Tivoli. Look at Tidal. Look at Cybermation. They all have different strengths, so depending on what your organization needs, YMMV.
We standardized on Tidal. Yep, it's expensive. Our IT budget for this year is in the $1M range, so it's time to step up into the big leagues.