Slashdot Mirror
Linux Needs Resource Management For Complex Workloads
storagedude writes: Resource management and allocation for complex workloads has been a need for some time in open systems, but no one has ever followed through on making open systems look and behave like an IBM mainframe, writes Henry Newman at Enterprise Storage Forum. Throwing more hardware at the problem is a costly solution that won't work forever, he notes.
Newman writes: "With next-generation technology like non-volatile memories and PCIe SSDs, there are going to be more resources in addition to the CPU that need to be scheduled to make sure everything fits in memory and does not overflow. I think the time has come for Linux – and likely other operating systems – to develop a more robust framework that can address the needs of future hardware and meet the requirements for scheduling resources. This framework is not going to be easy to develop, but it is needed by everything from databases and MapReduce to simple web queries."
Newman writes: "With next-generation technology like non-volatile memories and PCIe SSDs, there are going to be more resources in addition to the CPU that need to be scheduled to make sure everything fits in memory and does not overflow. I think the time has come for Linux – and likely other operating systems – to develop a more robust framework that can address the needs of future hardware and meet the requirements for scheduling resources. This framework is not going to be easy to develop, but it is needed by everything from databases and MapReduce to simple web queries."
I know you're afraid of the garbage collector, but it won't bite. I promise.
Yes, it will. It's not common, but it happens - and when it happens, it's nasty. Pretty nasty.
But not so nasty as micromanaging the memory by myself, so I keep licking my wounds and moving on with it.
(but sometimes would be nice to have fine control on it)
Lisias@Earth.SolarSystem.OrionArm.MilkyWay.Local.Virgo.Universe.org
Boobs.
> the first ones used threads, semaphores through python's multiprocessing.Pipe implementation.
I stopped reading when I came across this.
Honestly - why are people trying to do things that need guarantees with python?
The fact you have strict timing guarantees means you should be using a realtime kernel and realtime threads with a dedicated network card and dedicated processes on IRQs for that card.
Take the incoming messages from UDP and post them on a message bus should be step one so that you don't lose them.