Slashdot Mirror
Linux Needs Resource Management For Complex Workloads
storagedude writes: Resource management and allocation for complex workloads has been a need for some time in open systems, but no one has ever followed through on making open systems look and behave like an IBM mainframe, writes Henry Newman at Enterprise Storage Forum. Throwing more hardware at the problem is a costly solution that won't work forever, he notes.
Newman writes: "With next-generation technology like non-volatile memories and PCIe SSDs, there are going to be more resources in addition to the CPU that need to be scheduled to make sure everything fits in memory and does not overflow. I think the time has come for Linux – and likely other operating systems – to develop a more robust framework that can address the needs of future hardware and meet the requirements for scheduling resources. This framework is not going to be easy to develop, but it is needed by everything from databases and MapReduce to simple web queries."
Newman writes: "With next-generation technology like non-volatile memories and PCIe SSDs, there are going to be more resources in addition to the CPU that need to be scheduled to make sure everything fits in memory and does not overflow. I think the time has come for Linux – and likely other operating systems – to develop a more robust framework that can address the needs of future hardware and meet the requirements for scheduling resources. This framework is not going to be easy to develop, but it is needed by everything from databases and MapReduce to simple web queries."
Yes and no.
No, large (Linux using) companies like Google, Facebook, Twitter have always used some kind of Linux container solution, not virtualization.
Yes, policy is controlled by the cluster manager.
But for example Google uses nested CGroups for implemeting those policies for controlling resources/priorities on their hosts.
Virtualization is very ineffcient and Docker/Linux containers are a perfect example of how peole are starting to see that again:
https://www.youtube.com/watch?... / https://www.youtube.com/watch?...
Suppposedly, CPU utilization on AWS is very low, maybe even only 7%:
http://huanliu.wordpress.com/2...
The reason for that is, is that VMs get allocated resources they never end up using. Because the host kernel/hypervisor doesn't know what the VM (kernel) is going to do/need.
For their own services Google doesn't use VMs, but Google does offer VMs to customers and to control the resources used by VM they run the VM inside a container.
Here are some talks Google did at DockerCon that mentions some of the details of how they work:
https://www.youtube.com/watch?...
https://www.youtube.com/watch?...
New things are always on the horizon