Slashdot Mirror
One-Machine Linux Cluster
An AC wrote: Forget Beowulf ? clusters, Jacques Gelinas has made available a kernel patch to enable many virtual servers running on the same machine, even the same kernel. Read his original message posted to the Linux kernel list." Imagine what this will mean for hosting companies...
Honestly, you might be on to something here :)
A virtual Beowulf that is physically a beowulf node itself...Naah, that'd take some clever hacking, a cluster of cluster servers. Probably not possible.
"a cluster of clusters" sounds awfully like an array of arrays which is a multi-dimensional array. Hmm... A multi-dimensional cluster perhaps?
"Nature doesn't care how smart you are. You can still be wrong." - Richard Feynman
If I understand correctly (IIUC? Maybe I can start a new acronym trend here...) anything parallelizable must be custom-written with the Beowulf libraries in mind. I have absolutely no idea how to do this, but the command #include or comes to mind, something I read about it. I've got a handful of older systems sitting around, and gave clustering a thought, but I'm not confidant in my Linux skills to try and compile a custom clustered kernel.
haha.. better read the fine print and make sure you actuallu get your own *computer* including box and powersuppy, and motherboard-- or you may end up sharing your box with 100 other ppl :-P
Isn't this like the BDSs jail() syscall?
The "Lameness Filter" edited out my .h files!
#include "beowulf.h"
or
#include "cluster.h"
Windows 2000 Advanced Server comes with clustering built in via the "cluster" command, you probably don't want that though, because it wouldn't run on the 450 and probably pretty slow on the Duron.
As far as i know... this was supposed to be one of the big wins for the mainframes... i recall some note about 44000 linuxes running together on a single IBM mainframe? sorry dont have the link handy...
A crank is a little thing that makes revolutions
Slashdotted before I could read the whole thing. :( But, as a sysadmin for a smallish web devolopment/hosting company I could REALLY use some separation between certain clients. Sure, this isn't ready for production systems but one day it may be.
The patcher is right...modern CPUs (for my industry) have PLENTY of power. What I hate is having to run some third party app for a client (even in a Linux environment) that *might* affect the whole machine. This patch holds the promise that I won't have as much to worry about.
Yes, this is a good thing.
Here before all but 8486 of you.
Well hey, on a multiprocessor machine it might be interesting. Dedicate a processor for each node and you have four beowulf nodes in one box.
Not only would it be cool for developers to test Beowulf-enabled code, but it would be awesome to have each node independantly accessible from the network.
Ohh god...
type beowulf=packed array[0..255] of system;
type cluster=packed array[0..255,0..255,0..255] of beowulf;
Slightly recursive...
cluster[0[255],0[255],0[255]] = isPrime(bignumhere);
Thats if I remember my multidimensional linked arrays correctly. (and no, it isn't C++ either, sorta Pascal.)
Basically the same idea as Galaxy. Check it out for ideas.... http://www.openvms.compaq.com/availability/galaxy. html
This has just about zero to do with clustering, if anything this is the opposite of clustering. However this IS very very interesting for Web Hosts and just about anyone else that wants to create and maintain multiple environments for developement, test, etc. Image, being able to carve up a mid-range machine like you can an S390 (or other Mainframe class machine Like Sun's E10/15K). So suppose IBM takes this an runs with it. Linux is already ported to RS/6000 and AS/400, now you could get 8 processors of RS/6000 goodness, run production on 4 processors, Test on 2 processors, and Dev on 2 processors.
The devil will be in how you refresh test and dev from production, but that can probably be done inside Logical Volume Manager.
This is very very cool stuff it will be very ineresting to see how it stacks up against the big boys in Virtual machine space.
What if it is just turtles all the way down?
... of clustering. Its... slicing your box up...
I believe this package is very popular with webhosts. One user can totally hose the machine, the rest are not impacted. Trust me, I know.
check out mosix -- it looks like the simplest way to combine the two transparently. I haven't used it, but it looks easy. This would be helpful for some tasks more than others (ie. povray or orher intesive math, but not quake), and of course, things that are multi-process (the same requirement needed for SMP). There was a ./ article about this a while ago.
HIV Crosses Species Barrier... into Muppets
Can anyone tell me how this is different than User Mode Linux?
Opus: the Swiss army knife of audio codec
Someone doesnt understand the reason for a cluster
/. article to hosting companies. This is like the Sandinavian ISP that replaced a bunch of Sun boxen with a much smaller number of larger IBM mainframes. The mainframes run IBM's VM (stands for Virtual Machine), which is roughly analogous to the "root-kernel" dicussed in the linked-to articles, then runs many independent copies of Linux under VM. Each user gets his own copy of Linux to screwup^H^H^H^H^H^H^H do as wishes with. VM 'fools' the copies of Linux into thinking they each have their own machine. Here we have Linux replacing VM, so now a Free product can do some of this (assuming it works, which, being brand new, I wouldn't bet my income on. But I'll bet it will work well soon.), though not all: VM lets you run multiple OS's on the box, which this doesn't. But it does a lot - separate root logins/passwords and process lists for the 'vurtual Linuxen', for example.
I think that was whoever wrote the headline. This doesn't buy you what a cluster buys you, which is more MIPS and RAM working on the same problem. This buys you multiple relatively independent environments on one machine. Hence the reference in the
"that's not encryption - it's a new perl script that I'm working on..." - from some Matrix parody
Hostpro, now Interland has this sort of thing for freeBSD. It used to be called vserver. The new improved version is called Freedom. It's been out for years.
I wonder how this would work with mosix... it could be a dream system!
You could use mosix to combine the compute resources of several boxes to look like one box. And then, you could use this divy up the space so that people don't step on each other. When anyone (working in thier own space) kicks off a large compile, the load would transparently be distributed among all the boxen.
Of course, I have zippy experience with any of this, but it sounds possible.
HIV Crosses Species Barrier... into Muppets
What many people think it means is often something like a parallel file system. which is not the same.
If I recall right, backups can be a pain, but that would vary and depend on the software
"It is a greater offense to steal men's labor, than their clothes"
what would happen if you ran a fork bomb on one of the virtual servers? would it bring the whole physical machine down or just the virtual machine?
I SURVIVED THE GREAT SLASHDOT BLACKOUT OF 2002!
Jail isn't the same as this. If you read the jail manpages, it gives lots of examples how running with a jail involves very interesting problems for some uses. This different technique has different problems for other uses, and does some things nicely that jail does not. And user mode Linux is different, and better for yet other purposes.
Firstly, you're an idiot if you still haven't realized staff comments aren't in italics.
Secondly, Beowulf clusters were only mentioned because they are the complete opposite of the subject matter:
Beowulf clusters bring the computing power of several computers together for a single task, whereas with this a single computer could be used for several isolated tasks.
Some of you should hand over your geek badges, right now.
Two common uses for a cluster:
To answer your question, a web server can use either proxying or round-robin DNS without any special support from the main web server software. Of course, you have to have the proxy / DNS server running correctly. (Also, you obviously have to have either a shared or a synced filestore for your actual web site.)
"How can you claim that you are anti-crack, while still writing a window manager?" — Metacity README
Even if this was to be used to "simulate" a cluster, even that would be fairly useful to some people. If you want to learn to write programs optimized for clustered systems, yet don't have enough capital to be able to access one easily, what can you do?
Well, for one, you can now setup a virtual cluster to test out your apps...
XML is like violence. If it doesn't solve the problem, use more.
IBM has been doing this for some time on their mainframes, where it actually makes sense because of the massive amounts of processing power.
(Goodbye, karma.)
I know this is completely off-topic, but here it is anyway.
Your understanding of the prefix "meta-" is incomplete. In addition to indicating syntactic self-reference (see Hofstader), it can also indicate semantic self-reference (see... well, Hofstader; he talks about this, too, in his discussion of GOD: God Over Djinn).
SGI has a device for connecting crossbar routers together to form large single-system-image computers. It's called a metarouter:a router for routers.
Likewise, a cluster of clusters would be properly called a metacluster. Since "Beowulf" is commonly synonymous with "cluster," the term "meta-beowulf" is pretty much correct, even though it makes me cringe.
And you can do most of the same in Linux, with good old-fashioned chroot plus capabilities. No need for a separate system call.
Restricting to a specific IP address is a nice touch, and one thing Linux capabilities can't do, but it seems rather application-specific. It only allows one IP alias to the jailed process, and doesn't seem to cover any non-IPv4 addressing. And WTF do you have to specify a hostname? The kernel needs this information? (Or does the (2) in jail(2) not actually mean it's a syscall, like it doesn't in AIX?)
I'm also not sure if Linux capabilities are fine-grained enough to keep root from escaping a chroot without totally crippling it - but then again, jail() seems pretty crippling too. A virtual host server really shouldn't need root privs, other than bind-to-low-ports, and Linux capabilities do have that granularity.
"How can you claim that you are anti-crack, while still writing a window manager?" — Metacity README
Think about a system where you want to use IP filter to control what a network host/ports a service (or the hacker that has cracked your service) accesses.
If it addresses many of the issues that normal chroot has, then it may be good.
Isolation of applications against each other.
It's going to be intresting to see how much overhead this has when compared to vmware, usermode linux, or just chroots. (Tried 'em all).
If the overhead of this is not higher than chroot then it will be a big win.
Are you paranoid if you know that they just want to know everything you say and do?
Co-location is kinda pricey, but use linux enabled with domains, and you could split the cost with other people. Then nobody has to bitch about ROOT access. If I ran a small ISP, I could offer linux domains for cheap, virtual servers. If someone messed up a domain, just restore from the nightly backup and they are up and running. Get a couple dual proc boxes, with dual nics (inet/nfs+backup), and a 60 gig (raided?). Make 2 gigs per domain, and NFS mount the /home dirs on some nas. If you could get the backups working, where handling domains are like files, just copy and go, this could be some powerful tool for the busy admin.
-
Sometimes I've believed as many as six impossible things before breakfast. -Lewis Carroll (1832 - 1898)
What!?!? What happens when you bind, say, sshd to port 22 on multiple servers? Do you also need multiple ethernet cards and ip addresses? The docs don't say anything about that...
std::disclaimer<std::legalese> sig=new std::disclaimer; sig->dump(); delete sig;
When we upgrade databases, we assign a dedicated server because we dont want to use the all the cpu horsepower. If we ran a linux with virtual domains, we could upgrade on the same box and not use up all the resources, and the box could stay in production. Thou the article said there is a draw back on a shared file system, allow some kind of snapshots of file systems and you could make a very powerful combination. If you could move a snapshot file system to a domain, you could test, upgrade, whatever. Interesting idea.
BTW, if you have a Sun StarFire with domains plugged into an EMC terabyte storage, you spent millions to do just that!
I am displeased to see so many of you people replying and asking why this would be a good idea.
The point is that this is *l337*. *That* is the point.
Somehow I'm sure I could find a similar phrase in a "famous last words" collection somewhere ;-)
echo '[q]sa[ln0=aln80~Psnlbx]16isb572CCB9AE9DB03273snlbxq' |dc
It seems like this is the same kind of thing this person is talking about...? Is this more general in some way? What exactly is my host using anyway?
Yeah! Let's take single machine, split it into 4 logical machines, and then combine them into 1 logical machine! DOH!
Having these calls available to non-root opens up a can of worms. The system provided looks clean, except he should limit its execution with yet another capability.
Linux can natively be configured to enforce disk quotas and (with more difficulty) manage network bandwidth without any special virtual server software. Also the native unix process scheduling algorithm does reduce the priority of CPU bound tasks. The getrlimit(2) system call can be used to set various limits per process (not per virtual server unless the virtual server runs as one process I guess.) I know of no way to specifically limit disk bandwidth on Linux.
Freeware such as s_context and user mode linux provide no control over how much resources one virtual server gets over another besides disk usage. Other limited resources like CPU, disk and network bandwidth (RAM?) are shared just like they would be shared by separate processes under a single Linux system.
FreeVSD is not a virtual server, but a collection of scripts, binaries and multiple copies of hard-linked read-only filesystems for the common system environment. It is has the best chance for winning the total performance award but has no extra features for resource limits between systems.
True virtual machines. (E.g., vmware) provide very good isolation, but this leads to little sharing of excess unused resources between virtual servers I believe. They also have poorer performance in general because so much emulation is done.
The commercial, proprietary Private Server product from Ensim seems good from the marketing blurbs which say that they have "their own guaranteed share of the servers resources, including CPU, memory and bandwidth". I wonder what the performance penalty for this is and how much does it cost? Can anyone comment?
There is an article (spanish only) commenting this kernel feature here:
d &name=Sections&file=index&req=viewarticle&artid=2.
http://www.hispacluster.org/modules.php?op=modloa
In fact, this article was generated collecting the opinions of many users who post comments about this topic.
I hope it could give you some ideas about the implication of this important feature in the Linux future.
greetings,
lekter
http://www.hispacluster.org
It's the control over it.
Mainframes have insane amounts of control over user processes (a Linux image essentially becomes same), as well as the ability to allocate more resources, fewer, provide fine-grained process accounting, shut down processes, migrate them elsewhere (part of the IBM dataceter Linux concept is the ability to migrate nodes around the country as needed).
What a mainframe doesn't have to offfer is insane amounts of processor power or memory. Disk, and disk I/O are quite another matter -- the amount of aggregate bandwidth a z390 has to offer is impressive.
PC-based virtualization clearly has some advantages, through not all of those offered by a mainframe. A rack of virtualized PCs probably does offer a higher processor density than the equivalent mainframe, however.
What part of "gestalt" don't you understand?
Much respect to this guy. He's taken something thats big, hairy and complex and looked at it from a different direction. Because he's got access to the source he's been able to do something novel with it in what appears to be an efficient and simple way...you couldn't do that with any of the closed source OSes out there today!
/bin /lib etc. from a generic set but users can modify them if they need to - this would allow a sysadmin to keep the default system current while not preventing 'owners' of an individual image from being able to change things if they need to....I vaguely remember something like this for CDs - anyone got the details? Time for a bit of experimentation ;-)
The beauty of this is that there's *one* kernel running so, apart from any overhead of selecting the environment, you pretty much get the same performance as running native. This has got to have 1001 applications.
One of the things I'd personally like to see is some kind of overlaid filesystem so each image by default gets
No, it's a single kernel running compartmentalised. If you introduce a development kernel driver and it crashes, all the virtual machines go down with it.
I wonder if it would be practical to associate absolute CPU time limits or CPU usage percentages with a security context id in order to prevent a certain security context from hogging all CPU ressources.
A similar thing would be desireable for resident set size (real RAM usage) and virtual size (process size) per security context.
Jail SYNOPSIS jail path hostname ip-number command ...
DESCRIPTION
The jail command imprisons a process and all future descendants.
Please see the jail(2) man page for further details. .... ....
FreeBSD 4.4 April 28, 1999
Forget arguing about the definition of a 'cluster'. This is the technology that differentiates between PCs, servers, and mainframes.
IBM and Unisys mainframes (perhaps others, I've worked with these) have hardware partitions where CPUs are divided up. Linux is there now too.
Chroot jails have their problems an annoyances. I've been toying around for a bit with the idea of using User Mode Linux as a security sandbox. This cluster-on-one-system looks even better, and a sibling comment to this one indicates that maybe User Mode isn't a safe jail, anyway.
Not having enough of a home DP Center to dedicate one box for a firewall, I end up running local services (properly configured for local ONLY access in addition to firewalled for local only) on the same machine. I think I've done a good job, but there's always that nagging doubt. Putting my local services in a -safe- virtual OS would give me an additional level of comfort. Chroot jails are ok for standalone things like BIND, but once you have several services interacting like a mail system, it gets a bit messy.
The living have better things to do than to continue hating the dead.
Wow, that's enough karma to make you an official "Kool Kid" (tm)! Your opinions sure count now!
No, moron, it means that I don't have to whore for points. Your mommy should have pointed that out to you when she read you the article to which you replied.
"that's not encryption - it's a new perl script that I'm working on..." - from some Matrix parody
Unix and Linux have always had the chroot() system call. This call was used to trap a process into a sub-directory. After the system-call, the process is led to believe that the sub-directory is now the root directory. This system call can't be reversed. In fact, the only thing a process can do is trap itself further and further in the file-system (calling chroot() again).
And...
The vserver is trapped into a sub-directory of the main server and can't escape. This is done by the standard chroot() system call found on all Unix and Linux boxes.
But, I thought you couldn't (safely) run root processes in a chroot jail, because escape is easy if you can call chroot? Eg, create a subdirectory in your jail and chroot to that (keeping the same current directory), then chroot("../../../../") to get out of jail. Is it really safe to give someone the root password to a vserver in this system?
Could this be used to give every remotely downloaded app a virtual machine, sort of like a java VM? As an advantage to java, with IPv6, you could give every app its own class C network off your 1 billion IP block.
ok then your [sic] infringing on my copyright! Could you as [sic] me next time before STEALING my comments for your own?
Another use that hasn't been mentioned here is testing your failover systems. Now, instead of buying two machines, you can buy one and simulate crashes to test the failover. Very useful stuff.
Note: for most packages there are ways to do this anyway, but they can become a PITA.
-no broken link
Yes, the patch doesn't support hardware dedication. But my SUN background makes me ponder a line of thought.
:)
In Solaris, there are the psr* family of commands for processor administration. psradmin -f 0 will turn off processor 0. As long as this isn't physical powering down of processors, and simply instructions to the scheduler to disregard p0, you could, on the above vm, do something like:
Prod: psradm -f 4,5,6,7
Test: psradm -f 0,1,2,3,6,7
Dev: psradm -f 0,1,2,3,4,5
Leaving procs 0-3 for Prod, 4-5 for Test, and 6-7 for Dev.
Along the same lines, at boot time you can explicitly state memory ranges to the kernel, if linux can't detect your memory right, or you have known bad memory you want to avoid. With the same thought, the Prod, Test, and Dev kernels can be brought up explicitly stating the 0-2G, 2-3G, and 3-4G ranges as usable memory addresses.
You run into more problems when it comes to peripherals in the box, but how many serial ports do you really need? Just specify ttyS0 in the VM with the addresses of ttyS0,1,2 of the physical server.
Am I smoking crack, or should I just stick with my much-more-hardware-flexible Sparc architecture
This space for rent. Call 1-800-STEAK4U
This is much like the jail() of BSD. This does not give any of the benefits of a clustering arrangement. That is, the benefit of having a cluster is that you can distribute process across multiple machines and run from a common storage server. Although this technology is very useful (and can be applied in all sorts of ways- We run Bind in a jail) it does not provide extra process space if only running on one machine.
Having sufficient RAM is the largest factor in commodity grade webhosting services, so having mutlitple instances of a cluster on the same machine does not really make sense, when the whole point of a cluster is to give faster computation and access time.
btw- we offer both of these services here, and we do it on FreeBSD.
Or just imagine - 99% of CPU hogged by an application you cannot kill. Or is there some form of 'master' context that can influence all others?
Well, for those kinds of scenarios we could use a method frequently used by a renowned professional commercial platform. Just hunt down an MCSE, and tell them to fix the problem. I assure you, it will take only a few seconds.
-- Another senseless waste of fine bytes.
Sun has machines entirely built around the concept of virtual hosts. Of course, they stole the idea from the mainframe world, where this has been going on for decades. I don't know of any systems that currently allow splitting a single CPU between domains, but I honestly don't see it as much of a benefit.
Which is definitely not to say that it's a bad or late thing--it's nice to see Linux playing with the Big Boys (tm) now and again. Just don't think that it's ground-breaking technology.
"People who do stupid things with hazardous materials often die." -- Jim Davidson on alt.folklore.urban
During the course of my job, I have to recreate customer environments to duplicate their problems. Often these environments involve firewalls, NAT, or simply multiple subnets that are difficult and time-consuming to get past our IT guys (and even once we convince them of the need, it may be a week before any real progress is made on their end to set things up).
However, using this, I can hopefully get a single box set up with several "systems" using internal virtual networks that will allow me to have something like...
1: Server / Client Gateway
2: Client
3: Firewall / NAT / Router
4: Client Gateway
5: Client
With 1 and 2 being "inside" the firewall and 4 and 5 being "outside".
This would allow me to eliminate TONS of bureaucratic red-tape paperwork BS, and give a self-contained environment for some of my other coworkers to educate themselves on (95% of which have never done any actual hands-on work with a firewall or even done any routing).
While all this could be easily accomplished using a multi-processor Sun box and their domain / partitioning scheme, those tend to be a little more pricey than the dual-P2 I've already got sitting idle in the corner of my office...
Any comments on the feasibility of this scenario?
Would this package work for what I'd like to do?
Thanks...
-l
How does this compare to both
umlinux (I suspect this is not what's going on)
and
freevsd (Check it out)
http://www.freevsd.org
Some extremely large, well known hosting companies have trouble providing "reasonable" (trustworthy, timely, competent) support to corporate website virtual hosting clients, and I have repeatedly seen all hell break loose in the case of deadlines to push staging to live server from many time zones away.. for some reason I still don't understand, we were never allowed to touch the live server so we never even knew its directoy contents. Sheer hell. In this situation you seldom know if it's going to work on the live server (which is *not* the same as staging no matter what was promised) until D-Day.
I was even offered root access once to fix this provider's host but I had to refuse due to responsibility for all the other clients with virtual websites. The problems generally do not come from things that would crash a kernel, but from the economics of getting individuals to apply appropriate knowledge in the right place at the right time, within the context of a number of companies working together with their own agendas.
Using this virtual server patch, I can see a *lot* of time, effort, danger, stress, complaints, etc. all swept away into history!
Run a virtual servers on your local dev box and the remote staging and live machines, then use rsync from local to staging. Ssh into staging and rsync again, or have the admin staff do so. But the live machine can be isolated network-wise, so it is more likely that the user could be allowed onto the live server themselves.
Of course, 1) it isn't a real compartment with your own filesystem and everything, so for example installing Perl modules, tweaking boot scripts, or tweaking security ain't happening. 2) it isn't strong compartmentalization. And 3) you have to be running a linux kernel in the first place. And 4) the above problems tend to involve Suns, not linux boxes.
Seems this kind of idea is killer though. It would be really interesting if we could run something like this or (maybe better yet) user-mode-linux efficiently as a process inside a SunOS environment. Sounds like IBM had the right idea running a bunch of complete OS images on their heavy iron.