Slashdot Mirror
Managing Linux and Virtual Machines?
deijmaster asks: "For a couple of months we have been hearing (as a major consulting firm) IBM people pushing the possibility of installing a Z/Linux VM setup at one of our biggest clients (financial). To a Linux user such as myself this sounds great, at first. Now, I am a bit reluctant when it comes to managing this kind of infrastructure, with little or no local expertise at IBM. Has anyone gone through a Z/Linux VM corporate installation and lived through the management of such a solution?"
I once saw Amdhals version of Unix running on a mainframe at New England Telephone. The ps command yielded about 20,000 running processes and the guy I knew told me that it was just one of 6 VM systems running on the same hardware.
I was impressed.
SG
Oh, and the Marist linux-390 listserver is well worth subscribing to.
Well I used to work at similar financial company where IBM was pushing something similar as well. What it boiled down to was the following issues.
1. for the equivalent # of VM's it was more cost effective to buy new Intel hardware. The annual maintaince cost for the IBM more than paid for all new hardware.
2. Software availability. The only thing you could run it would be home grown apps or existing opensource apps. No commercial software was available. This company was an all Oracle shop, no DB2. They're primary opensystems backup solution was Netbackup. Which at the time had no client for linux on Z. (a year ago).
3. In house expertise. They had no linux expertise and very little Unix (solaris & HP) (jr admins at best) expertise. Let alone running linux on a Z.
So to sum it up. It's a very expensive, somewhat propritary and inflexiable environment. If you have a specialized use for it and can justify the cost go for it. Otherwise stick with commodity Intel/AMD hardware. It'll be cheaper and easier in the long run.
I haven't worked with a Z/Linux VM before. However, I have used User Mode Linux to create a dozen or so virtual servers per host server. And I'd imagine that the benefits offered by UML would also apply to Z/Linux VMs.
For example, with UML you're able to get much better resource utilisation. e.g. most of the time the machine is idle. When one of the UML servers need the host server's resources, they're there (CPU, network, disk IO, etc). That means you can have multiple UML servers bursting up to the performance potential of the host server. Certainly a better resource utilisation than having several host servers running mostly idle.
Another benefit of virtual machines are their logical separation from the host server. Each virtual server has their own users (including root), applications, file systems, IP address, etc. That means that if security is compromised on one, the others are unaffected. Ditto resources can be allocated to each virtual server according to need. And any mis-configuration on one doesn't affect the other. This compares to running multiple applications on the same server for different purposes (e.g. running HR and Account systems on one server, if email goes down them both systems are affected. In a virtual server setup, only one of the other would be affected.
So... Thumbs up to server virtualization software in general. Particular kudos to UML. And good luck finding out about Z/Linux!
- P
RimuHosting.com - Linux VPS Hosting
I work for a big financial firm in NYC that is using Z/Linux pretty heavily. I have to say that while we are very happy with the results, it is VERY important to have VM people on staff who are also Linux savvy. IBM has been great in getting us set up, but they don't live with the systems. We do. You'll need to be very careful about what you're using the Linux instances for, and take alook at how they'll use hardware resources, like the OSA cards.
With careful planning, and the expectation that it will be a bumpy start, you'' find that it's a very rewarding experience, both personally and professionally.
I wouldn't get hung up on the whole "local" thing. You just have to understand how IBM works. There's no concept of "local" at IBM. At any one point in time, 50% of IBM employees aren't in a traditional work place.
If you have problems, contact IBM and they will get their best people on it. IBM is all about customer service. You never get fired for buying IBM. From an engineers perspective, it's a pita. The best people in a department end up spending most of their time working on customer problems.
Hell, IBM still supports OS/2. If a Z-Series seems to solve your problem, go for it. IBM will take care of you.
http://www.linuxcare.com/
Chrisd
Co-Editor, Open Sources
Open Source Program Manager, Google, Inc.
Actually VM itself is fairly straight forward to administer
The biggest hurdle will be:
This is also where security comes in. Each SVM is really isolated from each other.
One thing to remember too is that VM was (and still is) used by many Universities and colleges -- not as much as it was back in the 70's and 80's, but it still has a presence.
Anyway... just some comments from an old timer VM sysprog
I sold and installed the very first Linux application on the S/390 --a Multiprise running VM and it worked great. We used the TurboLinux port and then finally wound up with SuSE.
We compiled the source code and it ran just like it did on a big Intel box. IBM helped with hardware issues which related to load balancing amongst the VM instances. One of their business partners supported the customer, Winnebago Industries with regard to Linux and OS 390.
IBM wasn't much of a factor as far as needing support. They supported the mainframe, the OS and VM just fine. SuSE installed without a single issue.
Some other issues arose in getting the user to learn IBM mainframe lingo, such as IPL instead of boot, and DASD. But, that didn't require much effort. The IBM Redbook on running Linux on the S/390 was all we needed to transfer knowledge. We downloaded it for free in pdf format.
The main benefit I discovered was the ability to consolidate servers. We replaced a bunch of M$ Exchange servers and ran a suite of Open Source apps such as Cyrus IMAP, Open LDAP, Exim, Apache, etc. We were able to get rid of a bunch of distributed servers and put them on one instance.
I suggest that IBM can help, but I don't think you'll be dependent on them. They're very expensive. With Linux on the zSeries or S/390 you can do everything yourself. -- That might not be what IBM wanted, but then they championed Linux, didn't they!
Familiarity with Linux will not help you setting up the zLinux environment. It works like this: You dedicate a few processors of your mainframe to Linux. These processors will run VM, which has:
The users are defined in a "user directory". There, you can specify how much memory, disk and CPU share you want to give to each user. These users, remember, are in fact virtual machines that will boot an image of Linux compiled for the zSeries processor architecture.
If you want to create and take down Linux images frequenlty, you'll have to install and customize some VM scripts that will do the job for you. When the scripts are installed, you can setup a new Linux image (complete with its own disks, IP address, etc.) with a single operator command.
Most sysadmins of a zLinux machine spend a lot of time in VM. So learning VM is essential if you are going to do this job. VM was created 30 years ago and is somewhat primitive in places, but the resource virtualization mechanism is incredibly powerful and makes up for it.
Finally, make sure that people understand that there might be dozens of virtual CPUs defined under VM but only a few real CPUs. If you have 4 CPUs, a Linux user with an absolute CPU share of 25% will have the equivalent of one CPU. If the Linux image is used for recompiling its kernel, it might be a tad slow. The mainframe has great I/O performance but only run-of-the-mill raw CPU speed.
Good luck.
Wintel hardware is crap and not at all scalable. It's like comparing a ferrari (z hardware) to a pinto (wintel) and saying "well, they're both cars". Sure the ferrari costs more, but it's a hell of a lot more likely to be able to win in a race.
Reasoning by analogy is always fraught with pitfalls.
The Ferrari can't carry more than two people. The IBM machine is designed for fast I/O. The Ferrari breaks down a lot. The IBM is designed to be highly reliable.
Perhaps a better, but still rather imperfect analogy would be to a tractor trailer--lots of horsepower, but not a speed daemon. Lots of cargo space. A decent diesel engine that can stand up to abuse.
IBM thinks that if you replace 20-30 Intel CPUs , all running at 5% utilization, with a single zSeries CPU running at 85-90% utilization, you'll save money and aggravation. On the other hand, if those 20-30 Intel CPUs are rendering CGI for a film, or modeling a jet engine (and thus running near 100% load), a zSeries CPU would only be able to take on the work of 4-5 Intel CPUs, if that.
I'm an IBM'er currently on assignment at the world's largest insurance company. I was brought in because they wanted to consolidate servers to a mostly-Linux solution. After piloting Samba 2 beta on zLinux last summer, they balked at the heavy reliance on Z.
The key is for people to realize that the type of workload is critical when deciding to try zLinux, and any barking about Athlon vs. G6 is useless. Also, vendors need to realize that once you compile an app on Linux on any one platform, you're usually a recompile away from running it under Linux on any other platform. Hence my reasoning that any complaints about software availability from a year ago is also useless. More apps are being ported to zLinux everyday.
Linux on Z has a role, it just needs to be explored by more brave souls. Besides, I've always said that if I leave the company, I'd like to create an "ISP in a box" using a z800 and some ESS disk to host a few thousand virtual web servers. I implore people to please visit Linux@IBM for more information.
Intelligent Life on Earth
Ahh yes, grasshopper, but when that one uber-box dies(hard disk, fan, power supply, whatever), gets powered off by accident, network cable unplugged, yadda yadda- it affects ALL the virtual machines.
Granted in the Big Iron, you've got lovely hot-swap capabilities and such(processors, memory, etc)...but nothing is foolproof or 100% reliable. It's the old joke with pilots about twin-engine airplanes; the door swings both ways and there's no such thing as a free lunch. On one hand, you've got a spare engine if one dies, but you're 2x as likely to have a failure, you've got a lot of added complexity, and sometimes it still won't save your bacon(twin engine planes have an abysmal survival rate for engine failure in part because of the really shitty way they fly with one engine down). This is VERY applicable- because managing this big IBM server is much more complex(the whole point of this article) than seperate hardware.
Best example I can think of in how hot-swap can still not save the bacon is with the Cisco PIX 5-something(The 1U pizza-box one). It has FULL failover- if you've got two, and one shits the bed COMPLETELY, the other one takes over absolutely everything, including active connections; they share ALL state information for what's called stateful failover. Aside from a momentary blip where things stop for a sec...nobody's the wiser that a piece of very expensive hardware just let the Magic Smoke out. The problem is that the PIX OS version we had was buggy and would crash randomly- and because they were sharing connection tables and everything, they'd BOTH die, which was REALLY bad since the boxes didn't have hardware watchdogs(!). We turned off fully-stateful failover, and the problem went away; we'd notice they'd ping-ponged(there's an 'ACTIVE' led to show you which is live) and we'd power-cycle the other.
So ask the tough questions; instead of asking what's N+1, ask what's NOT N+1, and do a very careful breakdown of what exactly it will cost to run this big huge box, and figure out what the 'per [virtual] machine' costs are...
Please help metamoderate.
We've got a production linux instance running under VM alongside our production VSE system. Since the box is fairly underpowered we get a minimal slice of the CPU. This makes the system respond like a 286 with the 'turbo' button turned off.
When the VSE instance bombs out for some reason, and we get effectively 100% of the CPU it responds like a pentium... maybe. Think P166.
Unfortunately in our circumstance we can't 'turn on' more MIPS because then our VSE instance is running on a 'bigger' machine and we end up doubling our licesing costs. Other alternative is to turn on the ILF (integrated linux facility) which dedicates 120Mips to linux only, without affecting other licesning, but that costs $150k. You can buy a lot of 2-way or 4-way pentium boxes with decent RAID arrays and get much better performance for that kind of money.
So if your shop is run by some sort of morons and you've got 100's of spare MIPS to burn, then Linux on the mainframe probably makes some sense. Otherwise, just get some intel boxes. Any savings the mainframe provides in terms of power, cooling, and ligher administration is going to be offset by massive complexity, poor performance, and a lack of easy support for a bizarre platform that few developers have access to.
"But actually trying to use m4 as a general-purpose langage would be deeply perverse" --ESR
IBM mainframe complexes basically never go down. There are installations that have been running 24x7x365 for decades. That's the whole point of owning one.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
Grain of salt, yada yada...
d bo okAbstracts/sg246824.html?Open
:(
I second the idea that it is very important to have VM skills on site for a customer looking at this. Presumably the customer is already a z/Series account, so they probably already know a thing or two, but they may have bought into the "VM is going away" speech and gotten rid of their VM stuff years ago and gone to z/OS.
Even if they have VM skills from 5-7 years ago - that will still do. VM hasn't changed all that much, it just has some more bells and whistles. So one or two refresher courses for whoever is still around in their shop will get them up to speed on z/VM 4.4 if they knew it 'Back in the day'.
And yes - Linux on VM is still young. Most shops appear to do a lot of 'roll your own' solutions to the administrative problems. Get hooked into the Marist linux390 mailing list, there are a lot of smart folks there who have at least thought about any problem youre likely to have.
I've run/tested every one of SLES 7, 8, RedHat 7.1, 7.2, RHEL3 beta, TurboLinux (old and crusty now) and Debian with pretty much any IBM middleware you could think of. From the linux side - it doesn't know anything about VM, or care. So you as the administrator must make sure it plays politely with the others it lives with. You probably should not just throw 2 Gigabytes of storage at it just because Websphere says it needs it. Running Linux with VM does require some understanding of how to make the most of shared resources. Check out this redbook:
http://publib-b.boulder.ibm.com/Redbooks.nsf/Re
It makes a lot of these points better than I can.
--Anonymous Coward cause I forgot my password
They're a heck of lot less wasteful (electrickery into heat) than they used to be, and require a lot less space (again, compared to the past).
Clusters ... I don't know where you get the 'faster and cheaper' line, unless you're talking about applications specifically designed for clusters. When you start writing apps designed for a few thousand simultaneous users, the benefits of the mainframe become apparent. Stability. Speed. The ability to hold gobs of info in ram. Which, BTW, makes them the nearly ideal web server. Security (hey, it's not M$!). Mainframes are a mature technology ... meaning lots of the annoying things (both hardware and software) still plaguing the small boxes have been fixed. (Admittedly, 'mature' often translates into 'f$cking obsolete pos' (i.e. panvalet).)
I don't worry about backups conflicting with apps on the mainframe. I don't worry about the details of storing things reduntantly (although that's quickly getting solved on the smaller boxes). For those things written on WinWhatever, the programmers need to worry about every little upgrade/patch from M$.
Now, most places still give mainframes a room of their own ... and it tends to be a bigger room than servers get. And, if you're happy with something a little slower and little less reliable, a good farm runs less than a mainframe.
But, to put things in perspective, one of my databases (non-mainframe) is moving to a USD 2.1 million machine. That's a fraction ... as in, from 1/4 to 1/20 (depending on options) of a mainframe.
I'm working in both worlds. I like the cost benefits of the smaller boxes. But it still freaks me out when users punch in a query and it takes several seconds (to minutes) for a response, when the delay on the mainframe is done by the time the enter key pops up.
Actually not quite true (that the VM person won't understand or like linux).
Linux has actually given VM a shot in the arm. VMers are well aware of that fact. I support a couple of VM systems during the day and play around with Linux at home at night.
Also in terms of culture, I think you would be very surprised with the VM culture. VM spent many years as the unwanted child, VMers had to rely on each other in order to be heard above the MVS roar.
If you take a look at some of the history of the internet you will find VM sitting there (BITNET was basically a collection of VM systems). The listserv concept was originally from VM (CERN was -- might still be -- a big VM site).
If you want to see some of the history of VM you can start here: http://pucc.princeton.edu/~melinda