Slashdot Mirror
IBM Saves $250M Running Linux On Mainframes
coondoggie writes "Today IBM will announce it is consolidating nearly 4,000 small computer servers in six locations onto about 30 refrigerator-sized mainframes running Linux, saving $250 million in the process. The 4,000 replaced servers will be recycled by IBM Global Asset Recovery Services. The six data centers currently take up over 8 million square feet, or the size of nearly 140 football fields."
This proves Linux has a smaller carbon footprint then other OS's!
Because they're using all that Microsoft IP without paying for it....
(it's a joke)
Do you have ESP?
for AIX on those mainframes! After all, AIX has more Unix IP than Linux, isn't it?
If you keep throwing chairs, one day you'll break windows....
Um, hello, while this may deserve the 'neat' tag, it's hardly newsworthy.
People are consolidating lightly (and heavily!) used servers into VMs all over the place.
I want to delete my account but Slashdot doesn't allow it.
We (Bigattichouse's Vectorspace Database) went through their Linux certification (as well as Grid cert), and they were a pleasure to work with - providing expert advice and patience in every step of the process. Not exactly on topic, I guess, but I thought I'd share. They really seem to embrace the engineering and spirit of Linux.
meh
The article says that the data centers required for the 4000 "small computer servers" aggregate to about 8 million square feet. It takes IBM 2000 square feet to house a small computer? Also, saving $250 million suggests that it costs them something over $60K per "small computer" even ignoring the price of the new mainframes. Amazing.
My employer recently 'consolidated' their server farm too. We used to have a room with fifty aging Dell PowerEdge servers, each running independently, requiring massive support, cooling, and electricity.
Now we have ten VM servers running all the migrated services, PLUS a room with about fifty aging Dell PowerEdge servers, each running independently, requiring massive support, cooling, and electricity.
I never thought 'consolidation' would require so much more space, electricity, air conditioning, and upgrades to core switches and UPS units.
"Sometimes, I think Trent just needs a cup of hot chocolate and a blankie." -Tori Amos on Nine Inch Nails
It's kinda hard to find technical specifications on these mainframes beyond marketing fluff. After some looking I found this brochure, which has some interesting information on the firmware and a few details of the I/O, but not much about the processing units, and why one of these would be able to replace 133 blade servers. It does mention up to 30 superscalar processors per box, but I'm not really sure what that means. (Maybe they go next to the inverting flux capacitor).
- Nothing to see hear.
When they came for the communists, I said "He's next door. Take him away. Goddam commies."
Lets see... $250M / 4000 = $62,500 per server being consolidated? I mean, I know floor space, buildings, racks, power, AC etc cost money... but that's still a *lot*. Anyone care to chime in on how close to normal that is?
The story here is about consolidation, virtualization, etc.
Linux is a small part of the technology involved here. z/OS is the real story here.
Am I part of the core demographic for Swedish Fish?
If you take hundreds of cabs and consolidate them down to 40 (with the associated consolidated storage) you are going to save millions. That has little to do with Linux. It is the modern mainframe that makes this kind of thing possible, which is why more people are moving to them. They must have a lot of servers spinning idle to get this done.
The reason why companies are in this pickle is because they thought more was better. They though "All we need to do is buy 4000 x86 servers and we can do tons of work." They didn't realize how HARD it is to get 4000 servers to operate in a cluster so you can take advantage of those individual systems as one body. So, they ended up with islands of computing power instead of a cluster. Naturally the mainframe consolidates these islands back to computing continents and you end up running the mainframes at near capacity all the time. Modern mainframes make this easy with dynamic CPU/RAM allocation, as well as dynamic storage. So you segment out the mainframe in to four or eight chunks. Chunk 1 is hot, chunk 3 and 5 are idle. Simply re-assign some of the CPUs from chunk 3 and 5 to 1 until the load goes down. You can take advantage of this in a big way if you segment your work load to match global demand. So chunk 1 might be data for the western USA, and chunk 7 might be EMEA. You can bounce resources between those segments much more easily. You can even script it. HP has an offering that does this automagically, I'm sure IBM has something similar.
Now, my personal opinion is why Linux? Some of the more advanced features like dynamic RAM, CPU, and IO allocation don't appear to be that solid to me. Perhaps IBM added these features to Linux or made them more robust? Maybe they run Linux inside an AIX virtualization container?
I had to maintain some software that was running on a aging 370 mainframe. The 370 was emulating a 360 which was emulating a 1401.
It was pension and payroll software and it was legally blessed.
It was such a frigging song and dance trying to get anything done that it was cheaper and faster for the company to emulate their butts off rather than trying to go through the management and the unions and the employees.
But I did learn about optimizing instruction fetches by scattering the compiled code around the circumference of a magnetic drum so that the drum would have rotated around beneath the read head in time for the next instruction.
Try and tell that to the young people of today, and they wont believe you, eh Obadiah?
MSBPodcast.com The opinions expressed here are my own. If you don't like 'em... Think up your own stuff.
Can I have the old ones?
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
I think a good part of that can be attributed to their being able to run one more or less standard OS across dozens of disparate systems. The real question is why they picked Linux vs. a proprietary design or some other open source OS.
The higher the technology, the sharper that two-edged sword.
These are machines that don't break, period. We're talking the types of machines that run the major banking systems of the world and the like. They simply do not go down. In this situation, if one of the 133 apps buggers up, it's only that VM that's shot. You just nuke it and restart it, the rest of the machine just keeps ticking along.
...si hoc legere nimium eruditionis habes...
Where I work we currently run two mainframes in a sysplex environment for all the core transactions. It's a very optimized environment and handles millions of financial transactions a day. In mid-2006, IBM started giving us zLinux engines to "try out" and they gave us all of the software we needed to make a go of it. Kind of like a playground drug dealer, they hoped that by giving us a bit for free we'd get hooked and become dedicated customers. The problem was, for the type of workload that typically runs on our servers (high CPU, moderate I/O) we were experiencing poor performance on the mainframe VMs. IBM sent all their engineers out to help make tweaks and tune all sorts of things. Despite all the tuning and tweaking that took place, we could never get a single engine to perform better than a $5,000 server. Keep in mind that a single engine was retailing for around $80,000 after discounts.
We did some calculations and determined that for the price of a zLinux engine we could buy an entire rack of high-end HP servers that would outperform the single engine by a factor of 200:1. Again, maybe it was just the workload we were doing, but even IBM couldn't figure it out and our server work profile isn't exactly uncommon. Granted you can cram a lot of guests onto a host system provided that none of the guests want to use more than 10% of their CPU at any given time, but that defeats the purpose. I could probably run a VMWare host with 100 guests and call it a success, provided they all sat idle.
It was kind of funny because the IBM engineers would shake their heads and admit that for our workload it just wasn't going to work out. Then the next week the sales guy would call and ask if we were ready to buy that third mainframe since he just read the engineer's report and our visit was obviously a smashing success.
I'm not knocking the whole Linux on the mainframe concept, I'm just sharing our experience and how the whole thing seemed to be like someone in IBM Marketing declared "we need to sell Linux on the mainframe" and the Dilberts were forced to sell a product that worked about as well as a chocolate fireguard. It was a very awkward experience and even the IBM engineers seemed like they were stuck in an uncomfortable position of supporting sales for a product that even moderately demanding customers wouldn't be able to run with.
Personally I consider Linux on the mainframe to be on par with running Linux on an iPhone. Sure you probably can, but does it actually do anything uniquely useful for the business? I have a hard time selling technology to the CIO on the grounds that because it's Linux it's a good business decision regardless of the context.
Not quite. They are engineered (as they have been for decades) for stability and were designed to handle that kind of load. Its CPU/RAM/storage are redundant, so that if something in the system goes down, new resources are allocated. Additionally, shops will have multiple mainframes just for that kind of redundancy. Its kind of like saying your car is a "single point of failure" - sure, it is, but they were engineered for the purpose of being reliable.
Because after OS/2 died IBM didn't want to get fucked by Microsoft again and Apple didn't have a business solution. Creating another ground up OS was definitely not a solution so they were left with two options: Linux and Unix. The only reason Linux is run on as many servers as it is today is because of IBM. They do Linux R&D, produce software, and provide B2B services for the OS. Linux is the official OS of IBM and without their support Microsoft would currently have a monopoly on server software. So, a simple answer to your question would be that IBM has a financial interest in Linux. But it goes deeper than that: IBM has committed to Linux - it's a marriage.
I think you're hesitant to accept IBM because of the whole 70's/80's "Big Blue" stuff, but after Microsoft swept the rug out from under their feet the company's strength was permanently compromised. The consumer market rejected them (hence the sale of the PC division to Lenovo) and until they committed to Linux software was a major vulnerability for them. The openess of Linux enabled them to get back in the game - their customers didn't have to worry about the future of the platform while their immense contributions to Linux enabled the OS to really threaten Microsoft. So yeah - as a Slashdotter, IBM are the good guys. They support Linux and they don't aggressively protect their many, many patents (they use their patents to protect themselves rather than trying to sue everyone they can for $$$). Personally, I think IBM is the most important tech company in the world.
I still haven't seen any conclusive evidence that Linux on mainframe is a good idea. I'm sure running 30 new mainframes is going to cost less than 4000 aging servers. Just about anything would be less expensive than 4000 aging servers.
But I bet that a small farm of modern medium sized servers running Linux on VMWare would be even less expensive. Or Solaris/Niagara. Why would you want to run an open source operating system, whose major benefits are openness and affordability on the what is literally the most expensive and most proprietary computing platform in the world!
These server consolidation projects are just giant boondoggles spawned because the server sprawl finally got insane. It's an endless cycle:
A. Giant server consolidation project that takes 4000 servers down to 30 servers.
B. Department B complains that Department A's application keeps hanging and consuming all of the CPU. They demand their own hardware "for availability reasons".
C. Vendor C demands dedicated hardware for licensing/capacity planning/supportability reasons. Rather than constantly bicker with the vendor over supportability they get dedicated hardware.
D. Department D complains that the IT department is charging outrageous prices for time sharing on the mainframe. After all a dedicated server only costs $XXX.
E. Suddenly there are 4000 servers again.
F. IT department spends some insane amount of money on infrastructure to manage the 4000 servers.
G. IT department budget gets insanely large trying to manage that much stuff.
H. Some CIO gets the idea that all of this money managing servers is ridiculous and we should do a server consolidation project.
I. IT department spends an even larger amount of money on the latest super high availability gear and consulting services so that the can run 4000 commodity servers inside a few big servers. All because it will "cost less to maintain".
J. Go back to A.
I should hope so!
8,000,000 ft^2 / 4,000 servers = 2,000 ft^2 per server
My god, those are large servers! They must still be using vacuum tubes...or maybe a 65cm manufacturing process.
@ASP.NET's parent-teacher meeting: "Little Johnny.NET is very bright, but he doesn't play well with others."
While I agree that IBM's mainframe systems are rock-solid (and, as a colleague is fond of saying, self-healing), accidents *DO* happen. I'm sure the mainframe is happily running its code just fine only seconds before a hurricane rips the roof off of the data center and hurls the machine into the next county....
It's those kinds of things that make disaster recovery necessary. If the apps were distributed across discrete servers, its possible that not all of them would have been destroyed. Remember the end of Twister? The barn was wasted, but the house was left intact.
actually, I think we need a Parallel Sysplex of these.
Before, .33% failure rate = 13 failures a day. You had well understood procedures for dealing with failures.
.33% failure rate = 1 failure per thousand days. This is a recipe for hell.
After,
But wait...
When you do have one machine fail- it takes down 133 virtual servers at the same time. You raised your risk enormously.
IBM will tell you all about fail-over just like they did our executives.
Half the country down for three days is the reality.
---
Still it is interesting to see a return to the centralized mainframe farm. Sure hope those multiply redundant communication lines don't go down.
She was like chocolate when she drank... semi-sweet at first and then increasingly bitter.
Last week, I attended a presentation at IBM's Australian Development Lab in West Perth, where a lot of the z/OS-related code is maintained and developed.
From what we were told, IBM z/OS mainframes are the *most* reliable platform to host software services (but of course, they'd say that).
The following is from memory, as best as I can remember it, and may not be 100% accurate:
The 'z' in 'z/OS' stands for 'zero downtime'. z System mainframes are engineered for 99.999% availability, or less than 3 minutes of downtime a year (we were actually quoted 'less than 5 minutes', but (1 - 99.999%) * 365.25 * 12 * 60 = 2.63). Apparently, they quite easily meet this requirement - we were told that it is not uncommon for systems to remain online for 10 years or more without failing.
Up to 32 z System mainframes can be clustered in a 'sysplex'. Each mainframe is divided into several LPARs (Logical Partitions), each which can host several VMs. If an application fails, the automated recovery service will attempt to restart it, either on the same VM, a different VM, a different LPAR or a different mainframe in the sysplex, as appropriate in the situation. It is also possible to host a redundant sysplex in a different site, which mirrors data and which the primary sysplex can failover to in the case of failure.
IBM mainframes are used in many major corporations around the world, particularly those where the cost of downtime is very high (think thousands of dollars a second).
There are a few reasons why the specs for mainframes are so hard to find.
n s.html
One is that the things you find on IBM's website are designed for CEOs and CIOs who don't really care about technical details -- only "solutions"
The second is that the specs themselves aren't well-defined. As an earlier poster pointed out, you don't buy one of these things off the shelf. You tell IBM what you want to do with it, and you work with them to construct not just a mainframe, but all of the storage and other add-ons that come along.
And finally, the third reason is that the specs don't line up with anything you likely work with normally... (If they did, you'd know where to find them.)
Here are some specs for the z9 Enterprise Class:
http://www-03.ibm.com/systems/z/z9ec/specificatio
Simplified you are looking at 54 CPUs with 512GB of memory.
The CPUs themselves are basically Power6 processors, but thats really simplifying everything down.
Each CPU is actually a "book" of CPUs. Several run at once on the same data. If any disagree, the instruction is rerun on a different CPU. Entire backup books (in addition to the 54) kick-in if a problem is detected.
Additionally, the z/Series comes with a bunch of "Specialty" CPUs. You can get 27 CPUs that do nothing but process Java work natively. Or ones that handle DB2 workload. Or even special processors optimized for the linux kernel. Oh and don't forget the built-in hardware crypto CPUs.
Memory and I/O and Power and everything else works pretty much the same way on a mainframe. And all of it is hot-swapable. (Even the Emergency Power Off switch can be replaced while the system is running).
The hardware specs are impressive, but the biggest deal about these boxes is that they don't go down. Most people I talk to question the idea of consolidating servers into one box because of "single point of failure" concerns. This is where the mainframe shines. These things have MTBF of decades, and will just churn away forever.
I work in a longtime "blue shop" perspective, not always from a software/OS perspective. While I like the 'concept' of running linux on zSeries, I think you could take a look at the requirements and choose a platform that can run the same Apps.
For example, for email we run Lotus Notes on a couple of BIG pSeries (AIX) servers. We could have run it on a farm (technical term) of windows boxes.
For webservers, which you could run on AIX, or linux on zSeries. We have multiple (read: many) x86 servers running linux+apache. Why? They connect to a backend app server (pSeries) which connects to a backend zSeries DB2 (I'd prefer Oracle however, to run Oracle on zSeries requires it to be run in a linux VM).
We definitely subscribe to the school of using VMs whether they are zSeries, pSeries, or VMWare on x86. Even if the x86 server is running ONE application, we still put vmware underneath, as it allows for us to move the image to a newer hardware platform when it's time to upgrade. Even some of the larger x86 servers run vmware but in each partition there is a single instance of apache. Makes for managing storage that much easier (fewer zones, cabling etc).
Would I consider moving our apache on linux on x86 to apache on linux on zSeries? Not really. It's a waste of CPU cycles (MIPS). I'd rather use zSeries MIPS for something a bit more critical like keeping my database up and running than serving out webpages (static or dynamic).
IT isn't not about religion, it's about finding the best tool considering your requirements. I have no problems telling IBM that product XYZ is trash. While my servers are IBM, you won't see IBM disk, or IBM tape, and atleast once a quarter some salesman from IBMs storage group is at my door. He buys me lunch and every quarter he is sent packing. You won't see ibm bladecenters as the thought of hundreds of additional servers to manage isn't appealing (but I'll gladly take 100s of VMs across larger x86/pseries boxes).
I know many of you were expecting to hear me say 5000 linux servers, but there are options for my requirements that did not lead to big "google style" linux farms.
BTW: I have no problems kicking out IBM on x86 if HP/Dell/Sun have a better product, and knowing this and letting IBM know this gives me a great advantage over them, as they very well know I'm capable of bringing in something more suitable. (I *used* to have IBM storage).
Yes, 4000 "small computer servers" times 2000 square feet equals 8 million square feet. But this is unlikely the arrangement. Consider instead a few buildings of data centers, each with 1 or more relatively small rooms. Within a room, there may be a few racks, all surrounded by walking space, and other perhipherals like AC units. Then outside of those rooms, more walking space for hallways. When you factor in all the human space and simple space for ventilation, and then cubicles and monitoring for support personnel it could average around 2000 square feet (40x50).
The same logic can be applied to costs.. $250 million / 4000 machines = $62.5K. Some of that is actual hardware, and software licenses. Some of that is ongoing support from their full time employees on staff to maintain the things.
There are 10 types of cliches in this world. Those that are new, and those that aren't.
How is that different from the hurricane ripping the roof off of the data center and hurling 400 blades into the next county? Or as we saw in SF, a power outage taking out an entire data center?
Any sect, cult, or religion will legislate its creed into law if it acquires the political power to do so.
Yeah, if somebody hits the Big Red Switch, there's going to be a problem. But, if they don't, well, it's a mainframe.
The Linux on these machines is running under z/VM, in multiple virtual machines. When one of them has a software fault, you reboot that one's VM and keep going; the other 132 Linux-running VMs run without noticing anything happened. (It is possible for z/VM to fault, sure. But it's an OS with 40 years of refinement in the "100% uptime" mainframe culture, and its task is just managing the virtual machines.) When something goes wrong with the hardware, the fault tolerance and self-healing features keep things running, and you fix the faulty element with a hot-swap. A properly set-up datacenter is going to minimize external risks, with backup power and such. Proper choice of datacenter location will minimize natural disaster risk.
So, yeah, the big risk is human failure, and these IBM-built, IBM-owned datacenters are presumably going to have extensively trained IBM-employed mainframe personnel, which minimizes that risk.
Now, if some cable company cuts the fiber optic lines . . .
The extra space is for sufficient ventilation for the sweaty Sysadmin. Nobody wants to go near him.
Cheers, Chris
"The six data centers currently take up over 8 million square feet, or the size of nearly 140 football fields."
I suppose when the US finally goes metric, they'll have to deal with units of area such as "millifields", "centifields" and "kilofields". In time, the measure will have to formalised e.g. "the distance a 100kg, 190cm man is able to kick a leather-encased rubber bladder...".
Or maybe the current generation of writers that thinks "140 football fields" is a meaningful substitute for "a really big chunk of space" will have died off by then.
IBM is a hardware company, always has been, they've been into open source software since before GNU existed.
http://en.wikipedia.org/wiki/SHARE_(computing)
Sure, they are an evil corporation with too much money on retainer, but they realized long ago, that software has an intrinsic value that crashes once the software is written.
For instance, the labor theory of value - the most influential of the intrinsic theories - holds that the value of an item comes from the amount of labor spent producing said item.http://en.wikipedia.org/wiki/Intrinsic_theory_of_v alue
Basically, once software is written, it's value rapidly approaches zero, because the ability to replicate that work is well within the skill levels of the neophyte. IBM conceded the value of software long before Bill Gates came around floating the idea that the value of software could be upheld by government interference, essentially creating a new fiat currency, and entering the business of printing money, they hired lawyers to back it up along with becoming an extremely predatory business entity.
While IBM may have quite a bit to lose going the free software route they have a lot more to gain. Once they own all the copyrights/patents they can do whatever they want and that (currently) includes GPLvX or greater.
Under the influence of Post-Cyberpunk Gonzo Journalism
Well, this has been the first /. flooding I've ever witnessed...
It is rather interesting that you should flood flood like you do, then bemoan cultural intolerance... I participate in a forum where several users (or "morons", as I dub them) demand and exercise their "right to flood", claiming that cleaning up their flood is denying them their right to free speech.
I just don't understand how do you find the time to do things like that...
Ignore this signature. By order.
There are a lot of errors in your comments, unfortunately. Of course you can run Red Hat and SuSE concurrently in a single LPAR under z/VM, and multiple versions thereof. This has always been true, ever since Linux began running on mainframes many years ago. You might want to have more than one LPAR to run more than one version of (first level) z/VM, but you don't need many. Two or three for z/VM and Linux is typical and just fine. And it's not as if LPARs are in short supply on mainframes: up to 60 are available on a single machine (30 on the smaller model), so "spending" 1 to 3 is no big deal.
Re: Investing in new mainframes, come on, get real. It's so easy to find market data because companies like Gartner and IDC publish it, and IBM just announced its 8th straight quarter of mainframe hardware growth, something that hasn't happened since before Y2K. It's impossible to do that with "a few showboat customers."
And no, you simply cannot approach the level of virtualization these machines offer on any other system, at least for typical business computing, and still offer reliable service to users. In fact, in IBM's case many of the software licenses are presumably "free," and they still found big cost savings by taking 4,000 machines down to 30. For the rest of the world the mathematics in such situations are even more compelling.
I was involved in a migration to the zOS architecture three years ago. I am currently involved in a similar exercise for a British telecoms company whose name escapes me. In both cases the principle was perfectly sound, but the reality rapidly starts to come down to what can be migrated, when, and why. At IBM application compatibility was a major consideration, and ultimately prevented key parts of the system from being migrated. At the current site, surprise surprise, the problems are the same, plus reluctance to do the work (upgrades, work required on the client's part, age of applications and Plain Old Politics). I wish IBM good luck, and perhaps because there is a better integration of operations and systems they might succeed, but I would be willing to bet that by the end of the process, they will have reached about 80% of their target.
...by switching to Geico.
I put the 't' in electrical engineering.
Actually, on a System z9 EC (Enterprise Class), a single CPU chip failure is not a "Call Home" repair event. Only the second CPU chip failure would result in an automatic call, while your business keeps running of course. (There are a minimum of two spares in each machine.) The average time to first failure for a particular machine is somewhere in the many decades range.
OK, just for fun (because it never actually happens in the real world), what happens with a triple failure? If you happen to have a "fully configured" mainframe -- all processors turned on -- then.... your business still keeps running. Yes, the system might lose some processing capacity, but it keeps running. The higher priority stuff (from a business view) takes precedence automatically, and life goes on. This is all on a single machine still.
If you've got an S18, S28, S38, or S54 model, then, at your business's convenience, the faulty hardware can be replaced. (You might do this at night, for example.) The repair technician tells the mainframe to "evacuate" memory on a portion of the machine while the OS and applications keep chugging along, possibly with reduced capacity, often not. (Depends on what configuration you choose.) When the evacuation is complete, the technician can pull a processor/memory group (called a "book"), insert the new one, bring the new one online, and... everything still keeps running. Again, this is all on a single machine -- no clusters required for any of this.
The six data centers currently take up over 8 million square feet, or the size of nearly 140 football fields.
In metric, that would be around 104 soccer fields.