Slashdot Mirror
Linux From A CIO's Perspective
An anonymous reader writes "CIO.com has a story on Linux and OSS in the enterprise from the perspective of the CIO of Cendant Travel Distribution Services, Mickey Lutz. 'In the summer of 2003, Mickey Lutz did something that most CIOs, even today, would consider unthinkable: He moved a critical part of his IT infrastructure from the mainframe and Unix to Linux. For Lutz, the objections to Linux, regarding its technical robustness and lack of vendor support, had melted enough to justify the gamble.'
His organization saved 90% in costs in so doing. Read on if you want to see how the top brass views OSS."
Makes us more liable than BSD (due to the GPL) but not particularly better? SIGN ME UP
Only three remote holes in the default install, in more than 10 years! OpenBSD
LP!!!
This pretty much sums it up:
;-)
Lutz's IT group rewrote a complex, real-time airline pricing application that serves hundreds of thousands of travel agents around the world and that also acts as the system of record for all of United Airlines' ticket reservations. When this application came up on Linux, it proved to be so demanding--it handles up to 700 pricing requests per second--that it completely redefined Cendant's expectations about what it would take to get Linux to work. "We have broken every piece of software we've ever thrown at this platform, including Linux itself," says Lutz.
With Big Iron you're paying a LOT of money. But you're not paying it for nothing. Big Iron will give you a lot of guarantees for stability, reliability, and thoroughput that don't exist on other systems. The key to this CIO's success is that he was willing to accept the challenges of doing Big Iron work on Little Brass systems. As long as you work all the details out yourself, this *can* work. (As Google has so eloquently proven.) The issue is that you're working without a safety net. If things go really wrong, there's no backup army of specially trained techs to run in and fix things. (And trust me, if you're paying enough money you'll have your own personal army of techs.)
The upshot to all of this is that if the gamble pays off, it pays off in a big way. All that money you were spending for a personal army, plus some other company's R&D now goes into your own pockets. You don't get away scott free (someone has to maintain the systems), but you see your rewards. And isn't that what business is about? Taking risks and making profits? If you've got the infrastructure to go for something like this, then go ahead and grab fate by the balls. No one ever got anywhere in life by playing it safe.
The "black box" of open source has transformed into something any CIO can appreciate: reliable performance and consistent uptime. The penguin can fly now.
Javascript + Nintendo DSi = DSiCade
Moving desktops to linux might be considdered revolutionary, but this isn't. The big iron market of servers and HPC machines has really been dominated by linux for several years now.
------ Take away the right to say fuck and you take away the right to say fuck the government.
The only thing that makes this news is that a CIO actually recognized it.
would consider unthinkable: He moved a critical part of his IT infrastructure from the mainframe and Unix to Linux.
This is actually quite thinkable. Now if some CIO moved all his desktops to Linux, I would be impressed. Moving the backoffice stuff from expensive licenses of Unix and mainframes to Linux is a no-brainer.
This news is all well and great, but it's been known for a while that moving from UNIX->Linux was cheaper. What should be focused on is the switch from Win -> Linux... Of course, that's not going to be showcased, because the costs of running either system are about equal.
Most C** are idiots. Is it now news that one makes/made an intelligent decision? I didn't read the whole article, but from my experience, some people 'convinced' him and now he takes the credit.
Any other ideas?
An interesting question that this article raises for me. Is what intel arch was being used (itanium/x86). For example could the costs have been reduced just by using linux on say a large scale IBM server similar to their other mainframe?.
It also goes to show that just because something is old does not mean its slow..
some guy name "bill" called from redmond. he wants to explain you why linux is more expensive...
What ? Me, worry ?
Actually, all of those bright, shiny websites (Expedia, Travelocity, and Orbitz) rely on a GDS (Sabre, Amadeus, Worldspan or Galileo) to provide their content.
Because you can't spell "slaughter" without "laughter"
Unix: $25 million
Linux: $2.5 million
These numbers were taken from a table in the article. Interestingly enough, the cost if something does break favors Linux as well. From the same table we get that the mainframe solution consists of 4 IBM mainframes, whereas Linux and Unix solutions require around 144 servers for Linux and 100 - 120 servers for Unix. If the hardware goes to hell it's so much easier to replace the single bad part than a mainframe.
Hopefully, more people will begin a transition to open source solutions when they realize it can be successful.
I switched my desktop and saved $90. Seriously though, do any slashdoters have experience switching their companies computers to, or even away from linux?
lol: You see no door there!
Now, let's get prepared to rebut any Microsoft officials whenever they talk about the common "Total Cost of Ownership" as far as Linux is concerned.
I tried switching the family over to Linux machines over the summer
vacation and the objections from the other family members was more
than enough to send all 7 machines right back to Windows ME, Windows
2000 and Windows XP.
So why all the troubles?
Afterall Linux users in COLA love to tell stories of how Linux is so
much better than Windows and they would never go back and so forth.
My conclusion after seeing real people in a real average Jane setting
revolt against Linux is that the Linux advocates are just plain lying
because Linux is really a step backwards for people used to using
Windows.
To make this short and simple, virtually NOTHING worked properly under
Linux.
Video cards could not get maximum resolution.
Capture programs, for my ATI All In WOnder and Video camera did not
work. In fact my ATI cards advanced features (remote control amongst
other things) didn't work at all.
My printer (Brother all in one fax/copier/printer) did not work.
My DSL connection did not work and when I called support they said
that Linux was not supported.
My mp3 and mpeg video and music files played but they skipped
horribly.
I couldn't log into my router via konqueror to change/view settings.
MANY, MANY, MANY web pages did not display correctly.
And it goes on and on for pages,but the bottom line is that Linux
lasted about 3 days in my house before I ditched it and went back to
the Windows versions I was using.
Thank goodness for Ghost which mad it easy to do.
Conclusion is that Linux is a birds nest of confusion. Linux seems
like it might be good until you actually try and use it and then it
shows it's ugliness, slowness and instability.
Why on earth ANYONE would use Linux for a home system is a mystery.
Stephanie
Like most critics, I'm not good at leading large companies. But I know good leadership when I see it. This guy Lutz has his head bolted on right.
The first thing most CIOs usually throw at their workforce is not to re-invent anything. If a product exists off the shelf at a reasonable cost, there are lots of disadvantages for taking the risk of inventing another one and few advantages if you succeed.
However, most of us workers have known that the "big iron" mainframe technologies of yesteryear are starting to "rust." It's getting difficult to find technical help who understand this stuff reasonably well. That brings me to the second point: Follow the technology market. The people will be there.
I suspect that in the not too distant future, many big-iron mainframers are going to be asking theselves whether the many millions they're spending are a good ROI. Open source databases and distributed computing are starting to look awfully attractive.
It's scary from a CIO's position because the old systems are working, even if they're not well understood any more. They're leaping from the systems they know, toward a high cost potential boondoggle. This guy apparently knew how to hire and retain good technical help, he knew how to organize that help, and he knew how to keep them focused on the goal.
Most leaders aren't that good. All too many businesses operate by habit. Only the red tape holds them together. Those organizations won't be making this leap until a certain critical mass has been reached to convince them one by one to make the effort.
We should be doing everything we can to encourage others like Lutz to push these efforts. This is how you really evangelize Linux. And when all this is over, the desktop will be an afterthought.
Nearly fifty percent of all graduates come from the bottom half of the class!
An interesting read. But very interesting for what was left unsaid as well. There was a fair amount of pain associated with the switch - aggravated by rolling out a new application simultaneously. The slowdowns and the associated costs are glossed over but I wonder how the business side feels about this change to only 25% of the entire infrastructure.
The time window seems fairly broad as well. No one disputes that lots of cheaper intel servers can do the same job as big iron. THe question is how many does it take and what happens with the applications involved.
Quite telling is the comment that they needed every bit of support possible. Although it is great that one CIO bit the bullet here....there is an ominous side to this story which means that few others will follow suit.
...how much money he would save if he moved to Microsoft!
Cheers,
RoadkillBunny
I was hoping to see a "Windows has lower TCO than Linux" ad that slashdot runs for Microsoft when I clicked the article.
Most C** are idiots.
:(
And I thought C# was bad enough. This naming scheme is getting out of hand
Irritable, left-wing and possibly humorous bumper stickers and t-shirts
But did he get a raise? Say about half of what he saved them.
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
Mainframe != Cluster.
The CPU speeds are very low on those things.
The advantage is that the CPUs can get the data they needs very quickly so it can do a whole bunch of transactions at once.
"In hindsight," says Lutz, "we shouldn't have tried to cut over to a new infrastructure at the same time we were deploying a new software application. It was too much at once."
They found that their Linux servers couldn't support the new application they had deployed at the same time. That doesn't mean it's less capable than the mainframes they replaced: they didn't even try running the higher-load application against the mainframes.
They should have first ported their servers to Linux on the mainframes, then switched them to Linux on clusters, then sent out new software that they could force back to the old behavior, then supported the new software in general.
That way, they'd have been able to isolate the problems more easily (which really turned out to be that the new application generated extreme peak loads, and nothing to do with Linux per se, aside from that they managed to improve the Linux performance to deal with it) and keep things stable while they fixed the issues.
The article did mention that Redhat and IBM were both part of the cutover team, so I guess they were the vendors.
Please be more specific about what ideological goals you're referring to.
Digital Citizen
Seriously, the biggest problem with mainframes is that switching them off is a big problem. These are not boxes you can easily - or safely - reboot, if there is a problem. There usually isn't, because the hardware is usually of very high calibre and massively redundant, but scheduled maintenance of, say, an Amdahl or a (when they existed) a Prime was not a trivial affair.
"Routine" maintenance wasn't much better - DEC would charge the Earth (and Mars) to swap tapes on even a humble VAX, and apparently had contracts with some places (such as my old University in Glamorgan, Wales) where absolutely nobody else was permitted to conduct such delicate operations.
This meant that no sane person ever did anything, if they could possibly help it. Which is one big reason that Big Iron started sliding into unpopularity the moment clusters started appearing.
The other problem with mainframes is that the manufacturer usually has the purchaser over a barrel. You can't exactly walk to Fry's and buy a new RAID controller for a CM-5 Connection Machine, or a new processor card for that Amdahl mainframe in the corner. The manufacturers know they have an absolute monopoly on parts, so charge the absolute maximum the market will bear.
For large clusters, the situation is very different. You probably wouldn't buy commodity off-the-shelf parts, if you could help it, but you COULD. That keeps the price somewhat checked on the higher-end higher-quality parts, because you can do fail-over. If you have twice the reliability, but over twice the price, it becomes more effective to use redundancy and hot-swapping.
Software is another important consideration. If you upgrade the software on a mainframe, you upgrade the WHOLE mainframe. If you upgrade a cluster, then so long as there is backwards compatibility, you can roll out the upgrade a node at a time, keeping the system as a whole running.
True, you don't usually do major brain-surgery on an IBM mainframe, as IBM isn't stupid enough to make severe enough changes to AIX to force a major overhaul on a regular basis, but (a) that limits how AIX can evolve (which will eventually kill it), and (b) major overhauls are a part of the computer business and do happen - you can't avoid them.
In today's world, though, it doesn't make sense to use ultra-specialized hardware and software. It isn't cost-effective and it isn't maintainable. A glance at a number of mainframe manufacturers show many have SOME kind of Linux offering, which (to me) shows they feel the same way. Eventually, Big Iron will become just a very fast component in a much larger, much more powerful super-cluster, rather than something significant in and of itself.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
The article seems to be comparing a $100 million implementation on a thing called UNIX against a $2.5 million dollar implementation using Linux on Intel. What's a UNIX? And why does it cost so much? A clearer definition of the hardware platforms being compared would be quite useful.
Additionally, in my opinion, the guy should have been canned by Cendant the moment United Airlines was off the air for 45 minutes. If YOU were responsible for this serious a screwup would you still have a job? Probably not.
lost it5 earlier betw3en each BSD crisco or lube.
They had serious problems, but this sounds like a sufficient safety net: "a 40- to 50-person cutover team of IBM, Red Hat and Cendant engineers brought the problems under control by throwing more servers into the mix."
Yeah ... like a 1970s era mainframe could run Linux! They could have LEASED enough Linux servers to do a full test run. Still, until you actually do the cutover, it's hard to really know what will break with a complex app.
Why is it that the article only thinks/talks about Intel and Linux? Linux has been to ported to almost every architecture imaginable e.g. PPC x86_64 ARM m68k, and probably more exotic architectures a well. For that matter why isn't AMD considered?
It may not support all of the latest sound and video cards, but it sure makes a better server.
In Soviet Washington the swamp drains you.
I bet a year later he's kicking himself from being the most important (visible to the Board because for every 1% he saved on that $100mil makes or breaks the quarterly reults) guy in the company to a nobody.
I find this article somehow amusing. Before becoming CIO at Cendant Travel, Lutz was at PHH which wrote one-off customized web applications. To Lutz it didn't matter if you wrote the same application over and over, he did not seem to appreciate a systems approach to development. He was more apt to approve of the "hack it and shove it out the door" method. Guess that's why he has since been put in charge of 'strategic projects' which at Cendant is tantamount to being put out to pasture. Last I heard he left to 'pursue other interests'.
via Stephanie Klugg Aug 15 2003, 6:33 pm
Someone is obviously paid to do this.
This article is badly focused. Their porting problems are not with the Linux platform. Their problems are related to lousy application architecture. (Or they would seem to be. Since I have not seen what they did I really can't say.)
... we do the up front analysis, and should have a pretty damn good idea that it will actually work when we build it. It sounds like this system might have been hacked.
...
Yes, distributing software is HARD, but it's something that can be modeled ahead of time with suprising fidelity. That's the difference between engineering and hacking
This seems to be a common issue and I don't understand why CIO types can't outgrow it. If you build the software correctly the platform becomes more or less irrelevant (E.g, Linux, BSD, SCO, LynxOS, whatever.) Why is this so difficult to understand? I've personally DONE this for several large systems (albeit not as large as GDS) and the guys in charge of the port are always locked into moving to a specific hardware/software platform. Maybe it just comes from an olde skool mainframe oriented mindset.
Unless, of course, this was truly a port and not a new architecture. I which case I have nothing but respect for these guys. It's always a miracle when those work at all
"There's no secret. You just press the accelerator to the floor and keep turning left." -- Bill Vukovich
The Fares application and infrastructure represent just 10 percent of the Galileo computing platform. The rest houses the massive collection of flight information for every airline, every route in the world, written in a 1970s-era mainframe language called the Transaction Processing Facility (TPF). "Unlike today's operating systems, TPF was designed almost exclusively for speed," says Wiseman.
... that raw power being speed. During top loads the system easily handles 4000 messages per second ... which will choke off any other OS I know of.
... apache has/is being ported to it ... a web server in TPF will then be every WebAdmin's dreams ... as it can never be slashdotted :-)
I am one of the programmers who develop for TPF. What is mentioned here, the part about TPF being designed for speed is 100% correct. It is still the fastest OS available IMO...
Most of the development projects are done only in Assembly, and even in that we generally concentrate a lot on minimizing the code base and MIPS. There is no way any other architecture now, which includes Linux, Unix, BSD's can compete with TPF in terms of raw power
And there is another new development in the horizon for TPF
That said, I am really hapy that Linux is improving it's position in the market. I would've been much happier if this was in the desktop rather than the backend server .
"In questions of science the authority of a thousand is not worth the humble reasoning of a single individual."
The moment that major CAD software operates reliably on Linux I'll start to pay very close attention. I said *major* software, not some homegrown thing that can draw only lines and circles.
Here, from TFA:The CIO decided not to TEST the system correctly?Their customers cannot access their new Linux system!They were LOSING money with their new Linux system.This guy made novice-level mistakes and it was only because Linux is so good that this became a huge success rather than a terrible failure.You always have a back-out plan. Always.
This guy took a huge risk
And the Linux system STILL saves him $$$MILLIONS$$$ every year and OUTPERFORMS his old system.
It's one thing when you're a genius CIO who plans and test for every contingency and deploys a working Linux system.
It's a completely different thing when you don't BUT YOU STILL SUCCEED BECAUSE OF LINUX.
This story is important because it shows the average CIO that, even if you aren't a genius and you DO make mistakes, Linux can STILL save you barrels of money and make you LOOK like a genius.
Tried to install SQL-Ledger to see if it was an alternative to Quicken.
After many hours, I gave up. Given that "production quality" Linux software contains installation instructions that say, in one step:
Install OBA
and the instructions to install OBA are a copy of the SQL-Ledger instructions, including the same line that says "Install OBA," I say "Screw it."
Linux is fine for wankers who want to spend their weekends screwing around with it. There are people who screw around with getting old cars to run, changing brands of vacuum tubes on their preamps, or trading matchbook covers. Fine. Everyone needs a hobby.
But for people who actually want to use a computer to do something productive, rather than using it to waste time, Linux simply doesn't work.
Sure, there will be some success stories as people convert to it. After all, having to learn JCL parameters was a waste of time too. But eventually, it too will fade, as the cost of maintaining such a mess consumes them.
Microsoft will win in the end, not because their products are good, but because when the day is done, people can get their jobs done using them. That still isn't true for Linux as long as the quality of products resembles what you find at a flea market.
There are a lot of things in the article that make me think that these guys are throwing a bit of smoke and confusion - or just don't know what they're doing. Here's one example:
According to Lutz, the number of possible combinations of flights and prices for all the airline carriers between two major cities has been estimated by researchers at MIT to be 10 to the 30th power.
Sure, if you want *every single* combination. Yes, I could fly from Denver to Las Vegas via Miami, New York, Chicago, Seattle, Los Angelas, Houston, Salt Lake, and *then* to Las Vegas - with up to 7 day layovers in every city - but that's just silly.
But by assuming just a few simple sanity checks (such as fewer than X stopovers, change carriers fewer than Y times, longest layover no more than Z hours, or "dont-use-puddle-jumper=1"), I would imagine that you could pull that number from 10^30 down to no more than 10^3 - perhaps less than 10^2. When you trim the size of your data set by 27 or 28 orders of magnitude before you even start your processing, then suddenly those 700 or so transactions/second start looking a lot easier.
Then, of course, there's yet another old standby of people needing lots of computations: Don't repeat your work. Insert a machine or process that watches each incoming request, and caches the results in a lookup table for future use. Yes, there are implementation challenges (such as marking entries as dirty when route information changes), but $50k in programming could save them more than $1m in hardware.
steve
Oh, you're not stuck, you're just unable to let go of the onion rings.
At 400w a piece, that will make for one nice electric bill; I wonder why they didn't put linux on the mainframe.
If this is the same Stephanie then it is none other that Gary flatfish Stewart who has been trolling COLA for years. Since Dec 2000 in fact. You will only find references to his earlist posts as he X-No-Archives them. His particular pathology is to post aliases composed of smutty wordpuns.
..
I do apologies that he's now discovered slashdot. I'll try and up his medication. Also take these aliases into consideration.
Richard C. Hymen aka Stephanie Klugg aka Patricia Fitzhenry aka Mike Cox aka Aftab Mooshoo Bong Singh aka Thomas J. Crapper aka Karen Livermore aka Sid Fidler aka elenacurva aka Peggy Wanker aka Peggy Wanka
I personally hate them, but their software is powerful and considered major CAD software.
The mainstream vendors, (Solidworks, Solid Edge, Inventor, et al.) are all married to the win32 API. For them, it will be a good long while just like Microsoft likes it.
However the big three, Dassaut, PTC, UGS all run on UNIX today, with one PTC Linux port. The others all claim too many support issues. (copout, support one distro and let your users sort it out.)
It's coming, but slowly.
Blogging because I can...
They could have their partitions running in modern mainframe environment first (the original stuff was from IBM so this is a no-brainer as IBM's mainframes are really well backward compatible, even on word length issues and such), then add Linux to the mix. After that, porting on Linux/x86(-64) would have been trivial.
This has been part of IBM's strategy for a while: run Linux from mainframe to the cheapest possible x86 hardware. The benefit? Single unified programming environment lets customers gradually migrate applications to hardware of choice. It's not that unusual to see Linux on zSeries as it makes perfect sense for consolidation in cases where VMWare ESX Server just does not cut it. The bottom line? Efficient IT operations management when system administration is focused on one operative environment (+ some mandatory Windows backoffice stuff).
Grabbing Fate's balls doesn't seem like such a good idea - I mean, if you were Fate, wouldn't that piss you off?
Can't I "reach for the brass ring" instead? And wouldn't that be much more hygienic anyway?
A fifth of the capacity of their system before they switched to Linux was on hardware purchased in 2001 to handle the rush of bookings when airports reopenned after September 11th. Most likely, routine upgrades in capacity and regular equipment replacement meant that the rest of their system was relatively recent as well; the savings on running a modern mainframe over running a 1970s era one (in terms of maintenance, power usage, and space occupied for the amount of computational power) would pay for buying a new one.
Well, it finally happened today. I switched to Linux. This morning I was writing a corporate email to send to everyone in our Exchange network and after 2 hours of drafting, typing, composing and formatting.... Outlook crashed. I called our department IT tech to see if anything could be done to recover my email but the whole outlook .pst storage file was corrupted!!!!! I didn't want this to happen again so I popped in my knoppix disk I had at my desk and fired up Evolution and started redoing my work. I got my work done thanks to an emergency boot disk and I am in the process of downloading Debian to install on my work computer. Windows sucks, I can't wait to show this to more co-workers who can be more productive. I can stick it to M$ with every PC at work I can switch to linux. Plus tuxracer is really fun to play for breaks :-)