Slashdot Mirror


Linux From A CIO's Perspective

An anonymous reader writes "CIO.com has a story on Linux and OSS in the enterprise from the perspective of the CIO of Cendant Travel Distribution Services, Mickey Lutz. 'In the summer of 2003, Mickey Lutz did something that most CIOs, even today, would consider unthinkable: He moved a critical part of his IT infrastructure from the mainframe and Unix to Linux. For Lutz, the objections to Linux, regarding its technical robustness and lack of vendor support, had melted enough to justify the gamble.' His organization saved 90% in costs in so doing. Read on if you want to see how the top brass views OSS."

22 of 163 comments (clear)

  1. Difficult, but big payoff by AKAImBatman · · Score: 5, Insightful

    This pretty much sums it up:

    Lutz's IT group rewrote a complex, real-time airline pricing application that serves hundreds of thousands of travel agents around the world and that also acts as the system of record for all of United Airlines' ticket reservations. When this application came up on Linux, it proved to be so demanding--it handles up to 700 pricing requests per second--that it completely redefined Cendant's expectations about what it would take to get Linux to work. "We have broken every piece of software we've ever thrown at this platform, including Linux itself," says Lutz.

    With Big Iron you're paying a LOT of money. But you're not paying it for nothing. Big Iron will give you a lot of guarantees for stability, reliability, and thoroughput that don't exist on other systems. The key to this CIO's success is that he was willing to accept the challenges of doing Big Iron work on Little Brass systems. As long as you work all the details out yourself, this *can* work. (As Google has so eloquently proven.) The issue is that you're working without a safety net. If things go really wrong, there's no backup army of specially trained techs to run in and fix things. (And trust me, if you're paying enough money you'll have your own personal army of techs.)

    The upshot to all of this is that if the gamble pays off, it pays off in a big way. All that money you were spending for a personal army, plus some other company's R&D now goes into your own pockets. You don't get away scott free (someone has to maintain the systems), but you see your rewards. And isn't that what business is about? Taking risks and making profits? If you've got the infrastructure to go for something like this, then go ahead and grab fate by the balls. No one ever got anywhere in life by playing it safe. ;-)

    The "black box" of open source has transformed into something any CIO can appreciate: reliable performance and consistent uptime. The penguin can fly now.

    1. Re:Difficult, but big payoff by stefanlasiewski · · Score: 5, Interesting

      The issue is that you're working without a safety net. If things go really wrong, there's no backup army of specially trained techs to run in and fix things.

      Well, there is a backup Army, and it's you.

      Google can have a 4000-node Linux cluster because they have enough staff to maintain and optimize the system (Keep an eye on their job pages to get an idea).

      Google also has some highly specalized needs-- some machines only crunch data for the DB, other machines only serve webpages, etc. It's in their interest to optimize the Kernel, OS, Database & Web applications as much as possible. Take a tweak which gains a 1% performance gain, multiply that against 4000 machines, and it's quite an advantage.

      There isn't a vendor in the world that can totally support their infrastructure, so Google does it themselves.

      --
      "Can of worms? The can is open... the worms are everywhere."
    2. Re:Difficult, but big payoff by Anonymous Coward · · Score: 5, Funny
      Take a tweak which gains a 1% performance gain, multiply that against 4000 machines, and it's quite an advantage.

      Let's see . . . that's . . . [pencil scratching] . . . 1%! Amazing!

    3. Re:Difficult, but big payoff by AKAImBatman · · Score: 3, Interesting

      Well, there is a backup Army, and it's you.

      No, you are the front lines army. The backup army was the reason you were paying the annual fees. Without those annual fees, there is no backup army. i.e. If you can't get it right, there's no one else to come in and fix it for you.

      Take a tweak which gains a 1% performance gain, multiply that against 4000 machines, and it's quite an advantage.

      That's something of a straw man argument. If 3 Sun machines and 10 LinTel boxes have the same Flop capacity, then a 1% increase in either one will add up to the same increase in computing power. The key difference is that there are only three Sun machines to update.

      There isn't a vendor in the world that can totally support their infrastructure, so Google does it themselves.

      That doesn't mean that there couldn't be. Google made their choice to go with a large number of decentralized systems. It works for them and it works well. But they have to do everything internally *because* of that decision. Had they gone the EBay route, they would be able to get that backup army, but then they would pay for the priveledge.

  2. unthinkable? by delirium+of+disorder · · Score: 3, Insightful

    Moving desktops to linux might be considdered revolutionary, but this isn't. The big iron market of servers and HPC machines has really been dominated by linux for several years now.

    --
    ------ Take away the right to say fuck and you take away the right to say fuck the government.
    1. Re:unthinkable? by delirium+of+disorder · · Score: 4, Insightful

      I donno....google seams pretty reliable. Akamai web caching is designed to ensure online reliability (and is Linux based). The standard for Internet based systems that need long uptime and robustness is GNU/Linux. This is because the web as a popular medium is only about a decade old. The only reason why other organizations (banks and the military for example) that need reliable systems have not migrated to Linux is because they use legacy code that would be expensive to port.

      The particular solution the CIO in the article choose gambled with reliability because they used 144 separate servers in 12 clusters. Well implemented clusters of x86 hardware can run seamlessly, but individual machines are likely to fail. Redundancy should mean that a couple blown power supplies or corrupted disks a year is no big deal, but it's still a slight risk and a pain to fix. There are Linux solutions that run on larger machines. They could have replaced their four IBM mainframes with four Linux mainframes. IBM supports Linux on the zSeries mainframes (formerly called System/390, before that System/370, which was the successor to System/360..its about as traditional a mainframe as you can still buy). The top high end computers are Linux based. The top 3 most powerful computers in the world at this time, two IBM eServers and a Sgi Altix, all run Linux. Linux offers the most flexible, powerful, and reliable solutions out there.

      --
      ------ Take away the right to say fuck and you take away the right to say fuck the government.
  3. Linux beats Unix on cost by bedroll · · Score: 4, Funny

    The only thing that makes this news is that a CIO actually recognized it.

  4. Considered unthinkable by Anonymous Coward · · Score: 3, Insightful

    would consider unthinkable: He moved a critical part of his IT infrastructure from the mainframe and Unix to Linux.

    This is actually quite thinkable. Now if some CIO moved all his desktops to Linux, I would be impressed. Moving the backoffice stuff from expensive licenses of Unix and mainframes to Linux is a no-brainer.

    1. Re:Considered unthinkable by Decaff · · Score: 4, Insightful

      Moving the backoffice stuff from expensive licenses of Unix and mainframes to Linux is a no-brainer.

      No it isn't. There are many very high volume commercial and financial websites that use features of commercial Unixes, such as memory and resource partitioning, self-healing, fault management and very high scalability. Linux will certainly get all these at some point, but until then it is certainly not a 'no-brainer' to move. Even with smaller systems there are many applications that require certain Unix versions.

    2. Re:Considered unthinkable by Scurrilous+Knave · · Score: 3, Interesting

      I'm not sure how to respond to the "no-brainer" label. My development team just migrated our company's production software development platform from a network of aging HP-UX machines (which served us quite well in their day, don't get me wrong) to a Linux network. The new development tools run 20 times faster (that's the actual figure, not hyperbole), our experience over the past two years is that the Linux network is much more reliable, and the server hardware is simply an increment onto their existing Windows server lease and support contracts, not a whole separate costly item like the HP's were. I don't know the exact dollar figure it'll save them every year, but I know it's major. For us, the developers, it was definitely a no-brainer.

      But management fought us kicking and screaming every step of the way. They weren't really sure what they wanted, they just knew they didn't really want Linux. After repeated patient, detailed explanations of the cost savings and productivity benefits were met with skepticism and outright disbelief, we finally had to create an extensive Six Sigma analysis of the business and technical case for our choice. I can't fault them for wanting that, but once they got it, they seemed rather sorry they'd asked, because it supported every word we'd already said. For them, it was far from a no-brainer.

      Now, how they were using their brains, I cannot say. To us, it looked like a child screaming about getting a vital vaccination. But despite their grumbling and repeated shots of the stink-eye our way, they now have their new system in place and it works great. The other developers love what they see.

      Of course, our whole team is being laid off and all development is moving to India, but that's beside the point. (That was in the works long before we were even condsidering the migration. Honest!)

  5. I'm reading through this and it's interesting. by Anonymous Coward · · Score: 5, Interesting
    There are two things I find really interesting here:
    1. Vendor support. OK, so if vendor support has gotten better, then which vendors does this CIO recommend? I don't see which ones he used in the article, maybe I just missed something.
    2. This quote
      "Open source is propelling us to adopt Java and a new way of programming," he says.
      Should be a bit of a cluebat for both Sun and the Open Source extremists. Java and Open Source can be extremely good for each other, it's just that both Sun and the Open Source community need to learn to cooperate on practical matters with those whose ideological goals differ... unfortunately it seems that neither Sun nor the Open Source community is extremely interested in realizing how much they could benefit from the other.
    1. Re:I'm reading through this and it's interesting. by thammoud · · Score: 4, Interesting

      Most companies (including us) that use the popular Linux/Java combination do so without caring about any Open Source ideolgies. The combination is simply outstanding.

  6. Re:Well by Halvard · · Score: 4, Insightful

    Most don't come out of geekdom, rather from business school and worked previously marketing, sales, or some other management area. The don't have the knowledge or skill to be geeks and must rely on them. To make a move like this one, you must have good ones that you trust not just with the business but **with your career**. That's ultimately more important since your family depends on it.

  7. hey Mickey... by C0vardeAn0nim0 · · Score: 3, Funny

    some guy name "bill" called from redmond. he wants to explain you why linux is more expensive...

    --
    What ? Me, worry ?
  8. Small point of correction by Alphabet+Pal · · Score: 3, Informative
    Lutz was in command of the alternative to those bright, shiny websites: an expensive, aging global distribution system (GDS) called Galileo.

    Actually, all of those bright, shiny websites (Expedia, Travelocity, and Orbitz) rely on a GDS (Sabre, Amadeus, Worldspan or Galileo) to provide their content.

    --
    Because you can't spell "slaughter" without "laughter"
  9. Cost breakdown by alvinrod · · Score: 5, Interesting
    Mainframe: $100 million

    Unix: $25 million

    Linux: $2.5 million

    These numbers were taken from a table in the article. Interestingly enough, the cost if something does break favors Linux as well. From the same table we get that the mainframe solution consists of 4 IBM mainframes, whereas Linux and Unix solutions require around 144 servers for Linux and 100 - 120 servers for Unix. If the hardware goes to hell it's so much easier to replace the single bad part than a mainframe.

    Hopefully, more people will begin a transition to open source solutions when they realize it can be successful.

    1. Re:Cost breakdown by AKAImBatman · · Score: 3, Insightful

      If the hardware goes to hell it's so much easier to replace the single bad part than a mainframe.

      Not to detract from your point, but mainframes don't break as a single piece unless the machine blows up or is otherwise completely destroyed. Big Iron systems are designed with redundant *everything* including motherboards, CPU, memory, network cards, power supplies, and disk drives. If any one part fails, the system will route around it. The part can then be powered down and ejected from the machine. To bring it back up to full capacity, you simply plug in the replacement part and walk away.

      In that light, Linux system failures are actually going to be more difficult to repair. However, the cost of repairing a Linux system is far less (disposable box) despite the inherent difficulty. :-)

  10. Spread the word! by bogaboga · · Score: 3, Interesting
    [...] Mickey Lutz did something that most CIOs, even today, would consider unthinkable: He moved a critical part of his IT infrastructure from the mainframe and Unix to Linux. For Lutz, the objections to Linux, regarding its technical robustness and lack of vendor support, had melted enough to justify the gamble.'[...] His organization saved 90% in costs in so doing.

    Now, let's get prepared to rebut any Microsoft officials whenever they talk about the common "Total Cost of Ownership" as far as Linux is concerned.

  11. The risks and the rewards by AB3A · · Score: 4, Insightful

    Like most critics, I'm not good at leading large companies. But I know good leadership when I see it. This guy Lutz has his head bolted on right.

    The first thing most CIOs usually throw at their workforce is not to re-invent anything. If a product exists off the shelf at a reasonable cost, there are lots of disadvantages for taking the risk of inventing another one and few advantages if you succeed.

    However, most of us workers have known that the "big iron" mainframe technologies of yesteryear are starting to "rust." It's getting difficult to find technical help who understand this stuff reasonably well. That brings me to the second point: Follow the technology market. The people will be there.

    I suspect that in the not too distant future, many big-iron mainframers are going to be asking theselves whether the many millions they're spending are a good ROI. Open source databases and distributed computing are starting to look awfully attractive.

    It's scary from a CIO's position because the old systems are working, even if they're not well understood any more. They're leaping from the systems they know, toward a high cost potential boondoggle. This guy apparently knew how to hire and retain good technical help, he knew how to organize that help, and he knew how to keep them focused on the goal.

    Most leaders aren't that good. All too many businesses operate by habit. Only the red tape holds them together. Those organizations won't be making this leap until a certain critical mass has been reached to convince them one by one to make the effort.

    We should be doing everything we can to encourage others like Lutz to push these efforts. This is how you really evangelize Linux. And when all this is over, the desktop will be an afterthought.

    --
    Nearly fifty percent of all graduates come from the bottom half of the class!
  12. A Bold Move by endeavour31 · · Score: 3, Insightful

    An interesting read. But very interesting for what was left unsaid as well. There was a fair amount of pain associated with the switch - aggravated by rolling out a new application simultaneously. The slowdowns and the associated costs are glossed over but I wonder how the business side feels about this change to only 25% of the entire infrastructure.

    The time window seems fairly broad as well. No one disputes that lots of cheaper intel servers can do the same job as big iron. THe question is how many does it take and what happens with the applications involved.

    Quite telling is the comment that they needed every bit of support possible. Although it is great that one CIO bit the bullet here....there is an ominous side to this story which means that few others will follow suit.

  13. The main mistake is changing everything together by iabervon · · Score: 4, Interesting

    "In hindsight," says Lutz, "we shouldn't have tried to cut over to a new infrastructure at the same time we were deploying a new software application. It was too much at once."

    They found that their Linux servers couldn't support the new application they had deployed at the same time. That doesn't mean it's less capable than the mainframes they replaced: they didn't even try running the higher-load application against the mainframes.

    They should have first ported their servers to Linux on the mainframes, then switched them to Linux on clusters, then sent out new software that they could force back to the old behavior, then supported the new software in general.

    That way, they'd have been able to isolate the problems more easily (which really turned out to be that the new application generated extreme peak loads, and nothing to do with Linux per se, aside from that they managed to improve the Linux performance to deal with it) and keep things stable while they fixed the issues.

  14. No, Linux saved that guy's a$$. by khasim · · Score: 3, Funny
    Like most critics, I'm not good at leading large companies. But I know good leadership when I see it. This guy Lutz has his head bolted on right.
    I'll have to disagree with that. He made a good choice in going to Linux from Unix, but he did so is such a fucked up way that it was only Linux's technological goodness that saved him from being a poster boy for Microsoft's "Linux sucks" campaign.

    Here, from TFA:
    The decision not to focus more on testing came back to haunt them.
    The CIO decided not to TEST the system correctly?
    Frantic calls began coming in from some of the 44,000 travel agency locations in 116 countries that were unable to access Fares.
    Their customers cannot access their new Linux system!
    Lutz would not comment on the financial losses incurred by United or Galileo during the downtimes.
    They were LOSING money with their new Linux system.
    "In hindsight," says Lutz, "we shouldn't have tried to cut over to a new infrastructure at the same time we were deploying a new software application. It was too much at once."
    This guy made novice-level mistakes and it was only because Linux is so good that this became a huge success rather than a terrible failure.
    Rather than falling back to the old platform at the first signs of trouble and reworking the new one, the engineers always thought the answer was around the corner.
    You always have a back-out plan. Always.

    This guy took a huge risk ... screwed it up royally ... and was saved by IBM, Red Hat and Linux.

    And the Linux system STILL saves him $$$MILLIONS$$$ every year and OUTPERFORMS his old system.

    It's one thing when you're a genius CIO who plans and test for every contingency and deploys a working Linux system.

    It's a completely different thing when you don't BUT YOU STILL SUCCEED BECAUSE OF LINUX.

    This story is important because it shows the average CIO that, even if you aren't a genius and you DO make mistakes, Linux can STILL save you barrels of money and make you LOOK like a genius.