Putting Linux Reliability to the Test

Frank writes "This paper documents the test results and analysis of the Linux kernel and other core OS components, including everything from libraries and device drivers to file systems and networking, all under some fairly adverse conditions, and over lengthy durations. The IBM Linux Technology Center has just finished this comprehensive testing over a period of more than three months and shares the results of their LTP (Linux Test Project) testing."

Comments

USE BAD HARDWARE!

[superpulpsicle]

You want to put any OS to the ultimate test, you should run cheap generic hardware. I swear it's an industry conspiracy that generic parts struggle a boat load. If your parts don't come from the big boys (DELL, gateway, etc), you are likely going to see issues down the line.

Get some ECS motherboard, generic RAM... bang. You're in for the evening.

Re:USE BAD HARDWARE!

[spikev]

IMHO, it's the big boys that have the conspiracy to sell crappy hardware. Try performance testing almost any (PC Chips mobos don't count) custom system against a Dell with a similar hardware configuration, and you'll see what I mean.

I've done it with my ECS board with generic ram, and I came out on top.

It's the big computer makers that sell the cheap generic hardware. Try getting anything that's essential and non-OEM, hardware or software, to work with one of those boxes.

You don't trust Microsoft to evaluate Windows...

[davidstrauss]

Why do you trust IBM's Linux Technology Center to evaluate Linux?

Not bad

[changelingyahoo.com]

This is nice to hear, but it would be even more valuable if the same tests were performed on a variety of operating systems in order to compare the results.

Brian

Needs to be done independantly

[ducomputergeek]

Well I have some Karma to burn tonight. I don't mind that its the 2.4 KERNAL even though 2.6 is ready. Why? We never put anything on our production server that hasn't been out for at least 6 months with exception of security upgrades.

Second off, If this were M$ testing 2k3 and publishing the paper, everyone here would be crying foul. But because its, "Linux" it must be 100% unbais and true.

I've been using Linux for 8 years now including under high stress enviroments, 3d graphics rendering mainly, and from experiance I have see very good things from Linux. We have had software glitches before, but the core software maybe has caused 3 - 5% of our downtime. Over 70% of our downtime involves human error and about 25% of failures are due to hardware giving out.

Still what my customers are wanting to see isn't benchmarks as "So easy Grandma could use it" in Linux. While the people in the datacenters want to know how well Linux will bear under a load, most end-users and SMB's don't need to worry about it, they just need something easy to use that works.

Re:Needs to be done independantly

[haruchai]

Well, their bias is clearly stated on their web page- this is the Linux Test project - which is dedicated to evaluating the capabilities and limits of Linux.

They aren't making a comparison to other OSes or saying that Linux is more suitable than such-and-such operating system; just that it is suitable for particular tasks or environments.

A comparison between different OSes should be carried out by an independent testing facility but, in this particular case, I don't see anything
wrong with their modus operandi.

Re:Needs to be done independantly

[antiMStroll]

Second off, If this were M$ testing 2k3 and publishing the paper, everyone here would be crying foul. But because its, "Linux" it must be 100% unbais and true.

An ironic assertion regarding bias. IBM isn't the author of Linux or any of its tools, add-ons, servers, etc. as Microsoft is of 2k3 and its support software. Microsoft also has a long and distinguished history of FUD. IBM doesn't have anywhere near the historical attachment to Linux that MS has to Windows, and IBM hasn't been caught lying about it yet. It would be irrational to treat the two equally at their word.

Re:Needs to be done independantly

[Samrobb]

Microsoft also has a long and distinguished history of FUD.

Pretty amusing that you would say that, considering the origin of the term:

Defined by Gene Amdahl after he left IBM to found his own company: "FUD is the fear, uncertainty, and doubt that IBM sales people instill in the minds of potential customers who might be considering [Amdahl] products."

Not that I disagree with your assertions - IBM doesn't have near the same ties to Linux that MS has to Windows. But it's amusing to see how much the technological landscape has changed, that a term coined to describe IBM can now be used to (in some sense) defend it.

Re:You don't trust Microsoft to evaluate Windows..

[davidstrauss]

Because what would they have to gain by lying?

They have much to gain: more corporate customers and more respect and funding by greater IBM. Just because IBM supports Linux doesn't mean its motives are pure (not financially driven). Another reason for bias is the division also stood to have huge setbacks if the tests were unfavorable. How could they justify expansion and better funding if their previous statements about Linux being enterprise-ready were unfounded?

Re:You don't trust Microsoft to evaluate Windows..

[Surazal]

So, ya reply to one point but ignore the rest? I think his (ultimate) point is valid. If the test was rigged, the folks involved with developing the kernel would catch on and take IBM to task for fudging the results. No, I'm not talking about the Slashdot/Fark crowd. I'm talking about REAL developers.

Also, Linux has weathered some unfavorable (and honost!) critiques before. Linus Torvalds said it best when he said (and I paraphrase since I am too lazy ATM to look up the actual quote) that it doesn't matter if there's negative publicity in the press about Linux. It just meant he got his bug reports from the Wall Street Journal as opposed to the regular kernel mailng list.

Re:Linux 2.4.19-ull-ppc64-SMP (SLES 8 SP 1)

[RALE007]

You're annoyed that they released the results for a 90 day stress test on a 2.4.x kernel, but not a 90 day stress test on a 2.6.x kernel? The 2.6.x kernel has been out for nine days. How would they have any results on the new kernel? By sending results to us using the Way Back Machine?

I think IBM used SuSE instead of Redhat because IBM Global Services and SuSE have been partners for almost two years.

Maybe you should stop hmmmmm'ing about these great mysteries and start googling.

Diagnosing software vs. hardware is easy.

[Inoshiro]

Hardware fails in random patterns (bits flipped by beta radiation, for example), software fails the same way in controlled instances (the same flawed logic fails the same way each time).

So when you run a test 5 times, and you get 5 results, the hardware is broken. When you run the same test 5 times, and it gets to the exact same point before sig11ing, you have a software flaw.

This is also why you do multiple tests to ensure you're getting an accurate picture of what's going on (flawed or not).

Re:Diagnosing software vs. hardware is easy.

[jmv]

software fails the same way in controlled instances

That's true... in theory. In practice, there are many ways software can fail in random (in the weak sense) ways. Many of these are related to timing. For example if you have many threads and fail to lock things properly, the result will depend on when the tasks are preempted. You can also have different results because of the way the interrupts (disk, net, ...) happen. There's also the not-initialize type of problem where the behaviour depends on whatever was there in memory before. There are probably many other ways for software to fail at random, including obscure combinations of events.

I'd say that the only kind of software that can't fail randomly is single-threaded and doesn't rely on any input other than regular files (and even then I'm not sure it's enough).

Here goes

[floydman]

FTA
The tests demonstrate that the Linux system is reliable and stable over long durations and can provide a robust, enterprise-level environment.

Ok, now i dont mean to troll here, so mod down if you wish, i really dont care.... BUT...
I am a linux user/programmer/lover for the past few years now, and i wanna see a company that is not SO IN LOVE with linux say what have just been said by IBM above.
In other words, i dont want to see companies who sell Linux, or who have benefit in selling Linux praise it. Does any one of you know of someone who fills in these criteria. Sun for one is not very fond of Linux, nor is MS ofcorse (despite the fact sometimes i doubt they have code in their stuff from Linux...)...to make a long story short
It would be really nice if such a judgment came from someone else besides IBM/REDHAT/ORACLE...

Scaleable.... really?

[tonyr60]

"The Linux kernel properly scaled to use hardware resources (CPU, memory, disk) on SMP systems."

Sorry, but how can the scaleability of the CPU resource be proven on a 2 CPU system? Show incremental results on 1, 2, 4, 8, 16, 32 etc. etc. and then CPU scaleability may be proven.

This is NOT an anti-Linux troll, rather the evaluation needs to justify it's outcomes or it starts to look like something from a company starting with M.

So what ?

[Alain Williams]

All that this shows is that a Linux based system works in the way that it should. Would you expect anything else if you ran your: TV, central heating, ... for a long period ?

The trouble is that, after a period of increased stability in the 1980's, in the last decade people have come to expect that computers fail, and they wonder with amasement if they don't.

OK: 30 years ago I remember it being a good day if the mainframe stayed up 12 hours. But things have moved on, today you expect your: MVS, VMS, Unix, Linux machine to stay working. The only OS vendor who's products have not matured is the one in Redmond - largely because of rampant infestation with new features.

The above is not intended to belittle the fantastic efforts of all those involved.

The problem lies not with reliability

[Lord Kano]

but with package and dependency madness.

I couldn't tell you the number of times I tried to install something and it fails because I was missing "X-Widget-2.41.so.1", so I try to install that "X-Widget-2.41" package and the "X-Widget-2.41-devel" package and they fail because they are missing several other depends as well.

Linux stability is fine. The GNU software stability is fine. We need a better way to install and maintain software.

LK