Linux Kernel Gets Fully Automated Test

An anonymous reader writes "The Linux Kernel is now getting automatically tested within 15 minutes of a new version being released, across a variety of hardware and the results are being published for all to see. Martin Bligh announced this yesterday, running on top of IBM's internal test automation system. Maybe this will enable the kernel developers to keep up with the 2.6 kernel's rapid pace of change. Looks like it caught one new problem with last night's build already ..."

Comments

Would be nice

[Anonymous Coward]

Wouldn't it be nice if Linux had an automated test platform for the kernel?

now all we need is automated....

[3seas]

code generation...

Re:now all we need is automated....

[caluml]

Actually, that could be done, could it not? Throw in some random functions, if/while/do loops in, return random variables, etc. It could create some funky new software. :)

Re:now all we need is automated....

[Baal Sebub]

I already got 1 million monkeys in my basement working on it.

Re:now all we need is automated....

[maxwell demon]

No problem. The following is an automated code generator. It generates a hello world program in C and writes it to stdout. (untested)

#include <stdio.h>
int main()
{
char const* program_pattern = "%s%s";
char const* include_pattern = "#include <%s>\n";
char const* function_declaration_pattern = "int %s(%s)";
char const* function_definition_pattern = "%s\n{\n %s;\n}\n";
char const* print_pattern = "printf(%s)\n";
char const* string_pattern = "\"%s\"";

char const* stdio_header_name = "stdio.h";
char const* main_function_name = "main";
char const* main_arguments = ""; // we don't read command line arguments
char const* output_string = "hello world!";

char string[15];
char print[23];
char main_decl[11];
char include[19];
char main_func[42];

sprintf(string, string_pattern, output_string);
sprintf(print, print_pattern, string);
sprintf(main_decl, function_declaration_pattern, main_function_name, main_arguments);
sprintf(main_func, function_definition_pattern, main_decl, print);
sprintf(include, include_pattern, stdio_header_name);
printf(program, include, main_func);
return 0;
}

Re:now all we need is automated....

[Curtman]

Actually, that could be done, could it not?

Apparently it works for Samba. :)

Re:now all we need is automated....

[Anonymous Coward]

Its called lisp ;-)

Re:now all we need is automated....

[petermgreen]

code generation is good for repetitive stuff especially if your language doesn't have much in the way of a built in preprocessor

say for example producing similar load on demand wrappers for a load of functions in a dynamic library.

p.s. /. seems to be restricting me to one post every 15 mins right now dunno why (the error says Slashdot requires you to wait 2 minutes between each successful posting of a comment to allow everyone a fair chance at posting a comment.

It's been 14 minutes since you last successfully posted a comment)

Re:now all we need is automated....

[Curtman]

code generation is good for repetitive stuff especially if your language doesn't have much in the way of a built in preprocessor

There's a fair bit of repetitive code in the kernel. I had to do some hacking to make some RS-422 cards we had work properly, and found that a lot of the char drivers especially contain very similar code, and structure. Code generation might help with older drivers that nobody cares about until they break. They tend to rot from the looks of things.

Re:now all we need is automated....

[MemoryDragon]

Whats so funny about it, code generation is used left and right in modern projects, this stuff is great to shift the grundwork away from the developers and not having to go into outsourcing hell.

Re:now all we need is automated....

[northcat]

Why hasn't anyone mentioned lex or yacc yet?

Re:now all we need is automated....

[kurzweilfreak]

Beautiful. Scale it up to output a new Linux kernel.

Re:now all we need is automated....

[pchan-]

holy crap, remind me to never hire you. if you know exactly what the string is going to look like, why didn't you just write it? if you expect it to change, why did you hard code the length values into the buffers?

#include <stdio.h>
int main(void)
{
puts("#include<stdio.h>\n"
"int main(void)\n"
"{\n"
" puts(\"hello world!\");\n"
" return 0;\n"
"}");
return 0;
}

Re:now all we need is automated....

[Anonymous Coward]

Automated code generation already exists...

It is called a compiler.

Re:now all we need is automated....

[Anonymous Coward]

Let's see if the same happens to me. (Posting AC since it's WAY offtopic).

The current time is 20:51:30

Re:now all we need is automated....

[Anonymous Coward]

The time is now 20:53:45

Re:now all we need is automated....

[Anonymous Coward]

Holy crap, remind me to never get hired by you.
Criticising people who "do it wrong" instead of helping them...no thanks.

Re:now all we need is automated....

[jrockway]

It was a joke, dumbass.

If you're going to used fixed-length buffers, though, at least use sNprintf!

Re:now all we need is automated....

[Anonymous Coward]

were they $0.05 each?
A million dead monkeys would be quite a problem.

Re:now all we need is automated....

[Anonymous Coward]

I do not have the sameproblem myself, but you're not the first one I've seen posting about this.

Re:now all we need is automated....

[3seas]

I think the kick in teh butt humor found in this is that of the computer auto generating the code, auto compiling it and auto testing it and regenerating code for improvement based on test results, ... loop it Johnny Mnemonic... uhhh ertrrr Neo..

Being one fully aware of the possiblities of auto coding or using code generators, both of which exist today in one form or another, just not so completely available wide scope on much of any user/consumer platform..

I was being serious but certainly found the humor others did.

Re:now all we need is automated....

[GraemeDonaldson]

Indeed. Longhorn, anyone?

Re:now all we need is automated....

[GraemeDonaldson]

*whoosh*

That was the sound of a joke going over your head.

Re:now all we need is automated....

[GraemeDonaldson]

*bzzzzt*

A compiler _translates_ human-readable code into something the machine can understand, it does not generate code.

Thanks for playing Slashdot, please try again soon.

Why has it taken so long?

[Beatlebum]

Why has it taken so long?

Re:Why has it taken so long?

[Anonymous Coward]

Bitkeeper.

Re:Why has it taken so long?

[Anonymous Coward]

How many tests have your written? That's why.

Re:Why has it taken so long?

[Anonymous Coward]

How many tests has his what written?

Re:Why has it taken so long?

[teh_cn]

mod me troll, but (free)bsd had this for years and not only for the kernel, but for world, too.

Re:Why has it taken so long?

[Uerige]

Because you didn't do it.

Re:Why has it taken so long?

[Anonymous Coward]

OpenSolaris will have had this for years, too.

Re:Why has it taken so long?

[ikewillis]

Good question, especially considering FreeBSD Tinderbox has been doing this sort of thing for years, and not just with the kernel but with the entire base system.

Re:Why has it taken so long?

[VStrider]

They had to. There isn't anyone left to do the testing.

Re:Why has it taken so long?

[Anonymous Coward]

This is more than just what tinderbox does. Tinderbox only does compilations.

Re:Why has it taken so long?

[putaro]

Real men don't test their code - they just post it on the Internet and let everyone else do it for them.

Within 15 Minutes? WTF

[LCookie]

"The Linux Kernel is now getting automatically tested within 15 minutes of a new version being released"

Would be much better to test it BEFORE a new version is being released, otherwise this is completely useless...

Re:Within 15 Minutes? WTF

[Anonymous Coward]

Why do you think they call it the 'bleeding edge'. Look at all that red.

Re:Within 15 Minutes? WTF

[Thing 1]

Great idea. You should ask IBM to integrate their test platform into Linus' processes. He might be dubious after BitKeeper (that idiot) about another company helping him, but in this case I think it's a great idea.

There may be (and probably are) other test beds out there, testing releases. It would be better for Linus (and the world) if he could release already-tested code to the world, instead of having the world duplicate all the testing effort, and IBM seems like a perfect solution.

Re:Within 15 Minutes? WTF

[DigiShaman]

Sounds like the solution to this problem is clear. Always use the second to latest kernel released. Stay away from the new one untill it's fully tested to your satisfaction.

Re:Within 15 Minutes? WTF

[doshell]

"Release" in the open source world has a broader sense than in commercial software. In open source not all "released" versions are meant for general public consumption; they include unstable versions targeted mostly at developers, so that severe isues can be detected and patched quickly.

Taking this into account, I believe this is meant to catch bugs mainly in nightly (unstable) builds and release candidates, not in "final" versions (those should, at least in theory, have no serious bugs left around as the latter have already been eradicated from release candidates).

Re:Within 15 Minutes? WTF

[oxfletch]

I automatically test every nightly -git snapshot release, so it's fairly well tied in anyway. This also means my heaviest usage of our machines is at night, when most of the (US) developers are asleep.

So it's fairly well tied in already ... and the whole -rc cycle should enable us to catch a lot of stuff.

Re:Within 15 Minutes? WTF

[Anonymous Coward]

A lot of the 'releases' are also point releases and -footree releases. Most of what's tested is probably development kernels anyway; before or after doesn't completely matters in those case.

Same goes for -rc and -bk releases (though I guess neither really exists anymore).

Think of it this way: x.y.z is tested by all the x.y.z-rc. releases coming before it.

Re:Within 15 Minutes? WTF

[Metteyya]

because they are nightly builds, that is - versions with applied patch, but untested yet.

Re:Within 15 Minutes? WTF

[DegeneratePR]

In any case, most people, especially in mission-critical processes, don't compile a new kernel as soon as it's released. Myself, I try kernels after a while, when no major issues are found. Even then, I test them out first in different test machines. So 15 minutes before, 15 minutes after, it's all the same.

Re:Within 15 Minutes? WTF

[ideut]

Which would mean, for the last several 2.6.x releases, that you are always using a version with a known root hole in it. Here's an idea: use your vendor's QA-tested kernel that they package for your distribution.

Re:Within 15 Minutes? WTF

[netdur]

> when most of the (US) developers are asleep
as far as I know (US) developers sleeping during the night time in... China

Re:Within 15 Minutes? WTF

[digitalunity]

Ummm...

If everyone did this, the newest kernels would never get tested. I think it is important that we have a diverse range of users using new, almost new, and older but well tested kernels.

Re:Within 15 Minutes? WTF

[fbjon]

Aer you a (US) developer, then? And what the hell is a (US) developer anyway, that's a weird-ass smiley if I ever saw one.

Re:Within 15 Minutes? WTF

[Anonymous Coward]

For those in the air force, a (US) developer is an "UnServicable" one. Ie, next to useless.

Re:Within 15 Minutes? WTF

[Thing 1]

But it's not all the same, though. Once it's "blessed" by Linus, it's released. If he had access to the test machines prior to releasing it, he could release higher-quality code.

And since the entire test run only takes 15 minutes, IBM (and the world) would benefit from allowing him multiple tests per release.

Question:

[bogaboga]

How were the previous kernels being tested? Were sources for improvement/change/modification, bugs and areas requiring refactoring being discovered by chance?

Re:Question:

[Anonymous Coward]

" How were the previous kernels being tested?"

Hey guys, new kernel is out, bang away at it and let me know what you think.

Re:Question:

[Anonymous Coward]

" How were the previous kernels being tested?"

You basically use the bleeding edge kernel, while doing other stuff on your computer.

Re:Question:

[ignorant_coward]

After a new kernel was released, power meters on mothers' basements everywhere saw a little blip. Add up all these blips, and you get a (somewhat) tested kernel.

Re:Question:

[ImaLamer]

In Soviet Union Kernel Tests You!!!

Re:Question:

[steve_l]

I thought it was "hello, here is a new release of fedora for you to install..."

Re:Question:

[xtracto]

So they where using the thousand moonkeys approach uh?

How much testing?

[anthony_dipierro]

This is good, and long overdue (I'm surprised it hasn't been around for years), but just how much testing is being done? Compiling? Booting? Or are there actual functional and reliability tests which are being performed?

Re:How much testing?

[oxfletch]

Compiles, boots, runs dbench, tbench, kernbench, reaim, fsx. If one test fails, it'll highlight it
in yellow, rather than green or red. I have a few of those in the internal tests, but not the external set.

This is only the tip of the iceberg as to what can be done. We're already running LTP, etc internally, and several other tests. Some have licensing restrictions on results release (SPEC) ... LTP is a pain because some tests always fail, and I have to work out the differential against baseline. Will come later.

What took so long

[Timesprout]

Most projects of any complexity use automated continuous build and testing as a standard development practise.

Re:What took so long

[Rakshasa Taisab]

They've been doing that the whole time, they call them "users".

Maybe...

[ratta]

automated performance regression tests may be useful too.

Re:Maybe...

[oxfletch]

The results are all there if anyone wants to play with them. Go to the results matrix, and click on the numerical part of the green box. Pick a test, and drill down to the results directory.

The numbers are there, it's just a question of drawing graphs, etc. I have some for kernbench already, but I'm not finished automating them. If anyone wants to email me code to generate them from the directory structure published there, feel free ;-) Preferably python or perl into gnuplot.

Re:Maybe...

[Nutria]

Instead of just reading a bunch of complaints, let me be 1 Slashdotter to thank you for your efforts.

It's too bad the Stanford Checker can't be integrated into your system.

This is awesome

[jnelson4765]

But it can't catch everything - the 1394 bus was screwed in 2.6.11. There are a lot of regressions that show up - and even that healthy cluster of systems will not show every problem.

Sound issues? Older network and SCSI cards? There are a lot of drivers that break, and no one notices it because there is nobody with the hardware testing the -rc or -mm kernels.

Wouldn't it make more sense to package these tools for someone to install on their collection of oddball equipment, and assist in the debugging/testing?

Where's the ARM, MIPS, and SH?

Re:This is awesome

[Meshach]

But it can't catch everything...

But that is not the point of automated testing. As a member of a qa team who is developing automated tests I get comments like that every day

Automated tests are not intended to catch everything or test strange permutations of pre-conditions. There purpose is to provide a mechanism for verifying that a build satisfies the basic requirements of the project.

More exotic configs need to be tested manually as usual but automated tests can provide a "failsafe" just in case a basic part of the build is broken.

Re:This is awesome

[zappepcs]

I agree with jnelson4765, new buids would be well served to be tested on a great many machines with a wide variety of hardware setups.

Who should map the hardware testing platforms? I don't know, but I do know that if the new kernel builds are tested for a generic group of hardware and released, then other testers report on their tests using hardware X, you would end up with a relatively quick listing of a new build against many variants of hardware. Published correctly, it would allow people to search for problems regarding the new build and any particular hardware that they might be using.

That should allow reasonable release schedules and capture errors with older/arcane/little-used hardware. I think that would be quite acceptable.

As for ARM and other platforms that are not exactly mainstream, they could also be in the second round of testing. If an ARM supplier wanted to be certified in the early build testing, they could support that with hardware and other resources, thus pushing certification earlier for their hardware. Even MB or HD manufacturers can support the testing with their hardware for early certification, so that it is a community effort rather than just that if a small group of testers.

All together, I can see the process as quite workable.

Re:This is awesome

[Cylix]

Unfortunately, organizing that kind of odd ball testing would be a management nightmare unless you want to go out and collect all of the hardware. Remember, some people do post patches and whole driver releases without stepping inside of the kernel team's realm.

The only real way to automate something like that would be a dummy load facility. Some software which would emulate the hardware being in place. Something conceptually similar to that effect anyway.

So then, for every driver for a device, you have a software kit which tests the driver to see if its functioning. I have no idea how practical this is as I've never really done any serious kernel hacking.

Re:This is awesome

[Nutria]

Where's the ARM, MIPS, and SH?

IBM doesn't sell any ARM, MIPS or SH-based systems. So, they don't test them.

The Debian buildd system is an automatic building and semi-testing system for, of course, all the archs that Debian supports, and that includes ARM, MIPS, and SH.

Re:This is awesome

[team99parody]

Wouldn't it make more sense to package these tools for someone to install on their collection of oddball equipment, and assist in the debugging/testing?

That's how the PostgreSQL build farm works. People with wierd hardware apply to be added to the automated test farm. ARM, MIPS, PARISC, Alpha, PowerPC, Sparc, etc. are all represented well in the postgresql automated tests.

Re:This is awesome

[Mr. Underbridge]

Automated tests are not intended to catch everything or test strange permutations of pre-conditions. There purpose is to provide a mechanism for verifying that a build satisfies the basic requirements of the project.

Isn't that what a compiler is for? ;)

Re:This is awesome

[rapidweather]

Sound issues?

That's one of my main problems as an end-user tester of livecd linux distros. I do have oddball equipment, to be sure.

Re:This is awesome

[Anonymous Coward]

It just shows that the "basic requirements" are set far, far, far too low.

At the moment I think the biggest problem with Linux is constantly breakin USB (or firewire, as with the parent) drivers, so (automated) tests are a must.

And binary ABI for drivers, but unfortunately that will not happen.

Re:This is awesome

[Meshach]

Isn't that what a compiler is for? ;)

I would have to respectfully say no.

A compiler only will identify syntax errors in the code. It will tell if the program will run. It will not tell you that it will produce the correct result.

The types of errors that a script can test for are (and there are way more):

• Are the required fields available in a window
  
• Are the bounds correctly enforced in a field
  
• Are values in fields retained
  
• ...
  

Compiling is one step to make sure a program is correct. It is only one step though...

Re:This is awesome

[Mr. Underbridge]

Meshach, meet joke. Joke, Meshach.

Re:This is awesome

[Meshach]

My applolgy

Nice to meet you

Re:This is awesome

[MegaFur]

But it can't catch everything

Who cares? It doesn't have to have 100% hits & correct rejections, 0% misses & false positives to be useful. As long as it can get a lot of hits and correct rejections, and not very many false positives or misses it will still be useful.

ARM Linux has something similar

[kyllikki]

ARM Linux has had something similar in Kautobuild for some time.

Although the testing and building is limited to the ARM platform.

The site also has a whos who thats worh looking at ;-)

Presumably...

[Kjella]

...the cross-platform, cross-hardware part? Setting up one machine to build automatically is easy. Setting up a whole bunch of them (and all unique, read administration nightmare) and tie them together to a system, that's quite a bit of work.

Kjella

Re:Presumably...

[oxfletch]

Indeed. The automation system I wrote is just a wrapper around an internal harness called ABAT that has a massive amount of work behind it. If systems crash it can detect that, power cycle them, etc.

Going from 90% working to 99.9% working is frigging hard. I had all this working 3-6 months ago, but the results weren't good enough quality to be published. Several people internally put a massive amount of work into improving the quality and stability of the harness.

Re:Presumably...

[Anonymous Coward]

so um... you're getting paid to devise more effective computer BDSM techniques. where do i sign up?

"improving the quality and stability of the harness." ... was leather involved?

Re:Presumably...

[TCM]

...the cross-platform, cross-hardware part?

It's magic! A single script and I can build a complete operating system for a big-endian 64bit architecture on a 32bit little-endian architecture, or any of the other 48 supported archs. More than that, I can build a complete NetBSD for any arch on any halfway POSIXish system.

build.sh bootstraps its own contained build utils (compiler, binutils et al) and builds the system with that. You can even build the complete system as non-root and get tarballs that you can use to install a complete system.

To think that my weak SPARCstation 2 should build its own system would be madness. Instead, I just run build.sh -m sparc distribution on a dumb, fast i386 box and have a world in an hour instead of week(s).

Oh, and NetBSD "feels" the same on any arch, no administration nightmare, no matter what arch you run it on.

Just FYI.

Re:Presumably...

[duffbeer703]

We've been playing with some IBM tools at work that automate server setup and provisioning... its pretty amazing stuff.

You can basically retask servers in something like 10-60 minutes depending on what you are doing, and its a completely automatic process.

Re:Presumably...

[Nutria]

1. How long did it take you to create+debug+tweak that script?
   
2. How much testing does it do, other than "compile +link"?
   

Re:Presumably...

[TCM]

1. The script is part of NetBSD.
2. http://cvsweb.netbsd.org/bsdweb.cgi/src/regress/ is supposedly used for regression testing. Ask a developer, I'm just a user :)

Re:Presumably...

[Bob_Robertson]

I don't remember who said it first:

The first 90% takes 10% of the time.

The last 10% takes 90% of the time.

I expect one could substitute "money", "labor", "effort" for "time" in the above.

Bob-

Re:Presumably...

[brettper]

I thought it was more like

The first 90% takes 90% of the time.

The last 10% takes the other 90%

but I may be optimistic

Re:Presumably...

[Knetzar]

It's generally known as the 80/20 rule. 80% takes 20% of the effort, while the other 20% takes 80% of the effort.

The idea is the same though.

Re:Presumably...

[Anonymous Coward]

No it's a joke...when 10% of the *actual* project time has elapsed, you *think* you have 90% of development done. But it turns out that "just 10% left" actually takes 9x longer than the "90% already done."

Re:Presumably...

[Anonymous Coward]

Static and dynamic analysis are fast becoming standard practices in any large software engineering project. IBM is working on many such projects, and is leading the way particularly in static analysis. Although I can't really comment (I work for IBM on such things) on specifics, IBM is doing a lot of work to make sure AIX stays ahead of Linux in reliability through improving code quality. Of course, Linux and AIX development are in separate buildings since the SCO thing, but I'm glad that the Linux people in IBM are improving their QA practices as well.

Related projects at OSDL

[anandpur]

Related projects at OSDL
http://osdl.org/projects/26lnxstblztn/results/
http://developer.osdl.org/cherry/compile/

News Flash

[sirReal.83.]

Red Hat (and probably Novell/SuSe, since they use over one thousand kernel patches) runs a myriad of tests on each of its own kernel builds nightly - and has been doing so for years. On more than just the 3 architectures covered by this test.

That said, pushing tests upstream is a great idea. Just not revolutionary or anything.

Re:News Flash

[Anonymous Coward]

Those internal tests could not keep pace with the rate of development of the Linux kernel however. This testsuite will test _every_ kernel release instead of just the few cherrypicked by the big vendors.

Re:News Flash

[bruthasj]

News Flash #2:

Redhat has several engineers that *are* upstream.

Re:News Flash

[sirReal.83.]

True. er... your point?

Re:News Flash

[bill_mcgonigle]

Man, I wish they'd test Fedora kernel releases on their test farm. Of a dozen different machines I've run 2.6 Fedora kernel releases on, I've lost 1394 on one, USB on another, the hardware clock, on a third, parallel port probing on the third, serial ports on a fifth, and the Compaq Smart Array on the sixth.

The other six machines seem OK. But that's a 50% buggered rate from various flavors of 2.6 upgrades, mostly from nightly 'yum update's. These are all IBM, Compaq, HP, and Dell machines, so somebody's test farm would be likely to catch these.

Long uptimes

[rice_burners_suck]

This is a very smart system. The Samba team uses something very similar. The key to finding regressions with this method is to create tests for every piece of functionality, and to integrate it with the rest of the testing suite, so that each function of the kernel will be continuously tested. For new features, it is preferable to create these tests as the features are being coded. For existing millions of lines of code, it is necessary for some brave souls to go in and create these tests.

I hope they are using code from the Linux testing suite. That piece of work has already formed a nice set of tests. Also, I hope that the kernel is automatically built with many different combinations of options. And with time, I hope this will become better. The more tests, with the more hardware configurations, with the more kernel configurations, with the more types of input data (including many imaginative forms of incorrect input data to test that the kernel handles it gracefully and thwarts attacks based on such methods), the better quality we will have in the kernel, and it is likely that Linux will be unmatched in quality, stability, efficiency (well, maybe not efficiency necessarily), and long uptimes.

through the looking glass...

[moviepig.com]

With an automated test suite, what happens when a class of bug is discovered to be untested-for? Presumably, the suite is modified to detect it. Then, is the resulting new suite itself subjected to an automated test suite? And, then...[divide-by-zero error...]

Re:through the looking glass...

[oxfletch]

There is indeed an internal self-test suite on the harness. It's not desperately sophisticated, and I wouldn't dare show it to anyone ;-) However, it does catch a lot of stupid bugs. It requires some manual intervention/inspection to work.

Plus, there's a separate development grid where we test new test-harness code before it's put onto the
production grid.

Re:through the looking glass...

[sirReal.83.]

divide by zero? what kind of crazy hacks did you put in that algorithm, boy? ;)

Re:through the looking glass...

[EvanED]

You're not looking at a divide-by-zero error, but a stack overflow from the infinite recursion.

Re:through the looking glass...

[moviepig.com]

You're not looking at a divide-by-zero error, but a stack overflow from the infinite recursion.

You're right, I made a mistake. I shall modify my test suite forthwith... [divide-by-zero error]

Does this mean...

[blixel]

Does this mean we'll get back to 2.6.x releases? Instead of new version of 2.6.x being released as 2.6.x.x every third day?

NOT FUNNY: Chinese Military Software

[Anonymous Coward]

We should consider keeping the automated test technology under the label, "top secret". The Chinese are aggressively trying to modernize their military. Computer software for coordinating the various pieces of battlefield hardware is integral to Chinese plans for the next-generation of warfare.

One downside

[Anonymous Coward]

One downside is that people can become sloppy if they rely too much in the test suite caching all bugs (it obviously cannot).

Anyway, I think that if put to good use it can be an step forward.

Regarding sharing code with other test projects... I hope they do not! What we need is as much different tests and testing methodologies as we can generate. The more the better, so let's go invent a better wheel!

Safety issues

[DruggedBunny]

Martin Bligh announced this yesterday, running on top of IBM's internal test automation system.

Hope he doesn't fall off and hurt himself.

Re:Safety issues

[rbarreira]

Geez man, you made my day... Or minute, at least :)

cool to see this publicly announced

[emmastrange]

I got to work on part of this system, which IBM calls Autobench, for my senior project at PSU. The system is a highly configurable framework which can download, compile, and run various benchmarks and profilers (for example while compiling a kernel). Its all centrally administered, so IBM can run a battery of tests on a variety of different machines at once.

I think Martin Bligh said that IBM has been using this for a while now, automatically downloading kernels upon release and testing them. The new thing is the regression matrix which is posted to the web. Very cool.

Re:cool to see this publicly announced

[Anonymous Coward]

You must mean Portland State University where there are NO grammer or English requirements! (UNST doesn't count)

2.6.12 on amd64

[scharkalvin]

needs work! The latest builds all failed!

Re:2.6.12 on amd64

[StupidKatz]

Considering that 2.6.12 hasn't been released yet, it just might be the case that they are still, oh, I don't know, working on it?

Re:2.6.12 on amd64

[Anonymous Coward]

Considering the VIA chipsets have cold-boot failures that weren't really fixed in 2.6.10, I'm surprised amd64 boots at all!

Call me cynical. My SK8V motherboard with ECC memory never boots on the first try. It always hangs at a random point in the kernel and needs a reboot before it will boot fully.

Yikes!

[Anonymous Coward]

Compiling is considered a test?

Re:Yikes!

[Anonymous Coward]

I remember when "It compiles. Ship it!" was a joke about Microsoft. Now it's codified as the Open Source Development Methodology.

Re:Yikes!

[anthony_dipierro]

Testing if it compiles on a certain platform is a test, yes. It's by no means the sole test which is sufficient to ship a product, but I'd definitely call it a test.

Re:NOT FUNNY: Chinese Military Software

[Anonymous Coward]

Let me guess... You're American?

"The least they are required to do"

[Anonymous Coward]

IBM's efforts vis-a-vis Linux are a real eye opener when you look at the scraps that Apple gives back to the open source community after taking so much.

Re:"The least they are required to do"

[Anonymous Coward]

Apple gives back plenty.... to the GPL licensed gcc compiler.

User Mode Linux builds

[Anonymous Coward]

I hope all this translates to easier compilation of linux. I had worked with User Mode Linux a while back, and it takes a trip to hell and back to compile.. you need just the right setup of gcc, config files to get it to compile, which took me forver to get right, and honestly a waste of time.

Linux enters the world of QA 101!

[mrkitty]

Years later and finally it is getting some *basic* QA testing done! What will they think of next!

Re:Linux enters the world of QA 101!

[Anonymous Coward]

Every good distribution is already doing this, and they all benefit from each others work.

If you want a rock solid system, why are you running a vanilla kernel? Grab a RHEL or a SLES kernel for that.

Re:Linux enters the world of QA 101!

[SlashMaster]

I'd expect the community to start advocating unit testing, an agile development practice, at some point to increase the reliabilty of code before it is even merged into the nightly builds.

I realize that this is not the same as testing the entire package on dissimilar hardware like he is doing here; For instance, there are bound to be a few issues when developers of code and its underlying code base both submit updates the same evening. IMHO, it'd especially help new developers if there existed unit tests distributed with the source packages allowing the developers to test to ensure that they aren't breaking anything else with their new/revised code.

In any case, if this is a start, its a nice start, and it means that linux is going to get a LOT easier to compile and install for people from my perspective...

Re:Linux enters the world of QA 101!

[LnxAddct]

Individual distros have been doing this for years. Red Hat is one company that is known for its extensive testing of the kernel (as well as many other OSS projects). Don't use a vanilla kernel if you're running a production environment.
Regards,
Steve

Re:Linux enters the world of QA 101!

[oxfletch]

Firstly, because both RHEL and SLES pull their base from mainline kernel. I'm damned if I'm going to fix bugs 3 times - RHEL, SLES, and back in mainline. Let's fix it once, before it spreads.

Secondly, it's MUCH, MUCH easier to fix a bug the night after it went in, not 3 months later. Everyone has context as to what's goin on fresh in their minds, and the change hasn't been buried under 7 tons of other crap.

Re:NOT FUNNY: Chinese Military Software

[dhakbar]

Um, he's the well known "phrusa troll" that usually only posts in China-related stories.

Sometimes, though, he interjects his posts into unrelated articles.

Don't sweat it

[Anonymous Coward]

GWB has been selling all sorts of technology to the chinese. This will end up in their hands so that halliburton can make a buck.

Re:Don't sweat it

[Anonymous Coward]

Too bad Clinton had already sold it to the Chinese.

Wait a minute...

[RoLi]

So let me summarize wether I understood it right:

You say it's "completely useless" because you have to wait 15 minutes when a kernel is released.

And this is modded "insightful".

Simple.

[Anonymous Coward]

Just get MS virus to work on Linux. Lots of code generation that way.

mod 0p

[Anonymous Coward]

fact came iNto d0n't feel that

Is this even worth anything?

[xenocide2]

One of the main goals appears to be whether the kernel builds or not. I shouldn't have to tell slashdot that build errors are among the most trivial of OS programming errors. They certainly exist, as the chart shows, but whoever is in charge of this project has a long way to go, by adding real tests of functionality. Consider it job security ;)

Re:Is this even worth anything?

[oxfletch]

For one, did you actually bother to look at the results at all, and what tests are being run, and
published?

For another, this is only the tip of the iceberg as to what can be done, but I'm not going to lock whatever I have now in some dingy dungeon until it's "finished". What's there is useful, ableit incomplete. Testing is *never* complete.

The main goal, as you put it, is to improve the quality of the linux kernel. If we can ensure the kernel builds, boots, and runs basic tests ... in a fully automated way ... then it frees up other resources to do more sophisticated tests, without getting dragged down by basic crap.

Onwards, and upwards ...

Re:Is this even worth anything?

[gromitcode]

personally I am absolutely stunned they didn't already have automated testing. People try to claim linux is mature and enterprise ready but then things like this pop up to show how far behind the commercial vendors it still is.

Re:Is this even worth anything?

[xenocide2]

Well, there is testing. It's just done by firms like RedHat and it's not made publicly available. There's also a very public and fairly thourough testing procedure done by Debian, but they don't specifically target the kernel, the way this particular system is.

Well, this time I am really unhappy!

[kompiluj]

1) the very need for such tests means that current 2.6.x kernels are very unstable - this means that Linux currently does not have any stable version - not good
2) remember Microsoft? they have been always doing nightly builds of Windows since the beginning of the time; the only thing sure is that it did not improve the quality of the code...

Re:Well, this time I am really unhappy!

[Anonymous Coward]

1) Automated build/testing should be a standard part of any large development project.

2) Riiiiight. I'm really sure 2000 and XP aren't an improvement over Windows 95.

Re:Well, this time I am really unhappy!

[unleashedgamers]

2) Riiiiight. I'm really sure 2000 and XP aren't an improvement over Windows 95.

2000 and XP are way diffrent than 95.

Windows '95, '98 and ME are descended from DOS and Windows 3.x, and contain significant portions of old 16-bit legacy code. These Windows versions are essentially DOS-based, with 32-bit extensions. Process and resource management, memory protection and security were added as an afterthought and are rudimentary at best. This Windows product line is totally unsuited for applications where security and reliability are an issue. It is completely insecure, e.g. it may ask for a password but it won't mind if you don't supply one. There is no way to prevent the user or the applications from accessing and possibly corrupting the entire system (including the file system), and each user can alter the system's configuration, either by mistake or deliberately. The Windows 9x/ME line primarily targets consumers (although Windows '95 marketing was aimed at corporate users as well).

The other Windows product line includes Windows NT, 2000 and XP, and the server products. This Windows family is better than the 9x/ME line; at least these versions use new (i.e. post-DOS) 32-bit code. Memory protection, resource management and security are a bit more serious than in Windows 9x/ME, and they even have some support for access restrictions and a secure filesystem. That doesn't mean that this Windows family is as reliable and secure as Redmond's marketeers claim, but compared to Windows 9x/ME its additional features at least have the advantage of being there at all. But even this Windows line contains a certain amount of 16-bit legacy code, and the entire 16-bit subsystem is a direct legacy from Microsoft's OS/2 days with IBM. In short, all 16-bit applications share one 16-bit subsystem (just as with OS/2). There's no internal memory protection, so one 16-bit application may crash all the others and the the entire 16-bit subsystem as well. This may create persistent locks from the crashed 16-bit code on 32-bit resources, and eventually bring Windows to a halt. Fortunately this isn't much of a problem anymore now that 16-bit applications have all but died out.

Of course Windows has seen a lot of development over the years. But in fact very little has really improved. The new features in new versions of Windows all show the same half-baked, patchy approach. For each fixed problem, at least one new problem is introduced (and often more than one). Windows XP for example comes loaded with more applications and features than ever before. While this may seem convenient at first sight, the included features aren't as good as those provided by external software. For example, XP insists on supporting DSL ("wideband Internet") networking, scanners and other peripherals with the built-in Microsoft code instead of requiring third-party code. So you end up with things like DSL networking that uses incorrect settings (and no convenient way to change that), scanner support that won't let you use your scanner's photocopy feature, or a digital camera interface that will let you download images from the camera but you can't use its webcam function. WiFi network cards are even more of a problem: where manufacturers could include their own drivers and client manager software in previous versions of Windows, users are now forced to use XP's native WiFi support. Unfortunately XP's WiFi support is full of problems that cause wireless PCs to loose their connection to the wireless access point with frustrating regularity. Also XP's native WiFi support lacks extra functions (such as advanced multiple-profile management) that manufacturers used to include in their client software. And of course applications (such as Internet Explorer and Outlook) have been integrated in the operating system more tightly than ever before, and more formerly separate products have been bundled with the operating system.

Re:Well, this time I am really unhappy!

[Anonymous Coward]

Of course Windows has seen a lot of development over the years. But in fact very little has really improved.

If you really think that Windows hasn't improved over the years, then you are very blind. It is true that Windows isn't perfect, and has a ways to go in some regards, but what OS is perfect in an all-around sense? Now you can try to claim that Linux might be more stable, lets you do this or that with the source code, but it has its downfalls with the interface or lack of configurability for the regular user (as well as lack of software). OS X is probably just as good as Windows in alot of regards, but might be lacking in software options in some areas as well. But atleast right now, Windows still has an incredible amount of software, which is what is keeping alot of people from switching over to other operating systems. What good is an operating system without any software?

The new features in new versions of Windows all show the same half-baked, patchy approach.

Their approach is surely is more well-thought out than Linux's half-baked user interface. Some flavors of Linux have come a long way in that regards, such as RedHat or Suse to name a few, but alot of times you end up having to go to the command-line to configure something. That is not the way an average user wants to do things.

For example, XP insists on supporting DSL ("wideband Internet") networking, scanners and other peripherals with the built-in Microsoft code instead of requiring third-party code.

Microsoft has digitally signed drivers, which have been tested in labs to work with their built-in drivers. It is not hard to use other drivers instead of those. In alot of cases they don't automatically install the software for you, and even if they did, you can always go to the control panel and change to your own drivers (which might be better or worse). In my experience, the custom drivers are rarely better, and mostly worse, than the default ones.

Also XP's native WiFi support lacks extra functions (such as advanced multiple-profile management) that manufacturers used to include in their client software.

Again, there are custom programs which you can use if the default ones with Windows don't work to your specifications. You make it seem as if you can't actually use the client software. I have used a custom wireless manager with my XP system in the past without any problems.

And of course applications (such as Internet Explorer and Outlook) have been integrated in the operating system more tightly than ever before, and more formerly separate products have been bundled with the operating system.

Internet Explorer has been tightly integrated into the OS for a long time now. Old story, nothing to see here. How exactly is Outlook integrated into the OS in the same fashion as IE though? I'm not debating this, just would like to see how you perceive this notion. In regards to bundling of software, do you mean such programs as Windows Media Player, Messenger, etc? How is this any different than most of the Linux distributions? They come with all types of software. You might argue that most of them are free, but even with the commercial Linux offerings, it is the same thing as far as I can tell.

Re:Well, this time I am really unhappy!

[posternutbaguk]

Current 2.6x very kernels unstable? Linux does not have any stable version? Obviously you havn't even used Linux in the last year or so.

Testing a product to make it better doesn't mean the product is bad to start with. Some code has higher aspirations than that.

Re:Well, this time I am really unhappy!

[toddestan]

1) the very need for such tests means that current 2.6.x kernels are very unstable - this means that Linux currently does not have any stable version - not good

If you think that the 2.6.x kernels are unstable, you can use the 2.0, 2.2, or 2.4 kernels. All those versions are still being maintained, and they are definently stable.

Re:Well, this time I am really unhappy!

[ICA]

Good try, but I still call FUD. You are not required to use Microsoft code for any of the pieces you mention above. Yes, it is on by default. HOwever, you can happily go to most manufacturer's website and download alternate versions. Many wireless cards now make it very simple with their client utilities or drivers to turn off Windows wireless zero configuration easily. Otherwise, do it yourself by unchecking the "Let windows manage..." box, or stopping the wireless zero config service.
Mostly the same goes for scanners. I can very easily take advantage of my photocopy feature by using the supplied manufacturer utility.

Lots of text with some true facts to not mean that you are not mostly spewing uninformed rants.

Re:Well, this time I am really unhappy!

[kompiluj]

The problem is that I have extensively and exclusively used Linux 2.6 the whole last year since I migrated from 2.4 because it lacked the features I needed. Back in the good old days it was that Linux had two versions: the development one - uneven numbers - like 2.1 and stable - even numbers - like 2.0. Now the development is in the "stable" branch - 2.6. And this results in big problems. You can get used to them on the desktop, you get mad having to drive to the server room because your 2.6.x server has literally FROZEN (no, not a nice kernel panic).
And as to testing - I would like to see Linux well tested, before it is released or I go elsewhere (no, not meaning Windows of course!)

Re:Well, this time I am really unhappy!

[kompiluj]

Now really dear sir, how am I to use 2.0, 2.2 or 2.4, since all new features, which I need are only in 2.6? Because everybody believes that 2.6 is a stable release (which in theory is indicated by the even release-subnumber) features got added to 2.6 and nobody bothers with backporting them to 2.4. And this is really a big problem. If there was a development version, then 2.6 would be rock-solid stable, I would get no freezes, panics and other nasty events and the features would go to the stable kernel - like it used to be.

Re:Well, this time I am really unhappy!

[unleashedgamers]

If you really think that Windows hasn't improved over the years, then you are very blind. It is true that Windows isn't perfect, and has a ways to go in some regards, but what OS is perfect in an all-around sense? Now you can try to claim that Linux might be more stable, lets you do this or that with the source code, but it has its downfalls with the interface or lack of configurability for the regular user

Linux has had major development in all the areas you say it lacks in. Mac over the last 5 years has also seen major development. I have seen very little come from windows since win 2000 (yes i have played with some of the longhorn beta's)

(as well as lack of software).

No doubt windows is farther in software than others, its the most used OS so why program for anything else?

OS X is probably just as good as Windows in alot of regards, but might be lacking in software options in some areas as well. But atleast right now, Windows still has an incredible amount of software, which is what is keeping alot of people from switching over to other operating systems. What good is an operating system without any software?
What does this have to do with windows development?

Their approach is surely is more well-thought out than Linux's half-baked user interface. Some flavors of Linux have come a long way in that regards, such as RedHat or Suse to name a few, but alot of times you end up having to go to the command-line to configure something. That is not the way an average user wants to do things.

When i said half baked patchy approach i meant that they sometimes have patches for patches.
and how is gnome or kde half-baked?

Microsoft has digitally signed drivers, which have been tested in labs to work with their built-in drivers. It is not hard to use other drivers instead of those. In alot of cases they don't automatically install the software for you, and even if they did, you can always go to the control panel and change to your own drivers (which might be better or worse). In my experience, the custom drivers are rarely better, and mostly worse, than the default ones.

But they still insist upon using the windows ones

Again, there are custom programs which you can use if the default ones with Windows don't work to your specifications. You make it seem as if you can't actually use the client software. I have used a custom wireless manager with my XP system in the past without any problems.

Both Gnome and KDE have it built in and you'll have major problems if you dont want to use Windows NDIS 5.0 drivers for your networking

Internet Explorer has been tightly integrated into the OS for a long time now. Old story, nothing to see here. How exactly is Outlook integrated into the OS in the same fashion as IE though?

Both are non removable both pose a security risk other programs in the OS can exploit them.

In regards to bundling of software, do you mean such programs as Windows Media Player, Messenger, etc? How is this any different than most of the Linux distributions? They come with all types of software. You might argue that most of them are free, but even with the commercial Linux offerings, it is the same thing as far as I can tell.

In windows your forced to have those along with things like pinball on your computer. In linux you can install and uninstall anything and everything almost

Re:Well, this time I am really unhappy!

[unleashedgamers]

Have you tried not using Windows NDIS 5.0 drivers? Not very easy.

Any other Open Source projects have similar?

[team99parody]

I think the PostgreSQL buildfarm is one of the coolest ones I've seen. It's distributed across a bunch of volunteer-run machines representing a broader selection of architectures than most any other automated-test projects I'm aware of. A nice article on it can be found here

Any other projects out there with similar transparency in their automated testing?

Furthermore, it prevents regressions

[xant]

Reliable, repeatable testing is a great way to prevent fixes in one area from causing bugs in another. When I fix A, I generally only test A manually. I don't test every other conceivable code path, even though my fix for A might well impact them.

An automated test for B will catch regressions caused by my fix in A, making it harder to backslide. Backsliding is very expensive because bugs are far removed from their cause. If an automated test sees that changes in A caused a regression in B, the cause is immediately obvious.

That is what aegis does

[nietsch]

http://aegis.sf.net/aegis.sf.net
and it can do a lot of other things too, like making sure that each change has an accompagning test and that all tests pass before anybody else is bothered with that change.

The biggest downside for aegis (as I see it) is that it needs to run on a central development server, it is not server based like CVS or the others(it has a cvs-like interface for reading). But OTOH, would it be so hare to have the kernel developers log into a central compile farm where the linux kernel is developed.

The same thing for NetBSD

[hubertf]

NetBSD has about the same thing - compiling of the whole operating system (kernel, userland, X) for ~50 platforms. Logs are available for developers to fix things.

- Hubert

Maybe a more readable link for the failures...

[blake182]

Could use some work on the failure console output.
3H[16;3H[16;3H[16;3H[16;3H[16;3H[16;3H[16 ;3H[16;3H [16;

(rest of irony deleted due to filter)

That would be good, except

[Anonymous Coward]

So they where using the thousand moonkeys approach uh?

I really dunno where you would find a thousand moon keys!

Re:That would be good, except

[HishamMuhammad]

The moon keys are hidden in the Star level 3, after you pass the warp zone in the Chocolate forest. You need a cheat to get a thousand, though.

Distributed Kernel Testing

[Anonymous Coward]

Here's an idea for a new automatic testing script some brave soul could write:

• Download the latest test version of the kernel (or any other program).
• Build it.
• Create a bootable ISO image that uses that kernel, along with testing programs.
• Burn the CD and reboot.
• After the automatic tests complete, send a short summary of the results to a central place where the data could be collected and analyzed.

This would have the benefit of doing basic testing on a wide variety of hardware. Users wouldn't have to change their computers to run the test versions. I know I'd be happy to run such a test at least weekly, if only to help ensure that future versions of Linux run well on my hardware.

exploits and securityholes?

[Ruede]

what about them ?

IBM's Rational Software

[erik umenhofer]

May not be 100% or even hardcore, but you can go from use case to code if you put in some time. It will also write Java code using your UML diagrams.

It's based off of Eclipse. Check it out if you can.

How about automated testing *before* release?

[noidentity]

The Linux Kernel is now getting automatically tested within 15 minutes of a new version being released, across a variety of hardware and the results are being published for all to see.

How about they test it 15 minutes before being released (or longer)? Otherwise you get a seriously broken release and find out 15 minutes later "it's majorly broken".

this is new?

[Anonymous Coward]

The Itanium Linux crowd have been doing this for months now:

http://www.gelato.unsw.edu.au/IA64wiki/KernelAutoB uild

there's even a freaking RSS stream:

http://www.gelato.unsw.edu.au/kerncomp/rss.php

oh well, good to see IBM testing linux on more common-or-garden x86 machines...

Stop playing the "then do it yourself" card, guys

[CarpetShark]

Because you didn't do it.

Please don't play this card all the time. We hear it way too often in the Free Software/Open Source communities, and it's really quite silly.

The grandparent post asked if it would make more sense to do it another way. That's a perfectly valid and logical question. Either he's right, and it does make more sense, or he's wrong (for a variety of reasons), and it's best to keep it the way it is. None of these require one person to do it incorrectly, and another to do it properly later. With that kind of argument, you're simply defending bad solutions.

In this case, though, it's simply a misleading article summary.

Automated testing

[robjejansen]

It's about time, that kernel.org that gets a tool lik this, beter late then never ;-) Mozilla.org tinderboxes working for years now.