Linux Kernel Gets Fully Automated Test

An anonymous reader writes "The Linux Kernel is now getting automatically tested within 15 minutes of a new version being released, across a variety of hardware and the results are being published for all to see. Martin Bligh announced this yesterday, running on top of IBM's internal test automation system. Maybe this will enable the kernel developers to keep up with the 2.6 kernel's rapid pace of change. Looks like it caught one new problem with last night's build already ..."

Comments

This is awesome

[jnelson4765]

But it can't catch everything - the 1394 bus was screwed in 2.6.11. There are a lot of regressions that show up - and even that healthy cluster of systems will not show every problem.

Sound issues? Older network and SCSI cards? There are a lot of drivers that break, and no one notices it because there is nobody with the hardware testing the -rc or -mm kernels.

Wouldn't it make more sense to package these tools for someone to install on their collection of oddball equipment, and assist in the debugging/testing?

Where's the ARM, MIPS, and SH?

Re:This is awesome

[Meshach]

But it can't catch everything...

But that is not the point of automated testing. As a member of a qa team who is developing automated tests I get comments like that every day

Automated tests are not intended to catch everything or test strange permutations of pre-conditions. There purpose is to provide a mechanism for verifying that a build satisfies the basic requirements of the project.

More exotic configs need to be tested manually as usual but automated tests can provide a "failsafe" just in case a basic part of the build is broken.

Re:Why has it taken so long?

[Anonymous Coward]

Bitkeeper.

Re:Within 15 Minutes? WTF

[DigiShaman]

Sounds like the solution to this problem is clear. Always use the second to latest kernel released. Stay away from the new one untill it's fully tested to your satisfaction.

Presumably...

[Kjella]

...the cross-platform, cross-hardware part? Setting up one machine to build automatically is easy. Setting up a whole bunch of them (and all unique, read administration nightmare) and tie them together to a system, that's quite a bit of work.

Kjella

Re:Presumably...

[Bob_Robertson]

I don't remember who said it first:

The first 90% takes 10% of the time.

The last 10% takes 90% of the time.

I expect one could substitute "money", "labor", "effort" for "time" in the above.

Bob-

Re:Within 15 Minutes? WTF

[doshell]

"Release" in the open source world has a broader sense than in commercial software. In open source not all "released" versions are meant for general public consumption; they include unstable versions targeted mostly at developers, so that severe isues can be detected and patched quickly.

Taking this into account, I believe this is meant to catch bugs mainly in nightly (unstable) builds and release candidates, not in "final" versions (those should, at least in theory, have no serious bugs left around as the latter have already been eradicated from release candidates).

Re:now all we need is automated....

[maxwell demon]

No problem. The following is an automated code generator. It generates a hello world program in C and writes it to stdout. (untested)

#include <stdio.h>
int main()
{
char const* program_pattern = "%s%s";
char const* include_pattern = "#include <%s>\n";
char const* function_declaration_pattern = "int %s(%s)";
char const* function_definition_pattern = "%s\n{\n %s;\n}\n";
char const* print_pattern = "printf(%s)\n";
char const* string_pattern = "\"%s\"";

char const* stdio_header_name = "stdio.h";
char const* main_function_name = "main";
char const* main_arguments = ""; // we don't read command line arguments
char const* output_string = "hello world!";

char string[15];
char print[23];
char main_decl[11];
char include[19];
char main_func[42];

sprintf(string, string_pattern, output_string);
sprintf(print, print_pattern, string);
sprintf(main_decl, function_declaration_pattern, main_function_name, main_arguments);
sprintf(main_func, function_definition_pattern, main_decl, print);
sprintf(include, include_pattern, stdio_header_name);
printf(program, include, main_func);
return 0;
}

Furthermore, it prevents regressions

[xant]

Reliable, repeatable testing is a great way to prevent fixes in one area from causing bugs in another. When I fix A, I generally only test A manually. I don't test every other conceivable code path, even though my fix for A might well impact them.

An automated test for B will catch regressions caused by my fix in A, making it harder to backslide. Backsliding is very expensive because bugs are far removed from their cause. If an automated test sees that changes in A caused a regression in B, the cause is immediately obvious.

Re:Within 15 Minutes? WTF

[digitalunity]

Ummm...

If everyone did this, the newest kernels would never get tested. I think it is important that we have a diverse range of users using new, almost new, and older but well tested kernels.

Re:now all we need is automated....

[jrockway]

It was a joke, dumbass.

If you're going to used fixed-length buffers, though, at least use sNprintf!

Re:Well, this time I am really unhappy!

[posternutbaguk]

Current 2.6x very kernels unstable? Linux does not have any stable version? Obviously you havn't even used Linux in the last year or so.

Testing a product to make it better doesn't mean the product is bad to start with. Some code has higher aspirations than that.