Slashdot Mirror
Linux Kernel Gets Fully Automated Test
An anonymous reader writes "The Linux Kernel is now getting automatically tested within 15 minutes of a new version being released, across a variety of hardware and the results are being published for all to see. Martin Bligh
announced this yesterday, running on top of IBM's internal test automation system. Maybe this will enable the kernel developers to keep up with the 2.6 kernel's rapid pace of change. Looks like it caught one new problem with last night's build already ..."
code generation...
Why has it taken so long?
"The Linux Kernel is now getting automatically tested within 15 minutes of a new version being released"
Would be much better to test it BEFORE a new version is being released, otherwise this is completely useless...
How were the previous kernels being tested? Were sources for improvement/change/modification, bugs and areas requiring refactoring being discovered by chance?
This is good, and long overdue (I'm surprised it hasn't been around for years), but just how much testing is being done? Compiling? Booting? Or are there actual functional and reliability tests which are being performed?
Most projects of any complexity use automated continuous build and testing as a standard development practise.
Do not try to read the dupe, thats impossible. Instead, only try to realize the truth
What truth?
There is no dupe
automated performance regression tests may be useful too.
Wondering why i am doing so strange posts? I am trying to get a "+5,Flamebait" or "-1,Insightful" rating.
But it can't catch everything - the 1394 bus was screwed in 2.6.11. There are a lot of regressions that show up - and even that healthy cluster of systems will not show every problem.
Sound issues? Older network and SCSI cards? There are a lot of drivers that break, and no one notices it because there is nobody with the hardware testing the -rc or -mm kernels.
Wouldn't it make more sense to package these tools for someone to install on their collection of oddball equipment, and assist in the debugging/testing?
Where's the ARM, MIPS, and SH?
Why can't I mod "-1 Idiot"?
ARM Linux has had something similar in Kautobuild for some time.
Although the testing and building is limited to the ARM platform.
The site also has a whos who thats worh looking at ;-)
...the cross-platform, cross-hardware part? Setting up one machine to build automatically is easy. Setting up a whole bunch of them (and all unique, read administration nightmare) and tie them together to a system, that's quite a bit of work.
Kjella
Live today, because you never know what tomorrow brings
Related projects at OSDL
http://osdl.org/projects/26lnxstblztn/results/
http://developer.osdl.org/cherry/compile/
Red Hat (and probably Novell/SuSe, since they use over one thousand kernel patches) runs a myriad of tests on each of its own kernel builds nightly - and has been doing so for years. On more than just the 3 architectures covered by this test.
That said, pushing tests upstream is a great idea. Just not revolutionary or anything.
I hope they are using code from the Linux testing suite. That piece of work has already formed a nice set of tests. Also, I hope that the kernel is automatically built with many different combinations of options. And with time, I hope this will become better. The more tests, with the more hardware configurations, with the more kernel configurations, with the more types of input data (including many imaginative forms of incorrect input data to test that the kernel handles it gracefully and thwarts attacks based on such methods), the better quality we will have in the kernel, and it is likely that Linux will be unmatched in quality, stability, efficiency (well, maybe not efficiency necessarily), and long uptimes.
With an automated test suite, what happens when a class of bug is discovered to be untested-for? Presumably, the suite is modified to detect it. Then, is the resulting new suite itself subjected to an automated test suite? And, then...[divide-by-zero error...]
Seeing bad movies only encourages them. Watch responsibly
Does this mean we'll get back to 2.6.x releases? Instead of new version of 2.6.x being released as 2.6.x.x every third day?
We should consider keeping the automated test technology under the label, "top secret". The Chinese are aggressively trying to modernize their military. Computer software for coordinating the various pieces of battlefield hardware is integral to Chinese plans for the next-generation of warfare.
One downside is that people can become sloppy if they rely too much in the test suite caching all bugs (it obviously cannot).
Anyway, I think that if put to good use it can be an step forward.
Regarding sharing code with other test projects... I hope they do not! What we need is as much different tests and testing methodologies as we can generate. The more the better, so let's go invent a better wheel!
Martin Bligh announced this yesterday, running on top of IBM's internal test automation system.
Hope he doesn't fall off and hurt himself.
I got to work on part of this system, which IBM calls Autobench, for my senior project at PSU. The system is a highly configurable framework which can download, compile, and run various benchmarks and profilers (for example while compiling a kernel). Its all centrally administered, so IBM can run a battery of tests on a variety of different machines at once.
I think Martin Bligh said that IBM has been using this for a while now, automatically downloading kernels upon release and testing them. The new thing is the regression matrix which is posted to the web. Very cool.
needs work! The latest builds all failed!
Compiling is considered a test?
Let me guess... You're American?
IBM's efforts vis-a-vis Linux are a real eye opener when you look at the scraps that Apple gives back to the open source community after taking so much.
I hope all this translates to easier compilation of linux. I had worked with User Mode Linux a while back, and it takes a trip to hell and back to compile.. you need just the right setup of gcc, config files to get it to compile, which took me forver to get right, and honestly a waste of time.
Years later and finally it is getting some *basic* QA testing done! What will they think of next!
Believe me, if I started murdering people, there would be none of you left.
Um, he's the well known "phrusa troll" that usually only posts in China-related stories.
Sometimes, though, he interjects his posts into unrelated articles.
GWB has been selling all sorts of technology to the chinese. This will end up in their hands so that halliburton can make a buck.
You say it's "completely useless" because you have to wait 15 minutes when a kernel is released.
And this is modded "insightful".
Just get MS virus to work on Linux. Lots of code generation that way.
One of the main goals appears to be whether the kernel builds or not. I shouldn't have to tell slashdot that build errors are among the most trivial of OS programming errors. They certainly exist, as the chart shows, but whoever is in charge of this project has a long way to go, by adding real tests of functionality. Consider it job security ;)
I Browse at +4 Flamebait
Open Source Sysadmin
Any other projects out there with similar transparency in their automated testing?
1) Automated build/testing should be a standard part of any large development project.
2) Riiiiight. I'm really sure 2000 and XP aren't an improvement over Windows 95.
Reliable, repeatable testing is a great way to prevent fixes in one area from causing bugs in another. When I fix A, I generally only test A manually. I don't test every other conceivable code path, even though my fix for A might well impact them.
An automated test for B will catch regressions caused by my fix in A, making it harder to backslide. Backsliding is very expensive because bugs are far removed from their cause. If an automated test sees that changes in A caused a regression in B, the cause is immediately obvious.
It's rare that you're presented with a knob whose only two positions are Make History and Flee Your Glorious Destiny.
2) Riiiiight. I'm really sure 2000 and XP aren't an improvement over Windows 95.
2000 and XP are way diffrent than 95.
Windows '95, '98 and ME are descended from DOS and Windows 3.x, and contain significant portions of old 16-bit legacy code. These Windows versions are essentially DOS-based, with 32-bit extensions. Process and resource management, memory protection and security were added as an afterthought and are rudimentary at best. This Windows product line is totally unsuited for applications where security and reliability are an issue. It is completely insecure, e.g. it may ask for a password but it won't mind if you don't supply one. There is no way to prevent the user or the applications from accessing and possibly corrupting the entire system (including the file system), and each user can alter the system's configuration, either by mistake or deliberately. The Windows 9x/ME line primarily targets consumers (although Windows '95 marketing was aimed at corporate users as well).
The other Windows product line includes Windows NT, 2000 and XP, and the server products. This Windows family is better than the 9x/ME line; at least these versions use new (i.e. post-DOS) 32-bit code. Memory protection, resource management and security are a bit more serious than in Windows 9x/ME, and they even have some support for access restrictions and a secure filesystem. That doesn't mean that this Windows family is as reliable and secure as Redmond's marketeers claim, but compared to Windows 9x/ME its additional features at least have the advantage of being there at all. But even this Windows line contains a certain amount of 16-bit legacy code, and the entire 16-bit subsystem is a direct legacy from Microsoft's OS/2 days with IBM. In short, all 16-bit applications share one 16-bit subsystem (just as with OS/2). There's no internal memory protection, so one 16-bit application may crash all the others and the the entire 16-bit subsystem as well. This may create persistent locks from the crashed 16-bit code on 32-bit resources, and eventually bring Windows to a halt. Fortunately this isn't much of a problem anymore now that 16-bit applications have all but died out.
Of course Windows has seen a lot of development over the years. But in fact very little has really improved. The new features in new versions of Windows all show the same half-baked, patchy approach. For each fixed problem, at least one new problem is introduced (and often more than one). Windows XP for example comes loaded with more applications and features than ever before. While this may seem convenient at first sight, the included features aren't as good as those provided by external software. For example, XP insists on supporting DSL ("wideband Internet") networking, scanners and other peripherals with the built-in Microsoft code instead of requiring third-party code. So you end up with things like DSL networking that uses incorrect settings (and no convenient way to change that), scanner support that won't let you use your scanner's photocopy feature, or a digital camera interface that will let you download images from the camera but you can't use its webcam function. WiFi network cards are even more of a problem: where manufacturers could include their own drivers and client manager software in previous versions of Windows, users are now forced to use XP's native WiFi support. Unfortunately XP's WiFi support is full of problems that cause wireless PCs to loose their connection to the wireless access point with frustrating regularity. Also XP's native WiFi support lacks extra functions (such as advanced multiple-profile management) that manufacturers used to include in their client software. And of course applications (such as Internet Explorer and Outlook) have been integrated in the operating system more tightly than ever before, and more formerly separate products have been bundled with the operating system.
Of course Windows has seen a lot of development over the years. But in fact very little has really improved.
If you really think that Windows hasn't improved over the years, then you are very blind. It is true that Windows isn't perfect, and has a ways to go in some regards, but what OS is perfect in an all-around sense? Now you can try to claim that Linux might be more stable, lets you do this or that with the source code, but it has its downfalls with the interface or lack of configurability for the regular user (as well as lack of software). OS X is probably just as good as Windows in alot of regards, but might be lacking in software options in some areas as well. But atleast right now, Windows still has an incredible amount of software, which is what is keeping alot of people from switching over to other operating systems. What good is an operating system without any software?
The new features in new versions of Windows all show the same half-baked, patchy approach.
Their approach is surely is more well-thought out than Linux's half-baked user interface. Some flavors of Linux have come a long way in that regards, such as RedHat or Suse to name a few, but alot of times you end up having to go to the command-line to configure something. That is not the way an average user wants to do things.
For example, XP insists on supporting DSL ("wideband Internet") networking, scanners and other peripherals with the built-in Microsoft code instead of requiring third-party code.
Microsoft has digitally signed drivers, which have been tested in labs to work with their built-in drivers. It is not hard to use other drivers instead of those. In alot of cases they don't automatically install the software for you, and even if they did, you can always go to the control panel and change to your own drivers (which might be better or worse). In my experience, the custom drivers are rarely better, and mostly worse, than the default ones.
Also XP's native WiFi support lacks extra functions (such as advanced multiple-profile management) that manufacturers used to include in their client software.
Again, there are custom programs which you can use if the default ones with Windows don't work to your specifications. You make it seem as if you can't actually use the client software. I have used a custom wireless manager with my XP system in the past without any problems.
And of course applications (such as Internet Explorer and Outlook) have been integrated in the operating system more tightly than ever before, and more formerly separate products have been bundled with the operating system.
Internet Explorer has been tightly integrated into the OS for a long time now. Old story, nothing to see here. How exactly is Outlook integrated into the OS in the same fashion as IE though? I'm not debating this, just would like to see how you perceive this notion. In regards to bundling of software, do you mean such programs as Windows Media Player, Messenger, etc? How is this any different than most of the Linux distributions? They come with all types of software. You might argue that most of them are free, but even with the commercial Linux offerings, it is the same thing as far as I can tell.
http://aegis.sf.net/aegis.sf.net
and it can do a lot of other things too, like making sure that each change has an accompagning test and that all tests pass before anybody else is bothered with that change.
The biggest downside for aegis (as I see it) is that it needs to run on a central development server, it is not server based like CVS or the others(it has a cvs-like interface for reading). But OTOH, would it be so hare to have the kernel developers log into a central compile farm where the linux kernel is developed.
This space is intentionally staring blankly at you
- Hubert
Could use some work on the failure console output.6 ;3H[16;3H [16;
3H[16;3H[16;3H[16;3H[16;3H[16;3H[16;3H[1
(rest of irony deleted due to filter)
I really dunno where you would find a thousand moon keys!
Current 2.6x very kernels unstable? Linux does not have any stable version? Obviously you havn't even used Linux in the last year or so.
Testing a product to make it better doesn't mean the product is bad to start with. Some code has higher aspirations than that.
- Download the latest test version of the kernel (or any other program).
- Build it.
- Create a bootable ISO image that uses that kernel, along with testing programs.
- Burn the CD and reboot.
- After the automatic tests complete, send a short summary of the results to a central place where the data could be collected and analyzed.
This would have the benefit of doing basic testing on a wide variety of hardware. Users wouldn't have to change their computers to run the test versions. I know I'd be happy to run such a test at least weekly, if only to help ensure that future versions of Linux run well on my hardware.what about them ?
May not be 100% or even hardcore, but you can go from use case to code if you put in some time. It will also write Java code using your UML diagrams.
It's based off of Eclipse. Check it out if you can.
1) the very need for such tests means that current 2.6.x kernels are very unstable - this means that Linux currently does not have any stable version - not good
If you think that the 2.6.x kernels are unstable, you can use the 2.0, 2.2, or 2.4 kernels. All those versions are still being maintained, and they are definently stable.
The Linux Kernel is now getting automatically tested within 15 minutes of a new version being released, across a variety of hardware and the results are being published for all to see.
How about they test it 15 minutes before being released (or longer)? Otherwise you get a seriously broken release and find out 15 minutes later "it's majorly broken".
The Itanium Linux crowd have been doing this for months now:
B uild
http://www.gelato.unsw.edu.au/IA64wiki/KernelAuto
there's even a freaking RSS stream:
http://www.gelato.unsw.edu.au/kerncomp/rss.php
oh well, good to see IBM testing linux on more common-or-garden x86 machines...
Good try, but I still call FUD. You are not required to use Microsoft code for any of the pieces you mention above. Yes, it is on by default. HOwever, you can happily go to most manufacturer's website and download alternate versions. Many wireless cards now make it very simple with their client utilities or drivers to turn off Windows wireless zero configuration easily. Otherwise, do it yourself by unchecking the "Let windows manage..." box, or stopping the wireless zero config service.
Mostly the same goes for scanners. I can very easily take advantage of my photocopy feature by using the supplied manufacturer utility.
Lots of text with some true facts to not mean that you are not mostly spewing uninformed rants.
The problem is that I have extensively and exclusively used Linux 2.6 the whole last year since I migrated from 2.4 because it lacked the features I needed. Back in the good old days it was that Linux had two versions: the development one - uneven numbers - like 2.1 and stable - even numbers - like 2.0. Now the development is in the "stable" branch - 2.6. And this results in big problems. You can get used to them on the desktop, you get mad having to drive to the server room because your 2.6.x server has literally FROZEN (no, not a nice kernel panic).
And as to testing - I would like to see Linux well tested, before it is released or I go elsewhere (no, not meaning Windows of course!)
You can defy gravity... for a short time
Now really dear sir, how am I to use 2.0, 2.2 or 2.4, since all new features, which I need are only in 2.6? Because everybody believes that 2.6 is a stable release (which in theory is indicated by the even release-subnumber) features got added to 2.6 and nobody bothers with backporting them to 2.4. And this is really a big problem. If there was a development version, then 2.6 would be rock-solid stable, I would get no freezes, panics and other nasty events and the features would go to the stable kernel - like it used to be.
You can defy gravity... for a short time
If you really think that Windows hasn't improved over the years, then you are very blind. It is true that Windows isn't perfect, and has a ways to go in some regards, but what OS is perfect in an all-around sense? Now you can try to claim that Linux might be more stable, lets you do this or that with the source code, but it has its downfalls with the interface or lack of configurability for the regular user
Linux has had major development in all the areas you say it lacks in. Mac over the last 5 years has also seen major development. I have seen very little come from windows since win 2000 (yes i have played with some of the longhorn beta's)
(as well as lack of software).
No doubt windows is farther in software than others, its the most used OS so why program for anything else?
OS X is probably just as good as Windows in alot of regards, but might be lacking in software options in some areas as well. But atleast right now, Windows still has an incredible amount of software, which is what is keeping alot of people from switching over to other operating systems. What good is an operating system without any software?
What does this have to do with windows development?
Their approach is surely is more well-thought out than Linux's half-baked user interface. Some flavors of Linux have come a long way in that regards, such as RedHat or Suse to name a few, but alot of times you end up having to go to the command-line to configure something. That is not the way an average user wants to do things.
When i said half baked patchy approach i meant that they sometimes have patches for patches.
and how is gnome or kde half-baked?
Microsoft has digitally signed drivers, which have been tested in labs to work with their built-in drivers. It is not hard to use other drivers instead of those. In alot of cases they don't automatically install the software for you, and even if they did, you can always go to the control panel and change to your own drivers (which might be better or worse). In my experience, the custom drivers are rarely better, and mostly worse, than the default ones.
But they still insist upon using the windows ones
Again, there are custom programs which you can use if the default ones with Windows don't work to your specifications. You make it seem as if you can't actually use the client software. I have used a custom wireless manager with my XP system in the past without any problems.
Both Gnome and KDE have it built in and you'll have major problems if you dont want to use Windows NDIS 5.0 drivers for your networking
Internet Explorer has been tightly integrated into the OS for a long time now. Old story, nothing to see here. How exactly is Outlook integrated into the OS in the same fashion as IE though?
Both are non removable both pose a security risk other programs in the OS can exploit them.
In regards to bundling of software, do you mean such programs as Windows Media Player, Messenger, etc? How is this any different than most of the Linux distributions? They come with all types of software. You might argue that most of them are free, but even with the commercial Linux offerings, it is the same thing as far as I can tell.
In windows your forced to have those along with things like pinball on your computer. In linux you can install and uninstall anything and everything almost
Please don't play this card all the time. We hear it way too often in the Free Software/Open Source communities, and it's really quite silly.
The grandparent post asked if it would make more sense to do it another way. That's a perfectly valid and logical question. Either he's right, and it does make more sense, or he's wrong (for a variety of reasons), and it's best to keep it the way it is. None of these require one person to do it incorrectly, and another to do it properly later. With that kind of argument, you're simply defending bad solutions.
In this case, though, it's simply a misleading article summary.
Have you tried not using Windows NDIS 5.0 drivers? Not very easy.
It's about time, that kernel.org that gets a tool lik this, beter late then never ;-)
Mozilla.org tinderboxes working for years now.