Slashdot Mirror
New Tool to Track Kernel Testing Time
mu22le writes "Andrea Arcangeli has created a new tool, 'klive', to automatically track the amount of testing that each kernel gets before release. According to Kernel Traffic "There was some discussion [on making it a kernel config option] that public perception might put this in the "spyware" category", but still the ability to track a kernel usage and reliability would be valuable to both developers and users."
Sure they do... that's what trade expo's are for, right?
Watch for Penguins, they eat Apples and throw rocks at Windows.
http://www.kerneltraffic.org.nyud.net:8090/kernel- traffic/latest.html#6
http://klive.cpushare.com.nyud.net:8090/
They seem to be taking system stats and system uptimes and presenting it in a hard to understand table. Is that tracking testing?
If I turn on my computer and don't touch it for a year, it will have excellent uptime, but it doesn't really test very much. Same too, if I just start up Apache and let it do its thing.
Testing is a very important part of any development cycle and testing metrics are very useful in determining the quality of the delivered product. However, I've never heard of "testing time" used as a metric. Maybe "coverage" or "bugs over time", but the amount of time itself is never really a concern.
From what I've seen of the Linux kernel (just downloading the source from kernel.org and browsing through it), there doesn't seem to be much in the way of actual debug code thoughtfully and diligently placed throughout the code. There are a few litterings of debug code here and there, but for the most part, it seems like the developers just expect it to work without error.
Nothing wrong with that attitude, if reality backs it up. And luckily, with Linux, reality is right there to prove the developers correct.
Jesus saved me from my past. He can save you as well.
...besides our patience...
j0b.org - A famous domain name for sale
That would be the difference between this idea and spyware.
I don't particularly care if someone is getting anonymous data about my usage of the linux kernel for most of my boxen. It'll help improve the performance with good accurate real world information. However I don't want some sensitive boxen that I am responsible for to output data to any other source for good or ill.
So if I can chose, I'll be happy.
Tracking Kernel Usage
http://kerneltrap.org/node/5606
Microsoft probably puts their stuff through more testing than anybody. And it's needed -- even with all that testing, things slip through. (I'll skip talking about why there's so many bugs ... others can do that.)
The Linux kernel is much smaller than Windows. Far less testing is needed, though of course some testing is still a good idea.
And I know this is unrelated, but the article submission talks about Spyware? Well, not all Spyware is bad. Just because software reports home, that doesn't make it evil. It's only evil if it does this without your knowledge and consent. (And putting this into a huge legal click-through doesn't really count.)
An example of `good' spyware would be the Google Toolbar. Yes, it can contact google if configured to do so, but at least 1) this provides some benefit to you, and 2) google makes it very clear what it's doing, not burying it in some click-through legal agreement and 3) you can easily turn it off if desired.
Isn't that what firewalls are for. You have complete control over what packets leave your office/room/basement; since the cable passes through there.
If a 3rd party releases a kernel with modifications that allow them to track you without your knowing how nice for their revenue! Imagine redhat releasing a kernel for Fedora that gives feedback to companies on their users' computing habits...If you're doing all this at kernel level it will be too hard to track.
I wonder if this is similar to the tool used in my microwave to track Kernel popping time.
/sig
Keeping this as an external script is definitely the way forward. As pointed out, having a kernel flag and especially having the possibility of it defaulting to YES is a step too far IMHO.
This is definitely a very useful system however, and I for one would very much like to see something similar for distributions (ie. not just the kernel, but the whole damn caboodle).
Burns: We're building a casino!
McAllister: Arrr. Give me 5 minutes.
Where I work we have testing scripts that perform most of our testing. Couldn't most kernel features be tested by scripts or automated processes?
Reason #32767 not to use VB6: Integers are 2 bytes... Think about it!
Some of the code in windows XP hasnt even been seen by a human being for years upon years. From my perspective, a significant amount of exploits come from services or 'features' that are never used for anything. I several years for working at a college, 200+ PC's under my belt at any specfic time I find that our biggest problem with bugs, come from places that we hardly touch. Alas, Microsoft has admited that it focused more upon putting features into its products and pushing it out the door in a nice boxed product moreso than security. They only clammored about how they are focusing on security of recent when consumers started getting wise and yelling at the top of there lungs for microsoft to fix there products.
Microsoft DOES test its products. I recall Win95 having "the largest beta-testing stage in history".
Anyway, the security failures in WinXP are not due to lack of testing, but because of poor design decisions. Besides, security failures can't be detected by common beta-testing, but by heavy security audits (Not that I don't hate MS policies, I do, but there was no point in your comment).
In summary, your post wasn't informative, interesting, or insightful. Not even funny.
And I don't think it could be thought of as spyware.
Spyware is supposed to be unknowingly reporting information about you, whether it was mistakenly installed by you or it crept in from somewhere else.
An application, or kernel option you flick on like a switch, which you install, and that reports information you know about, to people you understand are going to use that information, can't be called spyware unless it also happened to report how much pr0n you have as well as the kernel's amount of usage.
I think it would be a neat option to have in the kernel in general. Off by default, all us geeks who want to say "look! here! I'm running Linux!" could turn it on and it could report our uptimes and what kernels we're running.
We could "stand up and be counted" to show our support for Linux and give the various distributions a rough idea of what we think about them.
His name is Robert Paulsen...
if you download and install it as of 10am PST today, its going to try and install a cron job that begins:
-*/10 * * * * ps x | grep...
which vixie cron (and presumably others) rejects as invalid. i just changed it to run every 10 minutes like:
*/10 * * * * ps x | grep...
hth
about sean dreilinger
I didn't know Phil Collins had a son.
That's ok, Jesus likes me anyway.
I thought there already was an article about klive in slashdot, but I might be wrong too. Already installed it ^-^
Actually, they most certainly don't. Microsoft's target market (desktops) have always been an area where time-to-market has been more important than perfection.
Indeed, I would say that Microsooft's single strongest competitive advantage is that they were the first company to correctly balance time-to-market against testing and quality. The bottom line is that most businesses really don't care (as proven by their purchase history) about viruses if they can get colorful shaded borders around windows.
Other companies (all the unix vendors, apple, etc) that wasted years on testing before releasing were really showing a very poor understanding of their market.
I think this is a fine idea - tracking and all - and I've been running klive since I saw it on kernel trap last week - however, I think that some people are correct when they question how uptime counts as reliability. It doesn't - in the sense of it testing load and the like - but it does because it takes a whole lot of kernel reliability/stability for it to boot in the first place, and it takes a bit for it to just gain uptime.
Personally, I would like to see it as an option in the kernel - but I'd like it to be off by default. I'd the statistics to be available to everyone (*NOT* IP addresses, hostnames, etc) but rather version, compiler, memory and load.
While I'm fine with just running some guys software for now, it's gonna turn into a huge mess . What happens if there's a bug? How's he gonna get it distributed to everyone? What if they want to track something else?
that public perception might put this in the "spyware" category", but still the ability to track a kernel usage and reliability would be valuable to both developers and users."
And the ability to track user web site visitation and buying habits is valuable to both comsumers and retailers. I still don't want that shit on my computer.
Spyware is spyware. Any time my computer does something without my permission, I consider is malicious and immediately remedy the situation by removing the offending piece of software.
It amazes me that such a widely used product takes 10 years to implement Quality Assurance 101. I can hear the headlines now
"Linux Kernel team introduces Requirement to defect tracking".
Believe me, if I started murdering people, there would be none of you left.
...it's called 'uptime'.
Modest doubt is called the beacon of the wise. - William Shakespeare
Well screw that. i was going to help by installing it but "python setup.py install" wants Zope installed. Fergit it!
My karma is not a Chameleon.
Microsoft probably puts their stuff through more testing than anybody. And it's needed -- even with all that testing, things slip through. (I'll skip talking about why there's so many bugs ... others can do that.)
That isn't really much of a statement ("probably"). Given the 'quality' of quite a few Microsoft products, they obviously haven't been tested thoroughly enough.
The Linux kernel is much smaller than Windows. Far less testing is needed, though of course some testing is still a good idea.
Linux is a monolithic kernel with practically all its drivers (apart from VGA drivers) built in, and the WinNT kernel is a microkernel with most drivers made by third parties. You really, really cannot compare the two that easily.
Tidbit: Linux 2.6 is now running on more than half the computers tracked.
Of course it's not spyware for a linux power-user. We tweak our kernels all the time: "Oh, damn it, my new bluetooth device need the module bt_frobniz, guess I'll just make menuconfig, etc. to install it."
However, linux is growing up. We have a number of distros out there that are supposedly targeting new or casual users, those that might never fiddle with their .config (even if they do upgrade their kernel, it's probably through an automated tool).
Now, sure with (or without) the patch, Linus and any other linux defender can point out that the kernel is not to blame, the distro is, but that may not amount to much if the "Linux is Spyware" "headline" gets any amount of press. That kinda crap gets on the "front page"; retractions/clarification are "buried in small print in a random position of page 3b". It doesn't matter the veracity of linux's deferenders claim if they don't catch the same amount of "mindshare".
If I was to throw my .02$ into the arena on LKML, I'd be against a kernel option, or at /least/ one that requires a userland application [1] to take advantage of. Not because of technological issues, but because of polical ones.
[1] Yes, this could also be packged by a distro, but then, hopefully, the "headline" would be "<userland-application> is Spyware". People confuse applications with linux less often than they confuse distros with linux
You know, I don't know what universe these folk are living in, but this "python-twisted" package or whatever it is called is absolutely NOT included in every Linux distribution.
Slackware - oldest living Linux distribution - does NOT have this twisted thing in it.
You would think that the developers would use a standard programming language - like C - for something like this...(gr&d)
Ron Gage - Westland, MI
The scripts involved generally come from previously found and corrected defects (making sure they don't come back) and possibly from things that the developers think might break. And a large part of it comes from what somebody thinks somebody might do with the software. In any event, much of it's usually heavily scripted so that nothing is left out, though it's very difficult to get a script going that tests everything (and in fact, it's probably impossible for something as complex as Windows.)
And then there's beta programs, where they give the unfinished package to end users who are willing to beta test, and the users are asked to submit bug reports. But this is generally much less formal, and it seems to me that most users don't submit bug reports at all (as long as they can work around the bugs found), though there are some that will report every little thing (and these users are great for making beta programs work.)
(Right or wrong, that's generally the way how it's done. Code reviews are another matter entirely.)
As for open source software, they generally skip the whole scripted human run test suite, though many software packages do provide a test suite that does much of the same thing automatically. (`make test' is a wonderful thing when done properly ...)
Of course, huge packages like Windows or Office are the exception rather than the rule, and these are the sorts of things that really need a scripted, human run testing phase. Redhat Linux (the entire package) and OpenOffice may get similar treatments, but most smaller packages don't.
That may be true (or may not. I'd really love to see a citation from Microsoft where they claim that they focus more on adding features than security. That just doesn't sound like the sort of thing that PR would let them say, even if it's true.)In any event, the number of developers and testers and such assigned to Windows and Office is huge, probably larger than the entire staffs of most software companies. A small percentage of those people are probably assigned to doing product testing, probably a similar percentage as is assigned to similar jobs in other software companies. But since the entire team is so large, more testing is likely to be done. (Now, perhaps far more testing is needed, but that's not what I'm talking about here.)
Really, the only software packages that I can think of that might get more formal testing than Windows and Office might be something like Oracle.
Failed the FP you have - yoda
And yet, Microsofts testing procedures still suck. I know this because Windows still ships with bugs that would have been caught by basic static analysis programs, of the kind that I know for a fact Microsoft has and uses. Granted that Windows is massive, but really, they've had what, 3 years now since they announced their new dedication to security? They could have done a total code audit of the Windows source in that time, including ensuring that it passed static analysis tools. This is totally aside from classic test suites - something that I am not convinced Microsoft uses, at least not in the Windows kernel and Win32 API.
And the post that I was replying to simply said ` oh wait.. they dont test anything..' which is patently incorrect. (Of course, the post has also been modded -1, which is probably exactly where it belongs.)
I'm not saying this is the way it should be, only that it's the way it is. And it does make sense ... people don't buy Windows because it's secure. They buy it because it's what they know, it runs their software, and it does what they want. Security is an after-thought to most people, especially the sort who uses Windows.
Perhaps, but that's not what I'm talking about. They may not be testing what needs testing, but it seems quite likely that Microsoft spends more time testing something about their product than anybody else does, if only because of their size and the breadth of their products.Coupled with meaningful "stress" metrics, "uptime" might be more useful. I don't know the Linux kernel terminology, but internal stats capturing information (averages, min, max, rate or whatever makes sense for the particular case) about run queue lengths, number of threads transitioning from blocked to run, number of threads suspended for a page fault, number of preemptive and non-preemptive thread context switches might be useful. In every OS there are odd edge cases which "hardly ever happen" except in rather unusual cases and stats on some of these may be useful (since they usually are "under tested" and often reflect a stressful or unusual application load).
Why is there an "insightful" mod and why isn't it "-1"? If I wanted insight, I wouldn't be reading
I really don't see how this tracks anything other than uptime. Microsoft could probably publish a page listing each development build of Vista as well, and it would more than likely have more uptime than this.
Its a pathetic attempt to hide the fact that IBM, Redhat and Suse don't devote resources to testing as its in their interest that you have to pay for support and services.
So I went back to windows.
I read the headline as "New Tool Track to Kernel Testing Time." I like Tool.
Laboratree - Scientific collaboration based on OpenSocial.
Your obvoiusly not a software developer then are you? In development of my web-based applications I scrutinize and TRY to physically break everything I can break in my webaps. Including giving bonuses to those who test for me, who can find security flaws and reproduce them. Security testing is major in my productions, and always should be. I am positive that it is for Apple's OS X.
Honestly. Maybe you don't like what it's doing, but you know what it's doing, and you can change it. Or, if you aren't a programmer and can't afford one, you can simply disable the feature.
Don't thank God, thank a doctor!
I suspect that Microsoft checks for security issues too. But I also suspect that this is less than 50% of their total testing. I did not say `software testing is not done for security'. I said `Most software testing is not done for security', and I believe this statement to be accurate.
Perhaps I was a little more vague that I should have been, but ultimately it depends on the application. Some will get more security-specific testing than others, but I suspect that overall (painting all of software testing with a very broad brush here) security-specific testing is a good deal less than 50% of all software testing done.
In the web based applications that I've written, security has always been a prime consideration. However, even then, I've probably spent more time verifying that things work rather than verifying that they work securely. To make sure things are secure, I've concentrated more on the code, making sure I don't trust the user's input until it's been validated/scrubbed, things like that. But that's not really `testing', that's more development. (And yes, I do some testing that things work securely afterwards, trying to break things, but not as much time as I spend making sure that things work correctly. Security is important, but so is functionality.)
And any sort of network or web application is a special case, requiring extra special attention to security. (Though one would certainly argue that a network aware OS like Windows should pay similar attention to security, and it's relatively clear that Microsoft does not.)
Good for you. But I never said it shouldn't be or isn't `major'. Ok, but in order for my statement to be inaccurate in the case of OS X, you'd have to show that over 50% of their testing is looking specifically for security related issues. I don't doubt that security is important to them, but I suspect that security checking doesn't take most of their testing time.You know, I spend way too much time defending statements on /. that really ought to be obvious. I should stop wasting my time -- I'm probably not convincing anybody, and I'm not even really entertaining myself.
How can something that you have to compile and configure yourself be a spyware?