Slashdot Mirror
Kernel Tracing With LTTng On Ubuntu Maverick
francis-giraldeau writes "Linux Tracing Toolkit (LTTng) provides high-performance kernel tracing for Linux. This is the killer app for system level debugging and performance tuning. It's now easier than ever to install, with packages released for Ubuntu Maverick. The short introduction to kernel tracing shows how to interpret a simple kernel trace and relate it to strace. I would like to ask Slashdot readers what they would expect as features for a kernel tracing analysis tool, because I'm starting my PhD on this topic and looking for ideas. Also, I wonder why LTTng is not mainline yet. Will Linus Torvalds see the light in 2011?"
What is the goal of your work? Do you want to compare kernel tracing solutions and identify critical features in the process of coming up with a reasonable taxonomy? Do you want to implement something? Do you have a specific application for kernel tracing (e.g. informing performance tuning measures in enterprise environments which would probably be of interest to businesses)? Just throwing together a list of desired features is not going to be of interest to anyone, I guess. You have to come up with a motivation for each of the features, argue why this feature is necessary for the application at hand or for any application of kernel tracing in general, cite literature that gives evidence for your assumptions and conclusions. Maybe if you told the people what kind of work you're interested in and what the interest of your advisor(s) is, in which reasearch context (department, university) you are working, they could make sensible suggestions as to which features might be interesting to you.
Yes, Linus will see the light in 2011. It will probably be in the form of light from a CFL, but it could be the light from sunrise. Heck, unlikely as it may be, it could even be when Melinda Gordon tells him to "go into the light". Anyway, he'll definitely see the light one way or another.
A problem already solved with DTrace on Solaris http://docs.sun.com/app/docs/doc/817-6223
Why does the OP mention the Ubuntu package when the project releases a tarball?
There is no need to make news distro-centric when it does not need to be. The submitter should check to see what other binary packages are available or not mention them at all.
"I believe in Karma. That means I can do bad things to people all day long and I assume they deserve it." : Dogbert
...now be a good boy and have your meds. Otherwise I'll have to call the bad big doctor.
(sorry: sometimes I can't help but feeding the trolls :)
http://docs.sun.com/app/docs/doc/817-6223
Maybe I'm reading slashdot too early on a weekend morning, but I find the last statement of the summary particularly offensive. It seems like everyone who has some sort of kernel widget wants a PR campaign to get it included in the mainline. How about you finish your Ph. D. first and provide some convincing evidence as to why every single person running Linux has to have the tool? The trace tools are available as a package for anyone who wants them now. Why should the mainline be burdened with maintaining the package unless a significant number of users need it?
Atlas stands on the earth and carries the celestial sphere on his shoulders.
Not much point in having a tracing tool if it is owned by Oracle :)
I really liked solaris, I have an Ultra 5 as upgraded as it can be running an old version of solaris 10. Oracle has since decided I don't even deserve to be able to download bios updates, nevermind the OS. Stuff like that makes me want to avoid anything that says SUN on it.
"see the light" - you are making the assumption of it being something that casts light; I suspect Linus Torvalds judges on presented evidence and so far apparently judges the argument hasn't been carried.
Also, I wonder why LTTng is not mainline yet
Well, a bit of searching would have answered your question
The LTTng maintainer has been working for months (years?) to get the kernel tracing into a decent shape. These days the Linux tracing support is wonderful, and not just for LTT - perf, ftrace and systemtap are awesome tools (and more powerful than LTTng in some ways). In fact perf can do all what the web page says and it seems to be more simple for my taste
With DTrace, you have to know what you are looking for in advance, while LTTng can trace in background in flight recording mode and record everything that is going on. Then, afterward you can have all the information you need, and this is invaluable when you have a hard to reproduce bug!
are you talking about a trace or a data analysis tool? if you plan to use LTT to get a trace and then help the user analyse it, maybe you are more into analysis than tracing. then your question could be a bit misleading. Anyway, you would probably end up trying it all out, adding some features to make it all easier to trace as you try to use the existing stuff and analyse the results and so on as you progress. And if you are into trace data analysis (as opposed to tracing) then your domain of kernel trace data analysis is just one application of data analysis. there you need to look into data analysis methods, statistical methods, machine learning, etc. depending on what kind of analysis you like and need. it is somewhat different depending on your goals such as performance data, behaviour trace and analysis, etc.. for some more behaviour related stuff you can look into domains of program comprehension, behaviour analysis and modeling in general, software reverse engineering, specification mining, etc. anyway, at least i would be interested to see some results on this kind of stuff if you go with it and have some means to follow on it and provide feedback..mainstream or not most of this stuff never ends up anywhere or is available at all.
Right. Because everyone knows the best place to develop, debug, and profile code is on a production machine, and the person doing the development should be a system administrator, preferably with minimal experience.
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
Right. Because everyone knows the best place to develop, debug, and profile code is on a production machine, and the person doing the development should be a system administrator, preferably with minimal experience.
I would say many people do know that the best place to understand the performance of a system in production is in production. If the vendors support techs can give an admin commands to run and know that a typo here or there will not result in a panic then that is a very useful feature.
As I recall, Linus has pretty strong opinions on why it's a Good Thing that kernel development isn't "so easy a caveman could do it".
You are confusing profiling with tracing.
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
Right. Because everyone knows the best place to develop, debug, and profile code is on a production machine, and the person doing the development should be a system administrator, preferably with minimal experience.
Shit happens; problems arise. I've had many cases where production was slow, and the problem could not be re-created in QA, DEV, or STG. Or environments where there is no "non-PROD" environment, like my Perforce servers. How many places have a STG or "DEV" server for their central source code system? What if commits were working fine, but all of a sudden they're slow and "nothing was changed".
DTrace is also useful for gathering general statistics on a system with minimal impact. It's accessible via Perl in Solaris as well, so you could write wrappers and such. You could have a version of top(1) which, when run, isn't the top process.
Often the non-PROD environments aren't as large as the PROD ones, so while the code works, it isn't to the scale that the developer thought it would be once it really gets hit hard.
In general, why would you purposely have less visibility on your system/s (regardless of the environment) rather than more?
OK. Now just tell me that you wouldn't chase down the problem, but you would have an inexperienced system administrator do it. Then tell me you would never use the tool in any other way, and I'll concede that the OP's point that the tool was useless was spot on.
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
The profiling/tracing issue is moot. Either way the ability to have robust tracing/profiling/debugging tools that can start from the beginning or attach to a process in progress and safely report as much as possible is great for production environments.
The quickest way to characterize any problem is with low level trace information. Trying to think through all the possible differences between a test and production environment *usually* can produce results eventually, but stack traces, syscalls, and more show the failure so precisely that a reproduction procedure is often trivial.
XML is like violence. If it doesn't solve the problem, use more.
We are waiting for decent kernel tracing since a decade, while LTTng is readily available today. It's better than any other tools like perf, ftrace and dtrace. Microsoft Windows has the Event Tracing for Windows since 2003, and if Linux wants to be taken seriously, it has to be mainline and available without kernel patching. And, I think that users should not be experts to use that kind of tools.
You might be Ph.D student, but apparently you are disconnected from industrial reality. Linux not being taken seriously? Are you f* kidding me? Is that going to be part of your problem statement at the start of your dissertation?
Seriously. You take the cake in the academic hyperbole department.
Yea, maybe it's just un Ubuntu problem, but on both my laptop (Inspiron 700m) and my desktop (with an Nvidia card) crash about 10% on the desktop and 40% on the laptop, when viewing videos and occasionally when listening to music. It's pretty sad, as I need this stuff for my classwork. I pulled an old celeron XP box out of the closet, tranplanted some ram into it, and everything works fine. Pretty sad.
One of the biggest selling points for DTrace is its scripting language. It is extremely powerful and you can find dtrace scripts shared by others that allow you to do very powerful system stats gathering (e.g. here) How about doing something similar for LTTng - you could even do something simple like Lua hooks for LTTng
No confusion here. DTrace is useful for both profiling and tracing. More details here
Slashdot: Put news in, take stupidity out. Zero actually useful comments.
Where does this leave mine?
As is LTT-ng, so I guess your "point" is pointless. Furthermore, more details can easily be found on the lttng website, including a comparison to DTrace, and it doesn't do Linux, making it completely useless as a competitor to LTT-ng. Also, it was your ridiculous claim that such a tool is of no value unless it can be used by an inexperienced system administrator that I was rebutting.
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
See systemtap.
DTrace is Open Source, Free Software (FSF certified), thus, the fact that it's owned by Oracle doesn't really matter much. You don't need Solaris to use it; DTrace is fully functional in MacOS X and FreeBSD (in the latter, userland dtracing is available from 8.2).
You can do the same with DTrace. Read about DTrace feature called "speculation".
You're missing the fact that DTrace is safe to use, so it's impossible to crash the system just by tracing it.
The best place to investigate a problem that manifests itself on a production machine and cannot be easily reproduced on a development environment may be that machine - especially when doing it is safe. With DTrace, it is. With e.g. SystemTap - it's not.
Excellent point in a thread about LTT
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
I certainly agree that you will never crash a Linux system using DTrace.
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
Will Linus Torvalds see the light in 2011?
Oh come on, Linus isn't inside coding all day, every day, you know.
I am not devoid of humor.