Slashdot Mirror


Kernel Tracing With LTTng On Ubuntu Maverick

francis-giraldeau writes "Linux Tracing Toolkit (LTTng) provides high-performance kernel tracing for Linux. This is the killer app for system level debugging and performance tuning. It's now easier than ever to install, with packages released for Ubuntu Maverick. The short introduction to kernel tracing shows how to interpret a simple kernel trace and relate it to strace. I would like to ask Slashdot readers what they would expect as features for a kernel tracing analysis tool, because I'm starting my PhD on this topic and looking for ideas. Also, I wonder why LTTng is not mainline yet. Will Linus Torvalds see the light in 2011?"

23 of 88 comments (clear)

  1. Crowd-sourcing a degree... by PsiCTO · · Score: 2, Funny
    Seriously? You're asking for a crowd-sourced original contribution to (I assume) Computer Science for your PhD? Are you going to defend "our" dissertation live on /. ?

    ;-)

    1. Re:Crowd-sourcing a degree... by maxwell+demon · · Score: 3, Insightful

      Well, I guess his PhD would not be about imagining those features, but about implementing them. He asked for ideas what to implement, not for ideas how to implement it.

      --
      The Tao of math: The numbers you can count are not the real numbers.
    2. Re:Crowd-sourcing a degree... by francis-giraldeau · · Score: 5, Insightful

      The reason is that I would like to make my research useful for tracing users, and I think the best way to do it is to ask people what they really need. I will give credits to those how helped my, why not? ;-)

  2. Goal of the PhD work? by Anonymous Coward · · Score: 3, Insightful

    What is the goal of your work? Do you want to compare kernel tracing solutions and identify critical features in the process of coming up with a reasonable taxonomy? Do you want to implement something? Do you have a specific application for kernel tracing (e.g. informing performance tuning measures in enterprise environments which would probably be of interest to businesses)? Just throwing together a list of desired features is not going to be of interest to anyone, I guess. You have to come up with a motivation for each of the features, argue why this feature is necessary for the application at hand or for any application of kernel tracing in general, cite literature that gives evidence for your assumptions and conclusions. Maybe if you told the people what kind of work you're interested in and what the interest of your advisor(s) is, in which reasearch context (department, university) you are working, they could make sensible suggestions as to which features might be interesting to you.

    1. Re:Goal of the PhD work? by francis-giraldeau · · Score: 4, Informative

      Kernel tracing instrumentation is ready, now we need decent analysis tools. The problem is that there is so much data, that it's hard to interpret them. For the project, I have to come up with something that is new and better that what is already known. For example, we could get a better analysis than bootchart, or auto detect bottlenecks in a system (disk, CPU, memory, network, etc...). There are some work done to integrate userspace and kernel space tracing, virtual machine and host traces, dynamic and static trace points. For a distro, they could record a trace in background and send this information allong with the core dump when an application crash occur. That's all ideas!

    2. Re:Goal of the PhD work? by hAckz0r · · Score: 2, Interesting

      Here is another idea for you. How about hardware assisted "dynamic" (aka dynamically hooked) tracepoints via a custom Xen-like bare metal hypervisor? The OS and therefore its contained malware would know nothing of the inspection process, and best of all it could be OS independent if done at the hardware level. The control/diagnostics software could be running in a VM right next to the OS under test. Boot the hypervisor from CD and then load the original machines OS. Stealth rootkits would be a thing of the past. Simply boot the monitor before loading the OS under test and have a blast uncovering all kinds of malware in any OS of your choice.

    3. Re:Goal of the PhD work? by Lally+Singh · · Score: 2, Informative

      Hello, I couldn't find another way to contact you, so here we are.

      I'm finishing up a PhD in scalability & performance analysis, and have done a lot of work in instrumentation. A userland instrumentation tool is part of my final research. Instrumentation is in a terrible, terrible state -- save a few points of light -- and I'm happy to see someone else in this area!!

      So, as you're starting out, some tips:

      1) If you haven't already done so, investigate dtrace. While available on Mac OS & FreeBSD, it's worth picking up a virtual machine image of opensolaris & playing with it there.

      2) Pick up a copy of: R. Jain, "The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling," Wiley- Interscience, New York, NY, April 1991, ISBN:0471503361. It's my new Bible.

      Good luck, and hit me up if you'd like to chat. For my email address, I'm [my first name].[my last name]@gmail.com

      Cheers,
      -ls

      --
      Care about electronic freedom? Consider donating to the EFF!
  3. That it is safe to use in a production environment by Dug · · Score: 5, Insightful
    Not that much point having a tracing tool if an inexperienced admin cannot safely use it on a live system which has a problem.

    A problem already solved with DTrace on Solaris http://docs.sun.com/app/docs/doc/817-6223

  4. Ubuntu Only? by hduff · · Score: 3, Insightful

    Why does the OP mention the Ubuntu package when the project releases a tarball?

    There is no need to make news distro-centric when it does not need to be. The submitter should check to see what other binary packages are available or not mention them at all.

    --
    "I believe in Karma. That means I can do bad things to people all day long and I assume they deserve it." : Dogbert
    1. Re:Ubuntu Only? by francis-giraldeau · · Score: 2, Insightful

      Binary packages are easy to install, that's it. I don't know of other LTTng integration inside a distro. If you prefer patching your own kernel and compiling tools from git repository, you're free to do it.

    2. Re:Ubuntu Only? by compudj · · Score: 3, Informative

      So far, LTTng has been mainly integrated in embedded distros: WindRiver Linux, Montavista Linux and STLinux currently ship with LTTng. The interesting news that is particular about Ubuntu here is that, by installing the LTTng packages from PPA, it is now possible to easily deploy the LTTng kernel and userspace tracers on a desktop-oriented distribution.

  5. Re:That it is safe to use in a production environm by Dug · · Score: 2, Informative
    Another go this time with a working link

    http://docs.sun.com/app/docs/doc/817-6223

  6. See the light? by codegen · · Score: 4, Insightful

    Maybe I'm reading slashdot too early on a weekend morning, but I find the last statement of the summary particularly offensive. It seems like everyone who has some sort of kernel widget wants a PR campaign to get it included in the mainline. How about you finish your Ph. D. first and provide some convincing evidence as to why every single person running Linux has to have the tool? The trace tools are available as a package for anyone who wants them now. Why should the mainline be burdened with maintaining the package unless a significant number of users need it?

    --
    Atlas stands on the earth and carries the celestial sphere on his shoulders.
    1. Re:See the light? by francis-giraldeau · · Score: 2, Insightful

      We are waiting for decent kernel tracing since a decade, while LTTng is readily available today. It's better than any other tools like perf, ftrace and dtrace. Microsoft Windows has the Event Tracing for Windows since 2003, and if Linux wants to be taken seriously, it has to be mainline and available without kernel patching. And, I think that users should not be experts to use that kind of tools.

    2. Re:See the light? by Anonymous Coward · · Score: 3, Insightful

      "if Linux wants to be taken seriously..."

      Funny the one thing needed to be taken seriously is, by magic, the subject of your thesis.

      Had you been working on, say, resizable ramdisks (I'm just making this up), then resizable ramdisks would have been the one thing needed in Linux for Linux to be taken seriously.

      Ever considered humility?

    3. Re:See the light? by h4rr4r · · Score: 2, Insightful

      Linux is taken quite seriously and to use this sort of tool and not be an expert is pointless.

    4. Re:See the light? by Anonymous Coward · · Score: 2, Interesting

      Actually ... While I was the maintainer, IBM's had a team of people working on LTT for a period of 3 years before pulling the plug on their involvement because they saw that all the money they were pouring in there wasn't leading to a mainlining.

      Why were they interested in kernel tracing? Well ... When a customer of theirs has one of his 10,000 servers misbehaving in production, they can't afford telling him to just take it offline for diagnostics. They have to find (and fix) the problem in the field. There are very few tools that allow you to do that. Oh, and having the source code and being able to rebuild is just not an option in those cases. After I passed on maintainership, Google did some work on kernel tracing with the LTTng developers with goals very similar to IBM's: misbehaving machines in server farms should not need to taken offline for diagnostics.

      As for shooting the performance, I suggest you read up on LTTng's literature. The current team has done a stupendous job at deconstructing that myth.

      You're right, Linux is being taken very seriously. Hence the need for these kinds of tools.

      Thanks,

      Karim Yaghmour

  7. Some googling by diegocg · · Score: 4, Insightful

    Also, I wonder why LTTng is not mainline yet

    Well, a bit of searching would have answered your question

    The LTTng maintainer has been working for months (years?) to get the kernel tracing into a decent shape. These days the Linux tracing support is wonderful, and not just for LTT - perf, ftrace and systemtap are awesome tools (and more powerful than LTTng in some ways). In fact perf can do all what the web page says and it seems to be more simple for my taste

    1. Re:Some googling by diegocg · · Score: 2, Informative

      Well, ftrace has a lockless ring buffer. And eventually all the ring buffers are going to be unified...

  8. Re:That it is safe to use in a production environm by francis-giraldeau · · Score: 2, Interesting

    With DTrace, you have to know what you are looking for in advance, while LTTng can trace in background in flight recording mode and record everything that is going on. Then, afterward you can have all the information you need, and this is invaluable when you have a hard to reproduce bug!

  9. trace or analysis tool? by hraponssi · · Score: 2, Insightful

    are you talking about a trace or a data analysis tool? if you plan to use LTT to get a trace and then help the user analyse it, maybe you are more into analysis than tracing. then your question could be a bit misleading. Anyway, you would probably end up trying it all out, adding some features to make it all easier to trace as you try to use the existing stuff and analyse the results and so on as you progress. And if you are into trace data analysis (as opposed to tracing) then your domain of kernel trace data analysis is just one application of data analysis. there you need to look into data analysis methods, statistical methods, machine learning, etc. depending on what kind of analysis you like and need. it is somewhat different depending on your goals such as performance data, behaviour trace and analysis, etc.. for some more behaviour related stuff you can look into domains of program comprehension, behaviour analysis and modeling in general, software reverse engineering, specification mining, etc. anyway, at least i would be interested to see some results on this kind of stuff if you go with it and have some means to follow on it and provide feedback..mainstream or not most of this stuff never ends up anywhere or is available at all.

  10. Re:That it is safe to use in a production environm by Dug · · Score: 5, Interesting

    "Not that much point having a tracing tool if an inexperienced admin cannot safely use it on a live system which has a problem. "

    Right. Because everyone knows the best place to develop, debug, and profile code is on a production machine, and the person doing the development should be a system administrator, preferably with minimal experience.

    I would say many people do know that the best place to understand the performance of a system in production is in production. If the vendors support techs can give an admin commands to run and know that a typo here or there will not result in a panic then that is a very useful feature.

  11. Re:Because by larry+bagina · · Score: 2, Interesting

    He has expressed similar sentiments more recently as well (eg from 2007 on git's use of c vs c++)

    C++ is a horrible language. It's made more horrible by the fact that a lot of substandard programmers use it, to the point where it's much much easier to generate total and utter crap with it. Quite frankly, even if the choice of C were to do *nothing* but keep the C++ programmers out, that in itself would be a huge reason to use C.

    --
    Do you even lift?

    These aren't the 'roids you're looking for.