Slashdot Mirror
Solaris DTrace To Be Ported to FreeBSD
daria42 writes "It looks like Sun's famous Dynamic Tracing tool - one of the best features in Solaris 10 - is getting ported to FreeBSD. Sun open-sourced the code back in January and it has been picked up by FreeBSD developer Devon O'Dell. The tool provides insanely great advanced performance analysis and debugging features for server software. Good to see some result come out of the Sun open-sourcing process." From the article: "O'Dell told ZDNet Australia the aim of the project -- which commenced a month ago -- was that all scripts and applications that utilised DTrace under its native Solaris environment should be able to run in FreeBSD with no changes. While FreeBSD's existing ktrace function was similar to DTrace, it was limited in scope, according to O'Dell. 'FreeBSD implements a somewhat similar facility for dynamically instrumenting syscalls for any given application,' he said."
For we that don't have a clue what DTrace is, here's what the has to say: DTrace allows to do performance tuning with applications and troubleshoot production systems--all with little or no performance impact. DTrace provides improved visibility into kernel and application activity, giving the user operational insights with which they can make performance gains..
The no performance penalty sounds really cool to me.
--
Superb hosting 4800MB Storage, 120GB bandwidth, $7,95.
Picaday! Soon to be open "Picture of the day web".
Hosting 20G hd, 1Tb bw! ssh $7.95
"insanely great" is well known. In fact, it's in the Jargon File
Igor Presnyakov stole my hat
if you're referring to http://oprofile.sourceforge.net/news/, you're sadly mistaken. Realtime system profiling is very far behind on Linux compared to Solaris.
/dev/null - Sure you'll see abount 1% cpu utilization, but again, guess work at whats actually using IO.
Can you monitor how much network bandwidth each process uses? -- Sure you can see listening ports and IP traffic, and ntop is fantastic at showing what network bandwidth is used for (i.e. spotting p2p and IM traffic, eg). However if you have a trojen and you see suspecious network activity, there is a certain amount of guess work to try to find the process. Solaris will show exactly what process is making what connection where and the bandwidth it is using.
Can you monitor how much IO utilization each process has? -- No, only IO wait and CPU consumption which is normally enough, but say you have a script thats just reading all content on the disk and redirects it to
Sure you're usually right making an educated guess but system profiling is far ahead on Solaris.
There have been bindings for PHP for a few days, now.
S
(soon to be merged in mainline)
s /linux-2.6.git;a=blob;h=f3ea492ab44dfcfc05e2fad402 7e303f9b69d9dd;hb=caf39e87cc1182f7dae84eefc43ca14d 54c78ef9;f=kernel/kprobes.c
Oh well, it's already there and it seems to have been there for ages: http://kernel.org/git/?p=linux/kernel/git/torvald
Yes, there is: SystemTap by Red Hat, IBM and Intel.
Perhaps you should read
http://groups.yahoo.com/group/solarisx86/message/
and
http://milek.blogspot.com/2005/08/linux-and-solar
Two discussions on some differences between SystemTap and Dtrace. (And yes, both links are in favor of Dtrace, and for good reason it appears.)
Slackware
kprobes is not comparable to dtrace, to see a comparison between dtrace and kprobes check out
dtrace vs. krpobes
systemtap is in its infancy and being designed without safety as a priority, dtrace was created to be 100% safe to run anytime, even in production. systemtap is being made for the kernel hacker to debug the kernel. With possibly some userland probes and safety as an after thought. Sure they talk about safety as a goal. But as documented dtrace_usenix.pdf
dtrace was created from the start to be safe and secure. They even sacrafice some functionality to keep production servers safe. Systemtap is like building a bank they build the building, bring in the money, and desks, and machines, and promise that top of the line doors, windows and safe will top of the line and installed any day now.
There is no overhead when off or need to pre-instrument points to be traced. Dtrace dynamically inserts a probe point into the code path wherever you want it, typically at a function entry/exit point.
The overhead when in use is low enough that you can turn on a blanket Dtrace of all functions in the kernel without killing the OS. If you target your trace points sensibly the overhead is low enough that its not an issue. Its designed to be safe to use, so the Dtrace scripts that do in-kernel filtering can't do anything bad.
I was there too and that was most certainly NOT the reaction IBM gave. There were some Sun guys in the audience that kept harping on the lack of equivalence to dtrace, and the IBM guys repeated again and again that systemtap was in its early stages of development. Apparently it wasn't good enough for the Sun guys in the audience who basically heckelled the presenters throughout. Pretty damned shameful behavior I have to say.
As the guy porting DTrace, I want to clear up a few questions that appear frequently in the comments here. First, though, please be kind to the blog -- it's hosted on our (OffMyServer's) network, which is on a T1. I dunno how bad it got when the story was posted, but just for reference, it'd be nice to not have our network connection die.
:)
FAQ #1 seems to be about the license. Obviously, the CDDL is `viral' in the sense that changes in the code need to be provided under the same terms of the CDDL. In my understanding, this applies only to files in which modifications take place. Extension of something CDDL by adding extra files seems to not require those files to be released under the CDDL. That said, this is a porting effort, and most of the changes I will make will be inside CDDL-licensed files. Thus, anything imported will be under the CDDL. This does not require anything external files to be under the CDDL and thus it can be shipped with FreeBSD without `infecting' other files.
FAQ #2 seems to be whether Sun is happy about this or not. If you have read the article, you would have seen that I've been encouraged to work on this by Sun kernel engineers. Whether Sun as a whole is happy about this is not known to me, but everybody involved in getting it this far has been, so I'm not terribly worried about the rest.
FAQ #3 is about performance incurrences. Certain aspects of DTrace incur performance penalties, but only when DTrace is running. DTrace by itself is a library and a userland tool. All instrumentation is done dynamically and when DTrace is not instrumenting something, no performance hits happen whatsoever. When it is running, we have several advantages to other tools because (unlike e.g. truss) we are instrumenting single processes. Processes which are not being instrumented will not take any performance hits other than the fact that they have a bit less CPU usage since DTrace is instrumenting something.
How do you not take a performance penalty when the profiler is off? You must be root to run DTrace. When you instrument functions inside an application, this is done on-the-fly by rewriting the code that is being executed. When it is not being executed, nothing is being rewritten and it's not even looking to rewrite something. It's just some code resident in memory (in fact, modules are even loaded as needed). It sounds like it might be prone to security flaws, but keep in mind that this has been working in production for a while now.
When will this be in Linux? I don't know. I won't be working on it. I challenge _you_ to do this
Is this vaporware? No. I'm continuing development from about a week off (since I lost my development machine) this evening.
Feel free to ask more questions, I'll try to address them as I see them. But please refrain from bad-mouthing Sun or myself out of spite, jealousy, or whatever. I know it's fun to troll (if you're a troll), but the rest of us really don't appreciate it.
--Devon
www.sitetronics.com/wordpress
Actually crux is closer to ports, and slackware + pkg-src is basically the same thing.
“Common sense is not so common.” — Voltaire
On 1):
/ 2003/eschrock.pdf
Quite a lot, actually. I've talked with Eric Schrock about his thesis work, which was implementing some lock analysis tools using DTrace. This allowed him to detect (very precisely) things like LORs, deadlocks, and the like. His thesis is available at http://www.cs.brown.edu/publications/theses/ugrad
On 2):
When I've seen demonstrations on this stuff, it has been Bryan Cantrill doing fun stuff with libumem, mdb, and DTrace. I suspect that, at the minimum, we'd need libumem to find and fix this stuff with the accuracy that it can be done in Solaris.
Hope this is useful information.
--Devon
www.sitetronics.com/wordpress
That's not a Solaris developer (though I am) -- it's a customer who's been using DTrace for quite a while. He actually knows what he's talking about.
Is there any reason why I shouldn't look at FBSD as if it were a flavor of Linux? Yeah, it has a different kernel. I guess FBSD might be a little faster? That is what the benchmarks say, but the difference isn't staggering. I certainly don't notice. Is it more stable? I haven't had many problems with Linux that couldn't be blamed on cheap PC hardware.
Yes, a very important reason - FreeBSD is not Linux, just as surely as SCO UnixWare is not Solaris. Their codebase is certainly not the same, and in fact FreeBSD's code lineage dates back many years before Linux.
FreeBSD and Linux, being F/OSS systems, share a very large base of F/OSS software, so looking at kde on X on FreeBSD really won't appear that different from looking at kde on X on linux. I could just as well ask why anyone would want to use Linux when it just looks like a derivative of FreeBSD, which predates it. but that would not be a fair assessment because Linux is a seperate work built by another party. Yes, it is a unix-like system. Yes, it strives to adhere to POSIX standards. Yes, it runs all the same software. But no, it is a different system.
I have been using FreeBSD and NetBSD for many years, and where I work all of our stuff is on SuSE. In my opinion, SuSE is impossible to upgrade, its package system is inadequate, and shorewall is a lousy attempt at ip filtering. If I had my way I'd probably replace everything with FreeBSD. But did you notice somehting about the attitude of my opinions? Wasn't your first thought "Well gee, you use FreeBSD all the time and you've probably barely given SuSE Linux a shot?" If it was, you would be right. Because I learned to accomplish tasks in FreeBSD, I favor it - the same way I favor speaking in english over german because english is my native language. I'm sure if you sit down and think about it, when you picked up FreeBSD you tried to do things in the Debian idiom, expecting Debian results. But you didn't get them. So you're underwhelmed. It's natural, but please don't try to attribute it to FreeBSD being an inadequate copy of your favorite system, because that simply is a lie.
On the packages/ports system, I think you've really overdramaticized your plight with the BSD way-of-doing-things. First, you can cvsup the ports tree and compile from source. But you can also use pkg_add to add binary packages. If you don't want to fetch the package tarball yourself, you can use pkg_add's remote fetching feature. Simply pkg_add -r and you're on your way. It will take care of dependencies and the package database will record the package's information. You can also install portupgrade and use it to magically update a port and its dependencies when it is time to upgrade. It's not a difficult or time consuming system to use. I'm unfamiliar with Debian's package system, so I can't make any comments on it, but FreeBSD's package system has always been very useful fo me, and it gets more powerful all the time.
Overall, though, Linux and BSD really do feed from eachother's growth. What's good for one is good for the other. I may use FreeBSD, but that doesn't mean Linux is useless; and the opposite is true as well. All this bickering is really pointless because both projects will continue on in their own directions; some people will favor the one while some people will favor the other. It's simply a matter of preference
perl -e "eval pack(q{H*},join q{},qw{70 72696e74207061636b28717b482a7d2c717b343 637323635363534323533343430617d293b})"
make package-recursive
well here is some more reading on the subject of how systemtap is flawed http://uadmin.blogspot.com/2005/08/systemtap-vs-dt race-debate-continues.html> systemtap debate
/usr/include and the includes in a 2.6.x kernel so
RUNTIME issues:
how do you sudgest you solve the halting problem?
do you have a solution for errant pointers?
and many more.
compile time issues:
You should read this latest tidbit from the systemtap mailing list, basicly it says that systemtap will not understand include files, so if you want to track data in a userland or kernel based struct. So you are then forced to use premade tapsets or expose your system to the unsafe mode. If and when userland probes are created there won't be anytap sets for your apps you created your self.
from the systemtap mailing list
> [...] Shouldn't systemtap be able to handle all of the standard
> include files in
> users can monitor the system? [...]
The debugging information associated with the kernel (and in the
future, user-land applications) contains a form of that information.
We already expose it to some extent, and will probably do so more as
we gain experience. It is unlikely for systemtap to ever have to
directly parse C files such as kernel headers.
These are just a few of the issues, that systemtap faces. currently they are using work arounds that involve using "guru mode" for stuff that should of been dealt with from day one.