Slashdot Mirror
Solaris' Dtrace in Detail
paulkoan writes "The Register has a further details about the new Dtrace systems utility bundled with Solaris 10, along with pictures of the authors, and user testimonials. It also highlights Suns vague assurances that this (if it lives up to the hype) amazing utility may or may not end up in the public domain."
It also highlights Suns vague assurances that this amazing utility may or may not end up in the public domain."
so, we are assured of a and !a ???
Trolling using another account since 2005.
It looks like it's an allround system monitoring and administration tool. But how is it different/better than a well-sorted collection of individual tools, each doing one job as good as possible?
Dtrace is a lovely mechanism, and once it's out in production Solaris (and maybe Linux) I'm going to write a new TPS/response-time monitor using it.
davecb@spamcop.net
This article seems like it was written by Sun. I bet it's one of them junket articles, Sun must be a major sponsor of theregister.co.uk or something...
The article says what this D-Trace does, but yet, doesn't... Not to mention it's neither objective, review like, and even fails to mention alternatives or relative tools
Definately one of the poorest software article's I've seen in a while
Slashdot is making me bitter (-;
Complete CCTV Security Cameras
Gamblers Forum
Bingo!
In fact, you can bet your life that it will or won't show up in public domain.
Now this is what I call a nice and reputable company! At least they keep what they promise!
So you're using Solaris 10 then, yeah?
Sounds great and I am quite impatient to see how it works (take a mental note to increase the priority of the Solaris upgrade on one of my toy SPARCs). I can see a growing need for a non-obtrusive profiling tool. As a matter of fact I started writing something which may be similar to dtrace for Linux and it is for a long time in my TODO to port it to Solaris and maybe IRIX.
An excellent article at sys admin mag.
I think pkgadd may be the command you're looking for squire :-)
No, linux does not already do this.
Solaris users could not care less whether Sun ports it to linux or not.
Sun are *not* evil because they don't immediately give away the source to a product which has taken them years and $$$s to develop and which gives them an edge in a competetive market.
Some people just prefer Solaris. If you prefer linux, that's fine.
"DTrace is a quantum leap over previous tools"
Doesn't that mean it is a very small, randomly occuring change? Silly marketing speak.
So, one guy uses this tool to analyse function calls in a program. Other times, it is used to analyse whether tools like Oracle or BEA are making system calls.
This is exactly what a profiler is for. Doesn't Solaris come with profiled libraries you can link your code to to do the same thing with something like gprof? Aside from the fact that this tool works with non-profiled binaries, I fail to see the revolutionary aspect.
I bet someone has to DTrace his slashdotted server real soon now..
So if some software isn't in the Gentoo portage tree you won't run it on your Solaris boxes at work?
This is proprietary software for a version of Solaris which isn't even out yet.
Remind me not to employ you as a Solaris sysadmin. Hell, remind me not to employ you to make the tea - we might have different cups at work and you wouldn't know what to do with them.
Chernobyl 'not a wildlife haven' - BBC News
No, we're vaguely assured that [i]true[/i].
It's a hugely complicated problem and it doesn't have a simple solution except in the minds of markedroids.
I'm not sure how much of an indicator the open sourcing of looking glass is for what Sun decide to do with dtrace. Looking Glass never seemed to be anything more than a nice piece of eye candy to showcase the java desktop (the real product they were flogging). dtrace on the other hand looks like being one of their biggest pieces of product differentiation for Solaris in years. We all know how finance departments like those transactions-per-second->cost-per-transaction figures in making procurement decisions...
I suppose we'll have to wait and see...
Grr. No-one understands what I meant.
Get your own free personal location tracker
With statements like
:-)
"assurances that this (if it lives up to the hype) amazing utility may or may not end up in the public domain."
how can we go wrong.
Not to be the devils advocate, but Microsoft has offered excellent detailed profile management via perfmon for quite some time now. You can inspect disk IO per-processes, memory per-processes, page faults, etc. etc.
It's a bit of a misnomer to characterize dtrace as the first application for doing this, although I will say it's among the first that I have seen that is scriptable (a huge advantage when troubleshooting problems remotely).
Eric Sarjeant
eric[@]sarjeant.com
Tunnel vision on a sysadmin, (I don't care if you're taking care of Windows, Linux, Solaris, BSD, or Mac for fucksake) is a bad thing [tm].
Sun's vacillation about open source reminds me of the philosophers in the Hitchhiker's guide.
``But who the devil are you?'' exclaimed an outraged Fook.
``We,'' said Majikthise, ``are Philosophers.''
``Though we may not be,'' said Vroomfondel waving a warning finger at the programmers.
``Yes we are,'' insisted Majikthise. ``We are quite definitely here as representatives of the Amalgamated Union of Philosophers, Sages, Luminaries and Other Thinking Persons, and we want this machine off, and we want it off now!''
``What's the problem?'' said Lunkwill.
``I'll tell you what the problem is mate,'' said Majikthise, ``demarcation, that's the problem!''
``We demand,'' yelled Vroomfondel, ``that demarcation may or may not be the problem!''
``You just let the machines get on with the adding up,'' warned Majikthise, ``and we'll take care of the eternal verities thank you very much. You want to check your legal position you do mate. Under law the Quest for Ultimate Truth is quite clearly the inalienable prerogative of your working thinkers. Any bloody machine goes and actually finds it and we're straight out of a job aren't we? I mean what's the use of our sitting up half the night arguing that there may or may not be a God if this machine only goes and gives us his bleeding phone number the next morning?''
``That's right!'' shouted Vroomfondel, ``we demand rigidly defined areas of doubt and uncertainty!''
They seem to have a lot of praise from users, not just market speak. And technical users yet.
I'd be very, VERY surprised if Sun allows Dtrace into the open source world, at least not for a while. If Dtrace really is the supertool it seems to be, and is actually and massively UNIQUE, it represents a reason peole will move to Solaris and buy Sun's hardware to do it. Maybe the closest thing to a Unix killer app that has existed for a while.
Now eventually (as in five years down the road) it will probably pay for Sun to open it up. If I were them, I'd milk it for all its worth on the "get people to move to Sun boxes" mantra, while the rest of the world trys (and probably fails) to duplicate the tool. Then, when Sun has gotten all the converts they are likely to, start making the tool even BETTER by opening it up and letting the world go to town on it. (GPL or something similar so Sun can incorporate back in the goodies.)
Of course, that's just an off the cuff theory by someone who doesn't know.
"I object to doing things that computers can do." -- Olin Shivers, lispers.org
Sun's "collect" program that ships with their compiler suite already does this. No special compile or link step is necessary.
Dynamic Instrumentation of Production Systems - this paper was presented at Usenix, and describes how Dtrace is actually implemented.
Dtrace user guide.
A collection of Dtrace scripts
Grr. No-one understands what I meant.
Which is what - you don't have an opportunity to play with it at home? But you do - you can get a Solaris 10 preview through their Sun Express program (linked in the article) and install it on another box, in VMWare, etc.
So what did you mean then?
That emerge should give you a "why you should use dtrace" document?
How does Dtrace for Solaris differ from Kprobes and Dprobes for Linux ??
Maybe Compaq patented it? HP could sue Sun. Wouldn't that be fun?
Give system catchy name, vaguely reminiscent of Eminem's rap group
I'm sure they did this to appeal to the disproportionately-large numbers of African-Americans in the field of computer science.
Not sure what all the hype is about, DTrace sounds alot like HP's Measureware and PerfView. Bundling it with the OS for no addition cost, that sounds good to me.
Slashdot - Where the slash is most definitely to the left.
Well you CAN get Solaris 10 preview from the Express program. It's free for non-commercial use and available for x86. Downloads available here. Check out the hardware compatibility list for x86 here (don't see one specifically for Solaris 10 Express, probably the Solaris 9 list applies).
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
The Kernel Instrumentation Process has already started creating the foundation for a kind of dtrace functionality. The profiling mechanisms appear to echo some of what Sun has done to Solaris to enable this kind of process-less profiling. Hope someone is still pushing this along...
DaGoodBoy
My God! It's full of Voids!
- The execution path is different between the profiled and non-profiled library, so you can not be 100% assured you are getting the same result.
- To engage a profiled library you have to stop and start the target process, and often play all sorts of pre-load and path games.
- Looking into the kernel requires even more invasion, and can mean rebooting the server.
Basically, the idea here is that you don't have to use a possibly "different" library, and you can instrument code on the fly (without stopping the process or the box). This is really nice for production boxes where any interruption or change of software is a really big deal.My head explodes when I read marketing speak, what does it do? what is it?
(any answers along the lines of it DTRACES your OS etc will make me come to your mothers house and explode my head all over her prize dinnerware set)
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
it gives you access to all Solaris' performance counters, system call tracing, etc. and gives you a command interface where you write simple scripts that easily manipulate and present that data in a way which is useful to your performance tuning or debugging purposes.
/dev/kmem.
You could derive work-alike programs for top, sar, ps, etc. etc. using simple dtrace scripts.
And a lot of the statistics would be normally nearly impossible to get without debugging the kernel or poking around in
By putting it all in one place, and putting a nice interface on it, all of that is in the past.
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
This already exists in the linux 2.6 kernel (I think it may be back ported to the 2.4 as well).
Its called OProfile.
While not identical to DTrace it lets you perform the same task.
And another advantage of open source - if you do find a resource hog, you can go in and "make it right".
Heck, for really silly bottlenecks you could fix it in hours/days rather than months.
[ Monday is a terrible way to spend one seventh of your life. ]
I don't think Sun's open sourcing the thing will help Linux users much anyway, the kernel design is probably very different, with more evolved semantics. Which means Linux developers will hae to do the work themselves if they want their admins to have these facilites.
Note that this is finally a tool to resolve finger-pointing *it's your app that's slow - no it's the kernel etc* . I guess IBM will find it advantageous to create a similar Linux tool for their systems, as they bill Linux as an application serving environment. So we should see it soon on our own boxen.
The sad thing is this proves once again that closed source companies retain an edge at innovating.
This is not a signature.
actually that's not true. Kerninst does exactly that.
Kerninst is a framework for dynamically splicing code into a running kernel, almost anywhere, anytime. Code can be removed and changed at will. Kerninst works on standard (unmodified) Solaris and Linux kernels, *no* kernel re-compilation is necessary.
I think your point was that wide availability of a tool will increase the number of people using it and thereby increase sales of (in this case) Sun products. While that is true to an extent I think you're hoping for a bit much for Sun to port it to Linux. Sun are a business. They have an OS and boxen to sell. They are generally a community-friendly company, sure, but if dtrace is as useful as they say then I can see them wanting to keep it proprietary and use it to draw people in to buy their OS and their boxen.
I'm not even sure how well other OSes could use dtrace, at least without a fair bit of work. It sounds to me like it's pretty tightly integrated with the OS (for obvious reasons).
And anyway, you can use it. You can download Solaris for x86 for free and get used to the tools that way. So basically they are doing what you want, but with their entire OS instead of just this one tool.
I'm so nice I'll even show you where you can get it. Right here.
Chernobyl 'not a wildlife haven' - BBC News
I'll have to start using that in my conversations about lusers. "I wouldn't call him stupid; let's just say he doesn't have a good relationship with his brain..." :)
Sinepaw.org: Grape Winos
Aside from the fact that this tool works with non-profiled binaries, I fail to see the revolutionary aspect.
try reading some of the related links (like the bigadmin dtrace site, the usenix paper, the com.unix.solaris post, or the sys admin magazine article) and you'll quickly realize why this is a revolutionary tool.
the profiling facility is only one of the information providers that dtrace has.
other providers include:
- a function provider (allows you to monitor calls into any kernel function entry or exit point )
- a system call provider
- io provider
- process scheduler provider
- process provider (allows you to place probes in userland processes.)
- the list goes on
these providers taken together allow you to place probe points all over the kernel and userland processes where each probe point can return whatever information you want. (ie, you can get function argument/return values, you can access globals, you can troll kernel data structures, you can collect you own statistics, etc.)
that's a lot of potential information, so dtrace provides you with an easy programmatic way to drop data your not interested in. how about only collecting data when a particular function returns an error after being called from a particular process with a certain parameter set. easily done in dtrace.
then your simple dtrace script can correlate information from all these providers and display it clearly. (with a language which was designed to make simple statistical analysis easy via built in functions for aggregations, averages, etc.)
and of course the best part is that you can do all of this on a production system, running production software, and under normal load.
obvious disclaimer: i work for sun.
There's an old saying, "you are what you hold yourself accountable to".
In that sense, Sun is no Microsoft. They are a hardware company that provides services, plain and simple. The only reason why they are cold to Linux is because it pits Sun servers directly against x86 commodity PC's - otherwise they are all for it.
Once the dust settles on their bread and butter revenue stream, you can better believe that they will be open-source all the way. But right now they need to force some differentation with their hardware because in most cases they simply can't compete against an x86 farm in the server space. All the rest of the BS about new accomplishment is just propaganda, I would ignore it.
..finally Sun gets innovative again. I guess Solaris
10 will be the best release since 5.7..
Sigh, so now the PHB will be running this and telling us what to fix.
I know I would pay big dollars, say like $28, for a box that ran infinitely fast.
This could be the diet pill for software bloat that we've all been looking for.
You have failed to define statement a. Try again.
(slightly offtopic, but is pertinent to this thread) Wasn't diversifying hardware and software one of the ways that Akamai defended itself against the massive DDoS attack?
Why don't they replace solaris with linux? They should start becoming more mainstream.
Sounds just like Shark for Mac OS X.
. ht ml
If DTrace is better than Shark, it must be an awesome tool. Shark does everything this article describes and far, far more:
http://developer.apple.com/tools/shark_optimize
The 4.0 version of Shark shown at WWDC was even better. In many cases, Shark can actually give you specific directions on how to optimize your code: both at the source and instruction level.
so, we are assured of a and !a ???
Sun has a long history of vacilating on whether and how to open their products. Hardware interfaces, SPARC-based hardware cloning, various software packages, and so on. Often this means half-opening, licensing and failing to renew, and so on.
It seems to be different every time. But the common thread is that they never commit. (I think the closest they came is Open Office where at least we have a fork.)
It was due to their refusal to open things sufficiently for me to hack my Sun hardware and software that I switched to Linux a few years back (and swore off anything but open source for my home machines), rather than trying to upgrade my Sun systems for the coming millenium.
Came Y2K not a lot of stuff broke. But I shut down the last two anyhow. Too late, guys!
(And too late for me too, apparently. Things have been so busy in the day job that I haven't had a chance to actually DO any hacking at home - except on the houses themselves. B-( )
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
... by the article's use of the word 'instrument', as in "Users have more places to instrument a box and an easier way to do so.". I had never heard it used as a verb before. So I looked it up in the dictionary:
tr.v. instrumented, instrumenting, instruments (-mnt)Can somebody tell me which of these definitions is meant in the article?
"You can download Solaris for x86 for free"
No you cannot. You can download it for no money but it certainly isn't free. Please stop confusing "free as in speech" and "free as in beer", it only confuses the newcomers to Slashdot.
After RTFA DTrace remind me of PTrace(3) (along with strace). I think ptrace can do the same thing, and more..
Alright brain. You don't like me and I don't like you, but let's just do this and I can get back to killing you with beer.
I try to use beer as a motivator for my brain (Do this and you won't have to think for a little while). I don't think it works.Homer
Looking at that Apple blurb (and reports I heard from WWDC attendees), Shark is a performance tweaking tool - a profiler. If you had ever bought the heavy-duty Sun compilers, you'd have a functionally similar set of tools (collect, analyzer, and cc with various flags that generates annotated source). Admittedly not nearly as polished as Apple's tools (and not as cheap), but Shark is also several years newer - and having done exactly the optimization that article describes on my own code with the Sun tools, I assure you they have been around for quite a while.
However, DTrace is NOT an application profiler - though it can be used for that. And claiming that it is a profiler would be like claiming C is just an extension to assembly language. It could be used as such, but you'd be ignoring all the power of the tool.
DTrace is far more powerful: it is a whole system profiling tool. It can give debugging data, performance data, or whatever, for any process, thread, driver, kernel module, or other system component. It includes a moderately complex programming language - YOU get to write code that runs in REAL TIME within any code path anywhere in the system. And, you get zero performance loss in any part of the system that isn't actively being traced.
The DTrace manual is extremely informative. Of course, actually trying it on Solaris 10 Beta is even more so :-)
A witty [sig] proves nothing. --Voltaire
"I wouldn't call him stupid; let's just say he doesn't have a good relationship with his brain..."
Nah, George Bush and Karl Rove are getting along just fine.
Weren't the Phiilosophers in this story eventually placated by the fact that they would be neeeded to explain various possible reasons for the machine's answer?
In english, or is generally used to specify one or the other, but not both. So, they where really saying 'a xor !a'. Granted, this still evaluates to true.
And it still dodging around the issue. I do enjoy how project looking glass (cool but fairly useless) was used as an example of the trend of Sun to go open source. I guess, as people speculated, Sun decided to dig through their stuff, find something 'sorta cool' and throw it to the public saying 'see, we're trying.
The Neo-Bohemian Techno-Socialist
I have this somewhat funky, older computer - it's a somewhat (1.5 Ghz) speedy Athlon, but the thing is it has this somewhat older Seagate 10K RPM SCSI hard drive in it. I kind of pieced this computer together from parts here, and parts there.
So of course, I had to install Gentoo on it. I did. It took a while. I burned out a stick of RAM. I am afraid of Gentoo. But anyway, I got it up and running, but I had to disable tagged queueing on the disk to keep it from freezing up all the time. So here I am, with a functional Gentoo machine, and this 10,000 RPM disk is very, very loud. You can hear it in the next room when it does an updatedb or anything like that. With the ext3 filesystem, I do believe that it was doing what you call "thrashing". Just really loud, scary type of many tiny ball bearings in a coffee can type noise. Well, I had had enough, and Gentoo really creeped me out. I didn't like the idea of a journaling filesystem that fscks every now and then and you never know what the fsck is going to find. The first fsck it did (after some 20+ reboots over the course of a month or two) required a reboot because of some inconsistency it found. Just not cool at all - maybe I am paranoid, but I would rather have the thing fsck right away - and I know you can set it that way, or set it not to fsck at all, but in any case - the real drag was how loud the disk was, and that it appeared to be thrashing when certain things were going on - make installs, file extracts, updatedbs. I didn't like it. It creeped me out.
So I installed FreeBSD 4.10. I love FreeBSD. And you know something, the disk is really, really quiet now. I went back and re-enabled tagged queueing. No problems. Same disk, same controller, different OS. The disk sort of sounds like a welder now (like you are welding something). It's not loud, it doesn't thrash, and the power got cut once, but it fsck'd right away and you know you're good to go immediately, not waiting for the next 28th reboot to find out there is something horribly wrong with your filesystem.
So it's really amazing to me, how much quieter the disk is with FreeBSD. I know it's a loud disk, it's kind of a funky computer. It even has a funky heatsink. But the difference is like night and day.
So in this case, switching from ext3 to ufs solved a problem that I had. The disk I/O is actually a tad slower with ufs, it appears, according to bonnie++, which I did on both OS's... but the disk is so much quieter that I can hardly believe it. I would have never imagined it.
So sometimes, it may not be so much that you know what is wrong, or where the problem is coming from, but it's your ability to do something about it. In my case, I chose ufs, because I didn't want the disk to thrash, even though I was getting just slightly over 1MB/s greater disk I/O with ext3. This is not something you can just do overnight, it required a complete OS reinstall.
Right now, though, it is just a really wonderful computer to use - my last uptime was 10 days, 10 hours, and 10 minutes and 39 seconds. (storm was coming in and I don't have a UPS). Wonderful, stable, smooth-running, and _somewhat_ quiet system.
But seriously, I still haven't figured out what it is about Gentoo that creeps me out. I wonder if Dtrace could help me out with that one.
Dtrace does sound cool though, I think it can be very useful, provided that you can make the changes that you need to make without reinventing the wheel, or refactoring your entire program. Maybe there would be some way that Sun could sell it to people, or port it to other OS's and let people buy binary versions of it for a nominal fee?
Or I guess I'm asking, how complex can these functions get? Would it be feasible to tie it in with other things so I could do my own interface to it? Sounds like fun if I can...
Yeah.. stop confusing that.. That sentence says it can be downloaded at no cost and not that it's free to use any way you want.
Fuck the what?
Give system catchy name, vaguely reminiscent of Eminem's rap group
I'm sure they did this to appeal to the disproportionately-large numbers of African-Americans in the field of computer science.
The majority of Eminem's fan base is white.
Its a dessert topping, AND a floor wax.
I was reading the examples of using DTrace to spot performance issues.
It seems to be that most of them could have been caught with code reviews.
Perhaps with tools like DTrace, you can spot performance issues more quickly. But I believe a good code inspection regime could catch these problems and more.
...with HP-UX, which has had MeasureWare and PerfTools for years. HP even makes a version available for Solaris (and has for some time), as well as AIX and Windows (and probably Linux by now, but I don't know for sure).
I know this looks like a shameless plug, but I hate it when companies push some mundane thing as being "revolutionary" when it has basically been around for over a decade. Hey Sun, you know what be revolutionary? Open source Java. End of rant.
Way to miss the joke there, Poindexter.
So how exactly does Dtrace benefit application optimization? I code simulation programs to model physical systems. The applications can be CPU or I/O intensive and I'd like to know if Dtrace will help me optimize my code more than it already is.
;)
Can it suggest areas of vectorization, loop unrolling, parallelization, etc?
Does it primarily give you stats on usage of functions, disk I/O, etc?
The numerical methods I use are probabilistic, making them very hard to debug sometimes.
(Imagine a bug occurring only 1e-4% of the time, although it will always occur before you get the answer you want.
Thanks for any info!
Favorite
Perhaps it has to do with how much warning for bottleneck fillage is available in the real world situations encountered? It sounded to me like this was a sort of ASAP kind of fixing-it problem.
:-(
Obviously your method would be great to pre-emptively schedule across all systems, but I suspect in a typical shop it might be one of those things the sysadmin means to do when the fires are all out.
-josh
I am begining to like professional unix more and Linux less.
I am about to try out solarisx86 and my asus motherboard, nic, and video card all happened to be supported with Solaris 9 oddly enough.
I am interested to give this thing a spin.
http://saveie6.com/
Can I get Visual Dtrace for my NT boxen?
"That sentence says it can be downloaded at no cost and not that it's free to use any way you want."
No it doesn't. It says that it's free when it isn't.
And they don't even redo it well. Now, if dtrace were user/admin friendly, I could see why a new tool had to be reinvented. But dtrace is a step above writing a c program to use probe system calls - no imagination when it comes to what is useful to the user community.
No doubt, dtrace is a powerful tool, but so is crash and adb and just as user-unfriendly. Sun should read the technology history books and listen to the user community before they tout capabilities that have existed in other OS's for decades.
Instead they display their infancy to user-friendliness and mature OS capabilities. Kind of scary for an OS that's been around for 20+ years.
dtrace appears to be another Sun internal science project to make it easier for Sun's own engineers to figure out what's going on in their OS. Their engineers, being young pups, have no clue that what they "invented" is not unique and pound their chests to display their inexperience in relating to the business world and user community.
While IBM is touting technology that allows computers to read brainwaves, Sun is happy to give hackers yet another system monitoring tool (YASMT) and makes it front page news - no less. This is the type of tool IBM and others treat as ho-hum because they have much more relevant technologies to show off.
Actually, MeasureWare sounds like the SE Toolkit (http://www.setoolkit.com/), formerly known as "Virtual Adrian", and now managed by Rich Petit. Also, Sun Management Center has some pretty good performance data aggregation capability.
Also, is MeasureWare similar to SGI's Performance Copilot?
I'm glad to see authors of software posting.
Mark my word. Solaris 10 will be the biggest product innovation from Sun since the E10K, not just the biggest Solaris rev since Solaris 7.
Yes, nice, but will this tool help with improving Java app performance...?