Slashdot Mirror
Visualizing System Latency
ChelleChelle writes "Latency has a direct impact on performance — thus, in order to identify performance issues it is absolutely essential to understand latency. With the introduction of DTrace it is now possible to measure latency at arbitrary points; the problem, however, is how to visually present this data in an effective manner. Toward this end, heat maps can be a powerful tool. When I/O latency is presented as a visual heat map, some intriguing and beautiful patterns can emerge. These patterns provide insight into how a system is actually performing and what kinds of latency end-user applications experience."
mapping latency in a system using colored maps representing throughput has been a tool of db and network sysadmins for many many MANY years.
How bout you RTFA before you make you're smartass comments, since yours is almost a direct fucking quote from it. However, this isn't about measuring network latency, it's about disk latency, something that until recently was extraordinarily hard to measure.
I guess shouting at systems to make them start working has the opposite effect. Who knew a server was so emotional.
"I use a Mac because I'm just better than you are."
You're = your in this case. Other than that, nice comment.
Have you heard about SoylentNews?
Informative article, all on one page, not chock full of ads. Now excuse me while I stock my bunker.
Back in the day, working at an instrumentation company as a mechanical guy, I stopped to watch the senior electronic design engineer who was doing something that looked interesting. He had an old persistence-type storage oscilloscope hooked up to the rack-mount computer for a new instrument system and was watching the scope display, which was producing some fascinating patterns. Knowing f'all about this stuff but intrigued, I asked him to explain what was happening. He explained (and I'll butcher the explanation with layman's terms) that he was using d/a converters on the high and low bytes of the program address? to drive the x and y axes of the scope, and watching to see where, in the software, that the processor was spending much of it's time. He pointed to a hot spot on the scope display and said that this was where he would concentrate on optimizing his code. Fwiw, I thought that was pretty cool.
These visualizations are used to condense the information gathered on one second intervals from running systems. Any graph of substantially advanced material is going to require explanation until you understand what is being measured, how it is being graphed, and how this information translates in real world performance.
Of course a casual reader from the net needs to read text to understand what is going on. These aren't sales figure pie-charts and shouldn't necessarily be accessible for uninformed parties.
On another note.. Do you think casual readers would have any more success interpreting the raw data files? Anyhow, I am interested in the technique as it is not one I am currently using. With a little practice this may be a good at a glance technique.
The article presented plenty of information related to it's topic. The topic was that using a heat map to describe latency is more useful than simple averages and maximums displayed as line graphs. The article then analyzed certain interesting cases were a heat map had information that would not have existed in a line graph. What you are griping about is that the topic itself is simple and that the article is full of individual analyses that provide support for the topic.
After reading the article, this idea of a "heat map" or frequency distribution mapping (of sorts) can (sort of) be summed in:
A particular advantage of heat-map visualization is the ability to see outliers.
I find this particularly interesting as this graphically now allows a way to "filter" the real outlier out from a sea of data. Also,
Instead of a random distribution, latency is grouped together at various levels that rise and fall over time, producing lines in a pattern that became known as the icy lake. This was unexpected, especially considering the simplicity of the workload.
And concluding the section on what they dub as the "icy lake"...
To summarize what we know about the icy lake: lines come from single disks, and disk pairs cause increasing and decreasing latency. The actual reason for the latency difference over time that seeds this pattern has not been pinpointed; what causes the rate of increase/decrease to change (change in slope seen in figure 5) is also unknown; and, the higher latency line seen in the single-disk pool (figure 4) is also not yet understood. Visualizing latency in this way clearly poses more questions than it provides answers.
Without actually seeing the data or knowing the specifics of latency, from a pure mathematical standpoint I wonder what would result if one treated the set of numbers (from each disk) as a random sequence, identifying outliers (as they did using this heat model)...then graphically mapping those using a "chaos game theory algorithm". By using a graph to statistically analyze/visualize the "outliers", perhaps more could be revealed on the "randomness" of how one disk or a pair of disks reacts relating to the whole system.
I do not claim to know much in this area at all and this is merely speculation on how the set of numbers "randomness" may be approached...
We should start a new Slashdot and return control to the geeks. It actually wouldn't be that hard to get some users to
ACM is a scholarly, research-oriented group. If you're looking for spoon-fed, PCMagazine types of charts and graphs, look elsewhere. Lots of text generally means that someone with brains has to interpret the data, because the interpretation is non-trivial.
I want to delete my account but Slashdot doesn't allow it.
Or female. What's your point?
For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
...it's a shame that instrumentation of things such as EMC's PowerPath are a little painful. I guess there will always be gaps where vendor meets vendor and closed source meets open source, but it remains rather complex to analyse what's happening in Solaris with PowerPath and some Storage Foundation stirred in for good measure. Impossible? No...but maybe we'd all benefit from a little more interoperabilty?
It's a great article though - Brendan's a DTrace authority is impressive.
something that until recently was extraordinarily hard to measure.
Really?
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
hda 0.00 0.00 114.85 0.00 0.45 0.00 8.00 0.73 6.28 6.34 72.87
Where await and svctm are average wait (milliseconds) for the disk & queue and service time for the disk.
Or do you mean something else?
Deleted
I really like ACM Queue, which regularly prints articles for practitioners about things which both we and our more academic colleagues care.
I recommend it, and on rare occasions, contribute.
--dave
davecb@spamcop.net
The data presented in the article are actually quite a bit more subtle and interesting than the summary data you've got there. It's probably be impossible to notice the effects of the "icy lake" phenomenon they describe with average summary data like that, or to appreciate the effect of shouting. (Most IO's happen relatively quickly during the shouting, so the average doesn't skew up very high. What's remarkable about the shouting is the sudden burst of outliers indicating a few accesses with terrible performance.)
Take, for example, AT&T Network performance: /. article)
Current: Snail
Expected, after customers leave in droves over data plan changes: Snail on meth (see yesterday
Expected, once AT&T upgrades equipment: Sloth on vallium
Any insufficiently advanced magic is indistinguishable from technology.
Heat kinda makes me slow too...
Do you actually think the concept of a heat map is new?
A great graph has, (1) a title, (2) labeled x- and y-axes, (3) a 3D figure should also have labels for intensity and the z-axis. All text should be readable or removed. Generally, difficult to interpret figures should have a paragraph below them explaining (a) what they show and--ideally--(b) what the researcher concludes form the graph when this is not obvious.
But almost all there concluding remarks on a figure are, "we don't understand this graph"
Crimey! Slashdot needs a "Hide all comments from UIDs >1M"
"When I am king, you will be first against the wall..."
the summary data you've got there
Funny. I recall the command syntax for that one lets you setup intervals per second. That would be the "black foot" that gets you out of the "icy lake" phenomenon you describe.
"When I am king, you will be first against the wall..."
As a compromise, consider capturing the return address in a timer interrupt and stuffing into a DAC or two. Of course that requires that you have an unused DAC (or two) on board, a timer, and processing time available in the interrupt, and the result probably won't be as smooth as the DAC directly on the bus. Still, if you can do it it's better than nothing.
If you can't, perhaps you can use a different peripheral and some external logic. SPI to a shift register, perhaps? Or I can see having a second processor and sending it serially, then the second processor outputs the DAC.
Anyway, I can see using variations of this technique on some of my projects. Pretty cool idea, really.
Cleverly disguised as a responsible adult.
How about "Hide all comments where $UID > $MINE " ?
And ye shall know the truth, and the truth shall make you free.
John 8:32(King James Version)
That's the point, a good engineer's (or scientist's) response to new data that they can't fully explain is generally unmitigated glee, it means they've found something new. My takeaway from the article is, "try this new technique/tool, you'll see new data".
On another note, I've done some very basic analysis of disk performance at work, and this approach would have allowed me to be much more confident in my results. As it was, basically all I could do when comparing disks and filesystems was use iozone to characterize the "knee points" the article keeps mentioning, and try to map changes in aggregate numbers to saturation of various interfaces and/or devices. This method for actually getting sampling data for latency, and potentially from real workloads even, would have been extremely helpful.
uh, "a good engineer's (or scientist's) response to new data that they can't fully explain is generally unmitigated glee, it means they've found something new." perhaps, but generally you don't ask others to read about it until you understand something about the phenomenon OR it has withstood several attempts to understand it.
You also wrote, "new technique" but what is new? Do you think they invented the heat map, or exploratory data analysis?
Yes -- I do things like these all the time, and I frequently have reason to go: "Oh, our systems behave like *this* in reality?" I wrote an article on the subject (with gnuplot as the visualization tool): http://snipabacken.se/~grahn/gnuplot_kicks_ass/
That wouldn't leave me with very many people to talk to ;)
then you sweep in like a donkey with claims that that didn't just happen...
you are the worst kind of stupid.
if i respond to an AC, and an AC responds back, i will assume it was the original AC until told otherwise. the burden of proof is not on my shoulders... i've already proven i'm the same person. you can't prove you aren't the same AC.
you are NOTHING.
you are NOTHING.
One small sentence and your key signature ? How disappointing. I guess your batteries are running down, by now - too much screaming will make you hoarse.
Shame, I'll have to find another source of amusement now.
What a depressingly stupid machine.