Slashdot Mirror

← Back to Stories (view on slashdot.org)

Monitor Linux Performance With The Tools At Hand

Posted by timothy on from the i-t-b-w-t-c-l dept.

Jan Stafford writes "Need to monitor Linux performance without purchasing a pricey diagnostic package? Try these simple, built-in command line tools. This article was written by site expert and author (Rapid Application Development with Mozilla) Nigel McFarlane."

27 comments

  1. Slow news day? by Anonymous Coward · · Score: 0, Offtopic

    Slashdot is pointing links to tip-of-the-day sites?

    Eek.
    maybe we should have a daily link to the latest dilbert

  2. Historic.. by noselasd · · Score: 4, Informative

    Most utilities mentioned are really great, but mostly realtime stats
    sometimes it's nice to see historic view on the machine as well.
    sysstat does just that.
    Now if only I can remember the thing that also use that statistics do
    draw graphs (with gnuplot iirc.) Anyone ?

    1. Re:Historic.. by Monster+Zero · · Score: 5, Interesting
      Ganglia Distributed Monitoring System is a nice tool for monitoring lots of systems, I use it to keep an eye on 66 servers. My current template has plots for Current Cluster CPU/MEM/Beowulf Queue, Cabinet Temperature, Free Disk Space for both RAID units, then shows the utilization of each of the 64 nodes. Its fairly customizable and works great. Shows hour/day/week/month/year plots selected by a drop down menu.

      It uses RRDTool to create the plots - very nice.

    2. Re:Historic.. by sydb · · Score: 3, Interesting

      Is there anything Free around to give per-process historical information a la BMC Patrol ("Draw me a graph of memory utilisation over the last 4 hours for PID 3325, and show me how much of it was in real memory and how much was in swap"). OK the graph is a nice to have, the data would suffice.

      Yeah, while true; do; ps -Fp3325 >> data.txt;sleep 1;done but I'm looking for something... Enterprise... I hate that word.

      --
      Yours Sincerely, Michael.
    3. Re:Historic.. by wideBlueSkies · · Score: 1

      And then if you know Perl, you can always use one of the nifty graphics packages like GD to create your graphs.

      Parse the datafile, plot your lines, and write the .png file. It's been a while since I've used these types of packages myself, but I remember the usage being almost trivial.

      Have fun.

      wbs.

      --
      Huh?
    4. Re:Historic.. by sydb · · Score: 1

      Yes I would be happy doing that. It's just when you speak to people who use the big proprietary Enterprise monitoring suites they kinda scoff at a quick and dirty perl script. Anyway, for my purposes I will indeed probably knock something up in Perl.

      --
      Yours Sincerely, Michael.
    5. Re:Historic.. by chthon · · Score: 1

      SAR!

    6. Re:Historic.. by sydb · · Score: 1

      Does it really do per-process statistics (in particular paging space utilisation)? OK, I'll read the docs properly!

      --
      Yours Sincerely, Michael.
  3. Paging and Swap - Whats the difference? by Anonymous Coward · · Score: 0

    The article mentions there is a difference between paging a swap. Can someone enlighten me on what that difference is? Whys is one OK and the other bad?

    (+5 informative for the best answer)

    1. Re:Paging and Swap - Whats the difference? by Anonymous Coward · · Score: 0

      Paging is good cause that means you still have a job and weren't 'swapped' for someone else.

  4. disk performance? by menscher · · Score: 1, Offtopic
    How about hints for improving disk performance? I've got a raid array that does 100M/s reads, 80M/s writes under Win2k3 Server, but only 40M/s reads, 80M/s writes under RHEL 3.0. Obviously something is screwy with the linux read speeds, but we can't figure out why.

    I won't even start on our network performance problems, which are around 15M/s transfers (our network is capable of 100M/s).

    1. Re:disk performance? by hords · · Score: 1

      I won't even start on our network performance problems, which are around 15M/s transfers (our network is capable of 100M/s).

      When you are saying M/s do you mean Megabit or Megabyte? 15MBps (megabyte per second) is pretty close to 100Mbps (Megabit per second.) Perhaps you are just getting the opposite (bit instead of byte) of what you are expecting in Linux?

      Maybe this is not the case, but I thought I would throw that out there just in case. Also, 100Mbps network cards don't always go 100Mbps. I've seen some that go closer to 30-40Mbps.

    2. Re:disk performance? by menscher · · Score: 1
      By M/s I mean MiB/s.
      By MiB/s I mean mebibytes per second.
      By mebibyte I mean 2^20 bytes.
      By byte I mean 8 bits.

      I hope that clears things up.

      (Our network is gigE, before someone tries explaining that 100Mb/s fast ethernet can't do 100M/s transfers.)

    3. Re:disk performance? by digitalunity · · Score: 2, Informative

      If it is an option for your situation, you should at least investigate a migration to kernel 2.6.x. The IO system is MUCH faster, as is thread allocation. You should see a fairly significant increase in RAID performance, and least for striping. You should also make sure DMA/32 bit mode is enabled on the drives, not all drivers will set this automatically. I'm sure you've checked these, just a mention though.

      As far as the network, it sounds like the driver is setting it to 100Mb mode instead of 1000Mb mode. You might try to find updated drivers for the cards. You might also check the documentation to see if the eth drivers support debug-info dumping. That might be a lot to read through, but you should find your answer in there somewhere.

      If you post more hardware info, I'll post more specific answers.

      --
      You can't legislate goodness. Let each to his own destiny, by will of his freely made choices.
    4. Re:disk performance? by menscher · · Score: 1
      This is for a production server, so we'd prefer to avoid 2.6.x if possible (it doesn't seem to have stabilized quite yet).

      We're using a hardware raid (16 SATA drives that present themselves to the host as a single SCSI drive), which I think rules out the DMA/32 issue (that's IDE-only, right?). In any case, we're getting twice the read performance under Windows 2003 Server without hardware changes.

      The network is definitely at 1000Mb mode. I can even get it to benchmark at 940Mb using iperf (both TCP and UDP). Obviously I should expect some slowdowns due to application overhead, but a slowdown by nearly a factor of 10 seems excessive. As a side note (in the off-chance it's relevant), our server has two gigE cards, channel-bonded. (I know the switch has limitations, but we still like the second for the redundancy it provides.)

      Any advice you can provide for either issue would be greatly appreciated. (We've been beating our heads against this issue for a long time, and are open to try anything. That 2.6 kernel is even looking good at this point.)

    5. Re:disk performance? by codepunk · · Score: 1

      The absolute first thing I would check with network is that if you are connected to a full duplex switch set the card to full duplex in modules.conf . Yes shit is supposed to autonegotiate but it never does.

      --


      Got Code?
    6. Re:disk performance? by digitalunity · · Score: 1

      The hardware support is constantly changing between 2.6.x releases, but if you have the ability to test it, I highly recommend it. The IO scheduling is vastly improved. If you find a 2.6 kernel that supports your hardware and tests without mishap, stick with it and don't upgrade unless you have to.

      If you would prefer to stick with a 2.4.x kernel, you still have other options. Besides checking for updated SCSI drivers, you should check out the man page for 'blockdev'. Tuning your drive read-ahead can improve read throughput in a huge way. Try bumping your read-ahead up about 25%. If throughput goes up and latency is still acceptable, try going further. You will find a sweet spot.

      For networking performance, if one application(iperf) can push 940Mb but during production it only moves 1/10th as fast, I'd look at packet size distribution. A bandwidth testing tool will often check bandwidth by pushing the largest packets possible. If your application is generating a flurry of small packets, then you may be packet-rate bound. One option around this might be to recompile your kernel and look for options relating to performance. What kind of network card are they?

      --
      You can't legislate goodness. Let each to his own destiny, by will of his freely made choices.
  5. stress test for free by cloudless.net · · Score: 1

    1. Setup a Linux web server
    2. Host an interesting story on Slashdot with large JPEGs.
    3. Post the story on Slashdot
    4. Watch your server being Slashdotted!

  6. Remote realtime monitoring? by theCoder · · Score: 1

    Maybe someone can answer this. I primarily use Gnome at home, and like it's system monitor for CPU and memory statistics. However, it only seems to work locally. Is there anyway to get it to display statistics for remote computers, possibly using something like RPC.statd (internal network, so insecurity isn't as much of an issue)?

    I know at work, I can use Solaris' perfmon tool to monitor our Linux server's current stats, but I can't seem to find anything like that for Linux on the client side. Anyone know of a way to do it?

    --
    "Save the whales, feed the hungry, free the mallocs" -- author unknown
    1. Re:Remote realtime monitoring? by Anonymous Coward · · Score: 1

      You could export your DISPLAY to point to your local machine and run wmcpuload from the remote boxes?

    2. Re:Remote realtime monitoring? by ajayrockrock · · Score: 3, Informative

      I use gkrellm2 (www.gkrellm.net) to monitor a handful of machines. It has a ton of stats that you can use and it's very helpful...

      --Ajay

  7. HOWTO Monitor Linux Performance by Spy+Hunter · · Score: 1

    # find /proc | xargs cat

    --
    main(c,r){for(r=32;r;) printf(++c>31?c=!r--,"\n":c<r?" ":~c&r?" `":" #");}
    1. Re:HOWTO Monitor Linux Performance by g-san · · Score: 1

      glad someone mentions /proc when discussing how to monitor performance. all your stats are there. and once you find out your performance can be better, you have to go into /proc to tune anyways.

  8. innacurate by Danny+Rathjens · · Score: 4, Insightful
    No offense, but this story is a joke. Did he even read the man pages for those utilities? The 'stat' in those tools names stand for 'statistics' not 'status'. And don't even get me started on stupid statements like "linux never swaps" and "If paging activity happens all the time, buy more RAM"

    These utilities are explained better in the man pages themselves or the various system administration guides and howtos at the linux documentaion project.

    Oh yeah, and he is missing one of the best tools for this type of thing: namely 'sar', the system activity reporter, which is enabled by default on all redhat distros. (I have an xpostit note dedicated to all the flags to sar for various things)

    As for the graphing/monitoring questions people are asking in other posts; look for tools like nagios and mrtg and sysmon and mon or just search freshmeat.net. It's quite a common task which has been done many ways. My personal monitoring/graphs are perl scripts I wrote to fetch stats via ssh which I plug into mrtg.

  9. Having problems with RH Linux for this by TheLink · · Score: 1

    The version iostat I have on RH Linux doesn't give you disk IO in bytes/s or MB/s. Have to work it out from the blocks/sec. Where the man says a block is of indeterminate size.

    Haven't figured out how to use netstat or some other built-in command to check the bandwidth usage for each network interface. I had to write a little snippet of perl to do that. Any suggestions?

    Seems a lot easier to do stuff like that on FreeBSD or even Windows 2000.

    --
  10. Test Tools by frobisch · · Score: 4, Informative

    This is a good summary of available performance/test tools for Linux:
    Linux Test Tool Matrix

  11. s/JPEGs/Dynamic pages by r6144 · · Score: 1

    You just tested the pipe size rather than the box's 1337-ness.