Slashdot Mirror
Ask Slashdot: MRTG and IP Accounting
Webdude asks:
"I run a server that has many IP aliases and have found a
very strange thing: all the aliases receive data but all
data is sent out through eth0. I have
MRTG up and running but it doesn't help me because all
traffic is going out eth0. I set up IP Accounting and found
that it records the packets traveling properly but now my
big question is how do I get MRTG (or something similar)
to graph the stats that are in the IP Accounting tables???"
First of all: Not an anonymous coward, just too lazy for registration (now) -- niedner@toppoint.de.
Your Posting seems to be the first relevant. I, too, had this problem a while ago, and since I found it imposible to solve with my means, I gave up: Virtual hosts do work despite of this, so who cares? But, to the kernel gurus: Is there a way to instruct the kernel to send an answer back via the same interface the "got" the packet? What happens if a specific process just binds to one (of the many) IP? If this process anwers, does where does the answer originate from?
I do believe that his situation is one interface with many aliases binded to it, I dont really see the point to load balancing across one interface.
However, linux does support "balancing" via the beowulfs project ethernet bonding code.
-d
Not quite right. MRTG can both use SNMP and shell scripts to gather data to draw the charts. I have seen a script which filters data from ip accounting so that mrtg can use it.
You can search for this on the mrtg mailing list archive or on Dejanews.
I dumped this script after I installed snmp agents to gather the data, which is a better solution IMHO.
cheers,
Faiz
faiz@dhiraagu.com.mv
I have written a simple set of programs which helps me to create MRTG graphs on one server and collect IP accounting information from several Linux routers. Each router has a simple daemon (invoked by [x]inetd) which shows IP chains. And a servers has a TCL scripts which collects this data, group them basing on a simple configuration file and displays in some form, depending on command line parameters. One form is designed for MRTG. This script also caches accounting information. I have made this package for my needs, it does not have a documentation, but if you are intereting - just send me a mail to nick@aanet.ru.
There is a multipath load balancer in 2.2.* ac
There are cisco compatible lan bundling things in
the beowulf patches too
Load-balancing may be done just before the hardware transmit queue, after logical address (e.g. IP address) and physical address (e.g. Ethernet station address) are added to the frame.
Fragmented packets, usually large UDP/IP packets, take advantage of the of the multiple paths.
Networks that fail completely first fill their hardware queues, and and are subsequently ignored.
this was taken from http://www.beowulf.org/software/bonding.html
From this it sounds like you could take multiple cards, say four, a cisco fast etherchannel switch, and have all traffic incoming/outgoing traffic "balanced" across the devices.
I have not implemented any of this, but it just seems like that would be possible from what I have read on this page and posts to usenet. Oh well, just a thought I guess.
-d
Note that Cricket is not a direct replacement for MRTG; it is a frontend for RRDtool, which is maintained by Tobi Oetiker as the "successor" to MRTG.
RRDtool manages the storage and retrieval of time-series data in an intelligent way, making it easy to write programs that manipulate such data. There are at least two frontends to it (including Cricket) that perform data collection and presentation functions. Hopefully there will be many other applications that make use of RRDtool.
Of course, you can read about all that on the
web page for RRDtool.
Cricket and RRDtool together pretty much give you the current functionality of MRTG, with a much better configuration scheme and a more extensible framework.
perl script was written bij tommy@break.org
/.*->.* - \d+ \d+ \d+ \d+ [ ]+ (\d+)/; /.*->.* eth\d+ \d+ \d+ \d+ \d+ [ ]+ \d+ [ ]+ \d+ [ ]+ (\d+)/;
http://break.org/tommy/linux/mrtg/
fix by itris@technologist.com if ipchains has ethernet cards specified:
replace
with
Hi,
I remember watching the stats on a dial up server with mrtg. On that server you had to bring up SNMP after you added the ppp interfaces, I expect this still holds for you too so don't start the snmp demon till you've added the interfaces.....
Hope this helps,
M
>MRTG just counts the traffic currently going by the interface when your cron job kicks in and tells it ;-)
>to look at the interface you specify - it doesn't count all the traffic that went by during the time
>period between cron jobs. MRTG creates nicer graphs though.
This is incorrect. MRTG does not read data from the interface. It reads data from an SNMP agent. The SNMP agent (cmu-snmpd, ucdavis-snmpd, etc) is what collects the data on the interface. The agent maintains a count of the number of bytes/packets/whatever that have crossed the interface. When MRTG reads the data, it calculates the delta between the current reading and the prior reading. The delta value is then used to calculate the average rate.
You should ask the service if they provide your traffic counts to the users. They are collecting and should at least give users the ability to look it up.
Target[graphname]: /1.2.3.4:public@machinename
Note the foward slash in front of the address. Now, if your ifIndex changes, MRTG will automatically adjust.
My local cable provider has something very similar. They have it setup so that we can setup email warnings (we can define at wut intervals we want to be warned at). Still, it's nothing that will forcefully prevent you from sending/recieving anymore packets after a certain limit :(
1) Install ucd-snmp
2) Write a script that uses your favorite
ip-account tool to read the stats
2) Configure snmpd.conf to run this script
and store the results in a pair of mibs (in/out, etc)
3) Ask mrtg to read these mibs too
This is the way we do ip accounting, disk usage,
mail queue usage, etc on dozen of servers via snmp
and mrtg
Pity, most replies don't answer the actual question but go in to detail how MRTG works, how to set it up, what are the alternatives, etc.
But the question is not about a problem with MRTG but with the counters kept in the kernel.
Very similar problem. .. look at notes in man ifconfig.
I have a firewall with one physical interface.
eth0 has internet IP and eth0:0 has private.
No matter what I do, ALL incoming and outgoing
packet go through eth0.
ipchains accepts rules for eth0:0 but packet
never go through them.
It looks like major code rewrite for handling
aliases
BranoZ
(branoz@pobox.sk)
Patched cmu-snump (not umich which comes
from rh5.1 and later) has a patch to get the
interface stats via IP accounting. I have mrtg
drawing the same type of stats you are asking
for.
How about using the dummy.o module instead of ifconfig eth0:xx to declare aliases. Would it then be seen as a different device?
The route command shows the different dummyXX interfaces but if you do ifconfig eth0:xx then all you see is eth0.
Maybe the dummy interface would work better.
Wouldn't it be better to set up an output firewall rule of no command (assuming ipchains, ISTR ipfwadm didn't like no command)? This way, you can still filter things later on in the chain if you need real output firewall rules.
Not from what I've seen. Most of the DSL and cable providers with limits seem to be adopting punitive charges for excess traffic. These are charges far and away above cost. It's a real shame, if you ask me. Metering is good idea but dammit, the overcharges have to reflect cost not some idea of apropriate punishment for exceeding the limit.
My cablemodem service is limited to 8GB of data transfers per month afterwhich stiff per *mega*byte fees apply. I currently have no way to know when I've hit the wall. Is there any sort of ethernet traffic quota monitoring software for linux that can shut down eth0 when the limit is reached? Warnings at set intervals would be nice too.
The ipchains code can count how much you use in /proc with a perl
Linux 2.2 so you can check
script or the like then change behaviour. Eg
mail you at 4, again at 6 and shutdown at 7.7
(dont go to 8, your cable co probably charge you
for mac headers and anything else they can scam)
we use something like this:
#!/usr/local/bin/perl
($In,$Out)=(split(" ",`/usr/bin/netstat -b -I $ARGV[0]`))[10,11];
$_=`uptime`;
/^.*up (.+),[^,]+user/;
$Uptime=$1;
$Host=`hostname`;chop($Host);
print (*)ENDE;
$In
$Out
$Uptime
$Host
ENDE
(*) Please insert two (smaller char) here, slashdot doesn't allow this neither as character nor as tag. *sigh*
Not that elegant though, but a quick hack that works.
(It's written for NetBSD's 1.4 netstat, uptime etc. output.)
In your mrtg.conf you can use than something like:
Target[some_name]: `path to skript interface`
MRTG 2.x can be found at this page. It discusses using MRTG and provides a few hints for installation and use. It is not a substitute for reading the documentation that comes with the source which is distributed under the GPL.
Active development on MRTG 2.x (currently 2.7.4) has essentially stopped. There have been occasional patches and slight feature enhancements over the past year or two, but little active development.
The reason active delevopment stopped was the MRTG uses a very simplistic data storage mechanism. Whenever MRTG 2.x runs, it must reading in its entire data file and write it back out. While this works for small to medium numbers of interfaces (up to a few hundred), it starts to slow down dramatically and becomes unusable. The solution for this has been to divide up the load by using multiple instances of MRTG.
To resolve this, Tobi started working on a data storage tool he called the RRD Tool, the Round Robin Database. Using this tool, you can support several thousand intefaces. It is also distributed under the GPL as is everything he distributes. You can find more details about it at the above noted site or in the USENIX presentation he made.
While there is technically no "MRTG 3.0," several data collecting frontends are already in production use for Tobi's RRD Tool backend. The above mentioned cricket is one of them.
The solution for your problem is here:
/.*->.* - \d+ \d+ \d+ \d+ [ ]+ (\d+)/;
/home/httpd/html/mrtg /mrtg/gif
mydata.pl:
#!/usr/bin/perl
# mydata.pl
#
# parse linux 2.2.x ip-accounting file
# return data for use by mrtg
#
# line 1: data in
# line 2: data out
# line 3:
# line 4: hostname
use strict;
#modify these
my $hostname="www.break.org";
my $ipaccfile="/proc/net/ip_fwchains";
if($ARGV[0] eq "") { exit(1); }
my $linenr=$ARGV[0];
#read and parse correct line of ip_fwchains
sub get
{
my $find=shift;
my $return=0;
my $count=0;
open(FL,"$ipaccfile");
while() {
if(/[ ]+$find.*/) {
$count++;
if($count==$linenr) {
#match byte-counters in ip_acct file
$return=$1;
last;
}
}
}
close(FL);
return $return;
}
my $in=&get("input");
my $out=&get("output");
print("$in\n$out\n\n$hostname\n");
and for your mrtg.cfg:
WorkDir:
Interval: 5
Icondir:
Target[all]: `/root/mrtg/mrtg/mydata.pl 1`
MaxBytes[all]: 1250000
Title[all]: Total TCP/IP Traffic
PageTop[all]: Total TCP/IP Traffic
There are two processes going on here. A packet receiver and a transmitter.
The packet receiver listens for packets on eth0. If it finds a packet with a destination address matching one of the host's addresses, it accepts the packet, logs the address it came to, and passes it to the application layer.
When an application (web server in this case) sends data out, the kernel looks at the destination IP address, looks at the routing table, sees that the default route is eth0, so all packets go to eth0. So when using ip accounting, all outgoing packets are logged with a destination of eth0.
What you want to do is to log the SOURCE address, not the destination address. In order to do this you must use source-routing, so that your routing table routes based on the packet's source address instead of just the destination address. Add a route for packets with each source IP and a destination of the corresponding eth0 alias, and then your packets will be logged the way you wanted.
How about asking how to make Linux balance the outgoing traffic on all interfaces. I hat to say it but NT (arghh) has software to do this from third parties and so do most bigtime OS's. I'm sure Sun's Solaris also has IP balancing software of all kinds too.
--Aaron
It's more of a new question, rather than an answer to the original post.
--Aaron
Check out ipac ( http://www.comlink.apc.org/~moritz/ipa c.html) which can create text-based and gif/html graphs based on data from ip accounting/ipchains.
---Vitaliy.
MRTG gets all of its statistical data via SNMP. You'll need to check into getting SNMP to corrctly report the traffic on all of the IP interfaces. Once that's done, MRTG will happily graph the way you want it.
With TCP/IP streams, there are about as many outgoing packets as there are incoming packets. So the graphs aren't going to look much different.
(BTW, anyone know of any inaccuracies in the data from /proc/dev/net when there are hundreds of virtual IPs? Does Linux always keep those statistics accurately?)
Forrest J. Cavalier III, Mib Software Voice 570-992-8824
The Reuse Rocket: Efficient awareness for software reuse: Free WWW site
lists over 6000 of the most popular open source libraries, functions, and applications.
--
I've had the same problem here. I asked 'Ask Slashdot' about it quite a while ago. We've got about 17 aliases, but everything only goes out over our main adapater. (We use token ring instead of ethernet (horrible, isnt it), so its tr0 instead of eth0 for us)
I never got an answer so i just gave up and assumed it couldnt be done
I have written a program that I use on my local Linux firewall to monitor our customer's bandwidth uses as well as usage for port-specific bandwidth on machines. I wrote it so you can get MRTG to dump data based on *ANY* ip chain you can create, whether it be for a single port in a single machine, a single port in a subnet, or an entire subnet, you can dump information on it to MRTG. If anyone is interested you can eMail me and I'll send you a copy (remove the nospam up top), and if you subscribe to linux-net, I posted it on that list about 3 months ago, so you can check it out.
-Robert Gash
-R
I have something working now but I am looking into upgrading to the 2.2 kenel where ipaccounting works differnt. what should I do. Check out http://stat.ifip.net I am curently using MRTG with a perl script that reads the /proc/net/ip_acc file.
either should work depending on which kernel you're using. Set up an output firewall rule of "allow" for each alias device and watch the counters. Works for me.
Look at http://kundip.copl.com.au/leonb/ipchains-MRTG-util s.html - it works for me, so far.
Got time? Spend some of it coding or testing
Ouch. I don't know what software to use here...but I thought that most cable modem services charged incrementally after limits were hit... i.e. if your limit is 8GB, and you use 12, you pay 1.5 times your normal monthly fee.
You could try Bandmin, www.bandmin.org. Its still kinda betaish, but it works fine. However it doesn't graph (yet), it only record the data in plain text.
my boss here at rutgers wrote his own stuff to count our ip's on the network, it goes by MAC addresses... we use MRTG and SNMP to do quite a bit... read to get more info at http://dorm.rutgers.edu/ipcount.shtml
I am a person fairly knowledgeable in TCP/IP, but have found few good howto's, doc's, or books on snmp. Can anyone help? Thanks.
jay2@home.com
I think that he was trying for another "Ask Slashdot" - it was more a question than an answer.
Load balancing over different cards would be really useful for some people - even if it wouldn't be useful for the original question-asker-guy
Regardless, if there's no data passing by the interface at the instant that the mrtg cron job is running nothing appears to show up in the graphs.
In order to test it (and make sure it worked when I was setting it up) I had to make sure to have traffic going through the interfaces I was monitoring in order to get anything on the graphs.
I didn't get anything on the graphs otherwise.
-- darron@froese.org
MRTG uses snmp to graph its stats. I had to download and install the cmu-snmp-linux (look at freshmeat for the actual url) in order to get snmp to work. MRTG also needs to know what physical interface to monitor so you have to tell it in the config file. If you have multiple interfaces it gets a little tricky to figure out which one to monitor but it can be done.
;-)
Once you've got the snmp stuff installed you need to find out how many interfaces it sees:
# snmpwalk localhost public interfaces
You should see something like this:
A lot of text scrolling by real fast - look specifically for this:
interfaces.ifTable.ifEntry.ifOperStatus.1 = INTEGER: up (1)
interfaces.ifTable.ifEntry.ifOperStatus.2 = INTEGER: up (1)
interfaces.ifTable.ifEntry.ifOperStatus.3 = INTEGER: up (1)
interfaces.ifTable.ifentry.ifOperStatus.4 = INTEGER: down (0)
I have four interfaces (lo, eth0, eth1 and eth2 [three are up and eth2 is down])
Look farther down the list for the statistics on that port ( look for interfaces.ifTable.ifEntry.ifInOctets.1 = COUNTER: some-big-number-here - this line counts the packets that come in over interface 1) and chose which number (1, 2, 3, or whatever you have) to put in your mrtg.conf file.
My mrtg.conf file looks like this:
Target[domainname]: 3:public@domainname.here
I've set it to monitor interface 3 in this config line. You can have multiple configs so that you can monitor multiple interfaces. I have both my main ethernet interfaces being monitored.
Something else you may want to look at to accomplish accounting for ip stuff is ipac (look at freshmeat for a url). It doesn't use snmp but instead uses the proc filesystem and counters that you define [you can watch any sort of traffic you want: nntp, smtp, www, pop3, imap - in any direction that you specify] to create graphs that show you you much traffic you've had pass through that machine.
MRTG just counts the traffic currently going by the interface when your cron job kicks in and tells it to look at the interface you specify - it doesn't count all the traffic that went by during the time period between cron jobs. MRTG creates nicer graphs though.
ipac actually graphs the amount of packets that went by - it doesn't matter if there's no traffic going by when you run the stats-fetching tool (fetchipac).
Hope that helped.
-- darron@froese.org
Firstly, you should probably upgrade to Cricket, as it is more flexible, easier to manage and under active development unlike MRTG.
( http://www.munitions.com/~jra/cricket/ )
As one of the previous posters mentioned, MRTG does indeed use SNMP to get its data. Now I'm assuming you use the CMU SNMP agent (or the UCD.. doesn't matter). You probably only have the MIB-II SNMP definitions supported by your agent.
What is probably happening is that your agent doesn't know anything about the data you are trying to collect. Now with Cricket or MRTG you can configure it to collect from a script. So you will probably need to write a script to ssh (or rsh) into the machine you are monitoring, collect the data and print it to stdout. Then it will happily graph that for you.
HTH HAND.
Joe
--
BSD has it setup correctly usually off the bat, but I don't know about Linux. What distrib are you running? Kernel ver?
-
ping -f 255.255.255.255 # if only
Webdude asked: "I run a server that has many IP aliases and have found a very strange thing: all the aliases receive data but all data is sent out through eth0."
Maybe you use a RedHat distrib, are you?
They are especially designed not to set routing on aliases (don't know for other distribs) :
If you consider this as a problem, you can return to a more normal operation by commenting those lines in /sbin/ifup (in RedHat 5.2):
those ones inI haven't yet tried in RedHat 6.0, but I think your have to remove [ "$ISALIAS" = no ] && from this line in /sbin/ifup (ifup-aliases is the same):
Note the way the init scripts rely on config informations that Linuxconf stores nobody knows were... Since I seen that, I removed this thing.
Afterall, if I wanted such crap, I would use Windows or Solaris...
One of the major design choices of Unix was to use simple text files for configuration, and that's a feature I especially care about
I'm not sure if I'm stating the obvious, but for traffic to go out of cards (virtual or real) other then eth0, shouldn't you tell route to do so? Usually the default gateway on linux and BSD points to eth0, and if a destination can be reached through the net attached to that particular card, it takes the default gateway. ;)
Problem with this of course is that to actually account traffic on a per-ethercard basis you will need to somehow dynamically add a route if an incoming packet is detected. Tricky I'd say.
G'luck tho
That won't help. Sure it will be balanced on outgoing, however it will simply mean that all etheraliases will get 1/4th of all traffic.
And, since the question-asker-guy says he has ip _aliases_ i.e. eth0:0 eth0:1 etc.. that all map to the same ethercard, that has no effect, it all goes through the same net connect, the kernel just calls it differently. It would help if you had 4 physical cards, and plugged em all into 4 ports on a switch, say.
Slashdot botched my formatting. The route command would be:
route add *ipadress* *the_card_used*