Slashdot Mirror

http://www.ntop.org/products/p...

Okay, there you go.

Here is a URL to a presentation on the issue of GPU-Based Network Monitoring.

BTW, with PF_RING and a DMA-enabled NIC driver (PF_RING DNA), one should have no problems capturing 10 Gbps on a single CPU modern server. I can capture/playback 4.5 Gbps no problem using this with four 10kRPM HDDs - 8 drives should give you 10 Gbps rate capture/playback.

For corporate traffic, Don't put a box in between that traffic. If it fails, everything is down. Get a TAP, as you hinted, but make sure to get one that fails 'open'. Then, run Ntop off the TAP port. If the TAP burns up, or port goes bad, you still have network access.

It sounds like your "client" is just wanting to basically monitor on his family, so in that case, get a 10/100 HUB (not a switch) to stick downstream of your modem. Plug in your linux box on port 1, and the router/modem into port 2. Don't put anything else on it because.. it's a hub. Run Ntop on the linux box.

Best way I've found to measure growth is to have a running history of traffic on each router. You don't need a $billion to do it. There are some decent enough FOSS tools out there to do it. MRTG or Cacti will work nicely and integrate with SNMP.

For a smaller network, you could run a span port and graph your own data with a shell script, or hook up NTOP. which will give you real-time views of traffic but you would need to implement something to save those reports daily.

You suggest some good tools, but they primarily measure network utilization rather than capacity. The question isn't "how much data is my network handling now" but "how much data could my network handle at peak"?

Best way I've found to measure growth is to have a running history of traffic on each router. You don't need a $billion to do it. There are some decent enough FOSS tools out there to do it. MRTG or Cacti will work nicely and integrate with SNMP.

For a smaller network, you could run a span port and graph your own data with a shell script, or hook up NTOP. which will give you real-time views of traffic but you would need to implement something to save those reports daily.

sorry didn't meant to post as AC before... anyway:

DD-WRT on your router + ntop running on another machine. Ntop gives you all sorts of pretty graphics and stuff. Very easy to use.

Yep, I've been very happy with Tomato, which shows both real-time and aggregate data bandwidth use.

If you want even more detail into what's taking up all your bandwidth (port / protocol / IP / etc.), you could put up a box running ntop (the web-based "ntop", not the console "ntop" similar to iftop that only gives instantaneous usage info).

You might also be able to forward traffic from your router to a sniffer on a real machine running these tools, if you search for "[router] SPAN port" or "port mirroring" or somesuch.

this story

Yeah, I read the arstechnica article a few days ago, and the comments there were much better than the ones here. Among the sentiments I enjoyed:

• The media coverage of these handfuls of SWAT raids are mostly to scare everyone into securing their access points, because then it makes it easier for the feds to convict you when someone breaks into your wireless access point and downloads CP or something else they don't like. If you have an open access point, they can't really "prove" it was you. But if you have some kind of encryption going, then as far as the court is concerned it just *had* to be you doing the nasty, since you're the only one with the secret keys and there's no wai anyone could possibly break into it, as trivial as we know it is to do.
• The police don't apologize for anything that might happen during a raid. As far as they're concerned, they can do no wrong. But they will get reprimanded by the courts for issuing too many "dynamic entry" warrants prematurely.
• For my part, I think that if enough of us continue running open APs, the police will eventually have to find better ways to cooperate with us in their investigations. I don't really want to live in a world with no open and shared wifi (even though I have a cell phone with tethering and pretty fast HSDPA service, so I don't even need open wifi most of the time)

To actually respond to the OP...

• Set up a separate wifi router. Maybe look into something that can support OLSRd or something so you can get some kind of community mesh network going... this will particularly become important to have lots of people with OLSRd nodes if the government ever decides to use their internet kill switch for some silly reason.
• Run that wifi through a spare wired computer with two NICs, so you can use wondershaper or something to limit the bandwidth going through it.
• Some other good monitoring tools: NTOP (the web-based thing, though the other console ntop is also nice), to log and display traffic type and endpoints SNORT, to help alert if bad things are happening iftop is a good console thingy for showing you what is taking up bandwidth right now. Wireshark, for the times you feel evil and want to do some packet inspection / logging, though you probably don't want to run this all the time.

Good luck and have fun, don't let the man keep you down! :P

2. The Linksys is re-secured - but I hadn't thought of that being owned - so I have to now do a firmware upgrade on that - Thanks for the suggestion.

You really should look into putting ntop and tcpdump on the router. (I'm assuming it is running Linux obviously) These tools are indescribably helpful when diagnosing intrusions.

http://www.ntop.org/news.html
http://openmaniak.com/tcpdump.php

If your router isn't running Linux, and you can afford the extra electricity costs, put together a Linux or BSD firewall for him.

Has anyone used the n2n peer VPN?

It would be neat if such solutions were built into the popular distros; with all the monitoring creeping up around us it is about time that our PCs defaulted to encrypted traffic.

Cacti.
Ntop.
Nagios.
MRTG.

1. Deny IRC traffic at your firewalls. If there is a business need for IRC then setup a IRC proxy, or inline authentication. This simple step will stop many of the bots out there from phoning home.

2. Enable reverse path detection on your network devices. This forces your internal routers to check whether the source ip address that the bot is sending, is available out the interface that your comprimised host exists on.

3. Enable DHCP snooping on your edge switches. By configuring this feature the switchport that your host plugs into passively observes what IP address was given to your computer. If traffic is spoofed (a common occurrence for botnets) the switchport effectively shuts your host down.

4. Monitor your network. There many free and commercial products that will make it clear that your traffic profiles have changed. Some good free tools for this are Cacti - http://www.cacti.net/, Nagios - http://www.nagios.org/ and NTOP - http://www.ntop.org/

5. Utilize update antivirus technology, hopefully one that reports to a central console. These are simple steps, that frankly most people do not use in their networks. If they would the botnet issue would be greatly minimized.

ntop
Nagios
MRTG
Cacti

Then NTOP http://www.ntop.org/ is your best bet, this breaks down all traffic on your network and should allow you to see who's being naughty and who's being nice.

Whatever devices are between the nics (no crossover cable) leave an opportunity to see whatever traffic is going between them. Even ntop will tell you what types of traffic it's seeing - not to mention if you are inside a bunch of hubs. 'Darknet' sounds spectacular, but it just comes down to another stupid protocol running on a non-standard port. If you're lucky, your best luck is to invent your own protocol, encrypt it, and don't share the source with anyone. Good luck getting anyone to trust you though.

I'll add ntop to the list. Plug a box running that into a monitor port and watch the traffic for a while.

As others have said good documentation of the Network is a must. I was thrown into a similiar situation a year or 2 back at my highschool (I graduated in 94, so it wasn't as a student). Aftering doing a walk through of the network and finding every single hub (there where 2 switches) and what was attached to it we could then easily locate some of the problems. In some cases they have hubs chained 8 deep (with 60-70 computers) and there was a ton of broadcast traffic, we isolated some of the labs and replaced some of the chained hubs with switches (Temp fix), removed some worms, virii and located a bad nic, we got the network running a lot better. My next step was to replace the little NAT box they where using with a Netra that I had sitting at home running OpenBSD and Squid. This way I could transparent proxy all of the net traffic and cut back on a lot of the stuff the school didn't want to come in, also it had the added benefit of speeding up some of their classes, since most of them where like, ok Kids everybody click on this url. So 30+ requests for a graphics heavy page will bog a 1mb/s DSL connection. We are finally upgrading the network (Should be done soon), to use a bunch of fully managed cisco switches with Gig Fiber backbone and much needed vlans and firewalls. (which is cool since I can get my Netra back )

Yes, a specialized kernel is needed. Yes, some other kernels, maybe QNX, might be better than plain vanilla BSD or Linux kernel. Yes, MS Windows isn't even anywhere remotely close to supporting this kind of technology. But...

...as with any other activity, the packet loss will be reduced or go away by tuning your software (in this case the kernel) to the task at hand. That includes choosing libpcap, netfilter, or something else. However, for low and medium speeds BSD/Linux do a good job.

There's a paper which discusses the problem quite well: http://luca.ntop.org/Ring.pdf

The thing which I found surprising is how awful Linux stood up to the tests. The standard Linux kernel + stack was dropping the majority of packets; and only with special tweaks was it able to get to capturing 93% of the packets. But 93% still isn't 100%, which is what commercial sniffers have been doing since the DOS days.

So the bottom line here is:
1. Don't use off-the-shelf BSD or Linux without serious tweaking.
2. Even then you'll still be dropping packets.

What is also interesting is that MS Windows isn't close to supporting this type of technology, which means the MS servers are going to be in serious trouble as more people adopt Gigabit networks.

Forget running ethereal or other packet sniffers, they're far too fine-grained for what you're trying to do: prove they're being abused.

Connect a small box running your favorite *nix running ntop to the service port (or whatever they call it, I'm half asleep) of the switch/router to which the box(es) in question are connected.

That's it.

Ntop will give you very nice graphs to print out for the guy who needs a clue, showing not only the IPs involved, but the ports, percentages of traffic per protocal/port/whatever.

If they're being used for SPAM, for example, you'll see tons of outgoing port-25 traffic.
Just make sure that's not what the box is supposed to be doing!

If this isn't enough for him, and you're sure this is something against the company's policy, bring it to his boss' attention immediately (calmly) with a full explanation of why you are doing so.
If you delay this action, you risk having your boss "poison" his perception of the situation, and end up thinking that you're a troublemaker (that is, if the politics are as bad as I'm guessing they are there).

So either fight it all the way, or drop it... or be prepared to find a new job (these situations always suck).

If you actually read the line below it says:

If I distribute GPL'd software for a fee, am I required to also make it available to the public without a charge?

No. However, if someone pays your fee and gets a copy, the GPL gives them the freedom to release it to the public, with or without a fee. For example, someone could pay your fee, and then put her copy on a web site for the general public.

What that means is that I CAN force you to pay whatever I want you to pay BUT then you can go off and undersell me. That makes it fine for contract work (one or two big payments) etc but not that good for Shrink Wrap as if it's get's popular someone else will try and sell it or give it away. What that means in real life is that if you try and GPL software you tend to sell the product + service which is standard once you get beyond shrinkwrap products. The besta example I can think of is ntop

Buy one linux server, and then discover the wonders that are ping and SNMP. Simple tools such as Nagios and MRTG (or NRG or Cricket) can do wonders for helping spot problem switches/routers and congestion spots.

For example, every device we have is pinged 3 times every minute, and queried for bandwidth usage every 5 minutes. This has helped in finding bottlenecks, and the occasional switch that reboots every few minutes. (MRTG alone convinced the higher ups to buy new gear for our Datacenter and give it a dedicated link to the Core).

Also, setting up a wonderful SNMP trap server can be very useful. It allowed us to find a switch that likes to reboot at random intervals (the switch is 5 years old and being replaced this weekend). Of course, having it send a trap whenever a switch reboots is just the start of what certain switches/routers can do.

Also the use of Snort to sniff traffic that can be potentially malicious can be very helpful in tuning firewalls and finding those script kiddies. (use ACID for a pretty front end)

Another nice tool is NTOP Does almost everything NetFlow does and has a pretty graphical frontend built in. (I recently used this to find out that one of our firewalls was sending gigs of syslog data to the wrong server.)

And with the mention of syslog, might as well throw out a link for syslog-ng. yet another useful tool.

Basically the point of this is to say that sometimes it's best to let your equipment do that talking. They'll usually tell you what's wrong, just as long as you've set them up to do so. I found that once we put a lot of these tools into full production, we were able to cut down on our need to sniff the line whenever problems came up. This isn't to say that Ethereal isn't needed. That's hardly the case. Its use is still huge and shown all the time.

Ntop is the way to go for fast analysis. It has a http daemon built in which presents an overall view of your network's activity. You can then examine in detail with tools others have mentioned. Take care about running ethernet wiring alongside the mains wiring.

You should check out NTOP. It is an extremely useful tool.

The best web-based version is ntop, which is another one of those "Oh my god, this is SOOO cool" tools, similar to ethereal. It lets you drill-down through a fair bit of data, and pages load fast and it's virtually real-time, so you can bang on the reload key and see a similar sort of data that etherape/iftop would give you. It has a daemon piece and a CGI piece, so installing it via a package (eg. apt-get install ntop) may be much prefered to installing it by hand.

ntop is also a good resource for doing this.

And here's one more.

I had thought of using Knoppix as an "appliance" distribution - I hadn't considered Apache, I was thinking for things like Squid, NTOP, Nagios, etc. Config information could be held on a floppy or flash drive.

Matt

Check out ntop for the per ip bandwidth utilization. Not exactly what you wanted, but it might do the job.

As for the bar on kernel.org, just click on it for the source. Dang kernel.org has been slower than balls for me lately.

Check out ntop. It watches traffic passively and generates quite a few pretty graphs. It has breakdowns by protocol, machine, time of day. All sorts of stuff. Extremely useful for troubleshooting the "my internet is broken" problems.

Ntop is sort of a souped up TOP for TCP/IP connections. Keeps logs of outgoing/incomimg traffic and what kinds, as well as other information. Presents it in a browser format if desired. Highly recommended.

There's no need to query each box. The information you want is on your local wire, so stick something like ntop on the network and let it collect your stats.

Netsaint I think netsaint i very cool. I.t checks for services in various network devices therefore reporting on uptime etc.. Sends out emails if one device goes down and so on. Very configurable. Love it, also ties in nicely with Cricket(link the devices with their respective cricket pages).A reporting tool for netsaint(Impress your boss!!)here

Cricket bases on rrdtool which is written by Tobi Oetikers(the guy who wrote MRTG. If you look at the rrdtool page you see various other frontends, I just happen to like cricket. Great for graphing routers and switches(and pretty much else) through snmp(you can configure it to graph other things, for example their is a package that creates graphs of the RTA's of devices in netsaint(look at the cricket contrib page.

ntop ntop, a sniffe with a web based interfaces(and a console one) were nice for monitoring various aspects of parts the network. Check out one of the newer cvs snapshots

I haven't had time to check out OpenNMS yet. Another nice tool is ethereal, a awesome gui sniffer.
One thing that is especially great about netsaint and cricket and netsaint is the great number of 3rd party addons, which make life a lot easier

Cricket - pretty much the same as mrtg, but a lot more advanced, based on rrdtool, in my opinion a lot more powerful then mrtg and easier to manage

NTOP a sniffer with a web interface, check out one of the newer snapshots. REALLY useful, probably one of the most useful programs i have ever come around(network admin wise) both are web based

Here we use two Linux based systems to monitor traffic, NTOP and MRTG

NTOP stands for Network TOP and displays usage broken down by machine and protocol. I have successfully implemented this on RedHat 5.1 running on a 486 with 6 meg of RAM and a 500 meg HDD. I install the NTOP servers between the LAN and the router, connected to a hub where they can look at the traffic. Check out http://www.ntop.org for screenshots, etc.

MRTG is the Multi-Router Traffic Grapher. MRTG interrogates devices such as routers, switches and servers by using SNMP, and displays the results for a day, week, month and year on a webpage. For MRTG you need a slightly more meaty machine - I'm currently monitoring 12 sites every five minutes, using a P133 with 32M of RAM and a 1Gig HDD. (Mandrake 6.1 for this one). Site for this one is http://ee-staff.ethz.ch /~o etiker/webtools/mrtg/mrtg.html

I don't know if the above will be of any use - I think you'll have more luck with MRTG.
Good luck!
Matt (matt_brunton@hotmail.com)