Constructing a Linux-Based Network Testing System?

10Brett-T asks: "Is it possible to build a Linux box with two interfaces to test a network's ability to carry traffic between two ports? I work for a company developing ethernet switching hardware. We need to test its ability to carry traffic correctly under a variety of conditions. Various vendors have expensive test platforms available that may or may not do the tests we want, but we have a tight budget. We decided to try building a Linux system with two ethernet interfaces, and modify the routes to force the traffic to go across the network. Testing would be as simple as running an FTP connection between a client on one interface and a server on the other. This would be a great victory for Linux in our company, if I could get it working. The problem is that I cannot figure out how to bypass the Linux kernel's TCP/IP stack routing optimization. All the combinations of routing table modifications and iptables that I've tried still don't make the packets flow out the interfaces and on the wire instead of within the stack. Has anyone else tried something like this before? How did/would you approach this?"

Comments

Use two machines?

[ZeroLogic]

Wouldn't the easy solution be to use two machines instead of one? Am I missing something?

/ZL

Two machines?

[ObviousGuy]

You want to send packets from one NIC and catch them on another? Why not use two machines?

Different subnets?

[SpaceLifeForm]

I know that I accidently 'tested' a cross-over cable once. I did not notice any problems with IP traffic. No kernel oops. But I agree with the two machine approach.

Mobile network testing

[sachachua]

A pair of Linux iPAQs with expansion sleeves may work fantastically in this environment. Get two CF or PCMCIA network cards, install them on the iPAQs (the old 3630s or maybe even the monochrome iPAQs will do just fine), and check away. You can even install netcat.

It's really portable and flexible, and you can test all sorts of things easily.

Incidentally, for wireless neworks - there are tools for helping you check wireless signal strength. Grab a supported WLAN card, plug it in, and wander around the building checking if your wireless network's okay.

Re:Mobile network testing

[ClickNMix]

Its a great idea, and one that in the past Iv considered my self (More for gewizz then pratical value all be it)

But, in an office enviroment, having to check wire by wire is hell. Iv done a network test like this using a Fluke Network tester. At the very least, you need a pair of radios to go with them for talking to the guy at the other end!

Having something that scans the network from a single point is a much more appealing idea. And of course, having something thats not ease to do means you can sell if for more!

What kind of switching hardware?

[Neck_of_the_Woods]

This could help answer my next question. If your switching hardware has van routing, or routing moduale say like a Cisco 6509 you could set up a NAT or PAT on the switch. This would allow you to have an outgoing address that was not on your ipstack that could be rerouted back to the box. Just a thought, I am very tired and could just be out of my mind here at 4am and 6 coffee's to the wind.

Blue Warrior, needs sleep badly....

Do a broadcast ping.

[inkfox]

Any direct traffic is going to be optimized out of the picture. You should be able to put the 2nd NIC on the same subnet, do a broadcast ping, and still see responses from the other interface however.

FWIW -- all UNIXes I've worked with have reported results from all responding systems on a broadcast ping, unlike Windows which only shows the first to respond. So you'll want to filter through grep if there are other systems on the network, as you'll get responses from many systems you don't likely care about.

User Mode Linux?

[Karora]

Could you do this with two firewalls on the machine running inside user-mode linux?

Other than that, I would say get into the network code. You might be better with an older kernel (2.2.x) since the 2.4.x have the fancy zero-copy network stuff which is probably what is your problem.

second box?

[smoon]

How about a second box? Hardware capable of saturating a 10MB ethernet connection should be really cheap, and even saturating a 100MB connection isn't hard.

Get two interfaces on each, one interface to an internal 'management' network, the other interface to the equipment you're testing. That way you can operate both boxes out of a single computer, possibly using SSH (or X or Telnet or whatever).

You might also try some other tools besides ftp -- some of the cracker tools can be tuned to blast out an amazing amount of traffic, and even ping can saturate a link if you make the packets big enough and have enough processes sending them.

I dunno. Just a thought.

forget Linux, try OpenBSD

[Permission Denied]

OpenBSD has a number of features to do the "transparent" routing which you desire - basically, you can plug in two interfaces and route from one to the other without changing the packets at all. The OpenBSD box basically looks just like a hub or a dumb switch to its peers, eg, it doesn't even have an IP address assigned to either interface, it doesn't decrement ttl, it doesn't do TCP optimizations, etc. As far as I know, this is not possible with Linux and it sounds like this is exactly what you want. See this page for more information.

Re:forget Linux, try OpenBSD

[Papineau]

OpenBSD has a number of features to do the "transparent" routing which you desire - basically, you can plug in two interfaces and route from one to the other without changing the packets at all.

That's not exactly what the guy wants. In your option, you have:
<machine>
12
<outer network>-- --<inner network>
Which means that the OpenBSD acts as a repeater (a hub) without any IP addresses on the 2 NICs, and without touching the packets it lets go through.
Have I correctly understood what you said OpenBSD can do?

The submitter wants this:
<machine>
1 2
<switching equipment>
His goal is to send packets on NIC 1, through his switching equipment, then trough NIC 2, and check for errors.
And what prevents him from doing this is that the Linux kernel sees that the destination IP address is in the same computer, and bypasses the 2 NICs and the switching equipment to be tested.

So the question is thus: how to bypass that? I don't think that any virtual machine (UML, VMWare) will work here, as the network traffic they generate will probably pass by the TCP/IP stack of the host, and won't work. Your best bet may be to take an older kernel version (2.2.21 has been released yesterday), which doesn't implement that enhancement.
Would it be possible to test at another layer? Raw IP, IPX, or even raw 802.3 frames? It will take a bit more of coding, but the switch should be able to speak those languages. Of course, it doesn't test the TCP/IP capability of the switch, but I don't think it really matters as the switch sees MAC addresses, not IP addresses. OTOH, if you don't test it, it may break silently...

Re:forget Linux, try OpenBSD

[Permission Denied]

Which means that the OpenBSD acts as a repeater (a hub) without any IP addresses on the 2 NICs, and without touching the packets it lets go through. Have I correctly understood what you said OpenBSD can do?

Yes, this is exactly correct.

His goal is to send packets on NIC 1, through his switching equipment, then through NIC 2, and check for errors.

You explained it more clearly than he did :)

This makes sense - it's much easier to write single a simple program that sends out data and then reads it back in, checking for errors, than it is to write two programs which run on separate machines to check the integrity.

My suggestion would then be to modify the kernel. I imagine it shouldn't be too hard to find the place in the kernel where it does this optimization and simply comment it out. Whenever I've needed to make a simple modification to the Linux kernel (like allowing people in group 80 to listen on port 80 to ensure my web server never starts as root, and simple stuff like that), I've found that the Linux kernel is pretty easy to grok and modify. In addition, this will show his management the power of having source code (one of his goals), as this sort of modification would probably be really hard to do with Windows :)

Re:forget Linux, try OpenBSD

[Papineau]

I've found that the Linux kernel is pretty easy to grok and modify.
Agreed for small stuff (as I never tried some big changes). I once implemented a small driver for reading the sensors on my motherboard (before I found lm-sensors). It took me about a day to figure out the doc for the chip, then how to make the kernel ask the chip and report the values in /proc, and finally adjusting the values of resistors to match the BIOS values. Then I found lm-sensors, and I never applied that small patch again.

... as this sort of modification would probably be really hard to do with Windows :)
Maybe the optimization that is causing problem is simply absent there... (don't know, don't have a Windows box)

Re:forget Linux, try OpenBSD

[10Brett-T]

You've understood the question perfectly.

We're trying to avoid using 2 boxes per station, because of the space available, and the sheer number of stations we're going to build. A cow-orker suggested pairing stations... so each station would have access to two Linux boxes, but only have one at each station. Each would have two NICs.

Re:forget Linux, try OpenBSD

[Anonymous Coward]

cow-orker? nice.

inexpensive test platform

[fist_187]

I work in a lab at The University of New Hampshire called IOL (InterOperability Lab) that does a lot of device testing for vendors in conformance with ITU and ANSI standards. I don't work in the Ethernet group myself (I do DSL), but I know that they have lots of tests written up that may be a big help in analyzing traffic. I'm not sure what the other "expensive" services you mentioned cost, but IOL's may be very competitively priced, certainly worth a look.

Re:inexpensive test platform

[Anonymous Coward]

UNH is the worst school on the F***ING PLANET!!!!

Be your own man in the middle

[Halvard]

You could do something along the lines of the Frame Diverter project but instead of just tcp port 80 for transparent caching proxy, you could divert everything so that you can test.

To summarize, you take a system with 2 nics and replace the destination mac address of all frames passing through with the Linux box's input interface. Bridge the 2 interfaces and run the tests of your choice.

When you want to take this out of if, God forbid, something breaks in hardware or software, if it, say, between to switches, you replace it with an Ethernet Crossover cable and your network is restored to operation.

Ditto

[Clue4All]

The problem is that I cannot figure out how to bypass the Linux kernel's TCP/IP stack routing optimization. All the combinations of routing table modifications and iptables that I've tried still don't make the packets flow out the interfaces and on the wire instead of within the stack.

Nor could I. I spent the past year working on a thesis-like project for undergraduates building a new queueing mechanism using the Linux kernel. Using only one 300 Mhz processor and saturating two 100 BaseT interfaces would suck down about 1/3 of the CPU, and I found no way to bypass the stack. FreeBSD and OpenBSD can do it transparently if you want to give them a try.

cheap IP testing

[Anonymous Coward]

Go to www.packetstorm.com
I've used them very, inexpensive easy to use, can create just about any IP test.

Re:cheap IP testing

[Anonymous Coward]

They have many solutions and prices. Make sure you ask for the lowest speed platform.

IBM Mainframe Linux

[Grotus]

I'd look into what IBM did to get multiple instances of Linux running on a mainframe. It might be possible to modify their code to run on a standard PC. Then it should be possible to associate each interface with one of the instances.

Re:IBM Mainframe Linux

[technos]

I wouldn't even bother, because there is no such code. IBM's mainframes run multiple copies of Linux not because of modifications to Linux but because of their master virtual machine program, which runs above Linux and is the only thing really running directly on the hardware.

Linux *can* run directly on the hardware, sure, but then your only choice for running mutiple copies under it are the same choices you have when you want to run mutiple copies of that port on PCs, and all of those use the stack and run into the zero copy optimizations.

Broadcast

[pehowell]

What if you sent out a broadcast out one interface? As long as the other interface is in the same VLAN, the switch should forward it.

This doesn't test backplane bandwidth, though. Just connectivity.

You need to modify ip_output

[The-Dork]

The problem can be restated as "Linux does not allow locally deliverable packets to go outside the machine." This is because of the IP layer code which detects the local destination and short-circuits the path of the packet.

I am currently working on a patch to enable this to happen, but I dont see this getting ready soon enough for your needs. If you are constrained for time, you might want to look at the kernel, espcially the ip_output(), ip_route_output() and corresponding routines.

Does some other OS really implement this? Would be interesting to see what then do at the IP layer.

Raw Frames

[Si F.]

How about writing raw frames? If you make your own IP Packets they should skip
any internal routing thats going on. Its not that tricky and it will give you
a lot more control if you want to experiment with stuff like TTL and ToS.