x86 Commodity-Hardware Router?

neomage86 asks: "I recently had to set up a router for a small company, only five users at any given time, and the needed VPN capabilities are built in. So, instead of using a Cisco or other embedded router, I decided to just install Linux and IPTables on an old 200 MHz PII I had lying around. It's been working fine, and I'm thinking about doing something like this for a much larger network (3000+ users). Does anyone have suggestions on how much I will have to beef up the hardware to provide IP Masquerading for about 1000 users on a T3; provide network-layer filtering of the transmission; and route between 4-5 internal subnets?"

Comments

VPN

[aeakett]

VPN can be a real resource hog... word is though, that the Via C3 has some sort of processor level instructions to help accelerate this. Has anbody else heard of this?

Re:VPN

[aeakett]

Ah! Here it is! It's the encryption that the C3 seems to rip through.

Re:VPN

[quinkin]

VPN should be offloaded to a seperate box/boxes (NB: boxen is not the plural of box, just as foxen is not the plural of fox - although that does imply that bixen should be a female box...).

Even with the higher end router/vpn embedded solutions there seems to be an appreciable slowdown in the other traffics response times and throughput when the VPN is being heavily used - and the hardware acceleration in these systems is liable to "better" than the C3 acceleration.

Does anyone know if the C3 can do h/ware accelerated 3DES? AES is good, but compatability is better... I would assume the RNG could also be applied to DES/3DES to at least improve performance.

Q.

Upgrade? Hell, you're already massively over-spec!

[Finni]

You'll be fine with what you've got right there!

No seriously, you're going to swamp your PCI bus if you're doing routing between internal subnets. Goodbye, LAN throughput. Not to mention what merry hell you'll play with the CPU with VPN and firewall rules.

Your solution is great for a small place, or even a large place in a dedicated niche (like only VPN and/or firewall, or monitoring/IDS.) I wouldn't do something that ambitious with PC hardware though.

Go BSD rather then Linux.....

[jsimon12]

I would personally go with a BSD flavor rather then Linux. Don't get me wrong Linux is great but BSD was designed with routing in mind. You will be able to get away with less hardware and out of box things like OpenBSD are going to be more secure then a commodity Linux.

Re:Go BSD rather then Linux.....

[frankm_slashdot]

i wanted to say that... but was feeling lazy.

now that ive hit refresh a few times and have read your comment i might as well add my own $0.02

openbsd with pf is, imho, 50x better (and easier to set up and manage rules for) than anything linux can offer.

1000+ Users????

[the eric conspiracy]

Do the math. If your homebrew system goes down, you will be burning the time of 1000+ people ($60,000) per hour. With those kind of numbers it doesn't pay to do it on the cheap. Get a redundant Cisco system with plenty of power backup.

Re:1000+ Users????

[ADRA]

Not that i'm arguing here, but a Cisco equiv. Is hella-bucks for what this guys is trying to do, and its only a Passive failover anyways. If you want a solution that is truly expensive, try any ACTIVE failover provider.

Anyways, I have been using netfilter/iptables for on my 30 user, >100mbs network, 6 active NIC's and I've never had a crash that I didn't cause!

All things considered, spend a couple hundred.

[WolfWings]

And for said couple-hundred, you're looking to pick a secondary network card, along with a 2Ghz or so Athlon or P4 of your choice with a motherboard with a built-in network card. The built-in network card is important for a router.

An Athlon-64 or above would be ideal, simply because you'd be able to mount ludicrous amounts of memory on the box, which is pretty much all that could ever matter for a router/firewall app, as Linux can easilly support logging anything you want to a remote boxen.

Realistically though, I've routed 8 T1's at 80+% capacity in both directions among 650 laptops before, including 3 seperated subnets, all routed through one box.

The box was a Celeron (P2) 800Mhz we'd downclocked to 633Mhz (standard practice at my company, downclock everything for live events for stability) and it used around 10% of the CPU at peak once configured correctly.

By 'correctly' I mean having the T1's all coming in on a seperate PCI bus from the actual network cards for the subnets. Specifically, the built-in ethernet turned out to be on a seperate PCI bus from the actual PCI slots in the case. Configuring the box to take advantage of this dropped CPU load from 80+% to ~10%.

So... for a T3 fully loaded? I'd say get a 2.0Ghz machine just for breathing room, and give it at least 2GB of memory, as neither is that expensive and will leave plenty of breathing room for things like IPSec or other fancier options down the road without any problems.

Re:All things considered, spend a couple hundred.

[WolfWings]

First off, the case itself was one of the 'all in one' deals, simple one-5.25 bay, one-HD bay, one-floppy, half-height PCI cards only, etc.

The P2 was a typo, and one I appologize for. P3 would be much more accurate, and overlooking the typo is inexcusable as I was simply typing quietly before I hit post, and didn't read the entire post from the beginning before hitting post.

As for the T1's, we didn't use any PCI T1 cards. We used an external 10/100/1000 switch with all 8 T1's plugged into it via normal T110/100 converters as a concentrator, with the uplink port plugged into the computer. Four 10/100 PCI half-height network cards + onboard, three + onboard used. Onboard led to the switch with the T1's on it, the individual network cards all led to individual subnets.

As for the downclocking, yes, we had to throw jumpers. And as I said, it was policy at the time, and one I didn't completely agree with but it did noticably lower the heat output on the CPU's, which was often a problem when we had to install these things under bleachers or in other areas with absolutely zero ventilation and little access. In one case, we had to repurpose a bathroom actually, speaking of those. For that specific reason, the downclocking made sense.

The configuration of the multiple T1's on one ethernet port was fairly simple, using the Aliasing features of Linux to pretend to be 8 seperate ethernet cards plugged into that one switch, leading to each of the 8 T1 cards.

And yes, the CPU had little cache, and slow cache to boot, but lots of memory, and with that configuration it wasn't dealing with much data, barely a fraction of the actual network traffic, because all the network cards we'd installed could copy data directly from their own buffers to other network cards. The fastcopy option under Linux Networking in the kernel IIRC.

If you have any more questions, feel free to post again though. :-)

Re:All things considered, spend a couple hundred.

[WolfWings]

The T110/100 was supposed to be "T1 to/from 10/100" with arrows pointing both ways. Slashdot ate the greater-than/less-than signs, along with the hyphen.

Re:All things considered, spend a couple hundred.

[WolfWings]

Okay, point-by-point again.

The 't1 to 10/100 converters' are just common T1 interface boxes that output ethernet instead of 24 voice/data jacks. Data-only T1 interfaces, essentially. Unfortunately, that was one aspect I had zero to do with, the site provided them and I haven't had a reason to use them since (we usually do satellite T1 links for remote sites, or use sDSL for medium-term fixed emplacements), so other than saying Netopia was branded all over the boxes, I can't help further than a Google search would.

And the direct copying can change the addresses, so MASQ can still function as I understand it. To be honest, the direct copying of packets didn't drop the CPU load anywhere NEAR as much as simply having the cards seperated across seperate PCI busses, so the CPU could talk to each of the groups at the same time, instead of having to shout down the same piece of tin-can-and-string to everyone at once.

We did do what you described though, all the firewalling/IPsec/what-have-you was a seperate set of rules between a pair of virtual ethernet devices.

The overall layout was this:

Arbitrary subnet gets VPNed/MASQed/etc to a virtual ethernet address. Virtual ethernet gets firewalled to another virtual ethernet. Second virtual ethernet gets dynamically MASQed with connection-tracking to the 8 T1's to send the traffic to the lowest-usage T1 over the last minute or so using QoS rules.

Most of that's just shuffling headers around, which are tiny, and the final copy boiled down to a single MASQ and either getting passed on or dropped on the floor, which still works with fastcopy.

And yes, tracking a couple thousand concurrent connections did eat up the memory. (2-4 per laptop, LONG story, client was using multiple bidirectional realmedia streams to push an IRC-like live QA session at the Detroid Auto Show one year for vendors, so the presenter could ask questions and get realtime answers back without having to resort to a 'show of hands' count. Yes, we told them it was a bad design.)

As for cooling... At detroit we had plenty of space, plenty of cooling, etc, etc. But to be quite honest we've literally shown up at a site, and been informed they 'repurposed' our space for storage, and found we can barely squeeze a folding chair and a laptop into the space left for us, even with setting things on shipping crates. We gave up complaining and learned to expect (and equip ourselves) to be crammed in the equivilant of a furnace room with zero ventilation and space for one person to stand unseen as our minimal requirements for getting a live press event running for up to 12 hours at a stretch. Live press-style events are a bitch, but we do fairly well at supporting them.

Re:What's good for the customer

[jhoger]

So you're saying that his customer should avoid vendor lock-in by locking in with a proprietary vendor?

Hmm... Linux routers and firewall rules are well described on the web. Any "competent network engineer" as you describe him/her is likely able to read...

It's do-able but...

[dcowart]

It's do-able but segment out the functions at that point. Do you really want to try to route between subnets as iptables is traversing the masq table? Get three boxes; one box for routing, one for vpn traffic and one for actual firewall/masqing. IBM has crypto boards for accelerating SSL/IPSec stuff with linux drivers IIRC for your vpn box. Also, with three boxes you can take down the vpn without taking down the internet connection.

I would suggest getting PIII's instead of PII's though, but check where bottle neck's may be PCI bus, CPU processing packets, NIC not doing so well... etc. Plus if one box is connected to multiple subnets, it can be dhcp and/or dns and/or wins for them (if you do DNS please use the forwarder's option to forward dns requests to an upstream DNS server if possible).

Re:no can do sorry

[Paul Jakma]

you simply cannot pump that much through a standard PC. .... Unless anyone knows if those quad cards can route between connectors at faster (much much muuuuuch faster) than the PCI bus will allow

If its 100baseT, 4x12.5MB/s = 50MB/s is easily within the capabilities of a standard 32bit/33MHz PCI bus (100MB/s sustained), at least in terms of transfer rate. Make sure to use a card that has drivers which support polling (aka NAPI on linux).

Buying a service, not a router

[dpilot]

Especially with a PC-based router the customer needs to understand that he is now buying a *service* instead of a machine. It's not too smart to leave *any* box live on the Internet, or even in a customer's office without some sort of maintenance, but for a Linux (or Windows, any flavor) box it's potentially dangerous.

The number of exposures for Linux doesn't particularly bother me, for a box that's being actively maintained. For a generally non-service box you don't even need to be paranoically prompt about getting fixes applied. But I'd get worried about an *appliance* PC.

Re:Buying a service, not a router

[jhoger]

Okay, I'll bite. You're saying that if you have a magic nicely shaped appliance it somehow won't require security patches like a Linux box does?

All software has bugs. All software, particularly that which runs on the edge of the network, must be maintained with patches. All hardware networking solutions of any reasonable complexity like a router or firewall run software. Therefore they too must be patched from time to time.

At least with a Debian box you could put a cron job that automatically apt-get's latest patches for itself, if you wanted to have a box which maintains itself (I would rather have an admin maintaining it, but whatever...).

This may or may not be a feature of whatever appliance the parent thinks is a better alternative.

T3 only?

[ADRA]

If you're just powering a T3 and 6 10/100 subnets, you could get by on

P4 2.xxGhz (assuming moderate VPN usage)
512MB-1GB RAM depending on how many simultaious connections you're working with. The more connections the more memory eaten up
Hard drives: minimal config.
Motherboard & NIC's: Depending on how much you're 10/100's saturate, you may want to get some 66Mhz 64bit PCI cards instead of regular 33/32's. Eg:
http://www.cisco.com/en/US/products/hw/vpndev c/ps2 030/products_data_sheet09186a0080189f0a.html
It all depends on how much simultanious traffic you're looking at. You can use the analogy that the PCI bus is a network switch's backplain. 66/64's can transmit a theoretical maximum of 4gbits/sec. so it should be enough for anything you throw at it. 33/32's maximum theoretical is 1gbits/sec. but in reality expect for much less.

Re:What's good for the customer

[Anonymous Coward]

I'll bite (the troll).

Instead of offering standardized equipment that can be managed via console, ssh or SNMP by any competent network engineer, you offer some customized linux router solution that will always need to be handled differently.

A customized linux router solution can be managed via console, ssh or SNMP by any competent network engineer.

What advantage does your solution offer?

1. More online documentation than every other router and firewall vendor combined. Docs ranging from step by step howtos to in-depth discussion of complicated setups.
2. An open system that is upgradable on your timeline not your vendors
3. An easy upgrade path. If you want IPv6 support (or some other feature) and have an old firewall you might have to purchase a whole new unit if a new firmware with those features isn't available for your unit.
4. An army of people who know how to use iptables

Is it worth "saving" a little money up front, only to need to seek out your consulting services later?

Since it'll be running on an open system, they can seek out anyone's consulting services they want including those that might be in their own organization.

Wrong Answer

[MerlynEmrys67]

PCI bus = 127Mbps = 15.8mbps
T3 = ~50mbps

Wrong - you got the division wrong
PCI Bus 127 MBytes = ~1Gbit/sec
T3 = ~45 Mbits/sec

Are you telling me the fastest a PC bus can go is 15 MBits a second ??? I know of Intel class hardware that can keep 100 MByte going over a Gbit NIC. Lets not even go into shipping PCI-X busses and soon to be shipping PCI-Express busses that are significantly higher throughput than this.

Now that we have that problem solved, what you will run across with multiple 100Mbit network cards running into your system is a higher latency than your low end cisco router, and lower reliability potentially (all though in both cases, I suspect software reliability is orders of magnitude lower than hardware reliability). If you can live with the higher latency going through a PC based router - go for it, you might save a few bucks...

Re:Upgrade? Hell, you're already massively over-sp

[LunaticLeo]

Uh, PCI bus is 128 Mega-BYTES per second maximum thruput. That is 1 Giga-BIT per second. And that is just for the standard 32bit at 33MHz speeds. There are plenty of Intel based servers with 64bit and 66Mhz PCI variations.

Nice idea, but the hardware won't cope

[Masarand]

Buying enterprise-class network equipment for the first time round is scary (it's a strange and complex world.)

If you're inexperienced, try to get everything from one vendor so that getting it all working together is their problem, not yours.

You could do worse than a http://www.nortelnetworks.com/products/01/passport /lan/.

Apple Airport Base stations

[adzoox]

The original Apple Airport Base Stations called "Graphite v1.0" actually had a 486DX100 AMD Equivalent - if I'm not mistaken these were called "Dave Processors".

The Airport Base Station (original) is a very good "take apart" to learn how to build your own router.They couldn't be more simplistic in design and implementation.

What are you people talking about?

[Gilk180]

Admittedly, the pci bus will probably be the first absolute roadblock with a good machine, but I think you are all underestimating it's ability.

I did a quick test on my home network to make sure. I easily got 97 Mbps using NFS to transfer (multiple simultaneous) files between 2 machines on 100 Mb ethernet. I think that is pretty conclusive evidence that the PCI bus will not be a limit even on a DS3(T3), which only goes 51 Mbps. One of these machines even has the video card on the pci bus.

Anecdotally, why would we even bother building Gigabit ethernet if unable to read the data (given, the bandwidth is shared, but anyway).

For stabilities sake, you should probably not try to do the routing among internal subnets with this box. However, if most of the internal traffic is accessing external hosts, this would also be possible, since most of the traffic will be crossing this box anyway.

Re:Go BSD rather than Linux.....

[Anonymous Coward]

Linux wins the speed race, hands down.

You will get 0wned much, much faster with Linux than with OpenBSD.

Don't use Linux for this

[phoenix_rizzen]

The packet filtering software on Linux is horrible. The syntax is just nasty. And there are no guarantees it won't change again with the next kernel release.

Use a BSD system, with a real packet filter. FreeBSD gives you the choice of IPFW, IPF, or PF. OpenBSD gives you PF. NetBSD gives you IPF or PF. All of those have much larger / better features sets than IPChains / IPTables, and work a *lot* better in NAT/PAT/MASQ situations. These packet filters are also truly stateful (last time I checked IPTables, it wasn't truly stateful without a bunch of extra patches).

Linux makes an OK home firewall. But I wouldn't use it anywhere near a business.

We use FreeBSD 4.9 on Pentium 166 MHz systems with 128 MB RAM using IPFW to server secondary schools with just under 300 student computers. Haven't had any problems yet with network slowdowns or dropoffs or anything. These are on T1s in the remote schools, and 8 Mbit cable in town.

(I had problems keeping a similar box running Linux and IPTables working on my home wireless T1-equiv link.)

Re:Don't use Linux for this

[hbackert]

Linux makes an OK home firewall. But I wouldn't use it anywhere near a business.

I have to question this. Given the few arguments named, it's easy to do so. I built lots of routers with and without VPN (FreeS/WAN and recently Racoon), proxy services (for http, ftp, mail), firewalls doing NAT, VPN and anything else you can imagine. Customers read about a feature on Cisco routers/PIX and they want to have this suddenly. It often does not make much sense, but the customer is king. (I don't connect to T1s directly though, always Ethernet-only, avoiding T1/T3 with provider supplied routers which output Ethernet and don't do anything else, and everything is data-only, no voice (VoIP counts as data)).

Problems so far: 0, except hardware failures. That's my about 5 years of experience in this business.

No router was 0wn3d, we patch them regularily when there is a security patch needed, the firewall is pretty safe and closed, no outages, even when hundreds of notebooks connect to them. Collegues have way more problems with Cisco PIX/Checkpoint firewalls. So if Cisco & Co is ready for business, then Linux is ready too.

I do know that *BSD is working well too and I very much like the ports system, but there is nothing wrong with Linux and e.g. Gentoo. Both run stable and outperform all but the most expensive Cisco/Checkpoint gears I know.

Dont bother

[moosesocks]

If your company can afford to pay 1000 people and run a T3, they have the money to buy a PROPER Cisco-based setup.

Oh. And hire an experienced professional to install it (i don't dobut that you could manage it, though). I wouldn't trust a job of this size to someone who 'did it once at home and it worked'. The enterprise works much differently than your basement.

If you set it up and something goes wrong, you, my friend, are screwed.

I'm already doing something similar

[David E. Smith]

There's a whole niche market for "stripped-down versions of Linux" that handle things like this.

Currently, I'm using Mikrotik RouterOS as a core router. It's at a small ISP -- 400 or so high-speed customers, 3000 dialup customers (400-500 of which are connected during peak times). Standard routing stuff (30 or so internal static routes, big deal). Couple hundred firewall rules (some for stopping Windows worms from spreading, some for general network security, some to help keep the nastier spammers in check). And BGP, taking a full BGP feed from our upstream, plus a couple multihops from places like Cymru's bogons project. And it doubles as a PPTP server so I can securely work from home (in a gesture of supreme irony, I can't get Internet connectivity from the company I work at).

And some other stuff I can't think of right now.

All this is running in a 1U system I got from eRacks (they make good cheap stuff), except for the hard drive, which I yanked and replaced with a 64MB IDE-flash drive from these guys. Celeron 1.3GHz, 512MB RAM. The system never ever, even during peak times, goes over 10% CPU load.

This isn't quite up to the specs the original author was looking for, mainly because this hardware isn't also doing the T1 stuff. (It's got plain old boring Ethernet to an older Cisco router, to which our four T1s are connected, but the Cisco is basically just a really big media converter.) But given how low the hardware utilization is on this unit, and how underpowered this system is as compared to current hardware, I think it shows that the notion is quite feasible.

Re:no can do sorry

[prisoner-of-enigma]

"true...sort of. if those 3000 users aren't doing much other than checking email and browsing the web. If they are doing some serious stuff; which they may not be who knows?; then chances are good that 3000 users means a heck of a lot of traffic."

You're just not grasping this concept very well, are you? Let me spell it out to you very slowly: the limiting rate here is his T3 connection! No matter what these 3000+ users are doing, they cannot generate more than 45Mbit/sec of traffic because that's the max the T3 will handle (actually it's slightly less than even that due to overhead). So, with a single 100Mbit Ethernet card for the internal net and a single 100Mbit Ethernet card for the external net (or a T3 PCI adapter, it doesn't matter which), what's the max traffic you're ever going to have to deal with? Bingo! 45Mbit/sec, which is well within the capabilities of a single 100Mbit Ethernet card. It sure as hell isn't a problem for the PCI bus, which maxes out at 133MB (bytes, not bits) per second. That's 1064Mbit/sec, compared to the T3's 45Mbit/sec.

So, in short, it doesn't matter whether you've got one user, 5,000 users, or 50,000 users -- they are restricted by the smallest pipe in the system, and that's the T3. This should be obvious, but for some reason you keep thinking that more users can somehow generate more than 45Mbit/sec of traffic through a T3. Sorry, it can't be done. Perhaps you're thinking about using a PC as a switch instead of a firewall or something, but as a firewall you are completely and totally wrong.