Slashdot Mirror
x86 Commodity-Hardware Router?
neomage86 asks: "I recently had to set up a router for a small company, only five users at any given time, and the needed VPN capabilities are built in. So, instead of using a Cisco or other embedded router, I decided to just install Linux and IPTables on an old 200 MHz PII I had lying around. It's been working fine, and I'm thinking about doing something like this for a much larger network (3000+ users). Does anyone have suggestions on how much I will have to beef up the hardware to provide IP Masquerading for about 1000 users on a T3; provide network-layer filtering of the transmission; and route between 4-5 internal subnets?"
VPN can be a real resource hog... word is though, that the Via C3 has some sort of processor level instructions to help accelerate this. Has anbody else heard of this?
No seriously, you're going to swamp your PCI bus if you're doing routing between internal subnets. Goodbye, LAN throughput. Not to mention what merry hell you'll play with the CPU with VPN and firewall rules.
Your solution is great for a small place, or even a large place in a dedicated niche (like only VPN and/or firewall, or monitoring/IDS.) I wouldn't do something that ambitious with PC hardware though.
If I was a potential customer of yours, red alarm bells would be going of in my head.
Instead of offering standardized equipment that can be managed via console, ssh or SNMP by any competent network engineer, you offer some customized linux router solution that will always need to be handled differently.
What advantage does your solution offer?
Is it worth "saving" a little money up front, only to need to seek out your consulting services later?
Conformity is the jailer of freedom and enemy of growth. -JFK
I would personally go with a BSD flavor rather then Linux. Don't get me wrong Linux is great but BSD was designed with routing in mind. You will be able to get away with less hardware and out of box things like OpenBSD are going to be more secure then a commodity Linux.
Do the math. If your homebrew system goes down, you will be burning the time of 1000+ people ($60,000) per hour. With those kind of numbers it doesn't pay to do it on the cheap. Get a redundant Cisco system with plenty of power backup.
Microsoft might be a place to start. It's entirely possible they have recommended specs for a win2k router (why anyone would do this, well extra computers and win2k are available, I don't know) against some set of expected clients.
I would probably take your available capacity for the same system running linux at something like 1.5,1.2 times what microsoft recommends if I wanted to be really conservative.
But considering the number of clients you're serving, would it really be so bad to overbuild it? What's a P4 with a gig of ram go for anyway?
And for said couple-hundred, you're looking to pick a secondary network card, along with a 2Ghz or so Athlon or P4 of your choice with a motherboard with a built-in network card. The built-in network card is important for a router.
An Athlon-64 or above would be ideal, simply because you'd be able to mount ludicrous amounts of memory on the box, which is pretty much all that could ever matter for a router/firewall app, as Linux can easilly support logging anything you want to a remote boxen.
Realistically though, I've routed 8 T1's at 80+% capacity in both directions among 650 laptops before, including 3 seperated subnets, all routed through one box.
The box was a Celeron (P2) 800Mhz we'd downclocked to 633Mhz (standard practice at my company, downclock everything for live events for stability) and it used around 10% of the CPU at peak once configured correctly.
By 'correctly' I mean having the T1's all coming in on a seperate PCI bus from the actual network cards for the subnets. Specifically, the built-in ethernet turned out to be on a seperate PCI bus from the actual PCI slots in the case. Configuring the box to take advantage of this dropped CPU load from 80+% to ~10%.
So... for a T3 fully loaded? I'd say get a 2.0Ghz machine just for breathing room, and give it at least 2GB of memory, as neither is that expensive and will leave plenty of breathing room for things like IPSec or other fancier options down the road without any problems.
It doesn't matter what sort of PC you are using...you simply cannot pump that much through a standard PC. 3000+ users? forget it. You are going to need a cisco my man. Unless anyone knows if those quad cards can route between connectors at faster (much much muuuuuch faster) than the PCI bus will allow.
/* oops I accidentally made a comment, sorry */
It's do-able but segment out the functions at that point. Do you really want to try to route between subnets as iptables is traversing the masq table? Get three boxes; one box for routing, one for vpn traffic and one for actual firewall/masqing. IBM has crypto boards for accelerating SSL/IPSec stuff with linux drivers IIRC for your vpn box. Also, with three boxes you can take down the vpn without taking down the internet connection.
I would suggest getting PIII's instead of PII's though, but check where bottle neck's may be PCI bus, CPU processing packets, NIC not doing so well... etc. Plus if one box is connected to multiple subnets, it can be dhcp and/or dns and/or wins for them (if you do DNS please use the forwarder's option to forward dns requests to an upstream DNS server if possible).
www.rdex.net
[ -f /etc/default/inetinit ] && . /etc/default/inetinit /etc/default/inetinit, otherwise
/dev/tcp tcp_strong_iss $TCP_STRONG_ISS
/etc/defaultrouter ]; then /etc/defaultrouter | awk '{print $1}' ` /usr/sbin/route -f add default $router /usr/sbin/route add default $router /usr/sbin/route -f /etc/defaultdomain ]; then /usr/bin/domainname `cat /etc/defaultdomain` /var/yp/binding/$nisdomain ]; then /var/yp/binding/$nisdomain /var/yp/binding/$nisdomain/ypservers /var/yp/binding/$nisdomain/ypservers /var/yp/binding/$nisdomain/ypservers /var/yp/binding/$nisdomain/ypservers /var/yp/binding/$nisdomain/ypservers /var/yp/binding/$nisdomain/ypservers /usr/bin/domainname $nisdomain
/etc/gateways exists.
/etc/notrouter -a $numdhcp -eq 0 -a \ /etc/gateways \) ]
/dev/ip ip_forwarding 1 /usr/sbin/in.routed ]; then /usr/sbin/in.routed -s /usr/sbin/in.rdisc ]; then /usr/sbin/in.rdisc -r /dev/ip ip_forwarding $forwarding /usr/sbin/in.rdisc ] && /usr/sbin/in.rdisc -s; then /usr/sbin/in.routed ]; then /usr/sbin/in.routed -q; /dev/ip ip_forwarding $forwarding
if [ $TCP_STRONG_ISS ]; then
ndd -set
fi
defrouters=`/sbin/dhcpinfo Router`
if [ -z "$defrouters" -a -f
defrouters=`grep -v \^\#
if [ -n "$defrouters" ]; thenpass=1
for router in $defrouters
do
if [ $pass -eq 1 ]; then
else
fi
pass=2
done
else
fi
fi
if [ -f
echo "NIS domainname is `/usr/bin/domainname`"
else
nisdomain=`/sbin/dhcpinfo NISdmain`
if [ ! -z "$nisdomain" ]; then
# create the domain, if it is a new one
if [ ! -d
mkdir
fi
# Check if ypservers are available and create the appropriate
# file ; otherwise the ypbind will be started with
# unsecure broadcast option
nisservers=`/sbin/dhcpinfo NISservs`
rm -f
if [ ! -z "$nisservers" ]; then
touch
chown root
chgrp other
chmod 644
for i in $nisservers; do
echo $i >>
done
fi
# finally set the domainname so that NIS will be started later
echo "NIS domainname is `/usr/bin/domainname`"
fi
fi
if [ -z "$defrouters" ]; then
defrouters="`netstat -rn | grep default`"
fi
if [ -z "$defrouters" ]; then
numifs=`ifconfig -au | grep inet | wc -l`
numptptifs=`ifconfig -au | grep inet | egrep -e '-->' | wc -l`
numdhcp=`ifconfig -a | grep DHCP | wc -l`
if [ ! -f
\( $numifs -gt 2 -o $numptptifs -gt 0 -o -f
then
echo "machine is a router."
ndd -set
if [ -f
fi
if [ -f
fi
else
forwarding=`/sbin/dhcpinfo IpFwdF`
if [ -z "$forwarding" ]
then
forwarding=0
fi
ndd -set
if [ -f
echo "starting router discovery."
elif [ -f
echo "starting routing daemon."
fi
fi
else
forwarding=`/sbin/dhcpinfo IpFwdF`
if [ -z "$forwarding" ]
then
forwarding=0
fi
ndd -set
fi
I decided to just install Linux and IPTables on an old 200 MHz PII I had lying around.
It was either 233MHz or 266MHz if it was a Penntium II, or it was a Pentium or Pentium Pro.
You're going to run into a hardware bottleneck, mostly because of the PCI bus. You simply can't throuhput more than your 10MBit card can handle, and you'll be lucky if you get that much through. No non-dedicated machine is going to be as fast (and by dedicated, I'm referring to something specifically designed to be a router/switch), they just aren't designed that way. Bus limitations aren't as important in a machine that will be limited by external factors such as a broadband connection.
Go invest in a good Cisco box...a router or a switch of some kind...that will do this for you. If you want, hook this box up internally to do some monitoring, though last time I checked Cisco boxen do most of that for you. Really, you're moving from a homebrew, college dorm room solution to the real world.
--trb
Especially with a PC-based router the customer needs to understand that he is now buying a *service* instead of a machine. It's not too smart to leave *any* box live on the Internet, or even in a customer's office without some sort of maintenance, but for a Linux (or Windows, any flavor) box it's potentially dangerous.
The number of exposures for Linux doesn't particularly bother me, for a box that's being actively maintained. For a generally non-service box you don't even need to be paranoically prompt about getting fixes applied. But I'd get worried about an *appliance* PC.
The living have better things to do than to continue hating the dead.
If you're just powering a T3 and 6 10/100 subnets, you could get by on
v c/ps2 030/products_data_sheet09186a0080189f0a.html
P4 2.xxGhz (assuming moderate VPN usage)
512MB-1GB RAM depending on how many simultaious connections you're working with. The more connections the more memory eaten up
Hard drives: minimal config.
Motherboard & NIC's: Depending on how much you're 10/100's saturate, you may want to get some 66Mhz 64bit PCI cards instead of regular 33/32's. Eg:
http://www.cisco.com/en/US/products/hw/vpnde
It all depends on how much simultanious traffic you're looking at. You can use the analogy that the PCI bus is a network switch's backplain. 66/64's can transmit a theoretical maximum of 4gbits/sec. so it should be enough for anything you throw at it. 33/32's maximum theoretical is 1gbits/sec. but in reality expect for much less.
Bye!
If you're going to be useing a PC (with NICs in the PCI slots) to drive the T3, you're already throwing away money. The PCI bus is unable to handle that much bandwidth.
PCI bus = 127Mbps = 15.8mbps
T3 = ~50mbps
Congradulations! You've just thrown away 35mbps! There is a reason to go with a Cisco router: the asic is able to handle that much bandwidth w/o overloading.
T3 = ~50mbps
Wrong - you got the division wrong
PCI Bus 127 MBytes = ~1Gbit/sec
T3 = ~45 Mbits/sec
Are you telling me the fastest a PC bus can go is 15 MBits a second ??? I know of Intel class hardware that can keep 100 MByte going over a Gbit NIC. Lets not even go into shipping PCI-X busses and soon to be shipping PCI-Express busses that are significantly higher throughput than this.
Now that we have that problem solved, what you will run across with multiple 100Mbit network cards running into your system is a higher latency than your low end cisco router, and lower reliability potentially (all though in both cases, I suspect software reliability is orders of magnitude lower than hardware reliability). If you can live with the higher latency going through a PC based router - go for it, you might save a few bucks...
I have mod points and I am not afraid to use them
Uh, PCI bus is 128 Mega-BYTES per second maximum thruput. That is 1 Giga-BIT per second. And that is just for the standard 32bit at 33MHz speeds. There are plenty of Intel based servers with 64bit and 66Mhz PCI variations.
-- I am not a fanatic, I am a true believer.
If you get something with PCI-X, instead of standard PCI, you'll have a lot more bandwidth.
PCI-X is 64-bit, and with multiple cards, they'll probably be running at 100MHz. Vs. standard PCI at 32-bit, 33MHz, that's 6X the bandwidth, or about 90mbs, more than enough.
Just make sure you get one with enough 100MHz PCI-X slots for all your NICs. Many boards come with, say, 2 100MHz PCI-X, 2 66MHz PCI-X, then some standard slots. (Note that it's 2 slots per bus, and for more slots, the mobo will have multiple buses.)
Of course you'll need PCI-X network cards to handle that. Does anyone make those?
This is starting to sound expensive. Not sure it would actually save you any money. It would be cool, though.
-Uberhund
If you're inexperienced, try to get everything from one vendor so that getting it all working together is their problem, not yours.
You could do worse than a http://www.nortelnetworks.com/products/01/passport /lan/.
The Airport Base Station (original) is a very good "take apart" to learn how to build your own router.They couldn't be more simplistic in design and implementation.
Yell & scream & rant & rave... it's no use... you need a shaaaave ~ Bugs Bunny
Admittedly, the pci bus will probably be the first absolute roadblock with a good machine, but I think you are all underestimating it's ability.
I did a quick test on my home network to make sure. I easily got 97 Mbps using NFS to transfer (multiple simultaneous) files between 2 machines on 100 Mb ethernet. I think that is pretty conclusive evidence that the PCI bus will not be a limit even on a DS3(T3), which only goes 51 Mbps. One of these machines even has the video card on the pci bus.
Anecdotally, why would we even bother building Gigabit ethernet if unable to read the data (given, the bandwidth is shared, but anyway).
For stabilities sake, you should probably not try to do the routing among internal subnets with this box. However, if most of the internal traffic is accessing external hosts, this would also be possible, since most of the traffic will be crossing this box anyway.
Linux wins the speed race, hands down.
You will get 0wned much, much faster with Linux than with OpenBSD.
The packet filtering software on Linux is horrible. The syntax is just nasty. And there are no guarantees it won't change again with the next kernel release.
Use a BSD system, with a real packet filter. FreeBSD gives you the choice of IPFW, IPF, or PF. OpenBSD gives you PF. NetBSD gives you IPF or PF. All of those have much larger / better features sets than IPChains / IPTables, and work a *lot* better in NAT/PAT/MASQ situations. These packet filters are also truly stateful (last time I checked IPTables, it wasn't truly stateful without a bunch of extra patches).
Linux makes an OK home firewall. But I wouldn't use it anywhere near a business.
We use FreeBSD 4.9 on Pentium 166 MHz systems with 128 MB RAM using IPFW to server secondary schools with just under 300 student computers. Haven't had any problems yet with network slowdowns or dropoffs or anything. These are on T1s in the remote schools, and 8 Mbit cable in town.
(I had problems keeping a similar box running Linux and IPTables working on my home wireless T1-equiv link.)
A little off topic, but why has no one on slashdot complained about Cisco using the term
"hackers" in their TV ads about bad guys out of networks?
my network is basically served by a Tandy Sensation 2, a 486sx/33 with a 487slc/33 coprocessor installed. 40MB RAM, 2GB hard disk. It runs router services for .. uh.. 4 computers currently, and has run services for 10-12 computers. It also sports the network's email server, for three domains that I receive mail at. And a MySQL server, that I haven't had much use for lately, but it used to gain a few thousand SQL requests a day.
"Champagne for my real friends - and real pain for my sham friends!" http://ericblade.postalboard.com/
You're probably thinking of an ISA buss, which did have trouble coping with 10Mb at flat splat. A PCI buss could probably copy with around 1200Mb with decent cards before the electrons started coming off at the corners (implying that you'd need two PCI busses to get that much traffic back out of the box again).
you, sir, are an ass. I guess I am too for responding.
If your company can afford to pay 1000 people and run a T3, they have the money to buy a PROPER Cisco-based setup.
Oh. And hire an experienced professional to install it (i don't dobut that you could manage it, though). I wouldn't trust a job of this size to someone who 'did it once at home and it worked'. The enterprise works much differently than your basement.
If you set it up and something goes wrong, you, my friend, are screwed.
-- If you try to fail and succeed, which have you done? - Uli's moose
There's a whole niche market for "stripped-down versions of Linux" that handle things like this.
Currently, I'm using Mikrotik RouterOS as a core router. It's at a small ISP -- 400 or so high-speed customers, 3000 dialup customers (400-500 of which are connected during peak times). Standard routing stuff (30 or so internal static routes, big deal). Couple hundred firewall rules (some for stopping Windows worms from spreading, some for general network security, some to help keep the nastier spammers in check). And BGP, taking a full BGP feed from our upstream, plus a couple multihops from places like Cymru's bogons project. And it doubles as a PPTP server so I can securely work from home (in a gesture of supreme irony, I can't get Internet connectivity from the company I work at).
And some other stuff I can't think of right now.
All this is running in a 1U system I got from eRacks (they make good cheap stuff), except for the hard drive, which I yanked and replaced with a 64MB IDE-flash drive from these guys. Celeron 1.3GHz, 512MB RAM. The system never ever, even during peak times, goes over 10% CPU load.
This isn't quite up to the specs the original author was looking for, mainly because this hardware isn't also doing the T1 stuff. (It's got plain old boring Ethernet to an older Cisco router, to which our four T1s are connected, but the Cisco is basically just a really big media converter.) But given how low the hardware utilization is on this unit, and how underpowered this system is as compared to current hardware, I think it shows that the notion is quite feasible.
and they seem to be doing pretty well. I went looking for weird NIC hardware and came across Imagestream. They make big routers with Linux at the core, on x86 hardware in industrial form factors. Definitely worth a look.
Also on the thread of interface cards, try Mikrotik. If you're doing wireless, the MiniPCI carrier boards will make your day.
Full disclosure: I'm not related to or affiliated with either of those companies in any way. I've never even bought anything from either of them. I just came across them while searching and thought they were bookmark-worthy.
Our main firewall for our hosting company is a 2Ghz P4. We are not doing vpn, which would be the most resource intensive, but our T-3 line comes directly into it and we have a ton of firewall rules. There is never a load on the box, except when nimda hit :).
And with bridging you can have two transparent firewalls (no ips) that are redundant, using Spanning Tree Protocol. Pretty cool.
As long as he;s not trying to do VPN encryption on it, he probably will. Personal experience tells me a P100 (running FreeBSD, not Linux) can easily firewall a 100Mb network link for a few dozen users, so anything P2 class shouldn't have any trouble at all.
No seriously, you're going to swamp your PCI bus if you're doing routing between internal subnets. Goodbye, LAN throughput. Not to mention what merry hell you'll play with the CPU with VPN and firewall rules.
The PCI bus is not an issue. PCI is ~120M/sec and even 100T ethernet is only 10M/sec.
Any P2 class PC with some decent network cards (I personally recommend Intel EtherExpress cards) should handle his setup _easily_. You'll probably run out of (ie: have to manually tweak) OS resources long before the hardware is stressed. The bottleneck is not going to be raw throughput per se, it's going to be how fast the machine can process individual packets.
dude.. dont listen to most of these clowns... the pix 520 was nothing but a glorified pII processor with @ 128mb of memory.. if you ever opened the thing up you would see.. I say the 4 port pci cards are great.. cisco uses the intel brand cards.. their ios is stored on a flash card.. ok get a decent long life cp flash with ide to flash adapter.. maybe a 2-4u case. Instant pix box.. just because it doesnt say cisco doesnt mean it is not good.. make sure whatever os or rolled firewall you use it is a good one.. I personally am an OpenBSD man myself..
you could mirror the hardware and use CARP for load balancing/failover.. hrm.. now that is a setup fit for kings..
well, to give you an idea of what can be done, i'm running a slackware based p100 as a gateway/firewall/router/name server for my entire home network. there are 12 computers between a bunch of people. it sits between the cable modem and two switches with 2 10/100 $10 ethernet cards in it. of course this is not a large scale network, but it shows you what a p100 can handle with ease. for security i disallow all incoming connections except ssh. and if you think it doesn't get much traffic, i keep gnutella running on one of my computers 24/7 with about 100 active downloads at any time.
course bumping the hardware to like most guys say.. 1.x gig range with more mem would not be a bad thing...for sure.. some NIC cards have their own ipsec encryption processor(if that is what they call them). so think about that too.. personally I dont think you will go wrong.. hell you can always go to the cisco thing if it doesnt work.. turn your hardware to rack mount servers and keep on burning..
Interesting, nobody has 0wned any of the linux systems I admin at work in the year I've been working there.
Just to clarify, this is a project for my High School. They are upgrading the network infrastructure, and I work with the tech-ed department through an internship class. I just wanted to make sure this was reasnoble, before I suggested it to my own bosses.
Pfft, I'll bet you just didn't notice. That's how 0wnz0rd you are.
I want my Cowboyneal
Love the thought but pc hardware is hardly up to mission critical status even with a stable OS on it ata drives fail cpu's overheat junk ram corrupts data a company of 3000+ people cant afford to have downtime from that crap chipset or failed ram and can afford to by something that is more likely to last