No. For you see, I have spent the last ten years on mars. In a cave. With a pillow over my head.
Bonus points if you caught the reference. I probably didn't get it verbatim, though.
"The higher-end routers can do something like 240 Gbps."
The figures from 240-720 gigabits are the backplane performance, not what it can actually route. Hopefully you're aware of the difference. But it will still crank out some serious numbers with CEF.
However... have I ever once talked about comparing CEF to Linux-based routing? Nope. I've been talking about scenarios where you're forced to process-switching. What sort of figures does your 7600 turn in then? Cisco's datasheet says... 20,000 PPS. Yep, 20,000. 30 million down to 20 thousand. What is that, a factor of 1500? Wow. A $2k Linux PC can route 7 times that much. Ouch.
So, it seems to me that my original ideas still hold true: Once you're process-switching, your Cisco is vastly overpriced and underpowered. You seem to assert that you can enable all kinds of features on your 7600 and still use CEF. They're certainly better than the low-end (even able to CEF-switch NAT). But that doesn't mean that they can do EVERYTHING with CEF.
So, moving on to PCs...
"As I said, standard 32-bit 33MHz PCI cannot even handle gigabit ethernet"
And, as I said, any decent machine has wider and faster PCI busses via PCI-X, and/or PCI-E - and usually several busses.
"And the CPU sure as hell is a bottleneck -- just how fast do you think it can talk to the PCI bus? "
It depends on the chipset in question, but it takes an AWFULLY poorly-designed PC to not have enough bandwidth to the chipset to keep a couple of measly PCI-X busses going, and CERTAINLY enough to make 20,000 pps look pretty wimpy.
"But try making a PC route 30 or 40 1-gigabit fiber interfaces like some of the midrange ciscos,"
You're confusing switching with routing. Show me a Cisco that can actually perform all routing functions (including firewalling, NAT, payload inspection, etc.) on 30 or 40 gigabit lines. Sure, you can perform some rudimentary routing functions on their Ethernet switches. Can you hook a few t3s into them? Maybe hook up a couple of OC12s? Can they channelize lines into voice and data? We're talking about different things here.
"The standard PC architecture is not capable of servicing even a single gigabit interface unless you use PCI-X, and even then the CPU is a major bottleneck."
To quote a certain idiot I've heard from, "Uh, dude, PCs come in more than 10-year old low-end shit."
The days when the CPU was a bottleneck for gigabit are long gone. Sure, you could turn off interrupt coallescing which would drive the interrupts up, but we're still not talking about 400 MHz CPUs any more. And guess what... interrupts can be a limitation on a Cisco in pathological cases, too. On a PC, when interrupts or CPU cycles are a problem, you spend a couple grand more and get a few more CPUs - including more interrupt controllers. On a Cisco, you add another zero to the price of your router.
These discussions, when they take place, go round and round on NANOG. People who don't use PCs as routers come up with every reason in the world why they won't work, and then the people who actually DO use them drop some performance numbers that are absolutely astounding.
The real reason why Cisco is still in business is that if you have a problem, they *will* solve it. That problem can be that you need a particular interface, a faster router, a software problem, or just some handholding. If you give them money in sufficient quantities, they *will* take care of it. That's not true of PCs. If you want to point out shortcomings in PC-based routers, that's where you should start.
"Sure, because having 14 unreliable fans is better than quality components and proper thermal analysis and engineering."
It's obvious that you're not an engineer.
"The problem is, fans suck in lots of dust. Which then tends make things worse."
Ah. So we should be fanless, then. Show me the fanless Ciscos.
"Not to mention, what kind of alarm alerts you that the fans are dying?"
Audible alarms? Check, they're there.
"Would you know if a PC started overheating due to, say, dust accumulation or a slowed-down fan?"
Sure. It's all there. I can monitor fan speeds, chassis temperatures, individual temperatures on all four CPUs, and individual temperatures on all ten disks. What's your point?
"Can you swap out the fans without powering down the machine?"
Yep. All fans specified as hot-swap.
"I think it's pretty obvious you don't work at a serious company. In a company with 1500 employees, a total network outage for one hour will cost more than $75,000 just in lost employee productivity."
Lick my sack. I could lose that much from a *single sale* if I was down for an hour. As in *one of hundreds*.
"Just the fact that you consider a 3-year uptime to be something exceptional speaks volumes about the general unreliability of PCs."
"For a router, an uptime of 5 years is not particularly amazing."
It would be for me. After about a year of uptime, one of my Ciscos starts showing rather bizarre behavior which can only be solved by a reboot. Thinking that it must have been something that I was doing wrong, we brought in expensive consultants, which jumped through all kinds of hoops, could figure nothing out, and submitted it to Cisco. But since we have to wait about a year for it to happen, Cisco can't (or just hasn't) had anything done about it.
"Yes, occasionally, ethernet cards *do* fail. But so do t3 cards.
Just don't say that at the interview if you ever want to work in a real-world IT department. You would _not_ be hired."
Yeah. Some pompous, arrogant idiot like you would look down their nose and say "This guy doesn't know what he's doing." Then when one of your interface cards from Cisco failed, your sales department would have to kiss my ass about the service interruption. Been there, done that. Sorry. Reality is on my side.
"If you have all these problems with Ciscos and others don't, what could be the difference between your situation and ours.... Oh yeah, it is YOU."
For the record, I haven't been bitten by any Cisco bugs other than their inability to make an ethernet device which can do autonetogiation. $5 no-names can do it, but not a $300 Cisco card.(*) But that's not to say that they don't bite people, they do surface.
"And as for longevity, 5 years is a minimum I would expect out of any enterprise level cisco gear."
And I'd expect no less from my PCs.:-)
"Wha wha WhaaT? I have never heard of such a guy. Is he like the bad analogy guy?"
I hear that pretty often when talking to colo engineers where the data center was originally all telco. They don't like to deploy anything new, so they make up stories about why they'll only use t3 and other telco interfaces to run a feed twenty feet across a room. Engineers from more modern centers don't pull that quite as often.
(*) The last time a really big IOS bug had to be addressed, the DoS bug, virtually everyone with a Cisco (at least those who cared) had to upgrade their OS. Of course, not all of your settings carry over, and usually your ethernet goes back to autonegotiation, which Cisco can't figure out to save their life. So, a fair number of people all across the world suddenly had flaky, bizarre connection problems. For a month after that, I got calls from end-users all across the country with transfer problems, and I traced *every single one* to their admin having upgraded IOS and not forced the ethernet speed. I've never had a single autonegotiation problem with *ANY* other ethernet device, only Cisco.
Now, I'm certainly not saying that's the only problem they have. It's just their most easily seen (and perhaps most easily solved) technical deficiency. Cisco does a lot of things right, but pretending that they do *everything* right (or have no bugs) is just as stupid as believing the same of Linux.
Look at the 3845 routers. As I recall, they can't even route a measly 45 megabits *if* you turn on services, so they're still not in the league of a modern PC. Here are some power dissipation figures:
AC Without IP Phone Support: 435W AC With IP Phone Support-System Only: 555W AC With IP Phone Support-IP Phones: 360W (1128 BTU/hr)
"What's acceptable down-time for a home PC is absolutely unacceptable for a Cisco-quality router."
True. But we're not necessarily talking about a home PC. Still, for vastly less than a Cisco, you can buy a VERY well-designed server-class PC. Like I've said, you can buy several of them, and have them in hot-failover, and still come out cheaper.
It's funny that people assume that it's mission-critical that the network stay up, but it's alright if the servers go south. For me, either way, the end product is not working, which makes it a wash.
I see that you're talking about the CPU being a bottlneck in conjunction with USB. We agree, USB isn't up to it.:)
The Linux and BSD IP stacks are tremendously good. That's the product of so many people looking at them. And from how much of the time on my Ciscos can be spent on interrupts in "IP input" if I'm doing ACLs and bandwidth restrictions, I think that the APIC controllers (not PIC) really aren't that bad.
"If you truly have a need to enterprise level routers, what it means is that millions upon millions of dollars worth of business is dependant on this, and you will not try to save $50k by going with an open source solution."
You will if the Open-source option has all of the features you want and need. IBM puts a loooooot of money into selling Linux on their enterprise-level servers. The same *could* be true of routing, but there would need to be someone of considerable size interested in putting that sort of effort into it - and the effort would be large indeed, considering the hardware needs as well.
"Aside from calling you out on what seems to be a blatant advertisement, here goes. Can Smoothwall run with a automated backup box if the primary box goes down (see Cisco's HSRP or OpenBSD's CARP)?"
I don't know about smoothwall, but there are various user-space packages for Linux which will monitor remote machines, and take over (even performing ARP spoofing if desired to take over the IP address) when needed. It might take five minutes of operator configuration if it doesn't come standard with smoothwall, but that's what you big-time network engineers get paid for, isn't it? And as I've pointed out before, you can buy twice as many Linux-based routers as Cisco routers, doubling your redundancy, and still come out money ahead.
"How many ethernet ports can Smoothwall handle?"
Being based on Linux, there is a finite number of ethernet ports. I don't recall what it is, but it's vastly more than you'd ever be able to connect to a machine anyway. You could happily stuff quad-controllers in every available slot, and not even come close to the limit.
As for LDAP and VPN, it's the same as the failover. If Smoothwall doesn't have it, it's trivial to install and configure. In fact, the flexibility of a Linux-based solution can only be higher than what you get from a router company.
In the end, as I've said so many times, software is not the problem. If anything, software is in favor of Linux. It's the hardware. You simply can't get the interfaces for a PC that you can for a Cisco, and when you can, they're not always as full-featured.
It's tought to beat the BSD IP stack. Really tough. But again, the real problem isn't software, it's the network interfaces. My guess is that the list of different network types to which you can attach with BSD is more or less comparable to that of Linux, which is far, far fewer than Cisco.
There are two breakpoints. The PC wins in cheap CPU cycles (and number of CPUs), memory bandwidth, memory size, etc. - but at a certain point, I/O becomes the bottleneck. At that point, the switching fabrics themselves start to get very expensive.
The second is when you need anything but the most common network interfaces. Let's say that you want a t3 interface. You can get a PCI t3 interface that will run with linux - for about half of the cost of a Cisco t3 interface - but you can only run it as clearchannel. With the Cicso, you could run voice and data on seperate channels, and all kinds of fancy things. And once you're talking about anything other than a t1 or t3, you'll be hard pressed to find an interface card for more than a few other networking architectures.
Can a Linux box route packets faster than a WiFi router? Of course. Look at some of the numbers for the LVS project. You're not just routing, but also doing load-balancing as well, and if you can't saturate a 100 megabit line (even with NAT), then you have *really* ancient hardware. Routing a full gigabit is a more of a challenge (only a LITTLE challenge, even somewhat modest hardware can push 600+ mb/s), but price out a Cisco that can route a full gigabit, and see what you think.
Buy an Opteron instead. Or buy the low-voltage Intel or AMD chips.
I have a router built from a dual P133 that's really old, but even routing 6 megabits in demanding situations it doesn't even break a sweat. The chips don't even have fans on them. I measured the power draw from the wall (including power conversion losses) at just 45 watts under load.
If power is a HUGE concern, the Eden/Crusoe/Nehemia chips are *really* low-draw, and at a gigahertz or so, can do a pretty decent job of routing. They don't have floating-point circuitry to consume power (even through leakage current), which is just fine - you don't usually do things like sqrt(x) while routing!
I highly doubt that CPU would be a bottleneck, I've seen far too much data being routed on a PC to even think about that. In any similar situation on a Cisco, you'd be even more limitted unless you spent GOBS more money. And if CPU cycles really are a limitation, well, the Linux networking code is very well multi-threaded (although I'm not sure about the routing daemons), and more CPUs are dirt-cheap. Recently I priced out a mid-grade Cisco and a T3 card, and for less, I could buy a 4-socket Opteron system with 8+ gigabytes of memory.
System memory being a limitation? Well, we're talking about the days when any decent server board will have at least 6 DIMM sockets (how many does your Cisco have?), and at least a 128-bit, DDR-400 memory controller. How about your Cisco? Yes, the VERY high-end ones can compete in that area, but again, you're talking at least an order of magnitude more money.
A plain-old PCI bus, at 100 MB/s real-world throughput (or slightly less, depending on the chipset), would be a limitation, but again, any decent server motherboard has at least one or two PCI-X busses (200-800 MB/s each) and a PCI bus. And recently, they've got PCI-E bandwidth out the wazoo. In fact, in any decent motherboard, the I/O limitation you'd hit first would be from the CPU to the chipset, which would be ~14 gigabits/sec on an Opteron, and anywhere from comparable to much less on an Intel, depending on how much of the FSB was eaten with memory access.
USB would, of course, be a bottleneck - another mechanism would have to be figured out. PCI tends to have a very low limit on the number of available slots, so it's not as ideal for highly-connected routers, but something could be worked out without too much trouble.
Again, like I've said in other ports, the only real problem preventing wider use of Linux-based routers is the interface cards. You just can't beat Cisco in that arena. No matter what strange interface you need, if you pay them money, they'll set you up. The same can NOT be said of PCs!
"Most of the white-box server manufacturers just buy cheap Chinese-designed crap for power supplies, fans, motherboards, and so on, with no real quality control, testing, or engineering involved."
As far back as any of my Ciscos and servers go (almost a decade), I've had *one* power supply failure out of 20-something servers that have been in use, and that was in a box that yes, was a cheap box - with ten of them in a load-balanced pool, we don't need the expensive stuff. But of any of the servers of any consequence - and the Cisco - I have yet to have a power supply fail.
As for fans, luckily, the Ciscos haven't died, either. But if they did, on some, it would take just *one* fan to fail, and the unit would be toast. As a comparison, many of my servers have 6-14 fans, in redundant push-pull pairs. To make it better, if a fan dies, the rest of the fans SPEED UP to compensate.
And you also have to look at the turnaround time for a replacement. If I don't have a spare for each sitting on a shelf, I can drive three blocks and have a temporary replacement for the PC in twenty minutes - but of course, with the cost savings over a Cisco, you could have an entirely spare machine in place anyway, and still be far ahead in terms of money.
"Yeah, well a Linksys router is vastly cheaper than a Cisco, too. The problem is, it's not very dependable."
Yeah. Those PC-based servers aren't very dependable. The 3+ year uptimes on a dozen machines that I had to kill just to move them to a different facility was just an illusion. And all of the Cisco bugs that have bitten people in the butt were an illusion, too.
You're like the guys who tell me that I need to use a t3 connection to transport data from one side of their data room to the other because ethernet "isn't reliable". I ask them point-blank when the last time they had an ethernet failure was, and so far, they haven't been able to give me a single answer. Yes, occasionally, ethernet cards *do* fail. But so do t3 cards.
"Uh, dude, a USB controller would be lucky to push 50 megabits. And that's with horrible latency and lots of lost packets. "
Did you not read when I said that I knew that it wouldn't work? In theory, they have gobs of bandwidth. In reality, they don't. I know that.
"The main problem is the lack of suitable software, and the bottleneck the CPU would create"
Bottleneck? How is a multi-gigahertz CPU with tens of gigabits of I/O to the chipset and gobs of gigaBYTES of memory I/O going to be a bottleneck?
I have an old Cisco 1721 where the CPU is maxed out on a 3-megabit line if you use bandwidth restrictions, just because it can't even keep up with the counters. On the other hand, I have a dual P133 that I picked up for $40, firewalling, translating, and filtering a 6-megabit line at full tilt barely even keeps the CPUs warm.
CPUs are only bottlenecks in Ciscos, because they like the thought of selling you, say, a 3640 instead of a 3620 just because the CPU is a few hundred megahertz faster. There's no competition. In the PC world, CPUs are ridiculously faster than necessary just for bragging rights.
You're right, I forgot. Make that "MASSIVELY" less expensive than a Cisco, since you won't have to pay yearly subscription fees to get OS updates. Either way, you need a person to manage them, so that's a wash.
"The number 1 problem with Xorp is that it supports only a tiny fraction of standard Internet routing protocols. They don't have the developers to support anything more than a bare-bones software router."
Once you've got OSPF, RIP, and BGP, I would think that would cover a pretty good percentage of cases that you're likely to run into. And those are already available for Linux, you don't need anything fancy for them.
There are companies releasing high-end networking products that are nothing more than a PC motherboard and their software. A while back, one of the load-balancer companies (I think it was f5, but I don't recall for sure) contracted with Tyan to build their motherboards, with 4 (or more ) gigE controllers, each on it's own PCI-X bus, and Tyan also sold the board to the public.
The main reason that Cisco doesn't use commodity PC parts in their low- to mid- end routers is that if people knew they were getting nothing more than a $4,000 PC for their $15,000, they'd be pretty pissed. Also, there would be that many more people trying to "crack" IOS to make it run on white-boxes, and that opens up a whole new line of revenue drain for Cisco. (Not that people don't obtain unlicensed copies for their Cisco hardware, though...)
Switches and routing are different things, you can't really compare the two. And again, in their router module, if you implement any sort of ACL, are you still avoiding process-switching?
As for "suspect hardware", are you telling me that Cisco's fans, power supplies, and network modules aren't just as susceptible to failure as a regular server? If so, I've got a bridge to sell you, too. And for less than the cost of a single Cisco, you can have a hot- or cold-spare in place as well.
Sure, there is a high-end where a white-box PC won't handle it. But for any job that a white-box *can* handle, it will be VASTLY cheaper than a Cisco.
... until you use an ACL (or any other useful feature of said Cisco), then you're back to process-routing, in which case, it's going to be orders of magnitude slower than a multi-GHz CPU with mammoth memory bandwidth.
The idea of Ciscos and others using custom hardware to accelerate the routing is, in great measure, over-hyped. Yes, they do have some hardware that GREATLY speeds things up, but in most cases, it only works if you're not using any of the features that make their expensive equipment truly useful. Most of the nice features will kick you from CEF to process-switching, and at that point, a modern PC has *gobs* of CPU cycles, memory bandwidth, and even I/O.
I/O used to be pretty pathetic for PCs, but when you look at motherboard chipsets with up to 32 PCI-E lanes coming off of them, that gives you a theoreticaly 80 GB/s bidirectional transfer. Realistically, the connection from the CPU to the chipset would be a limitting factor, but if you're talking about Opterons, you get 12.8 gigabits/second. You can find Cisco routers that will beat that, but you're talking about more than an order of magnitude in price difference.
The largest impediment is not software, but hardware. The two benefits to a Cisco are that (A) there is someone who *will* fix your problem for a fee, and (B) You can buy an interface card for ANY network type out there.
As for (A), the same will likely become available for this if it isn't already. (B) is a lot harder. When you get into odd network types and high-speed telco lines, it becomes a bit more difficult - it isn't as easy as just calling your Cisco salesmonkey and buying the card you need.
It should be noted, however, that adding a card to a Cisco isn't always painless. I've had to upgrade the OS - which involved upgrading both memory and flash - just to support another ETHERNET card. How many decades has Ethernet been around for, and they want an OS upgrade to support one? And only to support an additional card, the built-in ethernet worked just fine.
Right now, we're using a Linux router for ethernet routing within our data center, which it handles just fine. As soon as our Sangoma cards show up, it's also going to handle a T3 to our office as well - but only clearchannel, we can't split it between phone and data (as I'd like to do.)
A while back, I had a rather perverse thought. You can hook up a LOT of interfaces to a high-end Cisco, and most routed telecom isn't very high-bandwidth. A T3, at a measly 45 megabit, is still very small considering the throughput of today's hardware. An OC3, at 155 megabits, still isn't much. The perverse thought was that if someone would come up with T1 and T3 modules with integrated CSU/DSUs that connected via USB or firewire, you could stuff a machine chock-full of 4-port controller cards, and be able to hook up 20 or more interfaces very quickly, and easily. In theory, each USB controller card *should* be able to push the ~200 megabits without much trouble, and even a plain old 32/33 PCI bus could *almost* handle the 110 MB/s of all 20 lines at full-tilt. Realistically, however, I do know that USB has many deficiencies which entirely prevent it from fulfilling that task.
If you can't fix it yourself, you call someone who will charge you to fix it for you. Such support is available for nearly all medium-scale open-source projects. Asterisk is a perfect example, Digium saw the opportunity to not only sell the hardware to make it work, but to make money off of software support as well.
Slashdot Mirror
"Have you heard of the 7600 series, perchance? "
No. For you see, I have spent the last ten years on mars. In a cave. With a pillow over my head.
Bonus points if you caught the reference. I probably didn't get it verbatim, though.
"The higher-end routers can do something like 240 Gbps."
The figures from 240-720 gigabits are the backplane performance, not what it can actually route. Hopefully you're aware of the difference. But it will still crank out some serious numbers with CEF.
However... have I ever once talked about comparing CEF to Linux-based routing? Nope. I've been talking about scenarios where you're forced to process-switching. What sort of figures does your 7600 turn in then? Cisco's datasheet says... 20,000 PPS. Yep, 20,000. 30 million down to 20 thousand. What is that, a factor of 1500? Wow. A $2k Linux PC can route 7 times that much. Ouch.
So, it seems to me that my original ideas still hold true: Once you're process-switching, your Cisco is vastly overpriced and underpowered. You seem to assert that you can enable all kinds of features on your 7600 and still use CEF. They're certainly better than the low-end (even able to CEF-switch NAT). But that doesn't mean that they can do EVERYTHING with CEF.
So, moving on to PCs...
"As I said, standard 32-bit 33MHz PCI cannot even handle gigabit ethernet"
And, as I said, any decent machine has wider and faster PCI busses via PCI-X, and/or PCI-E - and usually several busses.
"And the CPU sure as hell is a bottleneck -- just how fast do you think it can talk to the PCI bus? "
It depends on the chipset in question, but it takes an AWFULLY poorly-designed PC to not have enough bandwidth to the chipset to keep a couple of measly PCI-X busses going, and CERTAINLY enough to make 20,000 pps look pretty wimpy.
steve
"But try making a PC route 30 or 40 1-gigabit fiber interfaces like some of the midrange ciscos,"
You're confusing switching with routing. Show me a Cisco that can actually perform all routing functions (including firewalling, NAT, payload inspection, etc.) on 30 or 40 gigabit lines. Sure, you can perform some rudimentary routing functions on their Ethernet switches. Can you hook a few t3s into them? Maybe hook up a couple of OC12s? Can they channelize lines into voice and data? We're talking about different things here.
"The standard PC architecture is not capable of servicing even a single gigabit interface unless you use PCI-X, and even then the CPU is a major bottleneck."
To quote a certain idiot I've heard from, "Uh, dude, PCs come in more than 10-year old low-end shit."
The days when the CPU was a bottleneck for gigabit are long gone. Sure, you could turn off interrupt coallescing which would drive the interrupts up, but we're still not talking about 400 MHz CPUs any more. And guess what... interrupts can be a limitation on a Cisco in pathological cases, too. On a PC, when interrupts or CPU cycles are a problem, you spend a couple grand more and get a few more CPUs - including more interrupt controllers. On a Cisco, you add another zero to the price of your router.
These discussions, when they take place, go round and round on NANOG. People who don't use PCs as routers come up with every reason in the world why they won't work, and then the people who actually DO use them drop some performance numbers that are absolutely astounding.
The real reason why Cisco is still in business is that if you have a problem, they *will* solve it. That problem can be that you need a particular interface, a faster router, a software problem, or just some handholding. If you give them money in sufficient quantities, they *will* take care of it. That's not true of PCs. If you want to point out shortcomings in PC-based routers, that's where you should start.
steve
"Sure, because having 14 unreliable fans is better than quality components and proper thermal analysis and engineering."
It's obvious that you're not an engineer.
"The problem is, fans suck in lots of dust. Which then tends make things worse."
Ah. So we should be fanless, then. Show me the fanless Ciscos.
"Not to mention, what kind of alarm alerts you that the fans are dying?"
Audible alarms? Check, they're there.
"Would you know if a PC started overheating due to, say, dust accumulation or a slowed-down fan?"
Sure. It's all there. I can monitor fan speeds, chassis temperatures, individual temperatures on all four CPUs, and individual temperatures on all ten disks. What's your point?
"Can you swap out the fans without powering down the machine?"
Yep. All fans specified as hot-swap.
"I think it's pretty obvious you don't work at a serious company. In a company with 1500 employees, a total network outage for one hour will cost more than $75,000 just in lost employee productivity."
Lick my sack. I could lose that much from a *single sale* if I was down for an hour. As in *one of hundreds*.
"Just the fact that you consider a 3-year uptime to be something exceptional speaks volumes about the general unreliability of PCs."
"For a router, an uptime of 5 years is not particularly amazing."
It would be for me. After about a year of uptime, one of my Ciscos starts showing rather bizarre behavior which can only be solved by a reboot. Thinking that it must have been something that I was doing wrong, we brought in expensive consultants, which jumped through all kinds of hoops, could figure nothing out, and submitted it to Cisco. But since we have to wait about a year for it to happen, Cisco can't (or just hasn't) had anything done about it.
"Yes, occasionally, ethernet cards *do* fail. But so do t3 cards.
Just don't say that at the interview if you ever want to work in a real-world IT department. You would _not_ be hired."
Yeah. Some pompous, arrogant idiot like you would look down their nose and say "This guy doesn't know what he's doing." Then when one of your interface cards from Cisco failed, your sales department would have to kiss my ass about the service interruption. Been there, done that. Sorry. Reality is on my side.
steve
"If you have all these problems with Ciscos and others don't, what could be the difference between your situation and ours.... Oh yeah, it is YOU."
:-)
For the record, I haven't been bitten by any Cisco bugs other than their inability to make an ethernet device which can do autonetogiation. $5 no-names can do it, but not a $300 Cisco card.(*) But that's not to say that they don't bite people, they do surface.
"And as for longevity, 5 years is a minimum I would expect out of any enterprise level cisco gear."
And I'd expect no less from my PCs.
"Wha wha WhaaT? I have never heard of such a guy. Is he like the bad analogy guy?"
I hear that pretty often when talking to colo engineers where the data center was originally all telco. They don't like to deploy anything new, so they make up stories about why they'll only use t3 and other telco interfaces to run a feed twenty feet across a room. Engineers from more modern centers don't pull that quite as often.
(*) The last time a really big IOS bug had to be addressed, the DoS bug, virtually everyone with a Cisco (at least those who cared) had to upgrade their OS. Of course, not all of your settings carry over, and usually your ethernet goes back to autonegotiation, which Cisco can't figure out to save their life. So, a fair number of people all across the world suddenly had flaky, bizarre connection problems. For a month after that, I got calls from end-users all across the country with transfer problems, and I traced *every single one* to their admin having upgraded IOS and not forced the ethernet speed. I've never had a single autonegotiation problem with *ANY* other ethernet device, only Cisco.
Now, I'm certainly not saying that's the only problem they have. It's just their most easily seen (and perhaps most easily solved) technical deficiency. Cisco does a lot of things right, but pretending that they do *everything* right (or have no bugs) is just as stupid as believing the same of Linux.
steve
Look at the 3845 routers. As I recall, they can't even route a measly 45 megabits *if* you turn on services, so they're still not in the league of a modern PC. Here are some power dissipation figures:
AC Without IP Phone Support: 435W
AC With IP Phone Support-System Only: 555W
AC With IP Phone Support-IP Phones: 360W (1128 BTU/hr)
That's not a power-sipping device there.
"What's acceptable down-time for a home PC is absolutely unacceptable for a Cisco-quality router."
True. But we're not necessarily talking about a home PC. Still, for vastly less than a Cisco, you can buy a VERY well-designed server-class PC. Like I've said, you can buy several of them, and have them in hot-failover, and still come out cheaper.
It's funny that people assume that it's mission-critical that the network stay up, but it's alright if the servers go south. For me, either way, the end product is not working, which makes it a wash.
steve
I see that you're talking about the CPU being a bottlneck in conjunction with USB. We agree, USB isn't up to it. :)
The Linux and BSD IP stacks are tremendously good. That's the product of so many people looking at them. And from how much of the time on my Ciscos can be spent on interrupts in "IP input" if I'm doing ACLs and bandwidth restrictions, I think that the APIC controllers (not PIC) really aren't that bad.
=)
steve
We'd need a t3, and it looks like they don't go that large. I appreciate the tip, though!
steve
"If you truly have a need to enterprise level routers, what it means is that millions upon millions of dollars worth of business is dependant on this, and you will not try to save $50k by going with an open source solution."
You will if the Open-source option has all of the features you want and need. IBM puts a loooooot of money into selling Linux on their enterprise-level servers. The same *could* be true of routing, but there would need to be someone of considerable size interested in putting that sort of effort into it - and the effort would be large indeed, considering the hardware needs as well.
"Aside from calling you out on what seems to be a blatant advertisement, here goes. Can Smoothwall run with a automated backup box if the primary box goes down (see Cisco's HSRP or OpenBSD's CARP)?"
I don't know about smoothwall, but there are various user-space packages for Linux which will monitor remote machines, and take over (even performing ARP spoofing if desired to take over the IP address) when needed. It might take five minutes of operator configuration if it doesn't come standard with smoothwall, but that's what you big-time network engineers get paid for, isn't it? And as I've pointed out before, you can buy twice as many Linux-based routers as Cisco routers, doubling your redundancy, and still come out money ahead.
"How many ethernet ports can Smoothwall handle?"
Being based on Linux, there is a finite number of ethernet ports. I don't recall what it is, but it's vastly more than you'd ever be able to connect to a machine anyway. You could happily stuff quad-controllers in every available slot, and not even come close to the limit.
As for LDAP and VPN, it's the same as the failover. If Smoothwall doesn't have it, it's trivial to install and configure. In fact, the flexibility of a Linux-based solution can only be higher than what you get from a router company.
In the end, as I've said so many times, software is not the problem. If anything, software is in favor of Linux. It's the hardware. You simply can't get the interfaces for a PC that you can for a Cisco, and when you can, they're not always as full-featured.
It's tought to beat the BSD IP stack. Really tough. But again, the real problem isn't software, it's the network interfaces. My guess is that the list of different network types to which you can attach with BSD is more or less comparable to that of Linux, which is far, far fewer than Cisco.
"So where's the breakpoint?"
There are two breakpoints. The PC wins in cheap CPU cycles (and number of CPUs), memory bandwidth, memory size, etc. - but at a certain point, I/O becomes the bottleneck. At that point, the switching fabrics themselves start to get very expensive.
The second is when you need anything but the most common network interfaces. Let's say that you want a t3 interface. You can get a PCI t3 interface that will run with linux - for about half of the cost of a Cisco t3 interface - but you can only run it as clearchannel. With the Cicso, you could run voice and data on seperate channels, and all kinds of fancy things. And once you're talking about anything other than a t1 or t3, you'll be hard pressed to find an interface card for more than a few other networking architectures.
Can a Linux box route packets faster than a WiFi router? Of course. Look at some of the numbers for the LVS project. You're not just routing, but also doing load-balancing as well, and if you can't saturate a 100 megabit line (even with NAT), then you have *really* ancient hardware. Routing a full gigabit is a more of a challenge (only a LITTLE challenge, even somewhat modest hardware can push 600+ mb/s), but price out a Cisco that can route a full gigabit, and see what you think.
steve
Because Cisco is a name that everyone knows, everyone associates with routing, and everyone knows is a major player.
(You could also say that it's because Cisco buys any potential competitor that they can, but that's slightly - only slightly - cynical.)
Buy an Opteron instead. Or buy the low-voltage Intel or AMD chips.
I have a router built from a dual P133 that's really old, but even routing 6 megabits in demanding situations it doesn't even break a sweat. The chips don't even have fans on them. I measured the power draw from the wall (including power conversion losses) at just 45 watts under load.
If power is a HUGE concern, the Eden/Crusoe/Nehemia chips are *really* low-draw, and at a gigahertz or so, can do a pretty decent job of routing. They don't have floating-point circuitry to consume power (even through leakage current), which is just fine - you don't usually do things like sqrt(x) while routing!
steve
I highly doubt that CPU would be a bottleneck, I've seen far too much data being routed on a PC to even think about that. In any similar situation on a Cisco, you'd be even more limitted unless you spent GOBS more money. And if CPU cycles really are a limitation, well, the Linux networking code is very well multi-threaded (although I'm not sure about the routing daemons), and more CPUs are dirt-cheap. Recently I priced out a mid-grade Cisco and a T3 card, and for less, I could buy a 4-socket Opteron system with 8+ gigabytes of memory.
System memory being a limitation? Well, we're talking about the days when any decent server board will have at least 6 DIMM sockets (how many does your Cisco have?), and at least a 128-bit, DDR-400 memory controller. How about your Cisco? Yes, the VERY high-end ones can compete in that area, but again, you're talking at least an order of magnitude more money.
A plain-old PCI bus, at 100 MB/s real-world throughput (or slightly less, depending on the chipset), would be a limitation, but again, any decent server motherboard has at least one or two PCI-X busses (200-800 MB/s each) and a PCI bus. And recently, they've got PCI-E bandwidth out the wazoo. In fact, in any decent motherboard, the I/O limitation you'd hit first would be from the CPU to the chipset, which would be ~14 gigabits/sec on an Opteron, and anywhere from comparable to much less on an Intel, depending on how much of the FSB was eaten with memory access.
USB would, of course, be a bottleneck - another mechanism would have to be figured out. PCI tends to have a very low limit on the number of available slots, so it's not as ideal for highly-connected routers, but something could be worked out without too much trouble.
Again, like I've said in other ports, the only real problem preventing wider use of Linux-based routers is the interface cards. You just can't beat Cisco in that arena. No matter what strange interface you need, if you pay them money, they'll set you up. The same can NOT be said of PCs!
"Most of the white-box server manufacturers just buy cheap Chinese-designed crap for power supplies, fans, motherboards, and so on, with no real quality control, testing, or engineering involved."
As far back as any of my Ciscos and servers go (almost a decade), I've had *one* power supply failure out of 20-something servers that have been in use, and that was in a box that yes, was a cheap box - with ten of them in a load-balanced pool, we don't need the expensive stuff. But of any of the servers of any consequence - and the Cisco - I have yet to have a power supply fail.
As for fans, luckily, the Ciscos haven't died, either. But if they did, on some, it would take just *one* fan to fail, and the unit would be toast. As a comparison, many of my servers have 6-14 fans, in redundant push-pull pairs. To make it better, if a fan dies, the rest of the fans SPEED UP to compensate.
And you also have to look at the turnaround time for a replacement. If I don't have a spare for each sitting on a shelf, I can drive three blocks and have a temporary replacement for the PC in twenty minutes - but of course, with the cost savings over a Cisco, you could have an entirely spare machine in place anyway, and still be far ahead in terms of money.
"Yeah, well a Linksys router is vastly cheaper than a Cisco, too. The problem is, it's not very dependable."
Yeah. Those PC-based servers aren't very dependable. The 3+ year uptimes on a dozen machines that I had to kill just to move them to a different facility was just an illusion. And all of the Cisco bugs that have bitten people in the butt were an illusion, too.
You're like the guys who tell me that I need to use a t3 connection to transport data from one side of their data room to the other because ethernet "isn't reliable". I ask them point-blank when the last time they had an ethernet failure was, and so far, they haven't been able to give me a single answer. Yes, occasionally, ethernet cards *do* fail. But so do t3 cards.
steve
"Uh, dude, a USB controller would be lucky to push 50 megabits. And that's with horrible latency and lots of lost packets. "
Did you not read when I said that I knew that it wouldn't work? In theory, they have gobs of bandwidth. In reality, they don't. I know that.
"The main problem is the lack of suitable software, and the bottleneck the CPU would create"
Bottleneck? How is a multi-gigahertz CPU with tens of gigabits of I/O to the chipset and gobs of gigaBYTES of memory I/O going to be a bottleneck?
I have an old Cisco 1721 where the CPU is maxed out on a 3-megabit line if you use bandwidth restrictions, just because it can't even keep up with the counters. On the other hand, I have a dual P133 that I picked up for $40, firewalling, translating, and filtering a 6-megabit line at full tilt barely even keeps the CPUs warm.
CPUs are only bottlenecks in Ciscos, because they like the thought of selling you, say, a 3640 instead of a 3620 just because the CPU is a few hundred megahertz faster. There's no competition. In the PC world, CPUs are ridiculously faster than necessary just for bragging rights.
steve
"Don't confuse the purchase price with TCO."
You're right, I forgot. Make that "MASSIVELY" less expensive than a Cisco, since you won't have to pay yearly subscription fees to get OS updates. Either way, you need a person to manage them, so that's a wash.
steve
"The number 1 problem with Xorp is that it supports only a tiny fraction of standard Internet routing protocols. They don't have the developers to support anything more than a bare-bones software router."
Once you've got OSPF, RIP, and BGP, I would think that would cover a pretty good percentage of cases that you're likely to run into. And those are already available for Linux, you don't need anything fancy for them.
steve
There are companies releasing high-end networking products that are nothing more than a PC motherboard and their software. A while back, one of the load-balancer companies (I think it was f5, but I don't recall for sure) contracted with Tyan to build their motherboards, with 4 (or more ) gigE controllers, each on it's own PCI-X bus, and Tyan also sold the board to the public.
The main reason that Cisco doesn't use commodity PC parts in their low- to mid- end routers is that if people knew they were getting nothing more than a $4,000 PC for their $15,000, they'd be pretty pissed. Also, there would be that many more people trying to "crack" IOS to make it run on white-boxes, and that opens up a whole new line of revenue drain for Cisco. (Not that people don't obtain unlicensed copies for their Cisco hardware, though...)
Switches and routing are different things, you can't really compare the two. And again, in their router module, if you implement any sort of ACL, are you still avoiding process-switching?
As for "suspect hardware", are you telling me that Cisco's fans, power supplies, and network modules aren't just as susceptible to failure as a regular server? If so, I've got a bridge to sell you, too. And for less than the cost of a single Cisco, you can have a hot- or cold-spare in place as well.
Sure, there is a high-end where a white-box PC won't handle it. But for any job that a white-box *can* handle, it will be VASTLY cheaper than a Cisco.
... until you use an ACL (or any other useful feature of said Cisco), then you're back to process-routing, in which case, it's going to be orders of magnitude slower than a multi-GHz CPU with mammoth memory bandwidth.
steve
The idea of Ciscos and others using custom hardware to accelerate the routing is, in great measure, over-hyped. Yes, they do have some hardware that GREATLY speeds things up, but in most cases, it only works if you're not using any of the features that make their expensive equipment truly useful. Most of the nice features will kick you from CEF to process-switching, and at that point, a modern PC has *gobs* of CPU cycles, memory bandwidth, and even I/O.
I/O used to be pretty pathetic for PCs, but when you look at motherboard chipsets with up to 32 PCI-E lanes coming off of them, that gives you a theoreticaly 80 GB/s bidirectional transfer. Realistically, the connection from the CPU to the chipset would be a limitting factor, but if you're talking about Opterons, you get 12.8 gigabits/second. You can find Cisco routers that will beat that, but you're talking about more than an order of magnitude in price difference.
steve
The largest impediment is not software, but hardware. The two benefits to a Cisco are that (A) there is someone who *will* fix your problem for a fee, and (B) You can buy an interface card for ANY network type out there.
As for (A), the same will likely become available for this if it isn't already.
(B) is a lot harder. When you get into odd network types and high-speed telco lines, it becomes a bit more difficult - it isn't as easy as just calling your Cisco salesmonkey and buying the card you need.
It should be noted, however, that adding a card to a Cisco isn't always painless. I've had to upgrade the OS - which involved upgrading both memory and flash - just to support another ETHERNET card. How many decades has Ethernet been around for, and they want an OS upgrade to support one? And only to support an additional card, the built-in ethernet worked just fine.
Right now, we're using a Linux router for ethernet routing within our data center, which it handles just fine. As soon as our Sangoma cards show up, it's also going to handle a T3 to our office as well - but only clearchannel, we can't split it between phone and data (as I'd like to do.)
A while back, I had a rather perverse thought. You can hook up a LOT of interfaces to a high-end Cisco, and most routed telecom isn't very high-bandwidth. A T3, at a measly 45 megabit, is still very small considering the throughput of today's hardware. An OC3, at 155 megabits, still isn't much. The perverse thought was that if someone would come up with T1 and T3 modules with integrated CSU/DSUs that connected via USB or firewire, you could stuff a machine chock-full of 4-port controller cards, and be able to hook up 20 or more interfaces very quickly, and easily. In theory, each USB controller card *should* be able to push the ~200 megabits without much trouble, and even a plain old 32/33 PCI bus could *almost* handle the 110 MB/s of all 20 lines at full-tilt. Realistically, however, I do know that USB has many deficiencies which entirely prevent it from fulfilling that task.
If you can't fix it yourself, you call someone who will charge you to fix it for you. Such support is available for nearly all medium-scale open-source projects. Asterisk is a perfect example, Digium saw the opportunity to not only sell the hardware to make it work, but to make money off of software support as well.