Slashdot Mirror
Ask Slashdot: Enterprise-Grade Linux Networking Hardware?
An anonymous reader writes "In spite of Linux's great networking capabilities, there seems to be a shortage of suitable hardware for building an enterprise-grade networking platform. I've had success on smaller projects with the Soekris offerings but they are suboptimal for large-scale deployment due to their single-board non-redundant design (eg., single power supply, lack of backup 'controller'). What is the closest thing to a modular Linux-capable platform with some level of hardware redundancy and substantial bus/backplane throughput?"
Try a Dell server.
Official Linux support - check
Redundant power supplies - check
Remote LAN console - check
Server-class motherboard with loads of bandwidth - check
Rack-mountable - check
1% APY, No fees, Online Bank https://captl1.co/2uIErYq Don't let your $$$ sit in a no-interest acct.
checkpoint, xangati and a bunch of others i've seen use linux. have you even looked?
http://www.vyatta.com/solutions/physical/appliances
I've used Supermicro equipment for years. Their 1U Atom based systems work great for firewalls, routers, or any other kind of Linux network device. Low power, mostly fanless (power supply has a fan), expansion slots, decently priced. You can go up the line to full blown Xeon based systems with all the redundancy you need.
Their support is good also. You get to talk to knowledgeable people who speak English.
Supermicro website
I have a friend who operates a small ISP in rural Iowa. I believe he's using ImageStream routers. Just a quick look at their lineup and I'm guessing that they can cover small to mid size businesses. They claim to be able to replace Cisco 3945 and 7206 routers. I'm not sure about hardware redundancy though.
"A plan fiendishly clever in its intricacies"- Homer Simpson
ALIX boards can run Linux or FreeBSD (Monowall, pfSense) and support PoE, so you can set up your own redundant power system. For board redundancy, just use two routers.
Actually, the Soekris boards seem to be similar - they both use x86 CPUs.
Pretty most software devices I've seen have either been a rebadged Dell or Supermicro, with the top end running custom cases, and the low end doing whitebox.
In terms of "real" networking kit though, there is a bunch of switches that run linux:
Arista (everything)
Extreme (everything running XOS, which is all current models)
Cisco (everything running IOS XE, the only switch being the 4500-X)
All Juniper devices that run JunOS are FreeBSD, this includes both the EX and QFX switch lines, as well as their SRX firewalls.
Also most of the openflow-aimed switches run Linux, eg http://www.pica8.com/
/* FUCK - The F-word is here so that you can grep for it */
I run a 50/50 mix of Dell and HP Proliant servers. About 30 of each brand. All of these are fairly new, within a few years of age.
By far, the Dells do break down more often. The HPs seem to only lose hot-pluggable hard drives every now and then, but the Dells lose drives, PSUs, cooling fans, RAID controllers and even had a motherboard fail. However, the latest batch of Proliants I bought seem to not be built as good as in the past either. We'll see how well they hold up. It's all Foxconn junk nowadays. The new servers do perform very fast however, you do have to give them credit there.
There's plenty of options, but relatively few that an individual might be able to purchase for a pet project or for a small number of prospective clients.
Off the top of my head, Dell offer an OEM scheme whereby they'll rebrand one of their servers with your logo and install your software on it before shipping it out to your customer; another company called NEI will do something similar. I've actually got an NEI box right next to me now - I'm the customer of a company that uses them.
Their Proliant line is still pretty good and supports *nix.
We've had good results with boxes from Penguin Computing. We get boxes with redundant power supplies, redundant NICs, and RAID. We've spent a lot of time qualifying these boxes before deploying them to our customers and currently have a lot of them in the field.
...but I use Sun Microsystems hardware for this task.
The X2100, X4100 series servers more than meet my needs, and are available on the used market for a song these days.
The lights-out management works great, the rackmount kits and cable management arms are first-class, the hardware is well-made, and they look cool. Heck, they're even certified to run RHEL 5 or so.
Best of all - buying used Sun gear and putting Linux on it pisses off Larry Ellison. What more could you ask for?
Do daemons dream of electric sleep()?
So anytime someone posts an Ask Slashdot question, it means that they're idiots? Nice to know. The OP would have done well to have been clearer as to whether he was talking about servers or networking gear. But aside from that, if the OP is looking at sinking a whole ton of cash, which he'd have to in order to get all the things he listed, such as redundant power supplies, backup controllers and so on, he is doing the right thing by asking around people who have made high value purchases to find out which ones work w/ Linux, and which don't.
So if you had this job, all you'd do is visit the websites of the vendors in question and look @ their online product catalogs, where they are bound to list their shortcomings as well as their strengths? Good idea!
Ok first thing first, I work for ImageStream as the Technical Support manager. So I might have a slightly biased viewpoint when it comes to the place I have been working for the last 16 years... But we have been doing Linux Based networking for the last 14 years.
What the OP wants to do is rather difficult for a few reasons. First, after shipping thousands of Linux based routers I can tell you that redundant power supplies that fit into standard PC hardware have a much higher failure rate than a standard Power Supply. Granted, if you have a failure you still have a functional power supply(which is now working twice as hard and is even more likely to fail).
Second, standard PC hardware just doesn't support multiple redundant components. Sure you can get redundant power supplies, but redundant buses or Cpu's your talking different about a totally different class of hardware(see below).
Third, If you truly have an Enterprise application, and your asking about hardware to support your application you are already in over your head. Sorry it's just the truth. The OP is talking about building a custom solution for a mission critical application and they have to ask on slashdot about hardware solutions. What happens when(not if) the OP has a problem. The real reason that many people buy our(ImageStream's) hardware is for the support. If something doesn't work they don't have try and troubleshoot a strange Pci bus condition or an obscure Linux Kernel issue that you only see when you have +5,000 networking interfaces in a system. It's one thing if your a Google and you want to build something that just doesn't exist like the OpenFlow switches they are using in their Gscale network. But for a normal organization you are going to spend money and time to develop your custom solution and in the end if anything doesn't work, you will spend more time fixing it.
Now if the OP still wants to do this... I would look at an ATCA (AdvancedTCA ) chassis. You can get support for a redundant dual loop back plane, multiple CPU cards, redundant power supplies and in most cases a out of band management module for the chassis. But this is VERY costly hardware. If your not budgeting at least $20k in hardware your likely not going to end-up with anything that had real redundancy.
Uh, IOS is Linux based. And it's not iOS - that's Apple's iToy OS. IOS is Cisco's OS for managing their routers.
Oh, and while on this topic, make sure it supports IPv6.
Current state-of-the-art in off-the-shelf ATCA gear (chassis, switch cards, compute cards, etc.) provides redundant 40-gigabit backplane connectivity on the data fabric. It's available with linux support.
It's telco-grade stuff, so redundant power supplies, redundant fans, redundant networking, redundant shelf management, etc.
You're going to pay for it though.
Arista Networks. 10GbE, insanely low latency, insanely low per port cost and last I checked was running a Fedora kernel and userland.
You don't define precisely what you mean by these things.
* If we're talking about switches, forget it. Cisco does it better and faster, easier to manage, with more robust hardware and a better service plan (limited lifetime warranty on all fixed configuration switches!). A non-Cisco switch doing anything of value on your network is a surefire way to convince me that you are bush league.
* Routers - it really depends. What are you going to do? Just route traffic between LAN interfaces? A Cisco L3-capable switch will probably be the fastest for this job, considering that many of its traffic routing tasks can be done in hardware which has been made to spec. But if you're looking to stick with Linux, you can configure a Linux server with the hardware you require and load it up with a network protocol you need it to run. A Linux server can certainly run OSPF or BGP. However, what else are you need? Do you also need a firewall, a VPN concentrator, an intrusion detector, a WAN optimizer, a small phone system? Because if you need those things as well, a hardware router will do these things at once in addition to its routing tasks, with a better performance:price ratio. Configuring it is not hard to learn to do. If you don't have time, you can always phone someone else who's contractually obligated to fix it.
* Firewall - this is wide open. Every single piece of firewall software seems to approach things in a totally different way, especially in terms of management interfaces. I would look around for the one that communicates to you in the way you find most intuitive, and then buy the gear that runs that. While I know Linux on a server will have some powerful firewalling capabilities, I simply can't use most of the Linux-based management packages because they just don't seem to think the way I do. Hopefully this is remedied soon, because most firewall vendors are incredibly overpriced and, in the case of Cisco especially, occasionally hard to even obtain at all.
I'm no Cisco fanboy, although I do rely on them for my income (full disclosure). I also don't want to be a Negative Nancy, as I understand that not everyone warms up to the whole "you should be grateful to have our logo in your rack" attitude you get from Cisco...I certainly don't. But there is a reason beyond simple groupthink that causes people to buy their stuff - frankly, there just is no serious alternative when it comes to switches or multi-function routers.
---don't make me break out my red pen.
Thanks for the info. Never heard of them, but nice to know.
Just checked out wiki - stand corrected
Here is something different to all the other experts.
It is absolutely useless to have redundant hardware, eg: Dual PSUs, Dual CPUs, Dual Motherboards, etc. on the same computer. You will never be able to 100% protect against a hardware failure as they will invariably share hardware to allow the interconnection between the redundant components to occur, it is unlikely to protect from things like a short circuit/power surge which would take out everything until the UPS. Then if a component does fail, to repair it your are going to have to take it offline to restore that redundancy anyway.
You are far better off getting two (or more) completely separate servers, geographically diverse if possible, which uses software to provide redundancy. If one goes down, the other(s) would be powerful enough to handle all the load, and when everything is rosey, it just load balances.
The real world difference is you are looking at a $5000 server with identical specs as a $20,000 but without all the redundant PSUs, etc. but you would be better off buying two $5,000 servers ($10,000 total), set them up to have redundancy of each other (So you truely have two COMPLETELY separate sets in redundant hardware of all components, and geographically separate too if possible), and as a bonus you have twice as much computing power (or scale down power draw when not needed) for when both servers are working. If you need to pull one down for maintenance, you don't need to shut off the whole thing.
If you are into Dual PSUs, etc. equipment in addition to also load balancing/fallover between other servers which also have redundancy, this is pointless because you should have ability to cope with the complete failure of a "redundant" server anyway, for the time it takes to replace the defective part the window that the other server(s) will have a failure in that time is not very high.
The only exception to this is Hard Drives, Hard Drives make sense for redundancy, not just because of their high rate of failure, but the fact that if there is a failure, it is a lot more work to recover from (Whereas other components are just a straight hardware swap) so it is saving extra work in the long run.
For a smaller environment where a small amount of downtime would be acceptable, You can even have a Cold Server, an exactly clone of the Main Server ready to go with all the software setup but powered off until needed if there is fault with the main server, the Cold Server can then be powered on to take over. There is no redundancy or fall over with this, but then again, in a smaller environment, your app might not support any kind of redundancy. With a Cold Server, just turn off the faulty server, switch on the cold server, restore the latest data set, and off you go. Microsoft doesn't require that Cold Servers hold a separate license either.
Stand somewhat uncorrected!
http://en.wikipedia.org/wiki/NX-OS
However, the company used FreeBSD, not Linux.
Still, it was one of the best routers the company I worked for (100.000+ employees hi-tech company) ever installed. We had mostly Cisco gear, but the FreeBSD-based routers (they used some special motherboards) were a pleasure to admin and came with some service-level routing capabilities as added bonus. Performance was stellar for the time.
"The agriculture ministry is not in charge of Gundam" - Japanese ministry official.
+1 for Arista. They are doing some pretty amazing stuff. Check out http://techfieldday.com/2012/arista-presents-networking-field-day-3/ for videos of a recent tech talk.
Never trust anyone who takes pride in being called a 'geek'....
These guys make the hardware that $VENDOR rebrands and sells as an appliance.
http://www.nexcom.com/Products/network-and-communication-solutions/mainstream-appliance
If they want networking hardware, linux *ISN'T* the way to go.
That depends on what you want. The most useful part of having Linux on my router is that I can make it do what I want it to do. QOS, firewalling, those just scratch the surface. Someone who knows Linux networking well, or is willing to put a little work into it, can make a router that does virtually anything.
Go with the cheap router and buy TWO or more.
Deploy using VRRP or other active/standby or active/active configuration.
This probably isn't the right path for the OP, but throwing this out since this is an option that might be suitable for some readers.
More "industrial grade" than "enterprise grade", but if you need a flexible high-slot-count solution you may want to look into PICMG 1.3 (System Host Board) based hardware. Instead of a motherboard with PCIe expansion slots, there is a passive backplane consisiting for a system slot and some number of PCI and/or PCIe slots (anywhere from 1 to 20 depending on the particular backplane). The system slot takes a "Single Board Computer" that performs all of the "motherboard" functions with options ranging from atom to dual xeon processors to suit most processing needs. Since the hardware is really nothing more than standard PC components on a card instead of a motherboard, just about any PC OS is supported.
If you go with a 3U or 4U chassis, you can easily find redundant power supply options and service is also easier (you can swap out the processor card as easily as any other card). The only "difficult" maintenance is a backplane failure, but even that is normally a much simpler process than a conventional motherboard layout. There is no bus-level redundancy (though with a "split backplane" you can actually have 2 indepenent units in a single chassis... but you are better off with 2 separate chassis anyway). You can easily put together a "spares kit" of processor cards, backplanes, and network cards.
These systems are mainly used in industrial settings, so they tend to more rugged than typical systems. Due to the level of customization, you would also end up spending quite a bit of time selecting a configuration and doing testing. Depending on the number of systems you anticipate needing, this might be more effort than you'd want to spend.
Many vendors to choose from, if you are interested in looking into the option here are a few starting points:
http://www.trentontechnology.com/
(look under Products: Board Products). They produce high-performance processor cards (single and dual socket "Core" and Xeon).
http://www.onestopsystems.com/
turnkey systems and some interesting PCIe bus extension products if you want to share a rack of cards
http://www.cyberresearch.com/
a wide array of cards, backplane, chassis options including "lower-power" cards (celeron/atom) as well as higher-end.
My X2100s have been running continuously since 2008, so I haven't even /tried/ to upgrade them.
Never saw a need to upgrade the X2100 M2s I have.
Besides the BMC software, are there any firmware upgrades that are necessary?
(Note - BMC software upgrade is not necessary, but it is nice)
Of course, I don't have a support contract either. I couldn't hang up the phone fast enough when I was quoted over $25,000 to bring an Ultra 5 "back into compliance" so that I could get a firmware upgrade I *did* need. I threw it out instead.
I worry less about x86 hardware in this regard. It's not like the PC industry is particularly good about this crap in the first place...
Do daemons dream of electric sleep()?
Hover over the Products tab. You get choices for the various product line numbers. But this is obscurity for the public market. The marketing director might know exactly what all those numbers mean. But those who are new to this company will not. That's not to say they must not list their products by number somewhere. But I am saying they need to list their products by what functions they do and what problems they solve, so that new customers can go right to the correct pages. Potential customers won't be, if they have to step navigate sequentially by going in and out of different pages. They be better off scrolling than doing that.
now we need to go OSS in diesel cars
Looking for 16 gigabit ports (a pair of 10-gigabit ports for upstream would be a big plus) in a 1U rack mount form factor. Just hardware that Linux supports with open source drivers.
now we need to go OSS in diesel cars
Looks like you can't buy one or two devices from them, unless someone can point me at a location.
Cheap storage VM.
Actually IOS XE is OpenBSD based, as is JunOS and Brocade's FOS is Linux based. However almost all of the heavy lifting is done by the backplane and the ASICs, that's why you can perform a non-disruptive software update on all of those platforms, the OS is just there for management. This is why one of the easiest ways to cause a production issue in these environments is to enable a feature that requires CPU processing of a significant amount of traffic, they processors just aren't capable,
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
Dont try and beat companies for switching with linux grade equipment - there just isnt a good reason to. I love junos, screenos and ios - they kick arse... I also like what huawei do (they are a little cheaper, but at the switching side, they're very good). I've been doing networking for 15 years as a job and i've been doing linux since '92.
However, im also very VERY keen on linux at the routing side... I've even written my own firewall/routing software for linux. At the layer 3, linux has one advantage cisco, juniper (screenos and junos), and basically everyone else cannot give you - adaptability. just about any 1ru server capable of supporting either 8 1gbps nics (2x4 pci-e) or 2-4 10gbps nics (either 1x2 or 2x2 pci-e) is fantastic. Modern cpu's and busses really dont change much between vendors, only generation so you shouldn't really be bothered looking for "which has the best bus" cause they all do (dell, ibm, hp, it doesnt matter). If you can get a server with a serial lom (not just a network-connected web-gui based piece of nastyness (because you DO want oob management) you'll be laughing. Generally speaking, most x86 hardware will have around the same life expectancy as dedicated hardware and by that i mean if you get a dell server with redundant power supplies and so forth, it'll have about the same uptime as a juniper srx650 with dual power supplies. The one thing you'll probably miss out on is hot-swap-ability.
Now you plug that machine into your switch, etherchannel and vlan trunk it to your server and you have an amazing device. What you do with it then is entirely up to you, and this is generally the harder decision then the hardware - what you'll put on it. You can go with a real bit of firewall gui (such as vyatta) or you can do something far more interesting - i recommend devil linux personally as its the most flexible of the lot without being a bitch to maintain (as in, centos, ubuntu, fedora, whatever - not good choices for networking equipment cause there is alot of config to manage at the machine side - very bad for networking). One reason i say i dont like most firewall distro's in linux is that they tend to limit you and if your going to do this, go get a juniper netscreen/srx, they're just not that expensive (there is one exception to this, and thats openwrt, it runs on x86 and has almost every component a normal linux distro has). Its also worth avoiding harddrives (except if your going to put a network cache in there) and there are good options out there for doing just that.
Linux's most valuable asset is its abilities to do unbelievably fantastic things at the network layer and then be adapted easily. With vendor enterprise kit you'll get ipv4, ipv6, routing protocols (isis, ospf, bgp, rip and add eigrp for cisco) policy based routing, some network serivces (dhcp, ra's, etc) add firewall/loadbalancing/vpn depending on the device. With linux you get all this and a hell of a lot more in one device, it is well worth your time checking out the younger and more intresting routing protocols (like babel, oslr, etc etc, theres a few) - the fun is bringing it all together.
There is one downside to all this, too many options and alot to learn. Do you want a network device that will do:
1) policy based routing
2) ipv4 and 6 firewalling
3) load balancing
4) routing protocols
5) vpn'ing
1+2 come from the same place, so you'll be quite ok with that, the rest though is up to you, each has 15 different options from 15 different ppl and it takes some experimenting to know which is best for you. You'll also find none of them will configure or look anything like one another so you will be learning 4 very distinctly different software stacks with 4 very distinctly different configuration paradigms.
Personally, i dont see that as an issue for myself - in an organisation it can be a bit harder.
I just prefer to have some pricing information before I start talking to a company, but maybe I will give them a call.
Cheap storage VM.
You're seriously using a consumer-level desktop chassis for enterprise routing? You're not doing enterprise *anything*. See the title of this post. If you showed up with anything except a 1U rackmount machine, I'd show you the door.
Please help metamoderate.
Layer 2 is bridging. Layer 3 is routing. Switching used to be doing bridging fast and cheaply using specialized hardware, but if they want to throw in routing features in the same box, that's still fine. And usually the routing in a Layer 3 switch is dumber than the routing in a router, though that's usually deliberate marketing (leaving out BGP so you still get to buy a Real Router.)
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks