Slashdot Mirror
Ask Slashdot: Enterprise-Grade Linux Networking Hardware?
An anonymous reader writes "In spite of Linux's great networking capabilities, there seems to be a shortage of suitable hardware for building an enterprise-grade networking platform. I've had success on smaller projects with the Soekris offerings but they are suboptimal for large-scale deployment due to their single-board non-redundant design (eg., single power supply, lack of backup 'controller'). What is the closest thing to a modular Linux-capable platform with some level of hardware redundancy and substantial bus/backplane throughput?"
Try a Dell server.
Official Linux support - check
Redundant power supplies - check
Remote LAN console - check
Server-class motherboard with loads of bandwidth - check
Rack-mountable - check
1% APY, No fees, Online Bank https://captl1.co/2uIErYq Don't let your $$$ sit in a no-interest acct.
Xeon/Opteron based platform. you want enterprise, you order enterprise--something with ECC, and you could probably use memory mirroring on such an networking platform. Then, get the latest Intel NIC's whic have ECC throughout the NIC's components (not sure if new feature, or only newly advertised).
then, choose a major vendor. IBM/HP/Dell/Supermicro
checkpoint, xangati and a bunch of others i've seen use linux. have you even looked?
http://www.vyatta.com/solutions/physical/appliances
I've used Supermicro equipment for years. Their 1U Atom based systems work great for firewalls, routers, or any other kind of Linux network device. Low power, mostly fanless (power supply has a fan), expansion slots, decently priced. You can go up the line to full blown Xeon based systems with all the redundancy you need.
Their support is good also. You get to talk to knowledgeable people who speak English.
Supermicro website
If you're looking for very high throughput switches and network equipement running Linux, I heard about Arista a while back which might be what you want:
http://en.wikipedia.org/wiki/Arista_Networks
http://www.aristanetworks.com/
Their switches essentially run a standard Linux userspace with extra daemons controlling the switching hardware.
I have a friend who operates a small ISP in rural Iowa. I believe he's using ImageStream routers. Just a quick look at their lineup and I'm guessing that they can cover small to mid size businesses. They claim to be able to replace Cisco 3945 and 7206 routers. I'm not sure about hardware redundancy though.
"A plan fiendishly clever in its intricacies"- Homer Simpson
You just need the hardware with the highest throughput and fastest switching. That is going to be something using FPGAs to help intelligently route gobs of traffic and using an iOS interface. What use would Linux be?
...a Beowulf cluster ?!?
ALIX boards can run Linux or FreeBSD (Monowall, pfSense) and support PoE, so you can set up your own redundant power system. For board redundancy, just use two routers.
Actually, the Soekris boards seem to be similar - they both use x86 CPUs.
Pretty most software devices I've seen have either been a rebadged Dell or Supermicro, with the top end running custom cases, and the low end doing whitebox.
In terms of "real" networking kit though, there is a bunch of switches that run linux:
Arista (everything)
Extreme (everything running XOS, which is all current models)
Cisco (everything running IOS XE, the only switch being the 4500-X)
All Juniper devices that run JunOS are FreeBSD, this includes both the EX and QFX switch lines, as well as their SRX firewalls.
Also most of the openflow-aimed switches run Linux, eg http://www.pica8.com/
/* FUCK - The F-word is here so that you can grep for it */
I run a 50/50 mix of Dell and HP Proliant servers. About 30 of each brand. All of these are fairly new, within a few years of age.
By far, the Dells do break down more often. The HPs seem to only lose hot-pluggable hard drives every now and then, but the Dells lose drives, PSUs, cooling fans, RAID controllers and even had a motherboard fail. However, the latest batch of Proliants I bought seem to not be built as good as in the past either. We'll see how well they hold up. It's all Foxconn junk nowadays. The new servers do perform very fast however, you do have to give them credit there.
There's plenty of options, but relatively few that an individual might be able to purchase for a pet project or for a small number of prospective clients.
Off the top of my head, Dell offer an OEM scheme whereby they'll rebrand one of their servers with your logo and install your software on it before shipping it out to your customer; another company called NEI will do something similar. I've actually got an NEI box right next to me now - I'm the customer of a company that uses them.
Their Proliant line is still pretty good and supports *nix.
We've had good results with boxes from Penguin Computing. We get boxes with redundant power supplies, redundant NICs, and RAID. We've spent a lot of time qualifying these boxes before deploying them to our customers and currently have a lot of them in the field.
...but I use Sun Microsystems hardware for this task.
The X2100, X4100 series servers more than meet my needs, and are available on the used market for a song these days.
The lights-out management works great, the rackmount kits and cable management arms are first-class, the hardware is well-made, and they look cool. Heck, they're even certified to run RHEL 5 or so.
Best of all - buying used Sun gear and putting Linux on it pisses off Larry Ellison. What more could you ask for?
Do daemons dream of electric sleep()?
HP, Lenovo, Dell... pick any major vendor. This is their bread and butter. Or is the issue that you don't really want to pay for enterprise-grade?
You might want to give Lanner x86 hardware a look:
http://www.lannerinc.com/x86_Network_Appliances
Then have a look at some of the hardware that WatchGuard supply. Look similar?
(I have no association with Lanner other than having considered their hardware for a similar project)
You might want to take a look at ImageStream's (http://www.imagestream.com/) line of Linux based routers.
Companies like Arista Networks make switches that run linux and that expose that linux to the end-user.
So anytime someone posts an Ask Slashdot question, it means that they're idiots? Nice to know. The OP would have done well to have been clearer as to whether he was talking about servers or networking gear. But aside from that, if the OP is looking at sinking a whole ton of cash, which he'd have to in order to get all the things he listed, such as redundant power supplies, backup controllers and so on, he is doing the right thing by asking around people who have made high value purchases to find out which ones work w/ Linux, and which don't.
So if you had this job, all you'd do is visit the websites of the vendors in question and look @ their online product catalogs, where they are bound to list their shortcomings as well as their strengths? Good idea!
Ok first thing first, I work for ImageStream as the Technical Support manager. So I might have a slightly biased viewpoint when it comes to the place I have been working for the last 16 years... But we have been doing Linux Based networking for the last 14 years.
What the OP wants to do is rather difficult for a few reasons. First, after shipping thousands of Linux based routers I can tell you that redundant power supplies that fit into standard PC hardware have a much higher failure rate than a standard Power Supply. Granted, if you have a failure you still have a functional power supply(which is now working twice as hard and is even more likely to fail).
Second, standard PC hardware just doesn't support multiple redundant components. Sure you can get redundant power supplies, but redundant buses or Cpu's your talking different about a totally different class of hardware(see below).
Third, If you truly have an Enterprise application, and your asking about hardware to support your application you are already in over your head. Sorry it's just the truth. The OP is talking about building a custom solution for a mission critical application and they have to ask on slashdot about hardware solutions. What happens when(not if) the OP has a problem. The real reason that many people buy our(ImageStream's) hardware is for the support. If something doesn't work they don't have try and troubleshoot a strange Pci bus condition or an obscure Linux Kernel issue that you only see when you have +5,000 networking interfaces in a system. It's one thing if your a Google and you want to build something that just doesn't exist like the OpenFlow switches they are using in their Gscale network. But for a normal organization you are going to spend money and time to develop your custom solution and in the end if anything doesn't work, you will spend more time fixing it.
Now if the OP still wants to do this... I would look at an ATCA (AdvancedTCA ) chassis. You can get support for a redundant dual loop back plane, multiple CPU cards, redundant power supplies and in most cases a out of band management module for the chassis. But this is VERY costly hardware. If your not budgeting at least $20k in hardware your likely not going to end-up with anything that had real redundancy.
Current state-of-the-art in off-the-shelf ATCA gear (chassis, switch cards, compute cards, etc.) provides redundant 40-gigabit backplane connectivity on the data fabric. It's available with linux support.
It's telco-grade stuff, so redundant power supplies, redundant fans, redundant networking, redundant shelf management, etc.
You're going to pay for it though.
Arista Networks. 10GbE, insanely low latency, insanely low per port cost and last I checked was running a Fedora kernel and userland.
You don't define precisely what you mean by these things.
* If we're talking about switches, forget it. Cisco does it better and faster, easier to manage, with more robust hardware and a better service plan (limited lifetime warranty on all fixed configuration switches!). A non-Cisco switch doing anything of value on your network is a surefire way to convince me that you are bush league.
* Routers - it really depends. What are you going to do? Just route traffic between LAN interfaces? A Cisco L3-capable switch will probably be the fastest for this job, considering that many of its traffic routing tasks can be done in hardware which has been made to spec. But if you're looking to stick with Linux, you can configure a Linux server with the hardware you require and load it up with a network protocol you need it to run. A Linux server can certainly run OSPF or BGP. However, what else are you need? Do you also need a firewall, a VPN concentrator, an intrusion detector, a WAN optimizer, a small phone system? Because if you need those things as well, a hardware router will do these things at once in addition to its routing tasks, with a better performance:price ratio. Configuring it is not hard to learn to do. If you don't have time, you can always phone someone else who's contractually obligated to fix it.
* Firewall - this is wide open. Every single piece of firewall software seems to approach things in a totally different way, especially in terms of management interfaces. I would look around for the one that communicates to you in the way you find most intuitive, and then buy the gear that runs that. While I know Linux on a server will have some powerful firewalling capabilities, I simply can't use most of the Linux-based management packages because they just don't seem to think the way I do. Hopefully this is remedied soon, because most firewall vendors are incredibly overpriced and, in the case of Cisco especially, occasionally hard to even obtain at all.
I'm no Cisco fanboy, although I do rely on them for my income (full disclosure). I also don't want to be a Negative Nancy, as I understand that not everyone warms up to the whole "you should be grateful to have our logo in your rack" attitude you get from Cisco...I certainly don't. But there is a reason beyond simple groupthink that causes people to buy their stuff - frankly, there just is no serious alternative when it comes to switches or multi-function routers.
---don't make me break out my red pen.
Last time I tried to do update firmware on an x2100 I needed an Oracle Support account.
I don't have, and won't have one of these, so my machine is stuck in 2008.
Look at the blade options from these guys.
Backplane - check
Integrated switching at the backplane - check or optional
Redundant power and fabrics - check
Modular design - check
Supermicro is more flexible in what they can turn out - this for example: http://www.supermicro.com/products/system/2U/2015/SYS-2015TA-HTRF.cfm can house four of these: http://www.supermicro.com/products/motherboard/ATOM/ICH9/X7SPT-DF-D525.cfm
Here is something different to all the other experts.
It is absolutely useless to have redundant hardware, eg: Dual PSUs, Dual CPUs, Dual Motherboards, etc. on the same computer. You will never be able to 100% protect against a hardware failure as they will invariably share hardware to allow the interconnection between the redundant components to occur, it is unlikely to protect from things like a short circuit/power surge which would take out everything until the UPS. Then if a component does fail, to repair it your are going to have to take it offline to restore that redundancy anyway.
You are far better off getting two (or more) completely separate servers, geographically diverse if possible, which uses software to provide redundancy. If one goes down, the other(s) would be powerful enough to handle all the load, and when everything is rosey, it just load balances.
The real world difference is you are looking at a $5000 server with identical specs as a $20,000 but without all the redundant PSUs, etc. but you would be better off buying two $5,000 servers ($10,000 total), set them up to have redundancy of each other (So you truely have two COMPLETELY separate sets in redundant hardware of all components, and geographically separate too if possible), and as a bonus you have twice as much computing power (or scale down power draw when not needed) for when both servers are working. If you need to pull one down for maintenance, you don't need to shut off the whole thing.
If you are into Dual PSUs, etc. equipment in addition to also load balancing/fallover between other servers which also have redundancy, this is pointless because you should have ability to cope with the complete failure of a "redundant" server anyway, for the time it takes to replace the defective part the window that the other server(s) will have a failure in that time is not very high.
The only exception to this is Hard Drives, Hard Drives make sense for redundancy, not just because of their high rate of failure, but the fact that if there is a failure, it is a lot more work to recover from (Whereas other components are just a straight hardware swap) so it is saving extra work in the long run.
For a smaller environment where a small amount of downtime would be acceptable, You can even have a Cold Server, an exactly clone of the Main Server ready to go with all the software setup but powered off until needed if there is fault with the main server, the Cold Server can then be powered on to take over. There is no redundancy or fall over with this, but then again, in a smaller environment, your app might not support any kind of redundancy. With a Cold Server, just turn off the faulty server, switch on the cold server, restore the latest data set, and off you go. Microsoft doesn't require that Cold Servers hold a separate license either.
If you search for x86 network appliances, you'll get a lot of results.
A few off the top of my head.
Axiomtek
Portwell
Lanner
Advantech
You can also go to linuxfordevices.com and search network appliances.
Most of the devices are comparable. I have an Axiomtek and Lanner device right now. I am more impressed with the build quality of Axiomtek, but it was pain to get. Lanner is easy to deal with.
However, the company used FreeBSD, not Linux.
Still, it was one of the best routers the company I worked for (100.000+ employees hi-tech company) ever installed. We had mostly Cisco gear, but the FreeBSD-based routers (they used some special motherboards) were a pleasure to admin and came with some service-level routing capabilities as added bonus. Performance was stellar for the time.
"The agriculture ministry is not in charge of Gundam" - Japanese ministry official.
These guys make the hardware that $VENDOR rebrands and sells as an appliance.
http://www.nexcom.com/Products/network-and-communication-solutions/mainstream-appliance
If they want networking hardware, linux *ISN'T* the way to go.
That depends on what you want. The most useful part of having Linux on my router is that I can make it do what I want it to do. QOS, firewalling, those just scratch the surface. Someone who knows Linux networking well, or is willing to put a little work into it, can make a router that does virtually anything.
Why Linux? is it just because you like open source, and those crap closed source vendors are just plain silly? Or do you want to do something that can be done specifically in Linux? For example, we're looking to deploy configuration management (make sure feature X is turned on for every switch) on our switches and we found some that have an embedded linux controller (but not actually passes packets around) which will make this easy for us.
What is the definition of enterprise grade? If redundancy and throughput are the definition, I don't think that is "enterprise grade". All switches should provide good throughput (though some are better). Enterprise grade to me are things like increased port density to support large datacenter, and easy management with existing tools. Things such as monitoring and configuration for example should have APIs and integration with standard tools, for example.
Go with the cheap router and buy TWO or more.
Deploy using VRRP or other active/standby or active/active configuration.
This probably isn't the right path for the OP, but throwing this out since this is an option that might be suitable for some readers.
More "industrial grade" than "enterprise grade", but if you need a flexible high-slot-count solution you may want to look into PICMG 1.3 (System Host Board) based hardware. Instead of a motherboard with PCIe expansion slots, there is a passive backplane consisiting for a system slot and some number of PCI and/or PCIe slots (anywhere from 1 to 20 depending on the particular backplane). The system slot takes a "Single Board Computer" that performs all of the "motherboard" functions with options ranging from atom to dual xeon processors to suit most processing needs. Since the hardware is really nothing more than standard PC components on a card instead of a motherboard, just about any PC OS is supported.
If you go with a 3U or 4U chassis, you can easily find redundant power supply options and service is also easier (you can swap out the processor card as easily as any other card). The only "difficult" maintenance is a backplane failure, but even that is normally a much simpler process than a conventional motherboard layout. There is no bus-level redundancy (though with a "split backplane" you can actually have 2 indepenent units in a single chassis... but you are better off with 2 separate chassis anyway). You can easily put together a "spares kit" of processor cards, backplanes, and network cards.
These systems are mainly used in industrial settings, so they tend to more rugged than typical systems. Due to the level of customization, you would also end up spending quite a bit of time selecting a configuration and doing testing. Depending on the number of systems you anticipate needing, this might be more effort than you'd want to spend.
Many vendors to choose from, if you are interested in looking into the option here are a few starting points:
http://www.trentontechnology.com/
(look under Products: Board Products). They produce high-performance processor cards (single and dual socket "Core" and Xeon).
http://www.onestopsystems.com/
turnkey systems and some interesting PCIe bus extension products if you want to share a rack of cards
http://www.cyberresearch.com/
a wide array of cards, backplane, chassis options including "lower-power" cards (celeron/atom) as well as higher-end.
Hover over the Products tab. You get choices for the various product line numbers. But this is obscurity for the public market. The marketing director might know exactly what all those numbers mean. But those who are new to this company will not. That's not to say they must not list their products by number somewhere. But I am saying they need to list their products by what functions they do and what problems they solve, so that new customers can go right to the correct pages. Potential customers won't be, if they have to step navigate sequentially by going in and out of different pages. They be better off scrolling than doing that.
now we need to go OSS in diesel cars
You forgot to include the name of the company/product.
Dont try and beat companies for switching with linux grade equipment - there just isnt a good reason to. I love junos, screenos and ios - they kick arse... I also like what huawei do (they are a little cheaper, but at the switching side, they're very good). I've been doing networking for 15 years as a job and i've been doing linux since '92.
However, im also very VERY keen on linux at the routing side... I've even written my own firewall/routing software for linux. At the layer 3, linux has one advantage cisco, juniper (screenos and junos), and basically everyone else cannot give you - adaptability. just about any 1ru server capable of supporting either 8 1gbps nics (2x4 pci-e) or 2-4 10gbps nics (either 1x2 or 2x2 pci-e) is fantastic. Modern cpu's and busses really dont change much between vendors, only generation so you shouldn't really be bothered looking for "which has the best bus" cause they all do (dell, ibm, hp, it doesnt matter). If you can get a server with a serial lom (not just a network-connected web-gui based piece of nastyness (because you DO want oob management) you'll be laughing. Generally speaking, most x86 hardware will have around the same life expectancy as dedicated hardware and by that i mean if you get a dell server with redundant power supplies and so forth, it'll have about the same uptime as a juniper srx650 with dual power supplies. The one thing you'll probably miss out on is hot-swap-ability.
Now you plug that machine into your switch, etherchannel and vlan trunk it to your server and you have an amazing device. What you do with it then is entirely up to you, and this is generally the harder decision then the hardware - what you'll put on it. You can go with a real bit of firewall gui (such as vyatta) or you can do something far more interesting - i recommend devil linux personally as its the most flexible of the lot without being a bitch to maintain (as in, centos, ubuntu, fedora, whatever - not good choices for networking equipment cause there is alot of config to manage at the machine side - very bad for networking). One reason i say i dont like most firewall distro's in linux is that they tend to limit you and if your going to do this, go get a juniper netscreen/srx, they're just not that expensive (there is one exception to this, and thats openwrt, it runs on x86 and has almost every component a normal linux distro has). Its also worth avoiding harddrives (except if your going to put a network cache in there) and there are good options out there for doing just that.
Linux's most valuable asset is its abilities to do unbelievably fantastic things at the network layer and then be adapted easily. With vendor enterprise kit you'll get ipv4, ipv6, routing protocols (isis, ospf, bgp, rip and add eigrp for cisco) policy based routing, some network serivces (dhcp, ra's, etc) add firewall/loadbalancing/vpn depending on the device. With linux you get all this and a hell of a lot more in one device, it is well worth your time checking out the younger and more intresting routing protocols (like babel, oslr, etc etc, theres a few) - the fun is bringing it all together.
There is one downside to all this, too many options and alot to learn. Do you want a network device that will do:
1) policy based routing
2) ipv4 and 6 firewalling
3) load balancing
4) routing protocols
5) vpn'ing
1+2 come from the same place, so you'll be quite ok with that, the rest though is up to you, each has 15 different options from 15 different ppl and it takes some experimenting to know which is best for you. You'll also find none of them will configure or look anything like one another so you will be learning 4 very distinctly different software stacks with 4 very distinctly different configuration paradigms.
Personally, i dont see that as an issue for myself - in an organisation it can be a bit harder.
These guys seem to be working on something like that. They are more known for their insanely fast disk and flash storage systems and clusters used by scientific computing and financial users.
You're seriously using a consumer-level desktop chassis for enterprise routing? You're not doing enterprise *anything*. See the title of this post. If you showed up with anything except a 1U rackmount machine, I'd show you the door.
Please help metamoderate.
Layer 2 is bridging. Layer 3 is routing. Switching used to be doing bridging fast and cheaply using specialized hardware, but if they want to throw in routing features in the same box, that's still fine. And usually the routing in a Layer 3 switch is dumber than the routing in a router, though that's usually deliberate marketing (leaving out BGP so you still get to buy a Real Router.)
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks