Powerful Linux ISP Router Distribution?

fibrewire writes "I'm building a Wireless ISP using commercial grade, low cost equipment. My main stumbling block is that I cannot find a decent open source ISP class routing distribution. Closest thing to even a decent tool is Ubiquiti's AIRControl — but even it doesn't play well with other network monitoring software. I've used Mikrotik's RouterOS for five years, but it just isn't built for what I need. I don't mind paying licensing fees, but $300K for a Cisco Universal Broadband Router is out of my budget. Has anyone seen any good open-source/cheap hardware/software systems that will scale to several thousand users?"

Comments

Just use any Linux distro

[ls671]

Just pick up your favorite Linux distribution and get back to me with your requirements. I think Linux can easily do what you need almost out of the box. It is only a matter of configuring it. I bet some would recommend looking at OpenBSD or FreeBSD as well.

Either way, you would definitely have a more flexible solution that any canned product will provide you with.

Re:Just use any Linux distro

[grub]

Does it have to be Linux?

Why not try OpenBSD and its excellent BGP implementation OpenBGP! It powers some pretty hefty businesses and ISPs.

-

Re:Just use any Linux distro

[ls671]

The title in the question was asking for a Linux distro.

Anyway, you have proven me right, if you read my OP very carefully, it states:

> I bet some would recommend looking at OpenBSD or FreeBSD as well. ;-))

Re:Just use any Linux distro

[grub]

Yeah, I meant to reply to the story, not your comment. :)

Re:Just use any Linux distro

[ls671]

blah, blah, that was just a typo.

Go fuck yourself !

Cheers,

Vyatta

[Anonymous Coward]

http://www.vyatta.com/about/press_releases.php?id=75

try the beta v6

Erm... Requirements?

[teqo]

So AirControl "doesn't play well with other network monitoring software" (which one, and why?), and MikroTik "isn't built for what [you] need" (what's that?) - other than that, you don't give us any idea what you really expect. What are your requirements? Suggestions out of the blue: OpenWRT with quagga/zebra, hostapd, radius, olsrd, b.a.t.m.a.n. etc. etc, or you might want to have a look at Vyatta (no affiliation).

Are you serious, or just killing time?

[jeffmeden]

So Cisco makes billions of dollars a year selling some ungodly expensive, ungodly powerful head end router like devices (not even routers in the IP sense) and somehow you suspect a Linux distribution with the same features is going to unpack itself and be everything you want it to be? You need to tell us what the rest of your platform looks like if you expect any answers that go beyond 'any linux distribution can act like a router!'. What subscriber equipment is in use? How much user control do you need (access on/off vs. bandwidth filtering, etc.) Details, details, details.

Re:Are you serious, or just killing time?

[dave562]

And beyond that, just because a Linux box might support all of the protocols and implementations that Cisco has leveraged in their own products, it does not mean that the Linux box is going to configure itself. A lot of the reason that Cisco makes money is because they provide solutions. The solutions themselves leverage established technologies in many cases (RFCs are in the public domain), but Cisco makes them work together. It's the old discussion about Open Source vendors. They aren't making money selling people Linux because Linux is free. They are making money selling people Linux configured to perform specific tasks, and then selling support to keep the solution functioning and up to date.

Re:Are you serious, or just killing time?

[b1t r0t]

The "same features"? You mean like ASICs that forward the data with low latency once the route is established? Yep, Linux is going to somehow magically add those to your computer, and that's one of the reasons people pay the extra money for Cisco over some old P3 tower PC and a CD-ROM with a penguin on it. Another is that they fit nicely in a rack.

The submitter apparently has his own unique idea of what "ISP class" means. Admittedly, this is for a wireless network, so there is already a bit of latency expected and maybe not as much total bandwidth as a wired ISP, but you can never remove latency, only add less. And as you have pointed out, "ISP class" should include things like metrics and controls for users.

Re:Are you serious, or just killing time?

[BobMcD]

When someone is giving away exceedingly powerful, linux-equipped servers for free, let me know where and when. Until then, software (as in Microsoft vs. Linux) is *not* equal to software/hardware (as in Cisco). Plain and simple.

So you've never heard of the FrankenPix, I take it?

I'd buy a claim of 'more stable', 'customized', or something similar, but 'exceedingly powerful' probably just isn't true. Barring some evidence to the contrary, I see a given Cisco device as about on par with an extremely weak desktop computer, in terms of pure 'power'.

I'm not undervaluing the total package. I just think you may have gone a tad too far with the Kool Aid.

Further, I don't think the question was asking about hardware anyway, so I'm not certain why you'd be muddying the waters with that part of it. Software to software to software, the comparison remains valid.

Re:Are you serious, or just killing time?

[rantingkitten]

Do you need a Cisco Catalyst to handle 3 desks on a fairly slow DSL line, who aren't doing outrageous sharing between each other? No.

Sheesh. I wish someone would tell that to our clients. My company provides service to (mostly) small businesses, and half of these little five-man operations have some totally over-engineered Cisco gear acting as their network edge because some smartass, self-styled "IT Guy" told them it was the best. Surprise, he vanishes after plugging it in and collecting his fee, and now the client has all these problems with our SIP service and of course they have no idea how to manage their own equipment, and WE end up looking like jerks because our stuff won't work out of the box with whatever equipment the client has.

Could you do the 3 desk operation with a Linux machine and 4 network cards? Sure. In this example, it's cheaper to pick up a cheap hub, than to take even a salvage machine and put 4 network cards in it.

Here, though, I disagree. At the same company I mentioned, when I joined, we were a three-person operation, and we used a Linux machine with two network cards and a switch as our router. It worked great as we scaled up in staff numbers, particularly when tools like ntop and tcpdump existed to let me see when some joker was ruining it for everyone by torrenting the entire internet. If you never plan to expand, then sure, some cheap little router toy from Dlink or Linksys will do fine, but if you intend to grow, may as well do things right the first time than have to re-engineer your network down the road.

Also, a hub? Who the hell uses hubs anymore? I can't even think of a use for them these days other than packet sniffing, and an inexpensive managed switch will let you do that.

Re:Are you serious, or just killing time?

[mattsday]

The RFCs may be in the public domain, but it's companies like Cisco that champion them. some examples of common RFCs Cisco has been involved with.

Thus, these guys are setting most of the major network standards, as well as implementing them.

Re:Are you serious, or just killing time?

[JWSmythe]

    I don't believe in overselling customers. I believe customers appreciate the fact that I'm looking to milk them for extra money. Really, I can score one big scale, or I can build a relationship and continue with them as needed. I've had customers not call for years because they didn't need anything, but the minute they do, I'm there for them.

    Growth is a funny thing. A lot of places I've seen have had 4 desks with the intention of growing, and years later they still have exactly 4 desks. One place had a dozen or so servers with high hopes for the future. Those high hopes were a serious understatement. Their partial T3 became multiple GigE circuits, and their dozen server became over 100. Even the first big growth spurt overgrew the agreed upon server naming convention and it had to be changed after two years.

    One place I worked at, which was growing rapidly, they were set up with a bunch of hubs (I'll explain the hubs thing in a moment), and terrible links between the suites (multiple suites in a complex). It was terrible. Literally, it was normal to have >100ms pings between suites on a good day. I got 6 Cisco Catalyst 2924XL-EN's with 4 port 100baseFX cards, deployed one switch per suite, and ran fiber between all the suites. Total expense was about $600. Then the economy took a dump. They started downsizing, and I believe they were down to something like 5 desks and 3 servers (don't ask).

    Ok, now the hubs thing. I say "hubs" for any low end consumer grade unmanaged "switch". For some manufacturers, it was a marketing ploy to say "switch", which just meant "auto speed switching", where it would handle 10baseT/100baseT/100baseTX, but was still a hub (you could see all traffic on all ports). Some really are switches, but usually not at the level of a real managed switch. If you can get 5 ports for $20, it's a hub. :) I have seen some recently that act like a hub, which is really sad. Well, not just act. They'll even have a single collision light on the front. Oh, there's a big hint. :)

Re:Are you serious, or just killing time?

[mysidia]

Show me the Franken' Catalyst 2950/6500 Sup720 3BXL, Franken Cisco 12006, or Franken Juniper M7i/M320, and then I'll be impressed. Your desktop PC will not contain TCAM or other components required for a minimal level of forwarding performance needed by an ISP.

After all these years, a desktop PC still cannot perform the task of a simple 8 port switch, at nearly the same packet rates as the switch. The packet rates that can occur on an Ethernet network easily overwhelm the desktop PC's limited interrupt capacity and memory I/O bus bottlenecks.

For an Enterprise branch office edge a desktop router is fine. Because Enterprises only buy a limited amount of capacity from an ISP. Also, Enterprise branch offices have only clients, not servers, so they aren't really subject to a DoS (rejecting unwanted packets is half as expensive as fully forwarding normal packets).

Of course, Enterprise server farms never use a firewall at the edge on the path into the servers, unless the periodic unavailability due to DoS attack taking out the firewall is not an issue.

But for an ISP, if you are planning on being a serious ISP, your core business is providing a professional service. Use a well-designed solution, not something you've cobbled together from off-the-shelf parts. You get real value buying gear that performs forwarding in hardware

In the long run, one 24 hour outage or service degradation, can cost more than engineering the network properly, and using good managed pieces.

The fact of the matter is the FrankenPIX was based on the original PIX platform, and Enterprise firewall, that used to be just a PC with some fancy packaging and a proprietary flash card. That platform has been obsolete for many years, and is not suitable for an ISP, anyways.

In case you didn't know, Firewalls like the original PIX can't handle that much traffic, and they are easily DoSed into oblivion by a simple flood.

Anyways, decent gear for service providers these days offloads work to hardware. And runs on a real-time OS that can provide something closer to a service level guarantee than a commodity OS can.

In case you didn't know... Linux is not a real-time OS, and cannot provide timing guarantees a RTOS can.

Generic Linux running on commodity hardware cannot provide proper separation between control plane and forwarding plane.

For certain very important functions, a commodity PC simply can't match the performance of a dedicated ASIC.

You can talk BGP all you want, but you can't reliably forward 30,000 pps through a commodity PC, or push speeds higher than approximately 200megs, due to interrupt contention.

There is also the matter of reliability of the hardware...

Commodity desktop parts are not designed to run 24x7, and they fail frequently. Physical failure in routers is rarer, unless there are environmental issues, or the equipment is old.

Re:Are you serious, or just killing time?

[mysidia]

Another is that they fit nicely in a rack.

And they provide a packaged solution, that most network engineers recognize and know how to manage, troubleshoot... meaning it will be easier to find/hire people to help manage it, than some custom home-brewed solution?

Lower long-term operational expenses, hardware is darn proven (fewer operational risks than you have buying commodity desktop parts), and you can get a support contract, usually (or opt to save money upfront by finding equipment and replacement parts in the aftermarket).

Many of the low-end routers 26xx are pure software switching. But they can still perform better than Linux, because the OS is designed solely for that purpose, which means performance optimizations too.

Linux is more of a jack of all trades. Forwarding performance and operation as a network device isn't a central design goal in the linux Kernel.

Re:Are you serious, or just killing time?

[Mad Merlin]

FWIW:

PCI / PCIe x1 are both ~1Gbps max throughput (not counting overhead, that's raw bus speed). All the other PCIe's scale linerly, thus a PCIe x4 is 4Gbps bus speed.

After communications protocol over the bus that speed drops (not sure how much). There are other factors as well but what it all comes down to is PCI or PCIe can really handle only about 500Mbps per link.

-nB

Not quite. One lane of PCIe v1 is 250M/s, double that of PCI. One lane of PCIe v2 is 500M/s, double that of PCIe v1. So, a PCIe v2 4x slot would be able to push around 2G/s, or 16 Gbit/s, which is slightly more than the 500 Mbit/s you state.

Furthermore, given that built in gigabit ethernet ports on any motherboard built in the last 5 years or so are connected via PCIe, and I've never had an issue saturating the whole gigabit, it doesn't make a lot of sense to say that PCIe is limited to half a gigabit.

Re:Are you serious, or just killing time?

[atamido]

The "same features"? You mean like ASICs that forward the data with low latency once the route is established? Yep, Linux is going to somehow magically add those to your computer, and that's one of the reasons people pay the extra money for Cisco over some old P3 tower PC and a CD-ROM with a penguin on it. Another is that they fit nicely in a rack.

A lot of router equipment is essentially an x86 PC. Add on cards are often just PCI or PCIe cards. You'd be surprised how commodity a lot of that equipment is. At least, for a big part of the mid range stuff.

Granted it's all specially chosen hardware and custom firmwared, plus Cisco IOS is a heavily developed and mature OS specifically written for routing, so you're not going to see anywhere near the same performance with some random Linux whitebox system.

Re:Are you serious, or just killing time?

[shaitand]

ASICs offload processing power from the cpu as dedicated hardware. But the PC you are running linux on likely has a CPU that is at least 10x as fast as that in the CISCO routers (and use a lot more power) so they can keep up.

This is one of those things that is as much about marketing as reality. There are no shortage of hardware appliance network boxes like BIG-IP LTM/GTM and Bluecoat ProxySG's that cost tens of thousands of dollars and are nothing more than BSD/Linux rack mounted PC's in a fancy case. These devices have no trouble handling enterprise loads (which is about the only place $50k+ pieces of equipment will be found in the racks).

The linux box does consume much more power to accomplish the task than the cisco with its ASICs but the raw power is definitely there. A more significant concern than the processing is the bus speed. I doubt that is a problem if he is concerned about an $800 software license (mentioned in a suggestion earlier) in that kind of budget range he isn't going to have links that could tax the bus.

Re:Are you serious, or just killing time?

[shaitand]

'Ok, now the hubs thing. I say "hubs" for any low end consumer grade unmanaged "switch". For some manufacturers, it was a marketing ploy to say "switch", which just meant "auto speed switching", where it would handle 10baseT/100baseT/100baseTX, but was still a hub (you could see all traffic on all ports).'

Your showing your age here my friend. This hasn't been true for many years.

Be more specific!

[dokebi]

Without more performance and cost requirements, it's really hard to figure out what would work for you.

Are your users all in one building? Over a large area? Are you talking about a dozen access points or hundreds?

Without some more specific information, only advice I can give is:
Soekris boxes with FreeBSD.

Good luck.

Re:Be more specific!

[fibrewire]

The Coachella Valley is the area - all of it. A large area.

A dozen to start but hundreds in the near future - i'm going to provide high bandwidth service for next to nothing. So the routing HAS to work for minimal bucks.

Mutually exclusive

[vawarayer]

I'm building a Wireless ISP using commercial grade, low cost equipment.

To me, some words in this sentence seem to be mutually exclusive.

To my humble opinion, a good ISP needs to have good reliable equipement. Sometimes, out of the box routers are better because they don't have moving parts and their firmware could be more stable than a full-blown OS (even if it is Linux).

Disclaimer: Not that I don't like Linux, I use it all the time.

Re:Mutually exclusive

[BobMcD]

To my humble opinion, a good ISP needs to have good reliable equipement. Sometimes, out of the box routers are better because they don't have moving parts and their firmware could be more stable than a full-blown OS (even if it is Linux).

If not for this reason, why do you suppose the question got asked?

Re:Mutually exclusive

[Fez]

You can have low-cost commercial grade services run using off-the-shelf hardware.

pfSense includes support for CARP, which lets you build high-availablity failover clusters. You can have two (or three or four...) cheap systems and if one dies, just fix/replace it as needed. The backup system(s) automatically take over and nobody would likely even notice the changeover.

When it's cheap, that is much easier to consider.

If you want no moving parts, you can use an ALIX box, Soekris, or perhaps even some atom-based boards. If you want to use server-grade boxes to make yourself feel warm and fuzzy, you can do that too. Supermicro even has a server-class atom board in a 1U rack which runs pfSense very well for us.

Re:Mutually exclusive

[turbidostato]

"To my humble opinion, a good ISP needs to have good reliable equipement."

To my humble opinion, a good ISP needs to have good reliable *service*.

Ask i.e. Google to learn the difference.

Re:Mutually exclusive

[Xenna]

These guys:

http://www.applianceshop.eu/

Sell embedded systems with monowall/pfsense preloaded.

Extremely easy to use and reliable.
I use a pfsense one at home, no idea how things would scale...

Few requirements given but... Vyatta?

[backtick]

Maybe Vyatta @ http://www.vyatta.org/ does what you want. I really don't have any idea what that is from the actual post, tho. You need some routing for thousands of users, and can't afford a Cisco UBR. I'm not sure exactly if you wanted to use the UBR for DOCSIS type support for some reason (a la cable modem) but the fact it'll be wireless leads me to believe it won't be. I'm assuming you don't need a lot of physical ports, just something to manage your VLANS, some routed subnets, a bit of BGP, etc. Maybe XORP is what you want, tho @ http://www.xorp.org/ so you may want to look there. IHeck, 'm not even sure if you want to take a server with a bunch of PCIe ports and slam multiport switchable fabric cards in there like the ones DSS @ http://www.dssnetworks.com/v3/gigabit_pcie_6468.asp makes, or do something else. Maybe these links will help, and hopefully there'll be a detailed followup so we can aim at the real target :)

What on earth are you trying to actually do?

[sirket]

Routing and ISP's are huge topics- what are you trying to do?

The main problem with routing isn't bandwidth- anyone can pump enough 1500 or 9000 byte frames per second to fill a gigabit pipe. The problem is when you have lots of small packets. At that point, dedicated routing hardware with a high-speed TCAM becomes really important.

What kind of line cards do you need? ADSL? Ethernet? OC12?

What kind of services do you need to run? BGP? OSPF?

What kind of bandwidth are you going to be pushing?

Re:What on earth are you trying to actually do?

[fibrewire]

Tons of multicast video data will eat up 1/4 to half of my last mile bandwidth, followed by voip and data.

I'm trying to balance Access point range to around 1/2 mile without dropping bandwidth, so Ubiquiti AirMAX equipment seems to work in trial runs.

i don't want to drop below 100Mbit lan speeds, rates are fixed so if a customer can't connect they won't kill all the bandwidth for everyone else.

Client's actual throughput will be about 10Mbit down / 2Mbit up + about 45Mbit of Multicast video overhead - 100 clients will share about 50Mbit of bandwidth, if it scales out ok then 300 clients will share 50Mbit

Re:Hire someone who knows what they are doing.

[Em Emalb]

The fact that you are asking on slashdot shows that you are not qualified, and what you're going to get back is a bunch of others, who aren't qualified, suggesting all sorts of half assed hacks to do it which will just result in a utterly shitty service overall.

Dude, don't shit on a well-known slashdot tradition! How dare you!

Ebay is your friend.

[jjeffries]

Start off small. Pick up some used Cisco stuff off Ebay at 1% list. Maybe a 6500 with a couple of SUP2s for your core switch, a couple or four 7200s for the upstreams/customer facing bits. Make lots of money, upgrade to newer stuff as needed.

pfSense

[mhab12]

Give pfSense a try. http://www.pfsense.org/ Also a VERY active user forum at http://forum.pfsense.org/

Re:Hire someone who knows what they are doing.

[lymond01]

The fact that you are asking on slashdot shows that you are not qualified, and what you're going to get back is a bunch of others, who aren't qualified, suggesting all sorts of half assed hacks to do it which will just result in a utterly shitty service overall.

I disagree. The Open Source community has a thousand hidden gems that a person might not have heard about. Proxmox VE for one: virtualization, with a GUI, with live migration, and if 2.0 turns out, with heartbeat and failover (high availability). Most people have never heard of this where I work even though half the place is virtualized with KVM, VMWare, Hyper-V, etc. I would think the Slashdot, with its plethora of experiences, might come up with a little-known or workable solution in an already developed product that you haven't heard of yet.

Re:m0n0wall is a great BSD distro

[clarkn0va]

I have to agree, although I registered a vote for PFSense above. PFS is based on m0n0wall and both are excellent routers filling slightly different niches. I currently use PFS at home for its packages (freeswitch, squid), but I recently worked for a growing WISP and got them onto m0n0wall, now serving something in the neighbourhood of a thousand customers.

If you want pure simplicity, go m0n0wall. Otherwise, I strongly recommend looking at PFSense for the squid caching and adjust-on-the-fly connection table size.

Big Sur Wireless

[north.coaster]

There's a small wireless ISP located in the Big Sur area of California that seems to have been up and running for a few years now. Maybe the OP wants to build a system like Big Sur Wireless. Their web site includes a lot of details about their homebrew system.

Re:Hire someone who knows what they are doing.

[Jeng]

Wait, isn't shitting on topics a well-known slashdot tradition?

Re:Hire someone who knows what they are doing.

[Em Emalb]

Nobody expects to get shit on!

(To answer your question, yes. Slashdotters shit on anything and everything. We're like a pack of wild pigeons when it comes to that.)

As others have said...

[KiwiGod]

What's your interface to the net, line cards, bandwidth expectations, etc. I spent 5 years building a fairly heavy duty wISP network on a stupid low budget from my boss. You can obtain used cisco stuff for cheap. For instance, you can get your hands on a 7206vxr with a NPE-G1 for $10k or less nowadays... If you need something with high redundancy do do less intensive switching, you can pick up a 6509 with a pair of SUP2-MFSC2 cards for less than $2k. As far as support contracts go, I can't imagine that you need the latest and greatest IOS, let alone a support contract that costs more than the replacement of a piece of hardware. On a side note... why are you asking about the uBR series? Are you not running an ethernet network? Last I checked, there's no such thing as "low cost commercial grade." Depending on where you are, unlicensed stuff may not cut it, dealing with interference etc. And licensed hardware is certainly not cheap. With wireless, as well as so many other areas, you get what you pay for.

Re:Hire someone who knows what they are doing.

[Ichijo]

The proper question is: How do I find someone qualified to do this for me?

The problem is, if you ask a Cisco person to do it, you'll get a Cisco solution, even if it isn't the best solution for the task.

Re:Hire someone who knows what they are doing.

[nine-times]

I think you have a good point, but I don't necessarily agree. First, we don't know what market the submitter plans on operating in or who his clientele are. We don't know what his experience is, how much resources he has, or exactly what level of service he intends to offer. Like the guy who criticized the submitter for refusing to buy a $300k Cisco router, I think you committed a common mistake in thinking that IT is just a series of 1-size-fits-all solutions, and that if you going to use the "right" solution to each problem, you shouldn't bother.

The era of entrepreneurship and hacking things together isn't over, and it probably never will be. Our tools and hacks may become more advanced, but hopefully there will always be people trying out new techniques and business models, testing new start-up technology, and finding different ways of accomplishing the same goals. The answer isn't always to pay an expensive expert or to use established tech.

As for this:

You could get by with this in the late 90s, but when you're going to compete with cell phone companies, cable companies and standard POTS companies, you probably need to have a bit of a clue.

That's true, but neither my phone company nor my cable company provide wireless access where I live. Cell phone companies provide wireless, but it's pretty spotty and slow, and I live in NYC. There are plenty of areas in the US where no service is available except through dialup. Obviously these large companies aren't interested in competing in all markets, so if you come up with a business model and think you can make it work, then I say go for it.

Re:Hire someone who knows what they are doing.

[GooberToo]

The proper question is: How do I find someone qualified to do this for me?

You mean because he's humble enough to realize he doesn't know every thing, you believe he's unqualified anything. I suggest you look hard in the mirror and read what you just wrote to yourself.

Re:Hire someone who knows what they are doing.

[TerribleNews]

I disagree, wholeheartedly. The secret ingredient to a successful business is elbow grease. The fact that this person has asked slashdot this question is not a good indicator of success one way or the other. The important thing is whether this person will be able to take a significant number of the suggestions provided and give'em the old college try.

just a thought

[khelix]

I did not see anyone suggest http://www.untangle.com/ . i have only played with it for a short time, but it might be worth checking out!

Re:Hire someone who knows what they are doing.

[Anonymous Coward]

Have you worked at a cell phone, cable, or standard POTS company lately?
What exactly do you think you are going to get there besides a bunch of unqualified, "half assed hacks to do it which will just result in a utterly shitty service overall."

Besides, many of the folks posting in this thread are probably those same unqualified, half-assed hacks who work at such companies. Corporations don't have any corporate voodoo that makes them special any more than someone working for the government makes them any smarter or able to perform miracles (free healthcare, news cars, and money for everyone, YAY!) Put down the kool-aid and open your eyes.

Re:DD WRT

[pak9rabid]

http://www.dd-wrt.com/site/index

It's Linux on low cost wireless routers.

Yeah, that's just what I'd want my ISP to run as a core router.

Re:Screw Linux

[Fez]

pfSense 2.0 will solve the multi-wan traffic shaping limitation, and it's in beta right now. As for the multi-wan glitches, I'm not sure when the last time you tried it was, but the outbound load balancer was redone in 1.2.3 and 2.0 will have even more changes as well.

I run an ISP and we use a pfSense CARP cluster in front of our servers and it's worked great for us, but admittedly we are a small ISP. We also use it at more than a dozen customer sites. Everyone loves it.

OBSD or pfSense

[Anonymous Coward]

My history is: started on OBSD (due to hardware support, ironically); played w/ FBSD; ended up on pfSense.

My observations:
OBSD is absurdly security conscious... for ISPs especially, this is a good thing.
OBSD tends to have a lot of focus on new network features (pf, carp)
most OBSD features get ported to FBSD... but take time (look into carpdev)

pfSense (built on FBSD) has some overhead vs FBSD raw (obviously), but has *nice* management UI, package support, etc
customizations are easy for pfSense (I added some features to dhcpd a while back)... easier than generating the diff and submitting it ;)
pfSense is more focused on network features (they're working on fixing carpdev for FBSD)

I like pfSense a lot... I use it for routing between ~6 VLANs, IPSec tunnels with another pfSense, PPTP server, *tight* firewall rules (given 6 VLANs).

pfSense 2 will be adding a lot of nice features for businesses (multiple admin accounts, different permission levels, etc)

Vyatta

[Anonymous Coward]

What about Vyatta? It's a good router based on linux and you can install it on any old box you want or buy their hardware for it. Even has a cisco like interface if you want.

A suggestion

[scottraynel]

RuralLink Ltd (yes, I work for them) does what you want, linux-based wireless network management. Get in touch with us at http://www.rurallink.co.nz/contact-us

There's not a lot of info about that side of things on the website, but if you contact us we'll be happy to chat - and don't worry, we're all techs, there's no sales droids here.

Cheers,

Scott.

been there done that bought the tee shirt

[Anonymous Coward]

I founded and operate a wireless ISP serving about 1000 wireless subscribers, and have my own embedded linux distro inside just about everything. It would be a fair statement to say that linux literally saved our business on more than one occasion, by giving us the tools to overcome manufacturer software bugs, by establishing 'known good' systems of various types, by enabling read-only compact flash based systems running on solar power, by bringing a high level of utility and reliability into the critical parts of the network, by allowing us to make it anything it needed to be.

As a CPE, my linux distro never lets me down and never puts customers of at risk of 'stone dead - lights on but nobody home', like linksys/netgear/etc always seem to. Never having to tell someone 'just pull the power and plug it back in' for their connectivity is a real saving grace. And when in a business situation, I can equip these customers with connectivity devices that _do not fail_ and make us look stupid, while at the same time giving them useful feature sets unavailable in higher end router manufacturer gear (cisco 2621 - excellent hardware with great stabillity, just weak on features I get with dnsmasq, openvpn, tcpdump and others.. trying to diagnose network connectivity issues without tcpdump is just dumb.). Its also never choked and zeroed out it's own flash config for no goddam rason, unlike the previously mentioned low-end consumer devices frequently do. Basically, that consumer stuff puts you at risk and is suicide.

As a network appliance, linux flings packets just fine and gives you great tools to filer, mangle and generally control how and what it does. The ebtables code is awesome, the iptables stuff is killer, openvpn rocks asses, dnsmasq kills, there's just too many useful and cool things just go right. I have a pppoe server running rp-pppoe + my patches and userspace tools, running for years now and hit with every kind of client side bug and malfunction imaginable, and just keeps trucking along. Freeradius backed up with mysql is sweet as can be, and quagga for distributing my routes internally is just a dream. I have it all on read-only compact flash, so they never write and basiclaly will run until there is a show stopper hardware problem, at which point I will more than likely be able to remove the flash and put it into another machine and away I go.

There is a lack of management interface, and there is a learning curve to this route, but the upside is very low dollar cost and an attainable level of flexibillity, reliabillity and stabillity you are unlikely to find in any commercial solution anywhere. Cisco IOS is awesome, but you won't power anything that runs it off a 12v battery and solar panel on the side of a mountain and flinging/filtering 20mbps of traffic.

Good luck.

Step in the right direction

[fibrewire]

I guess i'm looking for a scalable ISP-in-a-box solution. And if it doesn't exist, then let's build one. But Proxmox VE looks like it will fit well with managing computer resources between the handful of Dell 2950s slated for Zimbra, FreeIPA (Active Directory for Linux), Nagios, Cacti, and AIRControl. Still looking for a good FreeRADIUS server i can tie into FreeIPA - but i need lots of other stuff than just a router-in-a-box. A balance between smartest / practicality / economical directly translates into cost savings of the end user. Someday i will be able to provide free internet, but for now i am targeting $20-$40 a month for data, voice, video, and multicast TV. Some features of a good OSS router needing attention are:

* PowerPC vs X86 vs GPU - does routing perform better on PowerPC (Mikrotik / Vyatta / Cisco)? would an Nvidia Tesla solution work well?
* Easy to manage large scale routing implementations - speed of deployment, discovery of devices, failover, centrally monitored?
* Weatherproof - power outages, network hiccups, etc. nothing more irritating than going on-site to an AP to reboot in the middle of a storm

For more details about a specific area please ask.

Re:Go 2nd Hand Cisco

[POTSandPANS]

If you are just starting up, I'd suggest a couple of Cisco 3550 layer 3 switches with the IP Services image. They don't have all the features of the big routers, but they can handle a huge amount of traffic. I doubt you could build a linux router that would handle as much traffic for the same price as a 3550.

Re:Hire someone who knows what they are doing.

[tkjtkj]

Rude, but true.

Correct question?

[Runaway1956]

Alright - I read your question, then a couple responses - but it isn't clear here that you're asking the question correctly. Humor me for a moment, then decide whether you asked the right question.

You have access to the web, with a hardware router behind the modem. That hardware router services both wireless and wired LANs, right?

You want to set up a router behind that router? You still won't be able to monitor traffic going through that hardware router. You need to put your *nix router between the modem and the hardware router, so that you become the gateway for all traffic going to and from the internet.

Of course, that is still not satisfactory if you wish to monitor traffic within the LAN. For that, you want to eliminate the hardware router entirely. Install the hardware to make your *nix router serve the WIFI and the wired LAN, and eliminate that hardware router entirely.

You can only monitor and control traffic that is being gated through your router, so you want it ALL to be routed through your box!

Re:Vyatta has licensing issues

[sleeper0]

Community Edition is free comes as a binary or full git tree, lags a revision behind. You can't buy support or prof services for it, but I'm assuming you aren't in the market for those if you can't/won't spend $800

Need help

[Goody]

I'm building a WISP, too. Do you think I should get a T-1 or a DS-3 for Internet? I haven't been able to decide between BSD and Linux for my router operating system. I think I'm going to go with Linux because I think the penguin mascot is cuter than that Satan mascot, but it's easier to get BSD to run on a 486 these days.

Re:no DD WRT

[bartwol]

In my experience, I think there's something to what you say. The DD-WRT software is quite capable, but the CPUs in consumer routers are relatively slow and get bogged down when you fire up a bunch of chatty sessions, a good load of firewall rules, and try to pound data through too. Add monitoring of the router (which DD-WRT doesn't do much to support) and it doesn't take much to make the router start lagging and gasping for air. I've experienced such limitations in an office environment.

FreeBSD, BGP, OSPF, pf

[itzdandy]

Seriously, learn to love FreeBSD.

I am assuming that you will be doing a tree style network with a central location providing you bandwidth on a fiber link or T1/T3 etc.

Get a PAIR(at least, add more as necessary) of nice, quad core Dell Poweredge or HP DL series servers. FreeBSD+CARP them giving you as seamless load balancing/fail over as you can realistically get.
at each hub consider either buying commercial wireless routers or build your own. If you build just keep everything fanless as that is where your equipment will fail you.
Use OSPF on branches while being aware of scaling issues and where OSPF isnt ideal, kick in the BGP and you can link your OSPF clusters together giving an extra level on branch redundancy because traffic can hop to another branch if necessary.

OLSR in mesh cells, OSPF on the cells backhaul router linking these cells and providing multiple route options for redundancy, and BGP between groups of cells and between you and other ISPs etc etc.

You dont need to take the Mesh down to the client, only to the neighborhood AP level. The idea of mesh per client creates too many hopps and clients have too much latency. Ideally, you are no more that a 2-4 hops from the backbone, any more and you are going to be adding too much latency from the hops. When a backhaul link goes down and the OSPF saves your butt by routing traffic through a neighboring cell, you are already going to add latency and you dont need that complicated by 6 hops in the neighborhood and 5 more to the backbone (11 hops over wireless is just too many for broadband).

Re:DD WRT

[ntk]

They cut off your network access because of a report of infringement? Are you in the US? Do you think you could mail me at danny@eff.org with more info? We're always interested in the details of these incidents.

Re:Exactly what i wanted to hear!

[Glendale2x]

Sure, the 2800 and 3800 ISR series can take full tables easily. You can get a 3845 starting at $10k. NM-1T3/E3 module is about $6k. Both the 2800 and 3800 take DDR-266 ECC SDRAM (except the 2801); don't feel the need to pay Cisco's prices for commodity RAM if you really don't want to. The 3845 is recommended to handle up to 2 DS3's. According to people I've asked, you can push a 3845 to 100-150 Mb/s. You can go as low as a 2811 ($2k) and still take full tables, but only at fractional DS3 speeds. I would guesstimate a 2811 is good to 10-20 Mb/s, Cisco recommends it for 4xT1.

Also, consider that some ISP's will include equipment bundles with circuit orders if you haven't already explored that angle.

Re:no DD WRT

[operator_error]

The dd-wrt shop does have more powerful CPUs/throughput-hardware than is afforded by common WRT-class home routers. HOW much more powerful, or more throughput I do not know. Maybe someone else can comment, given the hardware available.

The prices are reasonable; it seems for about $75 you can buy a outdoor-unit that will blanket an area better than a home router.

http://www.dd-wrt.com/shop/catalog/