Slashdot Mirror

It's fairly simple. Get yourself an IP range, get some connections and do some BGP over it. Sure it's relatively expensive but if you need that type of connectivity on location, then you should be able to pay for it. There are many small data centers out there that have this setup. In open source you can find a solution called Zebra but many ISP's give/lease/sell you the hardware for it from Cisco or Juniper.

For other setups refer to this: http://linux-ip.net/html/adv-multi-internet.html - if you need site-to-site just set up two VPN's over the two connections and route through them. It's all very simple if you break it down.

HTB is Hierarchical Token Bucket, a CBQ (Class Based Queueing) discipline for Linux. It lets you create a hierarchy of queues for a network link. The "Token Bucket" part means each leaf and node in the tree has a "bucket" that constantly, slowly fills with tokens. Sending a byte removes a token. So, on average, you're only guaranteed the fill rate, but if you haven't used it for a bit, you can send a burst until your bucket is empty. Extra tokens can be borrowed between nodes if they're not used by the others, up to the max rate. Thus you get minimum guarantees, max limits, and bursts, such as being able to quickly fetch a web page even if the link is full from others' usage, if you haven't used up your tokens.

For instance, you could have Customer A, Customer B, and Customer C at the top level, and then they each have a second level of HTTP, BitTorrent, and SSH. Customer A and B get a rate of 128k, and C gets 512k since he pays extra as a business customer. They all have a max rate of 6M, since that's the speed of their DSL lines, and a burst size of 1MB. Then, they have SSH (with a small rate and a small burst), HTTP (with a high rate and a large burst), and BitTorrent (with a 1k rate, and a small burst).

As long as Customer C isn't using any bandwidth, A and B can use it all. As soon as C wants to use some, he first gets his guaranteed 512k - no matter what - and then they all split any leftovers in proportion to their committed rates (So A gets a share, B gets a share, C gets four shares). If C only wants 512k, A and B each get to split all the leftovers evenly.

If A is using BT like mad, but then opens an HTTP connection, it'll be allowed most of his net connection (it has a high rate, but still lower than the full line speed). BT will automatically (and instantly) be throttled until HTTP is done. When he types on the SSH connection, it'll use little bits of its burst speed to refresh the window instantly, but its small rate won't let it consume the whole net if he accidentally cats /dev/urandom.

Sounds great, right? There are a few gotchas: You can only queue packets like this when *sending*. What're you going to do, receive a packet from the slow link and then delay it before sending it over the fast one that's not saturated? (Well, yes, you can, and it makes a limited amount of sense to fine tune TCP's flow control, in addition to selectively dropping packets to make it back off, and other tricks.) It's good, but it doesn't necessarily make optimal tradeoffs between latency and bandwidth - HFSC is an attempt to address this. Also, this is a moderately heavyweight way to do things. It has to spend some CPU classifying packets, and memory to track the buckets' state, so other queueing disciplines and schedulers exist that work on other methods (such as statistical, instead of discrete tracking), that are more appropriate for very large ISPs. Also, as a large ISP, you're going to be using Cisco, not Linux, for routing. :) But Cisco has sophisticated QOS as well.

Despite how complex this sounds, even using the simplest case on your home router will make a huge improvement in the weak side of your DSL line, the uplink. Several of the open source WIFI router firmwares support it out of the box for this reason. I have survived having my web site on my DSL linked to the front page of a popular site known to bring servers to their knees, without any lag in SSH or games, or interruption of mail or other services. We only noticed because our bulk transfers slowed to a crawl, as intended.

Learn more:

HTB: http://luxik.cdi.cz/~devik/qos/htb/ (the user guide has a good overview and pretty graphs)

HFSC: http://linux-ip.net/articles/hfsc.en/ (More pretty graphs and good explanation)

Linux Advanced Routing and Traffic Control list: http://lartc.org/ (The howto is out of date, but very enlightening)

I have recently looked into this for a project and here is some information I found.

http://www.cyberciti.biz/howto/question/static/linux-ethernet-bonding-driver-howto.php
http://linux-ip.net/html/ether-bonding.html
http://www.automatedhome.co.uk/Internet/ADSL-Bonding-How-To-and-Review.html

If you want to use two DSL modems, the best option for this is to use actual PCI ADSL modems, such as the Sangoma S518. If you are using a stand-alone DSL modem/router you will be limited greatly by the hardware whithin it. Using an internal DSL card you will be able to directly connect to the ATM network without using multiple bridges between multiple technologies. This allows layer 2 bonding (if your ISP supports MLPPP) instead of just layer 3 bonding. This means you can load balance each alternating bit (much like RAID striping), instead of just by connection (as in the case of server load balancing).

In the US you can find a CLEC (Competetive Local Exchange Carier) in your area. The Public Utilities Commision in your state should provide a list of registered CLECs. Call them all and ask if they provide Bonded ADSL links, and how much they charge. Ask them if they are just a Reseller CLEC or if they are actually a Facilities-Based Colocation CLEC.

CLECs are smaller phone companies. In almost all cases they are much more flexible and customer-oriented. Their support staff are usually the same guys that actually go out in the field and hook people up, not just some outsourced company in India or Pakistan.

CLECs come in two flavors, Reseller and Facilities-Based. Reseller CLECS are just marketing companies, they don't provide any services and will not be able to provide anything beyond that which your ILEC provides. Facilities-Based CLECs actually have facilities and rely on the ILEC as little as possible for providing services. In many cases the copper lines going to your house are all owned by the ILEC so they will need to lease the last leg of the circuit from the ILEC, or your location may be outside of the area they provide service so they will lease a digital circuit to your location and provide the ISP portion of the Internet connection.

On the Colorado Public Utilities Commision website they provide a PDF document of all CLECs in Colorado:
http://www.dora.state.co.us/PUC/telecom/TelcomProviders.htm
Your state should provide a list as well in some form.

If you are using Cable Internet and ADSL to provide even greater redundancy (I would strongly suggest this if reliability is more important that speed) the cable modems out there usually are just a bridge device and therefore you can use one ethernet port for the Cable modem and one ADSL card (or use an ethernet port for the dsl modem, but make sure to turn off NAT on the DSL modem/router and _route_ [not DMZ] all trafic to the real gateway/router/firewall box... don't ever double-NAT as it is hard to troubleshoot and causes all sorts of problems). When using two different providers you will only be able to do Layer 3 connection-based bonding.

Another method is to use a consumer router designed to provide layer 3 bonding and failover. The Linksys RV042 router supports these features, as well as QoS, VPN, etc.

The basic OS install was more or less easy, once we battled through the serial port redirection setup (guess most linux users never used a serial port before. After all, why bother when the box sits under your desk). I stil like serial ports over video for one major reason: issue resolution (when bad things happen, having that panic string saved by a console server can really save the day)

So adding two lines to the grub config (one for grub itself and one for the kernel it starts) and setting the BIOS is too hard? I fail to see how it could be much easier under solaris.

Issue #1: multipathing drivers for the SAN. With solaris, you just plug the thing into the san and all of the storage that the host has access to just showed up. Multipathing was instant and I didnt have to do jack. I could see what devices mapped to which physical array with a simple command. I didnt have to guess which array /dev/sde came from vs /dev/sdf. When you have 20 luns mapped to the same host from two different arrays, its kinda important to know which drives come from which array and what the corresponding lun numbers are. That said, most linux admins I've talked to didnt have a clue about what I was talking about since they never had a san.

For part a in linux, unfortunatly they appear to have pushed multipathing on to the device driver layer, for the second part dev-mapper is your friend. Anyone who doesn't use it against a SAN is an idiot.

Issue #2: dynamically add luns: With solaris, you just change the mapping on the array and the host picks it up and auto creates the dev links. That was easy. On Linux? you've got to be kidding me... You get to echo some crazy strings into several spots in /proc and watch the fun start.

Or just run "scsiadd -s"

issue #3: IP Multipathing. With solaris, dladmin is used to create a bond (if it is going to the same switch and the switch supports bonding) or use the built in ip multipathing to do an active/failover setup if you are going to multiple switches. Very well documented and very easy to do. With linux... yeah, bonding is a fun task. Need to go to multiple switches? no such luck, you are screwed. I eventually used VCS to take over the systems main IP and uses its IPMultipathing agent to do the job for me. VCS on solaris just hands the task off to mpathd since the OS already does it for you.

Standard, in-kernel 802.1ad is too hard for you?

Issue #4: zones: dont get me started. I dont want to run another entire OS, I just want name space isolation and chroot is so primative it is not even funny. Zones gives me everything I want with minimal overhead. It would have been nice to have since there are a few oracle products that dont play nicely with clusters (*Warehouse Builder*) because they imbed the host name everywhere. We could put it under Xen, but this is an app that moves huge amounts of data around, not exactly a good candidate for virtualization. Zones let us get around Oracle's brain dead use of the hostname, no such luck with linux.

It's called linux-vserver. Yes it's a patch, but it works well.

Issue #5: 3rd party drivers vs the new kernel patch. If I install a 3rd party device driver in solaris and upgrade the kernel, I dont have to rebuild/reinstall the driver. Linux (even redhat 4.x with their "back port") forces me to rebuild/reinstall every damn time. Its great if the driver is standard with the kernel, but if you need something outside of that (lsi multipathing drivers to get around #1 and 10G NIC drivers in my case) and you are screwed. No wonder up2date ignores all of the kernel* rpm's by default.

Fair cop, in debian at least they keep binary compatability wherever possible in a stable release

Issue #6: Whats the s

It looks like that's even supported in the Linux kernel. But does it really work?

Basically, you'd be looking at doing the following things. Multiple outbound providers, which will need another routing table built for the second link. Then you'll need to dive into QoS to split up your traffic into your definitions of bulk (HTTP, FTP), priority (Gaming), and drop (P2P). I notice that you have no default set up, but I leave that up to you. Finally, you can use iptables to mark and NAT your traffic out the right interface.

Under Windows, you would need some advanced routing software I think. ISA may do it, but I doubt your budget allows it. By default, Windows does have the ability to enforce QoS terms, but you'd need something to apply those QoS marks (I doubt that games commonly mark their packets with ToS)...which means a bridge in front of the Windows router. Might as well use a Linux router instead.

If anybody knows of a way to get a Windows box to route based on ports, I'd love to hear it.

Oh, and a simple solution for the exact problem you describe (which I don't think is what you really want) would be a proxy for the HTTP and FTP link, and a router for the other link. All HTTP and FTP requests would be sent out the proxy, everything else would go the default route (to the router) which could be configured to drop P2P and route everything else. Optionally, you could do QoS on the router to prioritize certain traffic. If you go that route, I'm fond of AnalogX Proxy (for Windows) because it's free and simple. Of course, that does require client configuration....unless you use Transparent Proxying.

Basically, you'd be looking at doing the following things. Multiple outbound providers, which will need another routing table built for the second link. Then you'll need to dive into QoS to split up your traffic into your definitions of bulk (HTTP, FTP), priority (Gaming), and drop (P2P). I notice that you have no default set up, but I leave that up to you. Finally, you can use iptables to mark and NAT your traffic out the right interface.

Under Windows, you would need some advanced routing software I think. ISA may do it, but I doubt your budget allows it. By default, Windows does have the ability to enforce QoS terms, but you'd need something to apply those QoS marks (I doubt that games commonly mark their packets with ToS)...which means a bridge in front of the Windows router. Might as well use a Linux router instead.

If anybody knows of a way to get a Windows box to route based on ports, I'd love to hear it.

Oh, and a simple solution for the exact problem you describe (which I don't think is what you really want) would be a proxy for the HTTP and FTP link, and a router for the other link. All HTTP and FTP requests would be sent out the proxy, everything else would go the default route (to the router) which could be configured to drop P2P and route everything else. Optionally, you could do QoS on the router to prioritize certain traffic. If you go that route, I'm fond of AnalogX Proxy (for Windows) because it's free and simple. Of course, that does require client configuration....unless you use Transparent Proxying.

Please visit http://linux-ip.net and more specifically for your problem: http://linux-ip.net/html/ch-advanced-routing.html

That should get you started. It's be no means simple, but my understanding is that once you get it up, it works.

Please visit http://linux-ip.net and more specifically for your problem: http://linux-ip.net/html/ch-advanced-routing.html

That should get you started. It's be no means simple, but my understanding is that once you get it up, it works.