The article's conclusion is correct in a large scale environment, but it does not show any of the steps that get you there, or alternatives to putting everything behind a stateful firewall. Really, the thesis statement should have been "External facing servers should not be behind stateful firewalls".
In any large scale customer facing deployment, you have to leave a piece of the application facing the customers. However, you are best off limiting what is on that host (or hosts, as you are probably talking a load balanced solution) to just static content and calls to the application/database servers - which can and in many cases should be behind stateful firewalls. Protecting the customer facing box becomes an exercise in limiting attack scope - stateless router ACLs, hardening the box, and the like - things that protect against packet/session floods that may not fully saturate your actual bandwidth but could still cause a firewall to collapse under the number of new sessions that are being created/denied.
In short, make your external facing application be multi-tiered (preferably with redundancy) to achieve higher uptime and better resiliency against external threats. In my experience this model does seem to cause internal incompetency threats to break your application more often, however...
Except this is no longer true in a full-duplex world, you can approach 99% utilization on Ethernet at full-duplex. At the time token-ring was competitive, full-duplex Ethernet was just emerging. While IBM's marketing and some of the complexity of token-ring hurt it, what really killed it was the widespread emergence of full-duplex ethernet switches which basically eliminated the under-utilization problem while not having the complexity of dealing with a token-ring network.
Except they are not "throttling" you, they are just giving you lower priority IF you use over 80% of your bandwidth for 15 minutes AND the whole segment is over 70% utilization. This means that grandma can still get her mail when you are seeding the new release of Ubuntu, but you "lose" bandwidth if you actually hit 100% congestion.
Cable services are shared for the last mile between the homes that they pass. For Comcast, the last numbers I saw (from the fairshare information threads) were ~250 homes per downstream. The higher the per-household usage, the more they have to split up that grouping - which requires putting more cable in the ground, setting up equipment, etc.
This is the bandwidth crunch the cable companies have, not the core of the network. The article actually does not address that fact at all, and seems to assume infinite edge bandwidth with limited core bandwidth. This is true in an enterprise network, but is not true in a cable network today.
Network performance, in this context, is actually discussing the ability of the network to respond to change. This does nothing for the throughput of a non-congested link, but link-state protocols today can assign costs for links based on many factors, including availability and utilization. To the normal person on the street, this would not seem to matter, but in reality outages happen pretty often on large networks. The design of the networks today is often dictated by the limits of the link-state databases and the CPUs that power them - the bigger the database, the more CPU power it takes to run the algorithm to calculate the new best path when there is any change.
As the link-state database contains a list of every link in the network and the cost of crossing it, as well as every other link that it is connected to - you can see how that quickly becomes computationally heavy on a large network. Also, the larger the network, the more likely it is that you have some change in the network at any one moment - just probability working against you.
However, this adds a little bit of overhead into the actual database, and will make troubleshooting problems more complex in the real world. The implementation is a really key part of this, and will probably actually dictate if this gets picked up and used going forward. There is a real need for faster convergence time on modern networks as we move into more internet video feeds, as the loss of even a few packets can deeply impact the user experience - so the interest will be there, but the devil is always in the details.
After reading a portion of the PDF supplied, I am actually far more satisfied that this is new research and not a restatement of fundamental network principles. The PDF has the equations where he proves a few simple criteria can define the scope of any network topology changes based on the difference in cost of the new route. This allows you to limit the recalculation of routes, blocking them from most of the routers where the recalculation would have provided no change in the actual routing topology.
The challenges he states are real challenges, and many modern networks are defined by the limits of the link-state protocols. In essence, this is like auto-summarization of prefixes in bgp, only applied to links in the link-state database - a possible slight loss in accuracy for a large boost in routing performance. This would allow the (faster converging) link-state protocols to scale to larger networks, rather then having to divide them into areas or use BGP to route between different sections of the network (resulting in loss of convergence time).
To the end user, this would mean that the internet would respond faster to outages, and better route around congestion on any one link.
So, from reading the article, I see that the great leap forward here is "smaller routing domain in a link-state protocol leads to faster routing updates". But, looking at the existing link-state protocols, they were designed from the ground up with the ability to limit your routing domain manually so increase the convergence time and decrease memory footprint.
I guess that means the achievement here is to have a link-state protocol that automatically limits your routing domain by limiting propagation of routes. This however seems like it could lead to seriously suboptimal routing which is probably a bad idea in most network environments today.
And yes, when DOCSIS 3.0 gets here, it will be amazing. However, I would not expect widespread deployments before Q3, if not longer - the cable modems for DOCSIS 3.0 are still really new and the rest of the support is still immature. That is not to mention that you need to upgrade every modem on a segment before you can use many of those new QOS features, otherwise you cause the older modems to go offline.:\
Logistics is the enemy in a highly distributed environment, and slows everything down.
Still, even if you do put "limits" on your traffic for high priority, people will find those exact limits and ride them. This destroys the actual use of the high priority as eventually (when enough bit-torrent clients implement it with it on by default) all traffic is high priority and nothing else is.
What I am saying is: if there is a way for people to abuse the system, they will. It is the tragedy of the commons, over and over again.
While everything you said is absolutely correct for an IP, packet based environment, I would suggest that you read up on DOCSIS (and more generally TDM based transport protocols).
You cannot do "per packet" queuing in DOCSIS, because you are using a time divided protocol. The only way to prioritize bandwidth to an application in an shared TDM environment is to guarantee "slices" of the time to that particular modem that has an active flow for that application. Those slices are reserved for that node regardless of if they use it or not, so everybody else using that particular cable segment suffer.
It is not the upstream router segment where bandwidth contention generally occurs on a cable network. It is on the actual cable segments themselves, which are also the most expensive and hardest part of the network to upgrade. Even protocol upgrades (docsis 1.0 -> 1.1) are huge investments in money and equipment, because you have to upgrade every end user to a new cable modem if their current one does not support the new protocol.
Also, last monkey wrench to throw at you:
If you trust what a specific application is sending to give it higher priority, what is to say that another application will not use that precedence value even when it is not that type of service? It is trivial for me to mark all bittorrent traffic leaving my home router as precedence type EF!
Note this line:
Comcast has received a notification by a copyright owner, or its authorized agent, reporting an alleged infringement of one or more copyrighted works made on or over Comcast's High-Speed Internet service (the 'Service').
Comcast did not wiretap you, or generate this themselves. They got a legal notice from somebody else that this happened and passed it on to you. If you will note that they did not even give your physical address and information back to the company that filed the complaint for infringement. This is also a notification that a subpoena may be filed against Comcast to retrieve this information as part of the standard discovery process in a copyright infringement case. That is at the discretion of the company that filed the initial complaint, however.
Yeah, Powerboost is a gimmick as far as I can tell - it gives you a few seconds of very fast downloads. But for the majority of users, this is perfect - those pictures come down really fast from your email, or that youtube video you were linked loads up in moments, etc.
As for erratic, you are going cross country with most of those results. I would expect a good amount of erratic behavior, especially since you are also dealing with whatever load the speakeasy server has at that moment. The general trend of the results you posted was : the higher the ping time, the slower your maximum speed (Upload was affected far less, which really tells the tale). It is what I would expect, and the noticeable outlier of Tuscon is probably more of a site load issue when the test did the ping. The speed of light is fairly immutable in these matters!
The slower speeds to the west coast sites (Phoenix, San Fran, Mexico City) are all explained by TCP window issues.
Remember, the higher your round-trip time the slower TCP will let you send bandwidth unless you tune the TCP window upwards. For reference: http://en.wikipedia.org/wiki/TCP_Tuning
This happens with every ISP, and quite frankly when you start talking cross country distances the speed of light becomes a factor in how fast you can actually get round trip time down to.
The higher your latency to a site (aka, how long does it actually take a packet to get there) the slower your bandwidth will be without some extensive TCP window tuning. Reference:
That is why you get slower speeds the farther out you go, even assuming every link in between is not congested.
Also, most of what you post there is explained by powerboost - very fast downloads for the first few seconds, then they throttle it back. If you ran a sustained download (go get a linux distro!) you could see what the actual sustained download speeds are after about 5 minutes.
The Comcast terms of service actually state that you can have your service terminated for something to the effect of "impairing the service for others". From postings on the internet, this typically means 150-250gb+/month these days. This also does not happen in every case, so I suspect that they may not even look until people in your area are complaining about slow speeds during peak times.
Also do not confuse this with the letters that get sent due to RIAA/MPAA action, or the various remedies that cable companies have for hacking your modem's bootfile and upping your bandwidth.
As for your speed test server question, do not forget that distance to their server, load on their server, your default TCP window size, your network configuration, and many other factors can relate to how much bandwidth you get to a single speedtest site. Try more then one, and try some that are fairly close to you (geographically or via traceroute).
The problem with your math? Your "backbone" numbers only consider that highest.05% of users for the "2100" T1s.
Try figuring your numbers for the 11.5million customers on a bell curve of bandwidth usage with a peak at around 2gb/month (based on other posts). You will come up with numbers significantly higher then the "2100" T1s for your backbone - and thats not even considering that bandwidth usage is not steady, but (like any ISP) has peak and off-peak times. To have a non-congested circuit, you have to plan for "peak" latency.
Then, finally consider that the real bandwidth congestion that cable companies care about is the last mile - the shared bandwidth from the nodes. This can quickly become over congested because it is typically a 45mb/shared (downstream) and 8meg shared (upstream) connection on the cable plant using TDM (which means it is not even optimized sharing).
Not saying that this is a perfect situation, but the numbers you throw out there are misnomers when you actually scale things to real customer numbers and start adding in all the (possibly multi-use) infrastructure that lies between the ISP circuits and the customers.
Really, nothing you listed there should add all the way up to 287gb unless you have some massive files from your clients.
xbox hard drive is 15gb, so thats 30gb even assuming you filled it twice.
Linux ISOs? Let us assume the worst, 5x5gb - 25gb
MSDN ISOs? Just guessing, but I cannot figure thats going to be more then 20gb
Tivocast programs + extra youtube plus etc should not be higher then 40gb for the month (figure 120mb each clip, about).
So without your client backups, we are looking at what, 115gb about? Did you transfer 150gb of client data in a month?
Your math is off by an order of magititude atleast, in any modern installation. First, it would take longer then that to pull cable down a 5 mile road, unless they were doing it as the road was built. Digging holes, pushing pipe (conduit), and throwing cable through that. Now, your talking atleast 100000 just for the cable runs, not including the fiber from the upstream node back to the headend, or the conversion equipment, or repeaters. So, 200000 easy at that point (you would be suprised at what that equipment costs!) at this point, before we even talk about head end equipment!
For HSD, a CMTS that handles 3000-5000 people runs you right around $100000 by itself, and then you have the upstream network equipment (if you go cheap, $5000 router, running over preexisting fiber). Now, as thats about the cheapest CMTS you can buy, its what we use to support this whole rural area - your 238 homes + every other run they are going to do in the future. But the local cable board required they do high speed data in order to get the franchise, so they install it. However, to keep the data neat, we will "charge" 1/10th of the price off to those 238 homes. This is BEFORE video equipment - and thats 20000-60000 for analog equipment, and another 60000-80000 for digital equipment. Again, this is for an entire area, so we will prorate this down to 1/10th of the price.
So, more realisitic figures are:
Running cable for 5 miles : 200000 and 2-5 months, depending on terrain.
HSD service : 10500 (prorated for customers)
Video : anywhere from 8000 to 14000, prorated.
That is 220,000 right there.
And this is before you hire local techs to take care of that area, and assumes you have a preexisting head end with preexisting fiber to it. This does not include all the first 6 month promotions that you run into, or extra equipment costs, or costs to keep contracts for quick replacement of any failed equipment / spare equipment on site. (Video normally has spares - if the cable goes out on superbowl sunday, people get fired)
And I am sure I am underpricing some things. It is NEVER cheap to dig cable through the ground, especally not through residental areas. And these are (prorated!) upfront costs, so if you do not get the take-up you expect, your locked in to a 5-10 year contract with the city with equipment that costs more to run and maintain then profit you get out of the area.
Per the anrand tech review linked in the article, a 2100FP with an analog cable running at 1600x1200 will have very laggy displays. Moving to a digital (DVI) cable resolves this.
If you do not have a DVI out on your PC, there are quite a few cheap cards today that have DVI out - typically your geforce 4 line of cards.
Slashdot Mirror
The article's conclusion is correct in a large scale environment, but it does not show any of the steps that get you there, or alternatives to putting everything behind a stateful firewall. Really, the thesis statement should have been "External facing servers should not be behind stateful firewalls".
In any large scale customer facing deployment, you have to leave a piece of the application facing the customers. However, you are best off limiting what is on that host (or hosts, as you are probably talking a load balanced solution) to just static content and calls to the application/database servers - which can and in many cases should be behind stateful firewalls. Protecting the customer facing box becomes an exercise in limiting attack scope - stateless router ACLs, hardening the box, and the like - things that protect against packet/session floods that may not fully saturate your actual bandwidth but could still cause a firewall to collapse under the number of new sessions that are being created/denied.
In short, make your external facing application be multi-tiered (preferably with redundancy) to achieve higher uptime and better resiliency against external threats. In my experience this model does seem to cause internal incompetency threats to break your application more often, however...
Except this is no longer true in a full-duplex world, you can approach 99% utilization on Ethernet at full-duplex. At the time token-ring was competitive, full-duplex Ethernet was just emerging. While IBM's marketing and some of the complexity of token-ring hurt it, what really killed it was the widespread emergence of full-duplex ethernet switches which basically eliminated the under-utilization problem while not having the complexity of dealing with a token-ring network.
Except they are not "throttling" you, they are just giving you lower priority IF you use over 80% of your bandwidth for 15 minutes AND the whole segment is over 70% utilization. This means that grandma can still get her mail when you are seeding the new release of Ubuntu, but you "lose" bandwidth if you actually hit 100% congestion.
Cable services are shared for the last mile between the homes that they pass. For Comcast, the last numbers I saw (from the fairshare information threads) were ~250 homes per downstream. The higher the per-household usage, the more they have to split up that grouping - which requires putting more cable in the ground, setting up equipment, etc.
This is the bandwidth crunch the cable companies have, not the core of the network. The article actually does not address that fact at all, and seems to assume infinite edge bandwidth with limited core bandwidth. This is true in an enterprise network, but is not true in a cable network today.
Network performance, in this context, is actually discussing the ability of the network to respond to change. This does nothing for the throughput of a non-congested link, but link-state protocols today can assign costs for links based on many factors, including availability and utilization. To the normal person on the street, this would not seem to matter, but in reality outages happen pretty often on large networks. The design of the networks today is often dictated by the limits of the link-state databases and the CPUs that power them - the bigger the database, the more CPU power it takes to run the algorithm to calculate the new best path when there is any change.
As the link-state database contains a list of every link in the network and the cost of crossing it, as well as every other link that it is connected to - you can see how that quickly becomes computationally heavy on a large network. Also, the larger the network, the more likely it is that you have some change in the network at any one moment - just probability working against you.
However, this adds a little bit of overhead into the actual database, and will make troubleshooting problems more complex in the real world. The implementation is a really key part of this, and will probably actually dictate if this gets picked up and used going forward. There is a real need for faster convergence time on modern networks as we move into more internet video feeds, as the loss of even a few packets can deeply impact the user experience - so the interest will be there, but the devil is always in the details.
After reading a portion of the PDF supplied, I am actually far more satisfied that this is new research and not a restatement of fundamental network principles. The PDF has the equations where he proves a few simple criteria can define the scope of any network topology changes based on the difference in cost of the new route. This allows you to limit the recalculation of routes, blocking them from most of the routers where the recalculation would have provided no change in the actual routing topology.
The challenges he states are real challenges, and many modern networks are defined by the limits of the link-state protocols. In essence, this is like auto-summarization of prefixes in bgp, only applied to links in the link-state database - a possible slight loss in accuracy for a large boost in routing performance. This would allow the (faster converging) link-state protocols to scale to larger networks, rather then having to divide them into areas or use BGP to route between different sections of the network (resulting in loss of convergence time).
To the end user, this would mean that the internet would respond faster to outages, and better route around congestion on any one link.
So, from reading the article, I see that the great leap forward here is "smaller routing domain in a link-state protocol leads to faster routing updates". But, looking at the existing link-state protocols, they were designed from the ground up with the ability to limit your routing domain manually so increase the convergence time and decrease memory footprint.
I guess that means the achievement here is to have a link-state protocol that automatically limits your routing domain by limiting propagation of routes. This however seems like it could lead to seriously suboptimal routing which is probably a bad idea in most network environments today.
And yes, when DOCSIS 3.0 gets here, it will be amazing. However, I would not expect widespread deployments before Q3, if not longer - the cable modems for DOCSIS 3.0 are still really new and the rest of the support is still immature. That is not to mention that you need to upgrade every modem on a segment before you can use many of those new QOS features, otherwise you cause the older modems to go offline. :\
Logistics is the enemy in a highly distributed environment, and slows everything down.
Still, even if you do put "limits" on your traffic for high priority, people will find those exact limits and ride them. This destroys the actual use of the high priority as eventually (when enough bit-torrent clients implement it with it on by default) all traffic is high priority and nothing else is.
What I am saying is: if there is a way for people to abuse the system, they will. It is the tragedy of the commons, over and over again.
While everything you said is absolutely correct for an IP, packet based environment, I would suggest that you read up on DOCSIS (and more generally TDM based transport protocols).
You cannot do "per packet" queuing in DOCSIS, because you are using a time divided protocol. The only way to prioritize bandwidth to an application in an shared TDM environment is to guarantee "slices" of the time to that particular modem that has an active flow for that application. Those slices are reserved for that node regardless of if they use it or not, so everybody else using that particular cable segment suffer.
It is not the upstream router segment where bandwidth contention generally occurs on a cable network. It is on the actual cable segments themselves, which are also the most expensive and hardest part of the network to upgrade. Even protocol upgrades (docsis 1.0 -> 1.1) are huge investments in money and equipment, because you have to upgrade every end user to a new cable modem if their current one does not support the new protocol.
Also, last monkey wrench to throw at you:
If you trust what a specific application is sending to give it higher priority, what is to say that another application will not use that precedence value even when it is not that type of service? It is trivial for me to mark all bittorrent traffic leaving my home router as precedence type EF!
Note this line: Comcast has received a notification by a copyright owner, or its authorized agent, reporting an alleged infringement of one or more copyrighted works made on or over Comcast's High-Speed Internet service (the 'Service'). Comcast did not wiretap you, or generate this themselves. They got a legal notice from somebody else that this happened and passed it on to you. If you will note that they did not even give your physical address and information back to the company that filed the complaint for infringement. This is also a notification that a subpoena may be filed against Comcast to retrieve this information as part of the standard discovery process in a copyright infringement case. That is at the discretion of the company that filed the initial complaint, however.
Yeah, Powerboost is a gimmick as far as I can tell - it gives you a few seconds of very fast downloads. But for the majority of users, this is perfect - those pictures come down really fast from your email, or that youtube video you were linked loads up in moments, etc.
As for erratic, you are going cross country with most of those results. I would expect a good amount of erratic behavior, especially since you are also dealing with whatever load the speakeasy server has at that moment. The general trend of the results you posted was : the higher the ping time, the slower your maximum speed (Upload was affected far less, which really tells the tale). It is what I would expect, and the noticeable outlier of Tuscon is probably more of a site load issue when the test did the ping. The speed of light is fairly immutable in these matters!
The slower speeds to the west coast sites (Phoenix, San Fran, Mexico City) are all explained by TCP window issues.
Remember, the higher your round-trip time the slower TCP will let you send bandwidth unless you tune the TCP window upwards. For reference: http://en.wikipedia.org/wiki/TCP_Tuning
This happens with every ISP, and quite frankly when you start talking cross country distances the speed of light becomes a factor in how fast you can actually get round trip time down to.
The higher your latency to a site (aka, how long does it actually take a packet to get there) the slower your bandwidth will be without some extensive TCP window tuning. Reference:
http://en.wikipedia.org/wiki/TCP_Tuning
That is why you get slower speeds the farther out you go, even assuming every link in between is not congested.
Also, most of what you post there is explained by powerboost - very fast downloads for the first few seconds, then they throttle it back. If you ran a sustained download (go get a linux distro!) you could see what the actual sustained download speeds are after about 5 minutes.
The Comcast terms of service actually state that you can have your service terminated for something to the effect of "impairing the service for others". From postings on the internet, this typically means 150-250gb+/month these days. This also does not happen in every case, so I suspect that they may not even look until people in your area are complaining about slow speeds during peak times.
Also do not confuse this with the letters that get sent due to RIAA/MPAA action, or the various remedies that cable companies have for hacking your modem's bootfile and upping your bandwidth.
As for your speed test server question, do not forget that distance to their server, load on their server, your default TCP window size, your network configuration, and many other factors can relate to how much bandwidth you get to a single speedtest site. Try more then one, and try some that are fairly close to you (geographically or via traceroute).
The problem with your math? Your "backbone" numbers only consider that highest .05% of users for the "2100" T1s.
Try figuring your numbers for the 11.5million customers on a bell curve of bandwidth usage with a peak at around 2gb/month (based on other posts). You will come up with numbers significantly higher then the "2100" T1s for your backbone - and thats not even considering that bandwidth usage is not steady, but (like any ISP) has peak and off-peak times. To have a non-congested circuit, you have to plan for "peak" latency.
Then, finally consider that the real bandwidth congestion that cable companies care about is the last mile - the shared bandwidth from the nodes. This can quickly become over congested because it is typically a 45mb/shared (downstream) and 8meg shared (upstream) connection on the cable plant using TDM (which means it is not even optimized sharing).
Not saying that this is a perfect situation, but the numbers you throw out there are misnomers when you actually scale things to real customer numbers and start adding in all the (possibly multi-use) infrastructure that lies between the ISP circuits and the customers.
Really, nothing you listed there should add all the way up to 287gb unless you have some massive files from your clients.
xbox hard drive is 15gb, so thats 30gb even assuming you filled it twice.
Linux ISOs? Let us assume the worst, 5x5gb - 25gb
MSDN ISOs? Just guessing, but I cannot figure thats going to be more then 20gb
Tivocast programs + extra youtube plus etc should not be higher then 40gb for the month (figure 120mb each clip, about).
So without your client backups, we are looking at what, 115gb about? Did you transfer 150gb of client data in a month?
Your math is off by an order of magititude atleast, in any modern installation. First, it would take longer then that to pull cable down a 5 mile road, unless they were doing it as the road was built. Digging holes, pushing pipe (conduit), and throwing cable through that. Now, your talking atleast 100000 just for the cable runs, not including the fiber from the upstream node back to the headend, or the conversion equipment, or repeaters. So, 200000 easy at that point (you would be suprised at what that equipment costs!) at this point, before we even talk about head end equipment!
:
For HSD, a CMTS that handles 3000-5000 people runs you right around $100000 by itself, and then you have the upstream network equipment (if you go cheap, $5000 router, running over preexisting fiber). Now, as thats about the cheapest CMTS you can buy, its what we use to support this whole rural area - your 238 homes + every other run they are going to do in the future. But the local cable board required they do high speed data in order to get the franchise, so they install it. However, to keep the data neat, we will "charge" 1/10th of the price off to those 238 homes. This is BEFORE video equipment - and thats 20000-60000 for analog equipment, and another 60000-80000 for digital equipment. Again, this is for an entire area, so we will prorate this down to 1/10th of the price.
So, more realisitic figures are
Running cable for 5 miles : 200000 and 2-5 months, depending on terrain.
HSD service : 10500 (prorated for customers)
Video : anywhere from 8000 to 14000, prorated.
That is 220,000 right there.
And this is before you hire local techs to take care of that area, and assumes you have a preexisting head end with preexisting fiber to it. This does not include all the first 6 month promotions that you run into, or extra equipment costs, or costs to keep contracts for quick replacement of any failed equipment / spare equipment on site. (Video normally has spares - if the cable goes out on superbowl sunday, people get fired)
And I am sure I am underpricing some things. It is NEVER cheap to dig cable through the ground, especally not through residental areas. And these are (prorated!) upfront costs, so if you do not get the take-up you expect, your locked in to a 5-10 year contract with the city with equipment that costs more to run and maintain then profit you get out of the area.
Per the anrand tech review linked in the article, a 2100FP with an analog cable running at 1600x1200 will have very laggy displays. Moving to a digital (DVI) cable resolves this. If you do not have a DVI out on your PC, there are quite a few cheap cards today that have DVI out - typically your geforce 4 line of cards.