True. And 90% of the barrier is the "last mile". Stringing fiber, or copper, or building radio towers is an expensive, time consuming process. Unless you want the place to start looking like parts of South America, you really cannot have 37 different fiber trunks hanging from the utility poles. Plus, the guy who got there first, is going to do everything imaginable to make sure he's the only one.
In the end, we have TWC writing legislation preventing local government from building their own network -- as we have in NC. Verizon being granted a common-carrier exception for their fiber deployment so they don't have to obey any open access rules -- and the first thing they do when installing fiber... rip out the "open access" copper infrastructure, thus removing any possibility of a competitor.
Not entirely. While there have been no "franchise" agreements for wireless, it's the same sort of scarcity. And spectrum licenses are insanely expensive. What we find are companies with very deep pockets buying up vast swaths of spectrum and sitting on it, thus preventing anyone else from competing with them. (while crying about a "spectrum crunch" that doesn't actually exist.)
Really? As if Firefox hasn't done the same stupid shit? (granted, it's easier to rollback)
Safari? Seriously??? Unless you're on a Mac (or just like bloat), you don't run Safari.
Opera. Holy fuck, man. Those fools making sweeping UI changes in POINT RELEASES! It's the reason I've stopped using (mostly) and absolutely DO NOT UPDATE opera. (it's still my RSS reader, because I'm very lazy.)
This isn't entirely a peering or transit dispute. At the end of the day, the traffic going into verizon's network BELONGS THERE. It's traffic destined to verizon customers, that those customers specifically asked for. As a residential, consumer, broadband ISP, it's your job to keep your links from being saturated and greatly oversubscribed... or it used to be. With next to no competition, verizon knows they can use those eyeballs as leverage to make content providers (or their ISP(s)) pay through the nose to get to them. If people had an option, they'd take their money elsewhere -- but they don't. (even if you're blessed with multiple broadband options, you'll find everyone is playing their greed card.)
Let's make the (purely hypothetical) math simpler... I'm going to setup an ISP offering dialup. I buy two 2 T1's: one data, one voice. That'll work out fine as the 1.5mbps data line will easily handle 24 analog modems (24 x 40-48kbps = 960k to 1.2m) But I've sold far more than 24 accounts, so I have to add more voice lines... still one data T1, 7 voice T1's. Now we have a problem: 1.5mbps is just not going to cut it. So who should be paying for more infrastructure, "kittenpix" as the service my customers are trying to access? Or me, the idiot who's oversold the network by a factor of 4? Obviously, I'd prefer "kittenpix" buy connectivity from me to directly reach my customers, thus paying *ME* to get to my customers.
Back in the era of dialup, no one thought like that because there were many operators, and it was fairly simple (and cheap) to start a new one. That sort of thinking would drive away customers, and fuel the creation of additional ISPs. The modern broadband landscape is the exact opposite... very few players, using complex and expensive systems built on long standing protected monopolies -- land line phone, coax cable tv, and (rarely) power lines -- that they viciously protect.
It's a matter of keeping up with security notices. And as only one of them is an actual internet router (with as much turned off as possible), it's a minor vulnerability.
(Now, my stack of Cisco IOS based devices... there's a ball of exploitability. currently, ntp being the pain in the ass to reign in.)
a) It Works(tm). b) There are no published exploitable bugs in the builds I'm running. [and c) only one of them is actually connected as an internet router.]
They do have formal releases, they're just uselessly WAY out dated. Their development model is far too complex, and IMO, haphazard. To be fair, they are building software for thousands of devices, most of which they don't have at hand to test -- not that they have the lab resources to actually do that level of testing. OpenWRT does things a bit better, in that it's much closer to a typical linux distro where you can choose what to install; of course, that makes is a difficult system to work with.
Once you find something that (other people say) works... don't mess with it. I've not updated any of my WRT's in a long time.
Long ago, it was a fork of openwrt. Today it's almost nothing like it. And how the hell is ddwrt "proprietary"? The source is there; you can build it yourself. Yes, there are binary blobs in it due to manufacturers sitting on driver code. Yes, some platforms are commercial, but most aren't.
Someone has to be running BGP, but not necessary the customer... I've setup several customers that had PI space but didn't run BGP themselves. It's setup on the ISP side just like the ISP's RIR assigned space... announce it like you own it.:-) (they can multihome that, too, as long as everybody knows that's the plan. some ISPs get upset when they see "their" space being announced by someone else.)
The way I read that... your customers were multihomed and sending you traffic that belonged on another ISP's network? That's the definition of spoofing!:-) You're perfectly correct to drop that crap. If they aren't using address space you assigned them, or PI space they own -- and told you about, it's not your problem.
I've had several clients bring their own address space (PI) and a few even had their own ASN. In **VERY** rare cases, we'd allow a client to announce a non-portable assignment to another ISP -- we have to give written permission to that other ISP, and carve into the client's desk that when they leave, they don't get to keep that address space. (and v.v., but getting the other ISP's permission was always an exercise. customers are so stupid.)
Actually, you'd do it at the customer ingress point. So *your* ISP would say, "that's not your address, moron" and ignore the packet. It would never make it to your ISP, much less AT&T or Cogent. HOWEVER, there are too many ISPs that don't do that, so in your example, AT&T would have to validate the traffic from your ISP. And that's where it starts to fall apart as AT&T doesn't necessarily know with whom your ISP peers, or what their local preferences are; legitimate traffic from you could arrive at AT&T through any number of different paths AT&T may not expect. (Logically, an ISP should know what's behind every link, but that's one more thing you cannot count on.)
customer ingress validation; check the traffic as it comes from your customer. (i.e. drop packets as soon as possible, as close as possible to where it arrived.) often enough the router's builtin reverse path filter will do this for you, nearly processing free.
15-20 years ago, sure. Today, you're lucky if you can get anyone with half a clue at your own ISP to even "look into it." Chasing this rabbit more than one or two hops just doesn't happen. And if you could, the chase ends with ISPs that cannot be bothered to stop their customers from spoofing traffic. Maybe if you could get enough operators to disconnect these "lame" ISPs, but you're not going to because that'd be dropping paying customers.
(I don't want to count the number of people involved to cut off a World Famous, Well Known Spammer -- who was actively spamming, even hostmaster@isp (morons!))
And just how do we find these tens of thousands of zombies? They're spoofing traffic and their ISPs are allowing it. They aren't sending out gigabits of traffic so there's no abnormal amount of traffic to look for. (perhaps abnormal for a single, specific connection, but no one is applying heuristics individually to millions of connections. they look for the far easier to spot, road flare... a link that's 90+% utilized for several hours.)
The target for the DDoS is easy enough to spot -- it's the thing being knocked offline, after all. The amplification points -- dns previously, and ntp here -- are obvious enough to collect by sampling the hundreds of gigabits arriving at the target; in this case that's several thousand intermediates. From there, it gets "impossible". The actual sender of the request lied about who they are, filling that in with the DDoS target. All the ntp server operators can do is point at an interface on their router and say, "yep, came from ISP foo." ISP foo, if they can be bothered AT ALL to continue the chase, can only do the same thing. (odds are they (their clients) aren't the root source of the requests, certainly not all of them.) This global game of whack-a-mole will end with ISPs that just don't give a flying f*** -- which is the true source of the DDoS mess anyway... dumbass network operators that do nothing to prevent their customers from spoofing traffic.
INCORRECT. I see you've never experienced the joy of your IP address being reassigned when renewed -- dhcp renew is answered with NAK! Or you don't run services that are sensitive to address changes. (globally, I've found dynamic address to be fairly dynamic. on the local scale, *my* address doesn't change that often, but it does change.)
(per peer interface) "ip verify unicast source reachable-via any" This is the less desirable "loose" method, and doesn't work if you have a default route (with 0/0 matching everything.) It won't necessarily stop all spoofing, but it will significantly cut it down. "via rx" is always preferred, but in this case, each site may not prefer a given network through it's local connection, instead crossing an internal link to another site. (this is also asymmetric routing.)
Once again... the only way to completely end this crap is for every operator to take steps to prevent their own clients from lying about who they are. uRPF works over 99% of the time; the odd-man-out multihomed setups get a fully defined ACL.
Furthermore, each BGP peering session should itself be filtered to a list of allowable prefixes -- often managed in an automated fashion through RADB's. (every ISP I've dealt with filtered, as did I) That db can also be used to maintain ACLs. The address space I managed didn't change often enough for anyone (read: me) to automate it. The likes of UUNet and AT&T, their prefixes change constantly.
Incorrect. I'll say it again: It's not as complicated as the "haters" claim. I've maintained "BCP38" in an ISP network with transit links (aka. isp-edge routers with default routes.) While it's not perfect -- because "everything" is potentially on the other side, there are steps to be taken. (I ultimately cannot prove a packet with a source address of paypal actually came from them unless I'm directly peered with them.) You know what's inside your network (read: as the network operator, you d*** well better know), and thus, can prevent ingress traffic from what's already inside your network; and prevent everything inside your network from pretending to be someone else.
And that last bit is the key point. If everyone ensured their egress traffic isn't spoofed, these sorts of things would no longer be possible. Host A wouldn't be able to send packets to NTP servers with their source set to paypal (for example.)
(Also, there's two way to do transit... the way consumer do it -- a single default route -- and the way ISPs and big enterprise does it -- full route table from the upstream provider.)
When you're standin in front of the clerk? That's not going to happen. You have an arrest warrant; you get arrested. If this had come up at a traffic stop, MAYBE the officer would've let her go (I wouldn't bet on it, 'tho.)
In 2005, that video was "worth" ~$150 -- standard going rate for "not returned" tapes to every store I knew. (it's in the membership agreement, not that anyone read those things) HOWEVER, SC, apparently, has a g** d*** law against not returning a video tape.
Actually, the "power hungry fuck" was the video store owner.
Sounds like the statutes of limitation would have expired long ago, and the warrant was never recended. I'm not sure how this is going to proceed as there's no longer a plantiff.
As one who has maintained an ISP's peering, it is no where near as complicated as you make it sound. Enterprise class hardware (from Cisco, Juniper, etc.) have builtin support for unicast reverse path filtering (uRPF) that's effectively processing free -- based on the routing table ("FIB" -- forwarding information base) -- very effectively preventing traffic from entering (or leaving) your network that doesn't belong there.
(As an end user, uRPF presents a small problem as the ISP DHCP server is a 10-net host and I null route 10/8.)
Slashdot Mirror
True. And 90% of the barrier is the "last mile". Stringing fiber, or copper, or building radio towers is an expensive, time consuming process. Unless you want the place to start looking like parts of South America, you really cannot have 37 different fiber trunks hanging from the utility poles. Plus, the guy who got there first, is going to do everything imaginable to make sure he's the only one.
In the end, we have TWC writing legislation preventing local government from building their own network -- as we have in NC. Verizon being granted a common-carrier exception for their fiber deployment so they don't have to obey any open access rules -- and the first thing they do when installing fiber... rip out the "open access" copper infrastructure, thus removing any possibility of a competitor.
Not entirely. While there have been no "franchise" agreements for wireless, it's the same sort of scarcity. And spectrum licenses are insanely expensive. What we find are companies with very deep pockets buying up vast swaths of spectrum and sitting on it, thus preventing anyone else from competing with them. (while crying about a "spectrum crunch" that doesn't actually exist.)
Really? As if Firefox hasn't done the same stupid shit? (granted, it's easier to rollback)
Safari? Seriously??? Unless you're on a Mac (or just like bloat), you don't run Safari.
Opera. Holy fuck, man. Those fools making sweeping UI changes in POINT RELEASES! It's the reason I've stopped using (mostly) and absolutely DO NOT UPDATE opera. (it's still my RSS reader, because I'm very lazy.)
This isn't entirely a peering or transit dispute. At the end of the day, the traffic going into verizon's network BELONGS THERE . It's traffic destined to verizon customers, that those customers specifically asked for. As a residential, consumer, broadband ISP, it's your job to keep your links from being saturated and greatly oversubscribed... or it used to be. With next to no competition, verizon knows they can use those eyeballs as leverage to make content providers (or their ISP(s)) pay through the nose to get to them. If people had an option, they'd take their money elsewhere -- but they don't. (even if you're blessed with multiple broadband options, you'll find everyone is playing their greed card.)
Let's make the (purely hypothetical) math simpler... I'm going to setup an ISP offering dialup. I buy two 2 T1's: one data, one voice. That'll work out fine as the 1.5mbps data line will easily handle 24 analog modems (24 x 40-48kbps = 960k to 1.2m) But I've sold far more than 24 accounts, so I have to add more voice lines... still one data T1, 7 voice T1's. Now we have a problem: 1.5mbps is just not going to cut it. So who should be paying for more infrastructure, "kittenpix" as the service my customers are trying to access? Or me, the idiot who's oversold the network by a factor of 4? Obviously, I'd prefer "kittenpix" buy connectivity from me to directly reach my customers, thus paying *ME* to get to my customers.
Back in the era of dialup, no one thought like that because there were many operators, and it was fairly simple (and cheap) to start a new one. That sort of thinking would drive away customers, and fuel the creation of additional ISPs. The modern broadband landscape is the exact opposite... very few players, using complex and expensive systems built on long standing protected monopolies -- land line phone, coax cable tv, and (rarely) power lines -- that they viciously protect.
It's a matter of keeping up with security notices. And as only one of them is an actual internet router (with as much turned off as possible), it's a minor vulnerability.
(Now, my stack of Cisco IOS based devices... there's a ball of exploitability. currently, ntp being the pain in the ass to reign in.)
a) It Works(tm). b) There are no published exploitable bugs in the builds I'm running. [and c) only one of them is actually connected as an internet router.]
They do have formal releases, they're just uselessly WAY out dated. Their development model is far too complex, and IMO, haphazard. To be fair, they are building software for thousands of devices, most of which they don't have at hand to test -- not that they have the lab resources to actually do that level of testing. OpenWRT does things a bit better, in that it's much closer to a typical linux distro where you can choose what to install; of course, that makes is a difficult system to work with.
Once you find something that (other people say) works... don't mess with it. I've not updated any of my WRT's in a long time.
Long ago, it was a fork of openwrt. Today it's almost nothing like it. And how the hell is ddwrt "proprietary"? The source is there; you can build it yourself. Yes, there are binary blobs in it due to manufacturers sitting on driver code. Yes, some platforms are commercial, but most aren't.
Transit means you can hand me traffic destined to anywhere, not sourced from anywhere.
Someone has to be running BGP, but not necessary the customer... I've setup several customers that had PI space but didn't run BGP themselves. It's setup on the ISP side just like the ISP's RIR assigned space... announce it like you own it. :-) (they can multihome that, too, as long as everybody knows that's the plan. some ISPs get upset when they see "their" space being announced by someone else.)
"What??? I didn't assign you different address space. What the f*** are you doing?"
They broken their network. That's their problem.
The way I read that... your customers were multihomed and sending you traffic that belonged on another ISP's network? That's the definition of spoofing! :-) You're perfectly correct to drop that crap. If they aren't using address space you assigned them, or PI space they own -- and told you about, it's not your problem.
I've had several clients bring their own address space (PI) and a few even had their own ASN. In **VERY** rare cases, we'd allow a client to announce a non-portable assignment to another ISP -- we have to give written permission to that other ISP, and carve into the client's desk that when they leave, they don't get to keep that address space. (and v.v., but getting the other ISP's permission was always an exercise. customers are so stupid.)
Actually, you'd do it at the customer ingress point. So *your* ISP would say, "that's not your address, moron" and ignore the packet. It would never make it to your ISP, much less AT&T or Cogent. HOWEVER, there are too many ISPs that don't do that, so in your example, AT&T would have to validate the traffic from your ISP. And that's where it starts to fall apart as AT&T doesn't necessarily know with whom your ISP peers, or what their local preferences are; legitimate traffic from you could arrive at AT&T through any number of different paths AT&T may not expect. (Logically, an ISP should know what's behind every link, but that's one more thing you cannot count on.)
customer ingress validation; check the traffic as it comes from your customer. (i.e. drop packets as soon as possible, as close as possible to where it arrived.) often enough the router's builtin reverse path filter will do this for you, nearly processing free.
15-20 years ago, sure. Today, you're lucky if you can get anyone with half a clue at your own ISP to even "look into it." Chasing this rabbit more than one or two hops just doesn't happen. And if you could, the chase ends with ISPs that cannot be bothered to stop their customers from spoofing traffic. Maybe if you could get enough operators to disconnect these "lame" ISPs, but you're not going to because that'd be dropping paying customers.
(I don't want to count the number of people involved to cut off a World Famous, Well Known Spammer -- who was actively spamming, even hostmaster@isp (morons!))
And just how do we find these tens of thousands of zombies? They're spoofing traffic and their ISPs are allowing it. They aren't sending out gigabits of traffic so there's no abnormal amount of traffic to look for. (perhaps abnormal for a single, specific connection, but no one is applying heuristics individually to millions of connections. they look for the far easier to spot, road flare... a link that's 90+% utilized for several hours.)
The target for the DDoS is easy enough to spot -- it's the thing being knocked offline, after all. The amplification points -- dns previously, and ntp here -- are obvious enough to collect by sampling the hundreds of gigabits arriving at the target; in this case that's several thousand intermediates. From there, it gets "impossible". The actual sender of the request lied about who they are, filling that in with the DDoS target. All the ntp server operators can do is point at an interface on their router and say, "yep, came from ISP foo." ISP foo, if they can be bothered AT ALL to continue the chase, can only do the same thing. (odds are they (their clients) aren't the root source of the requests, certainly not all of them.) This global game of whack-a-mole will end with ISPs that just don't give a flying f*** -- which is the true source of the DDoS mess anyway... dumbass network operators that do nothing to prevent their customers from spoofing traffic.
Also, it won't do much to help. A decade or two ago, maybe. Today, there's no point even trying... move to IPv6 already.
INCORRECT. I see you've never experienced the joy of your IP address being reassigned when renewed -- dhcp renew is answered with NAK! Or you don't run services that are sensitive to address changes. (globally, I've found dynamic address to be fairly dynamic. on the local scale, *my* address doesn't change that often, but it does change.)
(per peer interface) "ip verify unicast source reachable-via any" This is the less desirable "loose" method, and doesn't work if you have a default route (with 0/0 matching everything.) It won't necessarily stop all spoofing, but it will significantly cut it down. "via rx" is always preferred, but in this case, each site may not prefer a given network through it's local connection, instead crossing an internal link to another site. (this is also asymmetric routing.)
Once again... the only way to completely end this crap is for every operator to take steps to prevent their own clients from lying about who they are. uRPF works over 99% of the time; the odd-man-out multihomed setups get a fully defined ACL.
Furthermore, each BGP peering session should itself be filtered to a list of allowable prefixes -- often managed in an automated fashion through RADB's. (every ISP I've dealt with filtered, as did I) That db can also be used to maintain ACLs. The address space I managed didn't change often enough for anyone (read: me) to automate it. The likes of UUNet and AT&T, their prefixes change constantly.
Incorrect. I'll say it again: It's not as complicated as the "haters" claim. I've maintained "BCP38" in an ISP network with transit links (aka. isp-edge routers with default routes.) While it's not perfect -- because "everything" is potentially on the other side, there are steps to be taken. (I ultimately cannot prove a packet with a source address of paypal actually came from them unless I'm directly peered with them.) You know what's inside your network (read: as the network operator, you d*** well better know), and thus, can prevent ingress traffic from what's already inside your network; and prevent everything inside your network from pretending to be someone else.
And that last bit is the key point. If everyone ensured their egress traffic isn't spoofed, these sorts of things would no longer be possible. Host A wouldn't be able to send packets to NTP servers with their source set to paypal (for example.)
(Also, there's two way to do transit... the way consumer do it -- a single default route -- and the way ISPs and big enterprise does it -- full route table from the upstream provider.)
When you're standin in front of the clerk? That's not going to happen. You have an arrest warrant; you get arrested. If this had come up at a traffic stop, MAYBE the officer would've let her go (I wouldn't bet on it, 'tho.)
In 2005, that video was "worth" ~$150 -- standard going rate for "not returned" tapes to every store I knew. (it's in the membership agreement, not that anyone read those things) HOWEVER, SC, apparently, has a g** d*** law against not returning a video tape.
Actually, the "power hungry fuck" was the video store owner.
Sounds like the statutes of limitation would have expired long ago, and the warrant was never recended. I'm not sure how this is going to proceed as there's no longer a plantiff.
As one who has maintained an ISP's peering, it is no where near as complicated as you make it sound. Enterprise class hardware (from Cisco, Juniper, etc.) have builtin support for unicast reverse path filtering (uRPF) that's effectively processing free -- based on the routing table ("FIB" -- forwarding information base) -- very effectively preventing traffic from entering (or leaving) your network that doesn't belong there.
(As an end user, uRPF presents a small problem as the ISP DHCP server is a 10-net host and I null route 10/8.)
CURRENT, low hanging fruit, root cause... take away windows(tm) and they'll target something else.