Slashdot Mirror
ISPs And Router Security
IPvNOT asks: "With all the script kiddies and distributed denial of service tools that spring up each week, there is an increasing use of spoofed network addresses. It would seem logical to me, to help control some of the problem, for ISP's to install a simple access control list (ACL) entry that blocks all packets that do not contain an address within their 'internal' network. How hard would this be to implement on a large scale? Would ISPs implement this?" I would think that an ISP would be able to block and drop anything they receive from the outside if its IP address starts with '192.168.', '127.' or '10.', and there are several others that can be screened for -- are there reasons that ISPs don't do this?
But if you start applying further checks to every packet then the network will surely start to slow and in such a competitive world isps dont want that.
In addition even if 90% of isps can be persuaded to implement it, there are enough that will disregard it and the attacks will surely continue.
They should block far more than that. They have an IP range. Anything going out with a source that's not in the IP range should be blocked, imho.
Many ISP's see ACL's as processing overhead on their border routers. Most of this may be due to badly designed ACL's. Reserved addresses (such as the 10.x.x.x,...,192.168.x.x and any hosts from within the network should be blocked ( it is unlikely that traffic coming into your network should have an ip from within your network as it's source ip) We had some high processor utilisation on our border router and after looking at our ACL's, have reduced the utiliztion by 30%. Quality not Quantity is the key.
That most ISP's don't own all of their network. They purchase a majority of their dialup network access from outside providers like UUNet, Gridnet, Level3, etc. These providers can be prone to changing ip addresses frequently.
Additionally most ISP's want their resources to be widely available. If you can't get your mail from both home and work because the ISP blocks access from your work ip address (on a frame-relay or dsl connection) you will most likely not want to continue doing business with that ISP.
Most routers do have ACL's that limit telnet access to certain ip's or subnets. Only a poorly configured router would allow telnet access to the router from any ip address.
-josh
At least one of the ISP I worked for rejected packets with private or internal addresses as their sources coming from outside on the border routers.
A good citizen ISP should even filter its dialup pool to reject forgeries from its customers. It's called ingressfiltering and is the most reliable way of getting rid of those smurf and other various DoS attacks.
Many networks already do filter RFC1918 packets on their border routers. An interestingly heated discussion on the pros and cons of this is to be found on the nanog list. first message of (LONG!) thread
Camaron de la Isla 'When I sing with pleasure, my
"None are more hopelessly enslaved than those who falsely believe they are free." -- Goethe
A second alternative is to have a check on the modem/ISDN/DSL lines, to ensure that the address on the packet is equal to the one assigned the device on that line.
Last, but not least, for only a little extra complexity, use authentication, such as IPSEC or SKIP.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
It's just as easy to block DDOS (smurf) attacks both if you are the target or an unknowing accomplice.
The problem is that ISPs don't want to slow down their connection by a few bits/second, and they don't want to do that. Slow connections mean fewer customers which means less money.
-- Ever notice that fast-burning fuse looks exactly the same as slow-burning fuse? I didn't... (Edgar Montrose)
and 155 mbps isn't the fastest there is. my ex-employer had a 655Mbps connection (they are an isp, among other things). i don't think any pc could saturate that connection without _very_ fancy extra hardware.
route 10.0.0.0 0.0.0.255 nul0
Create a nul interface (many routers support this) and update your routing tables to route RFC1918 into the nul interface. Easier and better than using an ACL.
Obviously, this is not always feasible. Many cable/DSL ISPs use RFC 1918 addresses for the Cable/DSL modem devices (there is DHCP option 82 to distinguish whether the request came from the customer's PC or their cable modem). So, you'd have to let the traffic get all the way to your border router before stopping it.
What sort of security *is* important on a router? I would've thought that a router would be relatively benign, but can't (once compromised) it provide a gateway to ip spoofing, source and destination address spoofing, packet sniffing, and ip source routing? (and denial of service attacks, of course, we won't forget those!) Why else secure a router if you've got a secure firewall?
I deal with security incidents all day. Every admin that I talk to that has been rooted or is being used as an amp start always say: "I should have..." And, they are right, they should have, but they didn't. Admins need to have some plan for security.
Quick and dirty list of security tips:
No replies made to AC posts. Please log in.
Unfortunately a lot of ISP network roll-outs are done by people with very little IP network experience, or by "high paid" consultants.
The people without network experience are somewhat excused, but they should have gotten someone to look it over, and actually try some sort of penetration tests. They'd probably find a lot more wrong with it than routing non-routable IP's.
The consultants don't tend to bother unless asked, as it adds to their already high workload, plus they most likely think "I'm not getting paid enough to do that as well!". Some even just assume that the routers won't even route this sort of traffic unless told to.
A lot of routers don't help in this situation either. The training courses and/or materials for setting them up in many cases are rather badly written, don't cover a huge number of setup scenarios, and usually don't even bother to bring up these sorts of things at all.
On top of all this, you get things like the Managing Director of the company connecting a modem up to his PC and dialling out to his home account cos the connection to the net through the filewall is too slow (which is actually cos someone is trying to launch a DOS attack on your firewall), and then someone gets to his local files cos of some piece of software that he shouldn't have on the laptop in the first place.
Regardless, even if they did block the non-routable IP's, you should still "trust no one" and block whatever you can. If it's connected to the outside world, then there is a possibility that somehow, you could lose out.
The only way to truely protect your data is to grind up your hard drive into powder, magnetize it all, then heat it into a liquid. Cool and grind it up again, scatter it into the wind, and just HOPE entropy does the rest.
Nothing wrong with a little paranoia, but your statement is just plain illogical.
-pf
Make affiliate bucks
One of the problems I see with your theory of blocking out all private addresses (for those of you keeping score at home, that's 10.x.x.x, 172.16.x.x through 172.31.x.x, and 192.168.x.x) is that you won't see these addresses trying to come in through your firewall (if you've built it right) as often as you'd think. With Network Address Translation (NAT) and Port Address Translation (PAT), you'll see the public IP addresses of these hosts being attacked more often than not. The ability of any given script kiddie to modify the TCP header will complicate this, but without prior knowledge of your network it's a hit and miss attack. For all they know, you could be running any combination of subnetted private and/or public IP addressing schemes behind your firewall.
The best defense against these attacks is a good ACL on a solid firewall platform. Block incoming traffic from private addresses, but be sensible and put internet accessible hosts on a DMZ. For general internet use, select one public IP address from your pool of public IP addresses and use PAT.
The info above was typed out haphazardly and may not make sense, but after working with Internet security in a myriad of environments, the best advice I can give you is that if even if you THINK you understand it, you should still seek consult (the more eyes the better). I would go into more detail, but unfortunately I have to go set up a firewall for a client :)
-- Stu
/. ID under 2,000. I feel old now.
However, I would say the biggest reason is that this kind of filtering can eat a *lot* of processor cycles on your router. These routers are not cheap, so you tend to buy the least you can get away with.
Also, the packet by packet comparison necessarily slows down traffic. Not a big deal for a small line, but when you're dealing with multiple OC-3's, people get a little antsy about a 10% performance decrease.
--
-- Slashdot sucks.
I worked at a major internet provider for over 2 years, and when I left I was Senior Network Engineer, with only the head of the engineering department above me, and above him was the CTO. We had over a dozen POPs (Points of Presence), and OC3 lines strung from MAE East to MAE West and many points between, and OC12's being installed. So, let's assume I can speak slightly to this issue.
With a major provider, your hardware is going to be big enough (BFR, GRF, etc) to handle 60,000+ routes AND do adequate security filtering. Don't accept the RFC'd routes in, and don't propogate them. Period. Don't accept internal routes from external sources. These are simple rules any major provider *can* handle if they can handle a full routing table. We're talking edge routers.
Smaller providers who are multi-homed and that lease dialups wholesale are a problem, though. Their dialups have IPs that don't belong to them. They often don't have the expertise to configure their ACLs correctly, and leave gaping holes in their security. Sometimes we'd scan our customers' routers with SNMP probes and find a lot of default SNMP passwords for read *AND* write access to their router, and we'd let them know to button up their router. One of our routers would occasionally get flooded with extra routes from a customer (we had lousy filtering) and the resulting flood of traffic would kill the customer's router. The first sign of this would be the customer's line going down. We were understaffed and used several different kinds of routers, so ACL's varied slightly between platforms because of the way they had to be written.
My point is that you need three things for merely minimal security (just by IP blocks):
Hardware: a router with enough CPU and enough RAM
Expertise: engineers that know how to write ACLs for the IOS you're using
Priority: your engineers have to have the time to actually sit down and get the ACLs updated on all the routers correctly
Unfortunately, I don't think there are many providers that have all of these.
Having worked at a number of ISP's and talking to those in the trade I've found that it's a matter of the SysAdmin. In general ISP's that are Linux or BSD based are far more likley to have the router set up this way already. ISP's that are microsoft based (And there are many) tend to shy away from touching the router more than they need to.
From experience, on the edge of the network, there is NO reason for a packet to come into the network that is not part of your address space. Edge is defined as a single-homed connection with no transit capability. We have a packet filter on our edge routers as well as our core multi-homed router to deny traffic with a source address that doesn't match one of our class-C's.
The problem with doing this on a Cisco is that it requires the CPU to observe the header of each packet going out, rather than have the interface DMA the packet to the destination interface. During the last big round of DDOS attacks, (January-February) we say many ISPs try to put filters in their core routers. The result was a 4x+ increase in CPU usage in the routers, and a router crash in a lot of cases.
We saw BGP traffic increase by over 10x as these routers came up and down all across the Internet. We have our core router set up to log faked ingress packets, and you wouldn't believe how many packets we see. Also be aware, it's not always a DDOS attack that causes spoofed packets. We see misconfigured windows boxes leak the Microsoft Ethernet addresses out PPP, misconfigured firewalls leaking internal addresses, etc. We see no issues with filtering these packets since there isn't a way for those packets to get back to us anyway, and it takes up more of our outgoing bandwidth...
Best bet is to filter as close to the edge as you can. For companies that sell bulk-dialup, their access servers can be configured to filter packets not on their address pools. The routers serving those modem pools could filter on the addreses for that data-center. Cable providers could filter based on the IP addresses assigned to that cable head-end. If we can filter right up to the edge of the transit-network, DDOS should be a thing of the past....
Interestingly enough, I have received legitimate packets across the internet from IPs in the range of 192.168.*.*, 172.16.*.* - 172.31.*.*, and 10.*.*.*. How? Traceroute!
It seems that some networks (including @home) are too cheap to assign an external IP to all of their routers, so when traffic goes through their network, the routers routing it have internal IP addresses (which are not noticed unless you do a traceroute or just ping with a low TTL set).
Example: try tracert'ing a few IPs in the 24.112.51.* (@home) range.
-- Sig, 120 chars --
Your friendly neighborhood mIRC scripter.
if (ismoderator(reader)) hidecomment(this);
* Q
P.S. If you don't get this note, let me know and I'll write you another.
I worked at a regional ISP until recently, and we did block bad IP's in the routers, along with source-routed packets. For example, RFC1718 addresses (10.0.0.0/8, 17.16.0.0/12, 198.168.0.0/16) are never routed to/from the Internet, and the IP block we owned would never be accepted as a source address from the Internet. Some ISP's do make the effort. (I'm not sure if downstream customer networks were blocked from spoofing each other's addresses, but I believe they were.)
Now, the question in my mind is whether the effort we made (to secure the networks from spoofing attacks) was typical or atypical of reasonably-sized ISP's? (When I started at this ISP, they had two T1's to one backbone provider; now they have four DS3's to four different backbone providers...)
Deven
"Simple things should be simple, and complex things should be possible." - Alan Kay
The original question (without Cliff's comments) is actually why ISPs don't ensure that outgoing packets from their internal networks are indeed sourced from their IP range.
For example, if I'm an ISP who has (to use a popular range ...) 24.226.0.0/255.255.0.0 , I would add a firewall entry that outgoing packets are denied if they are not from 24.226.0.0/16 (same netmask as above, incidentally).
For multiple source IP ranges (including private network addresses), this can be extended by making an ipchains output filter to allow packets going into the network (they've already passed the input and forward filters) and packets sourced from your IP ranges, then deny'ing the rest. Don't forget to add IP spoofing filters to your input chain.
Final note: I actually do administer a network and am doing this. I've turned on logging on those spoofed packets of course, to see if I'm denying anything that shouldn't be, but it works quite well.
- Michael T. Babcock (Yes, I blog)
This just came up on the NANOG mailing list.
The conclusion was that RFC1918 suggests that ISP's prevent reserved addresses from being forwarded, but does not require filtering.
"should take measures" != "MUST NOT"
Also, any packet based ACL is simply impractical on core routers, except for the very fastest (like Juniper M40). Sure, they use ACL's temporarally to fix/debug problems, but leaving them there full time would degrade performance unacceptably.
There's one thing that large ISPs simply cannot do, and that's to anti-spoof every dialup client connection separately, because that just doesn't scale well as dynamic dialpools grow in size -- that's just too many ACLs for current-day routers.
Consequently, at best they'll anti-spoof aggregated blocks, but even that can be impossible to do when the IP address allocation is fully dynamic as it has to be for really big ISPs, especially those that are virtualized into more than one branded product that must operate in disjoint dialpool spaces but using the same dialin hardware. And even where aggregated anti-spoofing is posssible, it does not provide full accountability --- you can nail the DoS to an ISP, but that's all, which doesn't help when DDoS attacks are synchronized to appear from multiple ISPs simultaneously.
As a result, until the NAS/RAS/HomeGateways etc do their own per-connection anti-spoofing, there will always be opportunity for attackers to randomize their source addresses to some degree. The problem at the moment is that there appears to be virtually no anti-spoofing on dialpools at all in practice, and that's real bad.
"The question of whether machines can think is no more interesting than [] whether submarines can swim" - Dijkstra
-pf
Make affiliate bucks
Many of the engineers who run the biggest networks work very closely with Cisco to develop, implement and document new features in IOS. Here is a Cisco document which explains how an ISP should implement these features:
http://www.cisco.com/warp/public/707/EssentialIO Sfeatures_pdf.zip
All opinions expressed are mine, if you want them it'll cost you.
However, your subnet mask (as is the correction you posted) is goofy, although it makes a nice wildcard mask. Let's give the Cisco kids out there some useful syntax that they can cut-and-paste into their routers (as long as they're in privileged exec/enable mode):
ip route 10.0.0.0 255.0.0.0 null0
ip route 127.0.0.0 255.0.0.0 null0
ip route 172.16.0.0 255.240.0.0 null0
ip route 192.168.0.0 255.255.0.0 null0
The point is that if every ISP filtered outgoing packets to make sure that they came from legitimate IP addresses, while it may not stop DOSes, it would at least make it a LOT easier to shut down the offending sites since they couldn't lie about where they were coming from. Currently when you get DOSes, it requires a hellish amount of effort from the backbone providers if you want to try to figure out where the (spoofed) packets are coming from.
Well, there is a reason to allow external IP's from 'inside'. For example, I am thinking of connecting my neighbors DSL to my own network (which already has DSL from a different provider), and having a linux router send the outgoing packets out in a round-robin fashion. This should effectively double our upload speed. If his ISP filtered for non-local IP's, this wouldn't work.
There is a way to do this without burning up all the CPU. In an access-list, the more netblocks you have, the more filter rules you have, and each one bogs down the CPU more an more. The correct solution is different than an access-list. But it needs to be coded directly into IOS and the interface processors.
The logic for this borrows from routing. When a packet arrives on an interface, it will eventually be routed by means of looking up the destination address in the routing tables (and route-maps where configured). What needs to be done first is to use the source address of the packet and do a lookup in the route table. But instead of selecting the best route (lowest metric) just compare the arriving interface with the interface of any valid outgoing route that would be used if the source were a destination. If it matches up, let the packet go on. If not, drop it on the floor.
Once implemented, it would be just a matter of turning it on. Actually, this feature should be on by default. Someone once told me that this feature did exist, but they couldn't say what the command for it was.
now we need to go OSS in diesel cars
Just permit your own IP blocks as source address for outgoing packets, and deny everything else. If you have too many netblocks to do that with, then you need to get aggregated. If you don't, then you're part of the problem (e.g. flooding BGP4 prefixes).
now we need to go OSS in diesel cars
I guess it must depend on the ratio of good packets to spoofed ones. If you only have to spend a few cycles to throw out a useless packet, that is a packet that isn't sucking bandwidth. The complication is, you don't just check that packet, you have to check them all. So,
1)at what point does the slow-down from checking packet headers meet the slow-down from transmitting spoofed packets?
2)Is there any way to do an imperical study to track some statistics?
Aah, change is good. -- Rafiki
Yeah, but it ain't easy. -- Simba
But, I fail to understand the problem. Private IP addresses don't belong on the Internet, right?
So, if they don't belong there, they're there either because of an error (ie. plugged Internet connection into the wrong ethernet card on my Linux firewall/router) or because of a malicious attempt to take advantage of a potential security hole.
In either case, there's no legitimate reason why you'd want 192.168.x.x to be on the greater 'net, right?
I'm still a Linux newbie, and I'm still new to the power and responsibility my Linux firewall/router gives me that Windows doesn't. But filtering out all external packets claiming to be from my internal LAN was one of the very first things I did in implementing my firewall; even if I only copied the scripts off a How-To and modified them for my needs.
Speaking of which, how do I set up hosts.deny and hosts.allow to allow all telnet (bad, I know, but I'll never log in as root remotely; I promise) and http requests to my webserver into my LAN? <grin>
Fire and Meat. Yummy.
Yup, I'm quite impressed.
I run a Dell P-133 machine, using an Allied Telesyn AT-1500 on my local LAN and a generic NE-2000 PCI card on my external connection.
With my DSL setup, I have to run PPPoE, so I've got Roaring Penguin's solution, and a fairly good firewall routine.
When I fire up top and then start downloading big files on any one of my Windows clients, I never show more than about 2% CPU useage on the Dell, and most of that is running top.
I've saturated my 10 Mbps Intranet a few times, mostly parking a few video files on the Linux server, and then playing them from the 4 Windows clients scattered throughout my house. Even so, the Dell never peaked out over 5% CPU useage. As long as I stay away from X, there's loads of power and memory for ipchains to do its thing.
of course I am running nice DEC tulip chips and no icky 3com junk, or realtek crud. It's been a while but I believe I could get around 60mbps, maybe higher, but this was also only from one source to one destination.AT-1500s are bus-mastering ISA cards; I wish I could get two of them to run in that same machine, but I've never been able to get two ISA network adapters to run in Red Hat 6.0. Even though they run fine together under Windows 95, I get an "Ieee! Killing interrupt handler!" message when I try to bring up eth1 with the PPPoE driver.
I like the AT-1500s because they've got great LBL factor. ("LBL" = "Little Blinking LED")
LEDs for Link, Collision, TX and RX. Very pretty when my roommates are online.
Anyone know of any good PCI network cards with lots of LEDs? (Scary that it's a prime purchasing criteria, isn't it?)
Fire and Meat. Yummy.
Even many DSL systems use PPP (wierd, but true; newer consumer DSL is IP over PPP over Ethernet over ATM over DSL.) So similar enforcement is permissable there. If you have several IP addresses on the line, PPP has to know it, though, which currently isn't well-supported. But that's an extra-cost option with most DSL vendors anyway.
PPP is CPU-intensive enough that checking the IP address in the PPP server shouldn't be noticed. It's probably better to handle this before the first router than trying to maintain validity tables in the router. Attackers locked down to a range of IP addresses can still use one belonging to someone else as long as it's in the valid block. So the DoS kiddies can still attack; they just have to change their strategy a bit.
Real LANs with WAN connections are another matter, as are the dumber flavors of cable modems.
That's why I won't work for Intel, if they treat consultants that way.
This is all dealt with in RFC 2267: Network Ingress Filtering: Defeating Denial of Service Attacks which employ IP Source Address Spoofing
\w0zz - OpenBSD - A Better Solution
It's simple, really. As a matter of security; Whenever a router is aware of all address space on one port, it should block inappropriate traffic; Core network routers cannot do this. ISP routers should all be able to do this.
What do you mean would it be called the internet? If you can route your assigned IP, it's internet.
Why should you be able to forge packets?
Could filter out all ICMP (ping, etc.) packets as part of your firewalling ruleset, but that will have other consequences, since ICMP is important to help negotiate a TCP connection when something goes wacky somewhere.
Just make the firewall ruleset watch for fragmented ICMP packets, as well as misformed ICMP packets (ie. the "ping of death").
A floodping just eats bandwidth, sometimes overloading and crashing routers and stuff along the way. Yank your internet connection to make damned sure this doesn't hit the internet, then from a Linux box, type "ping -f 192.168.x.x &" several times, where the x.x is the rest of the local IP address of a Windows box. Watch your ethernet lights, and the collision light on your hub. When I do this, my Windows 95B box slows right down, but it doesn't crash.
A DoS attack usually involves sending a floodping of malformed or fragmented packets, meaning that the system at the other end either experiences buffer over-runs (which, depending on how well the operating system protects memory, may or may not corrupt something else in RAM) or retries in vain to get the other part of the fragmented packets that the other system is sending.
At some point, the buffers either become full or start to over-write their boundaries. The blue screen of death is only around the corner.
BTW, don't take this as gospel, this is my understanding of what happens from burning the candle at both ends as I learn about network security in preparation for administering a webserver at the office.
Fire and Meat. Yummy.
This is known as reverse path filtering - Linux 2.2 and Cisco IOS with CEF (Cisco Express Forwarding) have this. On suitable Cisco routers the overhead of RPF is minimal, due to the way CEF works. However, it doesn't work with multi-homed customers, amongst other cases.
Thank you all for responding to that.
I'll certainly check out those, and the other suggestion for the D-Link DE-530.
Not that this has any sort of real relevant use, but my firewall/proxy has a free 5.25" bay, which I'm going to use to mount a hard disk drive. Since it's a 3.5" drive, I'll be using one of those adaptors, and it's got a faceplate.
I've already laid plans to drill a whole bunch of holes into it and have labelled LEDs for eth0 and eth1. Again, not that it's got any practical use, except that it'll make my gateway look really impressive sitting under my fax machine with a zillion blinking lights on it.
Fire and Meat. Yummy.