Slashdot Mirror
DDoS Larger Than the Spamhaus Attack Strikes US and Europe
mask.of.sanity writes "CloudFlare has been hit by what appears to be the world's largest denial of service attack, in an assault that exploits an emerging and frightening threat vector. The Network Time Protocol Reflection attack exploits a timing mechanism that underpins a way the Internet works to greatly amplify the power of what would otherwise be a small and ineffective assault. CloudFlare said the attack tipped 400Gbps, 100Gbps higher than the previous record DDoS attack which used DNS reflective amplification."
When you approach the business and say that a zombie network is DDossing the website with a Reflection attack, and that's why no-one can access the website.
Science advances one funeral at a time- Max Planck
The ISPs of the world keep letting this kind of crap happen.... It should be pretty obvious when someone is trying to DDoS a server. Even if they don't want to lose a "paying customer", simply cutting access to that server for x amount of time for that IP would be more than enough.
I went to firstlook.org this morning to see Glenn Greenwald's latest NSA story, and was surprised to first see a page from cloudflare claiming to be checking if I was a legit visitor. Could this be related? Have the spooks struck again?
So which one of them got on IRC and dissed someone's mamma?
Serious question. why are network providers allowing FORGED packets to leave their networks?
Laziness Syndrome.
PP: "Serious question. why are network providers allowing FORGED packets to leave their networks?"
Because: the Not My Problem syndrome.
They only care when lawyers come knocking re: piracy,
or customers get so clogged in a zombie state that they must call threatening to leave since total speeds feel slower.
The latter results in plan upsell opportunities for a pricier highspeed tier. Guess the ISP benefits from that last one huh?
It's not always laziness. I added outgoing filters to my routers so that it only allowed source addresses from my network. That was great at stopping DOS attacks, but as I found-out the hard way, several of my customers were sending outbound traffic with source addresses not on my network. That was in 1997. For the next several years, it was a huge hassle to keep adding additional source address ranges for customers. An ISP selling a high speed connection has to allow outgoing traffic from addresses they don't own. That's the entire point of selling transit.
For the next several years, it was a huge hassle to keep adding additional source address ranges for customers.
So you became lazy.
Carefully not clicking on any links - not reading any of the fine articles
I did have one site I normally visit (DPReview) get really slow, and then eventually go offline for a short time, I wonder if that was due to the attack.
At this point it seems like even massive attacks are not really doing much of a job in slowing down companies using something like CloudFlare or other distributed CDN's. I wonder how much longer it is before people will give up on DDOS attacks as ineffective.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
NSA and GCHQ when you need them?
good question.. this is like classifier rule #1(or damm near #1) at the ISP i work for. If its not in our block it doesn't leave.
I can see how that's a problem for U.S. subscribers.
And RPF would not work in your setup?
So you became lazy.
Because hiring people that can update cisco IOS configs are cheap and the updates are risk free. Also, customers are very understanding and patient when they can't send traffic after they change addresses. The GP is right that it is a huge hassle.
I would not call that lazy. Altruism only goes so far in the REAL world. If someone pays you for the effort, or legislation demands it of everybody, you will probably keep doing it. But if it only out of the goodness of my heart, eventually everybody will say fuck it.
So that's why there's unemployment. Lazy job creators refuse to hire people because it's a huge hassle.
Fuck the real world and the legislation it rode in on.
A better question is why are ISP's allowing forged traffic ENTER their network from end users? If they drop grandma's traffic that doesn't have grandma's srcip then grandma won't complain and the WWW would be a little safer. Of course their will always be end users who transit legit traffic.
As many as 4 $9/month residential connections in Tokyo were taken over in this remarkable exploit.
Why does it always take a team of tech to manually block the spamming IP numbers? Why isn't this automated? When this sort of flooding action takes place it should be pretty obvious... respond.
I've decided to stop wasting my time responding to AC trolls/sockpuppets... so if you want a response from me... login.
Our last inbound attack appeared to come from over 50 million very well spread out different IPs. Of course those are all spoofed IPs but either way you can't effectively block that many without blocking larger amounts of legit traffic.
Why does it always take a team of tech to manually block the spamming IP numbers? Why isn't this automated? When this sort of flooding action takes place it should be pretty obvious... respond.
Yeah, CloudFlare should learn how set up a Linux box with iptable in front of their server.. How hard can this be?
While I agree that it's almost impossible to implement those kind of filters in a complex ISP core network, it's also not the location where it should be done. You want to do this as close as possible near where it happens. So, the filtering of bogus sources should happen at the ISPs access routers and BRASes. The nice thing is, most ISPs fully automate the configuration of those devices, so it shouldn't be that hard to automate filters too.
Once this stuff seeps into the core network of an ISP, it is very hard to determine where it actually comes from and filtering it will become an even harder task.
But how many LOCs is that? Joking aside, I would have thought that nobody had to dumb down things that much before posting to Slashdot.
I've been a member of the NTP Hackers team for more than a decade, the mechanism that is being abused for these attacks is in fact a very useful debugging/monitoring facility:
You can ask an ntpd server about how many clients it has and how often each of them have been accessing the server. On old/stable ntpd versions this facility was accessed using a single pure UDP packet (ntpdc -c monlist), and in reply you got back information about up to 602 clients (the size of the monlist buffer), sent as a big burst of UPD packets.
Researchers have developed maps of the entire publicly accessible NTP networks using this facility, I have personally used it to map the status of our fairly big corporate network. I.e. it can be extremely useful!
A few years ago the development version of ntpd switched to a different protocol and method to query this information, using a nonce which meant that you can no longer spoof the source address: (ntpq -c mrulist). Since the mrulist buffer is configurable, I have setup my public ipv6 pool server (ntp2.tmsw.no [2001:16d8:ee97::1]) to keep monitoring info for the last 10K clients.
Today we recommend that you either upgrade to ntpd v2.4.7, or if you really cannot do this, insert a 'restrict default noquery' option in the ntp.conf configuration file. The 'noquery' indicates that clients can still use the server for regular time requests, but the monitoring facility is disabled.
Terje
"almost all programming can be viewed as an exercise in caching"
"Reflection attack exploits a timing mechanism that underpins a way the Internet works to greatly amplify the power of what would otherwise be a small and ineffective assault
..
I would have thought the DDOS attack were facilitated by all those compromised Microsoft Windows desktop computers out there on the Intertubes
IPS is sometimes great.
However, if an attack occurs from thousands of IP-adresses and with random requests.
Well, they fail and you fail.
DDOS is terrible to defend against.
DOS otoh is simple, just block the IP.
Interesting. What where they doing?
I had that attack in my network last week. It's not based on ip spoofing. It simply expoilts open ntpd servers and send them the payload which targets many more servers (reflection and amplification). I had to mitigate the attack by filtering ntp port just to few credible servers from pool.ntp.org.
The business was offering RESTful DDOS, as a service.
There has to be some sort of pattern. It can't be entirely random.
I've decided to stop wasting my time responding to AC trolls/sockpuppets... so if you want a response from me... login.
CloudFlare don't mention who was targeted. I think it was a music torrent site. I know this because I use them and they have been suffering denial of service attacks for a while. If they are the target I wonder who is paying for this DDoS attack - disgruntled user, or music industry?
Don't know if it can be stopped. We can still hope...
"I found-out the hard way, several of my customers were sending outbound traffic with source addresses not on my network."
You should lose those customers! Really.
No-one, I repeat no-one, has business sending packets with forged source addresses.
Refer them to a book on policy routing when they don't know how to route in a multihomed enviroment.
That is called ip spoofing. They send a request with a sender address of a victim, and the server sends the reply to the victim.
This would not be possible when the attacker's ISP would not allow source address spoofing.
This case mentions the use of NTP, but the idea of reflection attacks by now has propagated to TCP as well, even without amplification it seems worthwile.
Right now an attack is running on many webservers that sends SYN packets with source port 80 and 443 and destination port 80 from spoofed source address.
Apparently they want to overwhelm the victim with SYN ACK packets from reflectors.
However, those are the same size as the SYN packets sent by the attackers. Probably no issue, those attacks are likely sent from compromised systems and botnets as well.
It is about time that a blacklisting system is setup for providers that allow source address spoofing, similar to how providers running open SMTP servers were tarred and feathered until they fixed it.
It's not always laziness. I added outgoing filters to my routers so that it only allowed source addresses from my network. That was great at stopping DOS attacks, but as I found-out the hard way, several of my customers were sending outbound traffic with source addresses not on my network.
I'm not a networking wizard so I ask...why did the customers need to send outbound traffic using modified source addresses? Why should that be allowed as part of your service?
You are right. But i can't beleive that there are still ISP's out there which do not put filters based on their routing objects on their border routers. It's insane. And on the other hand their upstream providers allowing it. What is BGP good for then? Are network guys that lazy?
Because it is VERY difficult to ascertain whether the source of an inbound packet is forged unless it is very obvious (like an IP that should be inside your network or on a private subnet). Outbound traffic on the other hand should almost always have a source IP that belongs to your assigned ranges (or configured private subnets).
I think you are confusing addresses not part of the originating network's IP blocks with spoofed, when the customer might have legitimate rights to the IP range through another provider. it is quite common to do this for redundancy/qos reasons that are 100% legitimate. For example link redundancy with a preferred route (higher bandwidth/lower cost).
Forgive me if I'm wrong but given large volumes of traffic that are sold at the lowest rate, providers are not about to add hassle and overhead to their filtering...
So it's a "business decision" really. After all, is there anything to penalize network providers from not adding filters?
Personally I think this should really be in everyone's best interest given the implications of inaction, but how to start the ball rolling?
A 'singular oddity' is an event that cannot be explained and only happens when you are alone.
Users of the internet should send traffic from their assigned address.
When they have multiple addresses they should use the address that belongs to the interface they send it on.
Either they route the traffict to the interface that belongs to an address, or they assign the source address depending
on the interface they want to route on.
Don't adhere to this rule and you face blacklisting of your traffic.
It is similar to open SMTP servers. Used to be no problem, used to be common practice, is not acceptible anymore today.
You can watch DDoS attacks live as they are happening all over the world, including this recent attack from this website: http://www.digitalattackmap.com/#anim=1&color=0&country=ALL&time=16065&view=map
Absolutely love that website, ever since I've found it on Slashdot a year or so ago.
Because UDP is broken, by design.
Many people who have redundant connections will do load balancing. After all they are paying for two (or more) connections, why not use them. Not to mention, your rules would break their redundant connection... their primary connection goes down and they have to use their 2nd connection. Boom, they are down... so much for a backup connection :P "Thanks sucky ISP!" Posting anon to keep moderation.
Why not? They obviously use hijacked/trojan/virus infested PCs which will be distributed quite randomly.
Comment removed based on user account deletion
There are many legitimate use cases for a stateless protocol. It's not "broken".
You *might* want to read up on it here -> http://yro.slashdot.org/commen...
* :)
(You're welcome)
APK
P.S.=> Yes, it actually works....
... apk
Most commercial grade end-points support this already, like my ONT. The same service that tells my ONT what IP address I have also makes sure I only can send out packets with IP addresses that are from my network. You don't need to configure your core router for every possible IP that moves through your network.
Multi-homing is fine, but if you're going to do it, follow standard practice, don't just start sending out spoofed IPs.
See right here, at http://tech.slashdot.org/story...
Specifically, he noted that in http://archive.icann.org/en/co... that it's a simple task to check outbound packets and drop them if the return address is for someone else.
The open question is ISP motivations: I used to work for Canada's first big ISP, and my management would have freaked out if they thought they were frivolously enabling a DDOS attack. See the queue article and comments for more info.
davecb@spamcop.net
As someone above pointed out, load balancing and redundancy are valid reasons to send packets with source IPs not in the originating AS. That mostly doesn't apply to residential subnets where the zombies are, but one reason does. I sometimes use LTE tethering and my home internet connection simultaneously because the LTE is as fast or faster than my home connection during non peak hours. I don't know if it is doing load balancing between the two uplinks, but why shouldn't it?
refactor the law, its bloated, confusing and unmaintainable.
That's not Dice, it's the regular people who hate beta who are sick of the never-ending toddler temper tantrum of comments about it. The comments were read, possibly ignored and possibly considered, and nothing else is going to happen from continual screaming. Get over it and either leave, or switch off the beta and wait it out like the rest of us.
This space intentionally left blank
He was selling transit. It was customers of his customers. The customer of the customer had a valid source address in the customer of the customer's assigned netblock. The customer of the customer's netblock isn't one of the customer's netblocks.
Oolite: Elite-like game. For Mac, Linux and Windows
It's not always laziness. I added outgoing filters to my routers so that it only allowed source addresses from my network. That was great at stopping DOS attacks, but as I found-out the hard way, several of my customers were sending outbound traffic with source addresses not on my network. That was in 1997. For the next several years, it was a huge hassle to keep adding additional source address ranges for customers. An ISP selling a high speed connection has to allow outgoing traffic from addresses they don't own. That's the entire point of selling transit.
BGP Communities
blindly antisocialist = antisocial
He didn't say spoofing, he said transiting, so they are people who have their own IP blocks assigned and are using those. The advantage is that you can have multiple uplinks and use the second as backup if your primary goes down and all of the ips never change.
The real issue here is that you were filtering at the wrong point. It should have been your customers doing egress filtering in this case.
Because it is VERY difficult to ascertain whether the source of an inbound packet is forged unless it is very obvious (like an IP that should be inside your network or on a private subnet). Outbound traffic on the other hand should almost always have a source IP that belongs to your assigned ranges (or configured private subnets).
On the Internet there should also never be traffic with RFC1918 source IP addressing, which is easily filtered on ingress.
blindly antisocialist = antisocial
Can we use this NTP reflection amplification to DDOS beta?
I think you may be off base here. My guess is that the vast majority of users don't care about your little campaign to ruin slashdot. I will keep modding as troll so long as I have mod points.
Yup, edited my bookmark to http://slashdot.org/?nobeta=1 quite some time ago.
Filtering ingress packets with RFC1918 source IPs may be useful in some circumstances, but it doesn't help in amplified attacks.
The source in these cases will always be a legitimate uninfected machine that is just doing its job (such as a DNS or NTP server). The source IP will be whatever IP the requester expects to see, such as the destination IP of the initial request.
In amplified attacks, the forgery occurs in the initial request packets, all of which have the source IP of the DoS target, which must always be an actual external IP. This is where egress filtering is useful, because none of these requests should have an IP outside of the subnet serviced by the egress filter.
Filtering ingress packets with RFC1918 source IPs may be useful in some circumstances, but it doesn't help in amplified attacks.
The source in these cases will always be a legitimate uninfected machine that is just doing its job (such as a DNS or NTP server). The source IP will be whatever IP the requester expects to see, such as the destination IP of the initial request.
In amplified attacks, the forgery occurs in the initial request packets, all of which have the source IP of the DoS target, which must always be an actual external IP. This is where egress filtering is useful, because none of these requests should have an IP outside of the subnet serviced by the egress filter.
Agreed except that egress filtering is not practical for inter provider transit traffic and not all SPs filter their customers' traffic.
blindly antisocialist = antisocial
Transit only works if you have a BGP route going through. Use the announced BGP routes as a filter. If it is not announced, you don't route it.
It's not spoofing if you have your own AS and are sending traffic from your allocated IPs in that AS via multiple transit providers. When OP said:
...several of my customers were sending outbound traffic with source addresses not on my network
...he could have been indicating that he was providing transit to customers that had their own IP allocations, ergo the source addresses were not in his network.
Multi-homing doesn't necessarily mean "getting an account from two different retail ISPs and then NAT-ing different depending on which interface you're leaving on"; we could be talking about BGP multi-homing here, in which case static ACLs are a total bitch, but, as mentioned above, other solutions exist.
My ISP makes me advertise using BGP to send via IP addresses I don't own.
Use BGP to determine which AS's are advertising routes through you; and only let them go. Easy!
The way I read that... your customers were multihomed and sending you traffic that belonged on another ISP's network? That's the definition of spoofing! :-) You're perfectly correct to drop that crap. If they aren't using address space you assigned them, or PI space they own -- and told you about, it's not your problem.
I've had several clients bring their own address space (PI) and a few even had their own ASN. In **VERY** rare cases, we'd allow a client to announce a non-portable assignment to another ISP -- we have to give written permission to that other ISP, and carve into the client's desk that when they leave, they don't get to keep that address space. (and v.v., but getting the other ISP's permission was always an exercise. customers are so stupid.)
"What??? I didn't assign you different address space. What the f*** are you doing?"
They broken their network. That's their problem.
Someone has to be running BGP, but not necessary the customer... I've setup several customers that had PI space but didn't run BGP themselves. It's setup on the ISP side just like the ISP's RIR assigned space... announce it like you own it. :-) (they can multihome that, too, as long as everybody knows that's the plan. some ISPs get upset when they see "their" space being announced by someone else.)
Transit means you can hand me traffic destined to anywhere, not sourced from anywhere.