The Slashdot DDoS: What Happened?

What follows this introduction is a rough summary of the crazy hell that we endured with the intermittant DDoS[?] attacks we experienced last Thursday through Saturday. I'm sorry it took this long to put this together and tell you what happened, but as these things go, we were too busy trying to solve the problem to waste time talking about it. Big thanks to Andover.Net's Netops PatL, Martin and Liz, as well as Slashcode-wranglers PatG, Chris, Marc, Kurt and CowboyNeal, plus scoop (from freshmeat) and others who chimed in along the way. Tomorrow is part2: A good description of how the new Slashdot @ Exodus works.

What follows is more-or-less Pat "BSD-Pat" Lynch's account of the DDoS... Pat is our super 31337 BSD Junkie sysadmin. He wants everyone to know that the timeline below is little screwy, but things are more or less in sequential order. Things might not be exactly perfect, but hey, what do you expect after 30 hours without sleep?

Having moved the day before, none of us were truly familiar with exactly how the new hardware would handle the full burden of being 'slashdot.org'. The cluster (known affectionately as The Matrix) had handled its premiere day with flying colors, but we didn't really have an accurate feel of how things would react. Combine this with a couple of extremely high traffic stories posted on both Thursday and Friday, and it took us a awhile to determine that the problems were external, and not a flaw in some new component in the cluster."

The Attacks began Thursday morning. Most of it came in the form of SYN floods, from obvious /16's no less, and some /24's. We didn't have any zombie-killing software or a firewall installed because of certain network topology issues. Later on, a second wave came, this closer to 8 or 9pm and the load balancer (an arrowpoint CS-100) died under the load.

The DDoS, as far as I could see, was a lot of SYN and Zero port packets coming from various /16's and /24's as well as a bunch of RFC1918 reserved addresses (10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16) At one point we reached 109Mbits worth of traffic into our network.

Liz and I went back to Exodus and rebooted the Arrowpoint, then the site seemed "ok" for a bit. By 3 in the morning, Liz decided that the PIX (Cisco's firewall) could simply not do what it was supposed to do, so we went back and started building a FreeBSD box as a bridging firewall.

just before we went to plug it in, I tried to ssh into the vpn-gate and noticed that nothing was working right: while the site worked, outgoing traffic and source groups on the Arrowpoint was screwed. As if that wasn't enough, two ports died on it already!

At some unknown point (time blurs after 30 hours straight!) Martin and PatG show up (thank the gods!) and they force us to go to sleep, they bring the site up outside the Arrowpoint, while Liz and I watch from a hotel room.

As of Friday morning, the site is semi-working, but the adsystem can't be updated, and we have no access to the backend servers. I scream bloody murder to Arrowpoint, who eventually shows up to blame the router: a cisco 6509 switch with two RSM/MSFCs.

Liz and I do packet dumps and determine it's not the router, the little CS-100 had died the night before, and thats where it all started. The Arrowpoint guy insists we did something to make the Arrowpoint not work (CT: Explicit description of precisely where Liz and and Pat wanted to store the newly deceased Arrowpoint removed to keep things rated PG) By 7 the CS-800 CSS is up we're almost done for the day, but we stay to make sure. By 10pm we're exhausted but stable, although we're running 4 servers on a round-robin DNS while the new load balancer waits.

Netops (Liz , Martin and I) regroup, and do reintegration of new Arrowpoint CS-800 and installation of a new FreeBSD Firewall box instead of the PIX during Saturday Afternoon. Slashdot returns to normal. Sysadmins get well-deserved sleep.

So that was the story. It was a pretty hellish weekend for everyone involved, but thanks again to those that helped get our ducks back in a row. Again, Part #2 to this (which originally was gonna be run last Thursday, but with all this ddos stuff got pushed aside) is a fairly detailed description of the new Slashdot setup at Exodus, complete with all the changes mentioned above. Fun for the whole family if your family is really into clusters of web servers."

Comments

Defense in Depth

[Old Man Kensey]

Modern military command uses the concept of defense in depth. The essence of this is trading space for time.

The simplest case is building two small walls instead of one humongous wall. If you build a humongous wall, it takes a long time to get through... unless the enemy finds a single weak point -- then you're screwed. Two walls each take less time to get through, but if they're well-built using different techniques, the enemy may not get through to begin with and if they breach the first they lose time covering ground and then adapting. They're also very obvious as they traverse the open ground between barriers.

Network security can benefit from the same concept. Others have already mentioned heterogeneous "airgap" systems -- one of the most common and least excusable faux pas by so-called "security admins" is a single firewall protecting a herd of boxen. Second to that is identical airgap firewalls.

Of course real defense doesn't end with the walls. Even services running behind an airgap should be structured with an eye toward reasonable security, as others have pointed out. Many companies think their firewalls make them safe; come the day those firewalls are breached and the attackers make off with everything stored on the NT intranet server before wiping the drive, they'll find out differently.

Any server, no matter how well shielded, should start life in a lockdown configuration and then be made less secure only as needed ("do we really need to enable daytime on this box?"). Admittedly I haven't kept up with developments in secure distros, but does anyone make a "locked-down by default" distro based off Red Hat/Debian/*BSD? It'd be a real service to admins and if not it's something I might consider starting a project for. I know of Bastille Linux but that's (as far as I know) not so much a distro as a set of scripts to tighten up Red Hat.

The only thing we have yet to figure out is how to effectively make systems under attack "shoot back". The most they can do at the moment is call in an airstrike (i.e. alert the admins). Any return-fire capability would only be as good as the intermediate links let it be. It might not even be a good idea, as it would increase network traffic and make the attack that much more severe.

Re:What's the Cisco angle?

[buffy]

"They are probably much better off with the BSD box. Although it's not a good idea to advertise their security infrastructure layout to the world. (Hint, Hint, CmdrTaco!)"

I disagree 100%. Knowledge of an installation's infrastructure should never comprimise the security of the setup. If it does, then you're relying (to a certain extent) on security through obscurity. Security should be provided by a well thought out layered approach: network layering (multiple firewalls, screening routers, IDS, etc...), host-based security (tcp wrappers, service minimalization & replacement, tripwire, etc..), and application security (ie. authentication, verification, etc...)

In designing networking/server infrastructures it's best to think of it as an open source project, and you should be willing to get opinions and discussion from any number of sources that could include crackers who may at some point want to use that knowledge to attack your site. This is one of the things I like about TIS Gauntlet once upon a time..."crystal box" was the term they used to describe it.

You should prepare for an attack ASSUMING that the infiltrators know as much about your setup as you do. In the long run, if you know that your infrastructure can hold up to someone with that amount of knowledge, then you'll be doing pretty well.

My only question...did I actually see in a comment that they're using NFS to publish data to the distributed webservers??? Ew. Run.

-buffy

(Hmm...I seem to really like parentheticals, don't I? (well maybe not. (really!)))

Re:But who ... ?

[G27 Radio]

Who knows, even Bill may be a /. reader?!

I'd be suprised if he wasn't. I just wonder if he posts.

numb

Re:GODDAMIT A LINUX SITE SHOULD RUN LINUX!

[Darchmare]

---
OSX is covered cause even tho apple is a hugely proprietry company, everyone here loves microsoft competitors.
---

Well, Apple was a Microsoft competitor long before Slashdot started seriously covering them. I think it's more the hardware and Unix-based nature of their recent OS movements more than anything. Note that OSX is based on Mach/BSD, which goes to show you that they're not focused on Linux only.

---
Anyway, point was slashdot IS primarily a linux site.
---

If by that you mean that most of the people here have an interest in it, sure.

I'm not saying that Slashdot isn't incredibly biased toward Linux, but that doesn't mean the Slashdot editors won't use *BSD when the occasion warrants.

Anyhow, Slashdot may be Linux-oriented, but nowhere do they say that they are so to the exclusion of everything else (which was the point I was arguing).


- Jeff A. Campbell
- VelociNews (http://www.velocinews.com)

Re:Blame Canada

[Kurt Gray]

Exodus is getting $1million/year from us so they let us do whatever we want. They only thing they won't let us do is take a picture of our cage -- no cameras allowed anywhere in the facility! I guess they're afraid we're going to steal their soul. We were able to smuggle out this picture of PatG, PatL, Martin, and the Arrowpoint rep. Behind them you can see the current Slashdot setup.

Of course it was the router!

[Denor]

...Arrowpoint, who eventually shows up to blame the router: a cisco 6509 switch with two RSM/MSFCs.

Yeah, that Microsoft certification, you really should have known better....

Re:Of course it was the router!

[Vanders]

Sure you're not thinking of a MCSE? Minesweeper Consultant and Solitair Expert....

Re:Why a firewall?

[GC]

Hmmm,

The CPU of a server spends time on a packet before it has checked what service it is destined for.
Filtering Broadcast packets on a network is a great way to improve workstation & server performance.

In the old days with Token Ring networks with cheap and nasty MAUs you could bring down the entire network with broadcast packets...

Poetic Justice?

[JamesSharman]

I find it quite amusing that the site that has entered legend for it's own specialized form the of the DDOS (the slashdot effect) has itself fallen fowl of the more malicious variety.

Congratulations on getting the new servers up and running, I've just moved my badtech cartoon site to digital nation (The old location of the slashdot servers).

Re:Blame Exodus SQUARED

[zrk]

BEGIN rant

I would definitely look at Exodus for some of this trouble. At times, they have been less than helpful for the service level they claim they will provide.

-They changed their security policy a while ago, and neglected to tell us until after the fact. All visitors to your cage must be announced, and just try to get replacement parts in and out without a whole rigamarole. Previously, one person "on the list" could escort others in and out of the facility, but no more. Granted this makes some sense, but when we showed up the first time after they changed their policy, before informing us, we balked, and complained. The response was (I kid you not) - "Well, we're a big company now, so we can't give the same level of service we used to." WHAT KIND OF ORGANIZATION SHOOTS THEMSELVES IN THE FOOT LIKE THAT?

-Their HVAC is substandard, and they don't truly care what equipment is placed in a cage. I pity the poor sun techs who have to replace the Sun server at the bottom of a stack of 10 other machines (ie, no shelf).

-They continue to abide by their own notification procedures when their "monitoring" software reports trouble. We've gone over their policy several times with them, and verified they had correct contact information for us, and yet they still follow old ways of notification. In this case, it's paging one person instead of using the paging mechanism that contacts the actual people who will do the work - the effort is the same either way.

-The number of times that we've notified them of trouble before their monitors catch it - for example, try working with them to show DNS requests from the outside to their servers aren't being handled.

END rant

I could go on, but I won't.

Nice account, but who?

[alteridem]

That was a good account of what happened, but in part two, we want to hear what you are doing to track the bastards down. Knowing how you go about fixing the problem and then tracking down the culprits may help other people who run into the same problem in the future. We would understand if you need to keep the info secret until you have finished tracking them down, or for legal reasons, but at least tell us so.

Re:Nice account, but who?

[Kenelson]

Actaully in the few times I have faced a DoS attacks, we did manage to track the users down. Just because they are forging the packets does not mean that their machine was able to avoid contacting the target completely.

In our case, we tracked the user down to his source by the "other" packets which he sent. The person sending the DoS often will send a ping and/or a name lookup of similar request prior to the attack or each time they add a new host in. Although it is a considerable exercise in collecting enough data to figure out which connects were real "valid" user contacts and which came from the kiddies. As a result we managed to isolate the DoS to specific hosts and subaddress ranges.

Of course, if you are into real fun assuming that you can get one of their target machines (which using a DoS scanner and a rough idea what subnet they are in) you can often port scan for eggdrop bots and other toys. Once you can convince a physical sysadmin to send you those files you then have a map of the kiddies entire bot and DoS network. Once I reach this stage I then post guards on IRC channels which their bots used and with a small ammount of detective work get their ISP. Script kiddies like to brag about what they do and it enevitably leads then them to surrender their identities.

In all the cases in which I managed to get that level of penitration into the kiddies network, I always managed to shut them down. ISP are very friendly about taking out malicious users especially when you supply logs and the attackers home address. :-) I even have gotten offers to have the attacker arrested (to bad I don't have the cash to fly there and file charges). Thus I can conclude although it is not an easly task it is not entirely necessarily impossible. (That is assuming your attackers are 14 year old kids and not paid professionals.)

--Karl

Re:Why a firewall?

[Christopher Thomas]

Why are you installing a Unix-based firewall in front of some Unix-based public servers? Why not secure the servers in the first place?

Having a firewall in place to filter invalid packets and other crud thrown at the servers means that more of the servers' time is spent generating slashdot pages. Also, the simpler the Unix box, the easier it is to secure - hence, securing a stripped down firewall instead of a big, complex slashdot server.

MPAA

[austinij]

(CT: Explicit description of precisely where Liz and and Pat wanted to store the newly deceased Arrowpoint removed to keep things rated PG)

You'd better watch it with this comment... the MPAA might come after you too!

Timing

[348]

I'm curious about the timing with the port to the Exodus environment, was there any indication the attack was timed to take advantage of the different environment? Not saying that the security measures were better or worse than the old site, just that the timing seems rather convienent.

What's the Cisco angle?

[drteknikal]

I'm curious on one detail. What was it that the Cisco PIX was supposed to do and didn't?

Re:What's the Cisco angle?

[John Fulmer]

I'm not a slashdot admin (but I could play one on TV!), but I am painfully familiar with PIXs.

The idea behind the PIX, or any firewall-like object, is to allow 'good' traffic (http, smtp, etc) into the production network, and reject 'bad' traffic (oddball ports, like port 0, unauthorized UDP traffic, etc).

The problem with the PIX, is that it is essentially a fairly stupid router that can do network address translation and other bells and whistles, but it does it poorly. VERY poorly. It was designed as a network address translation system back in the mid 80's (anyone remember all the "We'll run out of IP's by 1997!") by a company that Cisco later bought. Cisco took the product, did a logic problem ( "Firewalls can do address translation. PIX does address translation. PIX is a firewall!"), and had themselves a firewall.

Its configuration makes a lot of sense to someone familiar with cisco router ACL rules, but no one else.

They are probably much better off with the BSD box. Although it's not a good idea to advertise their security infrastructure layout to the world. (Hint, Hint, CmdrTaco!)

jf

Re:Why a firewall?

[Fas Attarac]

I don't quite know if you're asking these questions because you're legitimately trying to learn something about security or if you just think you have all the answers and are considering the universities that teach this stuff and the highly trained corporate IT departments to be idiots..

I totally agree that systems need to be individually secured against obvious problems. In any production setting you have to safeguard at least a bit against unauthorized access (even if from your own network). Firewalls just allow that to be done in a single layer, with a single access policy and set of rules.

It's a lot easier to set up a firewall (perhaps composed of multiple systems for redundancy and load management, perhaps even built into the very routers you're using) that's been designed for this task than it is to go through and audit every system individually.

What if you don't want systems to be reachable from the outside world at all? Your solution would be to use ipchains/whatever and just block all of the ports with that?

Are you aware that there are regularly discovered stack flaws that allow people to disable or crash a system where they have a direct network path like this? What about OS fingerprinting? I would be very uncomfortable if my servers could be touched at all by packets originating from the Internet. Firewalls not only keep people from accessing what may be potentially insecure systems, but it keeps them from doing *anything at all* to them that isn't explicitely allowed. By putting this functionality into a firewall, you have only one type of system (by "type" I mean "firewall" versus "web server" versus "NFS server" or "database server") seen by the outside world, and no critically vulnerable services that they can even *see* much less get to. If you were to put the load of network security onto the individual hosts, there are tons of things somebody can do, even if the service itself is secure, network threats are still quite serious.

If you legitimately are curious about actual network setups and why things like firewalls are necessary and aren't just trying to be an ignorant troll, I'd suggest you take some networking classes at your local university. Depending on their setup, they may have a lab for people to play around with various types of setups, even to the point of letting you simulate your own DDoS attacks and hacking into your own systems. Fun stuff.

Owned?

[Remote]

I wasn't going to talk about this in public because of /. silence about the DDoS, for I thought things could be somewhat related.

This is what I got this morning when I asked for www.slashdot.org:

&lthtml&gt
&lthead&gt
&lttitle&gtNot Slashdot.org&lt/title&gt
&ltmeta name="keywords" content=""&gt
&ltmeta name="description" content=""&gt
&lt/head&gt

&ltscript language="javascript"&gt
&lt!--
if (top.frames.length != 0)
{
top.location=document.location
}
//--&gt
&lt/script&gt

&ltframeset
rows="*,90" marginwidth="0" marginheight="0"
framespacing=0 frameborder=no border=0
&gt
&ltframe
marginwidth="5" marginheight="2"
src="http://slashdot.org"
name=thepage framespacing=0 frameborder=no border=0
&gt
&ltframe
marginwidth="0" marginheight="0"

src="http://red.namezero.com/strip2/strip.jhtml? name=slahsdot.org&channel=www"
name=pb scrollbars=no scrolling=no
framespacing=0 frameborder=no border=0
&gt
&lt/frameset&gt

&ltnoframes&gt
Sorry
&lt/noframes&gt

&lt/html&gt

Weird. Did anybody else see this?

Blame Exodus

[snopes]

We didn't have any zombie-killing software or a firewall installed because of certain network topology issues.

Topology my ass. Exodus fights hard to make you use their 'value add' security services. Be honest guys, the reason you weren't protected was b/c those bastards were working you over for more money and don't want you running your own security, right? In fairness, there's some nice things about running out of an Exodus facility, but dealing with their physical and network security chimps is not one of the high points.

Re:Why a firewall?

[DirkGently]

Coupla reasons:

1. I'd say that they don't want to limit thier functionality. A tweaked firewall will let them keep useful schtuff turned on.

2.If the firewall uses its CPU to deflect the crap, then the web servers wont have to deal with it.

3. They have a BSD uberadmin who can make that BSD box walk the dog. If something else wierd goes on, it'll be in his back yard.

Dirk

Re:Why a firewall?

[Eivind Eklund]

Because Unix != Unix, as you should be well aware of. The Slashdot servers are AFAIK running Linux; what was installed in front of them was FreeBSD. FreeBSD has a number of features that make it better at handling attacks (dummynet, *_BANDLIM, etc) that I do not believe have equivalents for Linux. There are also some aspects of the basic kernel networking architecture in FreeBSD that might increase attack resistancy (but I suspect they do not make much difference for that case).

It might be possible to switch the main servers from Linux to FreeBSD, but as an interrim solution I think putting a FreeBSD firewall in front of them was a good tradeoff, giving time to evaluate whether an OS change on the servers themselves are warranted, or if there are reasons why keeping Linux make more sense.

Eivind.

Re:Curious about the setup...

[Kurt Gray]

For example, why are the servers serving images and static files segmented? Is there a lot you save from Apache configuration for dealing with one as compared to the other?

The web page servers run Apache+mod_perl+DBI+adsystem module, and the image servers run a much lighter Apache httpd with cache friendly headers.

Where does MySQL sit? Any "reason" behind Debian vs. RH other than "just because"?

MySQL is on the VA 3500 box which is also the Red Hat box. The servers all came with Red Hat and we installed Debian on them, expect the 3500, and I think that was because VA installed extra drivers and stuff we wanted to leave it as is.

Also, any chance you could go through some of the configuration choices made for your apache processes on each of these?

I think this will be in Rob's next post. If not we'll post in that forum.

Re:Why a firewall?

[YU Nicks NE Way]

First, any public system on the web should be behind a firewall. The amount of load that a firewall takes during an attack can easily drive even a very fast machine to 100% utilization; if you want your other servers to still be serving legitimate customers, you need a firewall.

Also, a firewall acts like a choke point -- any attack must pass through it. By monitoring the health of that one machine, you can monitor the health of the entire networks. In addition, if you want to allow remote administration of the items in the cluster, you can provide a secured path through the firewall; again, you have only the one point of failure.

It's usually wise to have stacked firewalls (an "airgap") in front of a popular site, though, and it's often best to use a variety of operating systems on those firewalls. Somehow, though, I can't see Slashdot doing the wise thing there, though, and putting a FreeBSD->W2K airgap at the front, with the Linux-based Slash behind it.

10.0.0.0 net

[Signal 11]

I've found this alarmingly common to be routed on networks. I wish router manufacturers would squish this once and for all - it's in the RFCs that these are NOT to be routed on the 'net at large.

I've had alot of portscans for 31337 and 12345 in the past week on the mediaone network, all from 10.0.0.0/16 networks. I am massively annoyed that they let this through and block ports 137:139. Umm.. is this solving the problem? No! Oh, and they've taken a liking to scanning their customers boxen.. but I digress.

DDoS is the direct result of sloppy upstream administrators. IF I were in your shoes, I would be suing every person upstream for atleast a few hops for passing those 10.0.0.0 packets along for gross negligence.

Part2: Can we get some pictures?

[a.out]

I know I'm not the only one who would like to see pictures of this whole setup :)

And while you're at it get CowboyNeal to give us a sexy pose *on* the servers (grin)

How does a DDoS attack screw up the / code???

[moopster]

Every time I tried to view the front page of /. it came up waaaaaaay funky. 1 - Did anyone else experiance this? 2 - Is there more to this problem then just a DDoS? mcd

What about the children?

[Anonymous Coward]

While I agree that the Slashdot DDoS attack caused many people quite a bit of annoyance and frustration, I think leaving the impact at that is very short sighted.

Firstly, I don't think the blame for this DDoS can be centered on just one person or group. Obviously, those who attacked Slashdot are to blame, as are Slashdot's sysadmins, and the people at Arrowpoint. And secondly, the costs of this are much greater than you might think.

I have an eight year old daughter. We had a family pet - a rabbit, black, named Midnight, and my daughter was very fond of it. Midnight, sadly, passed away about two months ago. A week or two after Midnight died, my daughter came to me in tears and asked me, "Daddy, why won't God bring Midnight back? I've been praying like Deacon Simmons told me to."

Naturally, I had to think about how to respond to this. I finally answered, "well, honey, God is a little like Slashdot. He can seem arbitrary, cruel, and unresponsive, but he's really a nice guy who's just a little out of touch and is a little slow at responding to requessts."

This was fine, and I thought that would be the end of it. However, when Slashdot went down last week, my daughter burst into my den, positively sobbing and wailing, and managed to choke out "Daddy! Daddy! I can't get to Slashdot!" "Honey," I said, "it's just a website." But, between sobs, she said, "but you said God is just like Slashdot, remember? Does this mean God is dead?"

I tried to console her as best I could, but nothing seemed to work. When Slashdot came back up, she seemed to return to normal, but she hasn't been quite the same since. She doesn't ask me about God so much any more, and she seems less interested in Church.

As a good Christian, I will turn the other cheek, and not call for the punishment of those responsible. But to the heinous criminals and negligents responsible for this, I must ask, how do you feel about destroying a small girl's sense of innocence and wonder about the world? About crushing her childish dreams and idealism? About shattering her faith in God and his benevolence? About possibly having crushed her soul and emotion forever, leaving her to live the rest of her days in spiritual agony as a broken, scarred husk of a person?

I hope all of you think long and hard about what you've done. What is the soul of a child worth, next to a few double-checks of the router?

Thank you.

No doubt it was MS/MPAA/RIAA/Metallica/Dr Dre

[auntfloyd]

I'm sure that these great enemies of the Slashdot Empire have found this to be a convenient time to strike. We must systematically seek and destroy all those suspected of having sympathies with the MPAA, RIAA, or Microsoft for security reasons.

Therefore, all

Windows users

CD listeners

Movie watchers

Metallica fans

are asked to please leave now or face prosecution.

thank you.

Re:Owned? - Nope

[Cpyder]

Maybe you should type more carefully, since you
requested http://slahsdot.org (slaHSdot) not
slashdot.org...

I registered that domain (for free @ namezero) to
help the people who couldn't type. Sorry if I scared you :-)

Cpyder@slahsdot.org
_
/ /pyder.....
\_\ sig under construction

Is that what took so long..

[ch-chuck]

waiting for Rob to toggle in a boot loader to IPL from the punch card reader?

RSM/MSFC definations

[Ed Bugg]

RSM - Route Switch Module
- Basically a router on a card in the switch for routing between VLANs

MSFC - Multilayer Switch Feature Card
- Once a route for a packet flow is figured out (from the first packet going through the router) all other packets from the flow get switched instead of routed.

Re:What's the Cisco angle?[ERRATTA]

[John Fulmer]

Sorry. Make that mid 90's...

jf

A little more detail on the hardware setup

[Kurt Gray]

Rob is going to post exact hardware specs later, but in the meantime just to give you a brief idea where the "Arrowpoint" sits in relation to all this... Slashdot now running on several machines, all VA FullOns, running Debian and few running Red Hat, Apache+mod_perl, MySQL. The database is on its own VA 3500 server. There are currently six VA FullOns serving web pages from an NFS server, and three other web servers serving images.

All of these machines were behind an Arrowpoint (CS-100) firewall/load balancer which took it on the chin when we got DDoSed, so basically the Arrowpoint was taking the full force of the attack. So as described above we replaced it with a CS-800 and a BSD firewall.

I guess we learned that if you're going to post a letter from a Microsoft attorney on your web site the same day you implement a few new troll filters you better be prepared for the fury of hell to rain down on you. Then again this is Slashdot, so we always should be prepared for the fury of hell to rain down on us.

DDOS != 10.0.0.0

[Christopher Thomas]

DDoS is the direct result of sloppy upstream administrators. IF I were in your shoes, I would be suing every person upstream for atleast a few hops for passing those 10.0.0.0 packets along for gross negligence.

Um, no.

DDOS simply requires that a lot of compromized boxes be able to send you packets. Spoofing to non-existant return addresses is an orthogonal issue. You reply that it's used to mask the souce boxes? Any _valid_ address could also be used for that, so filtering would gain you nothing against that.

I agree that filtering of reserved addresses should be done, but that would not hinder a DDOS attack.

Re:DDOS != 10.0.0.0

[InsaneGeek]

I believe that he was more going along the line of things outgoing packets that have a source address from outside my network should be dropped before it gets outside your own network (not just the reserved, but anything that isn't supposed to be outgoing over that router).

As long as you aren't in wild and wooly peering arangements, one should be able to know all the ipaddress that are inside ones network (and within each segment of the network). Once a router sees something that can't possibly be coming from inside that network, it should be dropped and throw up alarms, bells, flashing lights, etc. cause something just ain't right (either a misconfigured client or someone trying something bad).

Doing this type of filtering doesn't prevent your system from being used in a DDOS attack, but it prevents your system from being used in the attack with a spoofed address. Hence see 50mb/sec from host w.x.y.z, contact owner of that address block and get it stopped, since it is not forged they have a compromised box internally. If everybody started doing that the world would be a MUCH better place to live in.

A link or two

[Praxxus]

Charles Spurgeon's Ethernet Web Site

Jason Schwarz Ethernet Tutorial

Lantronix Networking Tutorials

You might also try typing "ethernet tutorial" or somesuch in your favorite web search engine. Hope this helps!

--

Re:Could someone point me to a networking tutorial

[Fishstick]

I would recommend you start here:
net3-4-howto
firewall howto
masq-howto

I have also heard that you can directly connect two NICs with a special cable. Do you need software changes to do this?

Yes, you can do this with a crossover cable and no you don't really need any special software to do this. I use one when I bring my laptop into work and want to hook it to my workstation. You can either make one yourself or buy one at any decent site like hardwarestreet.com.

Sorry I am so clueless.

:-) Try 'Networking for Dummies'. It is a pretty good reference for setting up a Q&D network. The examples are for windows, but the basic principles are the same. I started out with the intent to hook up my PC with my Wife's to share a printer, knew nothing at all about setting up a LAN. That book and those howto's and a lot of tinkering were pretty much all that were required. Now I have my whole house wired, I have a Linux box hooked up to my cable modem doing masquerading for the machines in my house. I set up a server to do SMB file and print sharing and stuff.

Anyway, good luck.

Re:Why a firewall?

[YU Nicks NE Way]

First, in the event of an attack, a single point of failure isn't necessarily a bad thing. If you know exactly what has fallen over, you're more than half way to knowing how to fix it. A firewall is easy to secure precisely because it isn't a general purpose box; the BOFH knows exactly what's running on it. The worker bees behind the firewall are a different matter; they presumably run a wide variety of different software. Failure analysis becomes much more complicated. (Not to mention that diverse software allows for interaction among the different components, which exposes flaws.)

In a DDOS attack, if your firewall falls over, then the odds are that your network would have fallen over, too. Slashdot "only" handles 100Mb/sec, though -- one high end machine should be able to handle a pipe that wide. But, if the pipe gets wider, then they can get a virtual "choke" with a load balancer in front of the firewall.

Re:PIX inusfficient?

[gavinhall]

Posted by BSD-Pat:

The problem here is that we only had one subnet to work with. The PIX we had wouldn;t to the type of filtering/bridging that I wanted.

Cisco wants a DMZ on these things.

I needed a bridge...why I didn't use linux...

It was quicker and easier for me... ipchains has always been a pain in my arse... ipfw and ipfilter I know best.

The other thig is that we fried an arrowpoint cs-100 (little itty bitty dinky thing that was being replaced with a bigger one)

the little arrowpoint couldn't take the traffic of 109Mbits , it wasn;t meant for that, we were waiting on arrowpoint to ship us the unit we were *supposed* to have.

*BSD fills the gap because I know it inside and out, and it was the quickest to get up at that point.

As far as the router, we can't do any type of stateful filtering on the 6509, due to some setup that exodus has with the HSRP stuff, I'm sure given enough thought I could figure out how to do it, however we were running on crisis mode.

The BSD firewall filled that gap for us...I can now do access lists on that, instead of the cisco.

and we still have a "DMZ" but its on the same subnet.

The arrowpoint CS-800 was emergency shipped to us that afternoon....its about as big as a cisco 6509...and ummm won't die under that type of traffic/content checking (its layer 5 remember)

-Pat

Prevent SYN floods @ the MSFC

[Anonymous Coward]

Why not do IOS Load Balancing from the 6500/MSFC itself? You can use SYNGuard with the Load Balancing to protect against SYN floods... refer to: http://www.cisco.com/univercd/cc/td/doc/product/so ftware/ios121/121newft/121limit/121e/121 e1/iosslb1.htm#xtocid446613 -- Anonymous Cisco Employee

Re:Why a firewall?

[Thomas Charron]

Most of the time, you'll have a backup firewall just waiting to be turned on. Load is only *part* of the issue that occurs. The first one, which is simple enough, is that it's easier to secure a box on the front end then one on the backend. My home firewall, and the one at work, quite literally have *NO* open ports at all. We really hope the console never dies, becouse we're forked if it does.. :-P But doing this was *really* easy. Now, a backend box is only as secure as it's weakest app, and in the case of things such as MySQL, Samba, etc, it simply makes more sense to have at least one box sitting in front with *everything* locked down. 'Spec when individuals use a system such as Linux's ability to IpMASQ. A good situation would be a front end firewall doing IP forwarding to a load balancing system. Have it only forward the ports required, aka, port 80. Anything else, rejected and logged. Then, even if they do manage to exploit something in your web server, they magically find themselves unable to telnet to that trojan backdoor they managed to get running.

The more hurdles you put in front of the kidies, the more likely they are to get bored..

Re:Anti-spoof filters on the Exodus network

[gavinhall]

Posted by BSD-Pat:

even more wacky, we were getting stuff from 0.0.0.0/8 (gee, how the F#@% do you filter that??!?!) lets filter the equivalent of "any", gee...

we have been talking to Exodus to get this problem resolved.