Slashdot Mirror
The Slashdot DDoS: What Happened?
What follows is more-or-less Pat "BSD-Pat" Lynch's account of the DDoS... Pat is our super 31337 BSD Junkie sysadmin. He wants everyone to know that the timeline below is little screwy, but things are more or less in sequential order. Things might not be exactly perfect, but hey, what do you expect after 30 hours without sleep?
Having moved the day before, none of us were truly familiar with exactly how the new hardware would handle the full burden of being 'slashdot.org'. The cluster (known affectionately as The Matrix) had handled its premiere day with flying colors, but we didn't really have an accurate feel of how things would react. Combine this with a couple of extremely high traffic stories posted on both Thursday and Friday, and it took us a awhile to determine that the problems were external, and not a flaw in some new component in the cluster."
The Attacks began Thursday morning. Most of it came in the form of SYN floods, from obvious /16's no less, and some /24's. We didn't have any zombie-killing software or a firewall installed because of certain network topology issues. Later on, a second wave came, this closer to 8 or 9pm and the load balancer (an arrowpoint CS-100) died under the load.
The DDoS, as far as I could see, was a lot of SYN and Zero port packets coming from various /16's and /24's as well as a bunch of RFC1918 reserved addresses (10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16) At one point we reached 109Mbits worth of traffic into our network.
Liz and I went back to Exodus and rebooted the Arrowpoint, then the site seemed "ok" for a bit. By 3 in the morning, Liz decided that the PIX (Cisco's firewall) could simply not do what it was supposed to do, so we went back and started building a FreeBSD box as a bridging firewall.
just before we went to plug it in, I tried to ssh into the vpn-gate and noticed that nothing was working right: while the site worked, outgoing traffic and source groups on the Arrowpoint was screwed. As if that wasn't enough, two ports died on it already!
At some unknown point (time blurs after 30 hours straight!) Martin and PatG show up (thank the gods!) and they force us to go to sleep, they bring the site up outside the Arrowpoint, while Liz and I watch from a hotel room.
As of Friday morning, the site is semi-working, but the adsystem can't be updated, and we have no access to the backend servers. I scream bloody murder to Arrowpoint, who eventually shows up to blame the router: a cisco 6509 switch with two RSM/MSFCs.
Liz and I do packet dumps and determine it's not the router, the little CS-100 had died the night before, and thats where it all started. The Arrowpoint guy insists we did something to make the Arrowpoint not work (CT: Explicit description of precisely where Liz and and Pat wanted to store the newly deceased Arrowpoint removed to keep things rated PG) By 7 the CS-800 CSS is up we're almost done for the day, but we stay to make sure. By 10pm we're exhausted but stable, although we're running 4 servers on a round-robin DNS while the new load balancer waits.
Netops (Liz , Martin and I) regroup, and do reintegration of new Arrowpoint CS-800 and installation of a new FreeBSD Firewall box instead of the PIX during Saturday Afternoon. Slashdot returns to normal. Sysadmins get well-deserved sleep.
So that was the story. It was a pretty hellish weekend for everyone involved, but thanks again to those that helped get our ducks back in a row. Again, Part #2 to this (which originally was gonna be run last Thursday, but with all this ddos stuff got pushed aside) is a fairly detailed description of the new Slashdot setup at Exodus, complete with all the changes mentioned above. Fun for the whole family if your family is really into clusters of web servers."
The simplest case is building two small walls instead of one humongous wall. If you build a humongous wall, it takes a long time to get through... unless the enemy finds a single weak point -- then you're screwed. Two walls each take less time to get through, but if they're well-built using different techniques, the enemy may not get through to begin with and if they breach the first they lose time covering ground and then adapting. They're also very obvious as they traverse the open ground between barriers.
Network security can benefit from the same concept. Others have already mentioned heterogeneous "airgap" systems -- one of the most common and least excusable faux pas by so-called "security admins" is a single firewall protecting a herd of boxen. Second to that is identical airgap firewalls.
Of course real defense doesn't end with the walls. Even services running behind an airgap should be structured with an eye toward reasonable security, as others have pointed out. Many companies think their firewalls make them safe; come the day those firewalls are breached and the attackers make off with everything stored on the NT intranet server before wiping the drive, they'll find out differently.
Any server, no matter how well shielded, should start life in a lockdown configuration and then be made less secure only as needed ("do we really need to enable daytime on this box?"). Admittedly I haven't kept up with developments in secure distros, but does anyone make a "locked-down by default" distro based off Red Hat/Debian/*BSD? It'd be a real service to admins and if not it's something I might consider starting a project for. I know of Bastille Linux but that's (as far as I know) not so much a distro as a set of scripts to tighten up Red Hat.
The only thing we have yet to figure out is how to effectively make systems under attack "shoot back". The most they can do at the moment is call in an airstrike (i.e. alert the admins). Any return-fire capability would only be as good as the intermediate links let it be. It might not even be a good idea, as it would increase network traffic and make the attack that much more severe.
-- Old Man Kensey
OK, ok, everyone point fun at me for being dyslexic! And anyway, how do you know they weren't falling over chickens!
Ok, so I'm dyslexic and get the spelling of words wrong sometimes (which a spell check helps with) and sometimes use the wrong word (which it doesn't) but there is a deeper issue here. Language is simply a means of communication, if the message is communicated then it has done it's job. Furthermore language is not defined by text books and dictionaries, these books record it. There is only something wrong with a statement when it fails to convey the intended message, not when a word is incorrectly spelt or a comma is out of place.
Referring to me as 'an ignorant looser' is nothing short of bigotry, if you really want a discussion I suggestion come out from behind the aptly named 'anonymous coward' hiding place, reveal your identity and discuss the matter without resorting to insult.
"They are probably much better off with the BSD box. Although it's not a good idea to advertise their security infrastructure layout to the world. (Hint, Hint, CmdrTaco!)"
I disagree 100%. Knowledge of an installation's infrastructure should never comprimise the security of the setup. If it does, then you're relying (to a certain extent) on security through obscurity. Security should be provided by a well thought out layered approach: network layering (multiple firewalls, screening routers, IDS, etc...), host-based security (tcp wrappers, service minimalization & replacement, tripwire, etc..), and application security (ie. authentication, verification, etc...)
In designing networking/server infrastructures it's best to think of it as an open source project, and you should be willing to get opinions and discussion from any number of sources that could include crackers who may at some point want to use that knowledge to attack your site. This is one of the things I like about TIS Gauntlet once upon a time..."crystal box" was the term they used to describe it.
You should prepare for an attack ASSUMING that the infiltrators know as much about your setup as you do. In the long run, if you know that your infrastructure can hold up to someone with that amount of knowledge, then you'll be doing pretty well.
My only question...did I actually see in a comment that they're using NFS to publish data to the distributed webservers??? Ew. Run.
-buffy
(Hmm...I seem to really like parentheticals, don't I? (well maybe not. (really!)))
Probably because they're running things behind the firewall like NFS and some flavor of SQL which won't be secure enough to expose to the Internet anytime soon.
Who knows, even Bill may be a /. reader?!
I'd be suprised if he wasn't. I just wonder if he posts.
numb
Not knowing if that's a joke or not...
/") for all of your unix boxes because it takes time to setup accounts, etc.
I may sound like too much of a bastard, but not having time is not an excuse, you aren't doing your job. Each of those routers had to be configured to begin with, and most networking guys keep the entire configs in a text file that they upload to a router, add a couple of lines to the code and your done. Not doing this stuff is akin to doing a ("chmod -R 777
It's amazing how much time the admin seems to get when a site realizes that 80% of a T3 is full of bad traffic (the old saying and ounce of prevention...). If you don't have time to do this type of stuff, you need to have a serious talk with your boss; because sometime soon you are going to spend a whole week cleaning up some crap that would have only taken you a couple of hours to do in the first place (not to mention boss yelling, legal dept. yelling, ceo yelling, customers yelling...).
This is something I've been wondering about recently. How do you have clustered web servers sharing storage? Sure, use NFS you say. But that introduces a single point of failure. If your NFS server goes down, you lose the entire cluster. Are there any solutions to this that don't involve spending vast quantities of money on a Sun HA failover system or an Auspex mirrored NFS system or similar?
"The invisible and the non-existent look very much alike." -- Delos B. McKown
This is a very misleading post. First of, it's 10.0.0.0/8 not 10.0.0.0/16. Second, the only net you could remotely finger is the originating net for not doing egress filtering on the private nets. Everyone else is just routing based on dest IP and switching based on the data link (MAC) info. But there's no requirement for them to be doing that. The real fault lies with the local network engineer for not doing ingress filtering of packets with a source on a private net. You've got to take responsibility for your own misconfigurations. You can't blame everything on somebody else. They should have had a firewall in place and Exodus should have been doing the ingress filtering at their border. See my other post for a suggestion as to why this wasn't happening.
---
OSX is covered cause even tho apple is a hugely proprietry company, everyone here loves microsoft competitors.
---
Well, Apple was a Microsoft competitor long before Slashdot started seriously covering them. I think it's more the hardware and Unix-based nature of their recent OS movements more than anything. Note that OSX is based on Mach/BSD, which goes to show you that they're not focused on Linux only.
---
Anyway, point was slashdot IS primarily a linux site.
---
If by that you mean that most of the people here have an interest in it, sure.
I'm not saying that Slashdot isn't incredibly biased toward Linux, but that doesn't mean the Slashdot editors won't use *BSD when the occasion warrants.
Anyhow, Slashdot may be Linux-oriented, but nowhere do they say that they are so to the exclusion of everything else (which was the point I was arguing).
- Jeff A. Campbell
- VelociNews (http://www.velocinews.com)
- Jeff
Exodus is getting $1million/year from us so they let us do whatever we want. They only thing they won't let us do is take a picture of our cage -- no cameras allowed anywhere in the facility! I guess they're afraid we're going to steal their soul. We were able to smuggle out this picture of PatG, PatL, Martin, and the Arrowpoint rep. Behind them you can see the current Slashdot setup.
-Denor
There are a variety of ways to trace DoS attacks back using the current infrastructure, including the 'manual traceback' technique that Christopher alluded to. However, they don't work very well for DDoS.
For DDoS, tracing back to he source still isn't good enough, as 'here's a list of 10,000 hosts that have been co-opted to do a DDoS' has made the problem simpler, but still pretty difficult (stopping those hosts from doing it again, making sure that a different set of 10,000 hosts are co-opeted, determining who co-opted the hosts in the original place, etc.). Also, I'm not convinced of Savage's trick with chunking working very well when you're talking about 10,000 traces.
Why are you installing a Unix-based firewall in front of some Unix-based public servers? Why not secure the servers in the first place?
-russ
Don't piss off The Angry Economist
Hahahahahaah! You Andover folks are more 1337 than I thought. Not only do you have uber-hacker John Walker on your team, you're running the site on a Univac 1107 -- say, you have any of those old 2 1/4 ton 100MB hard disks?
I am quite civilized, and I should be brought a beer immediately. -- Bruce Sterling
Exodus is full of people that don't pay or want the world for nothing. Digex on the other hand has great support, strict security and very effecient services. (No restrictions on bandwitdth, backup hardware/routers/nics/switches and enough power to last weeks after a nuclear strike :)
Go digex
Because what you don't see when you come to this site, and possibly look at the code, is that all the pages are dynamicly generated. I can't be sure, but I'm guessing the 'sections' on the front page could be located on more then one server. And the artciles are in a DB on another server, so if that can't be reached, you just seriously chopped down the size of your resulting HTML. (And output, since there is no longer a middle) =]
The webpage where I work is located in 5 different files (PHP), and joined together when the user loads. But when it's all together, and you look at the source, the page looks like it should be one file.
MSFC
On Cisco equipment this is touted as Netflow and on Cat5500s the feature resides on the Supervisor Module (III) - This is in Slot 1.
RSM
Essentially a Cisco series 7000 router, without any physical ports (except the backplane I suppose), like others have pointed out to route between VLANs within the VTP Domain of Cisco switches.
I have found that routing is pretty slow (as least for today's 1gb/100Mbit LANs) so you're probably better putting Gigabit cards on your servers that are capable of VLAN detagging, then just let the switch switch packets by physical address.
A lot of corporations will use the 10.* address space for thier internal networks... with routers and switches and the such. Most routers will not allow 10.* addresses to be routed unless specifically told to. Unfortunately, the problem occours when the dweebs at the upstream connection point (to the net) tell thier routers to go ahead and route 10.* address "since the rest of thier network does." This is just silly and very unresponsible. There is no problem using non-routables, however it must be done correctly!
Try installing Mandrake with default security level of 5. It shuts down just about everything you can think of. Any services you want, you have to explicitly turn them on.
In Soviet Russia, hot grits put YOU down THEIR pants.
Global Crossing / Global Center filters out all RFC 1819 (or it is 1918?, whatever) private networks on our core routers, as well as customer connected peripery routers. This includes DSL, DS1, DS3, OC, ISDN, and dialup customers.
Customers with BGP sessions are allowed to advertise these networks either.
So ya thought ya might like to go to the show.
To feel the warm thrill of confusion, that special geek glow.
got me some bad news for you, Sunshine.
Roblimo isn't well, he stayed back at the hotel,
And he sent us along as a surrogate hand.
We're gonna find out where you fans really stand.
Are there any MCSEs on the slashdot tonight?
Get 'em up against the wall. -- 'Gainst the wall!
And that one with all the karma, he don't look right to me.
Get him up against the wall. -- 'Gainst the wall!
And that one is in RIAA, and that one's in MPAA.
Who let all this riffraff to have their say?
There's one smoking a joint, and another with spots!
If I had my way I'd have all of ya shot.
(I guess Pink Floyd's going to sue me now)
Make life even easier for people and point that NameZero domain to http://oneilli.net/~sharky/entry/?slashd ot... It'll break away that damn annoying ad banner frame automagically... :-) (and it gives users a choice to keep the frame so NameZero can't get *too* mad before defualting to break it off after 10 seconds...)
--
sharkyfour.com
Just some posting lameness filters were added to discourage the lame trolls who post in all caps and crap like that. I personally enjoy reading the well crafted rants from the creative trolls who try to start flame wars -- I'm sorry but someone posted this one troll yesterday that the web should be a place for marketers to sell and the techie elitists should get lost -- now stuff like that is funny and I thank the troll who posted because I laughed my ass off.
It didn't screw up the code, it most likely blasted the hell out of the MySQL servers, and the code doesn't do a whole hell of a lot of error checking.
--
John Nagle
In a way, I think Slashdot is getting what it deserves. This is the site where the general consensus among posters has been that it's okay to DDoS a site if you don't like something they did. (Remember all the scripts people posted to attack eToys?) Maybe some troll got tired of being moderated down and took the other posters' advice. Or maybe RTMark decided Slashdot is immoral and staged a "sit-in". "Do unto others..."
So when does Kurt get his own weekely column, "Inside Slashdot?"
:)
lf.o
I find it quite amusing that the site that has entered legend for it's own specialized form the of the DDOS (the slashdot effect) has itself fallen fowl of the more malicious variety.
Congratulations on getting the new servers up and running, I've just moved my badtech cartoon site to digital nation (The old location of the slashdot servers).
Well, thanks to the /. crew for finally getting round to telling us what happened - so much for all the whiners who insist that CmdrTaco et al. are involved in some massive conspiracy to keep us in the dark about "important issues" :)
Any possibility on finding out more about the origin of the DDoS? I'm not really sure of the feasibility of doing anything myself.
Could someone point me to a decent networking tutorial on the web?
I use systems, and I understand IP (a bit). I do not understand the stuff between the nodes. Switches. Routers. Hubs. Firewalls. Addressing.
Most people don't have to deal with this crap casue a network guy sets it up and we plug in and use the IP address he gives us, but if I ever want to set up my own network (beowulf lab or home network) I need some more info.
I have also heard that you can directly connect two NICs with a special cable. Do you need software changes to do this?
Sorry I am so clueless.
ed
BEGIN rant
I would definitely look at Exodus for some of this trouble. At times, they have been less than helpful for the service level they claim they will provide.
-They changed their security policy a while ago, and neglected to tell us until after the fact. All visitors to your cage must be announced, and just try to get replacement parts in and out without a whole rigamarole. Previously, one person "on the list" could escort others in and out of the facility, but no more. Granted this makes some sense, but when we showed up the first time after they changed their policy, before informing us, we balked, and complained. The response was (I kid you not) - "Well, we're a big company now, so we can't give the same level of service we used to." WHAT KIND OF ORGANIZATION SHOOTS THEMSELVES IN THE FOOT LIKE THAT?
-Their HVAC is substandard, and they don't truly care what equipment is placed in a cage. I pity the poor sun techs who have to replace the Sun server at the bottom of a stack of 10 other machines (ie, no shelf).
-They continue to abide by their own notification procedures when their "monitoring" software reports trouble. We've gone over their policy several times with them, and verified they had correct contact information for us, and yet they still follow old ways of notification. In this case, it's paging one person instead of using the paging mechanism that contacts the actual people who will do the work - the effort is the same either way.
-The number of times that we've notified them of trouble before their monitors catch it - for example, try working with them to show DNS requests from the outside to their servers aren't being handled.
END rant
I could go on, but I won't.
That was a good account of what happened, but in part two, we want to hear what you are doing to track the bastards down. Knowing how you go about fixing the problem and then tracking down the culprits may help other people who run into the same problem in the future. We would understand if you need to keep the info secret until you have finished tracking them down, or for legal reasons, but at least tell us so.
Or until someone sniffs their router password and blows away their routing configuration....
If by sniff you mean write down while working there. That was an ex-employee, disgruntled and whatnot, that had access to the information. Not a technical exploit, but a social one.
Kintanon
Check out JoshJitsu.info for Brazilian Ji
So maybe he was a troll, but it's an obscure enough subject that somebody would take him for real.... and if I end up giving somebody somewhere the real picture, then I'll have done what I wanted to do.
--
Use the Force
Read the Source
send flames > /dev/null
Only 'flamers' flame!
You'd better watch it with this comment... the MPAA might come after you too!
I'm curious about the timing with the port to the Exodus environment, was there any indication the attack was timed to take advantage of the different environment? Not saying that the security measures were better or worse than the old site, just that the timing seems rather convienent.
More race stuff in one place,
than any one place on the net.
Sengan violated the biggest rule of Slashdot in that everyone can state their opinion on something. That wasn't nearly as bad as how he did it though.
He posted a flaimbait story, and disabled comment posting (the only story to EVER have this done that I know of, and I've been around since quite nearly the beginning of slashdot. Remember TCWWW anyone?), put his flaimbait opinion on it, posted some horribly incorrect information, and expected people to be happy with him about it. He marked it as a news piece, when it was more editorial than anything.
That's why you rarely see Sengan around anymore. After that he was constantly flamed on every story he posted (I think he continued to post for a little while longer).
I really don't see why that should be the case (in fact, it's obviously false), but, considering that the statement rhymes and uses alliteration nicely, I can see why someone might be convinced. Don't you hate it when you can't get stupid jingles like that out of your head? Especially that damned "Mmmm Bop" song...
-NooM
I'm curious on one detail. What was it that the Cisco PIX was supposed to do and didn't?
http://drteknikal.blogspot.com/
that's not true. for example, if you traceroute an erols dialup, you'll see the 10. for the ppp server, and no that doesn't break the rfc. iirc, the rule is you can route from 10., etc but not from (eg, icmp from a 10. is ok, but if you try to telnet to a 10. it shouldn't go past your local border).
Nope, these 10.0.0.0/8 address you see in the traceroute are badly configured machines. For example, it breaks MTU path discovery very often...
I wasn't going to talk about this in public because of /. silence about the DDoS, for I thought things could be somewhat related.
This is what I got this morning when I asked for www.slashdot.org:
<html>
//-->
? name=slahsdot.org&channel=www"
<head>
<title>Not Slashdot.org</title>
<meta name="keywords" content="">
<meta name="description" content="">
</head>
<script language="javascript">
<!--
if (top.frames.length != 0)
{
top.location=document.location
}
</script>
<frameset
rows="*,90" marginwidth="0" marginheight="0"
framespacing=0 frameborder=no border=0
>
<frame
marginwidth="5" marginheight="2"
src="http://slashdot.org"
name=thepage framespacing=0 frameborder=no border=0
>
<frame
marginwidth="0" marginheight="0"
src="http://red.namezero.com/strip2/strip.jhtml
name=pb scrollbars=no scrolling=no
framespacing=0 frameborder=no border=0
>
</frameset>
<noframes>
Sorry
</noframes>
</html>
Weird. Did anybody else see this?
Topology my ass. Exodus fights hard to make you use their 'value add' security services. Be honest guys, the reason you weren't protected was b/c those bastards were working you over for more money and don't want you running your own security, right? In fairness, there's some nice things about running out of an Exodus facility, but dealing with their physical and network security chimps is not one of the high points.
Who was talking about NAT? I'm suggesting that you run your public services on a public IP address and your private services on a non-routable private IP address.
The use of a firewall in itself offers little if any security!
Cool. Does it make me more correct if I use boldface?
And yes, geez, if you have one compromised host it can lead to other hosts being compromised. Should that surprise anyone?
-russ
Don't piss off The Angry Economist
Are you referring to?
; -- the corruption of government starts with its secrets. a truly free people keep no secrets. --
In theory, this is correct.
However, in practice, we have incompetant admins, ignorant management, and underpowered hardware. In many backbone cases, ingress/egress filtering (or, indeed, most any kind of filtering) of these types of IP addresses isn't an option, due to the volume of data that these routers handle. They wouldn't be able to handle it. So, unfortunately, we must rely on ISP's on a more local scale to not only block these packets from coming in to the network (and likely on to their customers), but block them from leaving their network (or, perhaps, keep their customers from introducing them).
Along a similar line, these filters could/should be expanded to include the list of IP addresses that that network services. If done correctly (and down to an appropriate level of granularity), not only will all IP spoofing be eliminated, but anyone attempting to do so can be tracked down rather easily.
The fact that IP spoofing still shows absolutely no signs of abating is proof enough that few ISP's are filtering a damn thing.
The web page servers run Apache+mod_perl+DBI+adsystem module, and the image servers run a much lighter Apache httpd with cache friendly headers.
Where does MySQL sit? Any "reason" behind Debian vs. RH other than "just because"?
MySQL is on the VA 3500 box which is also the Red Hat box. The servers all came with Red Hat and we installed Debian on them, expect the 3500, and I think that was because VA installed extra drivers and stuff we wanted to leave it as is.
Also, any chance you could go through some of the configuration choices made for your apache processes on each of these?
I think this will be in Rob's next post. If not we'll post in that forum.
That was a good account of what happened, but in part two, we want to hear what you are doing to track the bastards down.
Unfortunately, if I understand correctly, that can only be reliably done by manual traffic analysis by the sysadmins of the various routers en route, if I understand correctly. The origins and possibly routes of the incoming packets will have been forged, so you have to actually go from router to router looking for unusual traffic.
Disclaimer: I am not a networking guru.
Various modifications to routing software have been proposed that would make tracking easier (see the recent slashdot article). However, at present you're in for a lot of work and still probably out of luck.
I'm not sure if this qualifies, but take a look at www.dubbele.com
I've had alot of portscans for 31337 and 12345 in the past week on the mediaone network, all from 10.0.0.0/16 networks. I am massively annoyed that they let this through and block ports 137:139. Umm.. is this solving the problem? No! Oh, and they've taken a liking to scanning their customers boxen.. but I digress.
DDoS is the direct result of sloppy upstream administrators. IF I were in your shoes, I would be suing every person upstream for atleast a few hops for passing those 10.0.0.0 packets along for gross negligence.
I know I'm not the only one who would like to see pictures of this whole setup :)
And while you're at it get CowboyNeal to give us a sexy pose *on* the servers (grin)
Machines are cheaper than people. It's easier to configure N+1 machines all the same than to configure N machines one way and one machine a different way.
Not ignoring, just forgetting to dispense with it as an issue.
-russ
Don't piss off The Angry Economist
It's not that I'm a god. It's just that I've seen firewalls and the machines behind them, and I'm unimpressed by the way they work, and I'm unimpressed by the arguments for them.
Why isn't your router blocking traffic with an unroutable source address?
You mean they build insecure boxes and then put them on the net? Why did they waste their time?
-russ
Don't piss off The Angry Economist
Every time I tried to view the front page of /. it came up waaaaaaay funky. 1 - Did anyone else experiance this? 2 - Is there more to this problem then just a DDoS? mcd
----------
No army can withstand the strength of an idea whose time has come.
- Victor Hugo
While I agree that the Slashdot DDoS attack caused many people quite a bit of annoyance and frustration, I think leaving the impact at that is very short sighted.
Firstly, I don't think the blame for this DDoS can be centered on just one person or group. Obviously, those who attacked Slashdot are to blame, as are Slashdot's sysadmins, and the people at Arrowpoint. And secondly, the costs of this are much greater than you might think.
I have an eight year old daughter. We had a family pet - a rabbit, black, named Midnight, and my daughter was very fond of it. Midnight, sadly, passed away about two months ago. A week or two after Midnight died, my daughter came to me in tears and asked me, "Daddy, why won't God bring Midnight back? I've been praying like Deacon Simmons told me to."
Naturally, I had to think about how to respond to this. I finally answered, "well, honey, God is a little like Slashdot. He can seem arbitrary, cruel, and unresponsive, but he's really a nice guy who's just a little out of touch and is a little slow at responding to requessts."
This was fine, and I thought that would be the end of it. However, when Slashdot went down last week, my daughter burst into my den, positively sobbing and wailing, and managed to choke out "Daddy! Daddy! I can't get to Slashdot!" "Honey," I said, "it's just a website." But, between sobs, she said, "but you said God is just like Slashdot, remember? Does this mean God is dead?"
I tried to console her as best I could, but nothing seemed to work. When Slashdot came back up, she seemed to return to normal, but she hasn't been quite the same since. She doesn't ask me about God so much any more, and she seems less interested in Church.
As a good Christian, I will turn the other cheek, and not call for the punishment of those responsible. But to the heinous criminals and negligents responsible for this, I must ask, how do you feel about destroying a small girl's sense of innocence and wonder about the world? About crushing her childish dreams and idealism? About shattering her faith in God and his benevolence? About possibly having crushed her soul and emotion forever, leaving her to live the rest of her days in spiritual agony as a broken, scarred husk of a person?
I hope all of you think long and hard about what you've done. What is the soul of a child worth, next to a few double-checks of the router?
Thank you.
I'm sure that these great enemies of the Slashdot Empire have found this to be a convenient time to strike. We must systematically seek and destroy all those suspected of having sympathies with the MPAA, RIAA, or Microsoft for security reasons.
Therefore, all
Windows users
CD listeners
Movie watchers
Metallica fans
are asked to please leave now or face prosecution.
thank you.
Which operating systems forward source routed packets or tunnel packets without explicitly being configured to do that?
You say it's weak security, but you come up with a weak example of why it is.
-russ
Don't piss off The Angry Economist
No, I'm not trolling. I haven't seen an rationale for a firewall which is any better than "Well, we're too stupid and lazy to lock down N Unix hosts, so we're going to lock down one. Somehow we will become less stupid and lazy because there is only one machine."
If I can secure a firewall that I control, then I can secure a firewall that I control.
If X then X is true every time, but it's not much of an argument for a firewall.
I can't prevent the group behind the firewall from introducing vulnerabilities on their side of the street
If they're in public services, you're toast *anyway*, because your firewall is letting those services through. If they're in private services, then why for God's sake did you bind them to a public IP address???
Most of the things that people are using firewalls to protect against can be solved by using non-routable IP addresses and some small amount of filtering on your router.
-russ
Don't piss off The Angry Economist
This is different from mostly-passive traps like teergrube (FAQ; jargon) or Deception Toolkit or spider traps which sit around waiting for Bad Guys to attack them and react unexpectedly when attacked (e.g.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Maybe you should type more carefully, since you
:-)
/pyder.....
requested http://slahsdot.org (slaHSdot) not
slashdot.org...
I registered that domain (for free @ namezero) to
help the people who couldn't type. Sorry if I scared you
Cpyder@slahsdot.org
_
/
\_\ sig under construction
waiting for Rob to toggle in a boot loader to IPL from the punch card reader?
try { do() || do_not(); } catch (JediException err) { yoda(err); }
RSM - Route Switch Module
- Basically a router on a card in the switch for routing between VLANs
MSFC - Multilayer Switch Feature Card
- Once a route for a packet flow is figured out (from the first packet going through the router) all other packets from the flow get switched instead of routed.
-- Ed Bugg --You have freedom of choice, but not of consequences.--
And FreeBSD is immune to this effect? How can this be? Even if 50% of all FreeBSD users are experts, and 10% of all Linux users are experts, there is still (as I said earlier and it's still not a troll) more Linux expertise.
-russ
Don't piss off The Angry Economist
SYN flooding is a solved problem in modern Linux kernels. Try again.
-russ
Don't piss off The Angry Economist
Sorry. Make that mid 90's...
jf
Ya know what? You've gotta be right. It must all just be coincidence. MS would never stoop to making a coordinated attack on its critics, right?
Russ, why don't you stop being coy, just cut to the chase and say "FreeBSD sucks, Linux rules," and get it over with?
/. It seems more likely the point was to deny service. Which it did.
Not that I agree with that sentiment. They both rule, but in slightly different ways.
You also seem to assume the point of this attack was to own
Both Linux and *BSD are capable of operating in secure and efficient modes, and both are capable of being operated otherwise. So let's get past that.
Besides, the issue here has to do with network devices more than OSes anyway.
-------
Bill Gates Is My Evil Twin.
All of these machines were behind an Arrowpoint (CS-100) firewall/load balancer which took it on the chin when we got DDoSed, so basically the Arrowpoint was taking the full force of the attack. So as described above we replaced it with a CS-800 and a BSD firewall.
I guess we learned that if you're going to post a letter from a Microsoft attorney on your web site the same day you implement a few new troll filters you better be prepared for the fury of hell to rain down on you. Then again this is Slashdot, so we always should be prepared for the fury of hell to rain down on us.
DDoS is the direct result of sloppy upstream administrators. IF I were in your shoes, I would be suing every person upstream for atleast a few hops for passing those 10.0.0.0 packets along for gross negligence.
Um, no.
DDOS simply requires that a lot of compromized boxes be able to send you packets. Spoofing to non-existant return addresses is an orthogonal issue. You reply that it's used to mask the souce boxes? Any _valid_ address could also be used for that, so filtering would gain you nothing against that.
I agree that filtering of reserved addresses should be done, but that would not hinder a DDOS attack.
It matters!!
They should have used Open BSD!!!
----------
Stupid sexy Flanders.
Don't get me wrong. You guys (and gals) are WAY more knowledgeable about this stuff than I, and I don't want to seem like I'm denigrating your technical skills.
BUT. You didn't figure out what was wrong. You replaced some hardware and "it seemed to work". If this WAS a DDoS (which the floods and IPs seem to indicate), then the hardware problem was a symptom, not a cause. In which case you're still open to further problems.
Or is this firewall supposed to block the flooding? How is a FreeBSD desktop firewall different than the router (or whatever) you put it in front of?
--
Have Exchange users? Want to run Linux? Can't afford OpenMail?
Linux MAPI Server!
http://www.openone.com/software/MailOne/
(Exchange Migration HOWTO coming soon)
Charles Spurgeon's Ethernet Web Site
Jason Schwarz Ethernet Tutorial
Lantronix Networking Tutorials
You might also try typing "ethernet tutorial" or somesuch in your favorite web search engine. Hope this helps!
--
Okay, I got Linux installed. So where's the free beer everyone keeps talking about??
net3-4-howto
firewall howto
masq-howto
I have also heard that you can directly connect two NICs with a special cable. Do you need software changes to do this?
Yes, you can do this with a crossover cable and no you don't really need any special software to do this. I use one when I bring my laptop into work and want to hook it to my workstation. You can either make one yourself or buy one at any decent site like hardwarestreet.com.
Sorry I am so clueless.
Anyway, good luck.
There is much cruelty in the universe, John.
Yeah, we seem to have the tour map.
Oh, come on! Everyone uses 192.168.0.0/24. We're going to wear out this class C if we're not careful. I propose having the third octet be your house number mod 256. My home network is now 192.168.192.0/24 (I live at # 10432).
--
E_NOSIG
Posted by BSD-Pat:
The problem here is that we only had one subnet to work with. The PIX we had wouldn;t to the type of filtering/bridging that I wanted.
Cisco wants a DMZ on these things.
I needed a bridge...why I didn't use linux...
It was quicker and easier for me... ipchains has always been a pain in my arse... ipfw and ipfilter I know best.
The other thig is that we fried an arrowpoint cs-100 (little itty bitty dinky thing that was being replaced with a bigger one)
the little arrowpoint couldn't take the traffic of 109Mbits , it wasn;t meant for that, we were waiting on arrowpoint to ship us the unit we were *supposed* to have.
*BSD fills the gap because I know it inside and out, and it was the quickest to get up at that point.
As far as the router, we can't do any type of stateful filtering on the 6509, due to some setup that exodus has with the HSRP stuff, I'm sure given enough thought I could figure out how to do it, however we were running on crisis mode.
The BSD firewall filled that gap for us...I can now do access lists on that, instead of the cisco.
and we still have a "DMZ" but its on the same subnet.
The arrowpoint CS-800 was emergency shipped to us that afternoon....its about as big as a cisco 6509...and ummm won't die under that type of traffic/content checking (its layer 5 remember)
-Pat
Why not do IOS Load Balancing from the 6500/MSFC itself? You can use SYNGuard with the Load Balancing to protect against SYN floods... refer to: http://www.cisco.com/univercd/cc/td/doc/product/so ftware/ios121/121newft/121limit/121e/121 e1/iosslb1.htm#xtocid446613 -- Anonymous Cisco Employee
Because FreeBSD doesn't suck, it just doesn't have (tada!) as much expertise available.
-russ
Don't piss off The Angry Economist
Why aren't you guys setup with Above.net instead? They are an entirely BETTER organization. AND they let you do pretty much anything you want with your setup and will keep it humming along for eternity or until your check bounces.
Kintanon
Check out JoshJitsu.info for Brazilian Ji
By the same logic:
This is Unix, get real. The logic is completely different. Arguing by analogy is, like, stupid.
I think the point is that they have one really good BSD guy. That makes BSD expertise 'more available' to Slashdot than whatever else.
You're right, that's a good point, but how does that make what I wrote a troll?
-russ
Don't piss off The Angry Economist
Posted by BSD-Pat:
even more wacky, we were getting stuff from 0.0.0.0/8 (gee, how the F#@% do you filter that??!?!) lets filter the equivalent of "any", gee...
we have been talking to Exodus to get this problem resolved.
Liar! How dare you say I look like an Arrowpoint rep!
$1million/year - is that just for Slashdot or all the Andover.net systems?
1,000,000 USD = 671,015 UKP = 1,121,428 EUR BTW
Richy C.
--