Slashdot Mirror
The Slashdot DDoS: What Happened?
What follows is more-or-less Pat "BSD-Pat" Lynch's account of the DDoS... Pat is our super 31337 BSD Junkie sysadmin. He wants everyone to know that the timeline below is little screwy, but things are more or less in sequential order. Things might not be exactly perfect, but hey, what do you expect after 30 hours without sleep?
Having moved the day before, none of us were truly familiar with exactly how the new hardware would handle the full burden of being 'slashdot.org'. The cluster (known affectionately as The Matrix) had handled its premiere day with flying colors, but we didn't really have an accurate feel of how things would react. Combine this with a couple of extremely high traffic stories posted on both Thursday and Friday, and it took us a awhile to determine that the problems were external, and not a flaw in some new component in the cluster."
The Attacks began Thursday morning. Most of it came in the form of SYN floods, from obvious /16's no less, and some /24's. We didn't have any zombie-killing software or a firewall installed because of certain network topology issues. Later on, a second wave came, this closer to 8 or 9pm and the load balancer (an arrowpoint CS-100) died under the load.
The DDoS, as far as I could see, was a lot of SYN and Zero port packets coming from various /16's and /24's as well as a bunch of RFC1918 reserved addresses (10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16) At one point we reached 109Mbits worth of traffic into our network.
Liz and I went back to Exodus and rebooted the Arrowpoint, then the site seemed "ok" for a bit. By 3 in the morning, Liz decided that the PIX (Cisco's firewall) could simply not do what it was supposed to do, so we went back and started building a FreeBSD box as a bridging firewall.
just before we went to plug it in, I tried to ssh into the vpn-gate and noticed that nothing was working right: while the site worked, outgoing traffic and source groups on the Arrowpoint was screwed. As if that wasn't enough, two ports died on it already!
At some unknown point (time blurs after 30 hours straight!) Martin and PatG show up (thank the gods!) and they force us to go to sleep, they bring the site up outside the Arrowpoint, while Liz and I watch from a hotel room.
As of Friday morning, the site is semi-working, but the adsystem can't be updated, and we have no access to the backend servers. I scream bloody murder to Arrowpoint, who eventually shows up to blame the router: a cisco 6509 switch with two RSM/MSFCs.
Liz and I do packet dumps and determine it's not the router, the little CS-100 had died the night before, and thats where it all started. The Arrowpoint guy insists we did something to make the Arrowpoint not work (CT: Explicit description of precisely where Liz and and Pat wanted to store the newly deceased Arrowpoint removed to keep things rated PG) By 7 the CS-800 CSS is up we're almost done for the day, but we stay to make sure. By 10pm we're exhausted but stable, although we're running 4 servers on a round-robin DNS while the new load balancer waits.
Netops (Liz , Martin and I) regroup, and do reintegration of new Arrowpoint CS-800 and installation of a new FreeBSD Firewall box instead of the PIX during Saturday Afternoon. Slashdot returns to normal. Sysadmins get well-deserved sleep.
So that was the story. It was a pretty hellish weekend for everyone involved, but thanks again to those that helped get our ducks back in a row. Again, Part #2 to this (which originally was gonna be run last Thursday, but with all this ddos stuff got pushed aside) is a fairly detailed description of the new Slashdot setup at Exodus, complete with all the changes mentioned above. Fun for the whole family if your family is really into clusters of web servers."
- /8's,
/16's and /24's - SYN
- PIX
- Arrowpoint
- RSM/MSFC
- CS-100
TIAThere's nothing wrong with Linux (and don't you dare suggest there might be! jihad! jihad!) but we just happen to have network admins who could whip up a FreeBSD firewall for us in a pinch so yeah, whatever works, as long as we don't use an OS from a company that tries to corrupt an open protocol then sends us nasty lawyers letters about it.
Now I know how a heroin addict feels!
/me stops shaking uncontrolably
-- From my Best Friend (Written to me over ICQ): "i was gonna go to a party...but i had to reinstall windows"
The simplest case is building two small walls instead of one humongous wall. If you build a humongous wall, it takes a long time to get through... unless the enemy finds a single weak point -- then you're screwed. Two walls each take less time to get through, but if they're well-built using different techniques, the enemy may not get through to begin with and if they breach the first they lose time covering ground and then adapting. They're also very obvious as they traverse the open ground between barriers.
Network security can benefit from the same concept. Others have already mentioned heterogeneous "airgap" systems -- one of the most common and least excusable faux pas by so-called "security admins" is a single firewall protecting a herd of boxen. Second to that is identical airgap firewalls.
Of course real defense doesn't end with the walls. Even services running behind an airgap should be structured with an eye toward reasonable security, as others have pointed out. Many companies think their firewalls make them safe; come the day those firewalls are breached and the attackers make off with everything stored on the NT intranet server before wiping the drive, they'll find out differently.
Any server, no matter how well shielded, should start life in a lockdown configuration and then be made less secure only as needed ("do we really need to enable daytime on this box?"). Admittedly I haven't kept up with developments in secure distros, but does anyone make a "locked-down by default" distro based off Red Hat/Debian/*BSD? It'd be a real service to admins and if not it's something I might consider starting a project for. I know of Bastille Linux but that's (as far as I know) not so much a distro as a set of scripts to tighten up Red Hat.
The only thing we have yet to figure out is how to effectively make systems under attack "shoot back". The most they can do at the moment is call in an airstrike (i.e. alert the admins). Any return-fire capability would only be as good as the intermediate links let it be. It might not even be a good idea, as it would increase network traffic and make the attack that much more severe.
-- Old Man Kensey
Carpe diem!
Yes, I know how to use a search engine.
/. search for old sites are two of my favorites.
/. typically has a bunch of people reading that know how to quickly get me to the info I need. They have done this before, so why duplicate the effort.
/. could have a couple of old /. links and a few tutorials that would be useful for all /. stories, but then people would have do actually do some work to post a story.
Google and
I just know that
As a suggestion for posting stories, a lot of news sites post links to what's related for a given stories. Maybe
Thanks for the info, everyone-
ed
OK, ok, everyone point fun at me for being dyslexic! And anyway, how do you know they weren't falling over chickens!
Ok, so I'm dyslexic and get the spelling of words wrong sometimes (which a spell check helps with) and sometimes use the wrong word (which it doesn't) but there is a deeper issue here. Language is simply a means of communication, if the message is communicated then it has done it's job. Furthermore language is not defined by text books and dictionaries, these books record it. There is only something wrong with a statement when it fails to convey the intended message, not when a word is incorrectly spelt or a comma is out of place.
Referring to me as 'an ignorant looser' is nothing short of bigotry, if you really want a discussion I suggestion come out from behind the aptly named 'anonymous coward' hiding place, reveal your identity and discuss the matter without resorting to insult.
I know, I know, but I can't help it! I'm a junky!
As a good Christian, I will turn the other cheek, and not call for the punishment of those responsible. But to the heinous criminals and negligents responsible for this, I must ask, how do you feel about destroying a small girl's sense of innocence and wonder about the world? About crushing her childish dreams and idealism? About shattering her faith in God and his benevolence? About possibly having crushed her soul and emotion forever, leaving her to live the rest of her days in spiritual agony as a broken, scarred husk of a person?
Well, idiot, it's your own damned fault for telling her that God == Slashdot! When Slashdot crashed, as it was going to eventually, she equated that with God. Now, she believes that God crashes, too. The infants that DDoS'd /. aren't responsible for scarring your little girl, you are, you hypocrite!!!!
As a satanist, you wouldn't have the problem of reconciling an all-powerful and benevolent deity with a world full of evil. The deity you worship would be evil as would the "good" deity that is your divinity's enemy, capisce? Therefore, all would be right with the world, since all is evil. Maybe you should change religions to save your daughter's sanity?
Besides Hell uses OpenBSD! Satan knows what he's doing!!
:-) - for the parody impaired.
Just be sure to wear the gold uniform when you beam down -- you know what happens when you wear the red one.
"They are probably much better off with the BSD box. Although it's not a good idea to advertise their security infrastructure layout to the world. (Hint, Hint, CmdrTaco!)"
I disagree 100%. Knowledge of an installation's infrastructure should never comprimise the security of the setup. If it does, then you're relying (to a certain extent) on security through obscurity. Security should be provided by a well thought out layered approach: network layering (multiple firewalls, screening routers, IDS, etc...), host-based security (tcp wrappers, service minimalization & replacement, tripwire, etc..), and application security (ie. authentication, verification, etc...)
In designing networking/server infrastructures it's best to think of it as an open source project, and you should be willing to get opinions and discussion from any number of sources that could include crackers who may at some point want to use that knowledge to attack your site. This is one of the things I like about TIS Gauntlet once upon a time..."crystal box" was the term they used to describe it.
You should prepare for an attack ASSUMING that the infiltrators know as much about your setup as you do. In the long run, if you know that your infrastructure can hold up to someone with that amount of knowledge, then you'll be doing pretty well.
My only question...did I actually see in a comment that they're using NFS to publish data to the distributed webservers??? Ew. Run.
-buffy
(Hmm...I seem to really like parentheticals, don't I? (well maybe not. (really!)))
Probably because they're running things behind the firewall like NFS and some flavor of SQL which won't be secure enough to expose to the Internet anytime soon.
it was the router with the lead pipe in the library that killed colonel slashdot!
Woke up- smoked some pot checked slashdot- but it wasn't there, so I smoked more pot and surfed news sites. downloaded some mp3's, listened to them, hooked up to CNN, checked for slashdot smoked more pot, went to work checked for slashdot, occassionally getting something, no customized page, tho. went home, smoked more pot, checked slashdot. nothing. so I started working on some music project I have going. smoked alot more pot... went to sleep woke up. repeat, except for going to work.
-- "Perceptions create reality. By changing your perceptions you change your reality."
Exodus's "Value Added Firewall Service" is not a PIX Firewall, it's a certain sun based firewall..
{sic} he he
What do you despise? By this are you truly known.
Who knows, even Bill may be a /. reader?!
I'd be suprised if he wasn't. I just wonder if he posts.
numb
I'm using 192.168.0.0/22 at work (with 192.168.0.0/24 for the servers and 192.168.1-3.0 for the workstations) and 192.168.2.0/24 at home .. enough to keep this net from wearing out ?
--
Ignorance is no excuse
Ping is not a good indicator as to whether or not a site is up.
I proved that for certain at home last night. I have this little Pentium 133 box down on the workbench that runs Windows 95. (I know, I know, W95 is icky. It's a machine that talks to an EPROM programmer, a ROM emulator, other emulators and embedded development stuff through printer port kludges- not gonna work anytime soon with a Freenix.) Anyhow, last night I was upstairs in the office trying to connect to the C drive on that machine (wanted to pull up a bitmap of President Eisenhower that was being used as the wallpaper on that box.) It would ping just fine, but I couldn't connect to the shares. I went downstairs to figure out what the heck was wrong. The power cable to the hard drive had gone intermittant and the box was bluescreened with a fatal (cannot access C drive) error.
Yep. Basic TCP/IP services were running just fine under that bluescreen.
Well, enough of a rambling folkloric story for now.
Keep your power dry.
"There are currently six VA FullOns serving web pages from an NFS server, and three other web servers serving images. "
Far be it for me to question, but NFS?? Ew. If someone does penetrate your infrastructure that's just asking for trouble.
Anways, NFS' performance leaves a lot to be desired. Wouldn't it be better to just publish the data to each server? With 100/1000 Mbit networking even large datasets can be propagated quickly.
-buffy
I was tring to post a message like this last Thursday but at the time slashdot was down...
You can't invent more CPU speed by dedicating one machine to filtering packets. The same amount of CPU speed could be used to serve pages.
Quite true, however by dedicating one machine to security you free up the others to serve pages. Remember, Linux is multitasking, so every app running slows down the others a little bit. By freeing the page-servers themselves from having to worry about security, you let them do their task more efficiently. That speeds up the process more than simply throwing more servers at the problem.
You can't create CPU speed out of thin air, no. But you can make the process more efficient and speed things up that way.
Not knowing if that's a joke or not...
/") for all of your unix boxes because it takes time to setup accounts, etc.
I may sound like too much of a bastard, but not having time is not an excuse, you aren't doing your job. Each of those routers had to be configured to begin with, and most networking guys keep the entire configs in a text file that they upload to a router, add a couple of lines to the code and your done. Not doing this stuff is akin to doing a ("chmod -R 777
It's amazing how much time the admin seems to get when a site realizes that 80% of a T3 is full of bad traffic (the old saying and ounce of prevention...). If you don't have time to do this type of stuff, you need to have a serious talk with your boss; because sometime soon you are going to spend a whole week cleaning up some crap that would have only taken you a couple of hours to do in the first place (not to mention boss yelling, legal dept. yelling, ceo yelling, customers yelling...).
You even got somebody to moderate you up as "Insightful."
I normally don't care for this kind of stuff, but your tolling post about how everybody falls for trolls was a brilliant work of post-modernist art.
Kudos, Anonymous Coward.
Information wants to be anthropomorphized.
is it me, or has slahsdot been slashdotted?
nal 11
I like the idea of 192.168.192.0/24, but I don't know about the extra power consumption needed for those extra bits. It would keep the electrons from being bored though.
Sorry, I'm without a binary capable calculator right now, or I'd calculate the extra power needed over a year to maintain those extra bits, with an average network load of 5%...
l8er!
"History doesn't repeat itself, but it does rhyme." Mark Twain
This is something I've been wondering about recently. How do you have clustered web servers sharing storage? Sure, use NFS you say. But that introduces a single point of failure. If your NFS server goes down, you lose the entire cluster. Are there any solutions to this that don't involve spending vast quantities of money on a Sun HA failover system or an Auspex mirrored NFS system or similar?
"The invisible and the non-existent look very much alike." -- Delos B. McKown
This is a very misleading post. First of, it's 10.0.0.0/8 not 10.0.0.0/16. Second, the only net you could remotely finger is the originating net for not doing egress filtering on the private nets. Everyone else is just routing based on dest IP and switching based on the data link (MAC) info. But there's no requirement for them to be doing that. The real fault lies with the local network engineer for not doing ingress filtering of packets with a source on a private net. You've got to take responsibility for your own misconfigurations. You can't blame everything on somebody else. They should have had a firewall in place and Exodus should have been doing the ingress filtering at their border. See my other post for a suggestion as to why this wasn't happening.
hmm, I got this error on Sat. I was getting all excited cause the page was actually loading but then I killed Slash. I'm a bastard andy j.
Stupid Cheap Guitars
---
OSX is covered cause even tho apple is a hugely proprietry company, everyone here loves microsoft competitors.
---
Well, Apple was a Microsoft competitor long before Slashdot started seriously covering them. I think it's more the hardware and Unix-based nature of their recent OS movements more than anything. Note that OSX is based on Mach/BSD, which goes to show you that they're not focused on Linux only.
---
Anyway, point was slashdot IS primarily a linux site.
---
If by that you mean that most of the people here have an interest in it, sure.
I'm not saying that Slashdot isn't incredibly biased toward Linux, but that doesn't mean the Slashdot editors won't use *BSD when the occasion warrants.
Anyhow, Slashdot may be Linux-oriented, but nowhere do they say that they are so to the exclusion of everything else (which was the point I was arguing).
- Jeff A. Campbell
- VelociNews (http://www.velocinews.com)
- Jeff
Exodus is getting $1million/year from us so they let us do whatever we want. They only thing they won't let us do is take a picture of our cage -- no cameras allowed anywhere in the facility! I guess they're afraid we're going to steal their soul. We were able to smuggle out this picture of PatG, PatL, Martin, and the Arrowpoint rep. Behind them you can see the current Slashdot setup.
In general, Linux is bad with TCP/IP traffic. This includes, as you could guess, firewall and routing. This is also why Linux doesn't completely overshine NT in those web server tests.
Of course, if you are on a cable modem or other mid-bandwidth, then it should definitely be fast enough. I'm guess slashdot has a tremendous amount of traffic to deal with, and probably has a big pipe.
The BSD's are generally looked at as more secure and better performing firewalls.
Personally, I don't like the linux ipchains. The policy file (text editing) is not friendly when you have a complex set of rules, multiple hosts with different rule sets. And the reporting is bad. Catting a logfile is not how I want to look for attacks. A GUI can be better than the command line, if the GUI is done right (key point: done right). Creating the policy file and reporting is a major factor with every firewall vendor out there.
If I was going to use a free (cost) software firewall, I'd go with an OpenBSD box myself.
Mike
-Denor
There are a variety of ways to trace DoS attacks back using the current infrastructure, including the 'manual traceback' technique that Christopher alluded to. However, they don't work very well for DDoS.
For DDoS, tracing back to he source still isn't good enough, as 'here's a list of 10,000 hosts that have been co-opted to do a DDoS' has made the problem simpler, but still pretty difficult (stopping those hosts from doing it again, making sure that a different set of 10,000 hosts are co-opeted, determining who co-opted the hosts in the original place, etc.). Also, I'm not convinced of Savage's trick with chunking working very well when you're talking about 10,000 traces.
Why are you installing a Unix-based firewall in front of some Unix-based public servers? Why not secure the servers in the first place?
-russ
Don't piss off The Angry Economist
Just have a couple of random Indonesian students thrown in the slammer, and we'll all feel safe again.
--
Sheesh, evil *and* a jerk. -- Jade
called them up they said no problem, it was a bug in the ios I need to upgrade. well i upgraded same thing. I was promised it could handle this kinda traffic. I didn't see it. Riped it out replaced it with Nortel and everything is fonzie. It was my only cisco device and i think will be my last. they are getting bad as Microsoft anymore.
comming to you straight behind all Nortel!
Hahahahahaah! You Andover folks are more 1337 than I thought. Not only do you have uber-hacker John Walker on your team, you're running the site on a Univac 1107 -- say, you have any of those old 2 1/4 ton 100MB hard disks?
I am quite civilized, and I should be brought a beer immediately. -- Bruce Sterling
Exodus is full of people that don't pay or want the world for nothing. Digex on the other hand has great support, strict security and very effecient services. (No restrictions on bandwitdth, backup hardware/routers/nics/switches and enough power to last weeks after a nuclear strike :)
Go digex
#1 Trolls WANT to be moderated downward. I am a troll, I know. If your troll doesn't get to -1, you have failed. The solution of course is to eliminate downward moderation. This creates more points for upward moderation thus making the task of finding good posts easier.
#2 Many dissenting posts have both upward and downward moderation. The result is that a controversial post often goes nowhere. Personally, I'd much rather see a post that was mod'd up 4 times and mod'd down 4 times than one that is at 5.
#3 Registered users posting as AC. I used to post AC all the time, but I registered for the convenience of customizing / and haven't posted AC regardless of kharma - see my history I still troll. Why do SO DAMN MANY registered users put anything of dissent as AC? You fear downward moderation. You want to tell someone to "fuck off" because they debunked your post, but you post AC. Pathetic. Perhaps to scare the kharma whores, AC post moderation should be linked back to the user so only their ID is masked.
I still wholeheartedly support Anon posting. Though I make exceptions for kharma whores who are protecting a worthless number in a database.
Before anything else, I laud the Slashdot team for quickly resolving their network access problems. I've been through something similar and can appreciate the complexity of diagnosing these problems.
..
I spent a statistically average stint deploying servers at exodus for an ecommerce site. I had a couple of odd experiences that I could not completely explain.
First, I, too, received 10/24 packets from the Exodus network that Exodus denied could be coming through their routers. I was using a simple 2600 series router with ACLs to do my filtering/firewalling, so it was quite clear where the packets came from.
I worked with three different load-balancing devices including the F5 and the Alteon Networks devices. None of them met my evaluation criteria of sufficient features and believable reliability. I bought the Alteon device because I could make it work within four hours and it appeared to have the right features. Then, the trouble ensued. The device failed us, under a very meager load, not because it wasn't fast enough, but because the features of the device interact in very complex ways. The fundamental problem is that the mixture of a switch, router and load-balancer in once device doesn't agree with IP. ITOT, we had to use it for all three even though we had a router that we preferred to use as...a router, dummy.
Lastly, we used a Debian GNU/Linux box as a gateway through the firewall using the cipe VPN code. This was the most reliable piece. Although, to be fair, it wasn't maintenance free. There were times with the VPN link would drop without explanation. Fortunately, we never lost contact with the box and so we could always nudge the link.
Why don't you moderate UP posts instead of worrying about crap nobody would read unless they were INTO the troll discussion?
Extremely long, over-intellectualized ("the moderation system takes this subconscious prejudice and places it at the level of doctrine"), rambling post by an AC about something everyone already understands. Add in the irony factor of discussing (on one level) what it is doing (on another level) and you have near-proof.
/.'s future if I could see some stats on moderation/meta-moderation: are we losing idiots faster than we are gaining them?
I'd have a lot more faith in
--
Have Exchange users? Want to run Linux? Can't afford OpenMail?
Linux MAPI Server!
http://www.openone.com/software/MailOne/
(Exchange Migration HOWTO coming soon)
Because what you don't see when you come to this site, and possibly look at the code, is that all the pages are dynamicly generated. I can't be sure, but I'm guessing the 'sections' on the front page could be located on more then one server. And the artciles are in a DB on another server, so if that can't be reached, you just seriously chopped down the size of your resulting HTML. (And output, since there is no longer a middle) =]
The webpage where I work is located in 5 different files (PHP), and joined together when the user loads. But when it's all together, and you look at the source, the page looks like it should be one file.
the obvious question: does it run Beowulf? (I'm posting this with my debian X-Box 18 months before MS releases their UNIX/X-Free Game box... weeee!)
---
Has anyone opened up a PIX? We had one at work once and opened the sucker up.. It's essentially a PC.. I think it had 4 PCI slots, with normal Intel NICs in them.. There was a flash card in place of a hard drive..
You can accomplish the same thing (better, baby!) with a PC, OpenBSD, and Zebra.
The source is forged
Where? At SourceForge?
MSFC
On Cisco equipment this is touted as Netflow and on Cat5500s the feature resides on the Supervisor Module (III) - This is in Slot 1.
RSM
Essentially a Cisco series 7000 router, without any physical ports (except the backplane I suppose), like others have pointed out to route between VLANs within the VTP Domain of Cisco switches.
I have found that routing is pretty slow (as least for today's 1gb/100Mbit LANs) so you're probably better putting Gigabit cards on your servers that are capable of VLAN detagging, then just let the switch switch packets by physical address.
A lot of corporations will use the 10.* address space for thier internal networks... with routers and switches and the such. Most routers will not allow 10.* addresses to be routed unless specifically told to. Unfortunately, the problem occours when the dweebs at the upstream connection point (to the net) tell thier routers to go ahead and route 10.* address "since the rest of thier network does." This is just silly and very unresponsible. There is no problem using non-routables, however it must be done correctly!
Two words: NameZero Sucks. Their service is *always* down... As soon as I get some extra cash I'm taking sharkyfour.com away from them...
--
sharkyfour.com
I always thought that it went like this:
1 blue
2 blue/white
3 green
4 orange
5 orange/white
6 green/white
7 brown
8 brown/white
i.e. only pins 1,2,3 and 6 are used by ethernet. And the twisted pairs need to be matched to 1/2 and 3/6. To make a cross over cable, these are the pairs that you have to switch.
http://www.cisco.com/warp/public/779/edu/academy/c urriculum/demo/curriculumde mo.html
I'm in the CCNA training program and they're offering the first two lessons off their website for free. Really good info and nice looking flash.
-Hi
Thank you for sharing your troubles with your faithful readership. Aside from my deepest sympathies for having been attacked by some asshole, I congratulate you for trying your best to get back to service and also for letting us know and not trying to hide things away.
:)
We're on your side. And if you manage to find out who did that, give us his/her IP
Linux 2.2 does not. IP Filter outclasses IP Forwarding and IP Chains.
The message on the other side of this sig is false.
And what about the people who need to legitimately need to route those addresses? A router is not going to know whether its on the public Internet or a private network.
Oh, and by the way, it's 10.0.0.0/8, not 16, along with 172.16.0.0/12, and 192.168.0.0/16. Probably also should block 127.0.0.0/8, 0.0.0.0/8 and 255.0.0.0/8 for good measure.
Then MSFT makes a press release and publicly offers to help / run their site. They offer free software and maybe even kick-in for some hardware. Not to mention the fact that open source and Linux get bad press (due to very poor editorial control and the moderation thugs) / = Linux. This is how the media sees it. That is how I see it. It doesn't matter what % are running other OS's.
If you like MSFT and visit / then be prepared to ride in the back of the bus. It' more fun in the back anyways.
Who ships a router with egress filters in place? I'm not aware of anyone doing this, but would love to know who is. You're responsible for your net and what kind of crap passes its border. You can't blame the 'upstream dweebs' for your own inability to configure a router!!
Recall that a Cat5 cable is usually:
1 - whiteOrange - 1
2 - Orange - 2
3 - whiteBlue - 3
4 - Green - 4
5 - whiteGreen - 5
6 - Blue - 6
7 - whiteBrown - 7
8 - Brown - 8
Important points:
all pins connect "straight": 1 to 1, 2 to 2, etc.
1 & 2 are a pair; and 3 & 6 are a pair.
4 & 5 and 7 & 8 are pairs too. They're not used by Cat5, 100Mbit; but they are used by Cat5e, 1Gbit; so hook 'em up now to save trouble later.
Now, 1 & 2 are the "transmit" pair; think of one line as the signal, and the other as the ground return for that signal. Each line needs its own return in order to get the groovy benefits of the magic of twisted pair. Similarly, 3 & 6 are the "receive" pair. When you hook a PC to a hub (the usual state of affairs), the "receive" and "transmit" are the other way 'round, so the PC transmits to the hub's receive, and vice versa, so all is happiness.
But when you're hooking two PC's together direct, if you used a straight cable, you'd be hooking one PC's transmit to the other PC's transmit, and the same for receive. No workee. So we swap those pairs:
1 - whiteOrange - 3
2 - Orange - 6
3 - whiteBlue - 1
4 - Green - 4
5 - whiteGreen - 5
6 - Blue - 3
7 - whiteBrown - 7
8 - Brown - 8
So that's 1 -> 3, 2 -> 6, 3 -> 1, 6 -> 2.
4 & 5 and 7 & 8 are again not used, but hook 'em up anyway.
When you test your cable, you can buy a cheap cable checker that shows a little light for each line (try Weidmuller / Paladin Tools), or a multimeter, and this will tell you whether you have continuity on each line. However, to test the cable properly, you really need a much more expensive checker that tests if it's gonna work at 100Mbit. After all, at 100Mbit, each bit is only 3 metres long!
Thier
license makes it easy for them to fragment should vendors decide to start improving it.
that's funny considering that everyone and their brother has their own rpm based dist. the bsds are MUCH more organized than the various linux dists. linux is the one that's in serious danger of fragmenting. for some strange reason, everyone wants their own dist. it's just stupid and it's massive duplication of effort.
debian, redhat, and slackware are all the linux i'll ever need (forked arch specific dists not withstanding; eg, since redhat themselves don't support ppc, there's nothing wrong with the ppc (rpm based) reference release).
Try installing Mandrake with default security level of 5. It shuts down just about everything you can think of. Any services you want, you have to explicitly turn them on.
In Soviet Russia, hot grits put YOU down THEIR pants.
1) Several people have asserted that a firewall somehow magically has more resources to deal with an attack. Sorry, no. If you have N+1 hosts, calling the one a firewall doesn't create more resources to deal with an attack.
2) A firewall breaks the end-to-end communication paradigm of the Internet. The idea is that you place smarts in the middle. Sorry, no. Hosts should communicate with hosts, not with intermediaries.
3) C'mon, you're running Unix, stop acting so helpless. If you can secure a Unix firewall, you can secure a Unix server. This is not rocket science. If you have to communicate with a service that you don't want to expose to the world, you bind it to a private IP address on NET10.
4) More often than not, a firewall is used to hide insecure hosts, and then people laughably call it "security in depth".
Don't piss off The Angry Economist
Why is it that this part of the Internet industry is run by Snake oil salesmen. Exodus may be bad ,but Level3 is worse ,they promise the world and dont deliver .
Global Crossing / Global Center filters out all RFC 1819 (or it is 1918?, whatever) private networks on our core routers, as well as customer connected peripery routers. This includes DSL, DS1, DS3, OC, ISDN, and dialup customers.
Customers with BGP sessions are allowed to advertise these networks either.
The PIX, like any firewall, is only as good as it's configuration. Things you allow through, will go through. And the PIX, like any firewall on the premesis of a network, can do almost nothing to stop DDoS attacks. The problems is DDoS attacks are aimed at network bandwidth not a particular protected host. If you have a DS3 to the 'Net, and I fill that DS3 with spurious data, the legit stuff doesn't get through. It's as simple as that. To stop DDoS attacks, other technologies are needed (ISP filtering, RFC 2267 / 1918 filtering, IDS to detect the attacks, etc.) Network security is a system, not a firewall. If you deploy the right tools for the right jobs you'll have much better luck.
"Any sufficiently advanced technology is indistiguishable from magic." - Arthur C. Clarke
Yeah, failover systems are being added to the mix as well. Basically the database machine and the NFS machine will always have a hot spare twin ready to go. I think Rob's next post will explain that.
A firewall shouldn't be listening to TCP connections from outside at all - thus a SYN-flood against the firewall won't cause it any problems (other than consuming bandwidth). If the firewall can prevent the SYNs from the attacker from reaching the servers (which must be listening for TCP) while letting the valid ones through, then you've pretty much protected against the SYN-flood.
In this case, it looks like the source addresses were easily characterized, and so the firewall would easily be able to block them. In other cases, the firewall would have a much harder time telling good traffic from bad.
--Fzz
What I want to know is if the FBI is going to play a part in this like so many other major DDoS attacks of late. jason
So ya thought ya might like to go to the show.
To feel the warm thrill of confusion, that special geek glow.
got me some bad news for you, Sunshine.
Roblimo isn't well, he stayed back at the hotel,
And he sent us along as a surrogate hand.
We're gonna find out where you fans really stand.
Are there any MCSEs on the slashdot tonight?
Get 'em up against the wall. -- 'Gainst the wall!
And that one with all the karma, he don't look right to me.
Get him up against the wall. -- 'Gainst the wall!
And that one is in RIAA, and that one's in MPAA.
Who let all this riffraff to have their say?
There's one smoking a joint, and another with spots!
If I had my way I'd have all of ya shot.
(I guess Pink Floyd's going to sue me now)
Make life even easier for people and point that NameZero domain to http://oneilli.net/~sharky/entry/?slashd ot... It'll break away that damn annoying ad banner frame automagically... :-) (and it gives users a choice to keep the frame so NameZero can't get *too* mad before defualting to break it off after 10 seconds...)
--
sharkyfour.com
>By 3 in the morning,
What is the point of working THIS hard?
I mean slashdot isn't critial to anyone and while I admire dedecation on the job, get some sleep man!
The surprise isn't how often we make bad choices; the surprise is how seldom they defeat us.
That's right! One way or another, if the person setting up the router doesn't know wtf they're doing then something is going to be broken. I personally think the more 'socially' responsible thing would be to ship with egress filters config'd and let them sort out their internal issues. But you're right, either way you slice it it's a pain in the ass for somebody. But hey, this isn't kids stuff. You gotta know what you're doing.
So... despite that your solution is wrong and won't work, you'd still like action taken on a flawed plan?
One question: When are you running for Congress and can I be your campaign manager? [It's a 2-part question]
The code is having it's own issues as we're also updating Slashdot to use a new version Slash (to synch up with new release slashcode.com) and bugs are being caught and squashed every now and then, besides that there are the usual hiccups that go along with moving to new servers.
Eff U buddy. DealaWhere is good for what I do. Think of us as a suburb of Phillie.
/. No exams for me, mutha...
I just defended my Ph.D. I got some time on my hands to read/post to
And I am not a network guru or CS geek so I have questions about this stuff.
ed (-1 offtopic)
A standard Class C subnet with 256 addresses (254 actually available; the first is the network address and last is the broadcast address) uses a subnet mask of 255.255.255.0. This can be expressed in binary as 11111111 11111111 11111111 00000000. If you count the ones, there are 24 of them. A network address and a subnet mask can be written as 192.168.123.0/24, thus "/24" refers to a Class C.
A
Now, to find out how many IP addresses are available in a subnet, and what they can be. If the subnet is a Class C or smaller (255.255.255.anything), subtract the last number from 256 (255.255.255.0, 256-0=256, 256 possible addresses). Then subtract two (the network and broadcast addresses), and rememer that one of your IPs has to be your router/gateway.
For 255.255.255.240 (/28), 256 - 240 = a 16-IP subnet with 14 usable IPs. For 192.168.5.128/28, the network address is (obviously) 192.168.5.128, and the broadcast address is 192.168.5.143.
For larger subnets, it gets more complicated to figure it all out and you usually have to convert everything to binary. If you want to know a little bit more detail, feel free to e-mail me.
--
$x='S24;r)>63/* h@<5+oZ)32"5cz';$me='phroggy'x$];
$x=~y+ -xz+\0-Tx+;print$_^chop$me for split'',$x;
For the /8 /16 /24, its basically a way to denote a group or range of IP address. A /24 would be a range of 256 IP address (or what used to be class C) and /16 would be a group of 256*256 IP address (old class B). For instance, 192.168.1.0/24 would mean the range of 192.168.1.0 through 192.168.1.255.
A good read on IP addressing can be found at 3com , its a bit long but well worth reading.
--
--
Yeah, I have OpenBSD sitting on one of my personal machines, and I rather enjoy it. And mucking with the kiddies is fun too.
:)
Like responding back with an ACK for random ports at random times. That'll confuse em. Heehee.
Do you have trouble getting X clients to go in and out of the firewall?
Mike
Anybody notice that "Highest Scores First" doesn't really work? I always have "Highest Scores First" selected but I still find 3s 4s and 5s below 1s and 2s. I wish it worked, it's a nice idea.
/. really get a second? Can't you put a transaction time limit on a web server? Or maximum number of requests answered/minute? And do it dynamically, so when you see the huge surge of traffic, you implement the time limit, and when it dies down again, restore normal operating conditions...
/.'ed that we couldn't ftp our files out to a mirror...
And about the story, don't they have backup servers so that when one fails, another one comes up? I guess the attack was big enough to take down all of their servers... How many hits could
Well, I know what that's like, after we announced that Canvas was available, we were so
___________________________
Michael Cardenas
http://www.fiu.edu/~mcarde02
http://www.deneba.com/linux
hyperpoem.net
Ipfilter (ipf on the command line, not ipfw -- although new versions of ipfw are now statefull I'm told) is a statefull firewall because it actually keeps state of TCP connections. Most firewalls will looks at the flags on a TCP connection to see what state they are in (SYN, SYN+ACK, etc) and blindly trust the packet. There are various ways to get around normal firewall rules by exploiting this "feature". Most rules to block access to a certain port will only block TCP connections that have the SYN bit set. What happens if I send a TCP packet to the port without a SYN bit being set in the entire stream? The non-statefull firewall will not block it.
Ipf on the other hand, will keep track of all TCP streams so it knows what state the streams are really in. Yes, it's more overhead, but it's way more accurate, which is arguably more important.
-B
We had an Alteon in use when we first moved to digital Nation last summer. No. Never again will we use an Alteon. It caused horrendous problems under high loads by munging the tcp headers on the packet that closes a socket, and only after taking the Alteon out of the equation did we ever get sleep.
--
Yes, Virginia, there really is a CowboyNeal.
Have the attacks stopped, or is the setup just better able to cope now? It sounds like there is always a background level of ddos going on, it just affected you because of the change in system?
Just some posting lameness filters were added to discourage the lame trolls who post in all caps and crap like that. I personally enjoy reading the well crafted rants from the creative trolls who try to start flame wars -- I'm sorry but someone posted this one troll yesterday that the web should be a place for marketers to sell and the techie elitists should get lost -- now stuff like that is funny and I thank the troll who posted because I laughed my ass off.
RFC1918 only states that packets _destined_ for those private address blocks not be routed. There isn't anything in there that says packets _sourced_ from private netblocks need to be sent to Null0.
Whoo boy! We did try an Alteon once for a few weeks back in August -- what a fiasco that was! All the web servers were very unhappy at the networking layer so we removed the Alteon and put the web servers in DNS round robin and things ran fine. Naturally I blame Canada.
It didn't screw up the code, it most likely blasted the hell out of the MySQL servers, and the code doesn't do a whole hell of a lot of error checking.
--
Just look for anything from "anonymous monopolist".
It's a simple fact that more Linux expertise is available because more people are running Linux. How is it a troll to recognize this fact in public? Sheesh, some moderators are biased.
-russ
Don't piss off The Angry Economist
John Nagle
In a way, I think Slashdot is getting what it deserves. This is the site where the general consensus among posters has been that it's okay to DDoS a site if you don't like something they did. (Remember all the scripts people posted to attack eToys?) Maybe some troll got tired of being moderated down and took the other posters' advice. Or maybe RTMark decided Slashdot is immoral and staged a "sit-in". "Do unto others..."
So when does Kurt get his own weekely column, "Inside Slashdot?"
:)
lf.o
Hmm...Windows - The largest DoS attack ever?
I'm not a slashdotter, I just play one on Slashdot.
I saw a description of a cat5 cable in the replies that does not match what I have used in the past. The normal configuration for a cat5 cable is as follows
View from the non-clip side of RJ45.
Left side to Right side
with the clip pointing up and the cable coming down from the clip
The wires are color coded by having either a color with a narrow white stripe or white with a narrow color stripe. Pairs of wire have the same color with one being more color and less white and the other having more white and less of that same color. In my description the first color is the dominant and the second is the narrow stripe. IE: Blue/White is Mostly blue with a white stripe
White/Orange Orange/White White/Green Blue/White White/Blue Green/White White/Brown Brown/White.
The Pairs are usually labled as follows.
Pair One---->Orange wires
Pair Two---->Green wires
Pair Three-->Blue wires
Pair Four--->Brown wires
Ethernet uses the White/Orange Orange/White Pair and the White/Green Green/White Pair.
So to build a cross over cable the Orange and Green pairs are switched on one end of the cable while the other end is standard.
The crossed end would have this configuration White/Green Green/White White/Orange Blue/White White/Blue Orange/White White/Brown Brown/White
The only reason I posted this was that I saw someone else giving information that was not standard cat5 specs. This could lead to more confusion than needed for someone who is trying to learn the basics.
Jumping to correct solutions slowly is better than jumping to incorrect solutions quickly.
--
--
fat lenny's gonna lick your brain today.
I find it quite amusing that the site that has entered legend for it's own specialized form the of the DDOS (the slashdot effect) has itself fallen fowl of the more malicious variety.
Congratulations on getting the new servers up and running, I've just moved my badtech cartoon site to digital nation (The old location of the slashdot servers).
it looks like you all worked pretty hard to fix this - cool :)
cant imagine a web w/o slashdot...
Cybie! aka Ralph Bonnell
Are you sure you diddn't have the thing misconfigured? They have a pretty bizarre concept of what a port is. I tested a 180e a few months ago. It took a bit more configuration than I would have expected, but then It ran perfectly on the live site I tested it on. I haven't run one over a couple megabits/s so I don't know how it
does under heavy load.
Well, thanks to the /. crew for finally getting round to telling us what happened - so much for all the whiners who insist that CmdrTaco et al. are involved in some massive conspiracy to keep us in the dark about "important issues" :)
Any possibility on finding out more about the origin of the DDoS? I'm not really sure of the feasibility of doing anything myself.
Although it's not a good idea to advertise their security infrastructure layout to the world.
Then you deny the assumed security of Open Source? Please stop before you disillusion me! Knowledge of the sendmail Debug command or the fact that there's no guarantee a binary was produced by source that was distributed with it might destroy my comfortable little OSS Zealot worldview...
-- LoonXTall
~~~LXT~~~
Life is like a computer program: anything that can't happen, will.
and i was hoping it was "zero cool" and the gang.
or just a bunch of 14 y.o. script kiddies.
Be you Admins? nay, we are but lusers!
Doing this type of filtering doesn't prevent your system from being used in a DDOS attack, but it prevents your system from being used in the attack with a spoofed address. Hence see 50mb/sec from host w.x.y.z, contact owner of that address block and get it stopped, since it is not forged they have a compromised box internally. If everybody started doing that the world would be a MUCH better place to live in.
But this would require time on the part of the admin. Time that the admin does not have do to other problems in the system.
it was the 3rd most popular purchase at Amazon by M$ employees...
Ah-HA! A person that supports the use of market tracking! So YOU are one of those attempting to undermine the value of the Web by making it apparent that stats are in demand! What next, will you present us with the sex life of MS employees, gathered for "market research" and "demographics"?
I've been reading WAY too much YRO lately...
-- LoonXTall
~~~LXT~~~
Life is like a computer program: anything that can't happen, will.
Could someone point me to a decent networking tutorial on the web?
I use systems, and I understand IP (a bit). I do not understand the stuff between the nodes. Switches. Routers. Hubs. Firewalls. Addressing.
Most people don't have to deal with this crap casue a network guy sets it up and we plug in and use the IP address he gives us, but if I ever want to set up my own network (beowulf lab or home network) I need some more info.
I have also heard that you can directly connect two NICs with a special cable. Do you need software changes to do this?
Sorry I am so clueless.
ed
BEGIN rant
I would definitely look at Exodus for some of this trouble. At times, they have been less than helpful for the service level they claim they will provide.
-They changed their security policy a while ago, and neglected to tell us until after the fact. All visitors to your cage must be announced, and just try to get replacement parts in and out without a whole rigamarole. Previously, one person "on the list" could escort others in and out of the facility, but no more. Granted this makes some sense, but when we showed up the first time after they changed their policy, before informing us, we balked, and complained. The response was (I kid you not) - "Well, we're a big company now, so we can't give the same level of service we used to." WHAT KIND OF ORGANIZATION SHOOTS THEMSELVES IN THE FOOT LIKE THAT?
-Their HVAC is substandard, and they don't truly care what equipment is placed in a cage. I pity the poor sun techs who have to replace the Sun server at the bottom of a stack of 10 other machines (ie, no shelf).
-They continue to abide by their own notification procedures when their "monitoring" software reports trouble. We've gone over their policy several times with them, and verified they had correct contact information for us, and yet they still follow old ways of notification. In this case, it's paging one person instead of using the paging mechanism that contacts the actual people who will do the work - the effort is the same either way.
-The number of times that we've notified them of trouble before their monitors catch it - for example, try working with them to show DNS requests from the outside to their servers aren't being handled.
END rant
I could go on, but I won't.
Thanks,
--Carpe diem!
Have any ISPs been banned from slashdot's servers? I'm coming in through relaypoint.net myself.
Yum, Yum.
Oops, hope no one saw that.
Seriously, though, it would be nice, albeit unexpected, if people had their facts straight.
Even if you are in wild and wooly peering arangements, it is likely you have interfaces on at least some of your network equipment which have 'closed' networks, so you could determine that any traffic coming out of it that doesn't belong to a certain set of IPs is spoofed.
As long as all other networks you peer with also block spoofed traffic where their customers plug in, the world will be a better place.
Yes, that's a very big "as long as", but it sure would be nice.
They only thing they won't let us do is take a picture of our cage -- no cameras allowed anywhere in the facility!
Hmm. When I worked for a startup with a cage at Exodus, we put an AXIS Network Camera in our cage. Since our cage was right by the door, we could see anyone coming in or out of their facility....
That was a good account of what happened, but in part two, we want to hear what you are doing to track the bastards down. Knowing how you go about fixing the problem and then tracking down the culprits may help other people who run into the same problem in the future. We would understand if you need to keep the info secret until you have finished tracking them down, or for legal reasons, but at least tell us so.
Your strike back idea isnt so off..
Have a 56K or dialup link to a dofferent leg or T-1 to "strike back".. by having a computer set up (like a sniffer) that determines what kind of attack is happening, where is it coming from, ok pass that over to the attack server.. attack server 1 tries to determine the attacker (script kiddie running winblows? or a uber haxor running a beowulf of cray's connected by 300 t-3 lines) if it's a script kiddie, start data mining, try automated attacks that can extract data or just plain kill them.. (smurf,ddos,surface to surface IP guided missle) if it's a nasty then react by getting every bit of info possible from the secondary data stream from the attacked servers, so you can physically go to that persons location and thunk their heads.
it's possible, but very Un-PC in today's world.
Attack back? that isnt very Politaccly correct of you!
Do not look at laser with remaining good eye.
Or until someone sniffs their router password and blows away their routing configuration....
If by sniff you mean write down while working there. That was an ex-employee, disgruntled and whatnot, that had access to the information. Not a technical exploit, but a social one.
Kintanon
Check out JoshJitsu.info for Brazilian Ji
*ahem* I'm pretty familiar with the ArrowPoint line as a whole, and I am going to now soundly beat whoever designed this sorry excuse for a web farm over the head.
First off, the last I heard, the CS100 is discontinued. What moron bought used hardware? Secondly, the CS100 was replaced by the CS50 and CS150. So then some genius goes from a CS100 to a CS800, the 20Gbit backplane model.
Otay, yeah. That's intelligent network design at work. *sniggersnort* Apparently Andover can't find anyone willing to drive to Acton who has a clue. And what's really sad is that whether or not any of the Andoverians will admit it, they are in the same bloody building as ArrowPoint. I know - I was there a couple weeks ago.
So now, what, we're supposed to be impressed now by an ineffecient web farm design, using excessive servers and used hardware? Let's look at EXODUS then.
Last time I dealt with Exodus is when I told them to either fix their routing or deal with a network I was in charge of basically slamming theirs into the ground with BGP because THEY couldn't configure BGP correctly.
Exodus' track record is one of incompetence, ignorance, rediculously poor customer service (Verio rated higher than Exodus.) and obscene ripoff scams. "Added Security" for only $5k+/month more. Which is simply 'oh, we'll put a PIX firewall in front of you' which is totally ineffective. The Cisco PIX+ firewall never made it past a 3Mbit flood in my personal benchmarks. It died. So Andover, soon to be VA Linux, is paying Exodus $1mil/year to take it up the ass without Vaseline.
Is Andover *DELIBERATELY* trying to scare VA Linux away from buying them? Only an idiot choses Exodus, because everyone's realized that Exodus is made up of scam artists and ripoff gurus. Is Andover trying to show VA Linux their technical staff is inept, when it obviously *isn't*? (Just absolutely godawful *DUMB* in an emergency, obviously. Or is that show too?)
So now Andover is wasting $1mil/year, slashdot is absolutely *GODAWFUL* slow now, as if it wasn't before, and we're supposed to be *IMPRESSED*!?
I'm still trying not to throw up at the mention of a 6509. To be blunt, the 6509 is the equivalent of Ascend. It's pure trash. Anyone who would WILLINGLY put their network's entire reliance on a 6509 should be killed out of *mercy*. Can't Andover afford a 7206 or 7206VXR after their wildly successful IPO!?
Y'know, there's nothing I hate more than technical companies that brag about their knowledgebase, but when push comes to shove, it's not there...
I swear, if my holdings suffer because of Andover's stupidity, I will be *very* angry.
(DISCLAIMER - I *AM* a VA Linux shareholder. You are goddamn right I am watching Andover with a VERY critical eye.)
your company here.
shelby != ford
NFS is not perfect but's it's improving under Linux by leaps and bounds, and it works fine for us. On the list of bottlenecks and risks on Slashdot, NFS is very low on that list.
If it is the Exodus facility in Jersey City, I assure you that there are quite a few cameras there. Several very obvious digital video cams, and of course a few cleverly hidden ones.
that they would private peer with @home so I don't have to bounce through a box on another backbone to get to the Dixeg boxen while at home.
They do take security very seriously. Especially physical security. Their Beltsville facility was the only time I couldn't get to see core routers while on private tour. Every other company I visited showed me the goods. They did show me
the hosting rooms, but you had to stay within the yellow tape
Ye, right!
Yeap, you're correct.
A journey of a thousand miles starts with a brutal anal raping at airport security
I'll probably need to ask this again (and other related stuff) when part 2 comes up, but I'd love to find out why you guys made some of the design decisions you made... I'm sure it'd be instructive for all.
For example, why are the servers serving images and static files segmented? Is there a lot you save from Apache configuration for dealing with one as compared to the other?
Where does MySQL sit? Any "reason" behind Debian vs. RH other than "just because"?
Also, any chance you could go through some of the configuration choices made for your apache processes on each of these? What's your startup script look like (how many processes do you bring up)? No, I'm not lookin' to own slashdot, I'm just curious whether there are any *must-dos* and *must-don'ts* involved.
send flames > /dev/null
Only 'flamers' flame!
Lower your points filter, and keep an eye out for the "Natalie Portman" postings. You know it's him!
--
--
Don't like it? Respond with words, not karma.
This is really silly. Why bring your children up in a Santa Claus level of religion? A much better answer to your daughter is that you don't know. Really a hard thing for a lot of people to do.
Wovon man nicht sprechen kann, darueber muss man schweigen. Ludwig Wittgenstein
Me too me too!
I was going to post this. Isn't 12.1 experimental code? "Early Release" or somesuch?
Well, you're both kinda right.
-If- Exodus had blocked the RFC1918 traffic like they were supposed to at -their- routers instead of letting it get to the slashdot servers, then the site would've received less traffic. Good, right?
And then the DDOS attacker(s) would've started to use legitimate addresses, and slashdot would have had no idea what was valid traffic and what wasn't.
@home uses 10.x addresses extensively through their networks. Traceroute through them sometime. This is acceptable and encouraged if you want to spare IP addresses that the internet will never get to anyway (do you let the public telnet to your ciscos?) The difference here is that the RFC1918 addresses originate and end within one node of each other - and they never go out or come into the @home networks!
(I don't work there or even have any particular feelings about @home, I just knew one of their netadmins (retired now (bastard)))
You'd better watch it with this comment... the MPAA might come after you too!
pronoblem
I know when I've been in a crisis situation, it has been helpful to have something that you know will work. I remember having problems with Caldera, trying to get a web server installed, and finally giving up and installing Red Hat, which is what I normally used.
What I'm trying to say is, given that Pat is the "31337 BSD junkie", even if Linux and FreeBSD both had all the necessary stuff, it's logical that Pat used BSD 'cause that's what he's most familar with and helps get the site back up quicker (which is the goal, after all).
Remember, we're all friends here, people! Use the best tool for the job!
--Mythos
My guess is that we will see a lot more of this until two things happen: 1) The Internet is reduced to utter chaos. There are just not enough FBI agents to plug all the holes. 2) In reaction to the chaos, limits are placed on the actions of users and devices. This may lead to a leap-frog process. However, the consequence of that "security" may itself be another problem. A way to reduce the risk for this is to not take "revenge" against assumed foes. And work through the system. Yes, you should work with the FBI. Unfortunately, we have a generation reaching for power that doesn't always understand the consequence of their actions. And, remember, that at least 3% of any population are sociopaths. Enjoy your "freedom" while it lasts.
Wovon man nicht sprechen kann, darueber muss man schweigen. Ludwig Wittgenstein
I'm curious about the timing with the port to the Exodus environment, was there any indication the attack was timed to take advantage of the different environment? Not saying that the security measures were better or worse than the old site, just that the timing seems rather convienent.
More race stuff in one place,
than any one place on the net.
Sengan violated the biggest rule of Slashdot in that everyone can state their opinion on something. That wasn't nearly as bad as how he did it though.
He posted a flaimbait story, and disabled comment posting (the only story to EVER have this done that I know of, and I've been around since quite nearly the beginning of slashdot. Remember TCWWW anyone?), put his flaimbait opinion on it, posted some horribly incorrect information, and expected people to be happy with him about it. He marked it as a news piece, when it was more editorial than anything.
That's why you rarely see Sengan around anymore. After that he was constantly flamed on every story he posted (I think he continued to post for a little while longer).
I got document not found....
rhino
Because it feels like something I've done before, yeah I could fake it but I'd still want more...
I'm curious on one detail. What was it that the Cisco PIX was supposed to do and didn't?
http://drteknikal.blogspot.com/
that's not true. for example, if you traceroute an erols dialup, you'll see the 10. for the ppp server, and no that doesn't break the rfc. iirc, the rule is you can route from 10., etc but not from (eg, icmp from a 10. is ok, but if you try to telnet to a 10. it shouldn't go past your local border).
Nope, these 10.0.0.0/8 address you see in the traceroute are badly configured machines. For example, it breaks MTU path discovery very often...
I would think that if you had posted a story about it maybe you could have received some help for some of the more intellegent Slashdot readers. Also, Slashdot was to busy to tell its readers, but Slashdot had time to tell Wired.
Or until someone sniffs their router password and blows away their routing configuration....
I wasn't going to talk about this in public because of /. silence about the DDoS, for I thought things could be somewhat related.
This is what I got this morning when I asked for www.slashdot.org:
<html>
//-->
? name=slahsdot.org&channel=www"
<head>
<title>Not Slashdot.org</title>
<meta name="keywords" content="">
<meta name="description" content="">
</head>
<script language="javascript">
<!--
if (top.frames.length != 0)
{
top.location=document.location
}
</script>
<frameset
rows="*,90" marginwidth="0" marginheight="0"
framespacing=0 frameborder=no border=0
>
<frame
marginwidth="5" marginheight="2"
src="http://slashdot.org"
name=thepage framespacing=0 frameborder=no border=0
>
<frame
marginwidth="0" marginheight="0"
src="http://red.namezero.com/strip2/strip.jhtml
name=pb scrollbars=no scrolling=no
framespacing=0 frameborder=no border=0
>
</frameset>
<noframes>
Sorry
</noframes>
</html>
Weird. Did anybody else see this?
Go read the networking-howto (or is it ethernet-howto?) and the linux network administrator's guide (NAG) on www.linuxdoc.org.
As much as I'd like to blame the moderators, it is the site administration that is at fault. They want the troll war to continue. It all adds up to more page views, downloaded ads, posts. Of course, soon, competing sites will reach critical mass and slowly bleed off users from /. Supporting a larger community requires more critical thinking than these people have.
Arrghh this got moderated up?? Oh come one, these are just the source address, routing is based on destination address. Simply: when the post service delivers a letter, they are only interested in who its too unless it can't be delivered.
/8 /12 /16
These RFC 1918 addresses:
10.0.0.0
172.16.0.0
192.168.0.0
are filtered by all ISPs execpt the ones with no good networking people, it standard practice.
It is of course NOT up to router manufactures to block these addresses, it kind of worries me that you think every router made is going to be on the net. Even so, many ISPs use these addresses internally for such things as cable modems so they can be managed.
A journey of a thousand miles starts with a brutal anal raping at airport security
that's not true. for example, if you traceroute an erols dialup, you'll see the 10. for the ppp server, and no that doesn't break the rfc. iirc, the rule is you can route from 10., etc but not from (eg, icmp from a 10. is ok, but if you try to telnet to a 10. it shouldn't go past your local border).
Topology my ass. Exodus fights hard to make you use their 'value add' security services. Be honest guys, the reason you weren't protected was b/c those bastards were working you over for more money and don't want you running your own security, right? In fairness, there's some nice things about running out of an Exodus facility, but dealing with their physical and network security chimps is not one of the high points.
It doesn't really matter what pairs are in use, the important thing is that pins 1 2 3 and 6 are used for ethernet. Technically the telco standard for color coding is blue orange green brown slate. On a cross over cable 1 is crossed with 3, 2 is crossed with 6. When I make a cable, its always white/blue=1 blue=2 white orange=3 orange=6. None of the rest matter.
Just three more hours seapeople and you can finally take me away from this crappy God Damned planet full of hippies
Who was talking about NAT? I'm suggesting that you run your public services on a public IP address and your private services on a non-routable private IP address.
The use of a firewall in itself offers little if any security!
Cool. Does it make me more correct if I use boldface?
And yes, geez, if you have one compromised host it can lead to other hosts being compromised. Should that surprise anyone?
-russ
Don't piss off The Angry Economist
Thinking more about the issue, I can't help but wonder if many problems are solved by only allowing a post to be downward moderated once. This way, excessive points aren't wasted and the dissenting opinions - which should be read - won't get battered by clueless moderators.
Of course, I still think all downward mods are a waste. Upward moderation helps because it saves time. If you want filtered speech, AOL would be more suited for the majority of moderators.
Pardon thy typoes.
That should have been, "Customers with BGP sessions NOT are allowed to advertise these networks either."
Be nice if I spelled periphery right too.
The MSFC, the multi-layer switch feature card does the routing. There is a PFC (policy feature card) which will provide QoS, Load-Balancing, access lists, etc. There is no RSM for the 6509. Check Cisco's site for reference. The MSFC/PFC combination will forward 15Mpps if the routed traffic goes through the hardware. If there's some exception that doesn't allow it to get routed by the ASIC then the traffic will go through a software router that will do 200Kpps, the software router is equivalent to a 75xx series router in terms of speed.
I'm not trying to give SlashAndover a hard time for making a poor choice, but my opinion of Exodus went down quite a few notches right after I installed my stuff there. Did you guys do your research?
Let's see, what kind of problems have I had?
-Routing - routes not set up when EXDS said they were 'Good to go!'
-Firewalls (managed) - installed with no rules.
-Power - missing a circuit in our cage
I could go on.
And I will.
-HVAC - we've had some of our Suns turn themselves off because it got too hot it our cage
-Backups - they installed the client on one of our boxes, didn't put it in startup scripts, and after a month of the client not getting backed up, took it out of the backup rotation because "it wasn't responding" (without bothering to call us!) (never mind they only installed the client on one of the six boxes they were supposed to)
-More HVAC "it's 84 degrees in here, you're supposed to notify me at SEVENTY TWO" "oh, well the alarm kept going off so we set the threshold higher" (you'd think i'm kidding - i'm not.) >:(
So the question I usually ask myself when the conversation turns to Exodus is, "How many times would I rather pound my testicles with a wooden mallet than do business with Exodus again?"
We've talked to SVPs and Directors and other PHBs, and all we can ever get out of them is "we're working on making it better for you". Gee, thanks, that's such a consolation.
Of course, we only pay them $168k/year, so naturally we shouldn't expect much for that paltry sum, right?
Are you referring to?
; -- the corruption of government starts with its secrets. a truly free people keep no secrets. --
Hmmm, maybe, just maybe because it's in release 12.1 ??
/. or any real production site use IOS 12.1
As you work for Cisco would you *really* recommend that a site like
A journey of a thousand miles starts with a brutal anal raping at airport security
You didn't specify which OS... but if it happens to be Win 95/98, then here is a site I thought excellent for a beginner trying to put together a simple home network.... http://network.fament.com/helmig/j_helmig/faq.htm
HTTP header ad space for rent! Advertise to thousands of server log readers - only $50 a week per header! 1-800-SURFALOT
From my knowledge of routing IP and how things work, I can't conceive how 10.0.0.0 packets are routed. Or any spoofed IP for that matter. All the networks I've been on have never allowed me to send packets outside my assigned range of IP addresses. At what point does someone sneak a 192.x.x.x, 10.x.x.x or spoofed, non-reserved IP packet into a backbone? There must be a serious breach of policy somewhere. Also, when a router at the provider gets a packet, doesn't it know the range of IPs of those it is providing service too? Shouldn't it block transmission from other addresses it is not servicing? It seems natural that this should happen at every route point along the way. At what level do routers stop caring what IP the packet is coming from and just routes everything? Is it the circular, interconnected nature of the internet makes it so the highest level routers must route anything? It seems like a string of downstream routers need to be horribly misconfigured to allow a packet to reach a high level router that doesn't care about source IP. I have heard of selectively routing a packet through certain hosts, but I still don't see how a spoofed IP can leave the provider and it's routers in the first place in order to spoof via the selective routing. The whole issue of spoofing has bugged me for years but I can find no definitive explanation of how this kind of breach could be allowed.
In theory, this is correct.
However, in practice, we have incompetant admins, ignorant management, and underpowered hardware. In many backbone cases, ingress/egress filtering (or, indeed, most any kind of filtering) of these types of IP addresses isn't an option, due to the volume of data that these routers handle. They wouldn't be able to handle it. So, unfortunately, we must rely on ISP's on a more local scale to not only block these packets from coming in to the network (and likely on to their customers), but block them from leaving their network (or, perhaps, keep their customers from introducing them).
Along a similar line, these filters could/should be expanded to include the list of IP addresses that that network services. If done correctly (and down to an appropriate level of granularity), not only will all IP spoofing be eliminated, but anyone attempting to do so can be tracked down rather easily.
The fact that IP spoofing still shows absolutely no signs of abating is proof enough that few ISP's are filtering a damn thing.
The answer : Network Appliance Filer F740. Check their website at www.netapp.com.
Looking for a great online backup: Green Backup
And they regularly set up machines with the wrong domain name, forget to put the proper hostnames in the dns tables, don't upgrade exploitable versions of Netscape Enterprise, misconfigure Oracle Databases, and various and sundry other annoyances that create extra work for this lowly SysAdmin...
I don't know if these problems are specific to my company, but it just goes to show that no hosting company is perfect...
Unfortunately all the lameness filter has done has encouraged them to spam more to get around it, which they've figured out how to do already. Bit of a waste of time really :)
That was a good account of what happened, but in part two, we want to hear what you are doing to track the bastards down.
Unfortunately, if I understand correctly, that can only be reliably done by manual traffic analysis by the sysadmins of the various routers en route, if I understand correctly. The origins and possibly routes of the incoming packets will have been forged, so you have to actually go from router to router looking for unusual traffic.
Disclaimer: I am not a networking guru.
Various modifications to routing software have been proposed that would make tracking easier (see the recent slashdot article). However, at present you're in for a lot of work and still probably out of luck.
I had a similar experience, I was using explorer on "not my machine" and I kept getting time-outs on the links. Then I noticed that I could get pages from "www.slashdot.org" but not "slashdot.org"
All the links on the home page seem to drop the "www", I tried adding the www to some of the links and was able to get content.
I Don't know enough about IP to know if this means anything. I figured "www.slashdot.org" and "slashdot.org" would DNS to the same IP. Is this not the case ??
Is it possible that IE and Netscape have differnt rules for dropping the WWW.... no that can't be it, that comes from the HTML but possibly Netscape understands that www.slashdot.org == slashdot.org and IE doesn't??
Not trolling, just curious.
Fibre Channel SAN looks like it would work pretty well. Never used it before myself, I am sure someone else here could give some good info on implementing something like this.
Q.
Is anyone aware of a load balancer project? If there isn't one started, I'd be interested in getting it going on OpenBSD. Please email me at slashdot@remove-this-part.chaosmt.net
Democrats and Republicans only disagree about how to enslave you
Hi, /. suffers a major DDos attack _right_ after that Weston bloke at M$ strated making threats. You'll have to forgive my paranoia, it just seems a little well "timed".
am I the only person who finds it rather "convienient" that
Of course I'd never even think to imply that M$ had any involvement in this...
For those of you who haven't read about M$'s secret covert operations, read "The Microsoft File" - it was the 3rd most popular purchase at Amazon by M$ employees...
http://www.jonmasters.org/
This is certainly not 'Informative'(is there not a moderate this to 'Misinformative' selection?) and it certainly should not have shown up when I filtered down to +5 comments. Especially when I had to look at sub +5 comments to find that, in fact, his severe factual errors had been corrected. Do the moderators actually read the posts or do they just look for their buddies' names to give them 31337 credit?
I'm not sure if this qualifies, but take a look at www.dubbele.com
I know from personal experience the following backbone providers do not filter these addresses:
I do not deploy Linux. Ever.
No, it doesn't. Path MTU discovery is
unaffected when such addresses are used
judiciously for inner nodes of the network.
12.1 is wide-deployment release code now.
"When the president does it, that means it's not illegal." - Richard M. Nixon
Security by obscurity is not security.
Based on you description of your set up it appears that you are only located in a single data center. I am curious why you haven't diversified your dependency on a single data center and use some thing like Network Director or Hopscotch (if it is still around). I have never worked with Arrowpoint but am familar with Cisco Local Director (and MLBA). Can anyone comment on the comparison between the two
Gah. Flaimbait? Heh.
:) (Yes folks, the AC problem is a fairly recent thing)
I wasn't interested in the entire user thing at the time, but figured I probably should register. -Shrug-
Oh well. I still remember thinking Anonymous Coward was one person
That had no effect...
It's not just that traceroute won't get through... I can't connect to the site in netscape, I can't ping it, etc...
--
This
#3 is a very good point. If I come in as AC and post crap, if it gets moderated down there really isn't any incentive not to keep posting crap unless my karma also goes down. Do it enough and I post at -1 anyway...You listening Taco?
/. with the type of posting you do. I would refer more to the alt.syntax.tactical description of "usenet (or in this case /.) performance art." Most of the good ones I would classify this way.
Perhaps the moderators are on crack as of late, or maybe somewhere in the moderation guidelines there has to be in big letters DEFAULT THRESHOLD IS AT 1. Unless it's something that *really* is offensive, moderating it to -1 won't do much good. Especially if it's anonymous (After all, with the present scheme who gets hurt? NOONE!) Maybe the answer is to Metamoderate those people as unfair, as they unfairly squandered their points in a futile manner trying to sink something that's already well below most's radar.
A lot of times I've engaged people as AC simply because it's something that I feel doesn't need to be seen by incoming media-types or regular posters who have their thresholds set so as to avoid offtopic discussion. For the same reason, I almost always forego the +1 bonus I seem to have acquired, unless I feel it's something truly informative(which doesn't happen that often).
As far as you being a troll, it really isn't fair to say that on
Ontopic to the discussion, I'm glad to see the guys here finally put up an article about the DDoS and all. Thanks. Not only is my curiosity almost sated, but I feel like I've learned some stuff about networking as well (especially from the reader's links above).
Exoloss, as their security guys aren't the brightest fellows on the block, decided it was easier to ban ALL cameras rather than just flash ones.
Still, slipping one into your backpack circumvents that nicely (also ducking around that 'you must have a pass to get out with even a network cable' policy, come to that).
I've had alot of portscans for 31337 and 12345 in the past week on the mediaone network, all from 10.0.0.0/16 networks. I am massively annoyed that they let this through and block ports 137:139. Umm.. is this solving the problem? No! Oh, and they've taken a liking to scanning their customers boxen.. but I digress.
DDoS is the direct result of sloppy upstream administrators. IF I were in your shoes, I would be suing every person upstream for atleast a few hops for passing those 10.0.0.0 packets along for gross negligence.
My two problems with it are: 1) anyone who can get past the firewall and access the NFS servers at the network level can own you--read/write/delete...just connect directly to NFSD and make it your bitch^H^H^H^H^H^H^H^Hahem, do anything you want; 2) the performance of NFS is far far far less than optimal, compared to the performance of local filesystems, or even other distributed filesystems.
Just my opinions, but I've been doing large scale stuff for a very long time. And, again...I'm not complaining--it seems to perform quite well. I just have to think and write about these things all day, every day.
-Buffy
The God == Slashdot concept supports my theory that the song God Save the Queen was begun as a DDoS against God and his prayer-server. Millions of identical prayers, thousands of times a day...
"It's that guy!"
- - - -
The real Tetsujin 28 is a giant robot.
I know I'm not the only one who would like to see pictures of this whole setup :)
And while you're at it get CowboyNeal to give us a sexy pose *on* the servers (grin)
If you like you can also build your own, but the required tools are fairly expensive, unless you plan to make a living at it ;-)
No software required, other than networking protocols...
"History doesn't repeat itself, but it does rhyme." Mark Twain
Who was DoS'ing Slashdot? My conspiracy theory is that M$ was pissed at your resistance to removing those Kerberos posts.
Who knows, even Bill may be a /. reader?!
This post encoded with ROT26. If you can read it, you've violated the DMCA. Handcuffs please, sergeant.
I think the main reason is that in order to do packet filtering, the router has to unpack the packets, examine the origin and destination addresses, and then pass/drop the packet. Without having to examine the packets, the interface processor can examine the route table, decide which path to send out on, and forward the packet out the correct interface. The main CPU doesn't have to get involved.
That being said, the solution is to put filters at the edge. We have packet filters that drop any packets that don't have our IP addresses as the origin. It's not that big of a problem when we deal with it at the Mbps level. When we have to deal with it at the Gbps, it puts too much load on the processor.
We saw what happens when you try to do packet-filtering in already loaded routers with the first round of DDoS attacks. All week after that, we were seeing significantly more BGP router flaps than normal. (>50/s where normal is ~10/s) The ultimate answer is IPv6, until all the tools are there, we're stuck.
Is it only me, or is it a reaction to the recent Microsoft chest-beating? Monday and yesterday, I was getting "unable to connect", timeouts, etc. while attempting to access the site usimg MS Explorer 4.01, (and on the two occasions when the front page came up, none of the internal links to slashdot-hosted content worked) but the Netscape Beta worked first time, every time. Of course, the Netscape/Mozilla memory leak eventually ate all of the resources on my system...
Was this related to the DDOS or was it something else? Anyone else experience this?
This page accidentally left blank
Every time I tried to view the front page of /. it came up waaaaaaay funky. 1 - Did anyone else experiance this? 2 - Is there more to this problem then just a DDoS? mcd
----------
No army can withstand the strength of an idea whose time has come.
- Victor Hugo
While I agree that the Slashdot DDoS attack caused many people quite a bit of annoyance and frustration, I think leaving the impact at that is very short sighted.
Firstly, I don't think the blame for this DDoS can be centered on just one person or group. Obviously, those who attacked Slashdot are to blame, as are Slashdot's sysadmins, and the people at Arrowpoint. And secondly, the costs of this are much greater than you might think.
I have an eight year old daughter. We had a family pet - a rabbit, black, named Midnight, and my daughter was very fond of it. Midnight, sadly, passed away about two months ago. A week or two after Midnight died, my daughter came to me in tears and asked me, "Daddy, why won't God bring Midnight back? I've been praying like Deacon Simmons told me to."
Naturally, I had to think about how to respond to this. I finally answered, "well, honey, God is a little like Slashdot. He can seem arbitrary, cruel, and unresponsive, but he's really a nice guy who's just a little out of touch and is a little slow at responding to requessts."
This was fine, and I thought that would be the end of it. However, when Slashdot went down last week, my daughter burst into my den, positively sobbing and wailing, and managed to choke out "Daddy! Daddy! I can't get to Slashdot!" "Honey," I said, "it's just a website." But, between sobs, she said, "but you said God is just like Slashdot, remember? Does this mean God is dead?"
I tried to console her as best I could, but nothing seemed to work. When Slashdot came back up, she seemed to return to normal, but she hasn't been quite the same since. She doesn't ask me about God so much any more, and she seems less interested in Church.
As a good Christian, I will turn the other cheek, and not call for the punishment of those responsible. But to the heinous criminals and negligents responsible for this, I must ask, how do you feel about destroying a small girl's sense of innocence and wonder about the world? About crushing her childish dreams and idealism? About shattering her faith in God and his benevolence? About possibly having crushed her soul and emotion forever, leaving her to live the rest of her days in spiritual agony as a broken, scarred husk of a person?
I hope all of you think long and hard about what you've done. What is the soul of a child worth, next to a few double-checks of the router?
Thank you.
the guy in philipines that have been arrested because of the ILOVEYOU virus said that MS should be investigated because it's their fault if there's virus 'cause MS apps are too buggy.
--
BeDevId 15453 - Download BeOS R5 Lite free!
"Science will win because it works." - Stephen Hawking
I'm sure that these great enemies of the Slashdot Empire have found this to be a convenient time to strike. We must systematically seek and destroy all those suspected of having sympathies with the MPAA, RIAA, or Microsoft for security reasons.
Therefore, all
Windows users
CD listeners
Movie watchers
Metallica fans
are asked to please leave now or face prosecution.
thank you.
What exactly do you think is different between a 6509/RSM and a 7206?
All 72xx use PCI backplanes/interfaces, max of around 1Gbps of throughput, and the VXR have some higher speed cpus to throw at your interfaces.
A 6509 has something like a 16Gb full-duplex (marketroid-speak == 32Gbit) backplane, with the RSM/PFC/MSFC/whatever sitting on a very fast bus, and processing the packets at L3/L4 with dedicated ASICs, as opposed to general purpose CPUs.
Which do you think will work under load better?
I am honestly curious to hear your technical reasoning for arguing against a 6509.
Several people have tied the notion of Network Address Translation and the use of private IP address space to firewalls.
The use of private address space in itself offers little if any security!
All that is required to break in to a site using RFC-1918 space is a compromised account on a system that has local routing knowledge. That could be a router or a nearby server.
Since when does the typical slashdot flamer only flame invalid points? :)
Sponge
Posted by BSD-Pat:
This type of solution is currently being looked at, it was in the original spec, however we didn't have the support needed at the time, but we will soon =)
You're not gonna blame me for that are you???
"History doesn't repeat itself, but it does rhyme." Mark Twain
Hey- are those the new fangled scotch tape holographic drives in the background?
Sig??? I don't need no stinkin Sig!
This is different from mostly-passive traps like teergrube (FAQ; jargon) or Deception Toolkit or spider traps which sit around waiting for Bad Guys to attack them and react unexpectedly when attacked (e.g.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
I think so ...
I am, therefore you think.
Maybe you should type more carefully, since you
:-)
/pyder.....
requested http://slahsdot.org (slaHSdot) not
slashdot.org...
I registered that domain (for free @ namezero) to
help the people who couldn't type. Sorry if I scared you
Cpyder@slahsdot.org
_
/
\_\ sig under construction
waiting for Rob to toggle in a boot loader to IPL from the punch card reader?
try { do() || do_not(); } catch (JediException err) { yoda(err); }
RSM - Route Switch Module
- Basically a router on a card in the switch for routing between VLANs
MSFC - Multilayer Switch Feature Card
- Once a route for a packet flow is figured out (from the first packet going through the router) all other packets from the flow get switched instead of routed.
-- Ed Bugg --You have freedom of choice, but not of consequences.--
Got a couple of things to point out with the Cisco Switches-
1. There's a code problem with several versions of the IOS for the MSFC that causes funky problems, including locking up the MSFC, and preventing Fail-Over according to Cisco. I was in D.C. at a CCIE Lab prep class when we had this problem a couple of weeks ago, so I wasn't on the firing line, and don't know all the details. Just might be something you want to check out.
2. The 5.3(2) Supervisor image apparently doesn't support Gig E ports well. We found this out at the same time as the above problem with the guy at TAC said "Geez, you shouldn't even be working at all with that image and all those Gig ports." or words to that affect. Supposedly the 5.4 images fix this problem. Convient since we had 24 switches delivered in January with a minimum of 4 Gig ports configured, up to 24 Gig ports depending on the switch, and every one of them had the 5.3(2) image.
3. Lastly, I'm becoming concerned with Cisco's Quality control, especially with the Catalyst switches. Of 48 Supervisors delivery in January, I've had to RMA 8 already because of dead SAINTs or other dead ASICs. For the math challenged, that's a 20% failure rate in less than 4 months; definitely not conducive to the 5 9's uptime directive from On-High.
This was not a rant, or even a Cisco bitch- just wanted to let ya'll know there are some issues with the 6509's that may have played with your problems.
Sig??? I don't need no stinkin Sig!
Metamoderators shall have the first born of whoever moderated /that/ down.
---
script-fu: hash bang slash bin bash
[ approaching AI ]
Um, this isn't a "Linux site". This is a geek site, which often covers Linux (but also covers OSX, BSD, even Win2k).
- Jeff A. Campbell
- VelociNews (http://www.velocinews.com)
- Jeff
the router card in a 6509 is the RSP4 from the aging 75xx line. The 72xx VXR will kick it's ass every time.
kashani
- Why is the ninja... so deadly?
What is wrong with BSD?
All the cable modems and DSL modems I've seen or heard about have the Ethernet port already "crossed over" internally since they're almost always directly connected, you only need a regular CAT5 patch cord. That's why if you connect it to a hub you have to use the uplink port on the hub.
I myself, as well as others, would love to hear why the PIX wasn't meeting your needs, and how *BSD was filling that gap. Are you guys at a point where you can give us more detail? Thanks.
Read the third paragraph, the stuff about "community reinforcement and propagation", meaning only boring, party-line comments seem to float to the top these days. Shit tends to do that. The comment rambles on for a bit after that, but the author has a very good point.
These days I also tend to get increasingly annoyed at seeing yet another generic "why this is good for Open Source" (always with capital O&S, nice'n'proper) empty comment rated into the ionosphere. There's usually an AC reply below it, something along the lines of "suck on it, you karma whore". I don't consider it useful to add to the noise with comments like that, but I'm glad to see there are other people who are sick of highly rated feel-good commentaries.
Actually, the only thing I don't agree with in the above is the weird excuse for posting AC. If you feel so strongly about it, why not use your handle? But perhaps that's me. I'm not a regular poster and don't give a rat's ass about my karma level, or whatever it's called. Wish there were more people like that, though.
#define DEAD 1 void tired(horse){ while(horse==DEAD) beat; }
Syllable : It's an Operating System
yahoo, hotmail, sony japan, and cdrom.com all use FreeBSD for a reason. I'll take the BSD ipfilter over linux ipchains any day. ipfwadm was fucking horrible, trying to setup complicated firewall rules at the command line? ipchains is slightly better but nowhere near as good as ipfilter.
Only the State obtains its revenue by coercion. - Murray Rothbard
Sorry. Make that mid 90's...
jf
Ya know what? You've gotta be right. It must all just be coincidence. MS would never stoop to making a coordinated attack on its critics, right?
Um, slashdot is a pro linux site. OSX is covered cause even tho apple is a hugely proprietry company, everyone here loves microsoft competitors.
BSD is covered just cause, and Win2K is only covered when FUD about Win2K is discovered, i mean, the launch of Win2K wasn't even covered, however on the same day we got a nice article about linux 2.3141529 being released.
Anyway, point was slashdot IS primarily a linux site.
Ironic since Cisco has just finished acquiring Arrowpoint, and my guess is that they'll implement the Arrowpoint code in....none other than the 65xx series....pretty ironic ;)
-- Andreas
because the 6509 is way too under powered processor wise to do the kind of traffic slashdot is doing. Having built a system that pushes 250Mb/s out at peak, the Arrowpoints are really the only way to go especiialy if you plan on the usual exponential internet growth, converting to NAT, and like some sembelance of filtering.
kashani
- Why is the ninja... so deadly?
but does anyone make a "locked-down by default" distro based off Red Hat/Debian/*BSD?
OpenBSD is pretty well there in the world of "secure by default". You'll have to enable pretty much anything you want to use by yourself.
-Wintermute
I've more traffic then most through an Alteon Ace3 (250 Mb/s and 90k concurrent sessions) and it ain't that good at it. Numberous software problems and lot's of generally flakiness. I had the config checked out several time by ALteon and even swapped hardware. We'll see how their new 700 boxes and 8.0 code do, but if you're doing things today Arrowpoint is one of the better choices.
kashani
- Why is the ninja... so deadly?
schmeel:~$ traceroute slashdot.org
traceroute to slashdot.org (64.28.67.48), 30 hops max, 40 byte packets
1 eowyn.cglow.org (192.168.2.1) 2.783 ms 2.081 ms 1.996 ms
2 HSE-Toronto-pppxxxxx.sympatico.ca (216.209.54.1) 53.911 ms 35.685 ms 33. 193 ms
3 dis17-toronto63-fe1-0-0.in.bellnexxia.net (206.108.100.33) 16.352 ms 16.7 96 ms 14.961 ms
4 torcorr02-fe0-0-0.in.bellnexxia.net (206.108.100.162) 18.211 ms 18.331 ms 18.837 ms
5 core1-toronto63-pos11-1.in.bellnexxia.net (206.108.98.17) 19.211 ms 20.42 1 ms 17.368 ms
6 bx1-chicago23-pos3-0.in.bellnexxia.net (206.108.98.42) 28.007 ms 26.737 m s 27.155 ms
7 206.108.108.250 (206.108.108.250) 29.808 ms 30.245 ms 29.002 ms
8 bbr02-g1-0.okbr01.exodus.net (216.34.183.66) 28.347 ms 28.040 ms 28.688 ms
9 bbr01-p5-0.wlhm01.exodus.net (216.32.132.210) 53.271 ms 53.805 ms 51.739 ms
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
[etc...]
blech!
--
This
All of these machines were behind an Arrowpoint (CS-100) firewall/load balancer which took it on the chin when we got DDoSed, so basically the Arrowpoint was taking the full force of the attack. So as described above we replaced it with a CS-800 and a BSD firewall.
I guess we learned that if you're going to post a letter from a Microsoft attorney on your web site the same day you implement a few new troll filters you better be prepared for the fury of hell to rain down on you. Then again this is Slashdot, so we always should be prepared for the fury of hell to rain down on us.
Actually, there are two standard ways of doing it. Taking pin 1 to be the leftmost when you have an rj45 plug upside down, the most common way of wiring a cable is 1 light orange 2 orange 3 light green 4 brown 5 light brown 6 green 7 light blue 8 blue pins 1 and 2 make up one twisted pair, and 3 and 6 make up another (as you already noted.) the other two twisted pairs, pins 4/5 and 7/8, are used in 100Base-T4 and Gigabit-over-copper. (I think 100VG also uses the other 2 pairs in some applications) so if you're going to wire a crossover cable for this sort of environment, dont forget to swap these two as well (catch: pin 7 to pin 5, pin 8 to pin 4) So, for a 4 pair crossover, your 2 cable ends would look like end one end two 1. light orange light green 2. orange green 3. light green light orange 4. brown blue 5. light brown light blue 6. green orange 7. light blue light brown 8. blue brown. - S
If you are talking about the MSM (RSM, whatever), yeah probably. I got a little confused.
The MSFC/PFC is the daughtercard for the Sup module that should be directly connected to the backplane, using custom circuitry instead of the CPU-based RSP stuff.
DDoS is the direct result of sloppy upstream administrators. IF I were in your shoes, I would be suing every person upstream for atleast a few hops for passing those 10.0.0.0 packets along for gross negligence.
Um, no.
DDOS simply requires that a lot of compromized boxes be able to send you packets. Spoofing to non-existant return addresses is an orthogonal issue. You reply that it's used to mask the souce boxes? Any _valid_ address could also be used for that, so filtering would gain you nothing against that.
I agree that filtering of reserved addresses should be done, but that would not hinder a DDOS attack.
I say we blame Canada!
hmmm... Though, it could be aliens behind it. They're always up to no good. Green little buggers.
Ceci n'est pas une sig.
and then it might be fun to offset this against the millions that various companies throughout the world have gained as their tech staff found themselves with nothing better to do than work :)
~ppppppppö
Well, didn't help much...
You didn't scare me, you *embarassed* me! : ^)
I don't need a layout of /.'s systems to bring them down. You've been playing Metal Gear and watching The Real McCoy too many times.
I need the layout to FIX it. Fixing and breaking are not just opposite enterprises, they're completely different. The fix = -break idea is flawed in much the same way bad = -ungood.
Think for a second. If a network layout would be useful to breakins, then every site defacement would have to have been accompanied by a physical breakin. While it's true some are inside jobs, the analogies borrowed from the physical world are plain wrong.
The message on the other side of this sig is false.
Don't get me wrong. You guys (and gals) are WAY more knowledgeable about this stuff than I, and I don't want to seem like I'm denigrating your technical skills.
BUT. You didn't figure out what was wrong. You replaced some hardware and "it seemed to work". If this WAS a DDoS (which the floods and IPs seem to indicate), then the hardware problem was a symptom, not a cause. In which case you're still open to further problems.
Or is this firewall supposed to block the flooding? How is a FreeBSD desktop firewall different than the router (or whatever) you put it in front of?
--
Have Exchange users? Want to run Linux? Can't afford OpenMail?
Linux MAPI Server!
http://www.openone.com/software/MailOne/
(Exchange Migration HOWTO coming soon)
For the cable, Look at a pinout (Blackbox or other sources). Notice there are 4 pairs:
Swap Brown for Orange (Pair 1) and Blue for green (Pair 2) and you just made a crossover cable. You just wired the Transmit of one machine to receive of the other.
Actually, for most stuff the outer pairs (Brown & Orange) are not used.
Charles Spurgeon's Ethernet Web Site
Jason Schwarz Ethernet Tutorial
Lantronix Networking Tutorials
You might also try typing "ethernet tutorial" or somesuch in your favorite web search engine. Hope this helps!
--
Okay, I got Linux installed. So where's the free beer everyone keeps talking about??
net3-4-howto
firewall howto
masq-howto
I have also heard that you can directly connect two NICs with a special cable. Do you need software changes to do this?
Yes, you can do this with a crossover cable and no you don't really need any special software to do this. I use one when I bring my laptop into work and want to hook it to my workstation. You can either make one yourself or buy one at any decent site like hardwarestreet.com.
Sorry I am so clueless.
Anyway, good luck.
There is much cruelty in the universe, John.
Yeah, we seem to have the tour map.
The routers should toss anything originating from 10.x.x.x, 172.[16-31].x.x, 192.168.x.x. Why would these be getting through to you at all. Yell at your ISP to fix their screw ups. I'd bitch at my ISP if they let such obviously forged packets enter their network to get passed on to me.
i think most network outtages are REALLY due to someone tripping over the power cord and thus, killing the machine. all of this techie mumbo-jumbo sounds impressive, but i bet a janitor without any computer technical training tripped over the wire, which caused a power outtage, which was recognized by the system admins, which drew them away from their party until someone could drive up there and fix the problem (plug it back in).
that's just my conspiracy theory.
Why read the article when I can just make up a snap judgement?
I see a pattern.
pronoblem
My opinion of the Exodus network (which was quite high) just went down about 10 notches.
Why don't they have anti-spoof filters that drop all 192.168.x, 172.16.x, and 10.x addresses?
I know this wouldn't have helped in this attach since these weren't the only spoofed addresses used, but still that stuff should never have arrived unless it originated on the Exodus network to begin with.
Anyway - good job to get it cleaned up and stable Thanks to all of those who busted butt to get the site back online and stable. I know what it's like to get attacked and it's not fun.
FreeBSD -> W2K airgap a wise thing? Please!! DOn't you know nature abhors a vacuum? /. using W2K. Plus all the jokes about 65K bugs, and downtime(?)
OpenBSD (or a customized Linux distro with all s/w in wierd places, maybe?)would be better, if only to stop all the flames about
I can throw myself at the ground, and miss.
IYHO, would it be a good idea to advertise a false security infrastructure layout? :)
Or better yet, while you're at it, short a few thousand shares of Cisco and Arrowpoint, then Slashdot'em! (it's a joke I'm kidding, 'K? no need to deny, etc.)
Why not do IOS Load Balancing from the 6500/MSFC itself? You can use SYNGuard with the Load Balancing to protect against SYN floods... refer to: http://www.cisco.com/univercd/cc/td/doc/product/so ftware/ios121/121newft/121limit/121e/121 e1/iosslb1.htm#xtocid446613 -- Anonymous Cisco Employee
Bah! Just have a couple *nix boxen, and ping -f from one box to the other through the cable. A couple hundred thousand packets later (ie: 15 seconds or so), and if you have anything > 0% packet loss, the wiring is probably b0rked.
I didn't know about this till now but that day i couldn't get through to slashdot.
I had just signed up at a new ISP where the busy signal is a plague there and the modempool now and then allows itself to change protocol in the midst of a connection (to something expecting IPv6 to be enabled here so i get disconnected modprobing for net-pf-10 or with illegal LCP call errors)
Well: In the midst of preparing a complaint i saved the nslookup, ping and traceroute versus slashdot:
here's nslookup from that day:
nslookup www.slashdot.org
Non-authoritative answer:
Name: slashdot.org
Addresses: 64.28.67.64, 64.28.67.61, 64.28.67.62, 64.28.67.63
Aliases: www.slashdot.org
Here's what the ping looked like:
ping www.slashdot.org
PING slashdot.org (64.28.67.62) from 212.242.56.150 : 56 data bytes
From msx-osl-15.ppp.cybercity.no (212.242.48.37): Source Quench
From msx-osl-15.ppp.cybercity.no (212.242.48.37): Source Quench
64 bytes from 64.28.67.62: icmp_seq=1 ttl=241 time=2644.7 ms
wrong data byte #8 should be 0xb1 but was 0xaf
af 4f 1d 39 b3 6f 4 0 8 9 a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f
20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f
64 bytes from 64.28.67.62: icmp_seq=3 ttl=241 time=1200.0 ms
wrong data byte #8 should be 0xb2 but was 0xb1
b1 4f 1d 39 6f 70 4 0 8 9 a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f
20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f
64 bytes from 64.28.67.62: icmp_seq=4 ttl=241 time=940.2 ms
64 bytes from 64.28.67.62: icmp_seq=5 ttl=241 time=1520.4 ms
wrong data byte #8 should be 0xb4 but was 0xb3
b3 4f 1d 39 ba 6f 4 0 8 9 a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f
20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f
From msx-osl-15.ppp.cybercity.no (212.242.48.37): Source Quench
--- slashdot.org ping statistics ---
8 packets transmitted, 4 packets received, 50% packet loss
round-trip min/avg/max = 940.2/1576.3/2644.7 ms
and here a traceroute:
/usr/sbin/traceroute www.slashdot.org
traceroute: Warning: www.slashdot.org has multiple addresses; using 64.28.67.61
traceroute to slashdot.org (64.28.67.61), 30 hops max, 38 byte packets
1 msx-osl-15.ppp.cybercity.no (212.242.48.37) 127.059 ms 117.442 ms 598.169 ms
2 pop-osl-de1.cybercity.no (212.242.48.33) 135.448 ms 127.659 ms 118.734 ms
3 ro-osl-feth0.cybercity.no (212.242.48.25) 115.117 ms 118.309 ms 198.244 ms
4 Serial10-1-1.GW2.OSL1.ALTER.NET (146.188.35.221) 184.727 ms 177.547 ms 178.205 ms
5 422.ATM6-0-0.CR1.OSL1.Alter.Net (146.188.9.210) 954.049 ms 879.173 ms 898.407 ms
6 499.ATM3-0.BR1.NYC5.Alter.Net (146.188.14.254) 1015.430 ms 1060.149 ms 1038.496 ms
7 331.ATM3-0.GW1.NYC5.ALTER.NET (137.39.30.105) 1272.030 ms 1194.124 ms 1158.176 ms
8 151.ATM2-0.XR1.NYC1.ALTER.NET (146.188.177.226) 1159.550 ms 1237.909 ms 1098.529 ms
9 295.ATM6-0.XR1.EWR1.ALTER.NET (146.188.176.105) 975.240 ms 1076.418 ms 1097.999 ms
10 193.ATM8-0-0.GW4.EWR1.ALTER.NET (146.188.179.177) 1481.877 ms 1017.935 ms 1105.325 ms
11 exodus-ewr1-oc3.customer.ALTER.NET (157.130.15.194) 1087.343 ms 899.820 ms 891.461 ms
12 bbr01-g4-0.jrcy01.exodus.net (209.67.45.253) 868.639 ms 1157.474 ms 1259.498 ms
13 bbr02-p5-0.wlhm01.exodus.net (216.32.132.50) 1395.570 ms 1417.646 ms 1440.750 ms
14 dcr03-g2-0.wlhm01.exodus.net (64.14.70.65) 955.337 ms 937.974 ms 934.676 ms
15 64.14.80.154 (64.14.80.154) 995.027 ms 1082.273 ms 1133.553 ms
16 64.28.66.203 (64.28.66.203) 1455.196 ms 1940.193 ms 1440.243 ms
17 64.28.67.61 (64.28.67.61) 739.961 ms 1072.558 ms 779.768 ms
---
when i saw all this i thought it either was my ISP being fucked up beyond recognition, or some insane DNS error regarding the new Slashdot server.
No idea if this is of any value, and haven't time to read all the replies - perhaps someone has already posted the same observation. And perhaps it was all a result of how you all tried to defend yourself against real attacs. It's only that i see you have another IP today and that "Multiple IP's" thing has vanished, so thought i'd mention this since i happened to save the output.
Well. Too tired to think - tagging me redundant and to bed.
Kristin.
(hmm and if there's spaces between lines in this posting, thank Mozilla.)
Why aren't you guys setup with Above.net instead? They are an entirely BETTER organization. AND they let you do pretty much anything you want with your setup and will keep it humming along for eternity or until your check bounces.
Kintanon
Check out JoshJitsu.info for Brazilian Ji
Posted by BSD-Pat:
;)
We have been bitching
Honestly, who cares, just don't read it. I really like slashdot because it keeps me up on a lot of stuff but this whole rating system thing is for shit. I'm not trying to flame here but other people judging the validity and insightfulness of my post? Who cares? Actually I could make the same argument. I'm stuck looking at some post that some "moderator" thought was funny and I think is a complete waste of hard-drive space. Maybe I just don't fully understand the usefullness or need for this rating system but it all seems really juvenile.
Liar! How dare you say I look like an Arrowpoint rep!
$1million/year - is that just for Slashdot or all the Andover.net systems?
1,000,000 USD = 671,015 UKP = 1,121,428 EUR BTW
Richy C.
--