Slashdot Mirror
Undernet In Serious Trouble: Any Suggestions? (Updated)
An Undernet admin writes: "For the past 4 days, many of Undernet's servers have been hit with constant DDoS, massive stuff on the order of 100M/sec that doesn't look like it will clear up anytime soon. The major services with which Undernet is associated, including Uworld and the channel service bots X and W, have been removed because the ISP that hosts them cannot afford to have them online, and even with them offline, the ISP has continued to be hit with the DDoS. Several servers will be forced to delink permanently if this continues. And all of it's happening because a script kiddie in Romania has nothing better to do with his time, and with his head start, many other groups have decided to lend a hand and take out other servers while his main pummelling is going on. We're about to run out of new ideas, since we can only code in so much security so fast, and law enforcement isn't terribly effective. What does the Slashdot community say?" There's a notice on their Web site. Update: 01/08 09:49 PM by michael : The news story we linked to was ancient.
Face it. IRC is the universal home of Those Who Have No Hope Of Ever Having Sex.
Efnet, undernet, chatnet, all the big nets. the PFY's known as scriptkiddies (some of them not even youthful pimple faced youths anymore) go to IRC because it's somewhere that magically makes their penis extend two or three whole inches, just because they can find some person or some group of persons, cause them a great deal of displeasure, and say "Look what i did!" to their buddies.
What these twits would realize, if they had grey matter operating above the brainstem, is that by doing this, they're making everyone who has donated equipment and bandwidth to IRC networks question whether or not that was a good idea.
IRC networks are going to go away because of scriptkiddies, unless these kiddies, some of them over 20 these days (get a life, folks), knock it off.
Would YOU run a public irc server if it ment you were going to get DoSed into the stone age twice a week? I sure as hell wouldn't. Maybe that's why chatnet only has 4 servers in the US these days.
All that being said, undernet has always been a haven for oversexed, underage wankers anyway.
Go ahead, moderate this post as a flame. I'm just upset because my home channel, which has existed in one form or another since the previous bush administration, has been moving around from network to network lately trying to find one that doesn't get shut down constantly by angry users, or worse yet, angry ircops who are scriptkiddies themselves.
This is just like television, only you can see much further.
I personally find this article interesting for the simple fact that I'm a Systems Engineer at one of the Undernet sites that was forced to delink last week because of the DDoS on our Undernet server[1]. I've read most of the comments, and must say that most of them are lacking in the kind of content that the ordinator of the article has requested. In fact, most of them border on immature (which must be why most of them are moderated to a 1 or a 2). With that said, many comments had useful incites, though they are defiantly not news to anyone close to any IRC network.
First of all let me state that I have as little to do with the actual operation of the Undernet server or the network as a whole as possible. That role if fulfilled by another group who works very hard with a real task and literaily deals with IRC problems in their personal time, so it's hard for me to comment on the politics of their situation. I can however, comment on the politics, and a few technical details (For certain reasons, I'm more than a little vage in what we observed during the attack) of the situation I was involved with at the time. What follows is somewhat of a chronology of the event.
Hr 1 - 3. The attack started pretty slowly. So slowly that it really didn't set of any alarms, though some customers on remote parts of the network did notice high latency, and a bit of packet loss. This was enough to start looking around, but not really enough to suspect an attack.
3:00 - 3:15: Connectivity is lost to nearly any network that requires crossing a border router. The traffic stats from the border routers show that nearly every bit of connectivity is full company wide. It was clear that at this point that this was probably an attack, though it was unknown what was being attacked, or where it was coming from.
3:15 - 4:00: Using historical data the sources of the attack were identified. Using this data, we initiated contact with each provider we have connectivity from to request filters be placed in their network to block the attacks. At the same time the company's tech support call center is overwellmed with calls from customers experiencing various problems. Further, all the major application servers (mail, news, etc) are also nearly unusable since they no longer have connectivity to the remote machines they were talking to. As a topper, one of the noisier (literaily) network monitoring programs our NOCC uses has gone into "make random noises mode." This is due, in large part, to the nearly 600 alarms it thinks exist because of connectivity problems to the rest of the network.
4:45: I remove the FDDI cables from the FDDI card in the IRC server.
4:00 - 4:30: The attack is starting to dissipate. It's theorized that it's because the machine that was being attacked was no longer on the Net. Also about this time, the distributed filtering should start taking place.
6:00: After spending a couple of hours cleaning up the mess that such an attack leaves on all the other machines I receive the standard email from the security people requesting time estimates for my labor on this afternoon's Comedy Hernia Hit.
This chronology is reflective of nearly every other DDoS attack I've experienced in the last 12 months. It's clearly frustrating, and a complete waste of my time (especially since it was my last working day before a very rare vacation), and it should be pretty clear why I don't want IRC servers on a network I have to maintain.
Let me be clear, at no point was the server itself ever effected (other than, I assume it lost connectivity to it's hub during the attack), but nearly other major application was affected in some way, and it definitely caused a lot of paying customers to not get the service they pay for.
Someone suggested that we need to prevent people from "rooting" machines in order to prevent these attacks. The poster is correct, this is what we need to do. Anyone have any ideas how to prevent this? I know all the machines on my network are secure, but I can't control machines I don't maintain. And that's just the problem. This isn't about the host sites securing their network, most of them do and the ones who don't learn quickly that they have to. Adding (more) security features to the application (ircd) also isn't the answer, as the machine itself was never affected. Hunting down the initiator of the attack only prevents that person from attacking anything for a while, like the death penality I see no indication that it's a real deturiant to the crime. Quite honestly, I too am at a loss as to what, if anything, will ultimately solve the problem short of completely abandoning the technological foundations that the Internet was built on.
As for law enforcement, they are generally quite interested in such attacks[2], but they have clear guidlines in what they can and can not get involved in (you have to show a capial loss grater than a specificed amount). In this case I know these guildlines were met, but generally these investigations go nowhere because the trail often leads to cracked machines that have no usefull telemetry of the attack, or the intrusion. I have often thought that companies who fail the maintain basic security on their network should be held liable to damages to other networks in these situations, but even that is quite troublesom.
Of course, there is one method that solves this problem, at least for me. It was to remove the service from our network. As a Sysadmin who has customer's who pay to use other services I have no trouble with this. As someone who tries to be a useful member of the "Internet Community" I have serous issues with this method. In this case, no good deed goes unpunished.
[1] In fact, I personally pulled the FDDI cables out of the machine during the attack once we determined the machine that was the
[2] Though, sometimes you have to work to make contacts with people smart enough to care.
I expect this is the Trinity attack that is described in considerably detail here by X-Force. You can find the actual article and anlysis of the Stacheldraht tool here written at the University of Washington. The author of that article claims that he wrote a program that detects Stacheldraht on a system. Of course, getting the ISPs that are sending these DDOS messages to actually use some security might be a bit difficult. By the way, this is old news, since the CERT advisory is dated June 99.
Thalia
But, if you don't feel like reading it, I'll sum it up here. and add a bit, now that I think about it.
-------
I used to be a script kiddie, then I hit puberty.
You either understand that last statement or you dont. Kids are kids, and having worked with emotionally hadicapped (not retarded) in a highschool setting, I know what they do with computers. I'm the one who had to fix them. (macs, no less)....
There's 3 reasons I've found that kids like to break things
1. They don't own it, so they cannot comprehend that it has value to someone. This is perfectlly normal for kids between the ages of 2-6, it varies in it's severity, but it usually goes away before kids are injected into the social realm of dealing with other people in school, so it's not a big problem.
2. Kids between the ages of 6-18 more commonly express their destructive skills on something because they do not understand it, and feel that by breaking it they have power over someone who does know how to use it. Ownership isn't a factor in this, I've seen kids break their own things because they cant make it work (you see this very commonly with "broken" toys in younger children.
Again, most kids will stop, or mellow down by the time they've hit puberty.
The third case is most common in mentally or emotionally challenged children:
3. "If I can't have fun with it, no one can." This is more common among older kids and extends beyond material items. This is the only case where I've found that ownership REALLY matters, but not in all cases. most people, however, grow out of this phase as well.
So what is someone who hasn't outgrown this state well past the time they should have? The police and doctors call them Sadists and Sociopaths. In this case however i would feel reluctant to use either of those terms. I think in this case it's more a case of a pre-pubescent pissing match between himself and another channel.
Back in my own script kiddie days on IRC I witness MAJOR network wars included the disabling of about 50% of the @home network in san diego, cutting down telephone poles, cutting off power to NOC's, angry kids beating the SHIT out of the kid who nuked him at school, calling in bomb threats to places, ANYTHING and EVERYTHING they can do to disable an ISP even if only for a second.
just long enough
All that shit I saw, was _ALL_ related in one way or another to "channel takeovers" some of them over things as petty as who's allowed to flirt with the only girl in a channel, platform debates, music debates... rarely over anything more mature than a 6th or 7th grade level.
Which brings up this point: most of the people who do this are still kids (under 18) so unless they nuke a military server or something, all their gonna get in most cases is a warning, maybe a fine.
So, what's to be done? I say it's time that the more mature half of the internet joins together to fight this in a way that younger kids have no controll over. I've had AMAZING success tracking down script kiddies and calling their parents. People who are clueless, or who have something to lose by being related to a kiddie, are VERY helpful.
Here's some ideasI've used and had VERY good success with.
1. Fight back online - Pro: it's fast and can be effective. Con: lowers you to their level.
2. Call their parents/employer/school*** - Pro: Can be VERY effecting in the long term. I've had people fired, grounded, suspended, and reprimanded with one phone call. Con: Can take a while, or you get someone who just doesn't care.
3. Call the ISP from which the attacks orginate.* - Pro: Admin's will always know what you're talking about, and they're usually helpful as DDOS through their systems reflects badly upon them, costing them dollars. Con: most dialup/residential ISP's dont really care or log things, so it's hit or miss.
4. Shut it all down, and walk away for awhile. - Pro: Best idea if you can afford this option. Most kiddies get bored after a few days, or when school starts. Con: depending on who you are, shutting down your system and doing something else may not be possible.
So, there you go... those are my loosely compiled thoughts and ramblings on the subject of Script Kiddies.... ciao
-Doug
Q. What's it take to get a story posted on
I've seen some amusement on this thread, amusement at the very fact that Undernet has been DoS'd.
/.'ers out there who know what a close-knit channel is like and how much it sucks when stuff like this happens.
:P) There's another Linux-related channel on Undernet which a few people split off of for one reason or another, and those people started our channel. There was some degree of disdain amongst our channel because of some of the policies of the first channel. (I like the place, though. :) But the two channels are cooperating on some of the DoS issues. We're all about Linux and getting a good place for our users to chat.
Well, don't be. It's not funny. There are people losing money because of this; there are people who are becoming absolutely brainless and deciding "Gosh, it'd be fun, let's go the way of the skript-kiddie and and help the DoS'ing be even worse!"
Then there are dedicated channel ops and owners who are building bots, starting channels, writing mailing-list software to help their members and fellow ops deal with the crap that's going on. I'm a 200-level op on one of the linux channels on Undernet (check my user info for more information) and while there are those here who feel IRC is a waste of time, I believe it's one of the best ways to communicate with people all around the world about a common interest. If you don't like IRC you don't have to use it. I can see how some people think it's a waste; but it's something I enjoy. And so do 20-odd other ops and regulars in this channel.
I met these people because they helped me install Linux over two years ago; there are ops and regulars who are good friends of mine from Australia, New Zealand, Canada, the US, UK, Malaysia, Germany, Greece to name a few. We put faces to the names via webcams; we know who's going out with who, we comfort our friends when they're going through crap, and we came together and cooperated with a mailing list and new bots and new policies once W went on the blink.
Someone tried to compromise our channel yesterday (a takeover, for the unschooled) but order was restored. With W (X for other channels; we happened to have W when he was still around) the oplist, auto-kicks, and bans are very easy to store; without W, the guy managed to get ops by pretending to be one of us. Could have done some damage, but thanks to some IRCops (Thank you seti and saralee!) order was restored, new bots put in place, and new channel policies. I know there are other
Right now there's rumors that W and X will never come back. If they don't Undernet is dead...and where is a channel to go? Some IRC networks have strange ident issues; some are dying out; and some have a structure such that it's hard to even keep hold of a channel because of skript kiddies. Right now Undernet splits a lot--too many users and not-so-perfect routing. It's also hard to connect to a server. There's a lot of lag.
And now I get to a point I think bears hearing: Forking doesn't mean animosity. (Are you reading this, RMS?
To the skript kiddies out there who are continuing to pummel Undernet because you think it's cool: Stop acting lower than dirt and get a life. You can find something better to do than cost people time and money.
"The GIMP Girl"
Angry IT woman in big clompy boots. And talking lint!.
quick story
I remember getting TONS of spam from a machine a major university. It appeared to be a machine running in the astronomy dept. I sent a nice friendly e-mail about it, as our users were getting 20 to 30 spams a minute through it and wanted to stop being told where to get Viagra (Bob dole already told us thank you). The official response from the sys admin was a none to polite, "Fuck you and mind your own god damn business".
My response was to cc that with a letter asking a bunch of questions to 2 local newspapers and 1 TV station and the president of the alumni association. The open relay got closed *magically*
What the point to my incessant yammering you ask? Sometimes ISP's (especially smurf sites in Japan *ahem*) need to be bullied into doing some of the most obvious, easy things. Some ISPs claim that filters cause problems, increase router load etc, etc, etc. The problem usually is that no one has brought it to their attention, or rather no one has screamed at them loudly enough.
"Science is about ego as much as it is about discovery and truth " - I said it, so sue me.
DUMP THE ROUTE As soon as possible stop advertising the affected block to your peers, this is the fastest way to prevent the traffic entering your AS and saves bandwidth on your internal lines. It under your control and its faster than informing all your peers and waiting till *they* get filters in place, its not their problem and even if they filter the traffic it still takes their external bandwidth.
This depends on your BGP config and a few things will happen, firstly if you're a large ISP you're going to lose other customers as you're not advertising their IP addresses and depending on peering agreements the minimum could be as large as a /20 or /19 but its better than lossing the whole network and all your customers! If upstream peers from you are not aggregating your routes this will in effect remove the route from the whole net (might take a little while to converge the whole net) and the traffic from the attacking DDOS machines won't get very far (their own subnet). If your routes are aggregated upstream and you've withdrawn the route the traffic stops with the upstream ISP anyway.
This should give you breathing time without the loss of your whole network and (at least you'll have bandwidth to telnet to your routers) identify which machines were getting attacked. Talk to the upstreams and get them to dump the host(s) specific route to null.
I meet far to many network admins that think they know everything there is too know about networking that just state "what can I do but put filters on the border", which is fairly useless for preserving external bandwidth which of course is what your customers are paying for.
BTW, while I'm here, anyone want to give me a job?
Will configure routers for food.
A journey of a thousand miles starts with a brutal anal raping at airport security
Hrm. Bad analogy.
More like if you decided to drive an unsafe car on the road. And no, you don't have that right (at least not in North America).
--
All men are great
before declaring war
A government is a body of people notably ungoverned - AC