EFnet Hits Turbulence

Lots of submissions regarding a bumpy week for EFnet, mostly short on fact and long on rumor. Several high-capacity servers have either dropped off entirely or limited their connections to local clients due to DOS attacks. We got one good link about the situation; anyone else have more info? Is this a real problem or just normal roughness? I'm not an IRC regular these days but I've never seen a stable IRC network.

Comments

yes, real problem

[Brian Feldman]

This is a real problem. I'd say a very good portion of the traffic I've noticed is servers netsplitting and synching up -- good that it can recover at all, but a painful recovery. Hybrid is being modified to accept higher loads (in various ways; inter-server protocol enhancements are one, as is work on using asynchronous event mechanisms rather than select()/poll() and reducing the overall "weight" of the implementation), so hopefully even through DoS attacks, in the future EFnet will be better about it.

Note that this is just one aspect of recent EFnet suckage :) Before you flame me: no, I like EFnet still, even though it has recently sucked very much.

--

Blackened

[toofast]

I remember when blackened.org went offline because of DoS attacks. They had the ability to serve up to 7000 IRC clients. One of the main reasons for killing the server, IIRC, was because of an evening where a bunch of idiots threw tons of garbage down blackened's pipes, causing the entire state of (arizona?) to be deprived of internet access. Although I cannot find Matt's original letter, I did find the config of irc2.blackened.com:

oldcharred.blackened.com: AMD K6-2 @ 333mhz, 128M of ram, 18G-10k rpm scsi primary, 9G secondary. This server houses the origional irc2.blackened.com EFnet server, the largest EFnet server in the world before it de-linked. Still running with the origional IRCD, I, O, C/N lines and TCM.

It's a pity that, in blackened's case, volunteer workers such as mjr are forced to abandon what they love to do, because of immature kiddies flooding the network with useless garbage.

Nothing to see here, move along.

[pim]

Disclaimer: I'm fairly new to the efnet experience. I've been running Undernet servers[1] for the past two years and only recently linked a server to efnet[2].

I haven't yet found someone who has been able to figure out where these rumours have been coming from. We got a couple enquiries about "is efnet going to shut down" in our efnet mailbox, but that's nothing out of the ordinary (Imminent Death of Efnet Predicted - Film at 11). Haven't seen any mail claiming that anything really special is going on. A couple of servers changed their policy. As far as I understand, from my limited experience, there's nothing strange or extraordinary about that. IRC networks are dynamic in nature.

The amount of DoS-flooding that goes on directed at a typical server for a major IRC network is completely out of bounds. Scriptkiddies see themselves as Freedom Fighters and Mighty Warriors, but are slowly pushing IRC networks to the point where they either become unusable or virtual Police States. On some networks, ideas have already been coined to start using a mandatory user registration system. No admin likes the privacy implications of such a move, but it may turn out to be the only way to keep the idiots out.

Once in a while, we get lucky and one of these kids touches a site that a federal agency cares enough about to start a case and the world has to deal with one scriptkiddie less. Most of them never get caught, though.

HTH.
Pi

[1] saltlake.ut.us.undernet.org and haarlem.nl.eu.undernet.org
[2] efnet.vuurwerk.nl

Re:IRC = warez + child porn

[MartinG]

> , and all of the other IRC networks,

Getting a bit general aren't we?

Have you ever visited irc.openprojects.net for example?
- Lots of useful discussion regarding development etc.
- Frequent conferences held discussing the direction of open source projects
- Much much more. There's even a #slashdot channel. The one single file I have seen on this irc network for offer over dcc is a linux kernel patch in #kernelnewbies. That's hardly what I would call illegal or immoral.

I would also add that there are similar channels on #efnet. Just because there are a lot of bad goings on is not a reason to punish the legitimate users by getting rid of the networks.

It's a size problem

[Masem]

I've got no proof, but based on what I know on IRC networks and the types of host-leaf connections you need to set up, I'd say that maintaining the stability of a IRC network increases with O(n^2), n being the number of servers. I've found EFNet always relatively unstable, Dal somewhat so, and smaller networks rather stable. I think some of this has to do with the ability of smaller networks to stategically plan where to plug in new servers carefully, while larger networks have to haphazardly place new servers based on the best info available to them. There's also something about "too many cooks" .. Dal has fallen apart due to so many internet IRCOP conflicts, it's not funny.

The only disadvantage is that you can't have as many clients -- but save for help channels, how the heck are you supposed to have meaning conversations in an IRC channel with over 50 or 100 people in it???? I think smaller networks will make IRC a bit more 'worthwhile' in terms of it's original concept.

This is for real, unfortunately

[Phaid]

Since the end of August, EFnet has become a real pain to use. Some of the better servers, like core.com and primenet.com, have simply gone away, and others are just about impossible to get to. The ones you can get on go up and down all the time, there are endless netsplits, etc. The only semi-stable servers either belong to .edu's or are part of some network like mindspring or home.com and don't let anyone on who isn't a part of their network (understandable, but frustrating).

The article is correct in one thing: it's because of the packet kiddies. With hundreds of kids behind cable modems blasting away at servers all day long, it's no wonder that network admins take down IRC servers -- the turnover rate on EFnet servers has been amazingly high recently.

The one thing to take comfort in: despite its problems, EFnet is still "the" IRC network to most people, so if you're on another IRC network, it's taking the brunt of the assault...

Re:IRC is kind of dying in general...

[JatTDB]

Just because the "newbies" don't use it, that doesn't mean it's dying.

The "newbies" also don't generally go to tech-info-heavy text only websites. The "newbies" don't normally use FTP in a non-URL-based way. There are a lot of things out there in this wacky world of the internet that the newbies will never try out or understand. That doesn't mean that any of them are dying.

How can something with 40,000+ client connections at any given time, and often going to over 60,000 possibly be considered dying?

Re:Well,

[Kintanon]

EFnet has been a great resource for me for computer help, etc....though I've been told once or twice to RTFM. But the people there have been generally more helpful than irritating, so I'm upset to see them getting DoS attacks, etc.



If you get shunned by Efnet come to Undernet. #linux and #linuxhelp and #techies are prefectly great places to find info. Undernet is alive and well and relatively trouble free.

Kintanon

Re:IRC is kind of dying in general...

[FallLine]

Yes, I agree, but....IRC should be dead by all rights. It is an inferior protocol implimented on mediocre servers connected haphazardly and administrated mostly by idiots [There is hardly a meritocracy amongst Opers...it's who can kiss ass the most]. The only reason the 3 major IRC networks still survive is for social, not technical reasons. People stay on IRC for the "social" structure or fabric, if you will. Put simply, all their friends are on IRC network X, so they stay put. Even though there are superior alternatives out there for most every application, the loose knit groups can't, or won't, coordinate their movements and move at once. Thus, when the individual is given the choice between using superior protocol A on their lonesome, or using crappy protocol B with their "friends", most will choose B and put up with all the crap they have to endure.

Why EFnet sucks

[Fas Attarac]

In the beginning, there was IRC. IRC was good, people got along, and chatting was what people did.

Then there were some differences of opinion between administrators. It's OK, these things happen. Feelings got hurt, EFnet spawns a child network. Increasingly, this happens more and more, but typically the arguments revolve around the introduction of features to give the user a better chatting environment.

There are always two sides to the argument. There are those that want things like channel ownership, more IRC operator participation in the affairs of mortals and harsher, coordinated controls against abusers of the service. Then there are those that don't want anything to change. They view IRC operators as the keepers of the links, and that those keepers should never meddle in the affairs of the users. Let them sort (battle) out their own problems. EFnet splits. The liberal operators and servers (the ones wanting the change) spawn off a new IRC network, and the conservative/reactionary operators and servers stay behind.

Think of it as evaporative cooling. As EFnet experiences its civil wars, the proportion of "to hell with the users" attitudes rises.

Eventually, this attitude starts biting the opers and admins on the ass. EFnet turns into a war zone, with DoS attacks starting to show up. A few users think they're funny and DoS the opers too.

The ugly dragon rears its head.

Now the attitude becomes "fuck everyone but my fellow opers". IRC wars move from the IRC playfield to the Internet with DDoS attacks taking down servers for the purposes of channel warfare and retaliation against opers and admins. Sometimes the ill feelings are warranted, but mostly the packet kiddies are just trying to make a nuisance of themselves. Networks suffer, ISP customers suffer, ISP's de-link their IRC servers. A free service (IRC) should not--must not--impact the ISP's ability to reliably serve its customers.

Now at this point, EFnet starts getting a shortage of big servers. Naturally there are dozens of ill-experienced, IRC savvy packet kiddies that have "grown up" a bit and want to try their hand at running servers. A few are cautiously linked in, oper abuse (already rampant on EFnet) begins to rise even faster. The line between oper and kiddie twists around a bit, more servers run by "former" (or current) packet kiddies, Internet wars abound, servers are split, packet kiddies continue to attack. Legitimate, well-staffed servers jump ship.

EFnet, in short, goes to hell.

I've always said EFnet is the ghetto of IRC networks. I would wager the vast majority of people that use EFnet to chat nowadays do so only because their friends are there, or they don't know that there are alternatives. If there was a way to reliably migrate a person's group of friends instantaneously to another more mature network, most would do it. I would.

Re:On limiting to local clients...

[Fas Attarac]

Limiting a server to local clients has always been an acceptable and practiced policy among EFnet servers. Generally if a server is capable of being open and handling non-local clients, it should do so, but if an ISP has a sizable population of IRC users, and sufficient hardware to support those users and little more, it makes sense for that ISP to set up its own dedicated IRC server for its customers, and to link that server to EFnet.

AOL and Netcom are prime examples of large providers that have opted to build their own IRC servers for their own clients. Unfortunately (mainly in AOL's case), they didn't feel obligated to police their own servers, and abuse was quite rampant.

Re:On limiting to local clients...

[Fas Attarac]

It's an issue of trust. A rogue IRC server can introduce any command into the IRC network that it wants. If there weren't any other opers watching the network, that server could cause anarchy.

They also require servers with large pipes for a reason: the IRC2 protocol is not very efficient. It's entirely ASCII-based and depends on things like connections, quits and channel messages to be propogated throughout the entire network. Thus, your private server wouldn't necessarily just be seeing traffic it needs to see, it would be seeing ALL traffic across the network.

Ordinarily, this amount of data isn't too bad. An ISDN link could probably handle it. The problem is the connect bursts. When two servers split, as you know, each server on the local side of the split sends out QUIT messages for each client that has now disappeared on the other side. If this is a major hub, this burst in itself is quite large. In addition, when servers reconnect (such as when your private server connects to the network or when a split server reconnects), a much larger connect burst occurs, as each client on the "other" side is introduced to the local side. JOINs (well, SJOINs) have to be sent as well, so that all servers have enough state information about what clients are on what channels that they can provide that information to the local user.

In short, IRC2 needs "HUGE pipes".

Yes, it could be designed better. Yes, there are better IRC network designs on the way, but there's little that can be done to fix IRC2.