Slashdot Mirror
Kuro5hin - Bitter and Hopeful
Dylan Griffiths, known to Kuro5hin users as Inoshiro, gives us the sysadmin play-by-play:
"This started on Sunday night. Basically, I had been over at a friend's place, there had been a storm watch, and he's a ham radio guy. He's a member of Canwatch, which is a volunteer ham radio thing you can do once you get a license. We were out driving around all afternoon. We got home, watched some TV, and dropped me off at home. At that point it was pretty late and I was about to go to bed. Normally, I would just go to bed, but I sat in front of the computer to check out Kuro5hin, and I noticed that there were about nine stories in the moderation queue. I thought that was a bit odd, because we normally get one or two stories at a time, and they get voted on, so they either show up or disappear quickly. I went to the submission queue, and I saw one or two stories posted by people with handles, and the rest were all Anonymous Hero. I initially thought that perhaps some fellow had decided to post a few things on Sunday night so it would be there for Monday morning, because weekend traffic is about half of our weekday traffic. I figured I would just delete the extras. The subject lines for the submissions were all just random strings of text. I didn't know why that would be, so I deleted a couple of them, and noticed that a couple came back. So, I logged into the server and I was going to see if I could block the garbage submissions. I also logged into the IRC channel to see if anyone knew what was going on. That's where people told me about a user named Kano, and how he was angry that his story was voted down so quickly. In the interest of getting the facts, I wanted to block what was going on, and get more of the story. I blocked it, fired off a couple of mails to [Kuro5hin creator] Rusty (Foster), and talked with some of the guys on IRC because on the whole, they're nice people. Kuro5hin has a great bunch of people that helped me and Rusty through this. We talked about it, and one of the channel members mentioned that the machine the attacks were coming from looked like it had a bunch of ports open. When I traced it through the whois database, it was a part of a server farm in a hosting company. So, you'd think they'd only have web, and maybe ssh and telnet open for admin purposes, and everything else would be centralized, because that's what you do when you have 400 machines."
The team leaps to action
Inoshiro continues, "Rusty joined the chat on Sunday night, and the IRC channel users banded together. We banned two subnets, and the channel people helped us clean up the submission queue. The box on one of the subnets we banned was obviously cracked. In addition to ftp ssh and http, they had sunrpc open, nfs, mysql and irc. So, besides the obvious fact that mysql should be open like that and the Sunrpc services, irc is something you don't see on a webhosting farm. I don't think the spammer expected us to block him quite so quickly. It took me about 40 minutes for me to block him because a router between me and k5 went mad and was giving me 3000ms latency. It was the first time I'd actually had to do it. Once it was blocked, that's when the channel helped us clean up. Then, within 20 minutes, it started coming in again. That one was blocked within about ten minutes, and that was a proxy server. Everything else since then has been cracked boxes.
"I got it down to the point where we would see five scroll by, and when we got to the end, I basically ignored everything else I was doing, and blocked submissions as they came in. It wasn't until Monday night that the router between myself and k5 stopped giving us incredibly high ping times.
"I went to bed, and I slept in a little bit. I got up, joined the channel. Since I finished school earlier this month, I talk to people in the channel in the morning because most of the people I know are asleep or have a job. I've been sort of looking for employment recently, but I've been spending a lot of time working on k5. I usually talk to them in the channel, because Rusty was gone for two weeks and I was the only admin around. I'd been spending more time just talking to people. We had a bit of a chat, a few people proposed ideas about who they thought might have done it. Nothing was really resolved. Then I noticed that there was more stuff coming in the queue. I contacted Rusty at work, and he joined the IRC chat, and we talked about it. We spent Monday getting some of the scoop developers to disable anonymous story submissions, then we added logging to a bunch of things. Basically, Monday was the day when we were babysitting k5. The poster would switch their submission to a new cracked box. I was watching the output of the log and ipchains the subnet, look up the person responsible, and cc: it to Rusty. The people Rusty used to work for, intes.net, offered legal support. They've been really great about it because even though Rusty doesn't work for them anymore, they were still hosting the box until we get it all moved."
On Tuesday, the system abuse continued not only in the submission queue, but also in the commenting system used by readers to share their feelings or concerns about news items that Kuro5hin posts throughout the day.
More from Inoshiro: "I mailed [Slashdot Founder] Rob (Malda) on Tuesday morning, and I wasn't sure how he'd take it. Usually his replies are given out with as few words as possible. After a couple of replies, we were sending 8 or 9 paragraphs back and forth all day. He suggested a few things, and Rusty said he didn't realize it could have gotten that bad so quickly. My buddy from Sunday came over, and I watching Kuro5hin and he was helping me set up networking booting with an OpenBSD box I have here. It was ten o'clock, and we went to watch The Simpsons. While we watched, the guy had just been spamming the server more. he started spamming about fifteen minutes after we went to watch The Simpsons. How could someone do this? This is like proving a windshield is made of glass by smashing it."
So, at three in the morning at the Villa Hotel in San Mateo, Rusty Foster, Kuro5hin's creator, replaced his website with a black page telling the story of the denial of service attacks. I got a chance to speak to Rusty today while he was in his office at OpenSales.
Rusty said, "Today I'm bitter and hopeful. Yesterday I was bitter and depressed. It bothers me a lot, is the best I can put it."
The fact that Kuro5hin is entirely volunteer-run, added to the fact that they've got an active IRC presence and die-hard fans, lends itself to community building. People read Kuro5hin, post comments, and share their feelings and criticisms with people around the world. In the end, the Kuro5hin staff is resolved to not let the misguided destruction of one incident destroy the community they have built from the ground up.
"I think that we will get the site back up," Rusty said. "It will not be entirely the same as it was before. Anonymous access is gonna go. That's all there is to it. There's a place for anonymous access and I'm all for free speech, but there's also got to be a place for real people who will stand up and identify themselves, more or less. We're not even asking for identities, we're asking people to create a pseudonym and use it. Slashdot pretty much has the market cornered on free and open access, and I'm a lot more impressed now with the crap you put up with."
I'm aiming for a month. I'm leaving in August to go to Italy, and then immediately after that, my sister's getting married. I won't be back here with reliable access until the middle off August. There are a bunch of great developers that work on the code, and I'm going to put together a list of things that need to be done. Knowing them, they'll probably do most of them. Whatever remains, I'll do when I get back, and then we will re-launch amid great fanfare. I got a lot of great E-mails from people supporting the site, and a lot of them supprting my decision to close it until we've taken care of the problem, and I would like to thank them collectively for all their support, making me feel better, and inspiring me to actually get the site back."
Update: 07/26 08:59 PM by CT : Just wanted to throw my 2 bits in... VA Linux Systems is gonna help with some hardware since the Kuro5hin system really was strugglign to keep up with their existing hardware. That doesn't address the spam attacks which we've also spent quite a bit of time discussing. I'm personally finding this really interesting since I've gone through it all with Slashdot over the years, and seeing it done to someone else with the benefit of hindsight and experience is quite interesting. The frusteration you feel when something you work so hard on is screwed with by troublemakers is hard to describe: especially when you're just a volunteer. Slashdot wouldn't have survived that stage without help from a lot of people... Best of luck to you guys, and I hope to see ya pull through this.
Apparently this script was used to spam K5, and the guy that created it has a web site, although it offers no explanation on WHY they did this. Maybe having the script will help you block it. The address of the script was posted as another anonymous message in this thread.
He claims he was inspired by Slashtroll, a similar script for trolling Slashdot. The author of Slashtroll (zk65) removed the program after seeing what happened to K5, and posted a message here.
I think this was metioned in the other story, but it's such a good idea that it bears repeating here.
How about making kuro5hin based on a trust metric?
Here's how it might start out. rusty and Inoshiro and a few trusted other (perhaps loyal kuro5hin readers) would start off as the web of trust. As people begin to submit stories and get them moved to the front page, they can get "moderated" up to be trusted to submit reasonable stories. Perhaps as people gain trust, they can have their stories moved to the front page faster. Presumably, these same people would eventually be included in the trust web and extended "moderation" privileges. And soon you would have enough people that the load would be distributed evenly.
Of course, there could also be an increasingly (exponential) penalty for submitting crap, eventually cumulating in the banishment of the user/IP from submitting stories for some amount of time. If the banishment is not for all time, then the trust would have to be slowly extended back to this person. This would hopefully prevent cyclical occurences of spammation.
I think this preserves the idea of kuro5hin, allowing the community to decide what gets posted, while limiting the community to something reasonable. The same idea could even be applied to comments as well, to prevent people from screwing the comment queue as well.
Thinking of it in Slashdot terms, for those of you who are die-hard Slashdot fans, the trust web is akin to karma.
I really miss kuro5hin. This was the first idea that popped into my head for fixing things.
What do people think?
It's because they let geeks run the site unfettered, they need to get some suits to sit in their big chair and anticipate such things.
Blender And Linux Fan
It was ten o'clock, and we went to watch The Simpsons. While we watched, the guy had just been spamming the server more. he started spamming about fifteen minutes after we went to watch The Simpsons. How could someone do this?
Damn right! Doesn't this cracker have any sense of cultural literacy? I bet he watches the Home Shopping Network for fun.
"If one is really a superior person, the fact is likely to leak out without too much assistance" -- John Andrew Holmes
But here's how I see it, /. is on hardcore equipment, and pays people to run it. If I ran a server (NT jokes aside) that was this unreliable I would be fired in about a week.
How about the odd story that at least tells us what is going on. Just throw something in the quickies like Hey we had some problems due to a mySQL misconfig, here's what happened and why. Not only would this satisfy a lot of us /bitchers, but it may provide a learning experience for all of us using similar tools.
Why did we think public-comment websites would be substantially different from Usenet? The only real social diff here is that Usenet has a much bigger group of volunteers trying to keep it working (cancelbots, etc.). It seems like the experiments in trust-based submission networks haven't given use the best answer yet.
I feel really bad for Kuri5hin. But as a denizen of one of the hotter parts of Usenet for the last decade, it is all eerily familiar, and in these web-spaces there are no killfiles to adjust.
Anyway, for all those who can't wait, basically, I appreciate all your support a whole lot. A bunch of people have offered various things, from hardware to bandwidth to security services, and they are all appreciated. I'm just trying to get on top of the whole situation right now, but I will get back to everyone who wrote. This community rocks, and is the reason I'm "bitter and hopeful" now rather than bitter and depressed. Thanks all.
--
There is no K5 cabal.
I am not the real rusty.
k5 and /. were never enemies. Its some sort of rumor. Don't spread it any more. /. has given tons of help in getting k5 back up and running, by donating servers and expertise. So /. is a great help. Don't say that there is some feud or that they hate us. they don't.
Here: roblimo actually ordered Inoshiro a pizza, bacuse he hasn't eaten yet today. Roblimo's in MD, Inoshiro's in Canada. That, I think, is above and beyond the call of duty. The conspiracy theory is not true, no other discussion is necessary.
--
There is no K5 cabal.
I am not the real rusty.
"Howdy. I've been reading k5 for a few months now, and I was really getting to enjoy it. Not just the site, but the community of people that read and posted there. Needless to say, I was saddened to find that k5 has been brought down by script kiddies. I'd like to do something to help, but I probably can't offer anything in the way of coding skills that you guys don't already have. Thus, I was wondering if I'd be able to send you guys some sort of monetary donations, to be put towards higher-end hardware or better net connectivity or whatever. The only other person I've talked to about this is interested in donating as well.
Hey, we're a community, right? And aren't community members supposed to help each other out in times of need?"
This pisses me off. Thanks to these 31337ers, I now have to go a whole month with no kuro5hin. What's sadder, is that kuro5hin is now getting so much publicity that it'll probably turn into another Slashdot, with firstposters, natalies and penis birds.
On a related note, what's up with Slashdot tonight, it seems slower than ever... Hello, am I reaching?
--
"Oppression and harassment is a small price to pay to live in the land of the free." -- Montgomery Burns.
In case you are a bit confused at this point, Dylan Griffiths' K5 nick is "Inoshiro", not "Iroshiro". Sorry, Emet. Erm, I mean, Emmett :)
I know, you're thinking, "but Fox shows Simpsons reruns every hour where I come from", but some backwards affiliates have cut down to showing the Simpsons only 11 (or even as few as 6!) times per week.
So don't be silly; the cracker was probably working from another timezone where the Simpsons had already ended or hadn't yet begun. I mean, just because he's an immature criminal vandal doesn't mean he's a complete monster!