Slashdot Mirror
A Day with an ISP Spam Investigator
scumbucket writes "Network World Fusion has an interesting article about an abuse investigator for ISP Earthlink and his job of tracking down spammers. It's nice to see that major ISP's are making an effort to shut spammers down and kick them off of their networks."
Not that interesting really. No specifics, not much technique. He calls offenders, cancels accounts, etc. Phishing is another department. He doesn't take action on pedophiles and refers them to cops.
Where's the beef?
"...all the labours of the ages, all the devotion, all the inspiration, all the noonday brightness..." yada yada
Well they dont do it because they wont to help the world. But spam means extra bandwidth, so extra cost. And maybe customers blame the ISP, so that might mean less customers. So it is the ISP's best interest to do something about spam.
http://www.virtualconcepts.nl/
This si interesting, but you have to say this guy is fighting a losing battle. You have to fight Spam at its source. Look at the Spamhaus statistics: it might sound trollish, but spam is evidently an American problem, which must be combated in America. The Spamhaus stats prove it. 90% of the spam you see is from 200 individuals, of whom 96% are Americans, operating out of america.
Clean up your act guys. When you're costing the world this much money, it just isn't funny anymore.
Meine Schwester ist sehr, sehr reizvoll - Nietzsche
SCO.com uses Linux
What's needed is every ISP having a consistently responsive abuse department. From what i've seen, everbody except the largest tier-1 ISPs do, with most of them having a substantive presence on abuse and anti-spam lists, and responsive to complaints.
.us tier-1 ISPs and most of .cn/.kr that are seriously culpable these days; from what I've seen working in the anti-spam arena these last six months, uu.net/MCI and their peers don't give a shit because, well, nobody's going to refuse to peer with them if they host spammers. Same thing in .cn/.kr, their broadband industries net the larger .us providers large $ over the longer term, and it's not in their best interests to be overly proactive.
.kr ASs in real time; we were given a demo at AusCERT 2004. The fact that they won't use this more proactively is depressing.
It's the major
Which is a shame; KISA (.kr equivalent of the FCC/ACA/etc) have got a great early-warning system set up, which shows transit traffic between
About 40% of my current spam corpus is from korea, the other 60% is about 30/30/40% china, uu.net, and comcast/verizon open proxies.
You're doing it wrong.
Yes, the typical spammer is a slashdot-reading geek who lives with his parents. ... Reminds me of a thing I read earlier warning parents about signs of their child engaging in dangerous hacking, such as use of Linux or requests for better hardware.
Just what a geek needs, another reason for parents to be suspicious of his computer usage. Help! I'm a computer addicted teenager who can't stop sending spam, and this is really a cry for help!
"He only reads the content of an e-mail in extreme cases, he says."
I've always found it safest to avoid reading email, unless I'm feeling really daring...
Online & Feelin' Fine
Well they don't do it because they wont to help the world. But spam means extra bandwidth, so extra cost.
I've heard many a system admin complain about the "cost of spam" to their networks, but have not seen a quantification of that cost. Given that spams are so small (the ones that I get average 4kB/spam), the storage costs of saving every spam (at 1$/GB) are about only 4 micro$/spam and the transfer costs (at $3/GB of transfer to pick a Google figure) are only 12 micro$/spam. Even CPU time is cheap. If a $2000 server CPU can handle only 10 messages per second (an underestimate?) then the cost in CPU time is only about 6 micro$/spam. In total, a million spams would cost an ISP maybe $20 or $30 which is far less that the burdened labor cost of one hour of a technician's time.
What am I missing here? Can any admins tell me the true dollar cost per spam? The only other reason, that I can think of, is that Earthlink fights spam to avoid blacklisting because blacklisting would drive up support costs when a million customers call at ask why their emails aren't getting through.
Two wrongs don't make a right, but three lefts do.
Well, with these kind of statistics, he'll be gainfully employed for years to come.
While he believes his job is important, Rush doesn't take the role of Internet cop too seriously. But he admits with a chuckle that his favorite computer game at the moment is called City of Heroes.
I'd sit back all day and play CoH, and tell my boss "Sure, I killed off 800 spammers today. But 30,000 more popped up. Guess I'll be seeing you monday."
I need me one of those gigs. Anyone offering?
Oh, and it's not censorship. He's not a government or publisher. The spammer can find other places to publish his work other than my mailbox. (Just like wannabe painters can't exhibit in my living room.)
One line blog. I hear that they're called Twitters now.
Incidentally, this bit:
was interesting to me. This sounds like the oft-repeated assertion that a US flag with a fringe in a courtroom means that you're under Admiralty law, not the law of the United States, and that anyone who appears before that court has lost most of their rights. Of course, They don't want you to know this...or that England still owns the US, or that there is a subtle yet vitally important difference between the United States and the United States of America that means you are 0wn3d by the government...
I tell you, there are worlds upon worlds of free entertainment out there on the Internet.
Carousel is a lie!
im no sysadmin or anything.
but if its 30 $ per day, its 10k per year.
further more you have to spend time and energy you have to spend sorting the mail. this is, ive heard, quite expensive in CPU time.
The best filters catch 99.9% of spam and only make 1 mistake in a thousand. ( i don't even think that they are that good).
1000 emplyoees gets 5 mails aday for a year thats 1.8 million mails, thats 1800 mails per year that goes down the drain. im not sure what that costs, but some of the are prolly quite expensive.
This is not absolute facts nor close, but my point is that the price of spam is more than the price of reciving spam.
spelling is for people who doens't know better...
I used to work at a small ISP -- lets say 5000 customers. We were getting lots of complaints about spam, so we decided to put in better spam filtering. That required a bigger server. Then the mail server went down for half an hour because of the volume of incoming spam, and there was a suddenly a big rush on getting the new server up and running.
The server was the cheap part: let's say $2000 (all figures Canadian) for the box, rackmount, hard drives, yadda blah. Thank God for Free software, because FreeBSD and SpamAssassin saved our asses. It took me, conservatively, three full days to set up and get it more or less right; I was doing a lot of learning on the job, and the regular sysadmin was away.
Now, don't forget that we were down for half an hour. This was from roughly 9am to 9:30am on that day, so that's a busy fucking time for us. There were tons of calls and only three people to handle them; fortunately, I was pressed into service trying to fix things, and wasn't on the phones. We probably lost a couple customers then, but most people were pretty understanding, especially when they were told it was fuckwad spammers who were causing the problem.
Complaints were a huge deal, both before and after the filtering was put in place; I was dealing with most of them, because I was doing abuse duties, and it wasn't fun. Complaints before the new server was installed went, "Why am I getting all this spam? Why can't you stop it?" Complaints afterward went, "Why am I still getting all this spam? Why isn't your filtering working? What do you mean, I have to set up my mail program to do more work?" (We set the threshold rather high, thinking that customers could use filtering in their mail client to set their own tolerance level. Ha! It is to laugh. Ever tried filtering on random headers in Outlook Express 5.0?)
Plus, there was maintenance of the server and software; upgrades were never fun; false positives happened and were dealt with; and now, my sources tell me, they've graduated to buying dual-fucking-xeon processors in order to handle spam filtering. Fuck me!
But hey, we were after a dollar cost, and I did get sidetracked. We already said $2k for the server. Three days of my time, $400 (deal!). Half an hour when everything in the company came to a halt because no one could send mail or do anything but answer the phones: $500, and that's probably very conservative. Two customers' worth of lost revenue for a year: say another $500. Spam complaints before took, oh, probably a good five solid days of my time: $650. Afterward was probably the same, so another $650. I know of at least one customer we lost afterward when the spam filtering wasn't the magic bullet I kept trying to tell them didn't exist, so $250. Bandwidth for all the spam we were accepting but kept from reaching the customers: let's say $50, for a nice round total of $5000.
Now this is very, very rough back-of-the-envelope calculations for a small dialup ISP I no longer work at; the managers there could probably tell you more about lost good will and so on. More importantly, it doesn't tell you about ongoing costs; that's just a snapshot from when I worked there. But that was $5000 spent by an ISP that was going down the tubes (true story), just to keep up (barely) with a denial-of-service attack that was slowly grinding us into the floor. I can't even imagine what it's like for AOL or Hotmail. Nor will we ever know what that time and effort and money might have done if it wasn't being spent on spam.
Goddamn fuckwad spammers piss me off.
Carousel is a lie!
Fun article for me. 25 years ago or so, I was the original "cable cop" in Michigan, USA (the job title was "system auditor"). This was before it was illegal to "steal" cable services, and the overall thrust of my work was to build a case for legislators.
About 50% of my time was indoors, pulling street-by-street printouts off our Tandem system and cleaning up/verifying account info by going back to original install paperwork. The rest of my time was spent climbing poles, verifying hookups and disconnecting the "non-subscribers." After a year of that, we had enough info to deliver numbers to the statehouse: 4% of all cable viewers weren't paying us for the service. That was enough for the legislators, and cable theft became a mid-range misdemeanor.
So then I started going after the midnight installers offering people "free HBO forever" at the low low price of $100 (or whatever). That was kinda fun...serveral times I was just hours behind these guys, removing service drops while the resident stood by watching, moaning eulogies for their recently departed 100 bucks.
I'm surprised that more ISPs don't have employees like the guy in TFA (or perhaps I'm surprised that we don't hear more about them)...losses due to spam are real, no? [In the case of cable, the "losses" were 99% paper; there was no extra drain on bandwidth, no guarentee these folks would have been paying us otherwise, and no real loss on the converters they were using (our collections folks did just fine charging 4X the cost for unreturned equipment). The only true "loss" was in tech-time, for the rare hookup that caused interference on a distribution line or radiated enough signal to breach FCC rules.]
Is the reason for this apparent lack of interest on the part of ISPs similar to that of the credit card companies during the early online days? Rather than appear inept at providing decent system integrity (easily spoofed card numbers, pitiful account verification, etc.), fraud and abuse were handled quietly, with costs taken off the bottom line. Or is the apparent less-than-vigorous investigation of spammers just part of the "?" step in the profit! formula...where bandwidth lost = cost of investigatory personnel, so screw the inconvenience to customers?
education is no substitute for intelligence
Several years back the local ISP for which I worked had a spammer force us to take our mail server down because his advertising bomb went off in our spool drive and completely filled it. It took a number of hours to manually clean it up, sift through logs to find and block the offender, and bring the server back on-line. Ask our business clients how much not having email available for several hours cost them. Just for illustration, that email was also only about 3k in size, but once it multiplied in the queue it consumed all 2GB of the spool.
More recently, the local ISP for which I often do admin work had to build three new incoming mail servers and purchase spam and virus filter software for each machine at the rate of at least $6000 ea. plus subscription. Without these machines, user mail spools were filling up with spam and viruses; the older the account the worse off it was. Ask these folks how much it costs.
I have seen spam perform the equivalent of DoS floods: causing servers to crash, filling up T1s, causing CPU loads on older but otherwise working machines to hit 98%, and more. I host a domain which sees 28,000 spams per week on average. We employ RBLs in our fight against spam, as well as blocking a number of countries known for delivering nothing legitimate to our servers.
We see the shit come from all directions. In one night I observed a spam run against a hosted domain attempt to deliver 5,821 messages -- all forensically identical -- in less than 100 seconds from roughly 15 sources.
Why should it be the burden of the ISP to provide extra bandwidth, CPU processing power, memory, and storage space just to accomodate what it clearly a theft of services? The dual 66MHz SPARC system that was running an ISP back in 1995 is still running, and in a normal environment handles incoming and outgoing email just fine. Without the introduction of a front-end server, or replacement altogether (money spent no matter how you look at it) the machine often ran at 75% load or more during times when historically it ran no more than 30%.
The attitude of "well, it's going to happen anyway, might as well deal with it" is garbage. Adopting such an attitude in the face of a hurricane, the forces of which cannot be stopped, is fully acceptable. But in the face of spam which should not exist in the first place, this attitude is comparable to rolling over and taking it right up the rectum rather than dealing with the source.