Slashdot Mirror
A Day with an ISP Spam Investigator
scumbucket writes "Network World Fusion has an interesting article about an abuse investigator for ISP Earthlink and his job of tracking down spammers. It's nice to see that major ISP's are making an effort to shut spammers down and kick them off of their networks."
Not that interesting really. No specifics, not much technique. He calls offenders, cancels accounts, etc. Phishing is another department. He doesn't take action on pedophiles and refers them to cops.
Where's the beef?
"...all the labours of the ages, all the devotion, all the inspiration, all the noonday brightness..." yada yada
Well they dont do it because they wont to help the world. But spam means extra bandwidth, so extra cost. And maybe customers blame the ISP, so that might mean less customers. So it is the ISP's best interest to do something about spam.
http://www.virtualconcepts.nl/
FTFA: One notorious spammer, whom EarthLink helped put behind bars, repeatedly used the names of sports such as baseball and football as his password.
Spammers are stupider than I realized.
This si interesting, but you have to say this guy is fighting a losing battle. You have to fight Spam at its source. Look at the Spamhaus statistics: it might sound trollish, but spam is evidently an American problem, which must be combated in America. The Spamhaus stats prove it. 90% of the spam you see is from 200 individuals, of whom 96% are Americans, operating out of america.
Clean up your act guys. When you're costing the world this much money, it just isn't funny anymore.
Meine Schwester ist sehr, sehr reizvoll - Nietzsche
SCO.com uses Linux
What's needed is every ISP having a consistently responsive abuse department. From what i've seen, everbody except the largest tier-1 ISPs do, with most of them having a substantive presence on abuse and anti-spam lists, and responsive to complaints.
.us tier-1 ISPs and most of .cn/.kr that are seriously culpable these days; from what I've seen working in the anti-spam arena these last six months, uu.net/MCI and their peers don't give a shit because, well, nobody's going to refuse to peer with them if they host spammers. Same thing in .cn/.kr, their broadband industries net the larger .us providers large $ over the longer term, and it's not in their best interests to be overly proactive.
.kr ASs in real time; we were given a demo at AusCERT 2004. The fact that they won't use this more proactively is depressing.
It's the major
Which is a shame; KISA (.kr equivalent of the FCC/ACA/etc) have got a great early-warning system set up, which shows transit traffic between
About 40% of my current spam corpus is from korea, the other 60% is about 30/30/40% china, uu.net, and comcast/verizon open proxies.
You're doing it wrong.
Yes, the typical spammer is a slashdot-reading geek who lives with his parents. ... Reminds me of a thing I read earlier warning parents about signs of their child engaging in dangerous hacking, such as use of Linux or requests for better hardware.
Just what a geek needs, another reason for parents to be suspicious of his computer usage. Help! I'm a computer addicted teenager who can't stop sending spam, and this is really a cry for help!
"He only reads the content of an e-mail in extreme cases, he says."
I've always found it safest to avoid reading email, unless I'm feeling really daring...
Online & Feelin' Fine
As this article from Satire Wire shows, spam can be a work of art or literature.
You could always send your resume out in a virus and see what offers you get. That seems to be the new thing. :)
One line blog. I hear that they're called Twitters now.
Not very much.
But even assuming they only made 50c per 1000 emails, when you're sending out 10+ million emails per day that's still $5,000+ per day or $1,825,000+ a year. Even at 1c per 1000 mails they still make $36,500+ per year.
Well they don't do it because they wont to help the world. But spam means extra bandwidth, so extra cost.
I've heard many a system admin complain about the "cost of spam" to their networks, but have not seen a quantification of that cost. Given that spams are so small (the ones that I get average 4kB/spam), the storage costs of saving every spam (at 1$/GB) are about only 4 micro$/spam and the transfer costs (at $3/GB of transfer to pick a Google figure) are only 12 micro$/spam. Even CPU time is cheap. If a $2000 server CPU can handle only 10 messages per second (an underestimate?) then the cost in CPU time is only about 6 micro$/spam. In total, a million spams would cost an ISP maybe $20 or $30 which is far less that the burdened labor cost of one hour of a technician's time.
What am I missing here? Can any admins tell me the true dollar cost per spam? The only other reason, that I can think of, is that Earthlink fights spam to avoid blacklisting because blacklisting would drive up support costs when a million customers call at ask why their emails aren't getting through.
Two wrongs don't make a right, but three lefts do.
They seem to monitor their user's passwords...
(Page 2)...One notorious spammer, whom EarthLink helped put behind bars, repeatedly used the names of sports such as baseball and football as his password...
I thought that passwords SHOULD not be easily unencrypted... or do they monitor them before encryption?
Your head a splode
Well, with these kind of statistics, he'll be gainfully employed for years to come.
While he believes his job is important, Rush doesn't take the role of Internet cop too seriously. But he admits with a chuckle that his favorite computer game at the moment is called City of Heroes.
I'd sit back all day and play CoH, and tell my boss "Sure, I killed off 800 spammers today. But 30,000 more popped up. Guess I'll be seeing you monday."
I need me one of those gigs. Anyone offering?
And how many spam messages pass trough a serious isp's network? I think you'd be surprised...
Part of the cost is also due to filtering and to the extra admin costs for implementing enough capacity to hold the extra spam..
Incidentally, this bit:
was interesting to me. This sounds like the oft-repeated assertion that a US flag with a fringe in a courtroom means that you're under Admiralty law, not the law of the United States, and that anyone who appears before that court has lost most of their rights. Of course, They don't want you to know this...or that England still owns the US, or that there is a subtle yet vitally important difference between the United States and the United States of America that means you are 0wn3d by the government...
I tell you, there are worlds upon worlds of free entertainment out there on the Internet.
Carousel is a lie!
You are neglecting the admin time and cost of keeping the server running. Monitoring it for problems, keeping the software up-to-date, making configuration changes, keeping it backed up, documenting the configuration so that disaster recovery is relatively painless and quick.
Mea navis aericumbens anguillis abundat
Most viruses go over 40 kb and can go to about 200 kb (that's what I get). Most annoying are the mailer daemon failures that i get for viruses that i did not (or anybody else from my domain) send.
I'll do the stupid thing first and then you shy people follow...
im no sysadmin or anything.
but if its 30 $ per day, its 10k per year.
further more you have to spend time and energy you have to spend sorting the mail. this is, ive heard, quite expensive in CPU time.
The best filters catch 99.9% of spam and only make 1 mistake in a thousand. ( i don't even think that they are that good).
1000 emplyoees gets 5 mails aday for a year thats 1.8 million mails, thats 1800 mails per year that goes down the drain. im not sure what that costs, but some of the are prolly quite expensive.
This is not absolute facts nor close, but my point is that the price of spam is more than the price of reciving spam.
spelling is for people who doens't know better...
All of which has to be performed whether the machine is handling spam or not, unless you're laying on extra hardware to take the extra load caused by the spam...
It's official. Most of you are morons.
Presumably you have a Gmail account,
...
and do not object to Google's policies
But many of us will not send mail to gmail.com
Problem 1: Gmail is nearly immortal
Google offers 1 gig of storage, which is many times the storage offered by Yahoo or Hotmail, or other Internet service providers that we know about. The powerful searching encourages account holders to never delete anything. It takes three clicks to put a message into the trash, and more effort to delete this message. It's much easier to "archive" the message, or just leave it in the inbox and let the powerful searching keep track of it. Google admits that even deleted messages will remain on their system, and may also be accessible internally at Google, for an indefinite period of time.
Google has been spinning their original position in press interviews, and with an informal page described as "a few words about privacy and Gmail." When we see fresh material from Google, we check the modification date at the bottom of the terms-of-use page and privacy page for Gmail. If these dates are still April 6 and April 8, we know that nothing has changed. Google can modify these pages too, any way they want and whenever they want, unilaterally. But at least these two pages carry slightly more legal weight than other pages, because Google should attempt to notify users of significant changes in these formal policies.
A new California law, the Online Privacy Protection Act, went into effect on July 1, 2004. Google changed their main privacy policy that same day because the previous version sidestepped important issues and might have been illegal. For the first time in Google's history, the language in their new policy makes it clear that they will be pooling all the information they collect on you from all of their various services. Moreover, they may keep this information indefinitely, and give this information to whomever they wish. All that's required is for Google to "have a good faith belief that access, preservation or disclosure of such information is reasonably necessary to protect the rights, property or safety of Google, its users or the public." Google, you may recall, already believes that as a corporation they are utterly incapable of bad faith. Their corporate motto is "Don't be evil," and they even made sure that the Securities and Exchange Commission got this message in Google's IPO filing.
Google's policies are essentially no different than the policies of Microsoft, Yahoo, Alexa and Amazon. However, these others have been spelling out their nasty policies in detail for years now. By way of contrast, we've had email from indignant Google fans who defended Google by using the old privacy language -- but while doing so they arrived at exactly the wrong interpretation of Google's actual position! Now those emails will stop, because Google's position is clear at last. It's amazing how a vague privacy policy, a minimalist browser interface, and an unconventional corporate culture have convinced so many that Google is different on issues that matter.
After 180 days in the U.S., email messages lose their status as a protected communication under the Electronic Communications Privacy Act, and become just another database record. This means that a subpoena instead of a warrant is all that's needed to force Google to produce a copy. Other countries may even lack this basic protection, and Google's databases are distributed all over the world. Since the Patriot Act was passed, it's unclear whether this ECPA protection is worth much anymore in the U.S., or whether it even applies to email that originates from non-citizens in other countries.
Google's relationships with government officials in all of the dozens of countries where they operate are a mystery, because Google never makes any statements about this. But here's a clue: Google uses the term "governmental request" three times on their terms-of-use page and once on their privacy page. Google's language means that al
The 'cost of spam' is not the cost of spam filters, extra storage, etc. The cost of spam is the cost to the end user of having to figure out which mail is real and which is spam.
Let's assume it takes a user only 1 second to determine if a piece of mail is spam, and deal with it, and let's assume the average user's time is worth $20 per hour. A million spams then cost the users:
$5555 = 1 second * 1 million / 3600 seconds in an hour * $20
You're right, the ISPs scared of being blacklisted. But they also view (correctly) keeping spam volume down as part of the service they sell. I know I have given up on some ISPs because of spam volume.
I used to work at a small ISP -- lets say 5000 customers. We were getting lots of complaints about spam, so we decided to put in better spam filtering. That required a bigger server. Then the mail server went down for half an hour because of the volume of incoming spam, and there was a suddenly a big rush on getting the new server up and running.
The server was the cheap part: let's say $2000 (all figures Canadian) for the box, rackmount, hard drives, yadda blah. Thank God for Free software, because FreeBSD and SpamAssassin saved our asses. It took me, conservatively, three full days to set up and get it more or less right; I was doing a lot of learning on the job, and the regular sysadmin was away.
Now, don't forget that we were down for half an hour. This was from roughly 9am to 9:30am on that day, so that's a busy fucking time for us. There were tons of calls and only three people to handle them; fortunately, I was pressed into service trying to fix things, and wasn't on the phones. We probably lost a couple customers then, but most people were pretty understanding, especially when they were told it was fuckwad spammers who were causing the problem.
Complaints were a huge deal, both before and after the filtering was put in place; I was dealing with most of them, because I was doing abuse duties, and it wasn't fun. Complaints before the new server was installed went, "Why am I getting all this spam? Why can't you stop it?" Complaints afterward went, "Why am I still getting all this spam? Why isn't your filtering working? What do you mean, I have to set up my mail program to do more work?" (We set the threshold rather high, thinking that customers could use filtering in their mail client to set their own tolerance level. Ha! It is to laugh. Ever tried filtering on random headers in Outlook Express 5.0?)
Plus, there was maintenance of the server and software; upgrades were never fun; false positives happened and were dealt with; and now, my sources tell me, they've graduated to buying dual-fucking-xeon processors in order to handle spam filtering. Fuck me!
But hey, we were after a dollar cost, and I did get sidetracked. We already said $2k for the server. Three days of my time, $400 (deal!). Half an hour when everything in the company came to a halt because no one could send mail or do anything but answer the phones: $500, and that's probably very conservative. Two customers' worth of lost revenue for a year: say another $500. Spam complaints before took, oh, probably a good five solid days of my time: $650. Afterward was probably the same, so another $650. I know of at least one customer we lost afterward when the spam filtering wasn't the magic bullet I kept trying to tell them didn't exist, so $250. Bandwidth for all the spam we were accepting but kept from reaching the customers: let's say $50, for a nice round total of $5000.
Now this is very, very rough back-of-the-envelope calculations for a small dialup ISP I no longer work at; the managers there could probably tell you more about lost good will and so on. More importantly, it doesn't tell you about ongoing costs; that's just a snapshot from when I worked there. But that was $5000 spent by an ISP that was going down the tubes (true story), just to keep up (barely) with a denial-of-service attack that was slowly grinding us into the floor. I can't even imagine what it's like for AOL or Hotmail. Nor will we ever know what that time and effort and money might have done if it wasn't being spent on spam.
Goddamn fuckwad spammers piss me off.
Carousel is a lie!
Upwards of 80% of our network traffic is mail. Of that, 70 - 80% of that is inbound spam, trojans and viruses. If we could eliminate them entirely from outside our network, we wouldn't require so much bandwidth and bandwidth is a major portion of our fixed operating costs. Office space is cheap compared to bandwidth.
Its not just the total number of received messages that affect cost. Delivery rate causes problems with network availability. Because of distributed attacks and mail bombs, we have to be able to scale well above our average consumption or risk losing connectivity. I don't mind losing a single service nearly as much as I mind losing a network.
You want a dollar figure? It depends on the incident. No two spams are exactly the same. Your figure of $1 per GB is misleading because it assumes that the traffic is distributed over a entire billing cycle. What happens if that 1GB is delivered over a period of 1 minute? Ever seen a clogged pipe?
We spend most of our time building the next generation of services to combat misuse of our resources so that our clients can get that occasional letter from Grandma.
Pull my finger for my public key.
"What good does it do if it is still completely and tragically uneffective?"
...... like limiting outbound email traffic on all new accounts. New accounts that hit your ceiling will be flagged for you to investigate, yet you will still be limiting the spam they can send and being a nice ISP.
Gotta agree with you there. Particularly at an ISP.
If you KNOW your actions are ineffective, wouldn't you re-evaluate your approach and look for more effective actions?
Say
From the article: "Yet canceling a spammer's account doesn't always solve the the problem. Serial spammers who have been kicked off the EarthLink network once will often jump back on, creating as many as four or five fraudulent accounts per day using stolen credit cards."
So if you limit new accounts to 1 email every 10 seconds (that's some fast typing), and put a ceiling of 200 emails a day, you'd quickly be able to spot the spammers. Yet those "four or five fraudulent accounts per day" would only be sending 1,000 spam messages a day.
The ISP's are not really serious about fighting spam. It does not cost them that much and they are probalby making money due to spam. So the only incentive they have to do anything about it is when the level of spaming gets to the point they are about to be blacklisted then they take action.
If they were really serious about curbing spam they would implement greylisting and greet_pause features in their MTAs. Both of these would block 99% of the spam being sent. The remaining spammers would then be much easier to track down since they would have to be running full blown MTAs which could then be blocked.
So why don't they do this? Because it does not make them any money and would cost them a little money to implement and maintain such features.
Ultimately the only way to eliminate spam is to make is unprofitable to the spammer. One option that I have never seen discussed is to track down the idiots that actually buy from spam and take their machines away and sterilize them so they don't reproduce.
They don't make any money, give me a break. Basing a business on just advertising is pretty difficult. I've seem some articles about spammers who claim to make a bunch of money and meanwhile they live in a trailerpark somewhere. Spammers just make money off the idiots who hire them to send out spam. They are just con artists.
-------------------------------------
Technically, we are beyond survival.
Fun article for me. 25 years ago or so, I was the original "cable cop" in Michigan, USA (the job title was "system auditor"). This was before it was illegal to "steal" cable services, and the overall thrust of my work was to build a case for legislators.
About 50% of my time was indoors, pulling street-by-street printouts off our Tandem system and cleaning up/verifying account info by going back to original install paperwork. The rest of my time was spent climbing poles, verifying hookups and disconnecting the "non-subscribers." After a year of that, we had enough info to deliver numbers to the statehouse: 4% of all cable viewers weren't paying us for the service. That was enough for the legislators, and cable theft became a mid-range misdemeanor.
So then I started going after the midnight installers offering people "free HBO forever" at the low low price of $100 (or whatever). That was kinda fun...serveral times I was just hours behind these guys, removing service drops while the resident stood by watching, moaning eulogies for their recently departed 100 bucks.
I'm surprised that more ISPs don't have employees like the guy in TFA (or perhaps I'm surprised that we don't hear more about them)...losses due to spam are real, no? [In the case of cable, the "losses" were 99% paper; there was no extra drain on bandwidth, no guarentee these folks would have been paying us otherwise, and no real loss on the converters they were using (our collections folks did just fine charging 4X the cost for unreturned equipment). The only true "loss" was in tech-time, for the rare hookup that caused interference on a distribution line or radiated enough signal to breach FCC rules.]
Is the reason for this apparent lack of interest on the part of ISPs similar to that of the credit card companies during the early online days? Rather than appear inept at providing decent system integrity (easily spoofed card numbers, pitiful account verification, etc.), fraud and abuse were handled quietly, with costs taken off the bottom line. Or is the apparent less-than-vigorous investigation of spammers just part of the "?" step in the profit! formula...where bandwidth lost = cost of investigatory personnel, so screw the inconvenience to customers?
education is no substitute for intelligence
Rush mentions that in one case he realized that the suspect was using a sports password scheme, does that mean that these people working at the ISPs can view our passwords? I happen to use maybe a set of 6 different passwords, but if someone can get one of them, they can access many things that are password protected for me. Its unreasonable to have a different password for every net logon you have, but I always thought that passwords were hashed so that even the system admin in most cases can't read them.
Now Zapp, you may ask: "What has that to do with anything?"
If you really don't know what staunch dfenders of free speech the Scientolgy[tm] "Church" is you might find some interesting reading at this link.
If you want to dig deeper then Xenu can guide you.
ich bin der musikant
mit taschenrechner in der hand
kraftwerk
Several years back the local ISP for which I worked had a spammer force us to take our mail server down because his advertising bomb went off in our spool drive and completely filled it. It took a number of hours to manually clean it up, sift through logs to find and block the offender, and bring the server back on-line. Ask our business clients how much not having email available for several hours cost them. Just for illustration, that email was also only about 3k in size, but once it multiplied in the queue it consumed all 2GB of the spool.
More recently, the local ISP for which I often do admin work had to build three new incoming mail servers and purchase spam and virus filter software for each machine at the rate of at least $6000 ea. plus subscription. Without these machines, user mail spools were filling up with spam and viruses; the older the account the worse off it was. Ask these folks how much it costs.
I have seen spam perform the equivalent of DoS floods: causing servers to crash, filling up T1s, causing CPU loads on older but otherwise working machines to hit 98%, and more. I host a domain which sees 28,000 spams per week on average. We employ RBLs in our fight against spam, as well as blocking a number of countries known for delivering nothing legitimate to our servers.
We see the shit come from all directions. In one night I observed a spam run against a hosted domain attempt to deliver 5,821 messages -- all forensically identical -- in less than 100 seconds from roughly 15 sources.
Why should it be the burden of the ISP to provide extra bandwidth, CPU processing power, memory, and storage space just to accomodate what it clearly a theft of services? The dual 66MHz SPARC system that was running an ISP back in 1995 is still running, and in a normal environment handles incoming and outgoing email just fine. Without the introduction of a front-end server, or replacement altogether (money spent no matter how you look at it) the machine often ran at 75% load or more during times when historically it ran no more than 30%.
The attitude of "well, it's going to happen anyway, might as well deal with it" is garbage. Adopting such an attitude in the face of a hurricane, the forces of which cannot be stopped, is fully acceptable. But in the face of spam which should not exist in the first place, this attitude is comparable to rolling over and taking it right up the rectum rather than dealing with the source.
Now, I suppose it is possible to get a T1 from Earthlink or some other ISP. Then, they may provide some services aside from just the data connection. And then there would be some TOS, some kind of service agreement and so on. But if you buy your service from the phone company I have never seen such a service agreement.
I expect this holds true for any sort of data connection from a telecommunications provider that is not providing any additional services, which means if you call SBC to get an OC48 they aren't going to ask you what you plan to do with it.
"I don't understand why all the focus on ISPs."
Because, unless you have a peering agreement, you are connecting to an ISP.
"You call the phone company (any phone company) and say you want a data T1 connection."
Okay. That's a chunk of money and it has a physical connection point that is recorded. It is completely different than a dial-up account.
"They give it to you and give you some IP addresses."
From their block. That means that they are your upstream provider. If someone complains about your behaviour, they will complain to your upstream provider who will then cut you off (or not).
"They do not process email for you, they do not give you web space and they do not respond to complaints about what you are doing with your T1."
They do respond to complaints about what you are doing.
"I expect this holds true for any sort of data connection from a telecommunications provider that is not providing any additional services, which means if you call SBC to get an OC48 they aren't going to ask you what you plan to do with it."
That is correct. They will not. But you ARE plugged into THEIR network.
One end of the line terminates at your location, the other end terminates at the phone company's location.
So, traffic coming from your line goes through the phone company's network. And people can see who licensed that IP range to you. They will complain to your upstream provider.
"Why is it the ISP's burden to accommodate this theft of services? Because it's only theft if it's stolen from _somebody_, and as an ISP in a competitive market, you'd rather spend the money to provide better quality services than lose customers to other ISPs, so that means it's stolen from _you_. "
Horseshit. That is along the same lines as the police department telling a hotel manager that he should bullet-proof the glass and walls in his establishment to help with the onslaught of drive-by shootings.
A stack of $250 2GHz Celerons is still money spent on a problem which should be stopped at the source. Fighting spam and viruses should not become a competitve-edge industry any more than fighting crime should be. The $250 spent on each machine is not advertising which will turn new customers; it is not increasing the features and usefulness of your product which will turn new customers. Instead we are having to purchase bigger and stronger wedges to keep people out who should not be entering in the first place. We have bought the locks, we lock them, and the intruders still try to get in... when does the burden shift?
In any argument, spam steals resources from the ISP which would normally be allocated for customer use. Even if spam only consumes a little more processing power or bandwidth than normal traffic, it is still an unwanted abuse of our purchased resource. If you look at the situation from the point of those who sell you the bandwidth in the first place, the money you have spent is really not for the bandwidth, or processing power, or storage, or whatever, the money has been spent for the ability to use the full resources. And when that ability has been lessened by incoming garbage, your ability has been reduced, the value of the purchased product has been reduced, and therefor the money you have spent goes down the toilet.
Next month, tally up all of the time you spend deleting spam and viruses, the amount of bandwidth spam and viruses uses in your pipe, and the cost of your anti-virus/anti-spam software, then call up your provider and tell them that you should not have to pay for xx% of your service because it was not useful data to you.
Even better is to try that on per-use providers, or telephone systems. While we are at it, the same should be done with pop-up ads, adware, in-page advertisements, etc. etc. etc...
No wait, call up your ISP and tell them that they should increase your mail box storage space because you get so much spam or viruses.
Nothing doing. We place too much burden on the end user to buy anti-spam software or services, and too much burden on the ISP to accomodate the massive amounts of garbade data coming into their systems. No. The burden should be on those who are unburdened by this scourge. If adequate punishments can be inflicted upon those who ignore the standard of neighborly etiquette, the problem will begin to disappear.