A Day with an ISP Spam Investigator
scumbucket writes "Network World Fusion has an interesting article about an abuse investigator for ISP Earthlink and his job of tracking down spammers. It's nice to see that major ISP's are making an effort to shut spammers down and kick them off of their networks."
Not that interesting really. No specifics, not much technique. He calls offenders, cancels accounts, etc. Phishing is another department. He doesn't take action on pedophiles and refers them to cops.
Where's the beef?
"...all the labours of the ages, all the devotion, all the inspiration, all the noonday brightness..." yada yada
Well they dont do it because they wont to help the world. But spam means extra bandwidth, so extra cost. And maybe customers blame the ISP, so that might mean less customers. So it is the ISP's best interest to do something about spam.
http://www.virtualconcepts.nl/
FTFA: One notorious spammer, whom EarthLink helped put behind bars, repeatedly used the names of sports such as baseball and football as his password.
Spammers are stupider than I realized.
This si interesting, but you have to say this guy is fighting a losing battle. You have to fight Spam at its source. Look at the Spamhaus statistics: it might sound trollish, but spam is evidently an American problem, which must be combated in America. The Spamhaus stats prove it. 90% of the spam you see is from 200 individuals, of whom 96% are Americans, operating out of america.
Clean up your act guys. When you're costing the world this much money, it just isn't funny anymore.
Meine Schwester ist sehr, sehr reizvoll - Nietzsche
"While sending spam is not against the law in most cases, it does violate EarthLink's use policy; not only can Rush terminate the account of a spammer, but he can also charge a $200 cleanup fee."
"Serial spammers who have been kicked off the EarthLink network once will often jump back on, creating as many as four or five fraudulent accounts per day using stolen credit cards
So - Earhtlink are fining victims of stolen credit cards, in other words!. Possibly a more long-lasting approach would be to geo-locate the connection and targt it with one of these
I have been a user for about 10 years. This ends Feb 2014. The site's been ruined. I'm off. Dice, FU
This is a great man.
I think I need a new sig here.
SCO.com uses Linux
What's needed is every ISP having a consistently responsive abuse department. From what i've seen, everbody except the largest tier-1 ISPs do, with most of them having a substantive presence on abuse and anti-spam lists, and responsive to complaints.
.us tier-1 ISPs and most of .cn/.kr that are seriously culpable these days; from what I've seen working in the anti-spam arena these last six months, uu.net/MCI and their peers don't give a shit because, well, nobody's going to refuse to peer with them if they host spammers. Same thing in .cn/.kr, their broadband industries net the larger .us providers large $ over the longer term, and it's not in their best interests to be overly proactive.
.kr ASs in real time; we were given a demo at AusCERT 2004. The fact that they won't use this more proactively is depressing.
It's the major
Which is a shame; KISA (.kr equivalent of the FCC/ACA/etc) have got a great early-warning system set up, which shows transit traffic between
About 40% of my current spam corpus is from korea, the other 60% is about 30/30/40% china, uu.net, and comcast/verizon open proxies.
You're doing it wrong.
Yes, the typical spammer is a slashdot-reading geek who lives with his parents. ... Reminds me of a thing I read earlier warning parents about signs of their child engaging in dangerous hacking, such as use of Linux or requests for better hardware.
Just what a geek needs, another reason for parents to be suspicious of his computer usage. Help! I'm a computer addicted teenager who can't stop sending spam, and this is really a cry for help!
I wonder.. how much does an average spammer earn doing his "job"? Can one make a living by sending spam and not doing any other work? Anyone knows, for example, how much does a spammer get for sending one, say, 1000 emails?
"He only reads the content of an e-mail in extreme cases, he says."
I've always found it safest to avoid reading email, unless I'm feeling really daring...
Online & Feelin' Fine
As this article from Satire Wire shows, spam can be a work of art or literature.
Well they don't do it because they wont to help the world. But spam means extra bandwidth, so extra cost.
I've heard many a system admin complain about the "cost of spam" to their networks, but have not seen a quantification of that cost. Given that spams are so small (the ones that I get average 4kB/spam), the storage costs of saving every spam (at 1$/GB) are about only 4 micro$/spam and the transfer costs (at $3/GB of transfer to pick a Google figure) are only 12 micro$/spam. Even CPU time is cheap. If a $2000 server CPU can handle only 10 messages per second (an underestimate?) then the cost in CPU time is only about 6 micro$/spam. In total, a million spams would cost an ISP maybe $20 or $30 which is far less that the burdened labor cost of one hour of a technician's time.
What am I missing here? Can any admins tell me the true dollar cost per spam? The only other reason, that I can think of, is that Earthlink fights spam to avoid blacklisting because blacklisting would drive up support costs when a million customers call at ask why their emails aren't getting through.
Two wrongs don't make a right, but three lefts do.
They seem to monitor their user's passwords...
(Page 2)...One notorious spammer, whom EarthLink helped put behind bars, repeatedly used the names of sports such as baseball and football as his password...
I thought that passwords SHOULD not be easily unencrypted... or do they monitor them before encryption?
Your head a splode
Well, with these kind of statistics, he'll be gainfully employed for years to come.
While he believes his job is important, Rush doesn't take the role of Internet cop too seriously. But he admits with a chuckle that his favorite computer game at the moment is called City of Heroes.
I'd sit back all day and play CoH, and tell my boss "Sure, I killed off 800 spammers today. But 30,000 more popped up. Guess I'll be seeing you monday."
I need me one of those gigs. Anyone offering?
And how many spam messages pass trough a serious isp's network? I think you'd be surprised...
Part of the cost is also due to filtering and to the extra admin costs for implementing enough capacity to hold the extra spam..
Incidentally, this bit:
was interesting to me. This sounds like the oft-repeated assertion that a US flag with a fringe in a courtroom means that you're under Admiralty law, not the law of the United States, and that anyone who appears before that court has lost most of their rights. Of course, They don't want you to know this...or that England still owns the US, or that there is a subtle yet vitally important difference between the United States and the United States of America that means you are 0wn3d by the government...
I tell you, there are worlds upon worlds of free entertainment out there on the Internet.
Carousel is a lie!
You are neglecting the admin time and cost of keeping the server running. Monitoring it for problems, keeping the software up-to-date, making configuration changes, keeping it backed up, documenting the configuration so that disaster recovery is relatively painless and quick.
Mea navis aericumbens anguillis abundat
Most viruses go over 40 kb and can go to about 200 kb (that's what I get). Most annoying are the mailer daemon failures that i get for viruses that i did not (or anybody else from my domain) send.
I'll do the stupid thing first and then you shy people follow...
im no sysadmin or anything.
but if its 30 $ per day, its 10k per year.
further more you have to spend time and energy you have to spend sorting the mail. this is, ive heard, quite expensive in CPU time.
The best filters catch 99.9% of spam and only make 1 mistake in a thousand. ( i don't even think that they are that good).
1000 emplyoees gets 5 mails aday for a year thats 1.8 million mails, thats 1800 mails per year that goes down the drain. im not sure what that costs, but some of the are prolly quite expensive.
This is not absolute facts nor close, but my point is that the price of spam is more than the price of reciving spam.
spelling is for people who doens't know better...
All of which has to be performed whether the machine is handling spam or not, unless you're laying on extra hardware to take the extra load caused by the spam...
It's official. Most of you are morons.
Presumably you have a Gmail account,
...
and do not object to Google's policies
But many of us will not send mail to gmail.com
Problem 1: Gmail is nearly immortal
Google offers 1 gig of storage, which is many times the storage offered by Yahoo or Hotmail, or other Internet service providers that we know about. The powerful searching encourages account holders to never delete anything. It takes three clicks to put a message into the trash, and more effort to delete this message. It's much easier to "archive" the message, or just leave it in the inbox and let the powerful searching keep track of it. Google admits that even deleted messages will remain on their system, and may also be accessible internally at Google, for an indefinite period of time.
Google has been spinning their original position in press interviews, and with an informal page described as "a few words about privacy and Gmail." When we see fresh material from Google, we check the modification date at the bottom of the terms-of-use page and privacy page for Gmail. If these dates are still April 6 and April 8, we know that nothing has changed. Google can modify these pages too, any way they want and whenever they want, unilaterally. But at least these two pages carry slightly more legal weight than other pages, because Google should attempt to notify users of significant changes in these formal policies.
A new California law, the Online Privacy Protection Act, went into effect on July 1, 2004. Google changed their main privacy policy that same day because the previous version sidestepped important issues and might have been illegal. For the first time in Google's history, the language in their new policy makes it clear that they will be pooling all the information they collect on you from all of their various services. Moreover, they may keep this information indefinitely, and give this information to whomever they wish. All that's required is for Google to "have a good faith belief that access, preservation or disclosure of such information is reasonably necessary to protect the rights, property or safety of Google, its users or the public." Google, you may recall, already believes that as a corporation they are utterly incapable of bad faith. Their corporate motto is "Don't be evil," and they even made sure that the Securities and Exchange Commission got this message in Google's IPO filing.
Google's policies are essentially no different than the policies of Microsoft, Yahoo, Alexa and Amazon. However, these others have been spelling out their nasty policies in detail for years now. By way of contrast, we've had email from indignant Google fans who defended Google by using the old privacy language -- but while doing so they arrived at exactly the wrong interpretation of Google's actual position! Now those emails will stop, because Google's position is clear at last. It's amazing how a vague privacy policy, a minimalist browser interface, and an unconventional corporate culture have convinced so many that Google is different on issues that matter.
After 180 days in the U.S., email messages lose their status as a protected communication under the Electronic Communications Privacy Act, and become just another database record. This means that a subpoena instead of a warrant is all that's needed to force Google to produce a copy. Other countries may even lack this basic protection, and Google's databases are distributed all over the world. Since the Patriot Act was passed, it's unclear whether this ECPA protection is worth much anymore in the U.S., or whether it even applies to email that originates from non-citizens in other countries.
Google's relationships with government officials in all of the dozens of countries where they operate are a mystery, because Google never makes any statements about this. But here's a clue: Google uses the term "governmental request" three times on their terms-of-use page and once on their privacy page. Google's language means that al
The 'cost of spam' is not the cost of spam filters, extra storage, etc. The cost of spam is the cost to the end user of having to figure out which mail is real and which is spam.
Let's assume it takes a user only 1 second to determine if a piece of mail is spam, and deal with it, and let's assume the average user's time is worth $20 per hour. A million spams then cost the users:
$5555 = 1 second * 1 million / 3600 seconds in an hour * $20
You're right, the ISPs scared of being blacklisted. But they also view (correctly) keeping spam volume down as part of the service they sell. I know I have given up on some ISPs because of spam volume.
I used to work at a small ISP -- lets say 5000 customers. We were getting lots of complaints about spam, so we decided to put in better spam filtering. That required a bigger server. Then the mail server went down for half an hour because of the volume of incoming spam, and there was a suddenly a big rush on getting the new server up and running.
The server was the cheap part: let's say $2000 (all figures Canadian) for the box, rackmount, hard drives, yadda blah. Thank God for Free software, because FreeBSD and SpamAssassin saved our asses. It took me, conservatively, three full days to set up and get it more or less right; I was doing a lot of learning on the job, and the regular sysadmin was away.
Now, don't forget that we were down for half an hour. This was from roughly 9am to 9:30am on that day, so that's a busy fucking time for us. There were tons of calls and only three people to handle them; fortunately, I was pressed into service trying to fix things, and wasn't on the phones. We probably lost a couple customers then, but most people were pretty understanding, especially when they were told it was fuckwad spammers who were causing the problem.
Complaints were a huge deal, both before and after the filtering was put in place; I was dealing with most of them, because I was doing abuse duties, and it wasn't fun. Complaints before the new server was installed went, "Why am I getting all this spam? Why can't you stop it?" Complaints afterward went, "Why am I still getting all this spam? Why isn't your filtering working? What do you mean, I have to set up my mail program to do more work?" (We set the threshold rather high, thinking that customers could use filtering in their mail client to set their own tolerance level. Ha! It is to laugh. Ever tried filtering on random headers in Outlook Express 5.0?)
Plus, there was maintenance of the server and software; upgrades were never fun; false positives happened and were dealt with; and now, my sources tell me, they've graduated to buying dual-fucking-xeon processors in order to handle spam filtering. Fuck me!
But hey, we were after a dollar cost, and I did get sidetracked. We already said $2k for the server. Three days of my time, $400 (deal!). Half an hour when everything in the company came to a halt because no one could send mail or do anything but answer the phones: $500, and that's probably very conservative. Two customers' worth of lost revenue for a year: say another $500. Spam complaints before took, oh, probably a good five solid days of my time: $650. Afterward was probably the same, so another $650. I know of at least one customer we lost afterward when the spam filtering wasn't the magic bullet I kept trying to tell them didn't exist, so $250. Bandwidth for all the spam we were accepting but kept from reaching the customers: let's say $50, for a nice round total of $5000.
Now this is very, very rough back-of-the-envelope calculations for a small dialup ISP I no longer work at; the managers there could probably tell you more about lost good will and so on. More importantly, it doesn't tell you about ongoing costs; that's just a snapshot from when I worked there. But that was $5000 spent by an ISP that was going down the tubes (true story), just to keep up (barely) with a denial-of-service attack that was slowly grinding us into the floor. I can't even imagine what it's like for AOL or Hotmail. Nor will we ever know what that time and effort and money might have done if it wasn't being spent on spam.
Goddamn fuckwad spammers piss me off.
Carousel is a lie!
We've heard it all before. Spam fills out inboxes day after day. Its is manageable if you have a 2 meg inbox, but take pity on those with boxes around the 1 gig mark. Of course I'm talking about GMail. Ps. I will trade karma for this link.
Noooo, Don't click Me!I couldn't think of a sig.
Upwards of 80% of our network traffic is mail. Of that, 70 - 80% of that is inbound spam, trojans and viruses. If we could eliminate them entirely from outside our network, we wouldn't require so much bandwidth and bandwidth is a major portion of our fixed operating costs. Office space is cheap compared to bandwidth.
Its not just the total number of received messages that affect cost. Delivery rate causes problems with network availability. Because of distributed attacks and mail bombs, we have to be able to scale well above our average consumption or risk losing connectivity. I don't mind losing a single service nearly as much as I mind losing a network.
You want a dollar figure? It depends on the incident. No two spams are exactly the same. Your figure of $1 per GB is misleading because it assumes that the traffic is distributed over a entire billing cycle. What happens if that 1GB is delivered over a period of 1 minute? Ever seen a clogged pipe?
We spend most of our time building the next generation of services to combat misuse of our resources so that our clients can get that occasional letter from Grandma.
Pull my finger for my public key.
Okay, shouldn't Earthlink have the phone number in their records?
So, Earthlink finds a spammer using a stolen credit card. Wouldn't they send the phone number and the credit card info to the FBI? Wouldn't the FBI trace that phone number to a physical location and arrest the spammer for fraud?
"What good does it do if it is still completely and tragically uneffective?"
...... like limiting outbound email traffic on all new accounts. New accounts that hit your ceiling will be flagged for you to investigate, yet you will still be limiting the spam they can send and being a nice ISP.
Gotta agree with you there. Particularly at an ISP.
If you KNOW your actions are ineffective, wouldn't you re-evaluate your approach and look for more effective actions?
Say
From the article: "Yet canceling a spammer's account doesn't always solve the the problem. Serial spammers who have been kicked off the EarthLink network once will often jump back on, creating as many as four or five fraudulent accounts per day using stolen credit cards."
So if you limit new accounts to 1 email every 10 seconds (that's some fast typing), and put a ceiling of 200 emails a day, you'd quickly be able to spot the spammers. Yet those "four or five fraudulent accounts per day" would only be sending 1,000 spam messages a day.
The ISP's are not really serious about fighting spam. It does not cost them that much and they are probalby making money due to spam. So the only incentive they have to do anything about it is when the level of spaming gets to the point they are about to be blacklisted then they take action.
If they were really serious about curbing spam they would implement greylisting and greet_pause features in their MTAs. Both of these would block 99% of the spam being sent. The remaining spammers would then be much easier to track down since they would have to be running full blown MTAs which could then be blocked.
So why don't they do this? Because it does not make them any money and would cost them a little money to implement and maintain such features.
Ultimately the only way to eliminate spam is to make is unprofitable to the spammer. One option that I have never seen discussed is to track down the idiots that actually buy from spam and take their machines away and sterilize them so they don't reproduce.
Running GroupWise at work. I had to dedicate a machine to running Guinevere and SpamAssassin and McAfee anti-virus.
And I have to make sure it is patched.
And I had to adjust the email server's threads (default set to either 2 or 4) for handling incoming email (increased to 50).
And tuning of SpamAssassin.
Typical, one person asks for cost, meaning marginal cost (how much more does one email cost) and the answer given is average cost (total expenses divided by the total number). If a networks trafic is 1 million an hour, and they spend $1K to process it the each message has an average cost of .001, assume that the trafic is 80% spam and that is then eliminated then the costs become .005 per message because the overhead expenses don't change much because they still need the capacity to meet peak demands.
Apocalypse Cancelled, Sorry, No Ticket Refunds
They don't make any money, give me a break. Basing a business on just advertising is pretty difficult. I've seem some articles about spammers who claim to make a bunch of money and meanwhile they live in a trailerpark somewhere. Spammers just make money off the idiots who hire them to send out spam. They are just con artists.
-------------------------------------
Technically, we are beyond survival.
Fun article for me. 25 years ago or so, I was the original "cable cop" in Michigan, USA (the job title was "system auditor"). This was before it was illegal to "steal" cable services, and the overall thrust of my work was to build a case for legislators.
About 50% of my time was indoors, pulling street-by-street printouts off our Tandem system and cleaning up/verifying account info by going back to original install paperwork. The rest of my time was spent climbing poles, verifying hookups and disconnecting the "non-subscribers." After a year of that, we had enough info to deliver numbers to the statehouse: 4% of all cable viewers weren't paying us for the service. That was enough for the legislators, and cable theft became a mid-range misdemeanor.
So then I started going after the midnight installers offering people "free HBO forever" at the low low price of $100 (or whatever). That was kinda fun...serveral times I was just hours behind these guys, removing service drops while the resident stood by watching, moaning eulogies for their recently departed 100 bucks.
I'm surprised that more ISPs don't have employees like the guy in TFA (or perhaps I'm surprised that we don't hear more about them)...losses due to spam are real, no? [In the case of cable, the "losses" were 99% paper; there was no extra drain on bandwidth, no guarentee these folks would have been paying us otherwise, and no real loss on the converters they were using (our collections folks did just fine charging 4X the cost for unreturned equipment). The only true "loss" was in tech-time, for the rare hookup that caused interference on a distribution line or radiated enough signal to breach FCC rules.]
Is the reason for this apparent lack of interest on the part of ISPs similar to that of the credit card companies during the early online days? Rather than appear inept at providing decent system integrity (easily spoofed card numbers, pitiful account verification, etc.), fraud and abuse were handled quietly, with costs taken off the bottom line. Or is the apparent less-than-vigorous investigation of spammers just part of the "?" step in the profit! formula...where bandwidth lost = cost of investigatory personnel, so screw the inconvenience to customers?
education is no substitute for intelligence
Working as a sysadmin for a national ISP, I can tell you the cost of spam is not in the storage. Doubling the amount of MTAs that you have to handle spam is a big cost. Purchasing Ironports to cut down on spam is a bog cost. Buying all the associated software and licenses (cluster software, SAN storage licenses, anti-spam licenses, etc...) is a big cost. Adding backup solutions for these new servers is an additional cost. And paying the sysadmins to administer those systems is a big cost. When the ISP receives hundreds of millions of emails daily, and 70% of it is spam, the associated needed infrastructure is nothing but a huge cost. Your numbers are also way off. 20-30$ for a million spam? Hell, it costs the helpdesk about 10$ per call, for when customers are complaining that their mail is slow, or they can't send mail cuz the MTAs are peaked. And that's just from the helpdesk perspective. Not to mention running an abuse team, which is nothing but a cost.
It's better to burn out than to fade away
Rush mentions that in one case he realized that the suspect was using a sports password scheme, does that mean that these people working at the ISPs can view our passwords? I happen to use maybe a set of 6 different passwords, but if someone can get one of them, they can access many things that are password protected for me. Its unreasonable to have a different password for every net logon you have, but I always thought that passwords were hashed so that even the system admin in most cases can't read them.
The First Amendment guarantees you protection of your right to free speech from encroachment by the U.S. government. It does not guarantee you the right to a venue or an audience. If a major ISP (or even a local ISP, for that matter) wants to prevent you from sending material they don't like using their equipment, that's not censorship - that's their right.
Now Zapp, you may ask: "What has that to do with anything?"
If you really don't know what staunch dfenders of free speech the Scientolgy[tm] "Church" is you might find some interesting reading at this link.
If you want to dig deeper then Xenu can guide you.
ich bin der musikant
mit taschenrechner in der hand
kraftwerk
I run my own mail server, so I see traffic before and after filtering. I receive, on average, one spam every eight minutes personally. Given ~130,000 users, your "million spams == $20-$30 cost" would accumulate on an hourly basis.
OH. MY. *FuCkInG* GOD. Is this true?
Several years back the local ISP for which I worked had a spammer force us to take our mail server down because his advertising bomb went off in our spool drive and completely filled it. It took a number of hours to manually clean it up, sift through logs to find and block the offender, and bring the server back on-line. Ask our business clients how much not having email available for several hours cost them. Just for illustration, that email was also only about 3k in size, but once it multiplied in the queue it consumed all 2GB of the spool.
More recently, the local ISP for which I often do admin work had to build three new incoming mail servers and purchase spam and virus filter software for each machine at the rate of at least $6000 ea. plus subscription. Without these machines, user mail spools were filling up with spam and viruses; the older the account the worse off it was. Ask these folks how much it costs.
I have seen spam perform the equivalent of DoS floods: causing servers to crash, filling up T1s, causing CPU loads on older but otherwise working machines to hit 98%, and more. I host a domain which sees 28,000 spams per week on average. We employ RBLs in our fight against spam, as well as blocking a number of countries known for delivering nothing legitimate to our servers.
We see the shit come from all directions. In one night I observed a spam run against a hosted domain attempt to deliver 5,821 messages -- all forensically identical -- in less than 100 seconds from roughly 15 sources.
Why should it be the burden of the ISP to provide extra bandwidth, CPU processing power, memory, and storage space just to accomodate what it clearly a theft of services? The dual 66MHz SPARC system that was running an ISP back in 1995 is still running, and in a normal environment handles incoming and outgoing email just fine. Without the introduction of a front-end server, or replacement altogether (money spent no matter how you look at it) the machine often ran at 75% load or more during times when historically it ran no more than 30%.
The attitude of "well, it's going to happen anyway, might as well deal with it" is garbage. Adopting such an attitude in the face of a hurricane, the forces of which cannot be stopped, is fully acceptable. But in the face of spam which should not exist in the first place, this attitude is comparable to rolling over and taking it right up the rectum rather than dealing with the source.
The major spammers have agreements with the ISP's so they can use major amounts of bandwidth.
I'm guessing the article was about dial-up accounts because I don't see anyone opening 4 or 5 dsl accounts a day.
So, the easy solution is to block port 25 from your dial up accounts. Or, at the very least, limit the out-bound connections on port 25 from those accounts. Either by number of connections (limit the number of spam messages sent) or by a fixed number of destinations (a lot of spam can be sent to a few addresses).
Earthlink should also be blocking port 25 access from dial-ups to known open relays.
Brother! TESTIFY! Woo! :-) With you 100%.
Carousel is a lie!
One notorious spammer, whom EarthLink helped put behind bars, repeatedly used the names of sports such as baseball and football as his password.
Did anyone else see the implications of that? It says, "Earthlink admins know your password." Every security system I know stores passwords using a one-way hash. It is supposed to be impossible for an administrator to discover the password from the stored data. But this admin just admitted he is that checking the cleartext passwords. Make certain you use a different password for every account.
Now the spammers know to use random passwords, so the admins have one less method to catch them. Did he kill every account that used "baseball" as the password? Probably not a bad idea, but not practical for a commercial ISP.
---
My Earthlink account only started getting obvious spam in the last year. The Subjects are variations on one of these:
"ORIGINAL SOFTWARES TO ALL COUNTRIES"
"V1AGRA, C1ALIS, XANAAX, VAL1UM AT CHEAPEST PRICE"
Is there any spam filter that would not catch them just by looking at the Subject? I receive only four each week, so maybe Earthlink is not doing too poorly.
I spend my life entertaining my brain.
I'd really like to see real investigators in action like those of Spamhaus (for example) who have entire biographies and histories of spammers. Guys who are geeks, not paper pushers (so to speak).
This was pedantic in the extreme.
"I don't understand why all the focus on ISPs."
Because, unless you have a peering agreement, you are connecting to an ISP.
"You call the phone company (any phone company) and say you want a data T1 connection."
Okay. That's a chunk of money and it has a physical connection point that is recorded. It is completely different than a dial-up account.
"They give it to you and give you some IP addresses."
From their block. That means that they are your upstream provider. If someone complains about your behaviour, they will complain to your upstream provider who will then cut you off (or not).
"They do not process email for you, they do not give you web space and they do not respond to complaints about what you are doing with your T1."
They do respond to complaints about what you are doing.
"I expect this holds true for any sort of data connection from a telecommunications provider that is not providing any additional services, which means if you call SBC to get an OC48 they aren't going to ask you what you plan to do with it."
That is correct. They will not. But you ARE plugged into THEIR network.
One end of the line terminates at your location, the other end terminates at the phone company's location.
So, traffic coming from your line goes through the phone company's network. And people can see who licensed that IP range to you. They will complain to your upstream provider.
I was a consultant who telecommuted to work, from the other corner of the country. I used Earthlink dialup to get to the Net. Among other things, I had my outgoing mail server set to the company's mail server, so that my headers would appear identical to those of any other person in the company.
One weekend, our corporate admin updated the email server software on our servers. Now, be aware that the server apparently had some issues (wouldn't accept fragmented packets as a security workaround to a packet filter that didn't properly analyze them, for example) that resulted in my connectivity to it (from the other corner of the counry) not always being flawless.
I soon realized that that morning, after this server update, I could no longer send email to customers through the company's server. Note that the only change to the infrastructure I used to send mail, that I was aware of at the time, was the corporate server update. It seemed logical to me, knowing what types of tradeoffs were considered acceptable by that admin, that the network had been further broken in some way as a security measure. So, I called the guy and said, "Hey, I can't send mail through the company's server any more. What's the deal with that new software?". He said he didn't know and would look into it.
Hours of puzzling later, he got around to monitoring the network to see if the packets I said I was sending were getting to the server in the first place. Answer: no.
The Earthlink fsckers had picked the same weekend to enable port 25 blocking.
Yeah, the admin loved me from that point on. Not.
The workaround, since it was not acceptable to have my email going out with Earthlink headers and too many personal cohorts knew my Earthlink address for me to want to change providers: Admin set up a second mail server listening on a random port number, and told me the port number.
Not. Not ever. We went through a long period where some folks at SpamCop decided we were spammers because of a subscribe-to newsletter that they didn't remember or appreciate. So, we were spammers. Lots of complaints were received here. Lots of complaints were sent to McLeod USA (who we had a T1 from at the time). Some news:
Spammers that use dial-up connections and cable modems can have problems with their ISP. You can complain to an upstream provider all you want and it is extremely unlikely to get anywhere. I suppose you might try complaining to Level3 about a spammer saying that SBC or McLeodUSA was unresponsive, but I don't think that is going to get anywhere either. Our current T1 cannot be terminated for any "TOS violation" - there is no TOS. This might not be true in all cases, but it is true in enough that spammers can operate with impunity over the network.
Until there is a clear definition that spam is illegal - which today there is not - spam will continue and network services will contine to be provided to spammers. If spam were somehow declared to be illegal - such as defining any email with a commercial message to be illegal - then network providers would be forced to do something. As it is today, there is really nothing that can be done about the origins of most spam.
Can you turn off the guy that decides to buy a list of 28 million email addresses and send email from their cable modem? Absolutely! Does this change the amount of spam that you can I get each day? No way.
Google Archive in HTML
Powerpoint format
Steve Atkins presentation to the ASRG: Google cache as HTML
Same as powerpoint
A graph of a random minute at a large email provider.
Each point is one host.
Those numbers are all very very real.
I can throw myself at the ground, and miss.
Gimme a break.
Here are 570 000 spams that CAME FROM earthlink...
I'm going to make some assumptions here, but your solution was part of the problem.
Anyone who uses SA for high-volume traffic knows that it is slow and a hog - perl, while being useful, is not known for it's speed.
DNSBL + Caching DNS server (such as dnscache, but if you're an ISP you probably have better solutions) will block a heckuva lot of email. Solutions like messagewall take this even further - filtering on headers, attachment extensions, content types and virus checking *while the message is in transit*, did I mention it's lightning fast?
In other words, keeping your guaranteed spam from SA solves a *lot* of your processing issues. I've been a target of "spam bombs" in the past and watched my mail server shoot from < 1 load to > 100 in minutes (I also use FreeBSD). Installing and properly configuring messagewall dropped that processing with a similar bomb kept it around 3. messagewall is truly an amazing piece of software and works with any SMTP server (although you'll get much better performance if your server supports pipelining).
Probably my only beef with messagewall is that it will not work with multiple interfaces on the same system accepting SMTP - I imagine this is fairly rare, but certainly not out of the question. However, a properly configured sendmail installation can do almost everything that messagewall can at the cost of speed and possibly security (messagewall instantly chroots itself and also does all of it's DNS async - it's also very, very small).
The result is that only negatives make it to the content analyzer, that being SA, which after relieving itself of all that processing time analyzing content becomes a much more trivial task. If you're using white/blacklists, moving your databases off to a remote server will also save you load on that server.
Keep in mind that if you're using qmail, running the perl version of qmail-scanner slows things down considerably. I believe magic-smtpd will do what you want, and I've also heard of a C replacement for qmail-scanner. The perl version is remarkably flexible however, and ditching it is definitely something to think about long and hard before committing to.
Not criticism of your work, I'm sure you are well-qualified and justified in your actions, but there's a lot of mail software out there and most of it sucks, so learning about new, good tools can be hard.
As far as Perl and speed goes, from what I remember it wasn't much of an issue; we used the spamd c-based daemon to pass email to just-the-one copy of SpamAssassin, and it wasn't that bad. There was a bit of delay, but it was nothing like before when we were using Procmail. One bad entry in Procmail could bring the whole thing grinding to a halt.
We considered using SA as a milter (I'm pretty sure that option is/was available), but for reasons I can't remember decided against it. The sysadmin was a fan of Sendmail, so switching wasn't an option (though I hear qmail is being considered now).
One thing I wanted to do but didn't have the time for was to set up DCC; I think that would've helped significantly.
If you're using white/blacklists, moving your databases off to a remote server will also save you load on that server. -- We had nothing that sophisticated; the priority right then was to get spam filtering off our customer-facing mail server (and DNS, and POP, and radius...ugh). No whitelists beyond very simple SA stuff, and certainly no individual preferences for SA.
The result is that only negatives make it to the content analyzer, that being SA, which after relieving itself of all that processing time analyzing content becomes a much more trivial task. -- Absolutely. There's a lot that could have been done better, and no doubt that I made some bad choices (learning, always learning). But, though I bitch and moan, it was better than what we had before.
Thanks kindly for the compliment and the suggestions...it's always good to have a second opinion.
Carousel is a lie!
Cursory research points to this:
i st ory/Scien149.html
http://www.rickross.com/reference/scientology/h
And also this concerning the other fellow (Reed):
http://www.slatkinfraud.com/elnk.shtml
Sky Dayton does indeed appear to be a follower of L. Ron, and his homepage still bears the quote mentioned in the article.
It's also because tracing spammers sufficiently well that you can haul them into court and force them to pay is usually a lot more expensive, has a low probability of success, and if they're in the US where you can prosecute them, it usually just results in an uncollectable judgement against some low-life living in a double-wide trailer.
The bandwidth doesn't actually cost you much, compared to the legitimate web-surfing traffic of your users. Admin labor costs a lot. Complaint handling costs a lot (assuming you do it well.) Servers themselves are pretty cheap - yes, that 66 MHz Sparcstation can handle the non-spam load, but you can buy yourself a stack of $250 2GHz Celerons to handle most of the crunching labor; your choice as to whether to distribute the load using DNS or make it work in a single Beowulf cluster.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
If you're in the business of providing people email service, as opposed to generic ISP connectivity services, then yes, spam is a major fraction of your bandwidth. But if you're a connectivity-type ISP, like the traditional dial ISP or a DSL ISP, usually most of your bandwidth is web browsing these days, or P2P file sharing traffic, and email is a much smaller fraction of the total bits. Spam may be 80% of your email, depending on how much you blacklist at the SMTP layer rather than accepting and discarding, and it may be 90% of your complaint traffic, but it's usually not a lot of bits. (Viruses are different, because they can really dog out the network for a while.)
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
We're a small company and spam costs our company about $3000.00 a month. Bandwidth and storage isn't the cost problem but labor is what cost.
I know I don't process a million spam complaits in a month so your guess of it costing $20 to $30 an month is WAY off base. Like I said I like to get paid just as anyone else does and this is the true cost of spam. Me sitting here at this workstation trying to keep the bastards out. Besides even if it does only cost 4micro$ per spam it is still THIEF OF SERVICES no matter how you cut it. Spamming is stealing. I'm sure whatever job you work at you don't work for free. Neither do I. Just as in anything the largest cost of something is normally in the labor.
If I had the money (I don't), I would pay for a professional hit on a few of the most notorious spammers. I'm not kidding.
I would pay big money for an experienced and expert hitman, to do the job carefully, patiently and thoroughly.
Once a couple of the well-known spammers were iced, I think we would see a serious decline in spam.
I don't feel all vigilante about other, more serious crimes. I don't think violence solves anything. I oppose the death penalty. I know this is an irrational position, but I don't care.
Does this make me a bad person?
I'm continually getting empty-body emails. I assume they are testing my address to see if it's alive. _Please_ could ISPs set up the MTAs to bounce empty messages.
I used to administer mail servers for a large ISP (millions of DSL/broadband connections). SPAM was punishing my servers and I would routinely get paged by my servers at 4am when the spools filled with outbound mail thanks to some sub that attented a MLM seminar somewhere. When I reported these clowns to my own abuse dept, little ever happened. I could produce tons of evidence that positively identified them as the sender, and still little happened. If the user finally had their account terminated, they would often signup for service again using an alias. I could easily spot this because I was keeping track of their account info as well. (10 different names, same street address, etc).
Anyway, I got tired of pissing in the wind so I started to fight them in a way that actually worked. I set their modems speeds to less than a 33.6k baud modem. If they called techsupt, nobody could help them because all the management tools showed their connection as CLEAN, good SNR, everything is GREEN... "Did you try rebooting?" "Can you open your network connections and verify the settings?" "thank you call again!"
Can't wait for wide-spread industry adoption of SPF!
I am uncertain Earthlink's SpamBlock works if you POP3 mail. I have it set to the "medium" setting, and have never received a "daily spam report".
(The "high" setting requires whitelisting everybody, which means giving Earthlink my address book. Even I do not have my address book; I just search for a previous email from somebody and click Reply.)
Whatever Earthlink is paying for the SpamBlock is too much if it does not do anything.
I spend my life entertaining my brain.
i've got karma to burn and I'm not guna sit idly by while watching clueless pundits get on their soapboxes and get modded as "insightful".
Do you have any fscking idea of what you're talking about? Would you care to enlighten me and give us a detailed list of "pro-active" steps they should be taking to prevent spammers from getting accounts in the first place? Did you read the article? Spammers have for years used stolen credit card accounts, fake business fronts to sign-up for connectivity and mail accounts from all pure-play ISPs out there and even, *gasp*, AOL. There's no way an ISP can have any idea of who opened the account and what they're guna do with it, until they start sending spam.
All ISPs have had their share of trouble with usenet. If you're big, if you're popular, if you're on-top of the flagpole, well yeah, you attract losers and get flack for it, not much you can do about that as an ISP. As a user, you can vote with your feet and go to a smaller ISP. I don't because I need the nationwide dial-up, I travel, and move around. However, the flip-side of being such a con-artist magnet is that you're in a great position to catch the fsckers red-handed, prosecute them and/or stick'em behind bars. Show me one another ISP who's had a better track record at doing this than earthlink. I'm all ears.
I'm amazed by all the slashdot pundits out there who are so outspoken against [insert whatever name of ISP who's trying to do something different], and seem to know so much about running an ISP. Why don't they offer links to their own ISPs and show us how they do things differently? I'm hardly being sarcastic, if you have the secret to making the internet a better place, then either let's hear it, or show me where to sign-up for an account.
I've looked at your other posts and I remember hearing about those cleartext password issues in the past. I really hope they've addressed them, thankfully, I don't use those account passwords for anything else. i'm not saying earthlink doesn't have issues they need to deal with. But if you're guna repeatedly get on your soapbox to bash them, try at least to stick to what you YOU KNOW. Who knows, maybe things have changed since they laid you off.
Extraordinary Vacations. Exceptional Prices
No, a big chunk of the CPU and network time of mailservers is eaten up by spam, even if you don't try to block. With over half of all email being spam these days, and much of that email bouncing and thus not being directly spam but still resulting from it, it's even worse. For many sys-admins, dealing with the spam when a particular burst of it exceeds any sane threshold they might have predicted for spam takes many hours, if not days, of expensive adminn time to clear up. And it keeps happening to small sites that can hardly afford the resources.
I know that I am at the bottom of the queue, but something I have a tendancy to do, and try to do a few times a week, (about 1 in 100 spam), is take the 5-10 minutes, to tracert to the website they are advertizing, and report it to them, their upstream, and their upstream's connection (getting to a backbone level isp)...
Also, sending an email to every contact on their whois (noting that more and more "hide" their whois, send it to the abuse at the registrar)..
If everyone did this for just one spam a day, it would make a *huge* difference... Also, if everyone would simply have an auth-relay for their smtp, and impliment SPF, that would go a long way too.. then people could actually reject based on spf. then blackhole lists could actually work.
Michael J. Ryan - tracker1.info