New Spam Frontier: Referer Logs

geoffsmith writes "Wired News is reporting that spammers are using referer logs as a cheap new way to spam small sites. Anyone running a website has probably already seen this phenomenon; I'm thinking of writing a script to remove these entries from my access_log by looking for hits that don't grab my images. (sorry lynx users!)"

Comments

They will never stop.

[SexyKellyOsbourne]

The entire internet will eventually go down in a deluge of spam unless it is made illegal and the laws are enforced!

Re:They will never stop.

[IIRCAFAIKIANAL]

Perhaps linking to or publishing the code to DeCSS would have been a better example.

Yes, it would have :)

You have no concept of either A)what a battlefield is, or B)what's REALLY going on in the world.

I agree that the Internet can be and is a battlefield. So can Wall Street or the TSE. Or major media sources. Or the telephone. I don't agree that it is the most active - perhaps from a first world perspective, but I try and think a little more globally than that.

Just a note: We have more attacks per day on one of our public .mil servers than we have had real contingency issues(to include disasters and humanitarian aid) in the whole theater all year.

And how many people died due to those attacks on the public .mil servers? (Yes, I am sure they are important for various reasons, but if I was *attacking* the USA, I would be hacking hospital databases - there is a scary potential for warfare there).

Has anybody else noticed that the internet is currently the most active battlefield in hostory?

Hacking a .mil server certainly qualifies as warfare but you basically said that the internet is a more active battlefield than, say, WW2. I disagree.

(And anyone considering invoking Godwin's law... piss off :)

I concede that the Internet certainly *is* a battlefield. However, considering that conflict on the Internet barely affects most of the people of the world, I wouldn't rate it so high.

Spam Lite

[Cyno01]

I don't know if i'm the only one, but has anyone else who doesn't filter their e-mail noticed a drop off in the amount of spam they recieve? For about the past 2 weeks, the amount of spam in my hotmail inbox has dropped from about 40 to around 15 a day. Anyone else had something similar to this happen?

Re:The spammer speaks...

[Ponty]

It is innovative. I was surprised and amused. It's awful, though. There's no rule that innovative things have to be positive.

Anyhow, unless the traffic is completely disabling, I don't see this as more than an annoyance that technology will filter out when it becomes sufficiently obnoxious.

Well..

[joyoflinux]

He just got a link posted on /. and Wired--I wonder how many spammers are going to target him now...This seems a little aganist logic

You can do better than that

[Subcarrier]

Actually it would be quite nice to see some of these "marketing gurus" put a little more thought into their spam. Today, some of the most carefully crafted content on TV is commercials (lamentably, also some of the worst). Watch and learn. I wouldn't mind receiving a spam that is fresh, funny, engaging, and didn't involve a virgin, my cock, a septic tank, or a gentleman from Nigeria. I wouldn't mind a funny beer commercial, for instance.

Sorry 'bout what?

[PissingInTheWind]

...(sorry lynx users!)

Sorry about what? Why should they care wether you keep them in your log or not?

Boost search engine ranking?

[j7953]

From the wired article:

... even though they ruefully admit that the log spamming may falsely boost their ranking on some search engines.

Umm, huh? I don't think the spammers actually link to the sites, they probably just send HTTP requests with faked referrer headers that contain the URLs of the spammer's web site. That won't boost your search engine rankings.

Re:*sigh*

[Dyolf Knip]

The few stats I've come across regarding spam 'success' suggests that if they get more than a dozen responses (excluding the fools who actually send back "Take me off your list") per one million emails they're having a good day.

[Wishful thinking mode ON!]
This implies that there are, maybe, all of 10,000 suckers who keep every spammer on the planet in business. If we find them and cut them off, spam response would drop to about 1 per billion and there's just no way they could make any money off of that.

Re:referer information should be disabled by defau

[Openadvocate]

There are many reasons, mostly for those who program websites. Sometimes you don't want people to see a page before another. this could also be solved with cookies, but some blocks those too.
Then there is the statistics, learn how people navigate around your site. referer can help you see a pattern and improve your layout.
Also it can prevent bandwidth hogs, mostly a issue with ad. graphics and pron sites where people use graphics from others servers on html pages on their own sites but also on free servers where people place graphics and files and link to those directly without using any html and then not showing any of the free servers ad's which provides them with money to run the sites in the first place.

Re:*sigh*

[AirLace]

I know why this problem is endemic. It's certainly down to more than the "10,000 suckers" you suggest.

I always use the example of my father, who is your archetypical pre-UNIX geek. He did all the PDP-11 stuff, worked with the VAXes and hacked machine code in ways that I don't yet understand -- an intensely intelligent man. Yet, every few months when I go to visit him, we get to talking about the internet and the first thing he does is talk about what he's bought online. For him, paying spammers is part and parcel of buying online -- he's paid spammers for search engine placings for his personal site, silly trinkets like water pumps and gardening tools and books.

To people who aren't part of the current 'geek' cognoscenti, spam is just another form of valid advertising, like the leaflets they get in the post and the billboards they walk past on their way to work. This isn't a specific group of people -- you can't "find them and cut them off" -- you need to target the problem at its source.

Re:referer information should be disabled by defau

[Dr. Awktagon]

Just also check for a magic string in the user agent and voila! trusted computing reinvented. To make it unhackable - just add a few more levels of obfuscation. ;))) The sad part of this, is that I have actually seen authentication schemes like this. Don't know whether I should cry or laugh :)

probably cry... what you described could easily be enforced with the DMCA.

If you use wget, watch out when using "--referer" and "--user-agent".... you just might be breaking TEH LAW!!!

Re:referer information should be disabled by defau

[phliar]

Do what I do: use Privoxy. Not only can you use it right now with whatever your favourite browser is, it's free. Not only does it block ads, it allows you to set Referer: on all outgoing requests to whatever you want. (I set it so Referer: is always the base URL of the page being viewed.)

Incidentally, I don't know why anyone bothers with logging referrer information. The only use sounds like what the bloggers do. If you're not a blogger, why do you even care who the referrer is? Half the time it's bogus or one of your own pages.

Re:referer information should be disabled by defau

[Permission Denied]

I can send them directly there, give them a specific welcome message if they haven't been to my site before, etc.

This is so damned annoying. If I'm searching for some specific information, I don't give a damn about your idiotic welcome page. I don't care what your website is about or what you have to say on your other pages - all I care about is the specific technical information that google told me you have.

More and more, I'm finding myself using googles cache instead of clicking on the actual links. I know you couldn't care less about my insignificant browsing habits, but the more people start doing annoying crap like this, the more people start using google instead of the web.

"This page is restricted to users of xyz.com. Please go there first."

Do you realize how stupid this is? You're trying to control how I use my browser. Of course I'm not going to go to xyz.com and try to use their idiotic navigation looking for a link to you. You're simply advocating another form of advertisement and I'm not interested. I care about the data you're providing, not how you're getting funded.

I can use the referrer to limit the damage done by only allowing the images to be referred by pages from my own site.

And this is, of course, broken behaviour. Did you know that when you open a new link in Netscape/Mozilla that the browser does not send any referer at all? This means that I can't open your images in new windows and I'm constrained to view your images one at a time. Also, the some browsers change the referer for images when you "save" images (eg, right-click and choose "Save as..." may not send the referer you're expecting).

If taken away one restricts opportunities for the site operator to personalize and protect content on their site.

If you're using this to restrict content to your site ... well, forget it. If you have something I really want, I'll open up a terminal and telnet to port 80. Yes, this is indeed effective restriction. (Quiz to see if you really know what you're doing: how would you set it up so that you know that a user has previously visited another site, with cryptographic confidence?)

As for "personalizing" content, please stop. The only times I've seen that word being used in a web context is to personalize advertising (and also restricting content because I'm not using IE, but don't get me started on that). I've never seen anyone "personalize" a site in a useful way, eg, "You're a C programmer who writes Solaris kernel modules, so you're probably not going to spring for my Herbal viagra scheme and I'm going to cut the marketing BS and give you only useful information."

Why do these "blogs" even keep logs of referer links? This is pure narcisism (and more importantly, a waste of disk space - even though disk is cheap, it's still worth more than someone else's paltry feeling of acceptance). If you're going to say something, just say it. Don't base your life around how many people like what you say. "Ohh, somebody linked to my journal, that means I'm special and I can now feel good about myself." Ahh - get a life.

I swear, "webmasters" piss me off.

Re:huh

[ShaunC]

and then people looking at the report would say, "hey, the page at wilddonkeysex links to my perfectly/valid/page and it's getting like 500 hits a day from there, woo! let's click on that url and see what the link to my page looks like!"

Actually it's even better than that. As you mentioned, many sites place their server logs online for public viewing; but take that a bit further. A lot of website stats packages will automatically turn referring URLs (and other data) into hyperlinks, to "pretty up" the stats pages. Because some search engines rank your page, in part, based upon how many other sites link to you... Well, you see where I'm going with this.

People don't have to visit the "victim" site at all, and they certainly don't have to browse the stats. The stats programs and search engine spiders will take care of everything. Got a low-ranking, poor traffic site that nobody links to? No problem, you can have 1,000 people linking to you by the end of the week, whether they know it or not. This really is nothing new, and the spamming side of it (i.e. repeatedly hammering a site) reminds me of how most TopSites work. These have been around forever, and so have the many methods of tricking them.

Placing your URL as the referer to sites with public stats can be quite helpful in boosting your rank, and a slightly hacked copy of wget or w3mir can make it an easy task. I guess the only real "news" here is that, once again, a few village idiots have failed to realize that some things are only good in moderation. There's neither a need nor an excuse to log yourself as a referer to any particular site more than once a month; and hundreds or thousands of times in a day is just plain stupid.

Shaun

Re:*sigh*

[ealar dlanvuli]

I actually bought something from a spam. It was a slightly topical T-Shirt that I thought was clever. Cost me $15 (PayPal).

The guy who sold it to me was obviouly a late teen, and was making ok money selling shirts at about $5 profit per when I called him.

I think most geeks have no problem with spam itself (in fact targeted spams that interest me often get clicks, I get about two of those a year), they have a problem with the number of scams that are sent using spam.

Backlinking

[CaptainSuperBoy]

Backlinking, or posting your referral logs, is doomed to failure and rightly so. It's just a glorified way of making your site into a link farm, with the expectation that your fellow bloggers will do the same. It is serendipitous that this practice is open to 'abuse' although I would never call the abusers spammers. They are just utilizing a method for submitting data that the site owners themselves have provided. I don't see any reason to call this 'spam' since the site owners are inviting users to submit data through HTTP referral headers.

Also, this quote from the article is ludicrous: "bloggers are not thrilled, even though they ruefully admit that the log spamming may falsely boost their ranking on some search engines."

There is no search engine that bases your rank on the number of sites that you LINK to. I believe the bloggers actually mean that they're sorry to see their backlinks (read: link farms) go, since those do in fact raise search rankings. What a travesty- Sites may have to rely on the actual quality of their content, rather than trading links!

Amidst the alarmist cries in the article, "spammers will destroy our practice of posting referral logs," nobody has even mentioned that there is a ridiculously easy technical solution. Before posting a referral link, why not just have your software visit the referring site and detemine if it actually links to your page? This will defeat the referral advertisers.

not spammers?

[luap2000]

Isn't that kind of like saying because someone has an email (a method for submitting data), it's okay to spam?

Plus, think of the numbers. If people are selling this 'service,' it's bound to have a negative effect on the overall quality of the web features like this offer.

Re:Why not fight back against the scumbags?

[t1m0r4n]

they would have HUGE bandwidth bills, and think twice about using the same marketting technique again

Actually, any smart spammer won't host the web page on a real server (at least in most countries). Even in the adult industry spam is strictly forbidden by service providers and sponsors.

The web page you are hitting almost always is on a temporary account with the intent of being shut down quickly. The web server is probably sitting in an apartment somewhere and bandwidth bills are no concern. In a couple of days the computer will be unplugged, reformatted, and connected to a different internet service provider.

Spammers usually aren't computer illiterate. By day they are highly skilled sys admins or consultants in the corporate work place. They send spam by night because it pays better.

Quite simply, if people didn't make purchases the spam would go away. But, alas, that is not the case.

Re:Referer checking for images

[Permission Denied]

So do you have an alternative proposal to prevent resource (i.e. bandwidth) theft?

Session cookies based a cryptographic hash of browser-identifiable information. Just hashing the IP and some secret string will prevent the bandwidth-stealing problem (not ideal since it breaks with NAT, but that's irrelevant if you're only trying to solve the deep-linking problem).

In php, setcookie('hash', md5($ENV[REMOTE_ADDR] . "TOPSECRET)) on page load, link to a file "image.php" instead of the .jpg and "image.php" does something like this: if (getcookie('hash') != md5($ENV[REMOTE_ADDR] . "TOPSECRET")) { header("Location: /error-documents/403.html"); exit(); }. This isn't complete (probably not even syntactically correct and be careful with what image.php allows one to download), but you get the idea. The actual image files can't be downloaded by apache, but can only be opened and sent to the browser through "image.php". For extra fun, re-generate the secret string from /dev/random every ten minutes (and keep around the last version of the key to avoid breaking on-going sessions).

This stops everyone from stealing bandwidth (including telnet-wielding network programmers like me) and it annoys no one.

How is that any good?

[Wakko Warner]

Keyboard nav is much better than links (use numbered links with "G," as in "25g" takes you to - but doesn't follow - link/text entry box #25 on the screen, etc.).

So, you mean you sit there and count how many links are on a page, then figure out where on the page #25 is, and then type all that in to go to it, instead of just scrolling down and clicking or something similar? How incredibly stone-age.

- A.P.