Slashdot Mirror
New Spam Frontier: Referer Logs
geoffsmith writes "Wired News is reporting that spammers are using referer logs as a cheap new way to
spam small sites. Anyone running a website has probably already seen this phenomenon; I'm thinking of writing a script to remove these entries from my access_log by looking for hits that don't grab my images. (sorry lynx users!)"
The entire internet will eventually go down in a deluge of spam unless it is made illegal and the laws are enforced!
"I'll adapt or I'll discontinue. I'm not planning on becoming the major annoyance of the blogging world.... I'm not too worried my reputation. Marketing is all about being innovative, different, adaptive, taking risks and knowing how to use the technology. I'm trying to be all that."
Heh, it's funny that this guy can make this statement and expect to be taken seriously. It's even more pathetic that he actually thinks he's "innnovative".
I don't know if i'm the only one, but has anyone else who doesn't filter their e-mail noticed a drop off in the amount of spam they recieve? For about the past 2 weeks, the amount of spam in my hotmail inbox has dropped from about 40 to around 15 a day. Anyone else had something similar to this happen?
"Sic Semper Tyrannosaurus Rex."
(sorry lynx users)
Don't worry. It's highly unlikely that any of the 4 current users will visit your website anyway.
He just got a link posted on /. and Wired--I wonder how many spammers are going to target him now...This seems a little aganist logic
Windows users are complaining that Microsoft is filling up their computer's System Event Log with spam about illegal exceptions and page faults.
I don't know who started it - but I find it very odd that browsers send referer info by default. Why? It does not provide anything extra for the user but problems. It is not once or twice that you find URLs to "confidential" pages if you browse through your webserver logs. And... I bet 95% of web surfers do not even know that they are sending this information all the time. Is there really any reason why the default is to send the referer info? I have seen people riot on much less important privacy issues. Why not about this? The referer plague exists in almost all browsers - and only in few browsers you actually can easily turn it off. What's going on?
Actually it would be quite nice to see some of these "marketing gurus" put a little more thought into their spam. Today, some of the most carefully crafted content on TV is commercials (lamentably, also some of the worst). Watch and learn. I wouldn't mind receiving a spam that is fresh, funny, engaging, and didn't involve a virgin, my cock, a septic tank, or a gentleman from Nigeria. I wouldn't mind a funny beer commercial, for instance.
"I have opinions of my own, strong opinions, but I don't always agree with them." -- George H. W. Bush
...(sorry lynx users!)
Sorry about what? Why should they care wether you keep them in your log or not?
A message from the system administrator: 'I've upped my priority. Now up yours.'
From the wired article:
Umm, huh? I don't think the spammers actually link to the sites, they probably just send HTTP requests with faked referrer headers that contain the URLs of the spammer's web site. That won't boost your search engine rankings.
Sig (appended to the end of comments I post, 54 chars)
In the regular prefs and the "quick prefs" (F12 under Windows version) Opera lets you turn off referrer logging. The only time I need to turn it on is certain sites, like my credit union, which is no big deal...
I should put something clever here. Maybe someday.
255.255.255.255 - - [27/Oct/2002:00:00:00 -0000] "GET /perfectly/valid/page/at/yoursite.html" 200 2467 "http://www.wilddonkeysex.com_for_Wild_Donkey_Sex/ " "(SpamBot5000)"
and then people looking at the report would say, "hey, the page at wilddonkeysex links to my perfectly/valid/page and it's getting like 500 hits a day from there, woo! let's click on that url and see what the link to my page looks like!"
-calyxa
Decay! Decay! Decay! -Helium
I'm not sure I understand. Does this mean the spammers put links on their own porn (or whatever) sites, and casual surfers will click into the blog from the porn site, thus making the porn site show up in the logs as the referer? That's how the referer is supposed to work, right?
Or are they just bots that hit random web sites and send fake referers along?
Either way, I have absolutely no clue why this would be abusive or even annoying? Can someone explain? Do people sit around checking their referers all day long?? (Then again, I don't understand why anyone would run a blog, so maybe I'm just out of touch).
I clean out all my outgoing referers (thanks squid), so maybe I subconciously assume everybody else does too. Never thought of the referers as anything but a silly waste of bandwidth, since they can be forged so easily.
[Wishful thinking mode ON!]
This implies that there are, maybe, all of 10,000 suckers who keep every spammer on the planet in business. If we find them and cut them off, spam response would drop to about 1 per billion and there's just no way they could make any money off of that.
Dyolf Knip
Yes, referrer information makes an excellent authentication scheme for highly confidential system dealing with transfer of mission critical information. ... Just also check for a magic string in the user agent and voila! trusted computing reinvented. To make it unhackable - just add a few more levels of obfuscation. ;))) The sad part of this, is that I have actually seen authentication schemes like this. Don't know whether I should cry or laugh :)
There are many reasons, mostly for those who program websites. Sometimes you don't want people to see a page before another. this could also be solved with cookies, but some blocks those too.
Then there is the statistics, learn how people navigate around your site. referer can help you see a pattern and improve your layout.
Also it can prevent bandwidth hogs, mostly a issue with ad. graphics and pron sites where people use graphics from others servers on html pages on their own sites but also on free servers where people place graphics and files and link to those directly without using any html and then not showing any of the free servers ad's which provides them with money to run the sites in the first place.
my sig
I think it's more than the web site's owner clicking on the page - a lot of bloggers post a list of "top referrers" on their web site as a way of thanking the referrers, and therefore they generate a lot of traffic to their referrers from their own visitors.
I know why this problem is endemic. It's certainly down to more than the "10,000 suckers" you suggest.
I always use the example of my father, who is your archetypical pre-UNIX geek. He did all the PDP-11 stuff, worked with the VAXes and hacked machine code in ways that I don't yet understand -- an intensely intelligent man. Yet, every few months when I go to visit him, we get to talking about the internet and the first thing he does is talk about what he's bought online. For him, paying spammers is part and parcel of buying online -- he's paid spammers for search engine placings for his personal site, silly trinkets like water pumps and gardening tools and books.
To people who aren't part of the current 'geek' cognoscenti, spam is just another form of valid advertising, like the leaflets they get in the post and the billboards they walk past on their way to work. This isn't a specific group of people -- you can't "find them and cut them off" -- you need to target the problem at its source.
The "solution" you mentioned wouldn't really work, as the spammers could simply download your images as well.
/dev/zero in order to place their refer entry, that's great, more power to them. If they don't download data, that invalid refer entry could easily be dismissed. Solution? I'm sure someone will crank out a spammer-refer-mod to include in apache.conf over this. :)
I see a solution in this. It would be the spammer's own DOS attack. If they willing to download
It's nice, as a site operator, to know where your guests are coming from. A good portion of my visitors come from Google and other search engines. The referrer log lets me know what they were searching for, and in nearly 95% of the cases they were looking for a specific topic on my site. I can send them directly there, give them a specific welcome message if they haven't been to my site before, etc.
Furthermore I can restrict traffic for some areas of my site (like some sites that block links from slashdot) for particular reasons or uses. "You just came from the page of an associate and are able to receive a discount." "This page is restricted to users of xyz.com. Please go there first."
Lastly, it protects my image content. My images are not stellar, and yet other sites continue to use them on their pages. I can use the referrer to limit the damage done by only allowing the images to be referred by pages from my own site.
Referrer information may be annoying to you, but it's an extrememly useful tool. If taken away one restricts opportunities for the site operator to personalize and protect content on their site. Not a huge loss, but it isn't really as great a privacy issue as you seem to believe.
-Adam
Just also check for a magic string in the user agent and voila! trusted computing reinvented. To make it unhackable - just add a few more levels of obfuscation. ;))) The sad part of this, is that I have actually seen authentication schemes like this. Don't know whether I should cry or laugh :)
probably cry... what you described could easily be enforced with the DMCA.
If you use wget, watch out when using "--referer" and "--user-agent".... you just might be breaking TEH LAW!!!
Incidentally, I don't know why anyone bothers with logging referrer information. The only use sounds like what the bloggers do. If you're not a blogger, why do you even care who the referrer is? Half the time it's bogus or one of your own pages.
Unlimited growth == Cancer.
As it says in the article, some blogs have automated lists of the top referrers, so that visitors can see who links to the blog. And yes, we're talking about bots sending fake referrers.
"The question of whether a computer can think is no more interesting than that of whether a submarine can swim" -EWD
It is extremely useful for security purposes.
No, not the security most people are thinking of. Checking to see if the user came from FeedBack.html before executing FormMail.pl is no security, since spammers can forge any referer they want.
I'm talking about security which stops a human user who is logged in to a particular website from being tricked into performing actions they didn't authorise. For instance: I log into my server's adminsitrative area. Then, in another window, I browse someone's blog. And I click on their "search" button. As it turns out, this search button is a trap, which sends me to my own admin area with a command to delete someone's account. I'm logged in, I have a valid network address, I'm active, there's no problem. Except that fortunately my browser sends "Referer: www.blog.org" instead of "Referer: www.admin.com".
That's why referer info is useful: to prevent a user from being hijacked.
Slashdot monitor for your Mozilla sidebar or Active Desktop.
This is so damned annoying. If I'm searching for some specific information, I don't give a damn about your idiotic welcome page. I don't care what your website is about or what you have to say on your other pages - all I care about is the specific technical information that google told me you have.
More and more, I'm finding myself using googles cache instead of clicking on the actual links. I know you couldn't care less about my insignificant browsing habits, but the more people start doing annoying crap like this, the more people start using google instead of the web.
"This page is restricted to users of xyz.com. Please go there first."
Do you realize how stupid this is? You're trying to control how I use my browser. Of course I'm not going to go to xyz.com and try to use their idiotic navigation looking for a link to you. You're simply advocating another form of advertisement and I'm not interested. I care about the data you're providing, not how you're getting funded.
I can use the referrer to limit the damage done by only allowing the images to be referred by pages from my own site.
And this is, of course, broken behaviour. Did you know that when you open a new link in Netscape/Mozilla that the browser does not send any referer at all? This means that I can't open your images in new windows and I'm constrained to view your images one at a time. Also, the some browsers change the referer for images when you "save" images (eg, right-click and choose "Save as..." may not send the referer you're expecting).
If taken away one restricts opportunities for the site operator to personalize and protect content on their site.
If you're using this to restrict content to your site ... well, forget it. If you have something I really want, I'll open up a terminal and telnet to port 80. Yes, this is indeed effective restriction. (Quiz to see if you really know what you're doing: how would you set it up so that you know that a user has previously visited another site, with cryptographic confidence?)
As for "personalizing" content, please stop. The only times I've seen that word being used in a web context is to personalize advertising (and also restricting content because I'm not using IE, but don't get me started on that). I've never seen anyone "personalize" a site in a useful way, eg, "You're a C programmer who writes Solaris kernel modules, so you're probably not going to spring for my Herbal viagra scheme and I'm going to cut the marketing BS and give you only useful information."
Why do these "blogs" even keep logs of referer links? This is pure narcisism (and more importantly, a waste of disk space - even though disk is cheap, it's still worth more than someone else's paltry feeling of acceptance). If you're going to say something, just say it. Don't base your life around how many people like what you say. "Ohh, somebody linked to my journal, that means I'm special and I can now feel good about myself." Ahh - get a life.
I swear, "webmasters" piss me off.
People don't have to visit the "victim" site at all, and they certainly don't have to browse the stats. The stats programs and search engine spiders will take care of everything. Got a low-ranking, poor traffic site that nobody links to? No problem, you can have 1,000 people linking to you by the end of the week, whether they know it or not. This really is nothing new, and the spamming side of it (i.e. repeatedly hammering a site) reminds me of how most TopSites work. These have been around forever, and so have the many methods of tricking them.
Placing your URL as the referer to sites with public stats can be quite helpful in boosting your rank, and a slightly hacked copy of wget or w3mir can make it an easy task. I guess the only real "news" here is that, once again, a few village idiots have failed to realize that some things are only good in moderation. There's neither a need nor an excuse to log yourself as a referer to any particular site more than once a month; and hundreds or thousands of times in a day is just plain stupid.
Shaun
Thanks to the War on Drugs, it's easier to buy meth than it is to buy cold medicine!
I actually bought something from a spam. It was a slightly topical T-Shirt that I thought was clever. Cost me $15 (PayPal).
The guy who sold it to me was obviouly a late teen, and was making ok money selling shirts at about $5 profit per when I called him.
I think most geeks have no problem with spam itself (in fact targeted spams that interest me often get clicks, I get about two of those a year), they have a problem with the number of scams that are sent using spam.
I live in a giant bucket.
Backlinking, or posting your referral logs, is doomed to failure and rightly so. It's just a glorified way of making your site into a link farm, with the expectation that your fellow bloggers will do the same. It is serendipitous that this practice is open to 'abuse' although I would never call the abusers spammers. They are just utilizing a method for submitting data that the site owners themselves have provided. I don't see any reason to call this 'spam' since the site owners are inviting users to submit data through HTTP referral headers.
Also, this quote from the article is ludicrous: "bloggers are not thrilled, even though they ruefully admit that the log spamming may falsely boost their ranking on some search engines."
There is no search engine that bases your rank on the number of sites that you LINK to. I believe the bloggers actually mean that they're sorry to see their backlinks (read: link farms) go, since those do in fact raise search rankings. What a travesty- Sites may have to rely on the actual quality of their content, rather than trading links!
Amidst the alarmist cries in the article, "spammers will destroy our practice of posting referral logs," nobody has even mentioned that there is a ridiculously easy technical solution. Before posting a referral link, why not just have your software visit the referring site and detemine if it actually links to your page? This will defeat the referral advertisers.
For now I'll delete the entries by hand, but if this increases it could get really annoying.
AlpineR
And this is, of course, broken behaviour.
So do you have an alternative proposal to prevent resource (i.e. bandwidth) theft? That is a very real problem, and no amount of arguing that the current solution is "broken" will get people to change unless you provide them an alternative.
15.1.3 Encoding Sensitive Information in URI's Because the source of a link might be private information or might reveal an otherwise private information source, it is strongly recommended that the user be able to select whether or not the Referer field is sent. For example, a browser client could have a toggle switch for browsing openly/anonymously, which would respectively enable/disable the sending of Referer and From information. Clients SHOULD NOT include a Referer header field in a (non-secure) HTTP request if the referring page was transferred with a secure protocol. Authors of services which use the HTTP protocol SHOULD NOT use GET based forms for the submission of sensitive data, because this will cause this data to be encoded in the Request-URI. Many existing servers, proxies, and user agents will log the request URI in some place where it might be visible to third parties. Servers can use POST-based form submission instead
Keyboard nav is much better than links (use numbered links with "G," as in "25g" takes you to - but doesn't follow - link/text entry box #25 on the screen, etc.).
So, you mean you sit there and count how many links are on a page, then figure out where on the page #25 is, and then type all that in to go to it, instead of just scrolling down and clicking or something similar? How incredibly stone-age.
- A.P.
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"