Slashdot Mirror
New Spam Frontier: Referer Logs
geoffsmith writes "Wired News is reporting that spammers are using referer logs as a cheap new way to
spam small sites. Anyone running a website has probably already seen this phenomenon; I'm thinking of writing a script to remove these entries from my access_log by looking for hits that don't grab my images. (sorry lynx users!)"
The entire internet will eventually go down in a deluge of spam unless it is made illegal and the laws are enforced!
Any way to bypass some of the obnoxious bots... Of course, will this affect the ever-dubious "Archive.org?" I think it doens't snag images, at least on the first pass.
"I'll adapt or I'll discontinue. I'm not planning on becoming the major annoyance of the blogging world.... I'm not too worried my reputation. Marketing is all about being innovative, different, adaptive, taking risks and knowing how to use the technology. I'm trying to be all that."
Heh, it's funny that this guy can make this statement and expect to be taken seriously. It's even more pathetic that he actually thinks he's "innnovative".
Spam will always find a way to evolve.
I don't know if i'm the only one, but has anyone else who doesn't filter their e-mail noticed a drop off in the amount of spam they recieve? For about the past 2 weeks, the amount of spam in my hotmail inbox has dropped from about 40 to around 15 a day. Anyone else had something similar to this happen?
"Sic Semper Tyrannosaurus Rex."
the article doesnt make it clear, how exactly is it done?
n ke ysex.com_for_Wild_Donkey_Sex!!
/Go_To_http://www.wilddonkeysex.com_for_Wild_Donke y_Sex!! HTTP/1.1" 404 5275 "-" "WildDonkeySex.com (SpamBot5000)"
http://www.yoursite.com/Go_To_http://www.wilddo
And your logs show:
255.255.255.255 - - [27/Oct/2002:00:00:00 -0400] "GET
??
(sorry lynx users)
Don't worry. It's highly unlikely that any of the 4 current users will visit your website anyway.
Well, what do you know? I just went and checked my website referral logs, and what do you know? SPAM!
Don't these people realize that by spamming me - by email, by false referrers, by pop up/under ads, etc., it virtually *gurantees* that they will never get my business?
"The natural progress of things is for liberty to yield and government to gain ground." - Thomas Jefferson
...just some blogger that was slashdotted, unwilling to believe that (s)he's really getting that many hits referred to from just one site.
He just got a link posted on /. and Wired--I wonder how many spammers are going to target him now...This seems a little aganist logic
Windows users are complaining that Microsoft is filling up their computer's System Event Log with spam about illegal exceptions and page faults.
For one thing, I only get about 2-3 legit emails a day, vs 20-30 spams.
On the other hand, I usually get a few thousand refer logs, and I *already* get a bunch of bogus refer logs from buggy browsers or something (like, a refer from a site I link to, I guess from people hitting the back button, that kind of thing).
On the other hand, I could see how it could get annoying for small sites.
The "solution" you mentioned wouldn't really work, as the spammers could simply download your images as well.
A more effective way to block these would be to scan sites in your logs and check to see if they link to you. It might take a while for huge sites, but then huge sites probably don't look through their refer logs as much.
OTOH, you would miss out on hits from sites that have random URLs or that kind of thing (like goggle's 'get lucky button')
Lonely?
Find love on the internet
I don't know who started it - but I find it very odd that browsers send referer info by default. Why? It does not provide anything extra for the user but problems. It is not once or twice that you find URLs to "confidential" pages if you browse through your webserver logs. And... I bet 95% of web surfers do not even know that they are sending this information all the time. Is there really any reason why the default is to send the referer info? I have seen people riot on much less important privacy issues. Why not about this? The referer plague exists in almost all browsers - and only in few browsers you actually can easily turn it off. What's going on?
I don't browse the blogs, but I make my own. In fact my Slashdot journal is a blog of a sort, and I haven't seen this type of spam activity, because as my journal clearly states I have a life elsewhere and don't care who looks at my journal.
Now please look at my journal and reply to me so my life has meaning.
Thank you for your time,
saskboy
Saskboy's blog is good. 9 out of 10 dentists agree.
Actually it would be quite nice to see some of these "marketing gurus" put a little more thought into their spam. Today, some of the most carefully crafted content on TV is commercials (lamentably, also some of the worst). Watch and learn. I wouldn't mind receiving a spam that is fresh, funny, engaging, and didn't involve a virgin, my cock, a septic tank, or a gentleman from Nigeria. I wouldn't mind a funny beer commercial, for instance.
"I have opinions of my own, strong opinions, but I don't always agree with them." -- George H. W. Bush
Suppose I have a form on my site to collect customer information.
Could filling this form with obviously bogus information and advertising be considered theft of services, or fraud, or something else?
I cant tell you how many times a referer log in my access log files contains someones email account pw for their web based email service... and being the ass that I am, i read their email. its quite fun.
Yeah, I havn't been getting much spam in my hotmail box anymore. But then again I did block all email into the thing...
Lonely?
Find love on the internet
...(sorry lynx users!)
Sorry about what? Why should they care wether you keep them in your log or not?
A message from the system administrator: 'I've upped my priority. Now up yours.'
No, I've noticed the opposite in my Yahoo account. I use their spamguard feature, but still, I've noticed an increase by a factor of 10x. I get ALOT of spam now.
:)
Perhaps hotmail is redirecting spam to Yahoo???
"Can of worms? The can is open... the worms are everywhere."
From the wired article:
Umm, huh? I don't think the spammers actually link to the sites, they probably just send HTTP requests with faked referrer headers that contain the URLs of the spammer's web site. That won't boost your search engine rankings.
Sig (appended to the end of comments I post, 54 chars)
Comment removed based on user account deletion
Is this damn pr0n referer.
Spam to Hotmail has decreased the past few days. I have 5 different accounts, and each one is getting about 70% less spam per day. The stuff I do get is more likely to get by my custom filters though because Hotmail has restricted the number of custom filters to only 10, unlike the previous 30something.
Saskboy's blog is good. 9 out of 10 dentists agree.
I have come across a few sites that use the refer info to let you access files and images, so other sites can't just link directly to the file.
That is one use I have seen and know of.
In the regular prefs and the "quick prefs" (F12 under Windows version) Opera lets you turn off referrer logging. The only time I need to turn it on is certain sites, like my credit union, which is no big deal...
I should put something clever here. Maybe someday.
The exact opposite for me. Spammers finaly figured out one of my new email address on my own mail server, I'm guessing through 'name combinations' (IE they know my domain, they know my name, and they spam every combination)
Lonely?
Find love on the internet
And why wouldn't you? The user is basically direct marketing his/her user credentials especially for you. Also ever wonder how these highly confidential web pages entered google. Yes, ofcourse google indexed a cool "these guys referred us page". And ofcourse the poor author of the page "for your eyes" only, did not think he would need to password protect it - because it will only be accessed by the 100 company executives (...who happily browse to pr0n sites to leave referral marks after reading the study on intranet security...)... I think I will pop!
I'm not sure I understand. Does this mean the spammers put links on their own porn (or whatever) sites, and casual surfers will click into the blog from the porn site, thus making the porn site show up in the logs as the referer? That's how the referer is supposed to work, right?
Or are they just bots that hit random web sites and send fake referers along?
Either way, I have absolutely no clue why this would be abusive or even annoying? Can someone explain? Do people sit around checking their referers all day long?? (Then again, I don't understand why anyone would run a blog, so maybe I'm just out of touch).
I clean out all my outgoing referers (thanks squid), so maybe I subconciously assume everybody else does too. Never thought of the referers as anything but a silly waste of bandwidth, since they can be forged so easily.
Haven't Microsoft started using brightmail to filter spam from hotmail?
According to MS themselves: Brightmail to Deploy Server-Side Technology on MSN Hotmail
This might be something to do with it...
Only server admins can access those logs. Since these admins are networking professionals, we can expect the same amount of discretion from them as from any other professional like your doctor or your lawyer.
...personally, I was hoping that one of the victims of the Washington, D.C., sniping spree would turn out to be a major spammer. Guess they were all humans instead.
You cannot apply a technological solution to a sociological problem. (Edwards' Law)
Yes, referrer information makes an excellent authentication scheme for highly confidential system dealing with transfer of mission critical information. ... Just also check for a magic string in the user agent and voila! trusted computing reinvented. To make it unhackable - just add a few more levels of obfuscation. ;))) The sad part of this, is that I have actually seen authentication schemes like this. Don't know whether I should cry or laugh :)
Actually, it should be o and double l. Not trool.
Hah. Since this is highly competed marketed, the first argument used when selling a 8-year old his first "web space" is "YES! Ofcourse you have full access to log files, what did you expect?!" "Don't believe us?! Let's look at this company's report as a showcase, just for you..boy...ermm. Sir."
T-r-o-o-l?
There is no excuse for poor spelling. Or as you would obviously put it, "S-p-e-e-l-i-n-g"
There are many reasons, mostly for those who program websites. Sometimes you don't want people to see a page before another. this could also be solved with cookies, but some blocks those too.
Then there is the statistics, learn how people navigate around your site. referer can help you see a pattern and improve your layout.
Also it can prevent bandwidth hogs, mostly a issue with ad. graphics and pron sites where people use graphics from others servers on html pages on their own sites but also on free servers where people place graphics and files and link to those directly without using any html and then not showing any of the free servers ad's which provides them with money to run the sites in the first place.
my sig
This isn't for authentication.
This is for people who don't want people "deep linking" to material within their sites. As an example www.gamefaqs.com allows people to link to some of the pages within their site, but not directly to the FAQs they host (which are merely sumbitted text files) by using the refer info. This stops people from bypassing the ads which pay for the site.
you're retarded, you didn't read the parent post at all, you're just looking to try and look smart by discussing a use of referral info that has NO bearing on what the parent poster said. So in summary: you're retarded, try reading instead of karma whoring.
Apache mod_rewrite (difficult for me to use because of the bizzare Unix shell syntax that I am not familiar with) does just that. I use it to keep people from stealing my images in my main web directory, but allow them to link to them freely in the Uploads directory.
Page faults happen all the time. They're probably happening to you right now. A computer without page faults is a computer without virtual memory. Page faults aren't going to reported in any System Event Log.
Actually, it should be o and double l. Not trool.
That would make it trooll.
Llooser...
It's nice, as a site operator, to know where your guests are coming from. A good portion of my visitors come from Google and other search engines. The referrer log lets me know what they were searching for, and in nearly 95% of the cases they were looking for a specific topic on my site. I can send them directly there, give them a specific welcome message if they haven't been to my site before, etc.
Furthermore I can restrict traffic for some areas of my site (like some sites that block links from slashdot) for particular reasons or uses. "You just came from the page of an associate and are able to receive a discount." "This page is restricted to users of xyz.com. Please go there first."
Lastly, it protects my image content. My images are not stellar, and yet other sites continue to use them on their pages. I can use the referrer to limit the damage done by only allowing the images to be referred by pages from my own site.
Referrer information may be annoying to you, but it's an extrememly useful tool. If taken away one restricts opportunities for the site operator to personalize and protect content on their site. Not a huge loss, but it isn't really as great a privacy issue as you seem to believe.
-Adam
Just also check for a magic string in the user agent and voila! trusted computing reinvented. To make it unhackable - just add a few more levels of obfuscation. ;))) The sad part of this, is that I have actually seen authentication schemes like this. Don't know whether I should cry or laugh :)
probably cry... what you described could easily be enforced with the DMCA.
If you use wget, watch out when using "--referer" and "--user-agent".... you just might be breaking TEH LAW!!!
mod dat sumbitch up
Incidentally, I don't know why anyone bothers with logging referrer information. The only use sounds like what the bloggers do. If you're not a blogger, why do you even care who the referrer is? Half the time it's bogus or one of your own pages.
Unlimited growth == Cancer.
That would only happen if one of those executives pt links to pr0n sites on the page.
<sarcasm>Of course, that would never happen. All executives are clean and honest.</sarcasm>
All I want is a kind word, a warm bed and unlimited power.
Unlimited growth == Cancer.
As it says in the article, some blogs have automated lists of the top referrers, so that visitors can see who links to the blog. And yes, we're talking about bots sending fake referrers.
"The question of whether a computer can think is no more interesting than that of whether a submarine can swim" -EWD
I.e. for something other than the WWW.
It is extremely useful for security purposes.
No, not the security most people are thinking of. Checking to see if the user came from FeedBack.html before executing FormMail.pl is no security, since spammers can forge any referer they want.
I'm talking about security which stops a human user who is logged in to a particular website from being tricked into performing actions they didn't authorise. For instance: I log into my server's adminsitrative area. Then, in another window, I browse someone's blog. And I click on their "search" button. As it turns out, this search button is a trap, which sends me to my own admin area with a command to delete someone's account. I'm logged in, I have a valid network address, I'm active, there's no problem. Except that fortunately my browser sends "Referer: www.blog.org" instead of "Referer: www.admin.com".
That's why referer info is useful: to prevent a user from being hijacked.
Slashdot monitor for your Mozilla sidebar or Active Desktop.
This isn't a bad way to keep other sites from "abusing" your content or bandwidth since it requires a client-side mod to get around. It, of course, does nothing against determined clients, but that is a different matter.
No, it isn't perfect, but it is one mechanism.
This is so damned annoying. If I'm searching for some specific information, I don't give a damn about your idiotic welcome page. I don't care what your website is about or what you have to say on your other pages - all I care about is the specific technical information that google told me you have.
More and more, I'm finding myself using googles cache instead of clicking on the actual links. I know you couldn't care less about my insignificant browsing habits, but the more people start doing annoying crap like this, the more people start using google instead of the web.
"This page is restricted to users of xyz.com. Please go there first."
Do you realize how stupid this is? You're trying to control how I use my browser. Of course I'm not going to go to xyz.com and try to use their idiotic navigation looking for a link to you. You're simply advocating another form of advertisement and I'm not interested. I care about the data you're providing, not how you're getting funded.
I can use the referrer to limit the damage done by only allowing the images to be referred by pages from my own site.
And this is, of course, broken behaviour. Did you know that when you open a new link in Netscape/Mozilla that the browser does not send any referer at all? This means that I can't open your images in new windows and I'm constrained to view your images one at a time. Also, the some browsers change the referer for images when you "save" images (eg, right-click and choose "Save as..." may not send the referer you're expecting).
If taken away one restricts opportunities for the site operator to personalize and protect content on their site.
If you're using this to restrict content to your site ... well, forget it. If you have something I really want, I'll open up a terminal and telnet to port 80. Yes, this is indeed effective restriction. (Quiz to see if you really know what you're doing: how would you set it up so that you know that a user has previously visited another site, with cryptographic confidence?)
As for "personalizing" content, please stop. The only times I've seen that word being used in a web context is to personalize advertising (and also restricting content because I'm not using IE, but don't get me started on that). I've never seen anyone "personalize" a site in a useful way, eg, "You're a C programmer who writes Solaris kernel modules, so you're probably not going to spring for my Herbal viagra scheme and I'm going to cut the marketing BS and give you only useful information."
Why do these "blogs" even keep logs of referer links? This is pure narcisism (and more importantly, a waste of disk space - even though disk is cheap, it's still worth more than someone else's paltry feeling of acceptance). If you're going to say something, just say it. Don't base your life around how many people like what you say. "Ohh, somebody linked to my journal, that means I'm special and I can now feel good about myself." Ahh - get a life.
I swear, "webmasters" piss me off.
I would agree with you, but for some reason the creepos at freerepublic.org love to link to my images. It's a giant, sudden bandwidth waste. Don't know why they do that, don't care, I stopped them and I needed their referer headers to do it.
There are many reasons, mostly for those who program websites.
Okay, but I expect my browser to do things for me, not for those who program websites.
Sometimes you don't want people to see a page before another. this could also be solved with cookies, but some blocks those too.
Okay, but that doesn't explain why the browser sends this information. Someone not wanting me to see something in the order I choose should not expect help from my software.
Then there is the statistics, learn how people navigate around your site. referer can help you see a pattern and improve your layout.
Okay, but that doesn't explain why the browser sends this information.
Also it can prevent bandwidth hogs, mostly a issue with ad. graphics and pron sites where people use graphics from others servers on html pages on their own sites but also on free servers where people place graphics and files and link to those directly without using any html and then not showing any of the free servers ad's which provides them with money to run the sites in the first place.
Okay, that's a reason why the web site would want this information, not why I would want to send it. If I want to follow a link directly to a web site then the last thing I want is for my browser to be actively undermining me.
For clarity, I think the original question was why the browser sends this information by default, not why people on the other end would want to receive it.
Lastly, it protects my image content. My images are not stellar, and yet other sites continue to use them on their pages. I can use the referrer to limit the damage done by only allowing the images to be referred by pages from my own site.
no it doesn't... it protects your BANDWIDTH. by keeping joesimagewhores.com from embedding your images directly in their html you protect your BANDWIDTH.. there is nothing you can do to keep me from copying your images from your site and using them in my site.. you can try the lame Java and Javascript solutions... those won't even slow down a web-user with 1/2 a brain.
so please, tell us the truth, you are protecting your bandwidth and rightfully so.
Me? I have more fun with it... I have a perl script that returns random porn if the photo is asked for from outside my site or it uses imagwmagick's mogrify to place "stolen from MEMEME.COM" in the center of the image... depending on my mood... (No I will NOT post my personal website on slashdot... I'm not about to get a huge bandwidth bill because of you guys!)
I dont care if they steal my images. I care if they try to steal my bandwidth though...
Do not look at laser with remaining good eye.
"Why? It does not provide anything extra for the user but problems."
Because it's nice for us site hosts to know where the traffic is coming from, helping us to realize just how few constituents are visiting our sites...
*mutter* Last damn time I put a URL in my sig...
Well, if you whent around putting image links to /dev/zero, I doubt many people would visit your website anyway.
And secondly, if you're making raw HTTP connections to the client (which they probably are, to fake the browser version) then there is no way to download anything that isn't on your computer. And, they don't even need to actually download it, they can just ignore whatever comes back from the pipe. As long as they make a request, it's in the logs.
Ultimately, there is no real way to tell the difference between a refer-spammer making a request and a regular user.
Lonely?
Find love on the internet
This sort of thing is why I normally have referer set to 'off' (Opera user). Also why someone proposed a third setting for Opera that I hope but doubt will be implemented: Instead of just 'on' or 'off', add an option that says 'claim site you're are visiting is the referer' -- so that no real referer info is sent, but that 'image protection' schemes get satisfied. I would use this setting if I had it available to me.
The internet is so often dealt with as if it were entirely novel. For the most part it's not, and simply complements telephone, fax, USPS, television, and so on for delivering information. (Granted, it is pretty neat.)
So at minimum the internet deserves regulatory parity with these other media. Abuse of telephones and faxes was dealt with years ago -- (albeit incompletely -- our phone rings off the hook, I'll rant another day). For some reason business was quick to push for the outright ban on junk faxes, but hasn't for email which must waste a lot of their employees' time and hassle IT, in the end costing them money. Money talks, so I which there was a more concerted effort by those businesses that would never themselves spam.
As with junk faxes (again, analogies everywhere) the injury from each incident is too small to do anything about; but we can act collectively through our government to attack the collective harm that is quite large.
I won't comment on the current political obsessions in DC on anything but domestic policy, but I hope we see something soon. I don't think state-by-state legislation will do the trick. Your opinion will count if you express it to the right people. Writing your congresspeople for one is NOT a futile activity: they carefully tally what their constituents are saying, and you will likely get at least a form letter in reply. (BTW, I think a real paper letter carries more punch than email.)
Exasperated outside DC, Andrew
Incidentally, I don't know why anyone bothers with logging referrer information.
It's good to know who is sending users to a dead link. Just by checking the referrer information for the 404 entries in your logs can determine what website is pointing users to a document that you have deleted or moved.
Not to knock your sig, which is admittedly funny, tecnicalities could lead one to conclude that regardless of Linux's price you will get zero performance. Perhaps I'm insane, but that's the order I use when converting ratios to division expressions. Peace.
I am feeling fat and sassy
I use it on my employer's site for a central feedback form that sends feedback to the proper person based upon what the refering page was
Oh, guess I still had some ranting left.
I'm sick of being modded down for being "overrated". My my anonymous post all you like.
That's not useful, that's obnoxious. If you put a link out, you should keep it alive using a redirect or whatever. If you continually expect other people to fix their links every fucking time you move shit around then forget it.
When I'm feeling bored, I'll take a look through some of the crap procmail catches, and visit a site being advertised (if it's still up). But I don't just visit once! No! I leave lynx visiting the biggest page I can find by starting a script on my server, then forgetting about it for a day or so.
;-) If you have a DSL connection at home (and you're not capped), why not use it to do some good when it would otherwise sit idle?
If only a few hundred more people started doing this - absolutely flooding these spammed Pr0n sites, and get-a-big-dick-quick scams they would have HUGE bandwidth bills, and think twice about using the same marketting technique again.
It's no use trying to email abuse depts, or reason with this scum, you have to hit them where it hurts, in the wallet. The only way to do this (for us at least) is to suck their bandwidth dry
Code, Hardware, stuff like that.
Ehh.. no it doesn't, it only stops the ones not able to do it correctly. Ie fake the referer in this case.
Glad someone said that, I was about to make the same comment. Dividing zero by a positive is not an error - the result is just zero.
Author, Shell Scripting : Expert Re
One of the primary uses I have for referrer information is locating bad or malicious links. If someone is sending large volumes of traffic to a particular page on my site, I'd like to know where that traffic is coming from. In addition, even to pages on my own site, if I see someone following a link to somewhere they either shouldn't be going or to a mistyped URL, the referrer information allows me to identify where they're coming from, and if it's a problem with my own site, it lets me correct it.
Perhaps referrer information should be released depending on the site's posted P3P privacy policies. If a site is interested in collecting information like this for marketing purposes, I can understand someone's reluctance to have their browser provide it. But for the rest of the sites (including those I maintain), the information is only ever used strictly for legitimate needs like those mentioned above. Please don't advocate that referrer information be restricted by default or for everyone, because that hampers my ability to troubleshoot problems.
The point is, if I design a site which steals your bandwidth by using your images, most people who view my page can't see the images (if you block external referers to your images) Unless that changes so that most people don't send HTTP_REFERER [sic] you're safe.
Author, Shell Scripting : Expert Re
Backlinking, or posting your referral logs, is doomed to failure and rightly so. It's just a glorified way of making your site into a link farm, with the expectation that your fellow bloggers will do the same. It is serendipitous that this practice is open to 'abuse' although I would never call the abusers spammers. They are just utilizing a method for submitting data that the site owners themselves have provided. I don't see any reason to call this 'spam' since the site owners are inviting users to submit data through HTTP referral headers.
Also, this quote from the article is ludicrous: "bloggers are not thrilled, even though they ruefully admit that the log spamming may falsely boost their ranking on some search engines."
There is no search engine that bases your rank on the number of sites that you LINK to. I believe the bloggers actually mean that they're sorry to see their backlinks (read: link farms) go, since those do in fact raise search rankings. What a travesty- Sites may have to rely on the actual quality of their content, rather than trading links!
Amidst the alarmist cries in the article, "spammers will destroy our practice of posting referral logs," nobody has even mentioned that there is a ridiculously easy technical solution. Before posting a referral link, why not just have your software visit the referring site and detemine if it actually links to your page? This will defeat the referral advertisers.
For instance, if a person goes from a Monster.com search page to his Yahoo mail account, Yahoo now knows where the person is looking for a job, what type of job he is looking for, etc. (it's all encoded in the URL). Yahoo also has access to his address book and all his email messages.
I see a scenario where Yahoo subtly threatens to email your boss to let him know you're thinking about quitting... unless you upgrade your account/add more storage space. It won't happen tomorrow, but Yahoo is sleazy enough to try something like that and they have the information... all they need is the technology to make that connection.
That's just one example, but it illustrates the point that referrer information is none of your business. You only want it because you can profit from it without any complaints from your audience.
Another example:
A lot of people apparently email the URL of my site to their friends. In my site logs, I often see the email addresses of the person who sent the message and the poor sap who clicked the link. These people have no idea they have divulged their email addresses to me via referrer info. If they wanted me to have that info, they would have given it to me. Sometimes I also see the subject of the message, which is particularly funny when it was sent by a competitor along the lines of "Have you seen what <insert_url_here> is doing?"
But as you said, "it isn't really as great a privacy issue as you seem to believe." It's worse than you realize.
Bottom Line
Companies will do just about anything to make an extra buck. So it shouldn't surprise anyone that they use technology against users to that end. But it's a two-way street -- people just need to wake up and start using technology to protect themselves.
Log only one hit per unique IP per $TIMEINTERVAL. If it's a small blog, log only one hit per small (16 hosts?) subnet per $TIMEINTERVAL.
Sure it's possible for a client or proxy to put whatever it wants in a referer string, that's the point of the article, but... Can someone use JavaScript on a web page to somehow convince the browser to fake the referer string? If not, checking the referer is pretty much guaranteed to prevent deep linking because no one is going to go around forging referer strings just in case they go to a site that does deep links.
I do not have a signature
But if there were no referer info, then there couldn't be cool tricks like the time Somethingawful.com redirected visitors from Slashdot to goatse.cx!
I found a site via Google that seemed to be using the referer data to highlight the keywords I was googling with.
I thought it was clever anyway!
I worked for a company that uses the referer as a security mechanism. I just rolled my eyes when I heard that. It seems that most of the people who do the www for a living or are on the whole blog bandwaggon are just art-school-dropout-losers who don't have the first clue about what they are doing
The only scheme for verifying the links that can't be fooled by the spammer is human moderation...
An engineer who ran for Congress. http://herbrobinson.us
I'm seein' double! Four Krusties!
there is hope! we need counter-exploitation! visit this site: http://www.cexx.org
my personal favorite is proxomitron! bye bye annoying popups!
I use Links (not Lynx) by ssh'ing into my home box from work. That way I can fulfill my text porn (slashdot & freshmeat) addiction without anyone knowing what I'm doing. Besides it is actually faster that using a browser believe it or not.
For now I'll delete the entries by hand, but if this increases it could get really annoying.
AlpineR
It really depends on what use you make of the information, doesn't it? I once wrote code for a website that made decisions (based on referer) as to whether or not it should ask you if you wanted to log in, or browse the site as an anonymous user. That was kind of cool... no javascipt, just plain-jane HTML; asked you once when you first entered the site, then left you alone thereafter. That was just about the only good use of referer I've ever seen.
Isn't that kind of like saying because someone has an email (a method for submitting data), it's okay to spam?
Plus, think of the numbers. If people are selling this 'service,' it's bound to have a negative effect on the overall quality of the web features like this offer.
J-Log: Journalism News, Media Views
>> I can use the referrer to limit the damage done by only allowing the images to be referred by pages from my own site.
> And this is, of course, broken behaviour. Did you know that when you open a new link in Netscape/Mozilla that the browser does not send any referer at all? This means that I can't open your images in new windows and I'm constrained to view your images one at a time. Also, the some browsers change the referer for images when you "save" images (eg, right-click and choose "Save as..." may not send the referer you're expecting).
You can also use the referer to not allow save as... from popular web browsers... If I didn't refer this IMG, then I guess it can't be downloaded. That may annoy you, but I think it's just another way to annoy the copy-cats.
And this is, of course, broken behaviour.
So do you have an alternative proposal to prevent resource (i.e. bandwidth) theft? That is a very real problem, and no amount of arguing that the current solution is "broken" will get people to change unless you provide them an alternative.
...I could give a good g*dd*mn about a "blog".
Spread the RC luvin'
My idea was to have a way to be able to construct backlinks from sites. At the time we had 100 users and the operating assumption was that all information put on the Web was public.
I did not write the code that implemented referer. However the security note I wrote did say that you should only send referer if you were actually following a link. The NCSA folk introduced the idea that it was a link to the 'last think you visited' which I consider to be buggy, there is no reason to reveal file: and bookmark: URLs.
We considered the privacy implications. Basically if someone wants to they can pass the linkage info explicitly via a query suffix.
There should be a toggle in my view, but folk seem to take a wierd view on security and privacy. Argument by analogy was the rule at the time. We can do BASIC password security because FTP does, DIGEST was written less than a week later, no interest of course because it was not compatible with the then rulling UNIX dogma of one way encrypted passwords, forget the fact that sending the password en-clair is a bigger problem. You can't have the password encrypted both on the wire and in storage without using public key which was patent encumbered at the time.
Oh yes, the mispelling was me. I am dyslexic. However we are petitioning to fix it. The next edition of the OED has the additional spelling referer which specifically describes my HTTP header. I am trying to get into the Guiness book of records first with the most serious spelling mistake ever.
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/
I invented the referer field before the IMG tag was proposed. So no, that application was not one that was ever considered. Nor is it the way I would have solved the problem.
I always thought the way IMG works somewhat broken but I happened to be asleep for the 8 hours it was up for review.
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/
You can also see which sites are using the images from your site on their own sites...which can sometimes be annoying. Particularly if you are running a college's web site and you have an image of a cheerleader somewhere that is getting linked to by every other cheerleader perv site out there. That's pretty damned annoying. Not that it has happened to my site or anything...
they will come... and rip it to shreds as fast as they can in any way possible.
It's the same deal if you have any kind of script that can be compromised. Example: FormMail.pl, if it didn't do strong checking someone could use it maliciously. There are a few ways to combat this, like setting a repetition checker so that if within n seconds if the same thing comes in m times ignore and remove it and/or ignore the ip address(es) it's coming from. You can also set it so it will only work for trusted people and you could have do some small monitoring to make sure none of the trusted people are flooding it. There are many ways to go about preventing the spammers from getting through you just have to think practically (ie: What do spammers do that would be different from your regular users) and do a little coding and your done... They obviously could care less about you, so there's no other way to really deal with them.
If you're using this to restrict content to your site ... well, forget it. If you have something I really want, I'll open up a terminal and telnet to port 80.
:)
You're right, it's not security. It's not a matter of security - it's a matter of convenience. How many people even know how to point a telnet seesion - and are they even in your web site's intended audience?
There are only two people I know of who actually do this. You don't strike me as someone who would EVER have a reason to visit any of my web sites, and the other person... well, the other one is me.
Now, that said, why should I design my sites for you instead for my users? They get some real usefulness out these techniques.
Websites are more than just file farms, and on many occasions referers are extremely useful in delivering the services many of them provide. If you can't imagine what possible use any services like that might offer, I have plenty of neat ideas. Just don't pretend your tastes are good enough for everyone - after all, you've already proven yourself technically exceptional by your use of telnet (believe it or not). I'm not trying to stop one person from doing certain things. I'm trying to stop a LOT of people. and, well, that works.
I agree about Google's cache, though. Gcache has proven invaluable to me on numerous occasions. I've even thought about POINTING people there FROM my website on occasion. great. go hard. That's what it's there for. It's not, however, what MY site is for.
If everyone only used technology the way it was intended, nothing would get invented at all.
Did you know that when you open a new link in Netscape/Mozilla that the browser does not send any referer at all?
Either you didn't understand the question, or you're just plain wrong. Netscape/Mozilla (along with every other browser I know of) will almost always send as referrer for an image request the address of the page with the image embedded. To make sure I wasn't talking out my butt, I just tested this in Mozilla 1.2beta, and I'm right.
It won't send a referrer for just an html page you open without clicking on a link of course, but the dude you're trying to make feel stupid wasn't talking about that, he was talking about people embedding his images. So you instead made yourself look stupid.
PS: You're a moron. Quit complaining about how noone respects how you use your browser and find a girlfriend.
You're not useful, you're obnoxious.
24 6 0.02% http://www.free-casino-gambling-online.com
I read the article, go check mine and sure enough 1 is there.
Jackass, he doesn't mean that it protects his image from somebody saving it or using it on their own site. He means it makes it so people won't put on their site, thus burning his bandwidth without giving him a visitor.
The only way to do it right is to generate pages on the fly, with all URLs in it being re-written to be cryptographically signed and timestamped. A link would look something like
The web server checks the signature and lifetime of every request before serving up the file. (I implemented Apache modules etc. to do all this at my job.)Unlimited growth == Cancer.
You are forgiven :) No, seriously the world is different now. Is W3 the organisation which could try to push it through? Or do we just have to believe that the browser vendors realize it in time before every site utilizes this in their inner logic and the change to better is thus impossible.
Where it == a change to how referrer information is sent. As there clearly is some benefits for the website developer/author for having the referrer info the situation after change could be an analogy to what is used with cookies :: the referrer info would be only sent if 1) the user is following a link (or similar mechanism) and 2) the link being followed resides in the same space.
Examples:
link from www.xyzzy.com/index.html to www.xyzzy.com/about.html - sent
link from www.xyzzy.com/index.html to www.othersite.com/ - not sent
link from www.xyzzy.com/~jukal/categories.html to www.xyzzy.com/~jukal/contents.html - sent
link from www.xyzzy.com/~jukal/ to www.xyzzy.com/~abuser/ - not sent
.
As you Zeinfeld, clearly are in the position to make a difference (being the "inventor" - or one of the inventors - of the mechanism) - what do you think about this? Do you know if w3.org for example is already considering this? If not, who should I, You, everyone else talk to?
I wonder if there is a vulnerability in here somewhere... people are displaying raw referrers on their sites, typically via a server script of some sort. Potential breeding ground for a new worm of some sort?
On the other hand, perhaps this is the first valuable use of spam: making people aware of the problem, and the smarter people shutting it down, before someone writes a worm to exploit it.
Read reviews of shopping cart software
15.1.3 Encoding Sensitive Information in URI's Because the source of a link might be private information or might reveal an otherwise private information source, it is strongly recommended that the user be able to select whether or not the Referer field is sent. For example, a browser client could have a toggle switch for browsing openly/anonymously, which would respectively enable/disable the sending of Referer and From information. Clients SHOULD NOT include a Referer header field in a (non-secure) HTTP request if the referring page was transferred with a secure protocol. Authors of services which use the HTTP protocol SHOULD NOT use GET based forms for the submission of sensitive data, because this will cause this data to be encoded in the Request-URI. Many existing servers, proxies, and user agents will log the request URI in some place where it might be visible to third parties. Servers can use POST-based form submission instead
How lame! Rather than blocking those offiste image references, just replace them by redirects to the goatse picture. Much more funny that way, and gets the point accross really quick...
Unless he got to his yahoo mail account by clicking on a link that he found at moster.com (highly unlikely), this won't happen. On most of today's browsers, if you enter an URL manually, or if you use your bookmarks, the referer field will be empty, rather than containing whatever page happened to be displayed in the browser window. It's called referer for a reason.
Some old versions of netscape sometimes did funny things with the referer, but who continues using netscape 3.01 nowadays?
Say no to software patents.
I have a Hotmail account that I registered with the intent of using for personal emails (it was my first and last name, big mistake), but I started receiving spam without even using the account. Ever.
;)
I would get anywhere from 25-35 unsolicited emails a day to this account... Then it filled up, around the 17th of this month. After I cleared everything out, I noticed that I was only receiving 3-8 spams a day.
I still haven't used the account, but I log in twice a day just so I can keep a manual record of the "spam bombardment", which I realize is nothing compared to most of you out there.
People will pass up steak once a week, for crap every day.
How many people even know how to point a telnet seesion - and are they even in your web site's intended audience?
I believe this kind of reasoning is a bit dangerous - it's very difficult to gauge your audience. You can, of course, gauge your intended audience, but you're always going to get some people in there that don't fit the demographic.
Example: a couple of years ago I wrote a bit of technical documentation for a system we were using internally in my department. I also linked it in from a couple of places, but I didn't restrict it to our local network since it wasn't any kind of secret.
I was quite surprised at some of the responses I got from that page. I, of course, didn't log referers, but I occasionally received mail from random people who found the site via a search engine. One mail was from a guy in India who needed some help with a technical problem and he found our site, and another mail was from an older gentleman in Japan that was not technically literate but found the site via an unexpected keyword search and was actually trying to research a literary text. The pages were written for our own users, but ended up being read by some quite unexpected people (and I thought this was pretty cool). All of our users had access to the latest browsers with all the gizmos, but if I had made that page using CSS2, DHTML and the latest javascript tricks, it may have not been accessible to some of the unintended readers.
Point being, although I'm not your intended audience, I might read some of your sites some day, so don't piss me off :)
"To make it unhackable - just add a few more levels of obfuscation. ;))) "
Obfuscation != security
now die PHP user
Except that any decent system should ask is you're sure about deleting that account. And while it does that is should give some nice random text as a hidden field and expect it when submitted before deleting the account.
Also I think the odds of having this sort of trap are minimal. And you should always be able to undelete accounts...
- Raynet --> .
You are correct in that the latest mozilla sends referers for images opened in new tabs/windows. But you're still talking out your butt.
1. Many linux builds (debs and redhat rpms) of current stable builds do not correctly send referers.
2. "Save as" may not send correct referers. It should save from cache, but instead it requests a header. This is a known bug.
3. Opening an image via "View image source" and the "View Page Info" dialog box will sometimes fail because it does send the correct referer. There are bugs filed for these, and they are being worked on.
4. Reloading an image may send an incorrect referer. This leads to all kinds of errors. Bugs are filed and this is being worked on.
5. Loading large image files may result in error messages, as image requests are being sent twice, once from the cache and once from layout(?)--not sure about the details, but there are bug reports and the problem is being worked on. There should be a solution in 1.2b, but I don't believe that it's been confirmed. Presumably the next stable release will have this fixed.
I kind of agree with the point that webmasters shouldn't have to compensate for the shortcomings of this or that browser, or the fancies of this or that user. However, it seems obvious that the use of referers to block access to images is kludgy and not in the spirit of the www as it was originally designed and promulgated. The parent comment is on target.
As for someones right to see where you come from, yes, you're right. Which is why it is up to you whether or not you use a client that allow you to turn the referrer header off or fake it. But on the other hand, it is up to the webmaster of the site you're trying to visit whether he'll then decide to prevent you from accessing his site.
Unsurprisingly, bloggers are not thrilled, even though they ruefully admit that the log spamming may falsely boost their ranking on some search engines.
So how is this, exactly? Search engines (think Google) may boost pages that are heavily linked to, but sending false referers to the website does *not* affect Google's rankings in any way.
Google goes by how many pages in its index contain a link to a site. It doesn't care what is in the site's logs, it wouldn't have any way to know this.
Just an observation...
NGWave - Fast Sound Editor for Windows
by Permission Denied on Sunday October 27, @07:02PM (Score:4) (#4543977) (User #551645 Info) [ Neutral ]
You are a Troll Permission Denied
It's not the size of your .sig that matters, it's how you use it.
Keyboard nav is much better than links (use numbered links with "G," as in "25g" takes you to - but doesn't follow - link/text entry box #25 on the screen, etc.).
So, you mean you sit there and count how many links are on a page, then figure out where on the page #25 is, and then type all that in to go to it, instead of just scrolling down and clicking or something similar? How incredibly stone-age.
- A.P.
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
> No I will NOT post my personal website on slashdot... I'm not about > to get a huge bandwidth bill because of you guys!)
:-)
Chicken.
Bugs Bunny was right.
You've to see things a little bit more from the average user's perspective.
When searching for some specific technical information page you may be right, but most people usually don't look for Solaris kernel modules (at least I assume that). And I can imagine lots of people are thankful if company XYZ presents related links and personalization that is of use for these people.
And on the other hand, I can't tell a customer:
Hey, don't know if your banner/marketing campagin was successful, we don't track those referers!
Why does this bother you so much? You've turned off your referrer, right? My site treats you like you appeared out of the blue sky, and you have access to everything that any other surfer does, without any "annoying", "stupid", "broken" fluff.
Do you honestly think that Yahoo is going to extort you based on a referrer log? You've got some pretty far fetched ideas there. Do you also use an anonymizer service? If not, then I've gained a ton more information about you than I could with the referrer log. All the referrer log tells me is that another web site has a link to my site, and that you clicked on it. Monster.com has no links to my site, except to my full resume - and it's fun, though not particularily useful, to see that there are people going to my resume page from monster. It saves me the trouble of creating several different links so I can track where people come from.
In other words, you are really stretching the point you're trying to make. Yes, it's technically possible that the referrer could be used in a way that makes your life less private. YOU have control over that, though, since you can turn it off. Saying that it should be off by default without providing some real, tangible benefit is shortsighted.
I'm sorry that 'webmasters' piss you off. I don't have a welcome page, the most I do is provide an extra line of text at the top of the page giving additional info to those who go there from specific sites and links. Instead of blocking my images I've decided to simply get rid of them - they don't really add anything to the content except where the content is the image, and I've lowered their size as much as possible. Speed of serving is more important than annoying the few idiots who do refer directly to the images on my site.
In short, like "rm", it's a tool. It can be used for good and bad. You either visit sites you don't trust frequently, or you are paranoid enough to leave it off all the time. I can understand that, and I agree that it's probably the best thing for you to do.
It is not a tool of pure evil though, and I'm afraid you've become somewhat like the RIAA in your argument. You assume that either (1) it can only be used for evil/bad/annoying/stupid purposes or (2) most people use it that way, and the few that use it for good can be as effective at delivering useful content without it.
-Adam
Silly me, I forgot that a proxy might do this automatically.
Very interesting to modify each outgoing URL to contain a time-stamp. Even better to add some authentication code to it. But very computationally expensive, no?
I guess if I had content that was that popular, where I suspected a lot of deep linking, I would probably go for a required registration or cookie-based scheme (as even requests for graphics return cookies, right?) before looking for more complicated solutions.
I do not have a signature
Putting ads in the User-Agent field for webmasters browsing their access_logs or server stats?
89% MSIE
8% Mozilla
2% Lynx
1% Visit My Porn Site! http://....
So much for my current User-Agent string of Mozilla/8.0 (compatible; This is the only ad-free space left on the internet)
Caveat Emptor is not a business model.
Check out http://www.spywareinfo.com/harvest_project/
They've got a project going to contaminate spammers' address databases with junk addresses. I set it up on http://www.samiam.com. It seems to me that every little bit helps.
On the dynamic HTML generation side, most app. server based sites have to do that anyway; there's just one additional step of replacing some URLs with signed ones. If you handle web sessions through URL-rewriting rather than some hack like HTTP Basic Auth or cookies (ugh!) you pretty much get it for free.
Unlimited growth == Cancer.
Isn't that the whole point of blogs? To see who is reading you? and I'm sure that most of the blogging people are preteen/teen kids so: "Ohh, somebody linked to my journal, that means I'm special and I can now feel good about myself.", shallow yes, but most likely true.
"It's not like your minds are as open as the source you love..." - Me to the majority of Slashdot.
The HTTP spec is not owned by the IETF. I have no intention to work on it in the near future, I am currently working on Web Services security.
The cookies model is the wrong one. We want to track across sites. It is important for the maintainer of CNN to be able to find out if the BBC has linked to their story.
The places where I would make changes is in the privacy area. The mechanism should be optional and be disabled by a switch. Referer links should never reveal the existence of a private document such as a bookmarks file.
However in terms of priorities I would put making popup windows optional much higher on the list. It should be possible to disable Javascript and Macromedia on a per site basis. IE is almost there in the later editions. I have killed a lot of popup ads by simply nominating their zone as being not authorized to run javascript. Jscript and Active-X should be managed the way images once were, the text would load and then you would press a button to load the images if you wanted them.
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/
If you are running any kind of web-based interface
' >
that can do dangerous things and normally requires
a password to login (such as webmin), referers
are very useful for protecting you from malicious
links.
For example, if you had webmin on localhost and
visited a web page with HTML like
<img src='http://localhost:10000/proc/run.cgi?cmd=rm+*
webmin will check the referer and save your files
from deletion. Without referer headers, there is
no way to tell the difference between the user
intentionally doing something, and the browser
being fooled into opening some dangerous URL!
Do you really believe that all links created to your site will be 100% correct at that time?
The world is full of bozos who make typos. Should we ignore them because of that? If your ecomm site was mentioned at NYT.com but they fubarred the URL would you say "oh well", or try desperately to get them to fix their page before the traffic drops off, or simply do something on your end to redirect the bad-url to the intended destination?
It would be maybe nice to have possibility in browsers to set whether referer should be actually sent and what should be sent inside. In fact, I know only about ELinks and Links now having this - you're free to set referer to "fake referer" (some string you're going to write down there is going to be sent), "normal referer" or referer containing URL of the page being loaded ("self-pointing referer" ;-). The "self-pointing referer" is set as default, and it helps workarounding most of the "protection" mechanisms, while effectively disclosing no possibly private information. Would be nice to see this in other browsers as well..
It's not the fall that kills you. It's the sudden stop at the end. -Douglas Adams
I think that perhaps you should investigate the workings of a local HTTP proxy called WebWasher (http://www.webwasher.de/). It has a feature for dealing with referer, and one of the options implements your suggestion, and it also allows content-filtering for things like Flash, JavaScript, etc., as well as wholesale domain-based access-filtering. It can also filter pop-ups, and I use it alongside Mozilla, which contains it's own pop-up, cookie, and image-filtering features.
Together, I feel that they make my internet "surfing" experience unbeatable, and safe, something it should always have been, before MS decided to invade.
They keep logs so they can see if anyone is looking. But i agree all that personalization stuff is stupid. People come to the web for information. If we asked your name, email address etc. everytime you walked into a real store you would never come back.
-------------------------------------
Technically, we are beyond survival.