Slashdot Mirror
New Spam Frontier: Referer Logs
geoffsmith writes "Wired News is reporting that spammers are using referer logs as a cheap new way to
spam small sites. Anyone running a website has probably already seen this phenomenon; I'm thinking of writing a script to remove these entries from my access_log by looking for hits that don't grab my images. (sorry lynx users!)"
The entire internet will eventually go down in a deluge of spam unless it is made illegal and the laws are enforced!
"I'll adapt or I'll discontinue. I'm not planning on becoming the major annoyance of the blogging world.... I'm not too worried my reputation. Marketing is all about being innovative, different, adaptive, taking risks and knowing how to use the technology. I'm trying to be all that."
Heh, it's funny that this guy can make this statement and expect to be taken seriously. It's even more pathetic that he actually thinks he's "innnovative".
I don't know if i'm the only one, but has anyone else who doesn't filter their e-mail noticed a drop off in the amount of spam they recieve? For about the past 2 weeks, the amount of spam in my hotmail inbox has dropped from about 40 to around 15 a day. Anyone else had something similar to this happen?
"Sic Semper Tyrannosaurus Rex."
(sorry lynx users)
Don't worry. It's highly unlikely that any of the 4 current users will visit your website anyway.
...just some blogger that was slashdotted, unwilling to believe that (s)he's really getting that many hits referred to from just one site.
He just got a link posted on /. and Wired--I wonder how many spammers are going to target him now...This seems a little aganist logic
I think their logic is that there are some stupid idiots that will fall for the tricks. They send spam to everyone they can, looking for those few...
Windows users are complaining that Microsoft is filling up their computer's System Event Log with spam about illegal exceptions and page faults.
For one thing, I only get about 2-3 legit emails a day, vs 20-30 spams.
On the other hand, I usually get a few thousand refer logs, and I *already* get a bunch of bogus refer logs from buggy browsers or something (like, a refer from a site I link to, I guess from people hitting the back button, that kind of thing).
On the other hand, I could see how it could get annoying for small sites.
The "solution" you mentioned wouldn't really work, as the spammers could simply download your images as well.
A more effective way to block these would be to scan sites in your logs and check to see if they link to you. It might take a while for huge sites, but then huge sites probably don't look through their refer logs as much.
OTOH, you would miss out on hits from sites that have random URLs or that kind of thing (like goggle's 'get lucky button')
Lonely?
Find love on the internet
I don't know who started it - but I find it very odd that browsers send referer info by default. Why? It does not provide anything extra for the user but problems. It is not once or twice that you find URLs to "confidential" pages if you browse through your webserver logs. And... I bet 95% of web surfers do not even know that they are sending this information all the time. Is there really any reason why the default is to send the referer info? I have seen people riot on much less important privacy issues. Why not about this? The referer plague exists in almost all browsers - and only in few browsers you actually can easily turn it off. What's going on?
Actually it would be quite nice to see some of these "marketing gurus" put a little more thought into their spam. Today, some of the most carefully crafted content on TV is commercials (lamentably, also some of the worst). Watch and learn. I wouldn't mind receiving a spam that is fresh, funny, engaging, and didn't involve a virgin, my cock, a septic tank, or a gentleman from Nigeria. I wouldn't mind a funny beer commercial, for instance.
"I have opinions of my own, strong opinions, but I don't always agree with them." -- George H. W. Bush
...(sorry lynx users!)
Sorry about what? Why should they care wether you keep them in your log or not?
A message from the system administrator: 'I've upped my priority. Now up yours.'
From the wired article:
Umm, huh? I don't think the spammers actually link to the sites, they probably just send HTTP requests with faked referrer headers that contain the URLs of the spammer's web site. That won't boost your search engine rankings.
Sig (appended to the end of comments I post, 54 chars)
In the regular prefs and the "quick prefs" (F12 under Windows version) Opera lets you turn off referrer logging. The only time I need to turn it on is certain sites, like my credit union, which is no big deal...
I should put something clever here. Maybe someday.
255.255.255.255 - - [27/Oct/2002:00:00:00 -0000] "GET /perfectly/valid/page/at/yoursite.html" 200 2467 "http://www.wilddonkeysex.com_for_Wild_Donkey_Sex/ " "(SpamBot5000)"
and then people looking at the report would say, "hey, the page at wilddonkeysex links to my perfectly/valid/page and it's getting like 500 hits a day from there, woo! let's click on that url and see what the link to my page looks like!"
-calyxa
Decay! Decay! Decay! -Helium
And why wouldn't you? The user is basically direct marketing his/her user credentials especially for you. Also ever wonder how these highly confidential web pages entered google. Yes, ofcourse google indexed a cool "these guys referred us page". And ofcourse the poor author of the page "for your eyes" only, did not think he would need to password protect it - because it will only be accessed by the 100 company executives (...who happily browse to pr0n sites to leave referral marks after reading the study on intranet security...)... I think I will pop!
I'm not sure I understand. Does this mean the spammers put links on their own porn (or whatever) sites, and casual surfers will click into the blog from the porn site, thus making the porn site show up in the logs as the referer? That's how the referer is supposed to work, right?
Or are they just bots that hit random web sites and send fake referers along?
Either way, I have absolutely no clue why this would be abusive or even annoying? Can someone explain? Do people sit around checking their referers all day long?? (Then again, I don't understand why anyone would run a blog, so maybe I'm just out of touch).
I clean out all my outgoing referers (thanks squid), so maybe I subconciously assume everybody else does too. Never thought of the referers as anything but a silly waste of bandwidth, since they can be forged so easily.
Haven't Microsoft started using brightmail to filter spam from hotmail?
According to MS themselves: Brightmail to Deploy Server-Side Technology on MSN Hotmail
This might be something to do with it...
[Wishful thinking mode ON!]
This implies that there are, maybe, all of 10,000 suckers who keep every spammer on the planet in business. If we find them and cut them off, spam response would drop to about 1 per billion and there's just no way they could make any money off of that.
Dyolf Knip
Yes, referrer information makes an excellent authentication scheme for highly confidential system dealing with transfer of mission critical information. ... Just also check for a magic string in the user agent and voila! trusted computing reinvented. To make it unhackable - just add a few more levels of obfuscation. ;))) The sad part of this, is that I have actually seen authentication schemes like this. Don't know whether I should cry or laugh :)
Hah. Since this is highly competed marketed, the first argument used when selling a 8-year old his first "web space" is "YES! Ofcourse you have full access to log files, what did you expect?!" "Don't believe us?! Let's look at this company's report as a showcase, just for you..boy...ermm. Sir."
There are many reasons, mostly for those who program websites. Sometimes you don't want people to see a page before another. this could also be solved with cookies, but some blocks those too.
Then there is the statistics, learn how people navigate around your site. referer can help you see a pattern and improve your layout.
Also it can prevent bandwidth hogs, mostly a issue with ad. graphics and pron sites where people use graphics from others servers on html pages on their own sites but also on free servers where people place graphics and files and link to those directly without using any html and then not showing any of the free servers ad's which provides them with money to run the sites in the first place.
my sig
This isn't for authentication.
This is for people who don't want people "deep linking" to material within their sites. As an example www.gamefaqs.com allows people to link to some of the pages within their site, but not directly to the FAQs they host (which are merely sumbitted text files) by using the refer info. This stops people from bypassing the ads which pay for the site.
Apache mod_rewrite (difficult for me to use because of the bizzare Unix shell syntax that I am not familiar with) does just that. I use it to keep people from stealing my images in my main web directory, but allow them to link to them freely in the Uploads directory.
I think it's more than the web site's owner clicking on the page - a lot of bloggers post a list of "top referrers" on their web site as a way of thanking the referrers, and therefore they generate a lot of traffic to their referrers from their own visitors.
I know why this problem is endemic. It's certainly down to more than the "10,000 suckers" you suggest.
I always use the example of my father, who is your archetypical pre-UNIX geek. He did all the PDP-11 stuff, worked with the VAXes and hacked machine code in ways that I don't yet understand -- an intensely intelligent man. Yet, every few months when I go to visit him, we get to talking about the internet and the first thing he does is talk about what he's bought online. For him, paying spammers is part and parcel of buying online -- he's paid spammers for search engine placings for his personal site, silly trinkets like water pumps and gardening tools and books.
To people who aren't part of the current 'geek' cognoscenti, spam is just another form of valid advertising, like the leaflets they get in the post and the billboards they walk past on their way to work. This isn't a specific group of people -- you can't "find them and cut them off" -- you need to target the problem at its source.
It's nice, as a site operator, to know where your guests are coming from. A good portion of my visitors come from Google and other search engines. The referrer log lets me know what they were searching for, and in nearly 95% of the cases they were looking for a specific topic on my site. I can send them directly there, give them a specific welcome message if they haven't been to my site before, etc.
Furthermore I can restrict traffic for some areas of my site (like some sites that block links from slashdot) for particular reasons or uses. "You just came from the page of an associate and are able to receive a discount." "This page is restricted to users of xyz.com. Please go there first."
Lastly, it protects my image content. My images are not stellar, and yet other sites continue to use them on their pages. I can use the referrer to limit the damage done by only allowing the images to be referred by pages from my own site.
Referrer information may be annoying to you, but it's an extrememly useful tool. If taken away one restricts opportunities for the site operator to personalize and protect content on their site. Not a huge loss, but it isn't really as great a privacy issue as you seem to believe.
-Adam
Just also check for a magic string in the user agent and voila! trusted computing reinvented. To make it unhackable - just add a few more levels of obfuscation. ;))) The sad part of this, is that I have actually seen authentication schemes like this. Don't know whether I should cry or laugh :)
probably cry... what you described could easily be enforced with the DMCA.
If you use wget, watch out when using "--referer" and "--user-agent".... you just might be breaking TEH LAW!!!
Incidentally, I don't know why anyone bothers with logging referrer information. The only use sounds like what the bloggers do. If you're not a blogger, why do you even care who the referrer is? Half the time it's bogus or one of your own pages.
Unlimited growth == Cancer.
Unlimited growth == Cancer.
As it says in the article, some blogs have automated lists of the top referrers, so that visitors can see who links to the blog. And yes, we're talking about bots sending fake referrers.
"The question of whether a computer can think is no more interesting than that of whether a submarine can swim" -EWD
I.e. for something other than the WWW.
It is extremely useful for security purposes.
No, not the security most people are thinking of. Checking to see if the user came from FeedBack.html before executing FormMail.pl is no security, since spammers can forge any referer they want.
I'm talking about security which stops a human user who is logged in to a particular website from being tricked into performing actions they didn't authorise. For instance: I log into my server's adminsitrative area. Then, in another window, I browse someone's blog. And I click on their "search" button. As it turns out, this search button is a trap, which sends me to my own admin area with a command to delete someone's account. I'm logged in, I have a valid network address, I'm active, there's no problem. Except that fortunately my browser sends "Referer: www.blog.org" instead of "Referer: www.admin.com".
That's why referer info is useful: to prevent a user from being hijacked.
Slashdot monitor for your Mozilla sidebar or Active Desktop.
This isn't a bad way to keep other sites from "abusing" your content or bandwidth since it requires a client-side mod to get around. It, of course, does nothing against determined clients, but that is a different matter.
No, it isn't perfect, but it is one mechanism.
This is so damned annoying. If I'm searching for some specific information, I don't give a damn about your idiotic welcome page. I don't care what your website is about or what you have to say on your other pages - all I care about is the specific technical information that google told me you have.
More and more, I'm finding myself using googles cache instead of clicking on the actual links. I know you couldn't care less about my insignificant browsing habits, but the more people start doing annoying crap like this, the more people start using google instead of the web.
"This page is restricted to users of xyz.com. Please go there first."
Do you realize how stupid this is? You're trying to control how I use my browser. Of course I'm not going to go to xyz.com and try to use their idiotic navigation looking for a link to you. You're simply advocating another form of advertisement and I'm not interested. I care about the data you're providing, not how you're getting funded.
I can use the referrer to limit the damage done by only allowing the images to be referred by pages from my own site.
And this is, of course, broken behaviour. Did you know that when you open a new link in Netscape/Mozilla that the browser does not send any referer at all? This means that I can't open your images in new windows and I'm constrained to view your images one at a time. Also, the some browsers change the referer for images when you "save" images (eg, right-click and choose "Save as..." may not send the referer you're expecting).
If taken away one restricts opportunities for the site operator to personalize and protect content on their site.
If you're using this to restrict content to your site ... well, forget it. If you have something I really want, I'll open up a terminal and telnet to port 80. Yes, this is indeed effective restriction. (Quiz to see if you really know what you're doing: how would you set it up so that you know that a user has previously visited another site, with cryptographic confidence?)
As for "personalizing" content, please stop. The only times I've seen that word being used in a web context is to personalize advertising (and also restricting content because I'm not using IE, but don't get me started on that). I've never seen anyone "personalize" a site in a useful way, eg, "You're a C programmer who writes Solaris kernel modules, so you're probably not going to spring for my Herbal viagra scheme and I'm going to cut the marketing BS and give you only useful information."
Why do these "blogs" even keep logs of referer links? This is pure narcisism (and more importantly, a waste of disk space - even though disk is cheap, it's still worth more than someone else's paltry feeling of acceptance). If you're going to say something, just say it. Don't base your life around how many people like what you say. "Ohh, somebody linked to my journal, that means I'm special and I can now feel good about myself." Ahh - get a life.
I swear, "webmasters" piss me off.
I would agree with you, but for some reason the creepos at freerepublic.org love to link to my images. It's a giant, sudden bandwidth waste. Don't know why they do that, don't care, I stopped them and I needed their referer headers to do it.
People don't have to visit the "victim" site at all, and they certainly don't have to browse the stats. The stats programs and search engine spiders will take care of everything. Got a low-ranking, poor traffic site that nobody links to? No problem, you can have 1,000 people linking to you by the end of the week, whether they know it or not. This really is nothing new, and the spamming side of it (i.e. repeatedly hammering a site) reminds me of how most TopSites work. These have been around forever, and so have the many methods of tricking them.
Placing your URL as the referer to sites with public stats can be quite helpful in boosting your rank, and a slightly hacked copy of wget or w3mir can make it an easy task. I guess the only real "news" here is that, once again, a few village idiots have failed to realize that some things are only good in moderation. There's neither a need nor an excuse to log yourself as a referer to any particular site more than once a month; and hundreds or thousands of times in a day is just plain stupid.
Shaun
Thanks to the War on Drugs, it's easier to buy meth than it is to buy cold medicine!
Lastly, it protects my image content. My images are not stellar, and yet other sites continue to use them on their pages. I can use the referrer to limit the damage done by only allowing the images to be referred by pages from my own site.
no it doesn't... it protects your BANDWIDTH. by keeping joesimagewhores.com from embedding your images directly in their html you protect your BANDWIDTH.. there is nothing you can do to keep me from copying your images from your site and using them in my site.. you can try the lame Java and Javascript solutions... those won't even slow down a web-user with 1/2 a brain.
so please, tell us the truth, you are protecting your bandwidth and rightfully so.
Me? I have more fun with it... I have a perl script that returns random porn if the photo is asked for from outside my site or it uses imagwmagick's mogrify to place "stolen from MEMEME.COM" in the center of the image... depending on my mood... (No I will NOT post my personal website on slashdot... I'm not about to get a huge bandwidth bill because of you guys!)
I dont care if they steal my images. I care if they try to steal my bandwidth though...
Do not look at laser with remaining good eye.
"Why? It does not provide anything extra for the user but problems."
Because it's nice for us site hosts to know where the traffic is coming from, helping us to realize just how few constituents are visiting our sites...
*mutter* Last damn time I put a URL in my sig...
The internet is so often dealt with as if it were entirely novel. For the most part it's not, and simply complements telephone, fax, USPS, television, and so on for delivering information. (Granted, it is pretty neat.)
So at minimum the internet deserves regulatory parity with these other media. Abuse of telephones and faxes was dealt with years ago -- (albeit incompletely -- our phone rings off the hook, I'll rant another day). For some reason business was quick to push for the outright ban on junk faxes, but hasn't for email which must waste a lot of their employees' time and hassle IT, in the end costing them money. Money talks, so I which there was a more concerted effort by those businesses that would never themselves spam.
As with junk faxes (again, analogies everywhere) the injury from each incident is too small to do anything about; but we can act collectively through our government to attack the collective harm that is quite large.
I won't comment on the current political obsessions in DC on anything but domestic policy, but I hope we see something soon. I don't think state-by-state legislation will do the trick. Your opinion will count if you express it to the right people. Writing your congresspeople for one is NOT a futile activity: they carefully tally what their constituents are saying, and you will likely get at least a form letter in reply. (BTW, I think a real paper letter carries more punch than email.)
Exasperated outside DC, Andrew
I actually bought something from a spam. It was a slightly topical T-Shirt that I thought was clever. Cost me $15 (PayPal).
The guy who sold it to me was obviouly a late teen, and was making ok money selling shirts at about $5 profit per when I called him.
I think most geeks have no problem with spam itself (in fact targeted spams that interest me often get clicks, I get about two of those a year), they have a problem with the number of scams that are sent using spam.
I live in a giant bucket.
Page faults happen all the time. They're probably happening to you right now. A computer without page faults is a computer without virtual memory. Page faults aren't going to reported in any System Event Log. you're right... my mistake. I meant Global Protection Faults !
Incidentally, I don't know why anyone bothers with logging referrer information.
It's good to know who is sending users to a dead link. Just by checking the referrer information for the 404 entries in your logs can determine what website is pointing users to a document that you have deleted or moved.
That's not useful, that's obnoxious. If you put a link out, you should keep it alive using a redirect or whatever. If you continually expect other people to fix their links every fucking time you move shit around then forget it.
When I'm feeling bored, I'll take a look through some of the crap procmail catches, and visit a site being advertised (if it's still up). But I don't just visit once! No! I leave lynx visiting the biggest page I can find by starting a script on my server, then forgetting about it for a day or so.
;-) If you have a DSL connection at home (and you're not capped), why not use it to do some good when it would otherwise sit idle?
If only a few hundred more people started doing this - absolutely flooding these spammed Pr0n sites, and get-a-big-dick-quick scams they would have HUGE bandwidth bills, and think twice about using the same marketting technique again.
It's no use trying to email abuse depts, or reason with this scum, you have to hit them where it hurts, in the wallet. The only way to do this (for us at least) is to suck their bandwidth dry
Code, Hardware, stuff like that.
Glad someone said that, I was about to make the same comment. Dividing zero by a positive is not an error - the result is just zero.
Author, Shell Scripting : Expert Re
Did you read the article? Some sites like to brag and show visitors where people are coming from--if you spam the referrer log, you can get your links on these kinds of pages...
One of the primary uses I have for referrer information is locating bad or malicious links. If someone is sending large volumes of traffic to a particular page on my site, I'd like to know where that traffic is coming from. In addition, even to pages on my own site, if I see someone following a link to somewhere they either shouldn't be going or to a mistyped URL, the referrer information allows me to identify where they're coming from, and if it's a problem with my own site, it lets me correct it.
Perhaps referrer information should be released depending on the site's posted P3P privacy policies. If a site is interested in collecting information like this for marketing purposes, I can understand someone's reluctance to have their browser provide it. But for the rest of the sites (including those I maintain), the information is only ever used strictly for legitimate needs like those mentioned above. Please don't advocate that referrer information be restricted by default or for everyone, because that hampers my ability to troubleshoot problems.
The point is, if I design a site which steals your bandwidth by using your images, most people who view my page can't see the images (if you block external referers to your images) Unless that changes so that most people don't send HTTP_REFERER [sic] you're safe.
Author, Shell Scripting : Expert Re
Backlinking, or posting your referral logs, is doomed to failure and rightly so. It's just a glorified way of making your site into a link farm, with the expectation that your fellow bloggers will do the same. It is serendipitous that this practice is open to 'abuse' although I would never call the abusers spammers. They are just utilizing a method for submitting data that the site owners themselves have provided. I don't see any reason to call this 'spam' since the site owners are inviting users to submit data through HTTP referral headers.
Also, this quote from the article is ludicrous: "bloggers are not thrilled, even though they ruefully admit that the log spamming may falsely boost their ranking on some search engines."
There is no search engine that bases your rank on the number of sites that you LINK to. I believe the bloggers actually mean that they're sorry to see their backlinks (read: link farms) go, since those do in fact raise search rankings. What a travesty- Sites may have to rely on the actual quality of their content, rather than trading links!
Amidst the alarmist cries in the article, "spammers will destroy our practice of posting referral logs," nobody has even mentioned that there is a ridiculously easy technical solution. Before posting a referral link, why not just have your software visit the referring site and detemine if it actually links to your page? This will defeat the referral advertisers.
For instance, if a person goes from a Monster.com search page to his Yahoo mail account, Yahoo now knows where the person is looking for a job, what type of job he is looking for, etc. (it's all encoded in the URL). Yahoo also has access to his address book and all his email messages.
I see a scenario where Yahoo subtly threatens to email your boss to let him know you're thinking about quitting... unless you upgrade your account/add more storage space. It won't happen tomorrow, but Yahoo is sleazy enough to try something like that and they have the information... all they need is the technology to make that connection.
That's just one example, but it illustrates the point that referrer information is none of your business. You only want it because you can profit from it without any complaints from your audience.
Another example:
A lot of people apparently email the URL of my site to their friends. In my site logs, I often see the email addresses of the person who sent the message and the poor sap who clicked the link. These people have no idea they have divulged their email addresses to me via referrer info. If they wanted me to have that info, they would have given it to me. Sometimes I also see the subject of the message, which is particularly funny when it was sent by a competitor along the lines of "Have you seen what <insert_url_here> is doing?"
But as you said, "it isn't really as great a privacy issue as you seem to believe." It's worse than you realize.
Bottom Line
Companies will do just about anything to make an extra buck. So it shouldn't surprise anyone that they use technology against users to that end. But it's a two-way street -- people just need to wake up and start using technology to protect themselves.
The only scheme for verifying the links that can't be fooled by the spammer is human moderation...
An engineer who ran for Congress. http://herbrobinson.us
Oh I agree. Like I said, Wishful Thinking. I actually do follow a couple of the more realistic ads. But for the love of Pete, I had to set my Hotmail account to automatically discard spam since it was filling up the entire account faster than the spam bucket was being emptied! We're talking three or four hundred spams in less than a week! And 99% of them fall into one of three categories: sex organ enlargement and various performance improvement widgets, though I'm quite satisfied with mine already; a fake college degree, though I already have a real one; debt consolidation, though I'm 5 years ahead on my college loans. I simply don't need this kind of harrassment.
I really do wish spammers would actually target their audience. I might get just as much junk, but it'd at least bear some resemblance to stuff that relates to me.
Dyolf Knip
For now I'll delete the entries by hand, but if this increases it could get really annoying.
AlpineR
Isn't that kind of like saying because someone has an email (a method for submitting data), it's okay to spam?
Plus, think of the numbers. If people are selling this 'service,' it's bound to have a negative effect on the overall quality of the web features like this offer.
J-Log: Journalism News, Media Views
And this is, of course, broken behaviour.
So do you have an alternative proposal to prevent resource (i.e. bandwidth) theft? That is a very real problem, and no amount of arguing that the current solution is "broken" will get people to change unless you provide them an alternative.
My idea was to have a way to be able to construct backlinks from sites. At the time we had 100 users and the operating assumption was that all information put on the Web was public.
I did not write the code that implemented referer. However the security note I wrote did say that you should only send referer if you were actually following a link. The NCSA folk introduced the idea that it was a link to the 'last think you visited' which I consider to be buggy, there is no reason to reveal file: and bookmark: URLs.
We considered the privacy implications. Basically if someone wants to they can pass the linkage info explicitly via a query suffix.
There should be a toggle in my view, but folk seem to take a wierd view on security and privacy. Argument by analogy was the rule at the time. We can do BASIC password security because FTP does, DIGEST was written less than a week later, no interest of course because it was not compatible with the then rulling UNIX dogma of one way encrypted passwords, forget the fact that sending the password en-clair is a bigger problem. You can't have the password encrypted both on the wire and in storage without using public key which was patent encumbered at the time.
Oh yes, the mispelling was me. I am dyslexic. However we are petitioning to fix it. The next edition of the OED has the additional spelling referer which specifically describes my HTTP header. I am trying to get into the Guiness book of records first with the most serious spelling mistake ever.
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/
I invented the referer field before the IMG tag was proposed. So no, that application was not one that was ever considered. Nor is it the way I would have solved the problem.
I always thought the way IMG works somewhat broken but I happened to be asleep for the 8 hours it was up for review.
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/
they will come... and rip it to shreds as fast as they can in any way possible.
It's the same deal if you have any kind of script that can be compromised. Example: FormMail.pl, if it didn't do strong checking someone could use it maliciously. There are a few ways to combat this, like setting a repetition checker so that if within n seconds if the same thing comes in m times ignore and remove it and/or ignore the ip address(es) it's coming from. You can also set it so it will only work for trusted people and you could have do some small monitoring to make sure none of the trusted people are flooding it. There are many ways to go about preventing the spammers from getting through you just have to think practically (ie: What do spammers do that would be different from your regular users) and do a little coding and your done... They obviously could care less about you, so there's no other way to really deal with them.
The only way to do it right is to generate pages on the fly, with all URLs in it being re-written to be cryptographically signed and timestamped. A link would look something like
The web server checks the signature and lifetime of every request before serving up the file. (I implemented Apache modules etc. to do all this at my job.)Unlimited growth == Cancer.
You are forgiven :) No, seriously the world is different now. Is W3 the organisation which could try to push it through? Or do we just have to believe that the browser vendors realize it in time before every site utilizes this in their inner logic and the change to better is thus impossible.
Where it == a change to how referrer information is sent. As there clearly is some benefits for the website developer/author for having the referrer info the situation after change could be an analogy to what is used with cookies :: the referrer info would be only sent if 1) the user is following a link (or similar mechanism) and 2) the link being followed resides in the same space.
Examples:
link from www.xyzzy.com/index.html to www.xyzzy.com/about.html - sent
link from www.xyzzy.com/index.html to www.othersite.com/ - not sent
link from www.xyzzy.com/~jukal/categories.html to www.xyzzy.com/~jukal/contents.html - sent
link from www.xyzzy.com/~jukal/ to www.xyzzy.com/~abuser/ - not sent
.
As you Zeinfeld, clearly are in the position to make a difference (being the "inventor" - or one of the inventors - of the mechanism) - what do you think about this? Do you know if w3.org for example is already considering this? If not, who should I, You, everyone else talk to?
I wonder if there is a vulnerability in here somewhere... people are displaying raw referrers on their sites, typically via a server script of some sort. Potential breeding ground for a new worm of some sort?
On the other hand, perhaps this is the first valuable use of spam: making people aware of the problem, and the smarter people shutting it down, before someone writes a worm to exploit it.
Read reviews of shopping cart software
15.1.3 Encoding Sensitive Information in URI's Because the source of a link might be private information or might reveal an otherwise private information source, it is strongly recommended that the user be able to select whether or not the Referer field is sent. For example, a browser client could have a toggle switch for browsing openly/anonymously, which would respectively enable/disable the sending of Referer and From information. Clients SHOULD NOT include a Referer header field in a (non-secure) HTTP request if the referring page was transferred with a secure protocol. Authors of services which use the HTTP protocol SHOULD NOT use GET based forms for the submission of sensitive data, because this will cause this data to be encoded in the Request-URI. Many existing servers, proxies, and user agents will log the request URI in some place where it might be visible to third parties. Servers can use POST-based form submission instead
Unless he got to his yahoo mail account by clicking on a link that he found at moster.com (highly unlikely), this won't happen. On most of today's browsers, if you enter an URL manually, or if you use your bookmarks, the referer field will be empty, rather than containing whatever page happened to be displayed in the browser window. It's called referer for a reason.
Some old versions of netscape sometimes did funny things with the referer, but who continues using netscape 3.01 nowadays?
Say no to software patents.
Pssst! it was a joke :)
Except that any decent system should ask is you're sure about deleting that account. And while it does that is should give some nice random text as a hidden field and expect it when submitted before deleting the account.
Also I think the odds of having this sort of trap are minimal. And you should always be able to undelete accounts...
- Raynet --> .
As for someones right to see where you come from, yes, you're right. Which is why it is up to you whether or not you use a client that allow you to turn the referrer header off or fake it. But on the other hand, it is up to the webmaster of the site you're trying to visit whether he'll then decide to prevent you from accessing his site.
Unsurprisingly, bloggers are not thrilled, even though they ruefully admit that the log spamming may falsely boost their ranking on some search engines.
So how is this, exactly? Search engines (think Google) may boost pages that are heavily linked to, but sending false referers to the website does *not* affect Google's rankings in any way.
Google goes by how many pages in its index contain a link to a site. It doesn't care what is in the site's logs, it wouldn't have any way to know this.
Just an observation...
NGWave - Fast Sound Editor for Windows
Keyboard nav is much better than links (use numbered links with "G," as in "25g" takes you to - but doesn't follow - link/text entry box #25 on the screen, etc.).
So, you mean you sit there and count how many links are on a page, then figure out where on the page #25 is, and then type all that in to go to it, instead of just scrolling down and clicking or something similar? How incredibly stone-age.
- A.P.
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
> No I will NOT post my personal website on slashdot... I'm not about > to get a huge bandwidth bill because of you guys!)
:-)
Chicken.
Bugs Bunny was right.
Why does this bother you so much? You've turned off your referrer, right? My site treats you like you appeared out of the blue sky, and you have access to everything that any other surfer does, without any "annoying", "stupid", "broken" fluff.
Do you honestly think that Yahoo is going to extort you based on a referrer log? You've got some pretty far fetched ideas there. Do you also use an anonymizer service? If not, then I've gained a ton more information about you than I could with the referrer log. All the referrer log tells me is that another web site has a link to my site, and that you clicked on it. Monster.com has no links to my site, except to my full resume - and it's fun, though not particularily useful, to see that there are people going to my resume page from monster. It saves me the trouble of creating several different links so I can track where people come from.
In other words, you are really stretching the point you're trying to make. Yes, it's technically possible that the referrer could be used in a way that makes your life less private. YOU have control over that, though, since you can turn it off. Saying that it should be off by default without providing some real, tangible benefit is shortsighted.
I'm sorry that 'webmasters' piss you off. I don't have a welcome page, the most I do is provide an extra line of text at the top of the page giving additional info to those who go there from specific sites and links. Instead of blocking my images I've decided to simply get rid of them - they don't really add anything to the content except where the content is the image, and I've lowered their size as much as possible. Speed of serving is more important than annoying the few idiots who do refer directly to the images on my site.
In short, like "rm", it's a tool. It can be used for good and bad. You either visit sites you don't trust frequently, or you are paranoid enough to leave it off all the time. I can understand that, and I agree that it's probably the best thing for you to do.
It is not a tool of pure evil though, and I'm afraid you've become somewhat like the RIAA in your argument. You assume that either (1) it can only be used for evil/bad/annoying/stupid purposes or (2) most people use it that way, and the few that use it for good can be as effective at delivering useful content without it.
-Adam
On the dynamic HTML generation side, most app. server based sites have to do that anyway; there's just one additional step of replacing some URLs with signed ones. If you handle web sessions through URL-rewriting rather than some hack like HTTP Basic Auth or cookies (ugh!) you pretty much get it for free.
Unlimited growth == Cancer.
The HTTP spec is not owned by the IETF. I have no intention to work on it in the near future, I am currently working on Web Services security.
The cookies model is the wrong one. We want to track across sites. It is important for the maintainer of CNN to be able to find out if the BBC has linked to their story.
The places where I would make changes is in the privacy area. The mechanism should be optional and be disabled by a switch. Referer links should never reveal the existence of a private document such as a bookmarks file.
However in terms of priorities I would put making popup windows optional much higher on the list. It should be possible to disable Javascript and Macromedia on a per site basis. IE is almost there in the later editions. I have killed a lot of popup ads by simply nominating their zone as being not authorized to run javascript. Jscript and Active-X should be managed the way images once were, the text would load and then you would press a button to load the images if you wanted them.
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/