Slashdot Mirror


AOL Blocks Links from LiveJournal

Evan Martin writes "LiveJournal.com is an open-source weblog site with over a million users, some of whom use AOL. Last week, AOL began blocking all HTTP requests with "www.livejournal.com" Referer headers. This is a common practice by image hosting sites to prevent off-site linking of their images and 'bandwidth theft'. However, in AOL's case, they're blocking everything, not just images, effectively breaking all links to any AOL member's site--but only from LiveJournal. To be clear: nobody on LiveJournal can even make a link to any AOL member site without getting a '404 Not Found' error. We've also heard reports of the same thing happening on AOL properties (Netscape, Compuserve). This concerns us because we have to deal with the support requests: it worked in the past for our users, and it continues to work for other sites, so our users think it's our fault."

Martin continues: "We've tried to contact AOL three different ways, all without success. We've also told our users to contact their tech support. At one point, an AOL staffer pointed out that FTP access still worked (which is probably because FTP has no "Referrer" concept), and so, as an interim fix, we're rewriting all HTTP URLs to use FTP on the AOL properties where that works instead. This means that users can again host their images on the AOL webspace they're paying for, but more importantly, it means they can simply link to their webpage.

We wouldn't be so upset if they were simply blocking images. Bandwidth use is a valid concern, after all, and we even provide step-by-step instructions for people to configure their webservers to prevent image "theft". However, because they're blocking all access, including regular links, this looks like it's either a mistake, or something more insidious (the conspiracy theorists have pointed out that AOL has just launched their own competing weblog product, also based on "journals").

Although CI Host sued AOL recently for being blocked, we really don't want to do that. We still suspect that this was all just a mistake, and hopefully, by making this public, we'll manage to get their attention, since all our previous attempts have failed."

40 of 396 comments (clear)

  1. Will this be what kills the referer header? by Anonymous Coward · · Score: 5, Insightful

    It's optional, so browsers don't need to send it. Mozilla/Firebird/etc (and Opera) can be easily modified to not send one, and the Google Toolbar could probably support blocking them, too (since IE isn't being updated). AOL is a big enough presence that this could have a significant impact on peoples' browsing.

    1. Re:Will this be what kills the referer header? by mosschops · · Score: 4, Interesting

      I use a program which allows me to change the referer header to appear to be from the same site I'm browsing. Not only does that protect my privacy but it works with the vast majority of sites that clamp down on content linking / bandwidth stealing. Sometimes simply removing the referer will kick you back out to an index page, which is a pain.

  2. Re:F12 by Nasarius · · Score: 5, Informative

    Gotta love Opera :)

    --
    LOAD "SIG",8,1
  3. Good. by Genghis+Troll · · Score: 4, Funny

    Anything that discourages "blogging" can't be all bad.

  4. AOL and Blogs by zangdesign · · Score: 5, Insightful

    Actually, you may want to investigate whether or not AOL has gone live with their blog offering ( article here). If so, it may be viewed as an intentional act.

    --
    To celebrate the occasion of my 1000th post, I will post no more forever on Slashdot. Goodbye.
  5. tinyurl? by Wavicle · · Score: 4, Interesting

    Could you get around this using tinyurl? I'm not sure if it changes the HTTP_REFERRER or not.

    --
    Education is a better safeguard of liberty than a standing army.
    Edward Everett (1794 - 1865)
    1. Re:tinyurl? by Chmarr · · Score: 4, Informative

      It TOTALLY depends on the browser you're using.

      If you're on web page A, click on a link to B and it redirects to C, some browsers will, when fetching C, have a referrer of A, and some will have a referrer of B.

  6. Well played... by tempest303 · · Score: 5, Insightful

    This is a really level-headed, well played move on LJ's part - primarily because they're following the universal principle of assuming stupidity before malice. ;)

  7. Sorta related by Tyler+Eaves · · Score: 5, Interesting

    Any else noticed that members.aol.com is sending an invalid content-type header?

    I've seen iso8859 and text/iso8859-html, neither of which Firebird likes...

    --
    TODO: Something witty here...
  8. Use link referers by EDA+Wizard · · Score: 4, Informative
    Until this mess gets sorted out, people should use a free link relocator service. Make A Shorter Link would work well for links to AOL pages.

    It wouldn't help people with embedded links to images at AOL, but at least it could get people to AOL without any additional clicking.

  9. They block slashdot too. by Anonymous Coward · · Score: 4, Interesting

    Well not the whole AOL network, but the former mozilla division blocked links from slashdot (and still does), (Example). Any sites that cause major bandwidth use should be blocked, I'm sure some frequest slashdotters get the infamous Pink page of death.

    1. Re:They block slashdot too. by mabinogi · · Score: 4, Informative

      No, the maintainers of bugzilla blocked links from slashdot.
      It has nothing to do with AOL.

      They did it because bugzilla is an entirely dynamic site, and an important tool being used by the developers.
      The last thing they want, is 50,000 slashdot users hitting it at once and preventing them from working.

      --
      Advanced users are users too!
  10. Why is it... by pongo000 · · Score: 5, Insightful

    ...that people bend over backwards to accomodate companies with draconian policies like AOL? If I were running an ISP, the loss of a few customers because they suddenly discovered they could no longer send e-mail to AOL customers through no fault of my own would most likely be offset by new customers who understand that the earth does not revolve around AOL. So they're blocking incoming HTTP traffic based on referrer? Are there not more pressing problems to attend to rather than trying to please the AOL gods?

    I'm not saying AOL is in the right. I'm simply saying that AOL (and companies like them) should be made to lie in the bed they make for themselves. Only when AOL customers start to be inconvenienced by AOL's own policies (rather than third parties patching together "workarounds" in a misguided attempt to protect the integrity of AOL) will they realize what AOL is up to...

  11. Wait a minute by s20451 · · Score: 5, Interesting

    Anytime there's an article that whines about deep linking, a few dozen people post replies saying that the company could use the referer header to block all such requests. Now that a company is actually doing it, it's suddenly a bad idea. Which is it -- good technical solution or bad censorship?

    I should also point out that some sites automatically block slashdot.org referers as a matter of self protection.

    --
    Toronto-area transit rider? Rate your ride.
    1. Re:Wait a minute by isorox · · Score: 5, Insightful

      My Libertarian side says AOL are free to do whatever the hell they want, it's their server. If you ask the AOL server for a page and it send you the goatse man, thats fine, thats their right. Vote with your wallet and dont buy their service.

      My more centrist side says this could be abusing a monopoly (or at least dominant position), OK they dont have a microsoft style monopoly, but they do have the monopoly over Joe Stupid.

      My cynical side says who gives a flying fuck

    2. Re:Wait a minute by infornogr · · Score: 4, Insightful

      I regret to inform you that all Slashdot comments are not posted by the same person. Not only are there different people on Slashdot, some of them have different opinions. It's a shocking revelation, I know.

    3. Re:Wait a minute by Sponge+Bath · · Score: 4, Funny
      Not only are there different people on Slashdot, some of them have different opinions.

      Agreed. There are those who think Microsoft sucks, and those who think Microsoft blows.

    4. Re:Wait a minute by danielsfca2 · · Score: 5, Insightful
      Deep linking seems to be a totally different issue here. Your average AOL user who wants to place a link on their LiveJournal to "members.aol.com/aoluser" would derive little value from a link to "www.aol.com" or even "members.aol.com"!

      This is webspace that AOL gives its users as part of their paid service. When you pay for webspace, the general idea is that it supports these things called hyperlinks. It stands to reason that you or anyone else should be allowed to link to your website from any other website. Any deviation from this traditional behavior should be documented in their terms of service, and is very shortsighted and/or stupid, as it threatens the very nature of the WWW, much like restrictions/penalties on linking to sites that are deemed undesirable.

    5. Re:Wait a minute by caferace · · Score: 4, Funny
      Waitaminnit. Since you're user #441 and posted at 4:42, and I'm user #442 I shoulda posted at 4:43. Now I've gone and fucked the whole power cycle up.

      I guess we'll have to try cold fusion again...

  12. And AOL wonders why..... by HutchGeek · · Score: 5, Interesting
    Seriously - they wonder why they get such a bad rap from the internet community at large. Most likey what has happend is that "Upper Management" made the decision to do it for some reason (although the journal conspiracy sounds quite probable), and they did't bother to ask the "real staff" what kind of an impact it would have. Now, once again, they've managed to piss a whole lot of people off. Makes you wonder what else they've blocked (censored) that thier users don't know about. I've heard rumblings on NANOG that they are trying to whitelist thier email too. There's a bright idea - a customer base the size AOL has, and their gonna whitelist mailservers. and my cutsomers wonder why I get ready to slap them when they suggest using AOL for a provider.

  13. Quick fix for HREFs viewed by MSIE by inertia187 · · Score: 5, Informative
    That's wrong of AOL, but if you're a LiveJournal user in a bind and really want to fix the links (but nothing else) fast, here's a JavaScript that you can load in all of your pages. You just need to load it once, and the page will work.

    Unfortunately, this trick really only works with MSIE. But it's better than nothing.
    <script language="javascript" type="text/javascript" src="http://www.martin-studio.com/js_tools/strip_r eferrer.js"></script>
    The above should all be on one line. Check for extra white space where the line feed got placed by Slashdot's bug (thanks alot).

    It should be strip_referrer.js with no space. Why does Slashdot do that??
    --
    A programmer is a machine for converting coffee into code.
    1. Re:Quick fix for HREFs viewed by MSIE by jamie · · Score: 4, Insightful
      Dear God in heaven, anyone who does this is nuts.

      Load a javascript off the website of someone else you don't trust? Cmon now... let's just invite a stranger into your home to watch all your websurfing, or post the contents of your cookie file to your LJ.

      Anthony, I'm sure you're a nice guy and all, but would you trust a random stranger's javascript on every one of your webpages?

      (The space added to the URL you pasted in is added to every long word at the 50-character mark, to make sure idiots can't break your browser rendering by typing very long words into their comments.)

  14. Re:F12 by randyest · · Score: 4, Insightful

    Well, if I were an AOL user right now (quite a stretch to imagine, I'm afraid), I'd be loving to check my contract to see if it provides for such limitations on user personal space. If not, I'd call to complain (I'd probably do that anyway, right before I called my new ISP). I know that my cable modem "free" hosted space included has a very specific contract that limits the monthly bandwidth usage, but does not mention anything about blocking access for links or clicks from other domains. I wouldn't be happy if they suddenly started returning 404 Page Not Found errors when anyone linked or clicked from foo.com.

    Maybe they can claim technical difficulties if called on it, or maybe the contract does let them yank their users around like this. I don't know. Does anyone here use AOL and will admit it long enogh to post a link or copy of the appropriate contract?

    Of course, there's the tinfoil-hat theory that AOL is planning to start thier own blogging service and wants to drag LiveJournal down from it's #1 spot a bit. Seems like a particularly blatant and non-clever way to do this though, or maybe that's the "beauty" of the whole ingenious plot? ;)

    Finally, I'm going to hope the /. editors checked this out somehow. I don't have AOL space. Can anyone else verify this is true and not just some EBKAC or hoax?

    --
    everything in moderation
  15. Referrer Header by Mrs.+Grundy · · Score: 4, Insightful

    I, for one, like the referer heading. It is useful to see where traffic is coming from and it really stinks that AOL is going to encourage people to mess with it, remove it, or spoof it. This will be the ONLY result of AOL's action. They may get a short break from livejournal links but people will work around it. The internet is about linking after all. If AOL want's to invent their own thing with their own rules they should make their own little private net like they used to have and they can remain one tight, happy, cloistered little clique. Of course if the referer header becomes useless maybe it would be a good opportunity to fix one of the most influential spelling errors in recent time and start using the refeRRer header instead.

  16. Common Decency Dictates.. by Plix · · Score: 4, Interesting

    That should AOL continue to block deep-linking (which they have the right to do so assuming that there is no contradictory clause in a user's contract) they should at least redirect users to a page explaining what is actually going on rather than leaving them to complain to LJ support.

  17. Bounce through a third party! by Anonymous Coward · · Score: 5, Informative

    A lot of websites let you bounce to other sites. Here are some demonstations

    Debian link to aol.com
    Yahoo link to aol.com
    Google link to aol.com
    Goatse link (yes, its true, goatse is useful!) to aol.com

    Hopefully, unless AOL wants to block the internet off, people will get around, and we can always set up p2p based redirection system (ala freenet). To get trough.

  18. Company People by philipkd · · Score: 4, Interesting
    Why can't we treat companies like people? If a friend of your group all of a sudden has all this money, but abuses it by playing little Napoleon, why can't you treat him with disrespect and ostracize him?

    When AOL needed help setting up their blogging software, who did they talk to? People like Dave Winer and other members of the net community.

    So shouldn't there be some sort of Karma here where we, the blogging community, ostracize a bad player. They do it to spammers all the time, why not to the big guys. They'll eventually realize that it's not profitable to do so, and conform.

    We could choose to disallow AOL urls into weblogs. We could prevent anybody with an AOL account having an RSS feed to a Blogger or LiveJournal. We could ban them from our conferences. Sounds like we're being assholes or "closed" by doing so, but I think it's important for people to check the bully to in the long-term enable the most openess possible.

  19. Re:hopefully by EvilStein · · Score: 4, Interesting

    Heh. Uh, AOL has more than enough bandwidth to handle freekin *livejournal linkage*....

    When I worked there (Netscape), we had 5 OC-48s in our building alone. OC-12s & DS-3 circuits for "redundancy."

    I highly doubt that it has anything to do with bandwidth.

  20. They Might Be Giants by Paradise+Pete · · Score: 5, Funny
    My Libertarian side says...
    My more centrist side says...
    My cynical side says...

    Hey look! It's Triangle Man!

  21. Killing referers kills EVERYTHING by strredwolf · · Score: 4, Interesting

    Unfortunately, killing the referer header breaks alot of sites which are blocking image pointing. We (KeenSpace) just put in header checking. We do it so that if a request for an image isn't from a webpage we host (eazy stuff to do), it's 404'ed.

    We cut our bandwith by 50% that way.

    --

    --
    # Canmephians for a better Linux Kernel
    $Stalag99{"URL"}="http://stalag99.net";
    1. Re:Killing referers kills EVERYTHING by Anonymous Coward · · Score: 5, Informative

      Then you are dickheads, plain and simple. The HTTP 1.1 RFC explicitly states that users should be able to turn off the Referer header. There are plenty of reasons for doing so. Furthermore, you aren't even using the right status code. It's 401 Unauthorized when you want to deny access, 404 means the content is missing (which it clearly isn't).

      There _is_ a fairly safe way of doing what you are after - let through empty strings and strings with spaces in. This lets through legitimate users who either disable the referer header, or have it set to "blocked by Norton" or whatever, whilst still stopping anyone from usefully using your bandwidth (since most of their visitors will still be providing the referer header).

    2. Re:Killing referers kills EVERYTHING by Anonymous Coward · · Score: 4, Interesting

      which is the main reason i dont goto any site on keenspace, one less visitor for you, one less comic for me. oh well, theres plenty more sites that dont block empty referrals.

      you may have cut half your bandwidth, but youll notice you also cut a third of your user base. hey, you might try blocking images for EVERYONE, then youd REALLY save some serious bandwidth!
      somehow i just dont envision there being millions of sites posting entire comic strips on their pages linking to your sites images...

    3. Re:Killing referers kills EVERYTHING by j7953 · · Score: 4, Informative
      Furthermore, you aren't even using the right status code. It's 401 Unauthorized when you want to deny access, 404 means the content is missing

      Yes, but...

      First, the "correct" status code would be be 403 Forbidden, 401 Unauthorized is used if "the request requires user authentication" and will cause the browser to prompt the user for login information.

      And for status 403, the HTTP standard (RFC 2616) says that "If the server does not wish to make this information [explaining why the request what not fulfilled] available to the client, the status code 404 (Not Found) can be used instead." The normal use for status 404 is if the server cannot find the requested resource, but according to the RFC it is also "commonly used when the server does not wish to reveal exactly why the request has been refused, or when no other response is applicable."

      So, either status code 403 or 404 are correct when trying to prevent precise ("deep") links from working. I agree that 403 is preferable.

      --
      Sig (appended to the end of comments I post, 54 chars)
  22. I, for one... by Anonymous Coward · · Score: 5, Funny

    ...welcome our new AOL overlords.

    I'm getting a ritual circumcision as required by AOL CEO Levin as we speakKKKKALRRRRRRRRRRRRRRRRRRRRR.

  23. All hail Proxomitron! by mad_dog3283 · · Score: 4, Informative

    Stupidity like this won't affect me at all. I use the Proxomitron, and I have the referrer field set to \u (which I think is the default setting). \u inserts the current URL into the referrer field. So, for example, if I hit a link on www.slashdot.org/foo.htm to www.aol.com/foo.htm, the Proxomitron will send www.aol.com/foo.htm and not www.slashdot.org/foo.htm to the server. This is especially helpful for sites that return 404's to requests with blank referrers (since the server always thinks the request is coming from its domain when in reality it may not be.)

    --
    Reprise the theme song and roll the credits!
  24. Re:Gee alot of ISP's use referrer blocking. by dasmegabyte · · Score: 4, Interesting

    Cheapasses is right. You get what you pay for, and if you're paying for impossible things, you should beware the catch. There is no such thing as unlimited bandwidth, or unlimited space...merely limits that you haven't hit yet.

    A t1 line is still over $700 per month, so burstable bandwidth starts at more than $2 per gigabyte. People who are on better pipes pay way less, of course, but then again they need to maintain them, and technicians start at about $25 per hour. Servers need to be powered, backed up and maintained to prevent hackattacks. So when somebody offers you unlimited bandwidth, unlimited space, unlimited email with 24x7 support for a pretty number like $7.77 or $5.55 or whatever, they're basically lying to you.

    Check your AUP. Somewhere in there you'll find a line saying that your unlimited bandwidth can be terminated at any time if you use too much of it. Unlimited really means "We're not telling you the limits. But you'll know when you hit them." Generally because your site takes off. You get popular, people start laughing at your jokes and caring about your weblog. Then your provider cuts the cord. Sucks, don't it?

    See, ISPs at all levels make money by overselling. They tell you you have a T1, when really it's fractional. They tell you you have 256 kbit upstream, then it maxes at 192. The most egregious example of this is the El-Cheapo webhost, an animal I despised so much that I started my own crummy service to combat it. If you have the know-how, and you have the time, I suggest you do the same. It can be a lot of fun and offsets the cost of big web projects. Just don't harbor any dreams of getting stinko rich.

    I remember the first time I had a site get "overnight popular." It was a certain web comic that we begged to come on board. In about two weeks ge went from moving 2 gig a month to over 50. And because we small timers get the short end of the bandwidth stick, his bill was about $200. Not his bill FROM us, but the bill TO us from our host for just his transfer. We didn't mark it up. That's a lot of money when you're a hobbiest. Shit, that was as much as we paid for everybody else's bandwidth that month.

    We have a policy of not touching people's sites or restricting tranfer, but if we hadn't known the guy (and known he was good for the money, which his new fans donated in droves, we even threw in $30), we probably would have had to use the "no contract" clause and take the site offline. Damned if I'm paying for somebody else's popularity...

    --
    Hey freaks: now you're ju
  25. Re:Solution: by carsont · · Score: 4, Insightful
    Don't use an ISP that is "broken". AOL has little to recommend it.


    Well, for one thing, AOL has been "broken" in many ways for many years, and yet they still have an enormous and loyal user base. So does Windows, for that matter.

    The problem is that for the average AOL user, who to put it bluntly is probably both too stupid to figure this out on their own and too lazy to read LiveJournal's explanation, it will appear just as likely that LiveJournal is "broken", not AOL. They will squeal "OMG WTF IT DONT WORK!!!!!1!!!11!!! :-(" and leap into the open arms of AOL's competing service. Whether or not AOL planned it this way is irrelevant.

    This is how the fragile and complex interoperability between pieces of computer software, which is opaque to most users, can subvert the workings of the free marketplace; if company A sabotages their product so that it won't work with company B's product, it is easy for customers to be fooled into blaming company B.

    Microsoft did this with their implementation of Java, and probably many other times. I doubt if this is some deliberate strategy on AOL's part, but the result will probably be the same regardless.
    --

    Ubi dubium, ibi libertas.
  26. Leave them feedback by BrynM · · Score: 4, Insightful
    You can leave them feedback regarding this decision at the AOL Feedback Page. Let them know that this decision will impact more than they expect it will and that you are disappointed with their actions. You don't have to be a customer. Most of us are technology experts who have influence on their potential and existing customer base. I've included my submission as an example, but please write your own.
    I find it disappointing that you are blocking content linked from such a popular site as LiveJournal based on referrer headers (see this slashdot discussion: http://slashdot.org/article.pl?sid=03/08/29/205724 2&mode=thread&tid=120&tid=187&tid=96&tid=9 9). I have recommended your services in the past, but that will stop now. Further, I will inform AOL customers whom I have recommended AOL to that their content is not available to sites that AOL seems to disapprove of for no stated reason.

    Changing your policy regarding this may defer my judgement about your "service", but your reputation has been irrecoverably tarnished in my view. Additionally your subsidaries, such as Netscape, will no longer be recommended by me.

    You may be thinking "Who cares? This is someone who isn't even our customer", but I have become the technology "guru" for over 100 people in my personal life and have input on technology decisions at my workplace. What should worry you is that for every peice of feedback you recieve like mine, there are hundreds of technologically literate people who will simply downplay or berate your services and will not do you the professional courtesy of informing you.

    Formerly Respectfully Yours,
    BrynM

    --
    US Democracy:The best person for the job (among These pre-selected choices...)
  27. A simpler, browser independent solution by Anonymous Coward · · Score: 5, Informative

    I don't have a problem with <obligatoryDerisiveness> AOhelL </obligatoryDerisiveness> preventing people from leeching images from their site, but there's a simple way to get around their prevention of direct links to their site: redirect using a META tag, which strips the referer header and makes it look like a direct request.

    For example:

    If you want to link from livejournal.com/myPage1.html to members.aol.com/~myOtherPage.html, then make the link go to livejournal.com/myPage2.html ..... in the header of myPage2.html, include this meta tag:

    <meta http-equiv="refresh" content="0; url=http://members.aol.com/~myOtherPage.html">

    It works accross all browsers and appears to AOL as if somebody just typed that URL directly into the address bar of their browser.

  28. You've Got Blog by annielaurie · · Score: 4, Funny

    My, undereducated guess is that AOL's done this because they have their own homogenated, cuticized, totally non-open-slammed-shut, AOL blogs. Yes, folks! According to their information:

    "Everyone has a story to tell; what's yours? Create an AOL Journal about your summer vacation, being pregnant or trying to find a new job. AOL makes it easy, fast and fun!"
    "Get Started Today
    Create a Journal
    Build your own blog
    with our cool tools. "

    It's all right there. I didn't have the heart to actually check out the blogs.

    What do we need with an open-source, customizable system like LJ's, complete with lively, growing user communities? Who needs software that might foster an actual original thought? Interacting with strangers is so...icky! We can pay lots of money instead to record our sanitized inmost feelings on the AOL version.

    They're probably staying awake nights figuring out how to block links from independent systems like Moveable Type. Could Slashdot be far behind?

    Anne

    --
    DUCT TAPE: The Election Supervisors' Secret Weapon