Slashdot Mirror
Best Practice For Retiring RSS Feeds?
GBJ writes "I work for an organisation that runs seasonal online competition events. Each event has its own news feed which becomes obsolete shortly after the event finishes. We're still getting RSS requests for some events as far back as 2004. I'd like to close a few thousand old feeds and remove the resource hit they cause, but I'm not sure what is the best approach. Currently I'm considering just returning a 404, but I have no idea if there is a better way to handle this. Uncle Google hasn't turned anything up yet, but sometimes it's hard to find something when you don't know what it's called ..."
Retiring feeds
A phase-out needs.
As facial bristles,
Or torn skin bleeds.
Burma Shave
Get thee glass eyes, and, like a scurvy politician, seem to see things thou dost not.--King Lear
Returning a 404 would be the first thing that I would do post competition. You still get the requests for a bit though...
sudo mount --milk --sugar
You should be able to do a 301 redirect to maybe some generic feed that just has one entry that says "This feed is out of date, please use try these feeds instead." Or whatever you want to let them know.
Reviewing just the first hour of video games.
The most appreciative thing you could do for the preservation of history is to place static simple RSS files at those addresses that include a link to your archives for the event.
IMarv
Trusting software vendors is no smarter than trus
If it's that old and no updates are going to be made to it, just stop providing it. Or throw one last update that says the feed is terminated and point them to a newer version of it. I doubt your users will cry foul.
RFC 2616 is the one to read. It specifies a "410 Gone" for resources that are gone for good.
Would it not be feasible to return a string of text along the lines of "This competition has been closed, and results are no longer available. Please visit our current RSS link to the current competition at ?"
I know that it will still be a hit, but a lightweight one, and more polite than a 404.
Quidquid latine dictum sit, altum sonatur.
If so, you can either archive the feeds on a web page (of only web page versions) and charge for "back issues" to cover the costs.
Still getting requests? Is it folks who are emailing yo explicitly for the feed or you're seeing traffic on those feeds in you logs?
If it's just traffic, maybe it's just some default setting on someone's phone program or something? In other words, those requests aren't "real".
Either way, I say archive them and if they really want to see them, let them bring up a browser. You're still offering service but you're not, I don't think, taking up much resources.
I am mildy retarded and lack any sort of thinking skills. Please tell me how to do my job.
Gone
Indicates that the resource requested is no longer available and will not be available again. This should be used when a resource has been intentionally removed; however, it is not necessary to return this code and a 404 Not Found can be issued instead. Upon receiving a 410 status code, the client should not request the resource again in the future. Clients such as search engines should remove the resource from their indexes.
use a 410 Gone more informative than 404
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.11
ERR 411[Max number of witty sigs reached]
The resource isn't "Not Found", it's "Gone". HTTP 404 is inappropriate in this instance.
Likely the best solution will be to ensure that people are notified first. If you're receiving a large number of hits, replace the content with a single RSS item that has a guaranteed-unique guid for every single request (say, based on the request time). This way, with each request, people will receive a "new" item that will display as unread, reminding them to unsubscribe from that particular feed.
After some amount of time, start returning HTTP 410 for all requests.
divert them all to a generic overall news type feed - upcoming events etc.
Maybe add an advert or 2
HTTP 410 is better than HTTP 404 in this case; Uncle Google and the like have a better understanding of what it means.
If you have a version of the feed that covers the whole event from start to finish, you might also want to offer a static version of that for download as an archive (but if you do this, put it on a different URL from where the feed used to be). This isn't strictly necessary, but I can see scenarios where people might appreciate being able to get at the feed's contents again.
Are there more than one contest going at once, that there's a need for multiple feeds? Or, more appropriately, can some of this content not be removed completely? Keeping a full feed archive seems a bit of overkill, especially for closed events from five years ago. Why not PDF the event archive for downloading and keep a single feed for active items? Overpreparation is a growing problem I'm seeing on the web. Far too few people/events/businesses are prepared to minimize anything for the sake of optimization.
Those things you're doing with that stuff you just bought? That's not what it's for! -
Just return an unending stream of crap from /dev/random on your server until it crashes the RSS reader at the other end. After a few days of this I'm sure they'll sort it out from their end :-)
Code, Hardware, stuff like that.
I may have missed something. But if its seasonal, why create a new feed for each event?
Why not reuse the same feed over and over again each year? If you are getting alot of old requests, its possible that one of your old feeds is featured someplace or getting good search results. (not to mention all the old subscribers that already had it). Everyone that was involved in the old event would automaticly get the new content.
Think about it
Im a gamer, not a grammer major. This post is full of spelling and grammer mistakes.
Put the feeds on an ice floe.
Best Slashdot Co
Use a "disposable" DNS record e.g. eventname.rss.domain.com
When time comes to bin the feed you can either alter the record to a server that just serves 404's or remove the DNS entry completely. Useful for feeds getting lots of hits/second as removing the record drops all the useless traffic from your site.
Just put the URL of $SHOCKSITE in the RSS feed.
You know you want too :-)
My company has had the same issue. We just wrote out a single item feed that explained that the feed had been discontinued and provided a link to the homepage. We wrote this feed over all the feeds to be taken down.
Interestingly, in the weeks following this action, our homepage views spiked followed shortly after by increased hits on more active feeds.
Your best bet is to think about it beforehand. Assign each feed a unique DNS hostname. When it's time to retire the feed, change the DNS hostname to point to 127.0.0.1 with a TTL of 30 days.
That won't help you with the feeds you've already created but it'll help you with the ones you're creating now.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
No way man. You can't just dump all those people like that!
Either redirect them to a feed that says "here are some other events you might be interested in subscribing to" or create a last post on the feed that says the same thing. Dumping them with a 410 is a great way to loose your traffic.
It sounds like in this case, a redirect is the proper thing. Just be forewarned that some readers (cough Google Reader) will redirect, but ignore the "permanent" in "permanent redirect" and will continue to hit the old feed.
It would be bad SEO and hitting up 127.0.0.1 will probably confuse some newsreaders.
Worse, now you've got to either maintain a buttload of random noise in your zone file, or you have to have a wildcard entry pointing to a wildcarded virtual host and maintain a buttload of feeds in a database. Either way, you've added a bit of complexity.
Newsreaders are amazingly stupid. Google Reader, for example, will ignore both your "301 Permanent Redirect" and ignore your Atom link="self" as well. Something tells me it will ignore your "410 gone" too. You should test what major newsreaders (MyYahoo, Bloglines, Google Reader) do when you deep-six a feed with a 410.
Even still, 410 is stupid. A 301 redirect to something generic would be better, even if the readers are too brain dead to get the message. 410="lost traffic".
This seems like more of an opportunity than a problem. People haven't deleted the feed, so they obviously still want to hear from you. Redirect requests to the 'dead' feeds to a general feed that announces new events.
You've got one feed for updates in each sport or event.
Imagine horse racing. You are making updates to racing scores throughout the day. Gamblers are monitoring each race. How would you structure this king of thing? One feed per day? One per season?
What about comments? You've got a feed per thread/story. When the story is closed for comments, what do you do with the feed?
What about auctions? You've got an auction and every bid triggers a new entry on your RSS feed. When the auction is over, what do you do with the feed?
Yes. Because if the newsreader flakes out, the *person behind the computer* might blame you and never subscribe to your feed again. We are dealing with people here, not machines. Were are not here to seek spite on stupid newsreaders or "lusers who dont unsubscribe". This is business. Petty bullshit only wastes time and loses money.
Even if the newsreader was well behaved and did the right thing, you've just stupidly removed your brand from being thrown in their face every time they opened their reader. If you've got like 10,000 subscribers sitting on a bunch of year-old feeds and you cruelly dump them with a 410/404, that is lost mindshare and lost traffic.
The best solutions are the ones that encourage feed users to keep using your services by nudging them to your new stuff.
Bottom line is, it is easier to keep existing customers than it is to get new ones. Think about it.
Are RSS feeds really causing an insane amount of traffic? Does that traffic cost more than it does to acquire new subscribers? I doubt it.
Let me qualify that though. Before I did anything drastic like 410'in their ass, I'd come up with the total number of subscribers to the entire pool of feeds. Most of the big-boy newsreaders will usually include a subscriber count in their User-Agent. Tally up the numbers and make sure you aren't dumping 100,000 subscribers in an attempt to save a piddly 1,000 feed-hits a day.
I had a problem with people requesting feeds several years after they were defunct. If I were making new feeds today, I would put them on their own A records, like news.feeds.example.com, comments.feeds.example.com, etc, and then simply remove the DNS when the feeds are obsolete.
How about the ones on /. home page? They still show old stories for MacCentral and MacSlash. :(
Ant(Dude) @ Quality Foraged Links (AQFL.net) & The Ant Farm (antfarm.ma.cx / antfarm.home.dhs.org).
It seems to me that the best practice would be to have a feed for each event that's re-used annually, rather than a feed for each event each year. Presumably folks interested in a given event in 2008 remain so in 2009.
Just pump 100 posts an hour with viagra/watches/penny-stock spam links into those feeds, they'll either get unsubscribed immediately or pull you a huge profit .. it could be fun finding out which ..
[no, really don't do this]
Sounds like you would benefit from a site level "current events" feed, that always has the most recent events. Then a per-event feed that, after the event, changes to "thank you for coming, see more at "current events". And that feed expires 3 months after the event.
If people are collecting your old feeds, but not getting value from them, is that actually valuable traffic? Are you interested in getting the word out, or counting fictional traffic?
You can also rename the feed to archive old information. Renaming the feed breaks the links to lazy readers. Anyone who really wants to research your old event data will search for it starting at your home page.
Since the content is static you can put it behind a caching CDN with a far in the future expires header. Let the CDN take the hits and keep the SEO.
Alternatively, you can redirect the requests to a copy of the feed from archive.org
Goatse.
This is Slashdot, and I guess some here view the world in some kind of mechanical way free from the constraints of the real world. So yeah, if he wants to tell them to fuck off and feed their newsreader streaming crap from /dev/urandom, by all means do so. But that is a child's thing to do. Adults view the world different.
A mature adult would see that the cost of bandwidth is minimal and the untapped potential in all those people sitting on ancient feeds outweigh the potential costs. A mature adult who strives to have a successful website would find a way to tap that potential, possibly by redirecting the feed to something that nudges them to the good stuff.
But no, go ahead and stream your fucking mp3s to their newsreader. That will teach them to ever cross path your mad skillz. It will teach them so good they'll never visit your site again, never click your ads and never buy your services. Who needs their money, right? After all, rent is cheap living in the basement!
(ps: booya)