Slashdot Mirror
When RSS Traffic Looks Like a DDoS
An anonymous reader writes "Infoworld's CTO Chad Dickerson says he has a love/hate relationship with RSS. He loves the changes to his information production and consumption, but he hates the behavior of some RSS feed readers. Every hour, Infoworld "sees a massive surge of RSS newsreader activity" that "has all the characteristics of a distributed DoS attack." So many requests in such a short period of time are creating scaling issues. " We've seen similiar problems over the years. RSS (or as it should be called, "Speedfeed") is such a useful thing, it's unfortunate that it's ultimately just very stupid.
RSS may be ultimatly stupid but you didn't get first post did you! rookie!
Does this have anything to do with /. problems yesterday
-ravan_a
another article
Can't one just write a small php script or something which returns an error (i.e. 500), less data to send back, and hopefully the reader would just try again later.
The readers should HEAD to see if the last modified changed... And the feed rendering engines should make sure their last modified is accurate.
A programmer is a machine for converting coffee into code.
...so could someone recommend a couple of really good ones for Windows and *nix?
This is helpful.
Rhymes that keep their secrets will unfold behind the clouds.There upon the rainbow is the answer to a neverending story
I don't really care for RSS either, but damn, was that necessary?
We've seen similiar problems over the years. RSS (or as it should be called, "Speedfeed") is such a useful thing, it's unfortunate that it's ultimately just very stupid.
And it seems to have gotten worse since the new code was installed- I get 503 errors at the top of every hour now on slashdot.
SJW: a person who perceives an injustice, and while correcting it, commits a greater injustice.
Since many clients request the new data every 30 minutes or so... how about a simple system that spreads out the load? A page that, based on some criteria (domain name, IP, random seed, round robin) gives each client a time it should check for updates (i.e. 17 past the hour).
Of course, this depends on the client to respect the request, but we already have systems that do (robots.txt), and they seem to work fairly well, most of the time.
"Faith: Belief without evidence in what is told by one who speaks without knowledge, of things without parallel." - A.B.
RSS just needs better TCP stacks. Here's how it would work: when your RSS client connects to an RSS server, it would simply leave the connection open until the next time the RSS data got updated. Then you would receive a copy of the RSS content. You simply *couldn't* fetch data that hadn't been updated.
The reason this needs better TCP stacks is because every open connection is stored in kernel memory. That's not necessary. Once you have the connecting ip, port, and sequence number, those should go into a database, to be pulled out later when the content has been updated.
-russ
Don't piss off The Angry Economist
Well maybe somebody should set something up to syndicate RSS feeds via a peer to peer service. BitTorrent would work, but it could be improved upon (people would still be grabbing a torrent every hour, so it wouldn't completely solve the problem).
Or did the RSS reader authors hope that their applications wouldn't be used by anybody except for a few geeks?
...is what one would say to the designers of RSS.
Mainly, IF your client is smart enough to communicate that it only needs part of the page, guess what? The pages, especially after gzip compression(which, including with mod_gzip, can be done ahead of time)...the real overhead is all the nonsense, both on a protocol level and for the server in terms of CPU time, of opening+closing a TCP connection.
It's also the fault of the designers for not including strict rules as part of the standard for how frequently the client is allowed to check back, and, duh, the client shouldn't be user-configured to check at common times, like on the hour.
Bram figured this out with BitTorrent- the server can instruct the client on when it should next check back.
Please help metamoderate.
"Despite 'only' being XML, RSS is the driving force fulfilling the Web's original promise: making the Web useful in an exciting, real-time way."
Err, did I miss the meeting where that was declared as the Web's original promise?
Anyway, the trouble is pretty obvious: RSS is just a polling mechanism to do fakey Push. (Wired had an interesting retrospective on their infamous "PUSH IS THE FUTURE" hand cover about PointCast.) And that's expensive, the cyber equivalent of a hoarde of screaming children asking "Are we there yet? Are we there yet? How about now? Are we there yet now? Are we there yet?" It would be good if we had an equally widely used "true Push" standard, where remote clients would register as listeners, and then the server can actually publish new content to the remote sites. However, in today's heavily firewall'd internet, I dunno if that would work so well, especially for home users.
I dunno. I kind of admit to not really grokking RSS, for me, the presentation is too much of the total package. (Or maybe I'm bitter because the weird intraday format that emerged for my own site doesn't really lend itself to RSS-ification...)
SO YOU'RE GOING TO DIE: The Comic for Dealing with Death
Here's a solution: Have the RSS readers grab data every hour or half hour starting from when they are started up, not on the hour. This would of course distribute the "attacks" on the server.
Why not have rss readers that check on startup, then check again at user specified intervals.. After a random amount of time has past.
user starts program at 3.15 and it checks rss feed.
user sets check interval to 1 hour.
rand()%60 minutes later (let's say 37) it checks feed
every hour after that it checks the feed.
simplistic sure, but isn't rss in general?
on an aside, any of you (few) non-programmers interested in creating rss feeds, i put out some software that facilitates it.
hunterdavis.com/ssrss.html
And there is a one word solution, peer to peer. The whole torrent concept is what is needed.
The basic problem with RSS is that it's a "pull" method - RSS clients have to make periodic requests "just to see". Also, there's no effective way to mirror content.
That's just plain retarded.
What they *should* do...
1) Content should be pushed from the source, so only *necessary* traffic is generated. It should be encrypted with a certificate so that clients can be sure they're getting content from the "right" server.
2) Any RSS client should also be able to act as a server, NTP style. Because of the certificate used in #1, this could be done easily while still ensuring that the content came from the "real" source.
3) Subscription to the RSS feed could be done on a "hand-off" basis. In other words, a client makes a request to be added to the update pool on the root RSS server. It either accepts the request, or redirects the client to one its already set up clients. Whereupon the process starts all over again. The client requests subscription to the service, and the request is either accepted or deferred. Wash, rinse, repeat until the subscription is accepted.
The result of this would be a system that could scale to just about any size, easily.
Anybody want to write it? (Unfortunately, my time is TAPPED!)
I have no problem with your religion until you decide it's reason to deprive others of the truth.
I seem to remember Windows scheduler being able to randomize scheduled event times within a 1 hour period. I think our RSS feeders need similar functions.
--You will rephrase your request for me to go to hell. Goto statements are not acceptable programming constructs
RSSOwl - http://rssowl.sourceforge.net/ is pretty good.
$ strings FTP.EXE | grep Copyright
@(#) Copyright (c) 1983 The Regents of the University of California.
On Windows I use RSS Bandit. Haven't found a non-sucky one for *nix, although I haven't looked all that hard. On OS X I use NetNewsWire, which while not great, does the job.
My guess is that InfoWorld is dynamically generating the RSS for each request. A simple host-side cache of the generated XML, so hits just talk to the HTTP server and not the app server, would probably make this a non-issue.
Or are they *really* getting more RSS hits than image requests? If -- somehow -- that's the case, spend $500/mo on Akamai or Speedera and point RSS stuff there, and give the CDN a reasonable timeout (30 minutes or something). That guarantees you no more than about 500 hits per timeout period, or maybe one every 10 seconds. Surely the app server can handle that.
Then again, what do I know? I only worked there for five years, including two on infoworld.com. It's been a few years, but unless things have changed dramatically, that is one messed up IT organization.
Cheers
-b
If I wanted a sig I would have filled in that stupid box.
If you're using NetNewsWire on OS X, try the Atom Beta, which, I'm sure it will come as no shock to you, adds support for Atom feeds.
Complaining about people connecting to your RSS feeds "impolitely" is missing the mark a bit, I think. Even RSS readers that *do* check when the file was last changed, still download the entire feed when so much as a single character has changed.
There used to be a system where you could pull a list of recently posted articles off of a server that your ISP had installed locally, and only get the newest headers, and then decide which article bodies to retrieve.. The articles could even contain rich content, like HTML and binary files. And to top it off, articles posted by some-one across the globe were transmitted from ISP to ISP, spreading over the world like an expanding mesh.
They called this.. USENET..
I realize that RSS is "teh hotness" and Usenet is "old and busted", and that "push is dead" etc. But for Pete's sake, don't send a unicast protocol to do a multicast (even if it is at the application layer) protocol's job!
It would of course be great if there was a "cache" hierarchy on usenet. Newsgroups could be styled after content providers URLs (e.g. cache.com.cnn, cache.com.livejournal.somegoth) and you could just subscribe to crap that way. There's nothing magical about what RSS readers do that the underlying stuff has to be all RRS-y and HTTP-y..
For real push you could even send the RSS via SMTP, and you could use your ISPs outgoing mail server to multiply your bandwidth (i.e. BCC).
SCO employee? Check out the bounty
/. is especially pissy with this but I want breaking news, not whatever is new each hour. So I hammer the shit out of it with my client (and get banned). I'd like to see a service where I download one client (that has front-ends in Gnome pannel, the Windows tray, etc.) that the site (/., cspan, etc.) _pushes_ new updates to when I sign up. Those w/ dynamic IPs could, when they sign on, have their client automagically connect to a server that holds their unique user ID with their IP.
I haven't posted in so long, my sig is out of date.
Am I the only one who finds it easier to get the information I want from the home pages of the sites I trust, rather than relying on an RSS feed? For one thing, in an RSS feed every story has the same priority ... stories keep coming in and I have no idea which ones are "bigger" than others. Sites like News.com, on the other hand, follow the newspaper's example of printing the headlines for the more important stories bigger. With RSS, it's just information overload, especially with the same stories duplicated at different sources, etc. Everyone seems really excited about RSS, but when I tried it I just couldn't figure out how to use it such that it would actually give me some real value vs. the resources I already have.
Breakfast served all day!
we need RHS... really HARD syndication
That's nothing compared to RMS, which (according to RMS) stands for GNU/Recursive Meta-Syndication.
"Why Subscribe?" Good question...
I'm using Liferea version 0.5.1 under Linux right now. Compiles from source fine on Fedora Core 2 and has worked great for me so far.
bbh
Overall traffic isn't what anybody is complaining about- as I noted, the 503 errors seem to come at the top of every hour (I just got through not being able to read slashdot for a few minutes), which means, essentially, slashdot is recieving a slashdotting. Do I know that RSS is doing it? Not from this location which has limited investigation tools or capability to figure out what's really going on. But it might explain recent behavior of the site.
SJW: a person who perceives an injustice, and while correcting it, commits a greater injustice.
However, the smart money is on Murphy. :)
One line blog. I hear that they're called Twitters now.
That is mind bogglingly inefficient. Its like POP clients checking for new email every X minutes. Polling is wrong wrong wrong! Check out the select() libc call. Does the linux kernel go into a busy wait loop listening for every ethernet packet? no! it gets interrupted when a packet it ready!
http://www.mod-pubsub.org/
The apache module mod_pubsub might be a solution.
From the mod_pubsub FAQ:
What is mod_pubsub?
mod_pubsub is a set of libraries, tools, and scripts that enable publish and subscribe messaging over HTTP. mod_pubsub extends Apache by running within its mod_perl Web Server module.
What's the benefit of developing with mod_pubsub?
Real-time data delivery to and from Web Browsers without refreshing; without installing client-side software; and without Applets, ActiveX, or Plug-ins. This is useful for live portals and dashboards, and Web Browser notifications.
Jabber also saw a publish/subscribe mechanism as an important feature.
The folks over at Netscape and/or UserLand should have studied the CDF standard first. Then they would have realized the value of specifying schedule information.
I won't argue with those who have posted here that some alternative to the "pull" technology of RSS would be very useful. But...
The biggest problem I see isn't newsreaders but blogs. Somebody throws together a blog, inserts a little gizmo to display one of my feeds & then the page draws down the RSS every time the page is reloaded. Given the back-and-forth nature of a lot of folks' web browsing pattern, that means a single user might draw down one of my feeds 10-15 times in a 5 minute span. Now, why couldn't the blogger's software be set to load & cache a copy of the newsfeed according to a schedule?
The honorable mention for RSS abuse goes to the system administrator who set up a newreader screen saver that pulled one of my feeds. He then installed the screen saver on every PC in every office of his company. Every time the screen saver activated, POW! one feed drawn down...
"Obviously, I'm not an IBM computer any more than I'm an ashtray" (Bob Dylan)
Random intervals. I already patched my desktop RSS reader to request new feed every 73+-13 minutes.
There you are, staring at me again.
But isn't this what TCP/IP multicast was invented for? I've never really understood why multicast has never really taken off. Too complicated? Instead of entering an rss server to pull from just join a multicast group and have the RSS blasted once every X minutes. Servers could even send out updates more often because there are only a few connections to send to. Of course I could be completely wrong and multicast may be the absolute wrong choice for this sort of application, it's been a while since I've read any documentation about it.
(B) + (D) + (B) + (D) = (K) + (&)
I recommend PulpFiction for an RSS/Atom reader on OS X. I much prefer the interface and how it treats the news compared to NNW.
All editorial writers ever do is come down from the hill after the battle is over and shoot the wounded.
it seems a few peoples here dont get it. RSS is the file format, not the transfer via HTTP The whole pull problem is a problem with HTTP, in theory you could make an irc like protocol and transmit via that, solving some of the subscription, distribution and pull problems.
If you don't use one computer all the time and you want to check your feeds from other places, I'd recommend going with a web-based news-agreggation service. I personally use BlogLines, but there are other services out there as well.
The main problem here is that RSS lacks any sort of distributed flow control, much as the Internet did back in the early days with tons of UDP packets flying around everywhere and periodically bringing networks to their knees.
One completely backwards-compatible fashion to add flow-control to RSS would be to use the HTTP 503 response when server load is getting too high for your RSS files. The server simply sends an HTTP 503 response with a Retry-After header indicating how long the requesting client should wait before retrying.
Clients that ignore the retry interval or are overly aggressive could be punished by further 503 responses thus basically denying those aggressive clients access to the RSS feeds. Users of overly aggressive clients would soon find that they actually provide less fresh results and would place pressure on implementors to fix their implementations.
I think this was more or less the first thought I had about RSS when I first looked into it and found out that it was a "pull" technology rather than a "push" as the early descriptions of it implied.
Yes, it's "cool" that I can set up a page (or now use a browser plug-in) to automatically get a lot of content from hundreds of web pages at a time when I really opened up the browser to check my e-mail.
What would have REALLY, been cool would be some sort of technology that would notify me when something CHANGED. No effort on my part, no *needless* effort on the servers part.
Oh wait... We HAD that didn't we, I think they were called Listservers, and they worked just fine. (Still do actually as I get a number of updates, including Slashdot, that way.) RSS advocates (and I won't mention any names) keep making pronouncements like "e-mail s dead!" simply because they have gotten themselves and their hosting companies on some black hole lists. Cry me a river now that your bandwidth costs are going through the roof and yet nobody is clicking though on your web page ads, because, guess what? Nobody is visiting your page. They have all they need to know about your updates via your RSS feeds.
Neither Windows nor Unix, but I've set up Feed on Feeds on my webserver and I like it!
It's a "PHP/MySQL server side RSS/Atom aggregator", so you can read your feeds wherever you are, you only need a web browser on the client side.
Pros:
1) you don't need to synchronize the state between the multiple workstations you might use.
2) no platform/os problem on the client side.
Cons:
1) you need some web hosting with PHP and MySQL available (I pay 45 a year for my domain name + 30MB Webspace + 30MB FTP + 30MB MySQL base + 100*25MB pop/imap accounts + SSL everywhere).
2) no installer so you'll need many computing skills to set it up (no that hard).
3) no automated update, you have to click "Update" so you may miss some news when you offline (see away from any internet access) for a long period...
Changed my online life as I no longer have to install anything on the client side (usefull when away from your home/office) or have to synchronize my feeds either with some removable storage (my USB key failed after 250+ daily syncs) or through the net (BottomFeeder, a smalltalk implementation which works on every platform I ever came accross, allows to sync with an FTP location).
Regards,
Poulpy.
True story:
We ran a network operations center to provide support for several hundred servers spread over two continents. Each hour, every server would 'phone home' to see if it needed updates or configuration changes. This was a fairly data-heavy operation, requiring many database lookups. We knew that we didn't want every server calling at the same time, so we had each server derive its own random integer between 1 and 59, and to use that as the minute of the hour to contact the NOC.
Before long we found that the NOC was dragging itself into a death spiral of overwork. The problem? By chance, an unusually large number of servers chose a very small range of numbers. Worse, they just happened to choose numbers close to 05, which just happened to be when some very large cron tasks were running as well.
Try rolling a die 100 times. Even though the odds are the same every time before you roll, the actual frequency of occurence of the individual numbers is not even. Leaving the choice of retrieval time to the client does not reliably reduce the chance of a server being overwhelmed. In fact, it more or less guarantees traffic spikes.
I'm not intimately familiar with RSS client or server implementations, but I suspect that it would be fairly easy to format a suggested refresh interval and refresh time on the server and send that to the client.
Crumb's Corollary: Never bring a knife to a bun fight.
Even for a poll at hourly intervals this should get staggered across an given hour according to when the client starts. Also, a client should probably not be polling every 3600 seconds (or whatever interval) but polling with a 3600 second gap between end of one poll and start of the next. In this way a loaded server will smear the clients out simply by having slower response, and the load will even out on its own.
It's always bad to have lots of agents doing things in synchrony when that involves an outside resource. Contact the client authors, give them a clue, let the upgrades push the bugfix out.
Finally, isn't RSS done over HTTP anyway? So why aren't these clients going through their ISP's proxy and doing Get-If-Modified? The target server should see only a fraction of the spike even with bad clients. Unless they're very very bad...
None of these things is a direct flaw in RSS, just crap quality of implementation in RSS clients.
Cameron Simpson, DoD#743 cs@cskk.id.au http://www.cskk.ezoshosting.com/cs/
>On Windows I use RSS Bandit
Pronounced "ArseBandit"?
That's priceless, to a Brit at least.
.
They will never know the simple pleasure of a monkey knife fight