In his note, he says that RSS is broken. I personally believe that at issue is not whether RSS is working or not. RSS is working but it has complicated the bandwidth issue. At issue is the fact that RSS feeds are generally generating more traffic to a site. Because RSS readers are polling the site to check if a feed has been updated, the traffic patterns change, with increased numbers of spikes on a hourly basis. This is similar to some of the issues network administrators started facing when Pointcast first appeared.
There are a number of ways to mitigate the issue.
HTTP Conditional GET for RSS
First of all, one of the things to consider when using RSS is to create conditional HTTP headers on RSS feeds. This helps mitigate some of the impact by ensuring that feeds are only served if the content has changed.
Feed Compression
The next item to think of is to use compression when serving feeds. By doing so, one reduces the size of the payload, which ends up being much better in terms of managing bandwidth. In my own experience, because RSS is primarily text, I’ve seen a reduction of 80% of the bandwidth when delivering RSS feeds in a compressed format. That represents a fairly large gain in bandwidth that can then accommodate more users.
Change the polling schedule
The RSS 2.0 specification already offers a number of optional elements to give RSS readers a better idea as to when to get content. For example, the pubDate element offers information as to when a feed was last published, as does the lastBuildDate one. ttl (aka. time to live) can also be used to indicate to the software that this feed should live for a certain amount of time. Finally, skipHours and skipDays offers more pointers as to when RSS reader software should not poll. With all those mechanisms in place, it looks like a lot of flexibility exists in the format to accommodate scalability.
When all else fails, reduce
If all of the above still fail, RSS publishers should look at reducing the size of their feeds. There are two ways you can do this. First, you can just say that you’re not going to offer full-text feeds. This seems to be the option that Scoble hates. Another way to do things is to offer both abbreviated feeds and full-text feeds or offer more detailed feeds, as I do on TNL.net.
An important consideration when doing something like this is how to address them. By default, users who just use the RSS autodiscovery feature will only get the abbreviated feed. However, they still have the option to go and get the full-text version. The compromise here is that users who just want to subscribe quickly can do so at a lower bandwidth costs, while power users can seek out the fuller feed and subscribe to that. The result, in my experience, is that most people use the autodiscovery feature, grabbing the smaller feed. Some power users do seek out the fuller feed and subscribe to that instead (based on the numbers, I’m seeing a 5% usage of the full-text feed as opposed to the default abbreviated one. This is a compromise solution that seems to accommodate everyone involved to date.
Final considerations
When publishing RSS feeds, your audience grows, which results in traffic growth too. One of the thing to realize is that RSS feeds are generally stickier than the rest of a site. What this means is that, for every new subscriber you get, you will see an on-going increase in your overall site traffic stats. This is not a bad thing as messages emanating from your site do get a higher passive readership. One of the thing that new syndication standards should consider is a follow-up on this. While RSS publisher know how many feeds are being pushed out, there is little, in the way of information as to what percentage of those feeds is being read. Stronger metrics need to be developed to get an understanding of passive vs. active subscribers (passive subscribers are subscribers that receive the feed but do not read it, while active subscribers are actually reading the content and clicking through). This, I believe, is one of the next challenges that needs to be addressed in order to make RSS a more viable and widespread distribution platform.