Combined RSS feeds from multiple sources, or backup feeds

[Troetelbiertje]

I'm a big fan of the rss downloader, I use it to let µTorrent download my favorite tv shows every week. In order to do so I made a choice between one of the major rss feed suppliers (EZRSS, ShowRSS, dailytvtorrents) and set up the rss downloader for every one of the shows I follow. The problem is that all of these feeds, without exception, have suffered from downtime and probably will do so again. In which case I need to reconfigure µTorrent to use a different feed.

So here's my idea. Make it possible to combine the different rss feeds into one aggregate. That way when one of the sites is down, your show will still be downloaded since µTorrent is no longer dependent on one single feed.

So basically what I want to achieve is this:

Let's say I want to watch the new Terra Nova show, so I go to "Add RSS Feed"

Normally I would use http://showrss.karmorra.info/feeds/374.rss but now I would also add some backups e.g. http://ezrss.it/search/index.php?show_name=Terra+Nova&show_name_exact=true&mode=rss and http://www.dailytvtorrents.org/rss/show/terra-nova (Maybe comma separated or it might be better to add a few extra fields, I leave that for the usability experts)

What I want is that µTorrent aggregates the information from the different feeds and combines it into one perfect feed, discarding any possible overlap.

Alternatively if that's to hard to achieve or maybe simply not possible. I would still want to be able to add the different feeds like this, but let the second and subsequent feeds serve as backups. So if feed one is down µTorrent automatically calls upon feed two.

[magao]

There are several ways to achieve this. The easiest is to just subscribe to multiple feeds. What does it matter if you get duplicates? uTorrent will just ignore them!

Other than that, probably the easiest is to use Google Reader and Feedburner (this is what I do, as I get the benefit of GR's retention of feed items).

1. Add all the feeds to Google Reader.

2. Put all the feeds into a single folder/tag.

3. Share that folder (i.e. make it a public folder).

4. Subscribe to the public folder's atom feed via FeedBurner. Make sure you add a count to the feed (I'd suggest at least 100 - by default GR only returns 20 items).

5. In the "Optimise" section of Feedburner (for the feed):

a. Turn off "BrowserFriendly";

b. Use "Convert Format Burner" to produce RSS 2.0.

6. Subscribe to the Feedburner feed.

There are a few caveats with this:

- Google Reader updates can be a bit irregular. If there is more than 1 subscriber to the feed, and it's reasonably active (new items most of the time it's polled) GR will probably poll the feed every hour or so. However, there have been periods when the feeds are responding but GR for some reason decides not to poll more than 1/day or something. You can always go into GR and refresh the feed though.

- There will then be an additional delay from Feedburner - it will poll GR approximately every 30 minutes. It's possible to force FB to poll if needed as well ("pinging" the feed).

- Might as well set the uTorrent update interval to 1 hour, as you won't get updates faster than that.

- Feedburner allows a maximum of ~550KB when polling a feed. So you need to experiment to see how high a count you can put on your GR feed - it will depend on how much info is in each item.

- You will get duplicates. However, since uTorrent will ignore the duplicates (other than logging) that's not a problem. In fact, if the torrent files themselves are on multiple servers, the duplicates can be useful (redundancy).

As examples of feeds, I have set up a feed that aggregates links to torrents for various EZTV sources (scraping the web page, ezRSS, TPB and Twitter). It has minimal information in it (titles and torrent links only) and so I've been able to pull 500 items at a time through Feedburner.

The Google Reader public folder is (Atom - not usable in uTorrent):

http://www.google.com/reader/public/atom/user%2F01948784923688372624%2Flabel%2FEZTV

or with a count:

http://www.google.com/reader/public/atom/user%2F01948784923688372624%2Flabel%2FEZTV?n=100 (note the ?n=100 on the end).

The Feedburner feed (RSS 2.0):

http://feeds.feedburner.com/eztv-rss-atom-feeds

BTW, the scraping and sanitisation of the original feeds/web pages/twitter is all done via Google App Engine ... so the entire system (GAE, GR, FB) is running on Google's infrastructure.

[magao]

I should add that with a little bit of coding you could do this all in GAE, using the data storage to ensure you don't get any duplicates. Something like:

1. URL for retrieving feed(s).

2. Scheduled (cron) tasks to retrieve the feeds and update the datastore.

3. Retrieve the feed either with uTorrent, or go through the Google Reader/Feedburner dance.

You wouldn't be holding any of the actual torrents - just the links to them. However, it would essentially be an indexer then.

The advantage here of GR/FB is that uTorrent doesn't retrieve RSS feeds using gzip compression whilst GR does, so the amount of data you're pushing out of GAE is much higher if you go direct to uTorrent. However, if it's only 1 person retrieving it then it still wouldn't come anywhere close to using up your free quota ... maybe 1%.