This post originated from an RSS feed registered with .NET Buzz
by Frans Bouma.
Original Post: RSS Feeds at weblogs.asp.net should be static
Feed Title: Frans Bouma's blog
Feed URL: http://www.asp.net/err404.htm?aspxerrorpath=/fbouma/Rss.aspx
Feed Description: Generator.CreateCoolTool();
Phil Winstanley blogs about the very slow pace of the SqlServer(s) behind the blogs here at weblogs.asp.net. I agree, the blogs are very very slow (posting a comment takes ages and reading the main feed is often resulting in a timeout).
I think part of the deal is that the RSS feeds here are produced dynamically. This means that if there are 10,000 users pulling the main feed's RSS data every 10 minutes, the engine will every 10 minutes produce 10,000 versions of the RSS feed dynamically, consulting the database and other logic. For not that popular feeds, it's not that bad. However with a highly popular feed like weblogs.asp.net, it's not that efficient.
Now, imagine that the engine would produce every T minutes a new RSS feed file. The 10,000 requests for this file will only be fulfilled if the file is changed. If the file hasn't been changed, the webserver will return a HTTP 304 message. If the file has been changed, no database activity has to be performed: just send the bits. Scott told me that due to some clustering issues this is not yet possible here, but perhaps that can be (and has to be) changed.
Using a fixed time window of T minutes, you can produce content which is always producible within a fixed time period smaller than T (otherwise you have to set T higher). This is the mechanism used by high traffic sites like Slashdot for years: every T minutes the complete content is generated. If something changes in that window, you can decide to generate the content dynamically into a cache (a comment for example) or wait for the next time window. For the main feed(s) for example, you could generate every 30 minutes a new file or when a new blog is posted.
That would greatly reduce the access to the database server, which seems the bottleneck at the moment.
Previous Topic
